Hand gesture spotting and recognition using hidden Markov models

Abstract

Advancements in computer vision techniques have made it possible to use hand gestures for human-computer interaction. Consequently, hand gesture recognition systems have received a lot of attention in recent years, where they have been applied in a number of applications. The main concern is to make human-computer interaction as natural and interactive as possible through the use of hand gestures, and focus on obtaining highly accurate and efficient systems. This research work presents a dynamic hand gesture spotting and recognition system which is able to cater for both isolated gestures and multiple gestures in continuous sequences. A hand segmentation technique which is based on depth maps is implemented to determine the location of the hand as it moves in time. The gesture is then fully represented by its corresponding 2D trajectory, where orientation, location and velocity features are used to describe the gesture. In order to recognize the particular gesture, Hidden Markov Models are used. For concatenated gestures, an extra procedure is required to segment out the key gestures from the video sequence and eliminate any transitional non-gesture movements which occur between gestures. This problem is also known as gesture spotting. For this reason, a non-gesture model which does not make use of any training data is developed to model transitional movements. Furthermore, a forward spotting scheme which uses a sliding window approach is implemented. The start and end points of key gestures are determined by considering the likelihood values produced by the gesture and non-gesture models for the observation symbols within the sliding window along the continuous sequence. In order to reduce the number of states of the non-gesture model, a relative entropy technique is used. Furthermore, for accurate determination of the end point in continuous sequences, a heuristic orientation backtracking technique is proposed and implemented in this work. To demonstrate the effectiveness of the system, a set of ten gestures, which represent typical commands used to control a TV system, are defi ned. A comprehensive dataset that includes both isolated and concatenated gestures is developed to train and test the models. For isolated gestures, a 100% recognition rate was obtained using an orientation feature and ten states for each gesture model. Experimental results on concatenated gestures showed that some gestures are more susceptible to recognition errors due to ambiguities between them. Using the pro- posed orientation backtracking technique and a sliding window of length 40, the recognition rate of concatenated gestures was increased from 79.4% to 87.8% and the reliability was increased from 76.35% to 82.99%. Comparisons with existing state-of-the-art solutions indicate that the results obtained are highly satisfactory.