A Hierarchical Model for Continuous Gesture Recognition Using Kinect

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

This paper proposes a framework of sign language to emotional speech conversion based on deep learning to solve communication disorders between people with language barriers and healthy people. We firstly trained a gesture recognition model and a facial expression recognition model by a deep convolutional generative adversarial network (DCGAN). Then we trained an emotional speech acoustic model with a hybrid long short-term memory (LSTM). We select the initials and the finals of Mandarin as the emotional speech synthesis units to train a speaker-independent average voice model (AVM). The speaker adaptation is applied to train a speaker-dependent hybrid LST-M model with one target speaker emotional corpus from AVM. Finally, we combine the gesture recognition model and facial expression recognition model with the emotional speech synthesis model to realize the sign language to emotional speech conversion. The experiments show that the recognition rate of gesture recognition is 93.96%, and the recognition rate of facial expression recognition in the CK+ database is 96.01%. The converted emotional speech not only has high quality but also can accurately express the facial expression.

The use of hand-based gestures has been proposed as an intuitive way for people to communicate with robots. Typically the set of gestures is defined by the experimenter. However, existing works do not necessarily focus on gestures that are communicative, and it is unclear whether the selected gesture are actually intuitive to users. This paper investigates whether different people inherently use similar gestures to convey the same commands to robots, and how teaching of gestures when collecting demonstrations for training recognizers can improve resulting accuracy. We conducted this work in two stages. In Stage 1, we conducted an online user study (n=190) to investigate if people use similar gestures to communicate the same set of given commands to a robot when no guidance or training was given. Results revealed large variations in the gestures used among individuals With the absences of training. Training a gesture recognizer using this dataset resulted in an accuracy of around 20%. In response to this, Stage 2 involved proposing a common set of gestures for the commands. We taught these gestures through demonstrations and collected ~7500 videos of gestures from study participants to train another gesture recognition model. Initial results showed improved accuracy but a number of gestures had high confusion rates. Refining our gesture set and recognition model by removing those gestures, We achieved an final accuracy of 84.1±2.4%. We integrated the gesture recognition model into the ROS framework and demonstrated a use case, where a person commands a robot to perform a pick and place task using the gesture set.

We present a novel technique for the recognition of complex human gestures for video annotation using accelerometers and the hidden Markov model. Our extension to the standard hidden Markov model allows us to consider gestures at different levels of abstraction through a hierarchy of hidden states. Accelerometers in the form of wrist bands are attached to humans performing intentional gestures, such as umpires in sports. Video annotation is then performed by populating the video with time stamps indicating significant events, where a particular gesture occurs. The novelty of the technique lies in the development of a probabilistic hierarchical framework for complex gesture recognition and the use of accelerometers to extract gestures and significant events for video annotation.

CG-Recognizer: A biosignal-based continuous gesture recognition system

Gesture recognition is a key research field in the human-computer interaction. At present, most of researchers focus on one-handed gesture recognition, but do not pay much attention to bimanual (two hands) gesture recognition. This paper presents a deep learning-based solution to tackle the self-occlusion and self-similarity. To solve this problem, this paper uses Kinect to collect many colour and depth images of different gestures, and each gesture contains multiple sample individuals. Colour images and depth images are used to train the recognition model of bimanual gesture respectively, and then the colour image and depth image are fused, and the bimanual gesture recognition model is trained based on colour image and depth image fusion. Then, the bimanual recognition effects of the three models are compared. The experimental results show that, regardless of the single gesture precision or the mean average precision, the bimanual gesture recognition effect of the fused model is better than the gesture recognition models based on either colour image or depth image.

Body gesture Recognition has been one of the interested research field for Human-Robot Interaction(HRI). Most of the conventional body gesture recognition algorithms used Hidden Markov Model(HMM) for modeling gestures which have spatio-temporal variabilities. However, HMM-based algorithms have difficulties excluding meaningless gestures. Besides, it is necessary for conventional body gesture recognition algorithms to perform gesture segmentation first, then sends the extracted gesture to the HMM for gesture recognition. This separated system causes time delay between two continuing gestures to be recognized, and it makes the system inappropriate for continuous gesture recognition. To overcome these two limitations, this paper suggests primitive body model encoding, which performs spatio/temporal quantization of motions from human body model and encodes them into predefined primitive codes for each link of a body model, and Selective/Asynchronous Input-Parallel State machine(SAI-PSM) for multiple-simultaneous gesture recognition. The experimental results showed that the proposed gesture recognition system using primitive body model encoding and SAI-PSM can exclude meaningless gestures well from the continuous body model data, while performing multiple-simultaneous gesture recognition without losing recognition rates compared to the previous HMM-based work.

In order to incorporate naturalness in the design of human computer interfaces (HCI), it is desirable to develop recognition techniques capable of handling continuous natural gesture and speech inputs. Though many different researchers have reported high recognition rates for gesture recognition using hidden Markov models (HMM), the gestures used are mostly pre-defined and are bound with syntactical and grammatical constraints. But natural gestures do not string together in syntactical bindings. Moreover, strict classification of natural gestures is not feasible. We have examined hand gestures made in a very natural domain, that of a weather person narrating in front of a weather map. The gestures made by the weather person are embedded in a narration. This provides us with abundant data from an uncontrolled environment to study the interaction between speech and gesture in the context of a display. We hypothesize that this domain is very similar to that of a natural human-computer interface. We present an HMM architecture for continuous gesture recognition framework and keyword spotting. To explore the relation between gesture and speech, we conducted a statistical co-occurrence analysis of different gestures with a selected set of spoken keywords. We then demonstrate how this co-occurrence analysis can be exploited to improve the performance of continuous gesture recognition.

The ability to recognize and classify human gestures bears a huge potential of improving haptic feedback in virtual environments: when it is possible to predict how a user wants to explore or manipulate his environment both the visual and the auditive feedback may be adapted in order to enhance the immersiveness of haptic displays. Within this work a software approach is suggested that allows for such a real-time classification of gestures in a continuous data stream. A visually based tracking system is used to record the hand movements, while Hidden Markov models are applied to analyze the data. After presenting the methodological background and the software implementation, the outcome of an evaluation study is discussed. It reveals satisfying results of gesture classification and, what is particularly important, the recognition is fast enough to enable multi-modal haptic feedback.KeywordsGesture recognitionHidden Markov modelMulti-modal haptic feedbackVirtual reality

In this article, a real-time dynamic finger gesture recognition using a soft sensor embedded data glove is presented, which measures the metacarpophalangeal (MCP) and proximal interphalangeal (PIP) joint angles of five fingers. In the gesture recognition field, a challenging problem is that of separating meaningful dynamic gestures from a continuous data stream. Unconscious hand motions or sudden tremors, which can easily lead to segmentation ambiguity, makes this problem difficult. Furthermore, the hand shapes and speeds of users differ when performing the same dynamic gesture, and even those made by one user often vary. To solve the problem of separating meaningful dynamic gestures, we propose a deep learning-based gesture spotting algorithm that detects the start/end of a gesture sequence in a continuous data stream. The gesture spotting algorithm takes window data and estimates a scalar value named gesture progress sequence (GPS). GPS is a quantity that represents gesture progress. Moreover, to solve the gesture variation problem, we propose a sequence simplification algorithm and a deep learning-based gesture recognition algorithm. The proposed three algorithms (gesture spotting algorithm, sequence simplification algorithm, and gesture recognition algorithm) are unified into the real-time gesture recognition system and the system was tested with 11 dynamic finger gestures in real-time. The proposed system took only 6 ms to estimate a GPS and no more than 12 ms to recognize the completed gesture in real-time.

A hidden Markov model (HMM) has been widely used as a cost-effective solution for time series based applications such as hand gesture recognition. A large number of parameters in an HMM often result in a degradation of classification accuracy due to overfitting. Furthermore, the accuracy is often affected by the initialization values of the parameters. Left-right banded (LRB) HMM is one of the representative models which improve the classification accuracy by reducing the number of parameters. However, LRB HMM still suffers from false negative misclassification when it is applied for hand gesture recognition. Misclassification in hand gesture recognition arise from similar trajectories of multiple gestures that are represented by hidden states. This paper proposes a new initialization process that reduces not only state transition parameters but also hidden state observation parameters. To this end, the gesture trajectory is predicted and then the parameters are masked out when their probabilities are negligibly small in the predicted trajectory. Experiment results show that the confusion rate is nearly 14% lower than the LRB HMM while maintaining the classification accuracy.

With the rapid development of human-computer interaction technology, gesture recognition becomes one of the key technologies of human-computer interaction. In this paper, we propose a new method of dynamic hand gestures recognition. The method adopts the hierarchical identification model for dynamic hand gestures recognition. First, we combine frame fusion with density distribution features for rough gesture recognition, second, we use the hausdorff distance or fingertip detection for accurate gesture recognition. The main innovation of this method lies in that we change the way of dynamic gestures recognition into the recognition of static image, improves the efficiency of gesture recognition effectively. Experimental results showed that our recognition rate is above 90%.

The main work of human motion gesture recognition is to recognize and analyze the behavior of human objects in the video. Although the current research in the field of human motion gesture recognition has achieved certain results, the human motion gesture recognition in real life scenes has great effects due to factors such as camera movement, target scale transformation, dynamic background, viewing angle, and illumination. This article first proposes a new method of constructing human motion posture features to describe human behavior. This method is based on deep convolutional neural network features and topic models. Experiments have verified that compared with the traditional feature map extracted from the convolutional neural network fully connected layer, the feature map extracted from the convolutional neural network convolutional layer is not only lower in dimension but also has higher discrimination. Secondly, based on the feature map of the convolutional neural network, the training map downsampling strategy is used to overcome the interference caused by the object's scale change and shape change. Finally, based on the basketball gesture recognition method, the behavior performance of the legs and arms in 9 basketball actions of walking, running, jumping, standing dribbling, walking dribbling, running dribbling, shooting, passing and receiving is analyzed. As well as the corresponding signal waveform characteristics, a two-stage data division method for basketball is proposed. The unit action data is extracted for analysis to realize feature extraction. In order to select the most suitable classifier for basketball gesture recognition, the constructed feature vector uses four Different classifiers are trained to construct different classifiers to realize the division of actions.

With the constant development of smart devices, Gesture recognition is applied on more and more fields. Gesture recognition devices currently on the market are inconvenient and expensive. A gesture recognition method based on Kinect is proposed in this paper. The camera of Kinect is used to get gesture images and then a hidden Markov model is established to recognize dynamic gesture so that the operation is simple. The assessment, decoding algorithms and training issues of hidden Markov model in gesture recognition is focused in this paper. Viterbi algorithm and Baum-Welch algorithm are used to solve the problems above. After simulation experiments, it is discovered that the gesture recognition algorithm based on Kinect proposed in this paper has a high accuracy rate for a variety of gestures, simultaneously the naturalness of human-computer interaction has been greatly improved.

Eye gaze and gesture form key conversational grounding cues that are used extensively in face-to-face interaction among people. To accurately recognize visual feedback during interaction, people often use contextual knowledge from previous and current events to anticipate when feedback is most likely to occur. In this paper, we investigate how dialog context from an embodied conversational agent (ECA) can improve visual recognition of eye gestures. We propose a new framework for contextual recognition based on Latent-Dynamic Conditional Random Field (LDCRF) models to learn the sub-structure and external dynamics of contextual cues. Our experiments show that adding contextual information improves visual recognition of eye gestures and demonstrate that the LDCRF model for context-based recognition of gaze aversion gestures outperforms Support Vector Machines, Hidden Markov Models, and Conditional Random Fields.KeywordsContextual informationConditional Random FieldsEye gesture recognitiongaze aversion