Graphical Models for real-time capable gesture recognition

This paper presents head and hand gestures recognition system for Human Computer Interaction (HCI). Head and Hand gestures are an important modality for human computer interaction. Vision based recognition system can give computers the capability of understanding and responding to the hand and head gestures. The aim of this paper is the proposal of real time vision system for its application within a multimedia interaction environment. This recognition system consists of four modules, i.e. capturing the image, image extraction, pattern matching and command determination. If hand and head gestures are shown in front of the camera, hardware will perform respective action. Gestures are matched with the stored database of gestures using pattern matching. Corresponding to matched gesture, the hardware is moved in left, right, forward and backward directions. An algorithm for optimizing connected component in gesture recognition is proposed, which makes use of segmentation in two images. Connected component algorithm scans an image and group its pixels into component based on pixel connectivity i.e. all pixels in connected component share similar pixel intensity values and are in some way connected with each other. Once all groups have been determined, each pixel is labeled with a color according to component it was assigned to.

Human communication relies on a large number of different communication mechanisms like spoken language, facial expressions, or gestures. Facial expressions and gestures are one of the main nonverbal communication mechanisms and pass large amounts of information between human dialog partners. Therefore, to allow for intuitive human-machine interaction, a real-time capable processing and recognition of facial expressions, hand and head gestures are of great importance. We present a system that is tackling these challenges. The input features for the dynamic head gestures and facial expressions are obtained from a sophisticated three-dimensional model, which is fitted to the user in a real-time capable manner. Applying this model different kinds of information are extracted from the image data and afterwards handed over to a real-time capable data-transferring framework, the so-called Real-Time DataBase (RTDB). In addition to the head and facial-related features, also low-level image features regarding the human hand - optical flow, Hu-moments are stored into the RTDB for the evaluation process of hand gestures. In general, the input of a single camera is sufficient for the parallel evaluation of the different gestures and facial expressions. The real-time capable recognition of the dynamic hand and head gestures are performed via different Hidden Markov Models, which have proven to be a quick and real-time capable classification method. On the other hand, for the facial expressions classical decision trees or more sophisticated support vector machines are used for the classification process. These obtained results of the classification processes are again handed over to the RTDB, where other processes (like a Dialog Management Unit) can easily access them without any blocking effects. In addition, an adjustable amount of history can be stored by the RTDB buffer unit.

To naturally interact with virtual environment by hand gesture, this paper presents a robust RGB-D data based recognition method of static and dynamic hand gesture. Firstly, for static hand gesture recognition, starting from the hand gesture contour extraction, the palm center is identified by Distance Transform (DT) algorithm. The fingertips are localized by employing the K-Curvature-Convex Defects Detection algorithm (K-CCD). On the basis, the distances of the pixels on hand gesture contour to palm center and the angle between fingertips are considered as the auxiliary features to construct a multimodal feature vector, and then recognition algorithm is presented to robustly recognize the static hand gestures. Secondly, combining Euclidean distance between hand joints and shoulder center joint with the modulus ratios of skeleton features, this paper generates a unifying feature descriptor for each dynamic hand gesture and proposes an improved dynamic time warping (IDTW) algorithm to obtain recognition results of dynamic hand gestures. Finally, we conduct extensive experiments to test and verify the static and dynamic hand gesture recognition algorithm and realize a low-cost real-time application of natural interaction with virtual environment by hand gestures.

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

With rapid developments in biometric recognition, a great deal of attention is being paid to robots which interact smartly with humans and communicate certain types of biometrical information. Such human–machine interaction (HMI), also well-known as human–robot interaction (HRI), will, in the future, prove an important development when it comes to automotive manufacturing applications. Currently, hand gesture recognition-based HRI designs are being practically used in various areas of automotive manufacturing, assembly lines, supply chains, and collaborative inspection. However, very few studies are focused on material-handling robot interactions combined with hand gesture communication of the operator. The current work develops a depth sensor-based dynamic hand gesture recognition scheme for continuous-time operations with material-handling robots. The proposed approach properly employs the Kinect depth sensor to extract features of Hu moment invariants from depth data, through which feature-based template match hand gesture recognition is developed. In order to construct continuous-time robot operations using dynamic hand gestures with concatenations of a series of hand gesture actions, the wake-up reminder scheme using fingertip detection calculations is established to accurately denote the starting, ending, and switching timestamps of a series of gesture actions. To be able to perform typical template match on continuous-time dynamic hand gesture recognition with the ability of real-time recognition, representative frame estimates using centroid, middle, and middle-region voting approaches are also presented and combined with template match computations. Experimental results show that, in certain continuous-time periods, the proposed complete hand gesture recognition framework can provide a smooth operation for the material-handling robot when compared with robots controlled using only extractions of full frames; presented representative frames estimated by middle-region voting will maintain fast computations and still reach the competitive recognition accuracy of 90.8%. The method proposed in this study can facilitate the smart assembly line and human–robot collaborations in automotive manufacturing.

In this work we present the results of a comparative user study of two radically different user interfaces for controlling infotainment applications in an automotive environment: a standard 2D interface and an innovative 3D interface. Based on a generic multimodal architecture, both systems can be operated by conventional key-console and touch-screen as well as by natural speech and dynamic hand and head gestures. Inspired by the advantages of classical Virtual-Reality interfaces, the idea of the new 3D design approach is to increase the overall usability when interacting with complex in-car information systems by applying pleasing display patterns. In a series of usability experiments we have evaluated both interfaces with regard to typical operation tasks in a simulated driving scenario. Thereby, we have found out that none of the two interface alternatives has clearly been preferred in general, but concerning selected design elements and the joy of use, the 3D visual front-end obtained significantly better ratings.

In this paper, we tackle advantages of cyclical movement patterns of hand gestures. The cyclical patterns are defined as closed-form which hand moves away from a rest position, follows one or more of a series of the movement of hand shapes and returns to its rest position. Due to the cyclical pattern characteristic, phase of gestures are supportive cues for deploying robust recognition schemes. We conduct a spatial-temporal representation of the hand gestures which takes into account both hand shapes and its movements during a gesture. The phase alignment then is deployed in the conducted space. The proposed scheme ensures inter-period phase continuity as well as normalizes length of the hand gestures. Three different datasets of dynamic hand gestures consisting of non-cyclical and cyclical patterns are examined. Evaluation results confirm that the best accuracy rate achieves at 96% for cyclical pattern that is significantly higher than results for typical gestures. The proposed method suggests a feasible and robust solution addressing technical issues in developing human-computer interaction applications such as using hand gestures to control home appliance devices.

This paper presents a hand gesture interaction system in virtual environment, in which mixed means static and dynamic hand gestures are combined for both navigation and object manipulation. Firstly, a simple average background model and skin color are used for hand area segmentation. Then a state-based spotting algorithm is employed to automatically identify two types of hand gestures. A voting-based method is used for quick classification of static gestures. And we use the hidden Markov model (HMM) to recognize dynamic gestures. Since the training of HMM requires the consistency of the training data, outputted by the feature extraction, a data aligning algorithm is raised. Through our hand gesture system, users can perform complicated operating commands in a natural way. The experimental results demonstrate that our methods are effective and accurate.

The research in the dynamic hand gestures recognition field has been promoted thanks to the recent success of the emerging Leap Motion controller (LMC) sensor, especially in the human-computer interaction field. During surgery, surgeons need to make more effective use of computers through a touchless human-computer interaction. In this paper, we present a set of gestures that are about commands to manipulate medical images. The public availability of hand gesture datasets for dynamic hand gestures, in particular with the LMC as an acquisition device, is currently quite poor. Therefore, we collect a more challenging dataset, called “LeapGestureDB”. The dataset is presented as text files, which contains 6,600 samples of 11 different dynamic hand gestures performed by over 120 participants. It is hosted online for the public interest, and especially to enhance research results. We suggest an approach of three-dimensional dynamic hand gesture recognition. We extract spatial, frequential and spatio-frequential features from the three-dimensional positions of the fingertips and the palm center delivered from the input sensor. These descriptors are fed into a support vector machine classifier to determine the performed gestures. We first evaluate our work on an existing dataset, namely “RIT dataset”, and then we use our collected dataset. We achieved a better accuracy rate of 93% comparing to results obtained from the “RIT dataset”.

The use of physical devices like mouse and keyboard to communicate to our computer has hindered the natural interface between the computer and its user. To eradicate this barrier, we have designed a virtual mouse which will act upon the various hand gestures the user provides and will help them interact with the computer in a better way. The virtual mouse control of our system will keep a track of the movements of the fingers and palm, thus providing us with a specific output based on the recognized action of our hand. The proposed model will recognize both static as well as dynamic hand gestures and the results will show that the model that has been proposed and built will intuitively interact with the computer and it can be achieved the minimum usage of any hardware requirements. Our model will also have a major advantage over the current existing models, which is, it also has the feature of voice command recognition, meaning that even if the user if not able to provide appropriate gestures using their hand due to some medical condition, they will still be able to interact with the computer via voice commands.

In this digital information explosion era, more and more products are proposed to enhance the quality and convenience of our life. However, vision-based hand gesture recognition is still a challenging problem to overcome. In this paper, an architecture system was proposed with dual-camera to construct the depth map and recognize dynamic hand gesture. It contains one static and two dynamic hand gestures. This design is implemented in SMIMS development board to verify and demonstrate the whole system.

People use gestures to communitcate with other people without difficutlty. As a method of computer/human interaction, gestures are more intuitive and more direct than existing input devices, such as keyboards, mice, etc., and futhermore, are more functional. Using hand gestures, which are important human gestures, is a key technique to build a friendly user interface.Some gestures are considered to be simbols, such as "OK" gesture, American sign language. Others have numerical meaning, i.e. how long, how large, how fast. etc. To deal with both kinds of hand gestures, we propose a hand shape representation method.The method represents a hand shape with 15 dimensional data of 5 finger tip positions in 3D. To implement both static and dynamic hand gestures in a graphic object manipulation system, at each measurement time the current hand shape is classified into 10 classes of hand shapes using subspace method and the hand movement is evaluated using recurrnet neural network.The system allows a user to manipulate graphic objects intuitively and directly.

The integration of more and more functionality into the human machine interface (HMI) of vehicles increases the complexity of device handling. Thus optimal use of different human sensory channels is an approach to simplify the interaction with in-car devices. This way the user convenience increases as much as distraction may decrease. In this paper a video based real-time hand gesture recognition system for in-car use is presented. It was developed in course of extensive usability studies. In combination with a gesture optimized HMI it allows intuitive and effective operation of a variety of in-car multimedia and infotainment devices with hand poses and dynamic hand gestures.

A comparative study of two state-of-the-art sequence processing techniques for hand gesture recognition

Human-Robot Interactions have to be more human like and human-human communications are multimodal interactions. In general communication, humans tend to use multiple modalities at a time to convey a message through. Multimodal interactions could be in many modalities such as gestures, speech, gaze and etc. Major multimodal human-human combination is the speech-hand gesture interaction. Hand gestures are used in a diverse range in these interactions. They add different meanings and enhance the understanding of the complete interaction in multiple dimensions. Purpose of this paper is to conduct a comprehensive analysis on multimodal relationship of speech-hand gesture interaction and its effect on the true meaning of interactions. Therefore this paper will focus on different aspects of each modality with regards to multimodal interactions such as vocal uncertainties, static and dynamic hand gestures, deictic, redundant and unintentional gestures, their timeline parameters, hand features and etc. Furthermore this paper discuss the effect of each speech-gesture parameter on understanding of the vocal ambiguities. Complete analysis of these aspects was conducted through detailed human study and results are interpreted through above multimodal aspects. Further vocal commands are analyzed using different vocal categories and different types of uncertainties. Hand gestures are analyzed though timeline parameters and hand feature analysis. For the timeline analysis, parameters were decided from the feedback of the participants on effectiveness of each parameter. Lag time, speed of the gesture movements and range of the gesture were considered for the timeline analysis.