Designs of human–robot interaction using depth sensor-based hand gesture communication for smart material-handling robot operations

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

To naturally interact with virtual environment by hand gesture, this paper presents a robust RGB-D data based recognition method of static and dynamic hand gesture. Firstly, for static hand gesture recognition, starting from the hand gesture contour extraction, the palm center is identified by Distance Transform (DT) algorithm. The fingertips are localized by employing the K-Curvature-Convex Defects Detection algorithm (K-CCD). On the basis, the distances of the pixels on hand gesture contour to palm center and the angle between fingertips are considered as the auxiliary features to construct a multimodal feature vector, and then recognition algorithm is presented to robustly recognize the static hand gestures. Secondly, combining Euclidean distance between hand joints and shoulder center joint with the modulus ratios of skeleton features, this paper generates a unifying feature descriptor for each dynamic hand gesture and proposes an improved dynamic time warping (IDTW) algorithm to obtain recognition results of dynamic hand gestures. Finally, we conduct extensive experiments to test and verify the static and dynamic hand gesture recognition algorithm and realize a low-cost real-time application of natural interaction with virtual environment by hand gestures.

Recently, human - machine interaction (HMI) becomes a hot research topic because of its wide applications, ranging from automatic device control to designing and development of assistant robot or even smart building at sparser scale. One of the most important questions in this research field is finding out an efficient and natural method of HMI. Among several channels of communication, hand gestures have been shown to be an intuitive and efficient mean to express an idea or to control something. For a successful hand gesture based interaction between human and robot, a vocabulary of hand gestures needs to be defined. To resolve this problem, in this paper, we propose a framework to study the behavior of Vietnamese in using of hand gesture in communication with robot. This study allows designing a hand gesture vocabulary for human - robot interaction (HRI) applications. In the literature, there are no works similar to ours. This makes our twofold contributions: (1) a proposed framework for hand gesture recognition: hand gesture vocabulary design, feature extraction for hand posture representation, hand posture classification, and hand gesture database construction; (2) some experimentations are realized to evaluate the defined hand gesture set that can be used in general situation of HRI. The experiment results show that the defined hand gesture set satisfies the both criteria: intuitiveness and recognisability.

Considerable effort has been put toward the development of intelligent and natural interfaces between users and computer systems. In line with this endeavor, several modes of information (e.g., visual, audio, and pen) that are used either individually or in combination have been proposed. The use of gestures to convey information is an important part of human communication. Hand gesture recognition is widely used in many applications, such as in computer games, machinery control (e.g., crane), and thorough mouse replacement. Computer recognition of hand gestures may provide a natural computer interface that allows people to point at or to rotate a computer-aided design model by rotating their hands. Hand gestures can be classified into two categories: static and dynamic. The use of hand gestures as a natural interface serves as a motivating force for research on gesture taxonomy, its representations, and recognition techniques. This paper summarizes the surveys carried out in human--computer interaction (HCI) studies and focuses on different application domains that use hand gestures for efficient interaction. This exploratory survey aims to provide a progress report on static and dynamic hand gesture recognition (i.e., gesture taxonomies, representations, and recognition techniques) in HCI and to identify future directions on this topic.

Gesture recognition as a natural, convenient and recognizable way has been received more and more attention on human–machine interaction (HMI) recently. However, visual-based gesture recognition methods are often restricted by environments and classical wearable device-based strategies are suffered from relatively low accuracy or the complicated structures. In this study, we first design a low-cost and efficient data glove with simple hardware structure to capture finger movement and bending simultaneously. Second, a novel dynamic hand gesture recognition algorithm (DGDL-GR) is proposed to recognize human dynamic sign language, in which a fusion model of convolutional neural network (fCNN) and generic temporal convolutional network (TCN) is fully utilized. The fCNN (fusion of 1-D CNN and 2-D CNN) is proposed to extract time-domain features of finger resistance movement and spatial domain features of finger resistance bending simultaneously. Moreover, due to the superiorities of TCN in sequence modeling task, this work proposes a novel hand gesture recognition method based on the TCN, which includes causal convolution, dilation convolution, and a residual network with appropriate layers. Both long- and short-time dependencies of the hand gesture features are deeply mined and classified in the end. Results of extensive experiments have demonstrated that the proposed DGDL-GR algorithm outperforms many state-of-the-art algorithms on the measure of accuracy, F1 score, precision score, and recall score with the real-world dataset. Moreover, the number of residual blocks and some key hyperparameters of the proposed DGDL-GR algorithm has been studied thoroughly in this work.

In this paper, we tackle advantages of cyclical movement patterns of hand gestures. The cyclical patterns are defined as closed-form which hand moves away from a rest position, follows one or more of a series of the movement of hand shapes and returns to its rest position. Due to the cyclical pattern characteristic, phase of gestures are supportive cues for deploying robust recognition schemes. We conduct a spatial-temporal representation of the hand gestures which takes into account both hand shapes and its movements during a gesture. The phase alignment then is deployed in the conducted space. The proposed scheme ensures inter-period phase continuity as well as normalizes length of the hand gestures. Three different datasets of dynamic hand gestures consisting of non-cyclical and cyclical patterns are examined. Evaluation results confirm that the best accuracy rate achieves at 96% for cyclical pattern that is significantly higher than results for typical gestures. The proposed method suggests a feasible and robust solution addressing technical issues in developing human-computer interaction applications such as using hand gestures to control home appliance devices.

Over the past 2 decades, vision-based dynamic hand gesture recognition (HGR) has made significant progresses and been widely adopted in many practical applications. Although the advent of RGB-D cameras and deep learning-based methods provides more feasible solutions for HGR, it is still very challenging to satisfy the requirements of both high efficiency and accuracy for real-world HGR systems. In this paper, we propose a novel method using the feature covariance matrix for effective and efficient dynamic HGR. We extract a set of local feature vectors that represent local motion patterns to construct the feature covariance matrix efficiently, which also provides a compact representation of a dynamic hand gesture. By tracking hand keypoints in three successive frames and calculating their motion features, our method can be extended to both 2D dynamic HGR and 3D dynamic HGR. To evaluate the effectiveness of the proposed framework, we perform extensive experiments on three publicly available datasets (one 2D dataset and two 3D datasets). The experimental results demonstrate the effectiveness of our proposed method.

In this project we've discussed a system which uses some dynamic hand gesture recognition technique to manage the media players like VLC media player. This project consist of certain modules which segments the foreground a part of the frame using detection of skin colour and approximate median technique. the popularity of hand gesture is finished by creating a call Tree, that uses certain features extracted from the segmented part. This hand gesture recognition technique introduces a replacement, natural thanks to interact with computers and is beneficial to several folks in our day-to-day life. Hand gestures are associated with the human hands in hand gesture recognition. This project is employed for controlling certain operations (Pause, Play, Volume up, Volume Down, Mute) on video player by mere hand gestures without getting in the rigmarole of pressing buttons or tapping onto the screen. this may be directly associated with our day to life like in presentations. We have considered hand gestures and their directional motion defines a gesture for the applying. during this application image retrieving is finished employing a Webcam. Some functions in VLC media players are used most often and these functions uses some predefined gestures. Shows the defined gestures in keeping with the VLC player control operation. We created a VLC Media Player Controller using Hand Gesture Recognition System to form ‘HUMAN LIFE EASY AND BETTER’. This project is implemented using two steps: (1.) Creation of Hand Gesture Recognition System: this is often done using image processing using OpenCV library. (2.) Controlling VLC Media Player using hand gestures: during this step we controlled the player using shell commands which were recalled using python commands through OS library

Hand gesture recognition is an important area of research in the field of Human Computer Interaction (HCI). The geometric attributes of the hand play an important role in hand shape reconstruction and gesture recognition. That said, fingertips are one of the important attributes for the detection of hand gestures and can provide valuable information from hand images. Many methods are available in scientific literature for fingertips detection with an open hand but very poor results are available for fingertips detection when the hand is closed. This paper presents a new method for the detection of fingertips in a closed hand using the corner detection method and an advanced edge detection algorithm. It is important to note that the skin color segmentation methodology did not work for fingertips detection in a closed hand. Thus the proposed method applied Gabor filter techniques for the detection of edges and then applied the corner detection algorithm for the detection of fingertips through the edges. To check the accuracy of the method, this method was tested on a vast number of images taken with a webcam. The method resulted in a higher accuracy rate of detections from the images. The method was further implemented on video for testing its validity on real time image capturing. These closed hand fingertips detection would help in controlling an electro-mechanical robotic hand via hand gesture in a natural way.

The research in the dynamic hand gestures recognition field has been promoted thanks to the recent success of the emerging Leap Motion controller (LMC) sensor, especially in the human-computer interaction field. During surgery, surgeons need to make more effective use of computers through a touchless human-computer interaction. In this paper, we present a set of gestures that are about commands to manipulate medical images. The public availability of hand gesture datasets for dynamic hand gestures, in particular with the LMC as an acquisition device, is currently quite poor. Therefore, we collect a more challenging dataset, called “LeapGestureDB”. The dataset is presented as text files, which contains 6,600 samples of 11 different dynamic hand gestures performed by over 120 participants. It is hosted online for the public interest, and especially to enhance research results. We suggest an approach of three-dimensional dynamic hand gesture recognition. We extract spatial, frequential and spatio-frequential features from the three-dimensional positions of the fingertips and the palm center delivered from the input sensor. These descriptors are fed into a support vector machine classifier to determine the performed gestures. We first evaluate our work on an existing dataset, namely “RIT dataset”, and then we use our collected dataset. We achieved a better accuracy rate of 93% comparing to results obtained from the “RIT dataset”.

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

We propose in this work a dynamic group sparsity (DGS) based time-frequency feature extraction method for dynamic hand gesture recognition (HGR) using millimeter-wave radar sensors. Micro-Doppler signatures of hand gestures show both sparse and structured characteristics in time-frequency domain, but previous study only focus on sparsity. We firstly introduce the structured prior when modeling the micro-Doppler signatures in this work to further enhance the features of hand gestures. The time-frequency distributions of dynamic hand gestures are first modeled using a dynamic group sparse model. A DGS-Subspace Pursuit (DGS-SP) algorithm is then utilized to extract the corresponding features. Finally, the support vector machine (SVM) classifier is employed to realize the dynamic HGR based on the extracted group sparse micro-Doppler features. The experiment shows that the proposed method achieved 3.3% recognition accuracy improvement over the sparsity-based method and has a better recognition accuracy than CNN based method in small dataset.

Human communication relies on a large number of different communication mechanisms like spoken language, facial expressions, or gestures. Facial expressions and gestures are one of the main nonverbal communication mechanisms and pass large amounts of information between human dialog partners. Therefore, to allow for intuitive human-machine interaction, a real-time capable processing and recognition of facial expressions, hand and head gestures are of great importance. We present a system that is tackling these challenges. The input features for the dynamic head gestures and facial expressions are obtained from a sophisticated three-dimensional model, which is fitted to the user in a real-time capable manner. Applying this model different kinds of information are extracted from the image data and afterwards handed over to a real-time capable data-transferring framework, the so-called Real-Time DataBase (RTDB). In addition to the head and facial-related features, also low-level image features regarding the human hand - optical flow, Hu-moments are stored into the RTDB for the evaluation process of hand gestures. In general, the input of a single camera is sufficient for the parallel evaluation of the different gestures and facial expressions. The real-time capable recognition of the dynamic hand and head gestures are performed via different Hidden Markov Models, which have proven to be a quick and real-time capable classification method. On the other hand, for the facial expressions classical decision trees or more sophisticated support vector machines are used for the classification process. These obtained results of the classification processes are again handed over to the RTDB, where other processes (like a Dialog Management Unit) can easily access them without any blocking effects. In addition, an adjustable amount of history can be stored by the RTDB buffer unit.

Hand gesture recognition forms a great difficulty for computer vision especially in dynamics. Sign language has been significant and an interesting application field of dynamic hand gesture recognition system. The recognition of human hands formed an- extremely complicated mission. The solution for such a difficulty requires a robust hand tracking method which depends on an effective feature and classifier. This paper presents a novel, fast and simple method for dynamic hand gesture recognition based on two lines (hundred) of features extracted from two rows of a Real-Time video. Feature selections have been used for hand shape representation to recognize the dynamic word for Kurdish Sign Language. The features extracted in real time from pre-processed hand object were represented through the optimization values of binary captured frame. Finally, an Artificial Neural Network classifier is used to recognize the performed hand gestures by 80% for training and 20% for testing with success 98%.

In this paper we have implemented a wireless vision based mobile robot control through hand gesture recognition based on perceptual color space such as HIS, HSV/HSB, HSL. Vision-based hand gesture recognition is an important problem in the field of human-computer interaction, since hand motions and gestures could potentially be used to interact with computers in more natural ways. The robot control was done purely based on the orientation histograms a simple and fast algorithm on the system which would recognize static hand gestures with HSV color spaces as major parameters. The wireless based mobile robot system using hand gestures is a new innovative user interface that resolves the complications of using numerous remote controls for various applications. Based on one unified set of hand gestures, this system interprets the user hand gestures into pre-defined commands to control the remote robot. The experimental results are very encouraging as the system produces real-time responses and highly accurate recognition towards various gestures under different lighting conditions.