Real-Time Monocular Skeleton-Based Hand Gesture Recognition Using 3D-Jointsformer.

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Recently hand gesture recognition via millimeter-wave radar has attracted a lot of research attention for human-computer interaction. Encouraged by the ability of deep learning models in successfully tackling hand gesture recognition tasks, we propose a deep neural network (DNN) model namely, Res3DTENet that aims to classify dynamic hand gestures using the radio frequency (RF) signals. We propose a scheme that improves the convolutional process of 3DCNNs with residual skip connection (Res3D) to emphasize local-global information and enriches the intra-frame spatio-temporal feature representation. A multi-head attention transformer encoder (TE) network has been trained over the spatio-temporal features to refine the inter-frame temporal dependencies of range-Doppler sequences. The experiments are carried out on the publicly available Soli hand gesture data set. Based on our extensive experiments, we show that the proposed network achieves improved gesture recognition accuracy than the state-of-the-art hand gesture recognition methods.

Hand gesture recognition is an important area of research in the field of Human Computer Interaction (HCI). The geometric attributes of the hand play an important role in hand shape reconstruction and gesture recognition. That said, fingertips are one of the important attributes for the detection of hand gestures and can provide valuable information from hand images. Many methods are available in scientific literature for fingertips detection with an open hand but very poor results are available for fingertips detection when the hand is closed. This paper presents a new method for the detection of fingertips in a closed hand using the corner detection method and an advanced edge detection algorithm. It is important to note that the skin color segmentation methodology did not work for fingertips detection in a closed hand. Thus the proposed method applied Gabor filter techniques for the detection of edges and then applied the corner detection algorithm for the detection of fingertips through the edges. To check the accuracy of the method, this method was tested on a vast number of images taken with a webcam. The method resulted in a higher accuracy rate of detections from the images. The method was further implemented on video for testing its validity on real time image capturing. These closed hand fingertips detection would help in controlling an electro-mechanical robotic hand via hand gesture in a natural way.

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

Recently, human - machine interaction (HMI) becomes a hot research topic because of its wide applications, ranging from automatic device control to designing and development of assistant robot or even smart building at sparser scale. One of the most important questions in this research field is finding out an efficient and natural method of HMI. Among several channels of communication, hand gestures have been shown to be an intuitive and efficient mean to express an idea or to control something. For a successful hand gesture based interaction between human and robot, a vocabulary of hand gestures needs to be defined. To resolve this problem, in this paper, we propose a framework to study the behavior of Vietnamese in using of hand gesture in communication with robot. This study allows designing a hand gesture vocabulary for human - robot interaction (HRI) applications. In the literature, there are no works similar to ours. This makes our twofold contributions: (1) a proposed framework for hand gesture recognition: hand gesture vocabulary design, feature extraction for hand posture representation, hand posture classification, and hand gesture database construction; (2) some experimentations are realized to evaluate the defined hand gesture set that can be used in general situation of HRI. The experiment results show that the defined hand gesture set satisfies the both criteria: intuitiveness and recognisability.

In this paper we have implemented a wireless vision based mobile robot control through hand gesture recognition based on perceptual color space such as HIS, HSV/HSB, HSL. Vision-based hand gesture recognition is an important problem in the field of human-computer interaction, since hand motions and gestures could potentially be used to interact with computers in more natural ways. The robot control was done purely based on the orientation histograms a simple and fast algorithm on the system which would recognize static hand gestures with HSV color spaces as major parameters. The wireless based mobile robot system using hand gestures is a new innovative user interface that resolves the complications of using numerous remote controls for various applications. Based on one unified set of hand gestures, this system interprets the user hand gestures into pre-defined commands to control the remote robot. The experimental results are very encouraging as the system produces real-time responses and highly accurate recognition towards various gestures under different lighting conditions.

In this paper, considering the characteristics of hand gestures that occur during a natural, unscripted human conversation, we report the results of experiments involving the data analysis of the video recordings of experiments. The recorded conversations between two subjects and the assistant in the experiments (the first author) are analyzed for findings that might be useful to future human-robot interaction (HRI), findings regarding the recognition of hand gestures in natural conversations. In the experiment, hand gestures that appeared naturally during the conversations were manually selected, and their elapsed times and number of appearances were analyzed. In addition, these unscripted conversations were analyzed with a voice recognition system, and the meanings of the hand gestures that appear in the recorded video were interpreted. It was observed that some gestures were made consciously and others unconsciously. Some gestures appeared multiple times with similar meanings and the same intention, especially gestures involving pointing at something to emphasize a sentence or word. Other observations included that there were times when (i) each subject made different hand gestures to convey the same meaning, (ii) both subjects made the same hand gesture but intended different meanings, and (iii) a specific gesture was made by the same subject but with different intended meanings. After considering the results of these experiments, we conclude that it is better to combine gestures with the utterances at the time in order to interpret their intended meanings correctly.

The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. However, real-time recognition of dynamic micro hand gestures from video streams is challenging for in-vehicle scenarios since (i) the gestures should be performed naturally without distracting the driver, (ii) micro hand gestures occur within very short time intervals at spatially constrained areas, (iii) the performed gesture should be recognized only once, and (iv) the entire architecture should be designed lightweight as it will be deployed to an embedded system. In this work, we propose an HCI system for dynamic recognition of driver micro hand gestures, which can have a crucial impact in automotive sector especially for safety related issues. For this purpose, we initially collected a dataset named Driver Micro Hand Gestures (DriverMHG), which consists of RGB, depth and infrared modalities. The challenges for dynamic recognition of micro hand gestures have been addressed by proposing a lightweight convolutional neural network (CNN) based architecture which operates online efficiently with a sliding window approach. For the CNN model, several 3-dimensional resource efficient networks are applied and their performances are analyzed. Online recognition of gestures has been performed with 3D-MobileNetV2, which provided the best offline accuracy among the applied networks with similar computational complexities. The final architecture is deployed on a driver simulator operating in real-time. We make DriverMHG dataset and our source code publicly available 1.1https://www.mmk.ei.tum.de/DriverMHG/

In this project we've discussed a system which uses some dynamic hand gesture recognition technique to manage the media players like VLC media player. This project consist of certain modules which segments the foreground a part of the frame using detection of skin colour and approximate median technique. the popularity of hand gesture is finished by creating a call Tree, that uses certain features extracted from the segmented part. This hand gesture recognition technique introduces a replacement, natural thanks to interact with computers and is beneficial to several folks in our day-to-day life. Hand gestures are associated with the human hands in hand gesture recognition. This project is employed for controlling certain operations (Pause, Play, Volume up, Volume Down, Mute) on video player by mere hand gestures without getting in the rigmarole of pressing buttons or tapping onto the screen. this may be directly associated with our day to life like in presentations. We have considered hand gestures and their directional motion defines a gesture for the applying. during this application image retrieving is finished employing a Webcam. Some functions in VLC media players are used most often and these functions uses some predefined gestures. Shows the defined gestures in keeping with the VLC player control operation. We created a VLC Media Player Controller using Hand Gesture Recognition System to form ‘HUMAN LIFE EASY AND BETTER’. This project is implemented using two steps: (1.) Creation of Hand Gesture Recognition System: this is often done using image processing using OpenCV library. (2.) Controlling VLC Media Player using hand gestures: during this step we controlled the player using shell commands which were recalled using python commands through OS library

With rapid developments in biometric recognition, a great deal of attention is being paid to robots which interact smartly with humans and communicate certain types of biometrical information. Such human–machine interaction (HMI), also well-known as human–robot interaction (HRI), will, in the future, prove an important development when it comes to automotive manufacturing applications. Currently, hand gesture recognition-based HRI designs are being practically used in various areas of automotive manufacturing, assembly lines, supply chains, and collaborative inspection. However, very few studies are focused on material-handling robot interactions combined with hand gesture communication of the operator. The current work develops a depth sensor-based dynamic hand gesture recognition scheme for continuous-time operations with material-handling robots. The proposed approach properly employs the Kinect depth sensor to extract features of Hu moment invariants from depth data, through which feature-based template match hand gesture recognition is developed. In order to construct continuous-time robot operations using dynamic hand gestures with concatenations of a series of hand gesture actions, the wake-up reminder scheme using fingertip detection calculations is established to accurately denote the starting, ending, and switching timestamps of a series of gesture actions. To be able to perform typical template match on continuous-time dynamic hand gesture recognition with the ability of real-time recognition, representative frame estimates using centroid, middle, and middle-region voting approaches are also presented and combined with template match computations. Experimental results show that, in certain continuous-time periods, the proposed complete hand gesture recognition framework can provide a smooth operation for the material-handling robot when compared with robots controlled using only extractions of full frames; presented representative frames estimated by middle-region voting will maintain fast computations and still reach the competitive recognition accuracy of 90.8%. The method proposed in this study can facilitate the smart assembly line and human–robot collaborations in automotive manufacturing.

The salient feature points of hand gesture play an important role for its representation and recognition. In this paper, a novel hand gesture recognition method based on salient feature point selection is proposed. The raw data of hand gesture is captured by the Kinect sensor and the hand gesture is segmented from the cluttered background. The shape feature of hand gesture is extracted from the contour, and the salient feature points are selected by a new algorithm to represent the hand gesture. Finally, the Dynamic Time Warping algorithm is modified and employed to find the best correspondence between two gestures. Extensive experiments are implemented on three benchmark databases to validate the effectiveness of our method. The experimental results verified the invariance of our method to translation, rotation scaling and articulated deformation. The comparison with state-of-the-art methods demonstrates the accuracy and efficiency of our method.

Recently, although radar sensors have been widely applied for hand gesture recognition (HGR) tasks, conventional radar-based HGR systems still have two major challenges. First, these systems rely on supervised learning approaches to learn gesture features, which normally require a large-scale labeled dataset to address the overfitting problem. However, the acquisition of such dataset is time-consuming. Second, the radar signature of hand movement is often influenced by micromotion caused by other body parts, which leads to distorted motion features, resulting in poor identification accuracy. To overcome these problems, we propose an unsupervised hand gesture feature learning method using the deep convolutional auto-encoder network to analyze hand gesture signal collected by a frequency modulated continuous wave (FMCW) radar sensor. First, via a convolutional encoder sub-network, input radar range profiles are transformed into lower dimensional representations. Then, the representations are expanded to reconstruct the corresponding input profiles by a deconvolutional decoder sub-network. In addition, to investigate the mechanisms of the proposed network and evaluate its performance, we conduct an in-depth study of the feature maps learned from various hand gesture experimental data and evaluate the corresponding classification performance. The results demonstrate that the proposed convolutional auto-encoder network is able to achieve high recognition accuracy with low training sample cost, which outperforms the state-of-the-art hand gesture recognition systems based on transfer learning VGGNet and fully connected-based auto-encoder network.

To naturally interact with virtual environment by hand gesture, this paper presents a robust RGB-D data based recognition method of static and dynamic hand gesture. Firstly, for static hand gesture recognition, starting from the hand gesture contour extraction, the palm center is identified by Distance Transform (DT) algorithm. The fingertips are localized by employing the K-Curvature-Convex Defects Detection algorithm (K-CCD). On the basis, the distances of the pixels on hand gesture contour to palm center and the angle between fingertips are considered as the auxiliary features to construct a multimodal feature vector, and then recognition algorithm is presented to robustly recognize the static hand gestures. Secondly, combining Euclidean distance between hand joints and shoulder center joint with the modulus ratios of skeleton features, this paper generates a unifying feature descriptor for each dynamic hand gesture and proposes an improved dynamic time warping (IDTW) algorithm to obtain recognition results of dynamic hand gestures. Finally, we conduct extensive experiments to test and verify the static and dynamic hand gesture recognition algorithm and realize a low-cost real-time application of natural interaction with virtual environment by hand gestures.

This research intends to understand whether users would adopt the interactive interface of hand gesture recognition for XRSPACE MANOVA in the virtual-reality environment. Different from the traditional joystick control and external sensors, XRSPACE MANOVA’s hand gesture recognition relies on cameras built into the head-mount display to detect users’ hand gestures and interact with the system to provide a more life-like immersive experience. To better understand if users would accept this hand gesture recognition, the current experiment compares users’ experiences with hand gesture recognition and joystick control for XRSPACE MANOVA while controlling for the effects of gender, college major, and the completion time. The results suggest that users of hand gesture recognition have better perceptions of enjoyment, satisfaction, and confirmation, which means that they have a relatively fun and satisfying experience and that their expectations of the system/technology confirm their actual usage. Based on the parametric statistical analyses, user assessments show that perceived usefulness, perceived ease-of-use, attitude, and perception of internal control suggest that, in terms of operating performance, users are more accepting of the traditional joystick control. When considering the length of usage time, this study finds that, when hand gesture recognition is used for a relatively longer time, users’ subjective evaluations of internal control and behavioral intention to use are reduced. This study has, therefore, identified potential issues with hand gesture recognition for XRSPACE MANOVA and discussed how to improve this interactive interface. It is hoped that users of hand gesture recognition will obtain the same level of operating experience as if they were using the traditional joystick control.

At the present time, hand gestures recognition system could be used as a more\nexpected and useable approach for human computer interaction. Automatic hand\ngesture recognition system provides us a new tactic for interactive with the\nvirtual environment. In this paper, a face and hand gesture recognition system\nwhich is able to control computer media player is offered. Hand gesture and\nhuman face are the key element to interact with the smart system. We used the\nface recognition scheme for viewer verification and the hand gesture\nrecognition in mechanism of computer media player, for instance, volume\ndown/up, next music and etc. In the proposed technique, first, the hand gesture\nand face location is extracted from the main image by combination of skin and\ncascade detector and then is sent to recognition stage. In recognition stage,\nfirst, the threshold condition is inspected then the extracted face and gesture\nwill be recognized. In the result stage, the proposed technique is applied on\nthe video dataset and the high precision ratio acquired. Additional the\nrecommended hand gesture recognition method is applied on static American Sign\nLanguage (ASL) database and the correctness rate achieved nearby 99.40%. also\nthe planned method could be used in gesture based computer games and virtual\nreality.\n