A real time system for dynamic hand gesture recognition with a depth sensor

Hand gesture recognition (HGR) systems using electromyography (EMG) bracelet-type sensors are currently largely used over other HGR technologies. However, bracelets are susceptible to electrode rotation, causing a decrease in HGR performance. In this work, HGR systems with an algorithm for orientation correction are proposed. The proposed orientation correction method is based on the computation of the maximum energy channel using a synchronization gesture. Then, the channels of the EMG are rearranged in a new sequence which starts with the maximum energy channel. This new sequence of channels is used for both training and testing. After the EMG channels are rearranged, this signal passes through the following stages: pre-processing, feature extraction, classification, and post-processing. We implemented user-specific and user-general HGR models based on a common architecture which is robust to rotations of the EMG bracelet. Four experiments were performed, taking into account two different metrics which are the classification and recognition accuracy for both models implemented in this work, where each model was evaluated with and without rotation of the bracelet. The classification accuracy measures how well a model predicted which gesture is contained somewhere in a given EMG, whereas recognition accuracy measures how well a model predicted when it occurred, how long it lasted, and which gesture is contained in a given EMG. The results of the experiments (without and with orientation correction) executed show an increase in performance from 44.5% to 81.2% for classification and from 43.3% to 81.3% for recognition in user-general models, while in user-specific models, the results show an increase in performance from 39.8% to 94.9% for classification and from 38.8% to 94.2% for recognition. The results obtained in this work evidence that the proposed method for orientation correction makes the performance of an HGR robust to rotations of the EMG bracelet.

In today’s ever-evolving healthcare landscape, our objective is to bring technology closer to human experience. In this paper, a vision-based hand gesture recognition (HGR) system is introduced as a human computer interface (HCI) that transforms natural movements into intuitive medium for device and application manipulation. This innovative approach is designed to empower the elderly and enhance patient monitoring by offering a seamless, nonintrusive interface. By bridging compassion with cutting-edge technology, this approach aims to redefine care and create a more connected, responsive healthcare environment. The study begins by reviewing both traditional and contemporary HGR methodologies, with a focus on vision-based systems, which have shown greater potential for practical applications compared to conventional sensor-based systems. Traditional approaches, such as contour analysis, colour segmentation, and template matching, have demonstrated limitations. In contrast, deep learning approaches have gained mass adaptation in vision-based systems for their ability to learn hierarchical features and better handling complex, dynamic scenarios. However, existing models struggle in real-time performance due to computational inefficiencies and environmental variations. This study introduces a deep learning-based detection framework which enables the practical application of HGR. For example, it supports the development of gesture-based virtual assistants for interacting with digital devices and allows elderly residents in care homes to give instructions to remote caregivers. The framework consists of two deep learning models, one for region of interest (ROI) detection and another for classification. The recognition process begins with capturing RGB images, chosen for versatility, affordability, and compatibility, which then passes through YOLOv8 (You Only Look Once version 8) for detection. The YOLOv8 model is trained on a subset, only 4.5%, of the HaGRID dataset, comprises over 550,000 RGB images, to accurately locate hand regions. Once detection is complete, several image processing techniques are applied to overcome typical HGR constraints. These include ROI extraction and gray scaling for faster computation, resizing for model optimization and distance viability, HSV conversion for lighting independence, and histogram equalization for enhanced feature extraction. The processed image is then fed into a convolutional neural network (CNN) for gesture classification. The network is designed with several convolutional layers, followed by dense layers with ReLU activation to introduce non-linearity, and SoftMax activation for classification. To improve the model's robustness, data augmentation techniques such as random flipping and rotation are applied. The HGR system demonstrates a staggering accuracy of 97% for training & validation. The proposed framework effectively addresses the limitations of conventional HGR systems—many of which have been developed and tested in controlled environments—demonstrating its applicability in real-life situations.

Hand gesture recognition is a relevant study topic for a reason that sometimes we may not be in position to communicate verbally. There is need to design Hand gesture recognition systems in order to help people adopt to nonverbal communication mainly sign language. However, there is no clue to understand the meaning of gesture through the computers directly. So this calls for definitions that generalize models in a computer. That is why the machine-learning approaches are implemented in recognition systems. There are generally two types of hand gestures recognition systems which researches have concentrated on. These include static and dynamic Hand gesture recognition systems. However, in building Hand gesture recognition systems, various machine learning approaches have been used. For implementing the proposed system, MS Kinect depth sensor was used as a hardware. The Kinect depth sensor is composed of an infrared camera. This is an advantage to the systems that are designed basing on the depth sensing because factors like color, clothing and background have less effect on the performance. So Kinect based depth sensor systems have a high accuracy and performance making them relevant and applicable in our daily lives. In this paper, we propose a static hand gesture recognition system in real time using two machine learning methods namely Support Vector Machine and Artificial Neural Networks. We use of the newly launched Microsoft Kinect sensor for image extraction. The sensor helps us to extract the hand images. We implement the system on a Matlab platform for reasons that Matlab is widely used by researchers in different fields and that can handle complex computations. In the training of the model, we collect a hundred depth-based Histogram of Oriented Gradient features per alphabet from the hand gesture images which we trained, tested and validated using Artificial Neural Networks (ANN) and Support Vector Machine (SVM). From this dataset, we can generate the generalized gesture model for each alphabet image. For the proposed system, the classification with ANN proves a higher performance then SVM.

At the present time, hand gestures recognition system could be used as a more\nexpected and useable approach for human computer interaction. Automatic hand\ngesture recognition system provides us a new tactic for interactive with the\nvirtual environment. In this paper, a face and hand gesture recognition system\nwhich is able to control computer media player is offered. Hand gesture and\nhuman face are the key element to interact with the smart system. We used the\nface recognition scheme for viewer verification and the hand gesture\nrecognition in mechanism of computer media player, for instance, volume\ndown/up, next music and etc. In the proposed technique, first, the hand gesture\nand face location is extracted from the main image by combination of skin and\ncascade detector and then is sent to recognition stage. In recognition stage,\nfirst, the threshold condition is inspected then the extracted face and gesture\nwill be recognized. In the result stage, the proposed technique is applied on\nthe video dataset and the high precision ratio acquired. Additional the\nrecommended hand gesture recognition method is applied on static American Sign\nLanguage (ASL) database and the correctness rate achieved nearby 99.40%. also\nthe planned method could be used in gesture based computer games and virtual\nreality.\n

DeReFNet: Dual-stream Dense Residual Fusion Network for static hand gesture recognition

Hand gesture is one of the method used in sign language for non-verbal communication. It is most commonly used by deaf & dumb people who have hearing or speech problems to communicate among themselves or with normal people. Various sign language systems has been developed by many makers around the world but they are neither flexible nor cost-effective for the end users. Hence in this paper introduced software which presents a system prototype that is able to automatically recognize sign language to help deaf and dumb people to communicate more effectively with each other or normal people. Pattern recognition and Gesture recognition are the developing fields of research. Being a significant part in nonverbal communication hand gestures are playing key role in our daily life. Hand Gesture recognition system provides us an innovative, natural, user friendly way of communication with the computer which is more familiar to the human beings. By considering in mind the similarities of human hand shape with four fingers and one thumb, the software aims to present a real time system for recognition of hand gesture on basis of detection of some shape based features like orientation, Centre of mass centroid, fingers status, thumb in positions of raised or folded fingers of hand

This paper presents a wearable hand gesture recognition (HGR) system, which can decode the information from surface electromyography (sEMG) and micro-inertial measurement unit μ-IMU. With the cooperation between sEMG and IMU, the number of sEMG electrodes is reduced to 2 pairs without scarifying the accuracy and recognition range, which significantly shorten the distance to practical applications. For low-power and high-security concerns, a capacitive coupled body channel communication (CC-BCC) module is also implemented in the system for wireless communication. Last, a modified deep forest algorithm is employed to predict the gestures from the signal sources with high accuracy and robustness. Finally, 16 hand gestures include 10 dynamic and 6 static gestures are recognized on two different subjects, the proposed system can achieve 96% accuracy, and the prediction time for each sample is less than 6 ms.

Gesture recognition plays a vital role in multiple applications; the applications may be related to simple aspects such as playing video games to complex tasks such as related to communication using sign symbols. In this paper, we propose a robust hand gesture recognition system using skin detection technique. Gesture recognition plays a key role for different applications. In this regard, we attempt to take out the type features of the fingers, for instance their orientation, their status in terms of raised or folded fingers and their respective image position, which require the processing of images as a matter of principle. In order to recognize the terms used for contact the program uses a complex approach to distinguish different hand movements. The dynamic sequence of the images that function as a contribution to the hand gesture recognition system is captures using the camera. GoogLeNet is used for hand gesture recognition training and detection. In order to train GoogLeNet, we use transfer learning. We also used American Sign Language (ASL) to map the sign to a particular alphabet.

In this study, the authors present a new system for sign language hand gesture recognition. Using video input, the system can recognise any spelled word or alphabetic sequence signed in American Sign Language. The three main steps in the recognition process include detection of the region of interest (the hands), detection of key frames and recognition of gestures from these key frames. The proposed segmentation algorithm distinguishes regions of interest from both uniform and non-uniform backgrounds with an efficiency of 95%. The proposed key frame detection algorithm achieves an efficiency of 96.50%. A rotation-invariant algorithm for feature extraction is additionally proposed, which provides an overall gesture recognition efficiency of 84.2%.

Hand gesture recognition is a quickly developing field with many uses in human-computer interaction, sign language recognition, virtual reality, gaming, and robotics. This paper reviews different ways to model hands, such as vision-based, sensor-based, and data glove-based techniques. It emphasizes the importance of accurate hand modeling and feature extraction for capturing and analyzing gestures. Key features like motion, depth, color, shape, and pixel values and their relevance in gesture recognition are discussed. Challenges faced in hand gesture recognition include lighting variations, complex backgrounds, noise, and real-time performance. Machine learning algorithms are used to classify and recognize gestures based on extracted features. The paper emphasizes the need for further research and advancements to improve hand gesture recognition systems' robustness, accuracy, and usability. This review offers valuable insights into the current state of hand gesture recognition, its applications, and its potential to revolutionize human-computer interaction and enable natural and intuitive interactions between humans and machines. In simpler terms, hand gesture recognition is a way for computers to understand what people are saying with their hands. It has many potential applications, such as allowing people to control computers without touching them or helping people with disabilities communicate. The paper reviews different ways to develop hand gesture recognition systems and discusses the challenges and opportunities in this area.

Hand gesture recognition systems often struggles when a gesture is occluded by the hand itself or by the other hand. To address this problem, multiview gestures may be used while training a gesture recognition system. Unfortunately, there are very few datasets available that contains different views of every single gesture. In this paper, we propose a method to handle occlusions in single-view sign gestures by generating multiple views from a single-view gesture using conditional multiview gesture synthesis. The generated views help to solve the occlusion problem, thereby enhancing the recognition performance. Additionally, we introduce a vision-based Multiview Hand Gesture Recognition (MHGR) framework that utilizes the generated multiview gestures for gesture recognition. Experiments conducted on the HGM-4 dataset demonstrate that the generated images are of high quality, photo-realistic, and significantly improve the recognition accuracy compared to some other existing methods.

Human-computer interaction introduces critical open door with the proceeds with improvement of wearable gadgets. Gesture recognition through smart devices is becoming a popular research direction. This paper proposes a hand gesture recognition and real-time game control system that is capable of continues human-computer interaction in view of an off-the-rack business wearable wristband. We utilize three-axis accelerator and gyroscope sensors embedded in smart band to collect hand motion information and use Kinect camera capture video information for manual segmentation during the training model phase. A continuous gesture segmentation algorithm based on sliding window and DTW algorithm is developed to detect meaningful gestures in the real-time game control stage. In addition, an android game named Fly Birds which is controlled by gesture recognition result is presented to simulate real-time human-computer interaction. Then, we classify the data in the window using common classifiers. Finally, our experimental results show that, we can accurately identify the designed gestures during the stage of static gesture recognition, and we also achieve a perfect interactive effect in the process of dynamic real-time game control. The experiment outcomes will advance the ascent of human-PC cooperation in view of hand gesture recognition and related applications will rise in vast numbers.

Gesture recognition is crucial for enhancing human-computer interaction and is particularly pivotal in rehabilitation contexts, aiding individuals recovering from physical impairments and significantly improving their mobility and interactive capabilities. However, current wearable hand gesture recognition approaches are often limited in detection performance, wearability, and generalization. We thus introduce EchoGest, a novel hand gesture recognition system based on soft, stretchable, transparent artificial skin with integrated ultrasonic waveguides. Our presented system is the first to use soft ultrasonic waveguides for hand gesture recognition. EcoflexTM 00-31 and EcoflexTM 00-45 Near ClearTM silicone elastomers were employed to fabricate the artificial skin and ultrasonic waveguides, while 0.1 mm diameter silver-plated copper wires connected the transducers in the waveguides to the electrical system. The wires are enclosed within an additional elastomer layer, achieving a sensing skin with a total thickness of around 500 μ m. Ten participants wore the EchoGest system and performed static hand gestures from two gesture sets: 8 daily life gestures and 10 American Sign Language (ASL) digits 0-9. Leave-One-Subject-Out Cross-Validation analysis demonstrated accuracies of 91.13% for daily life gestures and 88.5% for ASL gestures. The EchoGest system has significant potential in rehabilitation, particularly for tracking and evaluating hand mobility, which could substantially reduce the workload of therapists in both clinical and home-based settings. Integrating this technology could revolutionize hand gesture recognition applications, from real-time sign language translation to innovative rehabilitation techniques.

Gesture recognition has been studied for a while within the fields of computer vision and pattern recognition. A gesture can be defined as a meaningful physical movement of the fingers, hands, arms, or other parts of the body with the purpose to convey information for the environment interaction. For instance, hand gesture recognition (HGR) can be used to recognize sign language which is the primary means of communication by the deaf and mute. Vision-based HGR is critical in its application; however, there are challenges that will need to be overcome such as variations in the background, illuminations, hand orientation and size and similarities among gestures. The traditional machine learning approach has been widely used in vision-based HGR in recent years but the complexity of its processing has been a major challenge—especially on the handcrafted feature extraction. The effectiveness of the handcrafted feature extraction technique was not proven across various datasets in comparison to deep learning techniques. Therefore, a hybrid network architecture dubbed as Lightweight VGG16 and Random Forest (Lightweight VGG16-RF) is proposed for vision-based hand gesture recognition. The proposed model adopts feature extraction techniques via the convolutional neural network (CNN) while using the machine learning method to perform classification. Experiments were carried out on publicly available datasets such as American Sign Language (ASL), ASL Digits and NUS Hand Posture dataset. The experimental results demonstrate that the proposed model, a combination of lightweight VGG16 and random forest, outperforms other methods.

Automatic hand gesture recognition is the most important part of sign language translation. Its importance increases with the growth of deaf and hard of hearing population and cognitive computing. In this article, we propose an efficient system for automatic hand gesture recognition based on deep learning. The proposed system is based on a convolutional neural network (CNN). It employs a transfer learning of 3D CNN for hand gesture recognition. Three different datasets are used to evaluate the proposed system in signer dependent and signer independent modes.