A hierarchical structure for gesture recognition using RGB-D sensor

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

This article explores the application of YOLOv11 algorithm in gesture recognition field to evaluate its performance in human-computer interaction (HCI). By introducing the YOLOv8 model and corresponding dataset for training, we obtained the confusion matrix predicted by the model, which shows that the model can accurately recognize most gestures, although there are a few cases of misidentification. When the IoU threshold is 0.5, the average accuracy (mAP) of the model steadily improves with the progress of training, indicating that the overall performance of the model in gesture detection tasks has been enhanced. In addition, even under stricter evaluation conditions where the IoU threshold was increased from 0.5 to 0.95, mAP still showed an upward trend, although the growth rate was not as significant as mAP50, which still demonstrated the improvement in model performance. Through the detection of the test set images, we found that the YOLOv11 model can effectively recognize gestures and accurately interpret their meanings, demonstrating high accuracy. This study not only demonstrates the potential of YOLOv11 in gesture recognition tasks, but also provides a new technological path for the future development of HCI field. Overall, the YOLOv11 algorithm has demonstrated strong performance and accuracy in gesture recognition, providing a more natural and intuitive way for interaction between smart devices and humans.

Hand joints-based gesture recognition using a neural network provides excellent performance in hand gesture recognition. However, during the collection of sequential skeletal datasets, joints identification through hand pose estimations usually includes noise and even errors, which often diminish the accuracy of gesture recognition. To promote the availability of hand gesture recognition for such noisy datasets, this paper presents a nested interval unscented Kalman filter (UKF) with long short-term memory (NIUKF-LSTM) network to improve the accuracy of hand gesture recognition from noisy datasets. This nested interval method with the UKF changes the distribution of the sigma points based on two sampling intervals. By considering the information of previous frames, the nested interval method helps the NIUKF-LSTM network revise the noise in the sequential hand skeletal data and improve the recognition accuracy. The experimental results showing the removal of noisy skeletal data from the dynamic hand gesture dataset demonstrate the effectiveness of our NIUKF-LSTM network, which achieves better performance than do other state-of-the-art methods.

Gesture recognition is critical in human-computer communication. As observed, a plethora of current technological developments are in the works, including biometric authentication, which we see all the time in our smartphones. Hand gesture focus, a frequent human-computer interface in which we manage our devices by presenting our hands in front of a webcam, can benefit people of different backgrounds. Some of the efforts in human-computer interface include voice assistance and virtual mouse implementation with voice commands, fingertip recognition and hand motion tracking based on an image in a live video. Human Computer Interaction (HCI), particularly vision-based gesture and object recognition, is becoming increasingly important. Hence, we focused to design and develop a system for monitoring fingers using extreme learning-based hand gesture recognition techniques. Extreme learning helps in quickly interpreting the hand gestures with improved accuracy which would be a highly useful in the domains like healthcare, financial transactions and global business

The development of flexible pressure sensors for monitoring human motion and physiological signals has attracted extensive scientific research. However, achieving low monitoring limits, a wide detection range, large bending stresses, and excellent mechanical stability simultaneously remains a serious challenge. With the aim of developing a high-performance capacitive pressure sensor (CPS), this paper introduces the successful preparation of a single-walled carbon nanotube (SWNT)/polydimethylsiloxane (S-PDMS) composite dielectric with a foam-like structure (high permittivity and low elasticity modulus) and MXene/SWNT (S-MXene) composite film electrodes with a micro-crumpled structure. The above structurally modified CPS (SMCPS) demonstrated an excellent response output during pressure loading, achieving a wide pressure detection range (up to 700 kPa), a low detection limit (16.55 Pa), fast response/recovery characteristics (48/60 ms), enhanced sensitivity across a wide pressure range, long-term stability under repeated heavy loading and unloading (40 kPa, >2000 cycles), and reliable performance under various temperature and humidity conditions. The SMCPS demonstrated a precise and stable capacitive response in monitoring subtle physiological signals and detecting motion, owing to its unique electrode structure. The flexible device was integrated with an Internet of Things module to create a smart glove system that enables real-time tracking of dynamic gestures. This system demonstrates exceptional performance in gesture recognition and prediction with artificial intelligence analysis, highlighting the potential of the SMCPS in human-machine interface applications.

Non-verbal communication involves the usage of Sign Language. The sign language is used by people with hearing / speech disabilities to express their thoughts and feelings. But normally, people find it difficult to understand the hand gestures of the specially challenged people as they do not know the meaning of the sign language gestures. Usually, a translator is needed when a speech / hearing impaired person wants to communicate with an ordinary person and vice versa. In order to enable the specially challenged people to effectively communicate with the people around them, a system that translates the Indian Sign Language (ISL) hand gestures of numbers (1-9), English alphabets (A-Z) and a few English words to understandable text and vice versa has been proposed in this paper. This is done using image processing techniques and Machine Learning algorithms. Different neural network classifiers are developed, tested and validated for their performance in gesture recognition and the most efficient classifier is identified.

This ppaper investigates the application of the YOLOv8 model in conjunction with Mediapipe and Squeeze-and-Excitation (SE) blocks for Indian Sign Language (ISL). ISL with its inherent complexities, presents unique challenges in gesture recognition. YOLOv8, a cutting-edge open-source classification model has shown promise in various image processing tasks, including segmentation and classification. However, its performance in ISL gesture recognition remains underexplored. In this research, the YOLOv8 model is integrated with Mediapipe, known for its efficient hand tracking and gesture recognition capabilities, to capture dynamic gesture features more accurately. Additionally, the incorporation of Squeeze-and-Excitation (SE) blocks enhances the model’s ability to focus on critical features by adaptively recalibrating channel-wise feature responses, thus improving classification accuracy. The correct placement and integration of SE block with YOLO architecture affects the features that the attention layer learns. This hybrid model architecture thoroughly evaluates various configurations and hyperparameters, offering valuable insights into the strengths and limitations of YOLOv8 combined with Mediapipe and SE blocks for ISL recognition. The results point out the importance of a diverse dataset and the limitations of improvement the addition of attention layer can bring to the current YOLO architecture used.

While myoelectric pattern recognition is a prevailing way for gesture recognition, the inherent nonstationarity of electromyography signals hinders its long-term application. This study aims to prove a hypothesis that morphological information of muscle contraction detected by ultrasound image is potentially suitable for long-term use. A set of ultrasound-based algorithms are proposed to realize robust hand gesture recognition over multiple days, with user training only at the first day. A markerless calibration algorithm is first presented to position the ultrasound probe during donning and doffing; an algorithm combining speeded-up robust features and bag-of-features model being immune to ultrasound probe shift and rotation is then introduced; a self-enhancing classification method is next adopted to update classification model automatically by incorporating useful knowledge from testing data; finally the performance of long-term hand gesture recognition with zero re-training is validated by a six-day experiment of six healthy subjects, whose outcomes strongly support the hypothesis with about 94% of gesture recognition accuracy for each testing day. This study confirms the feasibility of adoption of ultrasound sensing for long-term musculature related applications.

This study presents a multimodal gesture recognition system based on the fusion of visual and strain sensor data, aiming to overcome the limitations of traditional gesture recognition methods in complex environments. A flexible strain sensor made from a CNTs-Graphene-TPU composite material was developed to create a smart glove with high sensitivity and stability, capable of accurately capturing finger flexion signals. Meanwhile, by integrating deep learning algorithms, a hybrid gesture recognition framework was constructed that combines visual and strain sensor data. Experimental results show that the system maintains high recognition accuracy (96.31%) across varying lighting conditions, significantly improving gesture recognition accuracy and system robustness. The system shows promising applications, particularly for gesture interaction and sign language translation in challenging lighting environments.

With the rapid development of human-computer interaction technology, gesture recognition becomes one of the key technologies of human-computer interaction. In this paper, we propose a new method of dynamic hand gestures recognition. The method adopts the hierarchical identification model for dynamic hand gestures recognition. First, we combine frame fusion with density distribution features for rough gesture recognition, second, we use the hausdorff distance or fingertip detection for accurate gesture recognition. The main innovation of this method lies in that we change the way of dynamic gestures recognition into the recognition of static image, improves the efficiency of gesture recognition effectively. Experimental results showed that our recognition rate is above 90%.

Real-time gesture recognition is mandatory for gesture based human-robot natural interaction. Several glove-based approaches are introduced for this purpose, but they are un-natural and force user to wear some cumbersome glove. This paper presents a vision-based gesture recognition method for human-robot interface. Hand poses and face are detected using multiple feature-based template matching techniques from the three largest skin-like regions that are segmented using skin color information from YIQ color space. Gesture is recognized using rule-based approach whenever the three skin-like regions at a particular image frame match with the predefined gesture. Gesture command is sent to robot through a knowledge based software platform for human-robot interface. In this paper, a method has also been proposed to detect left hand and right hand relative to face position as well as, to detect and locate its position. The effectiveness of the proposed method has been demonstrated over the interaction with a pet robot named AIBO

In order to incorporate naturalness in the design of human computer interfaces (HCI), it is desirable to develop recognition techniques capable of handling continuous natural gesture and speech inputs. Though many different researchers have reported high recognition rates for gesture recognition using hidden Markov models (HMM), the gestures used are mostly pre-defined and are bound with syntactical and grammatical constraints. But natural gestures do not string together in syntactical bindings. Moreover, strict classification of natural gestures is not feasible. We have examined hand gestures made in a very natural domain, that of a weather person narrating in front of a weather map. The gestures made by the weather person are embedded in a narration. This provides us with abundant data from an uncontrolled environment to study the interaction between speech and gesture in the context of a display. We hypothesize that this domain is very similar to that of a natural human-computer interface. We present an HMM architecture for continuous gesture recognition framework and keyword spotting. To explore the relation between gesture and speech, we conducted a statistical co-occurrence analysis of different gestures with a selected set of spoken keywords. We then demonstrate how this co-occurrence analysis can be exploited to improve the performance of continuous gesture recognition.

Gesture recognition has been the subject of intensive research in recent years owing to its wide applications. Unlike traditional systems, which usually require wearable sensors, many recent works have achieved the desirable gesture recognition performance using wireless channel state information from commercially available WiFi devices. However, existing works generally require training new models for different locations due to the location-dependent nature of channel state information. This paper proposes a location-independent system that can recognize gestures performed in a new location without training a new model. Our approach uses disentanglement that extricates location and other extraneous information from those needed for gesture recognition. The implementation is based on an unsupervised invariance induction framework consisting of feature extraction, a multi-output latent space, gesture recognition, and decoder modules. The key idea in designing this system is to separate gesture-dependent features from location-dependent features. Specifically, the feature extraction module consisting of a long short-term memory network is employed to select representative features; it essentially serves as an encoder to generate the latent space. During the training process, the network learns to cluster features representation for the gesture recognition and decoder by minimizing the total loss of the gesture recognition and decoder modules. We test our system with a dataset collected from various subjects performing four different gestures in multiple locations in seven rooms with different layouts. The results show that our location-independent gesture recognition system can achieve 88.69% accuracy for new locations.

Device free wireless sensing is an important research direction for next generation communication systems, which attracts increasing attention recently. Hand gesture recognition is one of the most popular applications. Although gesture recognition via commodity WiFi or Long Term Evolution (LTE) signal is well studied, seldom work exploits New Radio (NR) signal despite NR is already extensively commercialized. Compared with previous wideband communication system, the maximum bandwidth of NR is further improved to 100 MHz in Frequency Range 1. On this basis, we propose an NR based hand gesture recognition scheme in the presence of interference activities. We analyze the effect of time domain channel path leakage, and propose a multi-path extraction and combination method, which can separate the expected gesture from background movements. A prototype system is built to demonstrate the performance of gesture recognition in the presence of interference, and the influence of signal bandwidth is studied as well.

Surface electromyogram (sEMG) signal has been applied to gesture recognition successfully. However, performance of gesture recognition degrades due to noise, which is unavoidable in practical environment. This study aims to propose a gesture recognition system using a multiple classifier system (MCS) based on random subspace. Our method makes a decision using a signal which is less sensitive to noise. Each base classifier is trained using the subset of channels selected randomly. The influence of a base classifier by a noisy sample is measured by sensitivity, defined as the perturbation of an output of a classifier. The final decision of the MCS is determined according to the output of the based stable classifiers fused by majority vote. Experimental results show that the proposed method achieves better classification performance in comparison with its base classifier and the random subspace method in terms of accuracy and robustness to noise.