Light-Weight Deep Learning Techniques with Advanced Processing for Real-Time Hand Gesture Recognition

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Sign language identification and recognition technique is composed by the gesture images detection and the hand gestures recognition. Gesture images detection is to locate the image part of palm and fingers from the captured pictures, and rotating them to the appropriate gesture posture. Both of them are the important pre-processing for sign language identification and recognition. Lose them, the correctness rate of the sign language recognition algorithms will be dropped down to an unacceptable level. This paper presents novel processing algorithms for the gesture images detection and recognition. In the detection process, it rotates an askew gesture to right position, and to delete the elbow and forearm parts from the captured pictures. In the recognition process, it includes two phases with the model construction and the sign language identification. In the model construction phase, the static hand gesture of sign language is constructed by the Gaussian mixture model, and the unknown gesture image is identified by Gaussian model match. Based on this presented static sign language detection and recognition algorithms, the correct recognition rate is about 94% in average.

Speech impairment limits a person’s capacity for oral and auditory communication. Improvements in communication between the deaf and the general public can be progressed by a real-time sign language detector. Recent studies have contributed to make progress in motion and gesture identification processes using Deep Learning (DL) methods and computer vision. But the development of static and dynamic sign language recognition (SLR) models is still a challenging area of research. The difficulty is in obtaining an appropriate model that addresses the challenges of continuous signs that are independent of the signer. Different signers’ speeds, durations, and many other factors make it challenging to create a model with high accuracy and continuity. This study mainly focused on SLR using a modified DL and hybrid optimization approach. Notably, spatial and geometric-based features are extracted via the Visual Geometry Group 16 (VGG16), and motion features are extracted using the optical flow approach. A new DL model, CNNSa-LSTM, is a combination of a Convolutional Neural Network (CNN), Self-Attention (SA), and Long-Short-Term Memory (LSTM) to identify sign language. This model is developed for feature extraction by combining CNNs for spatial analysis with SA mechanisms for focusing on relevant features, while LSTM effectively models temporal dependencies. The proposed CNNSa-LSTM model enhances performance in tasks involving complex, sequential data, such as sign language processing. Besides, a Hybrid Optimizer (HO) is proposed using the Hippopotamus Optimization Algorithm (HOA) and the Pathfinder Algorithm (PFA). The proposed model has been implemented in Python, and it has been evaluated over the existing models in terms of accuracy (98.7%), sensitivity (98.2%), precision (98.5%), Word Error Rate (WER) (0.131), Sign Error Rate (SER) (0.114), and Normalized Discounted Cumulative Gain (NDCG) (98%) as well. The proposed model has recorded the highest accuracy of 98.7%.

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.

BISINDO is one of Indonesian sign language, which do not have many facilities to implement. Because it can cause deaf people have difficulty to live their daily life. Therefore, this research tries to offer an recognition or translation system of the BISINDO alphabet into a text. The system is expected to help deaf people to communicate in two directions. In this study the problems encountered is small datasets. Therefore this research will do the testing of hand gesture recognition, by comparing two model CNN algorithms, that is LeNet-5 and Alexnet. This test will look for which classification technique is better if the dataset conditions in an amount that does not reach 1000 images in each class. After testing, the results found that the CNN technique on the Alexnet architectural model is better to used, this is because when doing the testing process by using still-image and Alexnet model data which has been released in training process, Alexnet model data gives greater prediction results that is equal to 76%. While the LeNet model is only able to predict with the percentage of 19%. When that Alexnet data model used on the system offered, only able to predict correcly by 60%.
 
 Keywords: Sign language, BISINDO, Computer Vision, Hand Gesture Recognition, Skin Segmentation, CIELab, Deep Learning, CNN.

The interaction between human and machine can be achieved by implementing human gesture applications which are known as human computer interaction (HCI). It has been developed several years ago, and now we can easily find it in many applications in real world e.g. in virtual reality, games, robotics, sign language recognition, etc. Gesture recognition has the potential to offer a natural way of communication between humans and machines. The most popular methods in human gesture application are hand gesture recognition. Several of hand gesture recognition methods still have the same problem when trying to recognize hand gesture accurately because of the existence of the uncontrolled environment problem, especially complex background. This study offered a method that can handle complex background in hand gesture recognition by analyzing pattern in spatio temporal domain. The system could reduce irregular noise in complex background significantly that affected to recognize the hand gesture. The experiment had been conducted using 180 videos with a total of 15 gestures. The result showed that the proposed method could recognize 15 hand gestures with accuracy of 98.33%.

Hand gesture recognition is an important area of research in the field of Human Computer Interaction (HCI). The geometric attributes of the hand play an important role in hand shape reconstruction and gesture recognition. That said, fingertips are one of the important attributes for the detection of hand gestures and can provide valuable information from hand images. Many methods are available in scientific literature for fingertips detection with an open hand but very poor results are available for fingertips detection when the hand is closed. This paper presents a new method for the detection of fingertips in a closed hand using the corner detection method and an advanced edge detection algorithm. It is important to note that the skin color segmentation methodology did not work for fingertips detection in a closed hand. Thus the proposed method applied Gabor filter techniques for the detection of edges and then applied the corner detection algorithm for the detection of fingertips through the edges. To check the accuracy of the method, this method was tested on a vast number of images taken with a webcam. The method resulted in a higher accuracy rate of detections from the images. The method was further implemented on video for testing its validity on real time image capturing. These closed hand fingertips detection would help in controlling an electro-mechanical robotic hand via hand gesture in a natural way.

In today’s ever-evolving healthcare landscape, our objective is to bring technology closer to human experience. In this paper, a vision-based hand gesture recognition (HGR) system is introduced as a human computer interface (HCI) that transforms natural movements into intuitive medium for device and application manipulation. This innovative approach is designed to empower the elderly and enhance patient monitoring by offering a seamless, nonintrusive interface. By bridging compassion with cutting-edge technology, this approach aims to redefine care and create a more connected, responsive healthcare environment. The study begins by reviewing both traditional and contemporary HGR methodologies, with a focus on vision-based systems, which have shown greater potential for practical applications compared to conventional sensor-based systems. Traditional approaches, such as contour analysis, colour segmentation, and template matching, have demonstrated limitations. In contrast, deep learning approaches have gained mass adaptation in vision-based systems for their ability to learn hierarchical features and better handling complex, dynamic scenarios. However, existing models struggle in real-time performance due to computational inefficiencies and environmental variations. This study introduces a deep learning-based detection framework which enables the practical application of HGR. For example, it supports the development of gesture-based virtual assistants for interacting with digital devices and allows elderly residents in care homes to give instructions to remote caregivers. The framework consists of two deep learning models, one for region of interest (ROI) detection and another for classification. The recognition process begins with capturing RGB images, chosen for versatility, affordability, and compatibility, which then passes through YOLOv8 (You Only Look Once version 8) for detection. The YOLOv8 model is trained on a subset, only 4.5%, of the HaGRID dataset, comprises over 550,000 RGB images, to accurately locate hand regions. Once detection is complete, several image processing techniques are applied to overcome typical HGR constraints. These include ROI extraction and gray scaling for faster computation, resizing for model optimization and distance viability, HSV conversion for lighting independence, and histogram equalization for enhanced feature extraction. The processed image is then fed into a convolutional neural network (CNN) for gesture classification. The network is designed with several convolutional layers, followed by dense layers with ReLU activation to introduce non-linearity, and SoftMax activation for classification. To improve the model's robustness, data augmentation techniques such as random flipping and rotation are applied. The HGR system demonstrates a staggering accuracy of 97% for training & validation. The proposed framework effectively addresses the limitations of conventional HGR systems—many of which have been developed and tested in controlled environments—demonstrating its applicability in real-life situations.

Hand gesture recognition is a crucial aspect of how we interact with computers, allowing machines to understand and respond to our movements in a natural way. This paper explores an advanced method for recognising hand gestures using a 2D convolutional neural network (2D CNN), a type of deep learning technology. We built and tested our model on a specialised dataset, evaluating its performance through several metrics, including loss function, accuracy, precision, recall, root mean squared error (RMSE), mean absolute error (MAE), and mean squared error (MSE). We compared two different data splitting ratios, 80:20 and 70:30, to determine which one performed better. Our approach effectively captures the complexities of hand gestures, thanks to the spatiotemporal features in the input data. To enhance the model's robustness against varying conditions, such as lighting, backgrounds, and hand positions, we also employed data augmentation techniques. Our findings indicate that the model achieved an impressive accuracy of 80% and a precision of 85% on the training data, along with solid results on the testing data, including an accuracy of 79%. Ultimately, the 80:20 data splitting ratio showed the best overall performance, highlighting its effectiveness for our gesture recognition task.

The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. However, real-time recognition of dynamic micro hand gestures from video streams is challenging for in-vehicle scenarios since (i) the gestures should be performed naturally without distracting the driver, (ii) micro hand gestures occur within very short time intervals at spatially constrained areas, (iii) the performed gesture should be recognized only once, and (iv) the entire architecture should be designed lightweight as it will be deployed to an embedded system. In this work, we propose an HCI system for dynamic recognition of driver micro hand gestures, which can have a crucial impact in automotive sector especially for safety related issues. For this purpose, we initially collected a dataset named Driver Micro Hand Gestures (DriverMHG), which consists of RGB, depth and infrared modalities. The challenges for dynamic recognition of micro hand gestures have been addressed by proposing a lightweight convolutional neural network (CNN) based architecture which operates online efficiently with a sliding window approach. For the CNN model, several 3-dimensional resource efficient networks are applied and their performances are analyzed. Online recognition of gestures has been performed with 3D-MobileNetV2, which provided the best offline accuracy among the applied networks with similar computational complexities. The final architecture is deployed on a driver simulator operating in real-time. We make DriverMHG dataset and our source code publicly available 1.1https://www.mmk.ei.tum.de/DriverMHG/

Hand gesture is one of the method used in sign language for non-verbal communication. It is most commonly used by deaf & dumb people who have hearing or speech problems to communicate among themselves or with normal people. Various sign language systems has been developed by many makers around the world but they are neither flexible nor cost-effective for the end users. Hence in this paper introduced software which presents a system prototype that is able to automatically recognize sign language to help deaf and dumb people to communicate more effectively with each other or normal people. Pattern recognition and Gesture recognition are the developing fields of research. Being a significant part in nonverbal communication hand gestures are playing key role in our daily life. Hand Gesture recognition system provides us an innovative, natural, user friendly way of communication with the computer which is more familiar to the human beings. By considering in mind the similarities of human hand shape with four fingers and one thumb, the software aims to present a real time system for recognition of hand gesture on basis of detection of some shape based features like orientation, Centre of mass centroid, fingers status, thumb in positions of raised or folded fingers of hand

Hand gesture for communication has proven effective for humans, and active research is ongoing in replicating the same success in computer vision systems. Human–computer interaction can be significantly improved from advances in systems that are capable of recognizing different hand gestures. In contrast to many earlier works, which consider the recognition of significantly differentiable hand gestures, and therefore often selecting a few gestures from the American Sign Language (ASL) for recognition, we propose applying deep learning to the problem of hand gesture recognition for the whole 24 hand gestures obtained from the Thomas Moeslund’s gesture recognition database. We show that more biologically inspired and deep neural networks such as convolutional neural network and stacked denoising autoencoder are capable of learning the complex hand gesture classification task with lower error rates. The considered networks are trained and tested on data obtained from the above-mentioned public database; results comparison is then made against earlier works in which only small subsets of the ASL hand gestures are considered for recognition.

Gesture recognition is employed in human-machine communications, enhancing human life with impairments or who depend on non-verbal instructions. Hand gestures role an important role in the field of assistive technology for persons with visual impairments, whereas an optimum user communication design is of major importance. Many authors with substantial development for gesture recognition modeled several methods by using deep learning (DL) methods. This article introduces a Robust Gesture Sign Language Recognition Using Chicken Earthworm Optimization with Deep Learning (RSLR-CEWODL) approach. The projected RSLR-CEWODL algorithm majorly focuses on the recognition and classification of sign language. To accomplish this, the presented RSLR-CEWODL technique utilizes a residual network (ResNet-101) model for feature extraction. For optimal hyper parameter tuning process, the presented RSLR-CEWODL algorithm exploits the CEWO algorithm. Besides, the RSLR-CEWODL technique uses a whale optimization algorithm (WOA) with deep belief network (DBN) method for the sign language recognition method. The simulation result of the RSLR-CEWODL algorithm is tested using sign language datasets and the outcome was measured under various measures. The simulation values demonstrated the enhancements of the RSLR-CEWODL technique over other methodologies.

Abstract— Sign language is an essential mode of communication for individuals with hearing impairments, yet real-time translation remains a challenge due to complex hand gestures, facial expressions, and language variations. Traditional deep learning approaches, such as CNNs and RNNs, struggle with sequential dependencies and spatial feature extraction, limiting recognition accuracy. Recent advancements in Vision Transformers have significantly improved image-based learning by utilizing self- attention mechanisms to capture both spatial and temporal dependencies, making them highly effective for gesture recognition. This paper presents a bidirectional Sign Language Recognition and Translation System that employs ViT for sign recognition and a Pre-Recorded Gesture Database for text-to-sign conversion. The system captures real-time video input, extracts gesture features using ViT’s attention-based encoding, and converts recognized gestures into text. Conversely, it maps typed text to a corresponding pre-recorded sign animation, ensuring smooth and natural communication. By eliminating the need for gloss-based intermediaries and improving processing efficiency, the proposed system enhances accuracy, computational efficiency, and real-time performance, offering a scalable. solution for bridging communication gaps in the hearing-impaired community. KEYWORDS— SIGN LANGUAGE RECOGNITION, VISION TRANSFORMER, GESTURE-TO-TEXT, TEXT- TO- GESTURE, DEEP LEARNING, TRANSFORMER MODELS, REAL-TIME TRANSLATION

Hand gesture recognition is a quickly developing field with many uses in human-computer interaction, sign language recognition, virtual reality, gaming, and robotics. This paper reviews different ways to model hands, such as vision-based, sensor-based, and data glove-based techniques. It emphasizes the importance of accurate hand modeling and feature extraction for capturing and analyzing gestures. Key features like motion, depth, color, shape, and pixel values and their relevance in gesture recognition are discussed. Challenges faced in hand gesture recognition include lighting variations, complex backgrounds, noise, and real-time performance. Machine learning algorithms are used to classify and recognize gestures based on extracted features. The paper emphasizes the need for further research and advancements to improve hand gesture recognition systems' robustness, accuracy, and usability. This review offers valuable insights into the current state of hand gesture recognition, its applications, and its potential to revolutionize human-computer interaction and enable natural and intuitive interactions between humans and machines. In simpler terms, hand gesture recognition is a way for computers to understand what people are saying with their hands. It has many potential applications, such as allowing people to control computers without touching them or helping people with disabilities communicate. The paper reviews different ways to develop hand gesture recognition systems and discusses the challenges and opportunities in this area.