Hand Gesture Recognition via Lightweight VGG16 and Ensemble Classifier

MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

In today’s ever-evolving healthcare landscape, our objective is to bring technology closer to human experience. In this paper, a vision-based hand gesture recognition (HGR) system is introduced as a human computer interface (HCI) that transforms natural movements into intuitive medium for device and application manipulation. This innovative approach is designed to empower the elderly and enhance patient monitoring by offering a seamless, nonintrusive interface. By bridging compassion with cutting-edge technology, this approach aims to redefine care and create a more connected, responsive healthcare environment. The study begins by reviewing both traditional and contemporary HGR methodologies, with a focus on vision-based systems, which have shown greater potential for practical applications compared to conventional sensor-based systems. Traditional approaches, such as contour analysis, colour segmentation, and template matching, have demonstrated limitations. In contrast, deep learning approaches have gained mass adaptation in vision-based systems for their ability to learn hierarchical features and better handling complex, dynamic scenarios. However, existing models struggle in real-time performance due to computational inefficiencies and environmental variations. This study introduces a deep learning-based detection framework which enables the practical application of HGR. For example, it supports the development of gesture-based virtual assistants for interacting with digital devices and allows elderly residents in care homes to give instructions to remote caregivers. The framework consists of two deep learning models, one for region of interest (ROI) detection and another for classification. The recognition process begins with capturing RGB images, chosen for versatility, affordability, and compatibility, which then passes through YOLOv8 (You Only Look Once version 8) for detection. The YOLOv8 model is trained on a subset, only 4.5%, of the HaGRID dataset, comprises over 550,000 RGB images, to accurately locate hand regions. Once detection is complete, several image processing techniques are applied to overcome typical HGR constraints. These include ROI extraction and gray scaling for faster computation, resizing for model optimization and distance viability, HSV conversion for lighting independence, and histogram equalization for enhanced feature extraction. The processed image is then fed into a convolutional neural network (CNN) for gesture classification. The network is designed with several convolutional layers, followed by dense layers with ReLU activation to introduce non-linearity, and SoftMax activation for classification. To improve the model's robustness, data augmentation techniques such as random flipping and rotation are applied. The HGR system demonstrates a staggering accuracy of 97% for training & validation. The proposed framework effectively addresses the limitations of conventional HGR systems—many of which have been developed and tested in controlled environments—demonstrating its applicability in real-life situations.

Gesture recognition is crucial for enhancing human-computer interaction and is particularly pivotal in rehabilitation contexts, aiding individuals recovering from physical impairments and significantly improving their mobility and interactive capabilities. However, current wearable hand gesture recognition approaches are often limited in detection performance, wearability, and generalization. We thus introduce EchoGest, a novel hand gesture recognition system based on soft, stretchable, transparent artificial skin with integrated ultrasonic waveguides. Our presented system is the first to use soft ultrasonic waveguides for hand gesture recognition. EcoflexTM 00-31 and EcoflexTM 00-45 Near ClearTM silicone elastomers were employed to fabricate the artificial skin and ultrasonic waveguides, while 0.1 mm diameter silver-plated copper wires connected the transducers in the waveguides to the electrical system. The wires are enclosed within an additional elastomer layer, achieving a sensing skin with a total thickness of around 500 μ m. Ten participants wore the EchoGest system and performed static hand gestures from two gesture sets: 8 daily life gestures and 10 American Sign Language (ASL) digits 0-9. Leave-One-Subject-Out Cross-Validation analysis demonstrated accuracies of 91.13% for daily life gestures and 88.5% for ASL gestures. The EchoGest system has significant potential in rehabilitation, particularly for tracking and evaluating hand mobility, which could substantially reduce the workload of therapists in both clinical and home-based settings. Integrating this technology could revolutionize hand gesture recognition applications, from real-time sign language translation to innovative rehabilitation techniques.

Researchers have recently focused their attention on vision-based hand gesture recognition. However, due to several constraints, achieving an effective vision-driven hand gesture recognition system in real time has remained a challenge. This paper aims to uncover the limitations faced in image acquisition through the use of cameras, image segmentation and tracking, feature extraction, and gesture classification stages of vision-driven hand gesture recognition in various camera orientations. This paper looked at research on vision-based hand gesture recognition systems from 2012 to 2022. Its goal is to find areas that are getting better and those that need more work. We used specific keywords to find 108 articles in well-known online databases. In this article, we put together a collection of the most notable research works related to gesture recognition. We suggest different categories for gesture recognition-related research with subcategories to create a valuable resource in this domain. We summarize and analyze the methodologies in tabular form. After comparing similar types of methodologies in the gesture recognition field, we have drawn conclusions based on our findings. Our research also looked at how well the vision-based system recognized hand gestures in terms of recognition accuracy. There is a wide variation in identification accuracy, from 68% to 97%, with the average being 86.6 percent. The limitations considered comprise multiple text and interpretations of gestures and complex non-rigid hand characteristics. In comparison to current research, this paper is unique in that it discusses all types of gesture recognition techniques.

In this research work, an efficient sign language recognition tool for e-learning has been proposed with a new type of feature set based on angle and lines. This feature set has the ability to increase the overall performance of machine learning algorithms in an efficient way. The hand gesture recognition based on these features has been implemented for usage in real-time. The feature set used hand landmarks, which were generated using media-pipe (MediaPipe) and open computer vision (openCV) on each frame of the incoming video. The overall algorithm has been tested on two well-known ASL-alphabet (American Sign Language) and ISL-HS (Irish Sign Language) sign language datasets. Different machine learning classifiers including random forest, decision tree, and naïve Bayesian have been used to classify hand gestures using this unique feature set and their respective results have been compared. Since the random forest classifier performed better, it has been selected as the base classifier for the proposed system. It showed 96.7% accuracy with ISL-HS and 93.7% accuracy with ASL-alphabet dataset using the extracted features.

There has been growing interest in the development of new approaches and technologies for bridging the human-computer barrier. Hand gesture recognition is considered as an interaction technique having potential to communicate with machines. Human computer interaction (HCI) was never an easy task and lots of approaches are available to build such systems. Hand gesture recognition (HGR) using wearable data glove provides a solution to build a HCI system, but lags in terms of its computational time and poor interface. Pattern matching is one more solution which uses vision based techniques and provides strong interface to build HCI systems. But again, it requires complex algorithms which takes lots of computational time and hence limits its use in real time HCI applications. In this paper, we presented an eccentric approach for hand gesture recognition which is simple, fast and user independent and can be used to develop real time HCI applications. Based on proposed algorithm we built a system for Indian Sign Language recognition which converts Indian Sign numbers into text. The algorithm first captures the image of single handed gesture of speech/hearing impaired person using simple webcam and then using our proposed algorithm it classifies the gesture into its appropriate class. It uses simple logical conditions for gesture classification which make its use in real time HCI applications.

The Shape of Air: American Sign Language as Narrative Prosthesis in 21st Century North American Media

Hand gesture recognition is a rich area of research and covers a wide scope of applications from human machine interaction (e.g in games) to Deaf people computer interface and from 3D animationto control of mechanical systems. In this work a new algorithm for hand gesture recognition is proposed and evaluated. This algorithm uses moment invariants for feature extraction and neural networks for classification. To evaluate the algorithm a subset from the ASL (American Sign Language) is used, this subset consists of the 26 American one-handed sign language alphabet as a training set for the system. The system then tested using images that are different from the training set in size,orientation and position and the results for these tests are presented and discussed.

This paper presents an algorithm of Hand Gesture Recognition by using Dynamic Time Warping methodology. The system consists of three modules: real time detection of face region and two hand regions, tracking the hands trajectory both in terms of direction among consecutive frames as well as distance from the centre of the frame and gesture recognition based on analyzing variations in the hand locations along with the centre of the face. The proposed technique of ours overcomes not only the limitations of a glove based approach but also most of the vision based approach concerning different illumination conditions, background complexity and distance from camera which is up to 2 meters. Also by using Dynamic Time Warping Algorithm which finds the optimal alignment between the stored database features and query features, improvement in recognition accuracy is observed compared to conventional methods. Experimental results show that the accuracy is 90% in recognizing 24 gestures based on Indian Sign Language.

Hand gesture for communication has proven effective for humans, and active research is ongoing in replicating the same success in computer vision systems. Human–computer interaction can be significantly improved from advances in systems that are capable of recognizing different hand gestures. In contrast to many earlier works, which consider the recognition of significantly differentiable hand gestures, and therefore often selecting a few gestures from the American Sign Language (ASL) for recognition, we propose applying deep learning to the problem of hand gesture recognition for the whole 24 hand gestures obtained from the Thomas Moeslund’s gesture recognition database. We show that more biologically inspired and deep neural networks such as convolutional neural network and stacked denoising autoencoder are capable of learning the complex hand gesture classification task with lower error rates. The considered networks are trained and tested on data obtained from the above-mentioned public database; results comparison is then made against earlier works in which only small subsets of the ASL hand gestures are considered for recognition.

Communication for mute people has always been very difficult. Mute people use hand gestures, also known as sign language to communicate with normal people. Hands gestures have their assigned meanings which may differ from person to person and hence cannot be understood by normal people. To overcome this difficulty many vision-based hand gesture recognition systems and data glove based hand gesture recognition systems have been proposed. This paper illustrates about two different techniques of vision-based hand gesture recognition and one data glove based technique. The vision — based techniques are static hand gesture recognition technique and real-time hand gesture recognition technique. In both the techniques, MATLAB software is used for processing input images and no dataset is used for decision making which makes this system more accurate than the existing system designs. In data glove based technique, the glove consists of five flex sensors. The change in resistance of flex sensor is used to recognize the hand gesture. All the three techniques are performed on 10 subjects and compared to find the most accurate technique. It was found that both the vision based techniques showed 100% accuracy in bright lighting condition with a white background while the data glove based technique showed an accuracy of 86%. This which clearly shows that mentioned vision based is more stable and reliable compared to the data glove based technique.

Contents: Preface. Part I: Reminiscences. L. Fant, Two Memorable Meals With Ursula and Ed. R.M. Battison, American Sign Language Linguistics 1970-1980: Memoir of a Renaissance. B. Bragg, A Fish Story for Ursula and Other Remembrances. Part II: Historical and Comparative Analyses of Sign Languages. J. Woodward, Sign Languages and Sign Language Families in Thailand and Vietnam. D. McKee, G. Kennedy, Lexical Comparisons of Signs From American, Australian, British, and New Zealand Sign Languages. H. Lane, R.C. Pillard, M. French, Origins of the American Deaf-World: Assimilating and Differentiating Societies and Their Relation to Genetic Patterning. Part III: Language in the Visual-Spatial Modality. E.L. Newport, T. Supalla, Sign Language Research at the Millennium. P. Siple, Attentional Resources and Working Memory: A New Framework for the Study of the Impact of Deafness on Cognition. M. Wilson, K. Emmorey, When Does Modality Matter? Evidence From ASL on the Nature of Working Memory. P.B. Braem, T. Bram, A Pilot Study of the Expressive Gestures Used by Classical Orchestra Conductors. N. Frishberg, An Interpreter Creates the Space. Part IV: Linguistic Analysis of Sign Languages. S.D. Fischer, More Than Just Handwaving: The Mutual Contributions of Sign Language and Linguistics. R.B. Wilbur, Phonological and Prosodic Layering of Nonmanuals in American Sign Language. R.C. Loew, C.T. Akamatsu, M. Lanaville, A Two-Handed Manual Alphabet in the United States. E. Pizzuto, V. Volterra, Iconicity and Transparency in Sign Languages: A Cross-Linguistic Cross-Cultural View. M. Kuntze, Codeswitching in ASL and Written English Language Contact. S.K. Liddell, Indicating Verbs and Pronouns: Pointing Away From Agreement. Part V: Language Acquisition. J.G. Bettger, Viewing Deaf Children in a New Way: Implications of Bellugi and Klima's Research for Education. R.P. Meier, Shared Motoric Factors in the Acquisition of Sign and Speech. A. Abrahamsen, Explorations of Enhanced Gestural Input to Children in the Bimodal Period. D. Lillo-Martin, Early and Late in Language Acquisition: Aspects of the Syntax and Acquisition of Wh-Questions in American Sign Language. J. Reilly, Bringing Affective Expression Into the Service of Language: Acquiring Perspective Marking in Narratives. C.A. Padden, V.L. Hanson, Search for the Missing Link: The Development of Skilled Reading in Deaf Children. L.A. Petitto, On the Biological Foundations of Human Language. Part VI: The Neural Organization of Sign Language. A.R. Damasio, H. Damasio, Language and the Brain. D.P. Corina, Some Observations Regarding Paraphasia in American Sign Language. H. Poizner, D. Brentari, M.E. Tyrone, J. Kegl, The Structure of Language as Motor Behavior: Clues From Signers With Parkinson's Disease. V.A. Fromkin, On the Uniqueness of Language.

Sign language recognition is one of the vital research problems in computer vision systems. Sign language is used as a natural mode of communication by speech and hearing-impaired people in their daily life activities. This paper proposes a useful method to recognize static gestures with the application of Deep Learning. The pre-trained deep learning models trained on the ImageNet data and transfer learning technique are used to recognize 24 static alphabets of the American Sign Language (ASL) dataset. The publicly available benchmark American Fingerspelling dataset is used for the training of models. The paper presents the comparison of Convolutional Neural Network (CNN) models like VGG16, VGG19 and MobileNet on the same dataset and achieved average validation accuracy of 94.9% using the MobileNet model that gave good predictions using cross-validation. Also, the model has performed better than previous studies on the same dataset. The evaluation of the model performance is presented in the confusion matrix for the best trial fold that gives the prediction results of all 24 alphabets as recognition accuracy in percentage form.

In human interactions, hands are a powerful way of expressing information\nthat, in some cases, can be used as a valid substitute for voice, as it happens\nin Sign Language. Hand gesture recognition has always been an interesting topic\nin the areas of computer vision and multimedia. These gestures can be\nrepresented as sets of feature vectors that change over time. Recurrent Neural\nNetworks (RNNs) are suited to analyse this type of sets thanks to their ability\nto model the long term contextual information of temporal sequences. In this\npaper, a RNN is trained by using as features the angles formed by the finger\nbones of human hands. The selected features, acquired by a Leap Motion\nController (LMC) sensor, have been chosen because the majority of human\ngestures produce joint movements that generate truly characteristic corners. A\nchallenging subset composed by a large number of gestures defined by the\nAmerican Sign Language (ASL) is used to test the proposed solution and the\neffectiveness of the selected angles. Moreover, the proposed method has been\ncompared to other state of the art works on the SHREC dataset, thus\ndemonstrating its superiority in hand gesture recognition accuracy.\n

Publication in the conference proceedings of EUSIPCO, Bucharest, Romania, 2012