Recognition of Sign Language Using Leap Motion Controller Data

ABSTRACTSign language is a communication tool for deaf and dumb people that includes known signs or body gestures to transfer meanings. It uses shapes, directions, movements of hands, and also facial expressions. A sign not only transmits a word but also conveys a tone. Many of the deaf people are not only able to speak, but also not able to write or read a language, so developing sign language translation or in other words sign language recognition (SLR) system can be very vital in their life. The SLR is extremely desired because of its capability to overcome the obstacles between deaf and normal people. It is one of the most important research fields in the human computer interaction studies. Hence, this paper presents an overview of the recent main research works with the vision-based SLR system, and the existing recognition techniques are discussed. Next, we focus on video-based SLR system and perform continuous SLR within video sequences.

Sign language identification and recognition technique is composed by the gesture images detection and the hand gestures recognition. Gesture images detection is to locate the image part of palm and fingers from the captured pictures, and rotating them to the appropriate gesture posture. Both of them are the important pre-processing for sign language identification and recognition. Lose them, the correctness rate of the sign language recognition algorithms will be dropped down to an unacceptable level. This paper presents novel processing algorithms for the gesture images detection and recognition. In the detection process, it rotates an askew gesture to right position, and to delete the elbow and forearm parts from the captured pictures. In the recognition process, it includes two phases with the model construction and the sign language identification. In the model construction phase, the static hand gesture of sign language is constructed by the Gaussian mixture model, and the unknown gesture image is identified by Gaussian model match. Based on this presented static sign language detection and recognition algorithms, the correct recognition rate is about 94% in average.

Sign Language is a meansof communication between audibly challenged people. To provide an interface between the audibly challenged community and the rest of the world we need Sign Language translators. A sign language recognition system computerizes the work of a sign language trEvery Sign Language Recognition (SLR) System is trained to recognizespecific sets of signs and they correspondingly output the sign in the required format. These SLR systems are built with powerful image processing techniques. The sign language recognition systems are capable of recognizing a specific set of signing gestures and output the corresponding text/audio. Most of these systems involve the techniques of detection, segmentation, tracking, gesture recognition and classification. This paperproposes a design for a SLR System.

This article is dedicated to analyzing advanced approaches in temporal modeling and real-time gesture recognition within sign language recognition (SLR) systems. Sign glosses are expressed through the spatio-temporal characteristics of visual information, which requires the use of sequence-processing models for their automatic recognition. The study primarily evaluates the effectiveness of three key models: Long Short-Term Memory (LSTM) networks, Temporal Convolutional Networks (TCN), and Transformer-based architectures. The article also examines methods applied for real-time analysis of sign glosses, including: Sliding window segmentation of video streams; Self-attention mechanisms for identifying dependencies between gestures; Gloss mapping algorithms for linking sign movements to linguistic units; Ontological integration techniques for enhancing semantic accuracy. Practical results indicate that combining temporal modeling with semantic analysis and contextual verification algorithms ensures continuous and high-accuracy recognition of sign movements. In particular, multimodal systems (video + sensor + gloss) utilizing Transformer-based approaches achieved superior performance in real-time conversion of continuous sign gloss streams into text. The findings of this study hold practical significance for the development of smart assistive devices for automatic sign language translation, interactive interfaces for hearing-impaired users, and specialized SLR platforms for educational and instructional purposes.

Sign Language Recognition systems have been thoroughly researched in recent times to bridge the gap between the signer and non-signer community. This paper focuses on fulfilling the need for a thorough review of the works done in building such systems. The systems discussed herein for static Sign Language Recognition are broadly classified on Support Vector Machine, Convolutional Neural Network as classifiers. The identified phases for the reviewed systems include preprocessing, feature extraction and classification. The methods discussed in these phases include Principal Component Analysis, Histograms of Orientations Gradient, Hybrid Sign Invariant Feature Transformation. Further the limitations and improvements in approaches are relatively portrayed to give a better analysis for future work. The overall aim for this paper is to provide insight into making effective systems for Sign Language Recognition. This will make these systems easily available for educating the masses about static Sign Language. Thus, sign language will no longer remain a barrier.

An optical modeless Sign Language Recognition (SLR) system is presented. The system uses the HAusdorf-Voronoi NETwork (HAVNET), an artificial neural network designed for 2D binary pattern recognition. It uses adaptation of the Hausdorff distance to determine the similarity between an input pattern and a learned representation. A detailed review of the architecture, the learning equations, and the recognition equations for the HAVNET network are presented. Competitive learning has been implemented in training the network using a nearest-neighbor technique. The SLR system is applied to the optical recognition of 24 static symbols from the American Sign Language convention. The SLR system represents the target images in a 80 X 80 pixel format. The implemented HAVNET network classifies the inputs into categories representing each of the symbols, using an output layer of 24 nodes. The network is trained with 5 different formats for each symbol and is tested with all 24 symbols in 15 new formats. Results from the SLR system without competitive training show shape identification problems, when distinguishing symbols with similar shapes. Implementation of competitive learning in the HAVNET neural network improved recognition accuracy on this task to 89%. The hand gestures are identified through a window search algorithm. Feature recognition is obtained from edge enhancement by applying a Laplacian filter and thresholding, which provides robustness to pose, color and background variations.

Sign language recognition (SLR) system is helpful for individuals who can't talk or hear. One can easily communicate with this kind of people using SLR system. In this paper we are focusing on SLR approach where we have apply our proposed algorithm to recognize hand gestures of routine words like Morning, Night, Monday, Tuesday and so forth. Total 10 routine words we have recognize using our proposed technique. For recognize hand pose we have utilized skin color detection algorithm. Further we have applied correlation-coefficient algorithm to distinguish closeness features which are further continue in Neuro-fuzzy(NF) classification algorithm to recognize words. Presented algorithm tried on Matlab. With taken of 10 routine words our proposed framework of techniques got 92% correction rate.

Aiming at the feature of signer-independent sign language recognition the training data complexity caused by mass data and noticeable distinctions between different people data, the weighted KNN/HMM model is presented in this paper. This model is made of two blocks, which is part of sign language classification and recognition. In classing part, the KNN (K-Nearest Neighbor, KNN) is used to learn the training samples. Considering the different contributions of sign language features to pattern classification we give different weight to different characteristics. And the category of test sample is decided by the sum of weighted distance. In recognition part, weighted KNN classification result is taken as the state-input of HMM (Hidden Markov models, HMM) to implement sign language recognition, combine with the ability to temporal data modeling and fuzzy inference of HMM model. Experiment results show that weighted KNN/HMM sign language recognition model are efficient on either recognition speed or recognition rate.

In recent years, for recognizing sign language, several hardware approaches have been developed using the leap motion controller and Kinect sensors. The sensor-based approaches were costly, and the complexity of designing was high. In machine learning approaches, it was found that less accuracy of prediction occurs as compared to other approaches. As a solution, machine learning-based approaches using image processing evolved as an approach with better prediction and accuracy. In this paper, deep learning approach for sign language recognition (SLR) system has been proposed using AlexNet model. AlexNet is pretrained model, and it is being tested on Indian Sign Language (ISL) Dataset pf Stanford University comprising of alphabets (A–Z) and numerals (0–9). The proposed model has yielded an accuracy of 99.6%.KeywordsAlexNetDeep learningSLRRectified linear unit layer

Sign language recognition plays a crucial role in supporting communication between deaf communities and hearing individuals. In particular, Vietnamese Sign Language (VSL) recognition remains a challenging task due to the complexity of hand gestures and limited available datasets. However, most existing approaches address static gestures and dynamic sign phrases as separate recognition problems, often employing different feature representations and independent processing pipelines. This fragmentation increases system complexity and limits scalability for real-time sign language applications. This study proposes a unified hand-landmark-based deep learning framework for recognizing both static and dynamic VSL gestures within a single integrated system. The system begins by detecting hand landmarks through the MediaPipe hand tracking pipeline. Based on temporal motion analysis of landmark sequences, a gesture routing mechanism automatically determines whether the input corresponds to a static gesture or a dynamic sign phrase. Static gestures are classified using a convolutional neural network (CNN), while dynamic gestures are processed using a long short-term memory (LSTM) network to capture temporal dependencies. Experiments were conducted on a VSL dataset consisting of 23 static hand signs and 4 dynamic sign phrases collected from multiple participants with variations in hand shape and gesture execution style. The system performance is evaluated using accuracy, precision, recall, and F1-score. Experimental results demonstrate that the proposed framework achieves an average recognition accuracy of 92% for static gestures and 88.4% for dynamic sign phrases, outperforming traditional machine learning baselines. The proposed system provides a practical and efficient solution for VSL recognition and has potential applications in real-time assistive communication systems.

Sign Language Recognition (SLR) has become an essential area of research due to its potential to promote understanding between the deaf and hearing communities through communication. This paper provides an in-depth study of various methodologies and models employed in SLR, focusing on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). We analyze their application to datasets from various sign languages, such as Arabic Sign Language (ArSL), American Sign Language (ASL), and British Sign Language (BSL), and explore how these models improve the recognition of dynamic, multi-dimensional hand gestures. This research not only advances the understanding of deep learning applications in sign language recognition but also addresses critical challenges in data processing and real-time applications, paving the way for inclusive technologies in informatics and human-computer interaction. Despite the progress in applying deep learning techniques to SLR, several challenges remain, particularly in dataset limitations, handling large vocabularies, and ensuring consistent performance across diverse environments and signers. The paper also investigates the broader applications of SLR, such as virtual reality, healthcare, education, and accessibility, and discusses the integration of SLR with human-computer interaction systems. Furthermore, it highlights current limitations in the field, such as difficulties with video data handling, the need for standard datasets, and issues related to training computational models. Finally, the paper outlines future research directions, including developing more robust SLR systems that can function effectively in uncontrolled environments, improving data collection methodologies, and creating real-time, user-friendly applications to assist the community of deaf and hard-of-hearing individuals.

The essential research related to sign language recognition states that there is a need of remarkable progress in this domain. Selecting features is decisive to gesture recognition, as hand gestures are very fine in shape variation, motion and textures. For static posture recognition, although it is possible to recognize hand posture by extracting some geometric features such as fingertips, finger directions it is not reliable due to self -- occlusion and lighting condition. Silhouette and textures, however, they are inadequate in recognition process. Therefore this work focuses on two of the research problems comprising automatic sign language recognition, namely robust segmentation techniques for consistent detection, preserving the shape contour which is therefore useful for textural feature extraction. The discrimination ability of the two segmentation methods for texture computation is observed and compared by objective parameters. Experiments are done in various camera based images and that it explores the need for prior effective contour segmentation and textural features for constructing a precision based recognition system for sign language.

Sign language is the main communication language for deaf and dumb people. There have many difficulties for deaf and dumb people when they communicate with normal people. Only a few people can use Sign Language proficiently and most of the people don't know how to communicate the deaf people. From the point of view of computer vision, we can develop the sign language recognition for deaf and dumb people. Most of the sign language recognition systems are based on the static sign. In motion-based sign language recognition system, we proposed feature extraction method and recognition using machine learning. Every nation has its own sign language and our proposed system is based on Myanmar National Sign Language.

Communication between hearing-impaired and hearing individuals remains a persistent global challenge, primarily due to the limited understanding of sign language among the general population. This communication gap frequently leads to social exclusion, dependency on human interpreters, and significant difficulties in critical environments such as healthcare, education, and public services. To address this, the current study proposes an innovative Artificial Intelligence (AI)-based real-time sign language recognition and communication support system. Designed to enable inclusive, two-way interaction, the system utilizes advanced computer vision techniques to capture and preprocess hand gestures. Machine learning models, specifically Convolutional Neural Networks (CNNs), interpret these gestures to generate corresponding text and speech output. Conversely, speech input from hearing users is converted into text and animated visual sign representations to ensure seamless bidirectional communication. Furthermore, the application integrates an SOS emergency alert module and an interactive learning interface for basic sign language. By reducing the dependency on human translators and facilitating natural interaction, this proposed system aims to significantly enhance accessibility, user independence, and social inclusion for the hearing-impaired community. Keywords: Sign Language Recognition, Artificial Intelligence, Computer Vision, Bidirectional Communication, Convolutional Neural Networks (CNN), Accessibility, Deep Learning.

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.