Isolated Sign Language Recognition Based on Deep Learning

Ensemble transformer-based word-level sign language recognition with multi-modal input fusion

Isolated Sign Language Recognition (ISLR), which focuses on identifying individual signs from sign language videos, presents substantial challenges due to small and ambiguous hand regions, high visual similarity among signs, and large intra-class variability. This study investigates the adaptability of YOLO-Act, a unified spatiotemporal detection framework originally developed for generic action recognition in videos, when applied to large-scale sign language benchmarks. YOLO-Act jointly performs signer localization (identifying the person signing within a video) and action classification (determining which sign is performed) directly from RGB sequences, eliminating the need for pose estimation or handcrafted temporal cues. We evaluate the model on the WLASL2000 and MSASL1000 datasets for American Sign Language recognition, achieving Top-1 accuracies of 67.07% and 81.41%, respectively. The latter represents a 3.55% absolute improvement over the best-performing baseline without pose supervision. These results demonstrate the strong cross-domain generalization and robustness of YOLO-Act in complex multi-class recognition scenarios.

Human perception of sign language can serve as inspiration for the improvement of automatic recognition systems. Experiments with human signers show that sign language signs contain redundancy over time. In this article, experiments are conducted to investigate whether comparable redundancies also exist for an automatic sign language recognition system. Such redundancies could be exploited, for example, by reserving more processing resources for the more informative phases of a sign, or by discarding uninformative phases. In the experiments, an automatic system is trained and tested on isolated fragments of sign language signs. The stimuli used were similar to those of the human signer experiments, allowing us to compare the results. The experiments show that redundancy over time exists for the automatic recognizer. The central phase of a sign is the most informative phase, and the first half of a sign is sufficient to achieve a recognition performance similar to that of the entire sign. These findings concur with the results of the human signer studies. However, there are differences as well, most notably the fact that human signers score better on the early phases of a sign than the automatic system. The results can be used to improve the automatic recognizer, by using only the most informative phases of a sign as input.

Convolutional neural networks (CNN) have provided great advances for the task of sign language recognition (SLR). However, recurrent neural networks (RNN) in the form of long–short-term memory (LSTM) have become a means for providing solutions to problems involving sequential data. This research proposes the development of a sign language translation system that converts Panamanian Sign Language (PSL) signs into text in Spanish using an LSTM model that, among many things, makes it possible to work with non-static signs (as sequential data). The deep learning model presented focuses on action detection, in this case, the execution of the signs. This involves processing in a precise manner the frames in which a sign language gesture is made. The proposal is a holistic solution that considers, in addition to the seeking of the hands of the speaker, the face and pose determinants. These were added due to the fact that when communicating through sign languages, other visual characteristics matter beyond hand gestures. For the training of this system, a data set of 330 videos (of 30 frames each) for five possible classes (different signs considered) was created. The model was tested having an accuracy of 98.8%, making this a valuable base system for effective communication between PSL users and Spanish speakers. In conclusion, this work provides an improvement of the state of the art for PSL–Spanish translation by using the possibilities of translatable signs via deep learning.

A lightweight network-based sign language robot with facial mirroring and speech system

Sign language identification and recognition technique is composed by the gesture images detection and the hand gestures recognition. Gesture images detection is to locate the image part of palm and fingers from the captured pictures, and rotating them to the appropriate gesture posture. Both of them are the important pre-processing for sign language identification and recognition. Lose them, the correctness rate of the sign language recognition algorithms will be dropped down to an unacceptable level. This paper presents novel processing algorithms for the gesture images detection and recognition. In the detection process, it rotates an askew gesture to right position, and to delete the elbow and forearm parts from the captured pictures. In the recognition process, it includes two phases with the model construction and the sign language identification. In the model construction phase, the static hand gesture of sign language is constructed by the Gaussian mixture model, and the unknown gesture image is identified by Gaussian model match. Based on this presented static sign language detection and recognition algorithms, the correct recognition rate is about 94% in average.

A survey on recent advances in Sign Language Production

Sign language is the most natural and expressive way for the hearing impaired. Because of this, automatic sign language recognition has long attracted vision researchers. It offers enhancement of communication capabilities for the speech and hearing impaired, promising improved social opportunities and integration. This paper describes a gesture recognition system which can recognize wide classes of hand gesture in a vision based setup. Experimental results demonstrate that our proposed recognition system can be used reliably in recognizing some signs of native Indian sign language

A feature covariance matrix with serial particle filter for isolated sign language recognition

We present a novel and robust system for recognizing two handed motion based gestures performed within continuous sequences of sign language. While recognition of valid sign sequences is an important task in the overall goal of machine recognition of sign language, detection of movement epenthesis is important in the task of continuous recognition of natural sign language. We propose a framework for recognizing valid sign segments and identifying movement epenthesis. Our system utilizes a single HMM threshold model, per hand, to detect movement epenthesis. Further to this, we develop a novel technique to utilize the threshold model and dedicated gesture HMMs to recognize gestures within continuous sign language sentences. Experiments show that our system has a gesture detection ratio of 0.956 and a reliability measure of 0.932 when spotting 8 different signs from 240 video clips.

Sign Language Recognition (SLR) has become an essential area of research due to its potential to promote understanding between the deaf and hearing communities through communication. This paper provides an in-depth study of various methodologies and models employed in SLR, focusing on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). We analyze their application to datasets from various sign languages, such as Arabic Sign Language (ArSL), American Sign Language (ASL), and British Sign Language (BSL), and explore how these models improve the recognition of dynamic, multi-dimensional hand gestures. This research not only advances the understanding of deep learning applications in sign language recognition but also addresses critical challenges in data processing and real-time applications, paving the way for inclusive technologies in informatics and human-computer interaction. Despite the progress in applying deep learning techniques to SLR, several challenges remain, particularly in dataset limitations, handling large vocabularies, and ensuring consistent performance across diverse environments and signers. The paper also investigates the broader applications of SLR, such as virtual reality, healthcare, education, and accessibility, and discusses the integration of SLR with human-computer interaction systems. Furthermore, it highlights current limitations in the field, such as difficulties with video data handling, the need for standard datasets, and issues related to training computational models. Finally, the paper outlines future research directions, including developing more robust SLR systems that can function effectively in uncontrolled environments, improving data collection methodologies, and creating real-time, user-friendly applications to assist the community of deaf and hard-of-hearing individuals.

A way of communication on an equal footing with oral and written speech is sign language. Oral speech is a common way of communication, written speech has, with the development of information technologies, been taking up more and more space. In a parallel with them, communication can also take place through sign language, which is, to deaf people, as well as to all those who use it, a natural, simple and easy way to communicate. Relationship towards sign language has changed significantly throughout history from acceptance and isolated use, to complete rejection, and then to encouraging its adoption and emphasizing its importance for the cognitive, emotional, educational, social, and general development of deaf children. Serbian Sign Language (SSL) serves deaf people in Serbia as a means for everyday communication, for expressing desires, willingness, for learning, for intellectual discussions, for expressing personal style. Although the standardization of the Serbian Sign Language was completed in 2015, even nowadays we may still find certain gestures of expression varying in different regions. Different countries have different sign languages that are not reciprocally understood in use. They are distinguished by their own grammar (semantics, morphology, and syntax), different from the grammar of spoken languages. The distinguishing and recognition of sign languages in the world has led to changes in the field of education of deaf children. In bilingual schools, children acquire both sign and spoken languages, and teachers know both of the mentioned languages. The importance of sign language in the education of deaf children is emphasized.

Sign language is the primary mode of communication for individuals with hearing and speech impairments; however, the lack of sign language knowledge among the general population creates a significant communication barrier. Recent advancements in computer vision and deep learning have enabled the development of automated sign language recognition and translation systems. Despite this progress, many existing solutions depend on wearable devices, are computationally expensive, or lack real-time gesture-to-word and speech translation capabilities suitable for practical deployment. This paper presents the development of a sign language translator that maps gestures to words using a vision-based approach. The proposed system captures hand gestures through a camera, recognises sign language gestures using deep learning techniques, and converts them into meaningful text and speech output. The system is designed to be cost-effective, real-time, and user-friendly, eliminating the need for sensor-based gloves. By focusing on gesture-based translation, the proposed approach aims to enhance accessibility and enable effective communication between hearing-impaired individuals and non-signers in real-world environments.

Deaf and hard-of-hearing individuals use sign language as a means of communication. However, those around them, especially family members like the children of deaf adults, may face communication challenges if they are unfamiliar with sign language. This issue has prompted numerous researchers to conduct studies on sign language translation and recognition. However, there is currently no publicly available dataset specifically for Malaysian sign language. This article introduces an image dataset of the Malaysian Sign Language (MySL) hand gestures used in everyday situations. The dataset, named MyWSL2023, comprises 3,500 images of ten static Malaysian sign language words collected from five participants (two males and three females) aged between 20 and 21 years old. The data collection took place indoors under normal lighting conditions. The MyWSL2023 dataset, which has been made freely accessible to all researchers, serves as a valuable resource for not only investigating and developing automated systems for hearing-impaired and deaf individuals but also gesture and sign language recognition using vision-based methods. The dataset can be accessed for free at https://data.mendeley.com/datasets/zvk55p7ktd.

Human–Computer Interaction (HCI) refers to the interaction between the computers and human. One of the most important applications of (HCI) is sign language recognition. Several research works aimed to interpret and translate the sign language to a spoken language to help the hear impaired persons in integrating with communities. Sign language is the main way of communication for the deaf persons and hearing impaired which enable them to communicate with their societies and between each other’s. According to the World Health Organization, there are 466 million hearing loss people (i.e. 5% of the world population), 432 million or 93% of them are adults and 34 million or 17% of them are children. The hearing-impaired persons often need to the same level of mental capability as the normal persons. The main problem is the most of the hearing persons who cannot understand the sign language and most of hearing impaired cannot read or write our spoken language, this represents a barrier between the deaf persons and their societies so developing an automatic sign language recognition system is very necessary. This research introduces dynamic Arabic Sign Language recognition system using Microsoft Kinect. The recognition depends on using two machine learning algorithms (a) Decision Tree and (b) Bayesian Network then applied Ada-Boosting technique to enhance the recognition of the system, we compared the results with two direct matching techniques: (a) Dynamic Time Wrapping and Hidden Markov Model the system was applied on 42 Arabic gestures related to medical field. The experimental results showed that the proposed system recognition rate reached 91.18% for Decision Tree classifier, 92.50% for Bayesian classifier and 93.7% after applying Ada-Boosting.