Sign language recognition using modified deep learning network and hybrid optimization: a hybrid optimizer (HO) based optimized CNNSa-LSTM approach

In the discipline of hand gesture and dynamic sign language recognition, deep learning approaches with high computational complexity and a wide range of parameters have been an extremely remarkable success. However, the implementation of sign language recognition applications for mobile phones with restricted storage and computing capacities is usually greatly constrained by those limited resources. In light of this situation, we suggest lightweight deep neural networks with advanced processing for real-time dynamic sign language recognition (DSLR). This paper presents a DSLR application to minimize the gap between hearing-impaired communities and regular society. The DSLR application was developed using two robust deep learning models, the GRU and the 1D CNN, combined with the MediaPipe framework. In this paper, the authors implement advanced processes to solve most of the DSLR problems, especially in real-time detection, e.g., differences in depth and location. The solution method consists of three main parts. First, the input dataset is preprocessed with our algorithm to standardize the number of frames. Then, the MediaPipe framework extracts hands and poses landmarks (features) to detect and locate them. Finally, the features of the models are passed after processing the unification of the depth and location of the body to recognize the DSL accurately. To accomplish this, the authors built a new American video-based sign dataset and named it DSL-46. DSL-46 contains 46 daily used signs that were presented with all the needed details and properties for recording the new dataset. The results of the experiments show that the presented solution method can recognize dynamic signs extremely fast and accurately, even in real-time detection. The DSLR reaches an accuracy of 98.8%, 99.84%, and 88.40% on the DSL-46, LSA64, and LIBRAS-BSL datasets, respectively.

Sign language identification and recognition technique is composed by the gesture images detection and the hand gestures recognition. Gesture images detection is to locate the image part of palm and fingers from the captured pictures, and rotating them to the appropriate gesture posture. Both of them are the important pre-processing for sign language identification and recognition. Lose them, the correctness rate of the sign language recognition algorithms will be dropped down to an unacceptable level. This paper presents novel processing algorithms for the gesture images detection and recognition. In the detection process, it rotates an askew gesture to right position, and to delete the elbow and forearm parts from the captured pictures. In the recognition process, it includes two phases with the model construction and the sign language identification. In the model construction phase, the static hand gesture of sign language is constructed by the Gaussian mixture model, and the unknown gesture image is identified by Gaussian model match. Based on this presented static sign language detection and recognition algorithms, the correct recognition rate is about 94% in average.

Sign Language Recognition (SLR) has been an active area of research, but sentence-level SLR remains relatively underexplored. While most studies focus on recognizing individual signs, understanding full sentences presents greater challenges. This research proposes a sentence-level SLR using a combination of 3D Convolutional Neural Networks (3D CNN) for spatio-temporal feature extraction with sequential modeling using Long Short-Term Memory (LSTM) and Bidirectional LSTM (BiLSTM). Connectionist Temporal Classification (CTC) is also used to enable training without word-level annotations. In this study, we used the Indonesian Sign Language (BISINDO) dataset, specifically the DKI Jakarta version, consisting of 900 videos representing 30 sentences, which was expanded to 3600 videos through data augmentation techniques such as speed variation and brightness adjustments. All videos underwent preprocessing to ensure data quality, and Bayesian Optimization was applied for hyperparameter tuning to obtain optimal configurations for each model. Both models were trained with CTC loss and evaluated using Word Error Rate (WER). The 3DCNN-LSTM model achieved a WER result of 59.21%, while the 3DCNN-BiLSTM presents a significantly better performance with a WER of 2.77%. Despite these promising results, the models’ ability to generalize across different signers may require further research, as the dataset used in this research involved only a single signer.

A survey on recent advances in Sign Language Production

Sign Language Recognition (SLR) has become an essential area of research due to its potential to promote understanding between the deaf and hearing communities through communication. This paper provides an in-depth study of various methodologies and models employed in SLR, focusing on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). We analyze their application to datasets from various sign languages, such as Arabic Sign Language (ArSL), American Sign Language (ASL), and British Sign Language (BSL), and explore how these models improve the recognition of dynamic, multi-dimensional hand gestures. This research not only advances the understanding of deep learning applications in sign language recognition but also addresses critical challenges in data processing and real-time applications, paving the way for inclusive technologies in informatics and human-computer interaction. Despite the progress in applying deep learning techniques to SLR, several challenges remain, particularly in dataset limitations, handling large vocabularies, and ensuring consistent performance across diverse environments and signers. The paper also investigates the broader applications of SLR, such as virtual reality, healthcare, education, and accessibility, and discusses the integration of SLR with human-computer interaction systems. Furthermore, it highlights current limitations in the field, such as difficulties with video data handling, the need for standard datasets, and issues related to training computational models. Finally, the paper outlines future research directions, including developing more robust SLR systems that can function effectively in uncontrolled environments, improving data collection methodologies, and creating real-time, user-friendly applications to assist the community of deaf and hard-of-hearing individuals.

A way of communication on an equal footing with oral and written speech is sign language. Oral speech is a common way of communication, written speech has, with the development of information technologies, been taking up more and more space. In a parallel with them, communication can also take place through sign language, which is, to deaf people, as well as to all those who use it, a natural, simple and easy way to communicate. Relationship towards sign language has changed significantly throughout history from acceptance and isolated use, to complete rejection, and then to encouraging its adoption and emphasizing its importance for the cognitive, emotional, educational, social, and general development of deaf children. Serbian Sign Language (SSL) serves deaf people in Serbia as a means for everyday communication, for expressing desires, willingness, for learning, for intellectual discussions, for expressing personal style. Although the standardization of the Serbian Sign Language was completed in 2015, even nowadays we may still find certain gestures of expression varying in different regions. Different countries have different sign languages that are not reciprocally understood in use. They are distinguished by their own grammar (semantics, morphology, and syntax), different from the grammar of spoken languages. The distinguishing and recognition of sign languages in the world has led to changes in the field of education of deaf children. In bilingual schools, children acquire both sign and spoken languages, and teachers know both of the mentioned languages. The importance of sign language in the education of deaf children is emphasized.

This comprehensive review paper delves into the intricate landscape of sign language recognition (SLR) technologies, addressing the pressing need for effective communication solutions for the global population grappling with hearing impairments. With a meticulous examination of the past two decades of SLR research, the paper categorically explores isolated and continuous manual and non-manual SLR, shedding light on diverse modalities and datasets. The review not only emphasizes the significance of sign language as a fully developed language but also underscores the potential of advanced technologies, such as deep learning and unsupervised learning, in revolutionizing SLR. The inclusion of diverse research projects, ranging from real-time Indian Sign Language conversion to text to the development of deep learning models for American Sign Language recognition, adds practical depth to the exploration. The paper's contributions extend to presenting a tabular format for enhanced comprehension, addressing key facets like modality and sign language types, and offering insightful guidance for the trajectory of this burgeoning field. Overall, this review serves as a valuable and timely resource, advocating for continued innovation to foster a more inclusive and interconnected world for individuals with hearing impairments.

Communicating with hearing-impaired individuals poses a significant challenge. However, with the advancement of computer vision, automatic sign language recognition (SLR) is gradually addressing this issue and has made significant improvements. One of the key challenges in SLR lies in accurately capturing and interpreting the subtle nuances and variations in sign language gestures. In this study, our focus is on recognizing isolated sign language using the LSA64 dataset, which is a small-scale dataset of Argentinian isolated sign language. We concatenated CNN and LSTM into an end-to-end sign language recognition model for an isolated sign language recognition dataset, recognition of the Argentine Sign Language dataset (LSA64). We achieved promising results in our study, obtaining a high accuracy rate of nearly 97% while ensuring that the model remained compact in size.

Deaf and hard-of-hearing individuals use sign language as a means of communication. However, those around them, especially family members like the children of deaf adults, may face communication challenges if they are unfamiliar with sign language. This issue has prompted numerous researchers to conduct studies on sign language translation and recognition. However, there is currently no publicly available dataset specifically for Malaysian sign language. This article introduces an image dataset of the Malaysian Sign Language (MySL) hand gestures used in everyday situations. The dataset, named MyWSL2023, comprises 3,500 images of ten static Malaysian sign language words collected from five participants (two males and three females) aged between 20 and 21 years old. The data collection took place indoors under normal lighting conditions. The MyWSL2023 dataset, which has been made freely accessible to all researchers, serves as a valuable resource for not only investigating and developing automated systems for hearing-impaired and deaf individuals but also gesture and sign language recognition using vision-based methods. The dataset can be accessed for free at https://data.mendeley.com/datasets/zvk55p7ktd.

Speech impairment affects an individual’s ability to communicate. People affected by this problem, suffer by the communication barrier with the normal people. Hence, to express their emotions and thoughts people will use sign language for their communication and. Although sign language is common in world, there may be a difficulty for people who do not have knowledge about sign language to communicate with speech impaired people who knows the sign language. Now there is a prominent growth in the field of computer vision and deep learning, which has been tremendous development in the fields of sign and motion recognition. The proposed model focusses on overcoming the challenges in verbal communication between non-sign language speakers and sign language speakers by building a deep learning model to recognize alphabets of American sign language. The model was trained on the dataset collected from Kaggle website which consisted of 26 American Sign Language alphabets. The model achieved 99.3% mean average precision in recognition of sign and average probability value for test image achieved was 0.99. The deep learning method used to build the model achieved more accuracy than the previous solutions.

Aiming at the feature of signer-independent sign language recognition the training data complexity caused by mass data and noticeable distinctions between different people data, the weighted KNN/HMM model is presented in this paper. This model is made of two blocks, which is part of sign language classification and recognition. In classing part, the KNN (K-Nearest Neighbor, KNN) is used to learn the training samples. Considering the different contributions of sign language features to pattern classification we give different weight to different characteristics. And the category of test sample is decided by the sum of weighted distance. In recognition part, weighted KNN classification result is taken as the state-input of HMM (Hidden Markov models, HMM) to implement sign language recognition, combine with the ability to temporal data modeling and fuzzy inference of HMM model. Experiment results show that weighted KNN/HMM sign language recognition model are efficient on either recognition speed or recognition rate.

Sign Language is the dominant yet non-primary form of communication language used in the deaf and hearing-impaired community. To make an easy and mutual communication between the hearing-impaired and the hearing communities, building a robust system capable of translating the spoken language into sign language and vice versa is fundamental. To this end, sign language recognition and production are two necessary parts for making such a two-way system. Sign language recognition and production need to cope with some critical challenges. In this survey, we review recent advances in Sign Language Production (SLP) and related areas using deep learning. This survey aims to briefly summarize recent achievements in SLP, discussing their advantages, limitations, and future directions of research.

Sign Language Recognition (SLR) systems have gained significant popularity in recent years. Despite the existence of various Deep Learning (DL) models to classify sign languages, the deployment of these models on resource-constrained devices remains a challenge. Majority of the research on SLR focuses on popular sign languages, such as American Sign Language (ASL), German Sign Language (GSL) etc., while Kannada Sign Language (KSL) remains significantly underexplored. This study focuses on creating a word-level custom dataset of 33 classes consisting of 2319 videos and training three DL models, namely, LSTM, BiLSTM, and Encoder-only Transformer for dynamic KSL recognition. Trained models were then optimized with quantization techniques and converted to .tflite format, for efficient deployment on resource-constrained devices like smartphones. Dynamic Post Training Quantization (PTQ) model, for the BiLSTM model achieved the highest accuracy of 95.71%, followed by the LSTM model with 94.7%, and the Transformer-based architecture achieved 94.19%. Transformer model deployed on smartphone achieved the smallest inference time of 16.2 ms with a model size of 1097.77 KB, followed by LSTM recording 19.6 ms as inference time with a model size of 1254.49 KB and BiLSTM recording the highest inference time of 27.8 ms with a model size of 3862.01 KB.

Over four million individuals in Mexico face communication barriers due to hearing impairments. Sign language serves as an essential communication tool within the deaf community; however, automatic translation between sign and oral languages remains a significant challenge. This study proposes an approach for recognizing dynamic gestures from Mexican Sign Language (LSM) to support the development of assistive communication technologies. In collaboration with expert interpreters, an LSM corpus comprising 121 signs was developed, including a specialized lexicon focused on medical emergencies and accident scenarios. A standardized video acquisition protocol was implemented with both expert and non-expert participants. The proposed methodology consists of skeletal keypoint extraction using MediaPipe, data augmentation through frame sampling, and dataset normalization. Multiple deep learning architectures were evaluated, including ResNet, Simple RNN, LSTM, Bidirectional LSTM (BiLSTM), Gated Recurrent Units (GRU), a Transformer encoder, and a hybrid ResNet-Transformer model. Among the evaluated models, the ResNet architecture achieved the best performance, obtaining an F1-score of 0.948 under subject-independent evaluation, with an average inference time of 0.468 seconds. Hyperparameter optimization analysis indicated that performance improvements were primarily driven by training dynamics and regularization strategies rather than increases in architectural depth. The results demonstrate the effectiveness of deep learning-based approaches for dynamic LSM gesture recognition and highlight the importance of optimization strategies for robust generalization. This work contributes toward LSM-to-Spanish translation systems and provides a foundation for advancing data-driven sign language recognition technologies.

A lightweight network-based sign language robot with facial mirroring and speech system