Hand Sign Language Recognition and Conversion into Text Using Convolutional Neural Network

Sign language recognition plays a crucial role in supporting communication between deaf communities and hearing individuals. In particular, Vietnamese Sign Language (VSL) recognition remains a challenging task due to the complexity of hand gestures and limited available datasets. However, most existing approaches address static gestures and dynamic sign phrases as separate recognition problems, often employing different feature representations and independent processing pipelines. This fragmentation increases system complexity and limits scalability for real-time sign language applications. This study proposes a unified hand-landmark-based deep learning framework for recognizing both static and dynamic VSL gestures within a single integrated system. The system begins by detecting hand landmarks through the MediaPipe hand tracking pipeline. Based on temporal motion analysis of landmark sequences, a gesture routing mechanism automatically determines whether the input corresponds to a static gesture or a dynamic sign phrase. Static gestures are classified using a convolutional neural network (CNN), while dynamic gestures are processed using a long short-term memory (LSTM) network to capture temporal dependencies. Experiments were conducted on a VSL dataset consisting of 23 static hand signs and 4 dynamic sign phrases collected from multiple participants with variations in hand shape and gesture execution style. The system performance is evaluated using accuracy, precision, recall, and F1-score. Experimental results demonstrate that the proposed framework achieves an average recognition accuracy of 92% for static gestures and 88.4% for dynamic sign phrases, outperforming traditional machine learning baselines. The proposed system provides a practical and efficient solution for VSL recognition and has potential applications in real-time assistive communication systems.

Sign Language Recognition (SLR) has become an essential area of research due to its potential to promote understanding between the deaf and hearing communities through communication. This paper provides an in-depth study of various methodologies and models employed in SLR, focusing on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). We analyze their application to datasets from various sign languages, such as Arabic Sign Language (ArSL), American Sign Language (ASL), and British Sign Language (BSL), and explore how these models improve the recognition of dynamic, multi-dimensional hand gestures. This research not only advances the understanding of deep learning applications in sign language recognition but also addresses critical challenges in data processing and real-time applications, paving the way for inclusive technologies in informatics and human-computer interaction. Despite the progress in applying deep learning techniques to SLR, several challenges remain, particularly in dataset limitations, handling large vocabularies, and ensuring consistent performance across diverse environments and signers. The paper also investigates the broader applications of SLR, such as virtual reality, healthcare, education, and accessibility, and discusses the integration of SLR with human-computer interaction systems. Furthermore, it highlights current limitations in the field, such as difficulties with video data handling, the need for standard datasets, and issues related to training computational models. Finally, the paper outlines future research directions, including developing more robust SLR systems that can function effectively in uncontrolled environments, improving data collection methodologies, and creating real-time, user-friendly applications to assist the community of deaf and hard-of-hearing individuals.

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

Automating real-time sign language translation through deep learning and machine learning techniques can greatly enhance communication between the deaf community and the wider public. This research investigates how these technologies can change the way individuals with speech impairments communicate. Despite advancements, developing accurate models for recognizing both static and dynamic gestures remains challenging due to variations in gesture speed and length, which affect the effectiveness of the models. We introduce a hybrid approach that merges machine learning and deep learning methods for sign language recognition. We provide new model for the recognition of Kazakh Sign Language (QazSL), employing five algorithms: Support Vector Machine (SVM), Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural Networks (CNN) with VGG19, ResNet-50, and YOLOv5. The models were trained on a QazSL dataset of more than 4,400 photos. Among the assessed models, the GRU attained the highest accuracy of 100%, followed closely by SVM and YOLOv5 at 99.98%, VGG19 at 98.87% for dynamic dactyls, LSTM at 85%, and ResNet-50 at 78.61%. These findings illustrate the comparative efficacy of each method in real-time gesture recognition. The results yield significant insights for enhancing sign language recognition systems, presenting possible advancements in accessibility and communication for those with hearing impairments.

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

Sign language is a complex language that uses hand gestures, body movements, and facial expressions and is majorly used by the deaf community. Sign language recognition (SLR) is a popular research domain as it provides an efficient and reliable solution to bridge the communication gap between people who are hard of hearing and those with good hearing. Recognizing isolated sign language words from video is a challenging research area in computer vision. This paper proposes a hybrid SLR framework that combines a convolutional neural network (CNN) and an attention-based long-short-term memory (LSTM) neural network. We used MobileNetV2 as a backbone model due to its lightweight structure, which reduces the complexity of the model architecture for deriving meaningful features from the video frame sequence. The spatial features are fed to LSTM optimized with an attention mechanism to select the significant gesture cues from the video frames and focus on salient features from the sequential data. The proposed method is evaluated on a benchmark WLASL dataset with 100 classes based on precision, recall, F1-score, and 5-fold cross-validation metrics. Our methodology acquired an average accuracy of 84.65%. The experiment results illustrate that our model performed effectively and computationally efficiently compared to other state-of-the-art methods.

Sign language identification and recognition technique is composed by the gesture images detection and the hand gestures recognition. Gesture images detection is to locate the image part of palm and fingers from the captured pictures, and rotating them to the appropriate gesture posture. Both of them are the important pre-processing for sign language identification and recognition. Lose them, the correctness rate of the sign language recognition algorithms will be dropped down to an unacceptable level. This paper presents novel processing algorithms for the gesture images detection and recognition. In the detection process, it rotates an askew gesture to right position, and to delete the elbow and forearm parts from the captured pictures. In the recognition process, it includes two phases with the model construction and the sign language identification. In the model construction phase, the static hand gesture of sign language is constructed by the Gaussian mixture model, and the unknown gesture image is identified by Gaussian model match. Based on this presented static sign language detection and recognition algorithms, the correct recognition rate is about 94% in average.

Gesture recognition has become a significant part of human-machine interaction, particularly when verbal interaction is not feasible. The rapid development of biomedical sensing and machine learning algorithms, including electromyography (EMG) and convolutional neural networks (CNNs), has enabled the interpretation of sign languages, including the Polish Sign Language, based on EMG signals. The objective was to classify the game control gestures and Polish Sign Language gestures recorded specifically for this study using two different data acquisition systems: BIOPAC MP36 and MyoWare 2.0. We compared the classification performance of various machine learning algorithms, with a particular emphasis on CNNs on the dataset of EMG signals representing 24 gestures, recorded using both types of EMG sensors. The results (98.324% versus ≤7.8571% and 95.5307% versus ≤10.2697% of accuracy for CNNs and other classifiers in data recorded with BIOPAC MP36 and MyoWare, respectively) indicate that CNNs demonstrate superior accuracy. These results suggest the feasibility of using lower-cost sensors for effective gesture classification and the viability of integrating affordable EMG-based technologies into broader gesture recognition frameworks, providing a cost-effective solution for real-world applications. The dataset created during the study offers a basis for future studies on EMG-based recognition of Polish Sign Language.

Speech impairment limits a person’s capacity for oral and auditory communication. Improvements in communication between the deaf and the general public can be progressed by a real-time sign language detector. Recent studies have contributed to make progress in motion and gesture identification processes using Deep Learning (DL) methods and computer vision. But the development of static and dynamic sign language recognition (SLR) models is still a challenging area of research. The difficulty is in obtaining an appropriate model that addresses the challenges of continuous signs that are independent of the signer. Different signers’ speeds, durations, and many other factors make it challenging to create a model with high accuracy and continuity. This study mainly focused on SLR using a modified DL and hybrid optimization approach. Notably, spatial and geometric-based features are extracted via the Visual Geometry Group 16 (VGG16), and motion features are extracted using the optical flow approach. A new DL model, CNNSa-LSTM, is a combination of a Convolutional Neural Network (CNN), Self-Attention (SA), and Long-Short-Term Memory (LSTM) to identify sign language. This model is developed for feature extraction by combining CNNs for spatial analysis with SA mechanisms for focusing on relevant features, while LSTM effectively models temporal dependencies. The proposed CNNSa-LSTM model enhances performance in tasks involving complex, sequential data, such as sign language processing. Besides, a Hybrid Optimizer (HO) is proposed using the Hippopotamus Optimization Algorithm (HOA) and the Pathfinder Algorithm (PFA). The proposed model has been implemented in Python, and it has been evaluated over the existing models in terms of accuracy (98.7%), sensitivity (98.2%), precision (98.5%), Word Error Rate (WER) (0.131), Sign Error Rate (SER) (0.114), and Normalized Discounted Cumulative Gain (NDCG) (98%) as well. The proposed model has recorded the highest accuracy of 98.7%.

Communication between hearing-impaired and hearing individuals remains a persistent global challenge, primarily due to the limited understanding of sign language among the general population. This communication gap frequently leads to social exclusion, dependency on human interpreters, and significant difficulties in critical environments such as healthcare, education, and public services. To address this, the current study proposes an innovative Artificial Intelligence (AI)-based real-time sign language recognition and communication support system. Designed to enable inclusive, two-way interaction, the system utilizes advanced computer vision techniques to capture and preprocess hand gestures. Machine learning models, specifically Convolutional Neural Networks (CNNs), interpret these gestures to generate corresponding text and speech output. Conversely, speech input from hearing users is converted into text and animated visual sign representations to ensure seamless bidirectional communication. Furthermore, the application integrates an SOS emergency alert module and an interactive learning interface for basic sign language. By reducing the dependency on human translators and facilitating natural interaction, this proposed system aims to significantly enhance accessibility, user independence, and social inclusion for the hearing-impaired community. Keywords: Sign Language Recognition, Artificial Intelligence, Computer Vision, Bidirectional Communication, Convolutional Neural Networks (CNN), Accessibility, Deep Learning.

Every average person has the ability to see, hear, and respond to their surroundings. Some unfortunate people are without this significant blessing. These people primarily include the deaf and dumb, who interact with others by using sign language. Yet, since not all common people can understand their sign language, communicating with regular people is a significant limitation for them. Also, this will make it difficult for members of the deaf and dumb communities to engage with others, especially when they want to participate in educational, social, and professional settings. The goals of this research are to create a sign language translation system for hearing or speech challenged individuals utilising deep learning and neural networks to help those who are deaf or have trouble speaking or hearing communicate with others. The model utilised in this classification assignment is a pretty simple Convolutional Neural Network implementation for the methodology (CNN). This exploits the convolutional property, which was primarily developed for the analysis of visual imagery. Three layered CNNs were trained and tested in real time using segmented RGB hand motions. With the help of a personal device, such as a laptop webcam, simple photos of the hand were used to construct the image dataset for each gesture. I was able to get training accuracy of roughly 89% and testing accuracy of 98.5% with this CNN model. Keywords : Artificial Neural Networks (ANN), Convolutional Neural Networks (CNN), Gesture Recognition, Open CV, TensorFlow

Sign language plays a crucial role as a communication tool for both the deaf and hard-of- hearing communities, enabling them to engage and interact effectively within their own community as well as with others.However, communication barriers arise when individuals unfamiliar with sign language engage with those who rely on it, underscoring the need for inclusive solutions. Real-time sign language interpretation systems, leveraging machine learning and computer vision technologies, present a promising approach to bridging this gap. These systems convert sign language gestures into spoken or written language by utilizing gesture recognition algorithms, neural networks, and natural language processing. By analyzing hand movements, facial expressions, and body language, the systems provide accurate, context-aware translations of various sign languages, such as American Sign Language (ASL), with minimal delay. This enables seamless, natural interactions, making such technologies essential for fostering inclusive communication in diverse settings. Key Words: Sign language recognition, real- time interpretation, machine learning, com- puter vision, gesture recognition, neural networks, natural language processing, communication barriers, inclusivity, American Sign Language (ASL).

Sign languages are crucial in promoting independent living for deaf and hard-of-hearing individuals. They offer a means of communication that facilitates access to various aspects of daily life and supports autonomy. The deaf community in India, estimated at over 18 million individuals, faces significant communication barriers due to the limited use of Indian sign language (ISL) by the general population. This chapter explores the potential of machine learning (ML) for Indian sign language detection as a bridge between the deaf and hearing communities. The authors delve into existing research on sign language recognition (SLR) systems, analyze the challenges specific to ISL detection, and propose future directions leveraging advancements in ML. By examining the sociolinguistic landscape and the potential impact of ML-powered ISL detection, this chapter aims to contribute to a more inclusive society.

Sign language is a crucial communication tool for hearing and deaf communities, with each language containing unique hand positions, motion profiles, and shapes. To develop sign language recognition systems, common input data formats include RGB and depth, considering perspective, articulation distributions, joint visibility, and hand shape. However, difficulties endure in intricate and crowded settings, resulting in increased average errors. An improved deep learning method that makes use of Google's MediaPipe framework and combines Convolutional and Recurrent Neural Networks (RNNs) is suggested as a solution to these problems. While RNNs record temporal correlations in sign language gesture data, CNNs extract spatial characteristics from static images. MediaPipe is essential for preprocessing and feature extraction, as well as for supplying precise hand landmarks and enabling real-time processing. The proposed system shows a significant improvement in recognition accuracy and processing efficiency, with up to 5% improvement in recognition accuracy compared to traditional methods. This work contributes to the advancement of sign language recognition technologies, offering a more reliable and efficient solution for real-time sign language interpretation and text conversion.

Sign language is an expressive visual language that uses the hand and face and is an important tool for deaf individuals to communicate. Standard means of recognition objectives cannot recognize complex gestures effectively, in turn limiting accessibility and transactions of communication effectively. This project seeks to close communication barriers by translating hand gestures into speech, text, and even serve as an output to virtual assistants. To date, prior research in signing recognition has focused primarily on identifiable hand gestures, often selecting a subset of Indian Sign Language (ISL) for classification. In this work, a deep learning method is proposed to perform static sign recognition through Convolutional Neural Networks (CNN) with the VGG19 architecture. The system is trained through a dataset of ISL gestures for predicting and classifying signs with high confidence. The predicted sign is converted to a text output, to then a speech output, stored as an audio file. In conclusion, the system provides accessibility through detection, enabling a user user experience for both signers/non-signers to communicate and add encryption with accessibility. The model produces a far more particular/accurate performance in sign language recognition, and follows through with promising assistive technology solutions