SignBridge AI: A Multimodal Deep Learning Framework for Real Time Sign Language Translation

ABSTRACTSign language is a communication tool for deaf and dumb people that includes known signs or body gestures to transfer meanings. It uses shapes, directions, movements of hands, and also facial expressions. A sign not only transmits a word but also conveys a tone. Many of the deaf people are not only able to speak, but also not able to write or read a language, so developing sign language translation or in other words sign language recognition (SLR) system can be very vital in their life. The SLR is extremely desired because of its capability to overcome the obstacles between deaf and normal people. It is one of the most important research fields in the human computer interaction studies. Hence, this paper presents an overview of the recent main research works with the vision-based SLR system, and the existing recognition techniques are discussed. Next, we focus on video-based SLR system and perform continuous SLR within video sequences.

An optical modeless Sign Language Recognition (SLR) system is presented. The system uses the HAusdorf-Voronoi NETwork (HAVNET), an artificial neural network designed for 2D binary pattern recognition. It uses adaptation of the Hausdorff distance to determine the similarity between an input pattern and a learned representation. A detailed review of the architecture, the learning equations, and the recognition equations for the HAVNET network are presented. Competitive learning has been implemented in training the network using a nearest-neighbor technique. The SLR system is applied to the optical recognition of 24 static symbols from the American Sign Language convention. The SLR system represents the target images in a 80 X 80 pixel format. The implemented HAVNET network classifies the inputs into categories representing each of the symbols, using an output layer of 24 nodes. The network is trained with 5 different formats for each symbol and is tested with all 24 symbols in 15 new formats. Results from the SLR system without competitive training show shape identification problems, when distinguishing symbols with similar shapes. Implementation of competitive learning in the HAVNET neural network improved recognition accuracy on this task to 89%. The hand gestures are identified through a window search algorithm. Feature recognition is obtained from edge enhancement by applying a Laplacian filter and thresholding, which provides robustness to pose, color and background variations.

Hand gestures play a vital role in sign language, serving as a primary means of non-verbal communication for individuals who are deaf or mute, enabling them to express themselves and interact with others. Although numerous sign language recognition systems have been developed, most are either costly or lack adaptability for real-world use. Recent advances in deep learning, particularly Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), have significantly improved gesture recognition by enabling systems to learn complex patterns from large datasets of images or video sequences. This paper introduces a novel algorithm called Sign Language Accessibility Enhancement Using Ensemble of Deep Learning Techniques for Hand Gesture Recognition in Deaf and Dumb Communication (SLAEDL-HGRDDC). The model combines multiple components into a robust framework to enhance recognition accuracy and system scalability. Initially, Gaussian Filtering (GF) is used to reduce noise and improve image quality. Then, Inception-ResNet-V2 is adopted to extract intricate spatial features from gesture images. The recognition process utilizes an ensemble of three deep learning classifiers – Long Short-Term Memory (LSTM), Gated Recurrent Units (GRUs), and Temporal Convolutional Networks (TCN). Finally, the Modified Al-Biruni Earth Radius Optimization (MBER) algorithm is employed for optimal hyperparameter tuning. Extensive simulations demonstrate that SLAEDL-HGRDDC outperforms traditional systems in various performance metrics.

Real time Indian sign language recognition system

Communication between hearing-impaired and hearing individuals remains a persistent global challenge, primarily due to the limited understanding of sign language among the general population. This communication gap frequently leads to social exclusion, dependency on human interpreters, and significant difficulties in critical environments such as healthcare, education, and public services. To address this, the current study proposes an innovative Artificial Intelligence (AI)-based real-time sign language recognition and communication support system. Designed to enable inclusive, two-way interaction, the system utilizes advanced computer vision techniques to capture and preprocess hand gestures. Machine learning models, specifically Convolutional Neural Networks (CNNs), interpret these gestures to generate corresponding text and speech output. Conversely, speech input from hearing users is converted into text and animated visual sign representations to ensure seamless bidirectional communication. Furthermore, the application integrates an SOS emergency alert module and an interactive learning interface for basic sign language. By reducing the dependency on human translators and facilitating natural interaction, this proposed system aims to significantly enhance accessibility, user independence, and social inclusion for the hearing-impaired community. Keywords: Sign Language Recognition, Artificial Intelligence, Computer Vision, Bidirectional Communication, Convolutional Neural Networks (CNN), Accessibility, Deep Learning.

This work aims at presenting a novel Computer Vision approach in the development of a real-time, web-camera based, British Sign Language recognition system. A literature review focused on current (1) state of sign language recognition systems and (2) techniques used is conducted. This review process is used as a foundation on which a Convolutional Neural Network (CNN) based system is designed and then implemented. A bespoke British Sign Language dataset -containing 11,875 images - is then performed to train and test the CNN which is used for the classification of human hand performed gestures. Finally, the CNN architecture recognized 19 static British Sign Language gestures, incorporating both single and double-handed gestures. During testing, the system achieved an average recognition accuracy of 89%.

Sign language recognition (SLR) system is helpful for individuals who can't talk or hear. One can easily communicate with this kind of people using SLR system. In this paper we are focusing on SLR approach where we have apply our proposed algorithm to recognize hand gestures of routine words like Morning, Night, Monday, Tuesday and so forth. Total 10 routine words we have recognize using our proposed technique. For recognize hand pose we have utilized skin color detection algorithm. Further we have applied correlation-coefficient algorithm to distinguish closeness features which are further continue in Neuro-fuzzy(NF) classification algorithm to recognize words. Presented algorithm tried on Matlab. With taken of 10 routine words our proposed framework of techniques got 92% correction rate.

Sign language is an expressive visual language that uses the hand and face and is an important tool for deaf individuals to communicate. Standard means of recognition objectives cannot recognize complex gestures effectively, in turn limiting accessibility and transactions of communication effectively. This project seeks to close communication barriers by translating hand gestures into speech, text, and even serve as an output to virtual assistants. To date, prior research in signing recognition has focused primarily on identifiable hand gestures, often selecting a subset of Indian Sign Language (ISL) for classification. In this work, a deep learning method is proposed to perform static sign recognition through Convolutional Neural Networks (CNN) with the VGG19 architecture. The system is trained through a dataset of ISL gestures for predicting and classifying signs with high confidence. The predicted sign is converted to a text output, to then a speech output, stored as an audio file. In conclusion, the system provides accessibility through detection, enabling a user user experience for both signers/non-signers to communicate and add encryption with accessibility. The model produces a far more particular/accurate performance in sign language recognition, and follows through with promising assistive technology solutions

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

A hand sign for a disabled patient is a useful tool for human-to-human communication that can be used for a variety of purposes. They are frequently utilised by speech-impaired people around the world for communication since they are a natural form of engagement. In actuality, this group makes up around 1% of the Indian population. To enable easy communication among the signer and non-signer communities, an effective sign language recognition system (SLRS) can identify sign language motions. This research proposes a deep learning-based SLRS for classification that is computer vision-based. A sample of Auslan (Australian Sign Language (ASL)) signs is included in the dataset. Using top-notch position trackers, 27 examples of each of the 95 Auslan signs were recorded by a native signer. However, films captured Indian Sign Language (ISL) was filmed utilising numerous signs. Three individuals submitted 26 letters (A–Z) and 10 numbers (0–9). Few background removal techniques were performed on the samples and various skin color segmentation process was done. Third, a brand-new and reliable model for classifying ISL motions has been suggested utilising the mutation denoising autoencoder (MDAE). ISL and ASL datasets are publicly available and have been used to measure the effectiveness of the proposed system. In terms of measures like precision, recall, F1-Score, and accuracy, the proposed system is contrasted with well-known computer vision algorithms like the Support Vector Machine (SVM), Convolution Neural Network (CNN), Enhanced Convolution Neural Network (ECNN), and DAE. Python has been used to implement and test these categorisation techniques.

Sign Language is a meansof communication between audibly challenged people. To provide an interface between the audibly challenged community and the rest of the world we need Sign Language translators. A sign language recognition system computerizes the work of a sign language trEvery Sign Language Recognition (SLR) System is trained to recognizespecific sets of signs and they correspondingly output the sign in the required format. These SLR systems are built with powerful image processing techniques. The sign language recognition systems are capable of recognizing a specific set of signing gestures and output the corresponding text/audio. Most of these systems involve the techniques of detection, segmentation, tracking, gesture recognition and classification. This paperproposes a design for a SLR System.

The population of deaf-mute in Taiwan is increasing every year. These deaf-mute persons usually use sign language to communicate with each other. However, most hearing persons cannot understand sign languages because they have not learned it. In order to let normal persons understand sign language, a sign language recognition system is necessary. In this paper, such a sign language recognition system is proposed. In the proposed system, a depth image is captured by Kinect sensor, palm area is segmented from depth image, palm binary image is thresholded by using Otsu thresholding method, and background noises are removed by morphological closing operators, and SURF features and descriptors are extracted to identify the sign language. Experimental results show that the proposed method is effective to detect and identify the numerals and letters in the English Alphabet used in the sign language.

Sign Language Recognition systems have been thoroughly researched in recent times to bridge the gap between the signer and non-signer community. This paper focuses on fulfilling the need for a thorough review of the works done in building such systems. The systems discussed herein for static Sign Language Recognition are broadly classified on Support Vector Machine, Convolutional Neural Network as classifiers. The identified phases for the reviewed systems include preprocessing, feature extraction and classification. The methods discussed in these phases include Principal Component Analysis, Histograms of Orientations Gradient, Hybrid Sign Invariant Feature Transformation. Further the limitations and improvements in approaches are relatively portrayed to give a better analysis for future work. The overall aim for this paper is to provide insight into making effective systems for Sign Language Recognition. This will make these systems easily available for educating the masses about static Sign Language. Thus, sign language will no longer remain a barrier.

The current research looks into using convolutional neural networks, also known as CNNs, to develop a hand sign understanding and conversion system. The study addresses the urgent need for a precise and effective way to close the communication gap that exists between the hearing and the deaf communities. The study examines how well CNNs can identify and translate hand sign gestures into text, giving sign language users a real-time means of communication. We use three steps in our methodology: gathering datasets, training models, and thorough assessment. The outcomes show encouraging hand sign recognition accuracy, creating new opportunities for inclusive communication. Sign language is one of the oldest and most natural forms of language for communication, but since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real-time method using neural networks for fingerspelling-based American sign language. In our method, the hand is first passed through a filter and after the filter is applied the hand is passed through a classifier which predicts the class of the hand gestures. Our method provides 95.7 % accuracy for the 26 letters of the alphabet. Keywords-Sign Language, ASL, Hearing disability, Convolutional Neural Network (CNN), Computer Vision, Machine Learning, Gesture recognition, Sign language recognition, Hue Saturation Value algorithm.

Sign language recognition system for communicating to people with disabilities