AI - Based Two Way Sign Language Communication

Sign Language is a meansof communication between audibly challenged people. To provide an interface between the audibly challenged community and the rest of the world we need Sign Language translators. A sign language recognition system computerizes the work of a sign language trEvery Sign Language Recognition (SLR) System is trained to recognizespecific sets of signs and they correspondingly output the sign in the required format. These SLR systems are built with powerful image processing techniques. The sign language recognition systems are capable of recognizing a specific set of signing gestures and output the corresponding text/audio. Most of these systems involve the techniques of detection, segmentation, tracking, gesture recognition and classification. This paperproposes a design for a SLR System.

ABSTRACTSign language is a communication tool for deaf and dumb people that includes known signs or body gestures to transfer meanings. It uses shapes, directions, movements of hands, and also facial expressions. A sign not only transmits a word but also conveys a tone. Many of the deaf people are not only able to speak, but also not able to write or read a language, so developing sign language translation or in other words sign language recognition (SLR) system can be very vital in their life. The SLR is extremely desired because of its capability to overcome the obstacles between deaf and normal people. It is one of the most important research fields in the human computer interaction studies. Hence, this paper presents an overview of the recent main research works with the vision-based SLR system, and the existing recognition techniques are discussed. Next, we focus on video-based SLR system and perform continuous SLR within video sequences.

Sign language is a main communication tool for hearing impaired people. The sign language recognition systems are designed to convert it into text or speech, breaking the interaction barrier for deaf person. The paper reviews the state of the art in sign language recognition. The study focuses on data acquisition methods, because they put the major restrictions on a further recognition process. We perform the sign language recognition using data collected by the Leap Motion Controller. The device detects user's hands and transmits it into a three-dimensional model, providing hands and fingers location and posture. This paper presents a system that is capable of recognition and classification of the sign language using deep-convolutional neural network. We highlight the main features of the Leap Motion Controller that improves and simplifies feature extraction. Also, the paper discusses the problems we faced during the sign language recognition.

Sign Language Recognition systems have been thoroughly researched in recent times to bridge the gap between the signer and non-signer community. This paper focuses on fulfilling the need for a thorough review of the works done in building such systems. The systems discussed herein for static Sign Language Recognition are broadly classified on Support Vector Machine, Convolutional Neural Network as classifiers. The identified phases for the reviewed systems include preprocessing, feature extraction and classification. The methods discussed in these phases include Principal Component Analysis, Histograms of Orientations Gradient, Hybrid Sign Invariant Feature Transformation. Further the limitations and improvements in approaches are relatively portrayed to give a better analysis for future work. The overall aim for this paper is to provide insight into making effective systems for Sign Language Recognition. This will make these systems easily available for educating the masses about static Sign Language. Thus, sign language will no longer remain a barrier.

This article is dedicated to analyzing advanced approaches in temporal modeling and real-time gesture recognition within sign language recognition (SLR) systems. Sign glosses are expressed through the spatio-temporal characteristics of visual information, which requires the use of sequence-processing models for their automatic recognition. The study primarily evaluates the effectiveness of three key models: Long Short-Term Memory (LSTM) networks, Temporal Convolutional Networks (TCN), and Transformer-based architectures. The article also examines methods applied for real-time analysis of sign glosses, including: Sliding window segmentation of video streams; Self-attention mechanisms for identifying dependencies between gestures; Gloss mapping algorithms for linking sign movements to linguistic units; Ontological integration techniques for enhancing semantic accuracy. Practical results indicate that combining temporal modeling with semantic analysis and contextual verification algorithms ensures continuous and high-accuracy recognition of sign movements. In particular, multimodal systems (video + sensor + gloss) utilizing Transformer-based approaches achieved superior performance in real-time conversion of continuous sign gloss streams into text. The findings of this study hold practical significance for the development of smart assistive devices for automatic sign language translation, interactive interfaces for hearing-impaired users, and specialized SLR platforms for educational and instructional purposes.

Sign language is an expressive visual language that uses the hand and face and is an important tool for deaf individuals to communicate. Standard means of recognition objectives cannot recognize complex gestures effectively, in turn limiting accessibility and transactions of communication effectively. This project seeks to close communication barriers by translating hand gestures into speech, text, and even serve as an output to virtual assistants. To date, prior research in signing recognition has focused primarily on identifiable hand gestures, often selecting a subset of Indian Sign Language (ISL) for classification. In this work, a deep learning method is proposed to perform static sign recognition through Convolutional Neural Networks (CNN) with the VGG19 architecture. The system is trained through a dataset of ISL gestures for predicting and classifying signs with high confidence. The predicted sign is converted to a text output, to then a speech output, stored as an audio file. In conclusion, the system provides accessibility through detection, enabling a user user experience for both signers/non-signers to communicate and add encryption with accessibility. The model produces a far more particular/accurate performance in sign language recognition, and follows through with promising assistive technology solutions

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

Sign language recognition systems play an important role in improving communication between deaf or hard-ofhearing individuals and the hearing population. However, the lack of widespread understanding of sign language continues to create barriers in education, healthcare, and everyday social interaction. This paper presents a real-time vision-based sign language recognition system using Convolutional Neural Networks (CNNs) and computer vision techniques to translate hand gestures into text and speech. The proposed system captures gesture images through a webcam and processes them using OpenCV-based hand segmentation and preprocessing methods. A custom dataset consisting of 44 gesture classes, including alphabets and numerical signs, was created using grayscale images of size 50×50 pixels. Data augmentation techniques such as image flipping were applied to improve model performance and generalization. The processed gesture images were used to train a CNN model implemented using TensorFlow and Keras for accurate classification of hand gestures. The trained model enables real-time gesture prediction through live video input and converts recognized gestures into readable text and speech output using a text-to-speech module. Experimental results demonstrate that the proposed system provides efficient and accurate gesture recognition suitable for real-time applications. This work contributes to the development of an accessible, low-cost assistive communication system for sign language users.

An optical modeless Sign Language Recognition (SLR) system is presented. The system uses the HAusdorf-Voronoi NETwork (HAVNET), an artificial neural network designed for 2D binary pattern recognition. It uses adaptation of the Hausdorff distance to determine the similarity between an input pattern and a learned representation. A detailed review of the architecture, the learning equations, and the recognition equations for the HAVNET network are presented. Competitive learning has been implemented in training the network using a nearest-neighbor technique. The SLR system is applied to the optical recognition of 24 static symbols from the American Sign Language convention. The SLR system represents the target images in a 80 X 80 pixel format. The implemented HAVNET network classifies the inputs into categories representing each of the symbols, using an output layer of 24 nodes. The network is trained with 5 different formats for each symbol and is tested with all 24 symbols in 15 new formats. Results from the SLR system without competitive training show shape identification problems, when distinguishing symbols with similar shapes. Implementation of competitive learning in the HAVNET neural network improved recognition accuracy on this task to 89%. The hand gestures are identified through a window search algorithm. Feature recognition is obtained from edge enhancement by applying a Laplacian filter and thresholding, which provides robustness to pose, color and background variations.

This work aims at presenting a novel Computer Vision approach in the development of a real-time, web-camera based, British Sign Language recognition system. A literature review focused on current (1) state of sign language recognition systems and (2) techniques used is conducted. This review process is used as a foundation on which a Convolutional Neural Network (CNN) based system is designed and then implemented. A bespoke British Sign Language dataset -containing 11,875 images - is then performed to train and test the CNN which is used for the classification of human hand performed gestures. Finally, the CNN architecture recognized 19 static British Sign Language gestures, incorporating both single and double-handed gestures. During testing, the system achieved an average recognition accuracy of 89%.

Sign language recognition (SLR) system is helpful for individuals who can't talk or hear. One can easily communicate with this kind of people using SLR system. In this paper we are focusing on SLR approach where we have apply our proposed algorithm to recognize hand gestures of routine words like Morning, Night, Monday, Tuesday and so forth. Total 10 routine words we have recognize using our proposed technique. For recognize hand pose we have utilized skin color detection algorithm. Further we have applied correlation-coefficient algorithm to distinguish closeness features which are further continue in Neuro-fuzzy(NF) classification algorithm to recognize words. Presented algorithm tried on Matlab. With taken of 10 routine words our proposed framework of techniques got 92% correction rate.

Sign Language Recognition (SLR) has become an essential area of research due to its potential to promote understanding between the deaf and hearing communities through communication. This paper provides an in-depth study of various methodologies and models employed in SLR, focusing on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). We analyze their application to datasets from various sign languages, such as Arabic Sign Language (ArSL), American Sign Language (ASL), and British Sign Language (BSL), and explore how these models improve the recognition of dynamic, multi-dimensional hand gestures. This research not only advances the understanding of deep learning applications in sign language recognition but also addresses critical challenges in data processing and real-time applications, paving the way for inclusive technologies in informatics and human-computer interaction. Despite the progress in applying deep learning techniques to SLR, several challenges remain, particularly in dataset limitations, handling large vocabularies, and ensuring consistent performance across diverse environments and signers. The paper also investigates the broader applications of SLR, such as virtual reality, healthcare, education, and accessibility, and discusses the integration of SLR with human-computer interaction systems. Furthermore, it highlights current limitations in the field, such as difficulties with video data handling, the need for standard datasets, and issues related to training computational models. Finally, the paper outlines future research directions, including developing more robust SLR systems that can function effectively in uncontrolled environments, improving data collection methodologies, and creating real-time, user-friendly applications to assist the community of deaf and hard-of-hearing individuals.

Sign language recognition devices are effective approaches to breaking the communication barrier between signers and non-signers and exploring human-machine interactions. Wearable gloves have been developed for gesture recognition and virtual reality applications by employing flexible sensors for motion detection and machine learning for data analysis. However, most existing wearable devices present limited sign language translating capacity due to the sensors' design and distribution. Here, we propose a cost-effective dual-hand soft fiber optic glove system consisting of multimode soft liquid-core fiber optic sensors, gyroscopes, wireless printed circuit boards, and batteries for sign language translation. In combination with different deep learning techniques and recognition strategies, the glove system can recognize static gestures and dynamic gestures of American Sign Language, and deduce the meaning of sentences by the sequence of gestures. The soft glove system exhibits a broad sign language range (10 numbers, 26 alphabets, 18 words, and 5 sentences meaning prediction), and high recognition accuracy (98.6% for static gestures, 95% for dynamic gestures). The results also present the recognizing capacity for high-correlated gestures (e.g., “M” and “N”). Finally, we demonstrate its application for controlling the motion of a virtual character through 7 discrete commands in the VR interface.

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

Abstract: This paper presents a novel approach towards developing a unified Sign Language Recognition System catering to both American Sign Language (ASL) and British Sign Language (BSL) communities. Leveraging image processing techniques, our system aims to accurately interpret and translate sign gestures into textual or auditory outputs. By employing advanced algorithms and machine learning models, the system can effectively recognize complex sign patterns, allowing for seamless communication between signers and non-signers. We discuss the architecture and key components of our system, including image acquisition, preprocessing, feature extraction, and classification methodologies tailored to the distinct characteristics of ASL and BSL. Furthermore, we address challenges such as gesture variability, background noise, and lighting conditions, proposing solutions to enhance the robustness and accuracy of the recognition process. Through rigorous testing and evaluation, our system demonstrates promising results in real-world scenarios, showcasing its potential to bridge communication gaps and foster inclusivity for individuals within the deaf community.