Segmentation, Tracking and Feature Extraction for Indian Sign Language Recognition

ABSTRACTSign language is a communication tool for deaf and dumb people that includes known signs or body gestures to transfer meanings. It uses shapes, directions, movements of hands, and also facial expressions. A sign not only transmits a word but also conveys a tone. Many of the deaf people are not only able to speak, but also not able to write or read a language, so developing sign language translation or in other words sign language recognition (SLR) system can be very vital in their life. The SLR is extremely desired because of its capability to overcome the obstacles between deaf and normal people. It is one of the most important research fields in the human computer interaction studies. Hence, this paper presents an overview of the recent main research works with the vision-based SLR system, and the existing recognition techniques are discussed. Next, we focus on video-based SLR system and perform continuous SLR within video sequences.

Sign language recognition (SLR) system is helpful for individuals who can't talk or hear. One can easily communicate with this kind of people using SLR system. In this paper we are focusing on SLR approach where we have apply our proposed algorithm to recognize hand gestures of routine words like Morning, Night, Monday, Tuesday and so forth. Total 10 routine words we have recognize using our proposed technique. For recognize hand pose we have utilized skin color detection algorithm. Further we have applied correlation-coefficient algorithm to distinguish closeness features which are further continue in Neuro-fuzzy(NF) classification algorithm to recognize words. Presented algorithm tried on Matlab. With taken of 10 routine words our proposed framework of techniques got 92% correction rate.

This article is dedicated to analyzing advanced approaches in temporal modeling and real-time gesture recognition within sign language recognition (SLR) systems. Sign glosses are expressed through the spatio-temporal characteristics of visual information, which requires the use of sequence-processing models for their automatic recognition. The study primarily evaluates the effectiveness of three key models: Long Short-Term Memory (LSTM) networks, Temporal Convolutional Networks (TCN), and Transformer-based architectures. The article also examines methods applied for real-time analysis of sign glosses, including: Sliding window segmentation of video streams; Self-attention mechanisms for identifying dependencies between gestures; Gloss mapping algorithms for linking sign movements to linguistic units; Ontological integration techniques for enhancing semantic accuracy. Practical results indicate that combining temporal modeling with semantic analysis and contextual verification algorithms ensures continuous and high-accuracy recognition of sign movements. In particular, multimodal systems (video + sensor + gloss) utilizing Transformer-based approaches achieved superior performance in real-time conversion of continuous sign gloss streams into text. The findings of this study hold practical significance for the development of smart assistive devices for automatic sign language translation, interactive interfaces for hearing-impaired users, and specialized SLR platforms for educational and instructional purposes.

An optical modeless Sign Language Recognition (SLR) system is presented. The system uses the HAusdorf-Voronoi NETwork (HAVNET), an artificial neural network designed for 2D binary pattern recognition. It uses adaptation of the Hausdorff distance to determine the similarity between an input pattern and a learned representation. A detailed review of the architecture, the learning equations, and the recognition equations for the HAVNET network are presented. Competitive learning has been implemented in training the network using a nearest-neighbor technique. The SLR system is applied to the optical recognition of 24 static symbols from the American Sign Language convention. The SLR system represents the target images in a 80 X 80 pixel format. The implemented HAVNET network classifies the inputs into categories representing each of the symbols, using an output layer of 24 nodes. The network is trained with 5 different formats for each symbol and is tested with all 24 symbols in 15 new formats. Results from the SLR system without competitive training show shape identification problems, when distinguishing symbols with similar shapes. Implementation of competitive learning in the HAVNET neural network improved recognition accuracy on this task to 89%. The hand gestures are identified through a window search algorithm. Feature recognition is obtained from edge enhancement by applying a Laplacian filter and thresholding, which provides robustness to pose, color and background variations.

Sign language recognition devices are effective approaches to breaking the communication barrier between signers and non-signers and exploring human-machine interactions. Wearable gloves have been developed for gesture recognition and virtual reality applications by employing flexible sensors for motion detection and machine learning for data analysis. However, most existing wearable devices present limited sign language translating capacity due to the sensors' design and distribution. Here, we propose a cost-effective dual-hand soft fiber optic glove system consisting of multimode soft liquid-core fiber optic sensors, gyroscopes, wireless printed circuit boards, and batteries for sign language translation. In combination with different deep learning techniques and recognition strategies, the glove system can recognize static gestures and dynamic gestures of American Sign Language, and deduce the meaning of sentences by the sequence of gestures. The soft glove system exhibits a broad sign language range (10 numbers, 26 alphabets, 18 words, and 5 sentences meaning prediction), and high recognition accuracy (98.6% for static gestures, 95% for dynamic gestures). The results also present the recognizing capacity for high-correlated gestures (e.g., “M” and “N”). Finally, we demonstrate its application for controlling the motion of a virtual character through 7 discrete commands in the VR interface.

Abstract— Sign language is an essential mode of communication for individuals with hearing impairments, yet real-time translation remains a challenge due to complex hand gestures, facial expressions, and language variations. Traditional deep learning approaches, such as CNNs and RNNs, struggle with sequential dependencies and spatial feature extraction, limiting recognition accuracy. Recent advancements in Vision Transformers have significantly improved image-based learning by utilizing self- attention mechanisms to capture both spatial and temporal dependencies, making them highly effective for gesture recognition. This paper presents a bidirectional Sign Language Recognition and Translation System that employs ViT for sign recognition and a Pre-Recorded Gesture Database for text-to-sign conversion. The system captures real-time video input, extracts gesture features using ViT’s attention-based encoding, and converts recognized gestures into text. Conversely, it maps typed text to a corresponding pre-recorded sign animation, ensuring smooth and natural communication. By eliminating the need for gloss-based intermediaries and improving processing efficiency, the proposed system enhances accuracy, computational efficiency, and real-time performance, offering a scalable. solution for bridging communication gaps in the hearing-impaired community. KEYWORDS— SIGN LANGUAGE RECOGNITION, VISION TRANSFORMER, GESTURE-TO-TEXT, TEXT- TO- GESTURE, DEEP LEARNING, TRANSFORMER MODELS, REAL-TIME TRANSLATION

Indian sign language (ISL) is used by deaf people in India for their internal communication. Sign language recognition (SLR) system recognizes gestures of sign language and converts them into text and voice, thus enabling deaf people to communicate with others. In this paper, we have presented an automatic system for ISL hand gesture recognition. Modern features like Histogram of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), Fourier Descriptors, Hu Moments, and Zernike Moments have been extracted from training images of our own created dataset. Using these features, most commonly used classifiers like Support Vector Machine (SVM), Artificial Neural Network (ANN), K-Nearest Neighbors (KNN), and Linear Discriminant Analysis (LDA) have been trained and evaluated. Comparative performance analysis of these classifiers along with experimental results is the main topic of discussion included in this paper. Moreover, a user-friendly GUI of the system allows user to select any combination of features and classifiers to perform gesture recognition from both stored file and live camera. During experiment, it is found that the system is able to recognize gestures with a recognition rate up to 98.70%.

Sign language is the language used by the deaf, which is a comparatively steadier expressive system composed of signs corresponding to postures and motions assisted by facial expression. The objective of sign language recognition research is to see the language of deaf. The integration of sign language recognition and sign language synthesis jointly comprise a human-computer sign language interpreter, which facilitates the interaction between deaf and their surroundings. Considering the speed and performance of the recognition system, Cyberglove is selected as gesture input device in our sign language recognition system, Semi-Continuous Dynamic Gaussian Mixture Model (SCDGMM) is used as recognition technique, and a search scheme based on relative entropy is proposed and is applied to SCDGMM-based sign word recognition. Comparing with SCDGMM recognizer without searching scheme, the recognition time of SCDGMM recognizer with searching scheme reduces almost 15 times.

Sign is the way of communication when there is no appropriate verbal communication for deaf and dump persons. Sign Language Recognition System (SLRS) acts as a contrivance to perform the conversion of sign language into text or voice. Many classification techniques and algorithms with a variety of methods already exist in SLRS. But still only a few considerable contributions are there in Tamil Sign Language Recognition. The objective of this paper is to extract the features from the pre-processed image and propose an optimization based classifier to improve the classification accuracy in Tamil Sign Language images. The experimental result of the proposed work demonstrated that BPN-PSO outperforms existing classification techniques.

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

Sign languages are crucial in promoting independent living for deaf and hard-of-hearing individuals. They offer a means of communication that facilitates access to various aspects of daily life and supports autonomy. The deaf community in India, estimated at over 18 million individuals, faces significant communication barriers due to the limited use of Indian sign language (ISL) by the general population. This chapter explores the potential of machine learning (ML) for Indian sign language detection as a bridge between the deaf and hearing communities. The authors delve into existing research on sign language recognition (SLR) systems, analyze the challenges specific to ISL detection, and propose future directions leveraging advancements in ML. By examining the sociolinguistic landscape and the potential impact of ML-powered ISL detection, this chapter aims to contribute to a more inclusive society.

Deaf and hard-of-hearing individuals use sign language as a means of communication. However, those around them, especially family members like the children of deaf adults, may face communication challenges if they are unfamiliar with sign language. This issue has prompted numerous researchers to conduct studies on sign language translation and recognition. However, there is currently no publicly available dataset specifically for Malaysian sign language. This article introduces an image dataset of the Malaysian Sign Language (MySL) hand gestures used in everyday situations. The dataset, named MyWSL2023, comprises 3,500 images of ten static Malaysian sign language words collected from five participants (two males and three females) aged between 20 and 21 years old. The data collection took place indoors under normal lighting conditions. The MyWSL2023 dataset, which has been made freely accessible to all researchers, serves as a valuable resource for not only investigating and developing automated systems for hearing-impaired and deaf individuals but also gesture and sign language recognition using vision-based methods. The dataset can be accessed for free at https://data.mendeley.com/datasets/zvk55p7ktd.

Sign Language Recognition systems have been thoroughly researched in recent times to bridge the gap between the signer and non-signer community. This paper focuses on fulfilling the need for a thorough review of the works done in building such systems. The systems discussed herein for static Sign Language Recognition are broadly classified on Support Vector Machine, Convolutional Neural Network as classifiers. The identified phases for the reviewed systems include preprocessing, feature extraction and classification. The methods discussed in these phases include Principal Component Analysis, Histograms of Orientations Gradient, Hybrid Sign Invariant Feature Transformation. Further the limitations and improvements in approaches are relatively portrayed to give a better analysis for future work. The overall aim for this paper is to provide insight into making effective systems for Sign Language Recognition. This will make these systems easily available for educating the masses about static Sign Language. Thus, sign language will no longer remain a barrier.

Sign language recognition is one of the most promising sub-fields in gesture recognition research. A sign language is a language which, instead of acoustically conveyed sound patterns, uses visually transmitted sign patterns. Sign languages are commonly developed for hearing impaired communities, which can include interpreters, friends and families of hearing impaired people as well as people who are hard of hearing themselves. Developing a sign language recognition system will help the hearing impaired to communicate more fluently with the normal people. A simple sign language recognition system employing skin color segmentation and Neural Network has been developed. A simple segmentation process is carried out to separate the right and left hand regions from the image frame and in the preprocessing stage a simple vertical interleaving method has been proposed to reduce the size of the image. The 2D moment invariant features of the right and left hand interleaved image is obtained as features. Using the interleaved 2D-moment Invariant features, a simple neural network model has been developed. The system has been implemented and tested for its validity. Experimental results show that the system has a recognition rate of 92.58%.

Losing the ability to speak exerts psychological and social impacts on the affected people due to the lack of proper communication. Thus, Sign Language (SL) is considered a boon to people with hearing and speech impairment. SL has developed as a handy mean of communication that form the core of local deaf cultures. It is a visual–spatial language based on positional and visual components, such as the shape of fingers and hands, their location and orientation as well as arm and body movements. The problem is that SL is not understood by everyone, forming a communication gap between the mute and the able people. Multiple and systematic scholarly interventions that vary according to context have been implemented to overcome disability-related difficulties. Sign language recognition (SLR) systems based on sensory gloves are significant innovations that aim to procure data on the shape or movement of the human hand to bridge this communication gap, as the proposed system. The proposed model is a glove equipped with five flex sensors, interfacing with a control unit fixed on the arm, translating American Sign Language (ASL) and Arabic Sign Language (ArSL) to both text and speech, displayed on a simple Graphical User Interface (GUI). The proposed system aims to provide an affordable and user friendly SL translator system, working on the basis of Machine Learning (ML). However, it adapts to each person’s hand instead of using a generic data set. The system achieved 95% recognition rate with static gestures and up to 88% with dynamic gestures.