A Review on Sign Language Recognition and Learning System

There has been growing interest in the development of new approaches and technologies for bridging the human-computer barrier. Hand gesture recognition is considered as an interaction technique having potential to communicate with machines. Human computer interaction (HCI) was never an easy task and lots of approaches are available to build such systems. Hand gesture recognition (HGR) using wearable data glove provides a solution to build a HCI system, but lags in terms of its computational time and poor interface. Pattern matching is one more solution which uses vision based techniques and provides strong interface to build HCI systems. But again, it requires complex algorithms which takes lots of computational time and hence limits its use in real time HCI applications. In this paper, we presented an eccentric approach for hand gesture recognition which is simple, fast and user independent and can be used to develop real time HCI applications. Based on proposed algorithm we built a system for Indian Sign Language recognition which converts Indian Sign numbers into text. The algorithm first captures the image of single handed gesture of speech/hearing impaired person using simple webcam and then using our proposed algorithm it classifies the gesture into its appropriate class. It uses simple logical conditions for gesture classification which make its use in real time HCI applications.

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

Sign language is an expressive visual language that uses the hand and face and is an important tool for deaf individuals to communicate. Standard means of recognition objectives cannot recognize complex gestures effectively, in turn limiting accessibility and transactions of communication effectively. This project seeks to close communication barriers by translating hand gestures into speech, text, and even serve as an output to virtual assistants. To date, prior research in signing recognition has focused primarily on identifiable hand gestures, often selecting a subset of Indian Sign Language (ISL) for classification. In this work, a deep learning method is proposed to perform static sign recognition through Convolutional Neural Networks (CNN) with the VGG19 architecture. The system is trained through a dataset of ISL gestures for predicting and classifying signs with high confidence. The predicted sign is converted to a text output, to then a speech output, stored as an audio file. In conclusion, the system provides accessibility through detection, enabling a user user experience for both signers/non-signers to communicate and add encryption with accessibility. The model produces a far more particular/accurate performance in sign language recognition, and follows through with promising assistive technology solutions

Sign language identification and recognition technique is composed by the gesture images detection and the hand gestures recognition. Gesture images detection is to locate the image part of palm and fingers from the captured pictures, and rotating them to the appropriate gesture posture. Both of them are the important pre-processing for sign language identification and recognition. Lose them, the correctness rate of the sign language recognition algorithms will be dropped down to an unacceptable level. This paper presents novel processing algorithms for the gesture images detection and recognition. In the detection process, it rotates an askew gesture to right position, and to delete the elbow and forearm parts from the captured pictures. In the recognition process, it includes two phases with the model construction and the sign language identification. In the model construction phase, the static hand gesture of sign language is constructed by the Gaussian mixture model, and the unknown gesture image is identified by Gaussian model match. Based on this presented static sign language detection and recognition algorithms, the correct recognition rate is about 94% in average.

Abstract—People with hearing or speech impairments often face communication barriers in their day-to-day lives.The need for accessible communication tools for individuals with hearing or speech impairments continues to grow as digital communication and technology-driven interaction become central to everyday life. Indian Sign Language (ISL), being a primary mode of com- munication for many, requires automated systems that can inter- pret gestures accurately and efficiently. Motivated by this need, our project presents an AI-driven ISL Recognition and Transla- tion System that integrates Computer Vision, Deep Learning, and Natural Language Processing (NLP) within a unified web-based platform. The system employs MediaPipe to extract reliable hand landmarks and uses a Convolutional Neural Network (CNN) to learn and classify spatial gesture patterns from webcam streams or uploaded videos. For text-to-sign translation, the framework applies NLP techniques—including tokenization, lemmatization, and tense detection—to generate meaningful ISL animations. When a direct gesture animation is not available, the system automatically breaks words into alphabet-level signs to ensure complete translation. The preprocessing pipeline includes region- of-interest extraction and normalization to maintain consistent input quality. Implemented using a Django web framework with an intuitive user interface, the system supports educational, assistive, and accessibility-focused applications. Overall, this project delivers a reliable, user-centered solution that enhances communication and promotes digital inclusivity for the hearing and speech-impaired community Keywords: Indian Sign Language (ISL), Gesture Recognition, Convolutional Neural Network (CNN), MediaPipe, Natural Lan- guage Processing (NLP), Assistive Technology.

Sign Language is most accepted and purposeful way of communication for deaf and mute folks of the society. signing uses gestures, head/body movements and facial expressions for communication. it's a strong manner of Non-Verbal communication among humans. each country has its own developed signing. The language which is employed in Bharat is named as “Indian Sign Language (ISL)”. terribly lessresearch work has been carried out during this space as ISL has standardized recently. presently several researchers have primarily focused on gesture recognition that has been recorded underneath static hand gesture. solely few works have been reported for recognition of dynamic hand gesture. several ways are developed to recognize alphabets and numbers of ISL. The major steps concerned in planning such a system area unit gesture acquisition, trailing and segmentation, feature extraction and gesture recognition. This paper presents a survey on varied hand gesture recognition approaches for ISL. General Terms : Hand Gesture Recognition, Human computer interaction, YOLO3

Surface electromyogram (sEMG) signals associated with multiple patterns of hand gestures is commonly used as input for prosthetic control. However, placing a relatively large number of sEMG sensors on the residual limb of the amputees, particularly via the prosthetic cavity constitute a major challenge. In this article, a new acquisition system that integrates fewer electrodes is developed to acquire high-quality physiological information for adequate recognition of various hand gestures. The developed system utilizes the characteristics of the chip and special circuit design to enable simultaneous acquisition of sEMG signal and hybrid-impedance (HI) signal from a pair of differential electrodes. The results obtained based on the recordings from 10 subjects indicate that adding impedance information increases the sEMG recognition rate of hand gesture with fewer electrodes. This study might provide a new solution for achieving high-precision of the hand gesture recognition with fewer electrodes.

Recommending music based on a user’s music preference is a way to improve user listening experience. Finding the correlation between the user data (e.g., location, time of the day, music listening history, emotion, etc.) and the music is a challenging task. In this paper, we propose an emotion-aware personalized music recommendation system (EPMRS) to extract the correlation between the user data and the music. To achieve this correlation, we combine the outputs of two approaches: the deep convolutional neural networks (DCNN) approach and the weighted feature extraction (WFE) approach. The DCNN approach is used to extract the latent features from music data (e.g., audio signals and corresponding metadata) for classification. In the WFE approach, we generate the implicit user rating for music to extract the correlation between the user data and the music data. In the WFE approach, we use the term-frequency and inverse document frequency (TF-IDF) approach to generate the implicit user ratings for the music. Later, the EPMRS recommends songs to the user based on calculated implicit user rating for the music. We use the million songs dataset (MSD) to train the EPMRS. For performance comparison, we take the content similarity music recommendation system (CSMRS) as well as the personalized music recommendation system based on electroencephalography feedback (PMRSE) as the baseline systems. Experimental results show that the EPMRS produces better accuracy of music recommendations than the CSMRS and the PMRSE. Moreover, we build the Android and iOS APPs to get realistic data of user experience on the EPMRS. The collected feedback from anonymous users also show that the EPMRS sufficiently reflect their preference on music.

We propose a novel approach for recognizing hand gestures by analyzing the data streams generated by the sensors attached to the human hands. We utilize the concept of ‘range of motion’ in the movement and exploit this characteristic to analyze the acquired data. We show that since the relative ‘range of motion’ of each section of the hand involved in any gesture is a unique characteristic of that gesture, it provides a unique signature for that gesture across different users. Based on this observation, we propose our approach for hand gesture recognition which addresses two major challenges: user-dependency and devicedependency. Furthermore, we show that our approach neither requires calibration nor involves training.We apply our approach for recognizing ASL signs and show that we can recognize static ASL signs with no training. Our preliminary experiments demonstrate more than 75% accuracy in sign recognition for the ASL static signs.

Hand gestures facilitate user interaction and immersion in virtual museum applications. These gestures allow users to navigate virtual exhibitions, interact with virtual artifacts, and control virtual environments naturally and intuitively. This study introduces a deep learning-driven approach for hand gesture recognition using wearable vision sensors designed for interactive virtual museum environments. The proposed approach employs an image-based feature extraction strategy that focuses on capturing five partial occupancy areas of the hand. Notably, a deep learning strategy using the bidirectional long short-term memory (Bi-LSTM) model is adopted to construct an effective model for hand gesture identification. The bidirectionality of Bi-LSTM enables it to capture dependencies in both forward and backward directions, providing a more comprehensive understanding of temporal relationships in the data. The bidirectional nature allows the model to better capture the dynamics and complexities of hand motions, leading to improved accuracy and robustness. The performance evaluation includes experiments on publicly available datasets, considering virtual and real museum scenarios. The results highlight the Bi-LSTM-based approach’s superiority by accurately distinguishing various hand gestures. The experimental findings demonstrate that combining the five area ratios and Bi-LSTM classification enables robust recognition of diverse hand gestures and effectively discriminates between similar actions, such as slide left and right classes. Additionally, it shows promising detection performance compared to conventional machine learning models and state-of-the-art (SOTA) methods. The presented approach is promising for enhancing user interaction and immersion in virtual museum experiences.

Sign language is the communication process of people with hearing impairments. For hearing-impaired communication in Bangladesh and parts of India, Bangla sign language (BSL) is the standard. While Bangla is one of the most widely spoken languages in the world, there is a scarcity of research in the field of BSL recognition. The few research works done so far focused on detecting BSL alphabets. To the best of our knowledge, no work on detecting BSL words has been conducted till now for the unavailability of BSL word dataset. In this research, a small static-gesture word dataset has been developed, and a deep learning-based method has been introduced that can detect BSL static-gesture words from images. The dataset, “BSLword” contains 30 static-gesture BSL words with 1200 images for training. The training is done using a multi-layered convolutional neural network with the Adam optimizer. OpenCV is used for image processing and TensorFlow is used to build the deep learning models. This system can recognize BSL static-gesture words with 92.50% accuracy on the word dataset.

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

Barriers in expressive and receptive communication for individuals with hearing and speech impairments remain a significant challenge to achieving equitable social interaction and digital accessibility. These limitations restrict their participation in everyday conversations, education, and professional environments, emphasizing the urgent need for intelligent assistive communication technologies. This research introduces an AI-driven real-time Indian Sign Language (ISL) recognition framework that integrates advanced computer vision and deep learning techniques to translate hand gestures into textual outputs. The primary objective of this study is to design a lightweight, accurate, and real-time sign language translator suitable for Divyangjan, ensuring inclusivity in educational and social interactions. The proposed research employs Mediapipe-based hand landmark extraction to detect 21 key points in each gesture frame, followed by preprocessing and normalisation to create robust spatial representations. A hybrid Convolutional Neural Network processes these landmark vectors through a Bidirectional Long Short-Term Memory (CNN–BiLSTM) model that captures both spatial and temporal dependencies in gesture motion, allowing it to recognise static and dynamic gestures such as J and Z. The system was trained on a self-collected dataset of over 26,000 gesture images covering all 26 ISL alphabets. Experimental analysis demonstrates that the proposed model achieved 97.8 percent accuracy, 97.4 percent precision, 97.2 percent recall, 97.3 percent F1-score, and a validation loss of only 0.08, outperforming traditional classifiers such as Random Forest and SVM by a significant margin. The trained model performs robustly under varying lighting, background, and hand orientation conditions, ensuring high reliability for real-world deployment. The novelty of this study lies in the fusion of Mediapipe landmark extraction with a temporal deep learning framework for continuous ISL gesture translation within an interactive CustomTkinter GUI. This human-centric, computationally efficient design enables accessible, real-time communication for the deaf and hard-of-hearing community, contributing to socially assistive AI systems that promote digital inclusivity and empowerment for Divyangjan.

Breast density is one of the most significant factors that is associated with cancer risk. In this study, our purpose was to develop a supervised deep learning approach for automated estimation of percentage density (PD) on digital mammograms (DMs). The input ‘for processing’ DMs was first log-transformed, enhanced by a multi-resolution preprocessing scheme, and subsampled to a pixel size of 800 µm × 800 µm from 100 µm × 100 µm. A deep convolutional neural network (DCNN) was trained to estimate a probability map of breast density (PMD) by using a domain adaptation resampling method. The PD was estimated as the ratio of the dense area to the breast area based on the PMD. The DCNN approach was compared to a feature-based statistical learning approach. Gray level, texture and morphological features were extracted and a least absolute shrinkage and selection operator was used to combine the features into a feature-based PMD. With approval of the Institutional Review Board, we retrospectively collected a training set of 478 DMs and an independent test set of 183 DMs from patient files in our institution. Two experienced mammography quality standards act radiologists interactively segmented PD as the reference standard. Ten-fold cross-validation was used for model selection and evaluation with the training set. With cross-validation, DCNN obtained a Dice’s coefficient (DC) of 0.79 ± 0.13 and Pearson’s correlation (r) of 0.97, whereas feature-based learning obtained DC = 0.72 ± 0.18 and r = 0.85. For the independent test set, DCNN achieved DC = 0.76 ± 0.09 and r = 0.94, while feature-based learning achieved DC = 0.62 ± 0.21 and r = 0.75. Our DCNN approach was significantly better and more robust than the feature-based learning approach for automated PD estimation on DMs, demonstrating its potential use for automated density reporting as well as for model-based risk prediction.

The move to standardize Indian Sign Language has created an opportunity for researchers to focus on solving local problems, to increase its reach. In this paper, a survey and assessment of the techniques applied to the recognition and conversion of Indian Sign Language are performed. An overview of the techniques used in sign language recognition for Indian Sign Language is provided to understand the status of research in this field. Following this, a comparison of techniques aimed at rendering a more detailed picture of the research results is presented. The challenges faced by researchers, the limitations of current techniques, and the need for improved research in this area are highlighted. With the intent of spurring more in-depth research, key areas within the approaches and techniques in need of improvement are summarized.