A comparative analysis of Indian sign language recognition using deep learning models

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

The purpose of the study is to provide a literature review of the work done on sign language in Pakistan and the world. This study also provides a framework of an already developed prototype to translate Pakistani sign language into speech and text while using convolutional neural networking (CNN) to facilitate unimpaired teachers to bridge the communication gap among the deaf learners and unimpaired teachers. Due to the lack of sign language teaching, unimpaired teachers face difficulty in communicating with impaired learners. This communication gap can be filled with the help of this translation tool. Research indicates that a prototype has been evolved that can translate the English textual content into sign language and highlighted that there is a need for translation tool which can translate the signs into English text. The current study will provide an architectural framework of the Pakistani sign language to English text translation tool that how different components of technology like deep learning, convolutional neural networking, python, tensor Flow, and NumPy, InceptionV3 and transfer learning, eSpeak text to speech help in the development of a translation tool prototype. Keywords : Pakistan sign language (PSL), sign language (SL), translation, deaf, unimpaired, convolutional neural networking (CNN). DOI : 10.7176/JEP/10-15-18 Publication date :May 31 st 2019

Speech is the major way of human communication, but when it is limited, humans move to tactile kinaesthetic communication. People with speech-hearing impairments use sign language as an example of such adaptations. The deaf community uses Indian sign language (ISL) throughout India. In India, 250 licensed sign language interpreters are serving a deaf population of 1.8 to 7 million individuals. ISL interpreters are badly needed at institutes and places where persons with hearing impairments communicate. An Indian sign language picture database for English alphabets is established in this project. To prepare it for training, several pre-processing techniques were used. The effectiveness of deep learning neural networks is frequently influenced by the quantity of data available. As a result, data augmentation, a strategy for adding more and diverse samples to train datasets, was used to boost the effectiveness and outcomes of machine learning models. Our model is trained in CNN models utilizing transfer learning methodologies, with an accuracy of 95% for vgg16 and an accuracy of 92% for the inception model. More study on this research, as well as real-time implementation, has the potential to better connect people with hearing loss to society.

People around the world with speech and hearing impairment use a media of communication known as ‘Sign language’. In recent times, Sign language is omnipresent. However, there exists a challenge for people who do not know sign language, to communicate with people who can communicate exclusively using sign language. This gap can be bridged by using technologies of recent times to recognize gestures and design intuitive systems with deep learning. The aim of this paper is to recognise American Sign Language gestures dynamically and create an intuitive system which provides sign language translation to text and speech of various languages. The system uses Convolutional neural network, natural language processing, language translation and text-to-speech algorithms. It is capable of recognizing hand gestures dynamically and predicting the corresponding letters to form a desired sentence accurately.

In this paper, we present a real-time translation system that bridges the communication gap between the hearing and non-hearing communities. Our system converts English text to Indian Sign Language (ISL) and vice versa, using Natural Language Processing (NLP) techniques and deep learning-based gesture recognition. The system supports video-based gesture recognition for ISL and provides accurate text translations in real-time. This study addresses the technical challenges involved, including feature extraction from gestures and translating com- plex ISL sentences using neural networks like LSTM. Keywords- Indian Sign Language (ISL), Sign Language Translation, Gesture Recognition, Deep Learning, LSTM Model, Mediapipe Holistic, Text-to-Sign Conversion, Dy- namic Gesture Segmentation, Fingerspelling, Natural Lan- guage Processing (NLP), Pose Estimation, Hand Landmark Tracking, Real-Time Sign Language Recognition, Data Aug- mentation, Accessibility Technology

Barriers in expressive and receptive communication for individuals with hearing and speech impairments remain a significant challenge to achieving equitable social interaction and digital accessibility. These limitations restrict their participation in everyday conversations, education, and professional environments, emphasizing the urgent need for intelligent assistive communication technologies. This research introduces an AI-driven real-time Indian Sign Language (ISL) recognition framework that integrates advanced computer vision and deep learning techniques to translate hand gestures into textual outputs. The primary objective of this study is to design a lightweight, accurate, and real-time sign language translator suitable for Divyangjan, ensuring inclusivity in educational and social interactions. The proposed research employs Mediapipe-based hand landmark extraction to detect 21 key points in each gesture frame, followed by preprocessing and normalisation to create robust spatial representations. A hybrid Convolutional Neural Network processes these landmark vectors through a Bidirectional Long Short-Term Memory (CNN–BiLSTM) model that captures both spatial and temporal dependencies in gesture motion, allowing it to recognise static and dynamic gestures such as J and Z. The system was trained on a self-collected dataset of over 26,000 gesture images covering all 26 ISL alphabets. Experimental analysis demonstrates that the proposed model achieved 97.8 percent accuracy, 97.4 percent precision, 97.2 percent recall, 97.3 percent F1-score, and a validation loss of only 0.08, outperforming traditional classifiers such as Random Forest and SVM by a significant margin. The trained model performs robustly under varying lighting, background, and hand orientation conditions, ensuring high reliability for real-world deployment. The novelty of this study lies in the fusion of Mediapipe landmark extraction with a temporal deep learning framework for continuous ISL gesture translation within an interactive CustomTkinter GUI. This human-centric, computationally efficient design enables accessible, real-time communication for the deaf and hard-of-hearing community, contributing to socially assistive AI systems that promote digital inclusivity and empowerment for Divyangjan.

Sign language plays a crucial role in the lives of over 63 million people in India who live with hearing impairments. It is not just a method of communication; it is a vibrant, expressive language that allows them to connect with the world around them. However, the biggest challenge these individuals face is the absence of reliable, real-time translation systems specifically designed for Indian Sign Language (ISL). This gap leads to significant isolation in everyday situations, such as classrooms where lessons are delivered verbally, workplaces where meetings rely on spoken discussions, or even simple social gatherings were conversations flow without interpretation. Without tools to bridge this divide, hearing- impaired people often feel excluded, limiting their opportunities for education, employment, and social integration. To tackle this pressing issue, this paper introduces an innovative, vision-based system for real-time recognition of ISL. This system transforms live hand gestures captured by an ordinary webcam into readable text and audible speech, all powered by free, open-source software. No expensive hardware or proprietary tools are required, making it accessible to a wide audience. At its core, the system employs MediaPipe, a lightweight library from Google, to detect and track the key landmarks on hands—think of these as the joints and fingertips that form the building blocks of every sign. Once these landmarks are identified, a sophisticated deep learning model combining Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks steps in to analyze and interpret sequences of gestures. This hybrid model excels at understanding not just isolated signs but also the fluid, continuous phrases that make up natural signing

Hearing Impaired individuals routinely encounter limitations in their involvement in social interactions, access to intriguing information, and participation in everyday activities, among various other aspects. However, the hardest part of their interactions with regular people is communication, because sign language is the primary language of those who are hearing impaired. However, the general public is unaware of sign language. Each country has its own sign language. However, there are some striking similarities between them. In Sri Lanka, hearing impaired people use Sri Lankan Sign Language (SLSL) as their communication language. There is several research done on Sign Language recognition and translation. But no fully functioning system is utilized for Sri Lankan Sign Language translation. To find the gap in this area, we conducted a systematic literature review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method that analyses 12 studies on Sign Language Translation (SLT). As per the literature review, Image Processing (IP) and Convolutional Neural Networks (CNN) are the most used techniques for Sign Language translation. But these methods have limitations: not enough data, differences in how people use sign language, difficulty in translating in real-time, not capturing cultural aspects, needing specific equipment, and understanding the context of conversations. Recognizing and solving these problems is important, especially for languages like SLSL. Future research should focus on getting more data, making translation work for different cultures, and improving real-time translation. This will help hearing impaired people communicate better with others.

Abstract—People with hearing or speech impairments often face communication barriers in their day-to-day lives.The need for accessible communication tools for individuals with hearing or speech impairments continues to grow as digital communication and technology-driven interaction become central to everyday life. Indian Sign Language (ISL), being a primary mode of com- munication for many, requires automated systems that can inter- pret gestures accurately and efficiently. Motivated by this need, our project presents an AI-driven ISL Recognition and Transla- tion System that integrates Computer Vision, Deep Learning, and Natural Language Processing (NLP) within a unified web-based platform. The system employs MediaPipe to extract reliable hand landmarks and uses a Convolutional Neural Network (CNN) to learn and classify spatial gesture patterns from webcam streams or uploaded videos. For text-to-sign translation, the framework applies NLP techniques—including tokenization, lemmatization, and tense detection—to generate meaningful ISL animations. When a direct gesture animation is not available, the system automatically breaks words into alphabet-level signs to ensure complete translation. The preprocessing pipeline includes region- of-interest extraction and normalization to maintain consistent input quality. Implemented using a Django web framework with an intuitive user interface, the system supports educational, assistive, and accessibility-focused applications. Overall, this project delivers a reliable, user-centered solution that enhances communication and promotes digital inclusivity for the hearing and speech-impaired community Keywords: Indian Sign Language (ISL), Gesture Recognition, Convolutional Neural Network (CNN), MediaPipe, Natural Lan- guage Processing (NLP), Assistive Technology.

Objective: Development of Indian Sign Language video dictionary is essential in the today’s world of computerization. Though a lot of human video sign language dictionaries are available, we aim to develop the Indian Sign Language dictionary using synthetic animation which uses the computer generated cartoon rather than real human. Methods/Statistical Analysis: Sign Language cannot be spoken or written unlike other languages like English, Punjabi, Hindi, etc. The most commonly used words in Indian Sign Language are categorized and then these words are converted into the sign language writing notation (HamNoSys - Hamburg Notation System). This HamNoSys notation is then converted into SiGML (Signing Gesture Markup Language) using which the synthetic animation (using a computer generated cartoon) of the sign is generated. Findings: The synthetic animations are better as compared to human videos in terms of memory consumption, standardization, and flexibility. Synthetic animations can be modified as per the requirement whereas the human videos cannot be modified. The only drawback that seem is, these synthetic animations may lack the natural non-manual component of sign. Applications/Improvements: The synthetic dictionary created in this work can be used for translation system in which spoken or written sentence can be converted into the sign language animation. The dictionary created can be used to education to hard of hearing people. Display boards can be created for displaying the important messages in Indian sign language at the public gathering.

Sign Language is a way of communication used by hearing-impaired people. As many normal people don’t know the Sign Language it is difficult to understand by them to understand thus, increasing the communication gap between normal and hearing-impaired people. To overcome this problem many systems,using Machine Learning(ML), Artificial Intelligence(AI), Deep Learning(DL), computer vision methods, were developed and being developed. These systems are used to translate Indian Sign Language(ISL) to understandable text, facilitating the communication between normal and hearing-impaired people. This study briefly explains various type of Indian Sign Language(ISL) recognition systems.

Sign Language is the main communication medium for deaf people, in Indonesia itself there are several types of sign language, one of which is SIBI or Indonesian Sign Language System. This type of SIBI sign language is one type of sign language used in special school (SLB) environments. Deaf children are not yet able to use sign language and others have difficulty communicating with people around them. This research aims to help deaf children learn sign language, as well as help them communicate through sign language interpreters. The methodology applied to the application is by using a Convolutional Neural Network with the MobileNetV2 architecture. CNN is an algorithm that is included in the artificial neural network category which has the advantage of having a very high level of accuracy in classification, and MobileNetV2 is a form of Convolutional Neural Network (CNN) architecture that is able to overcome the need for excess computing resources created by Google researchers. so that it can be used on mobile devices or cell phones. The model training stage obtained an accuracy value of 98.99% with a total of 100 epochs. The model trained did not experience overfitting or underfitting so the model can be used. The testing stage uses black box testing to ensure the application is running correctly. The final stage is conducting research using TAM (Technology Acceptance Model) to measure the level of approval of the application using a Likert scale. Testing using black box testing was successful and met expectations, and the implementation of the application received an approval rate of 78.4% on average.

A hand sign for a disabled patient is a useful tool for human-to-human communication that can be used for a variety of purposes. They are frequently utilised by speech-impaired people around the world for communication since they are a natural form of engagement. In actuality, this group makes up around 1% of the Indian population. To enable easy communication among the signer and non-signer communities, an effective sign language recognition system (SLRS) can identify sign language motions. This research proposes a deep learning-based SLRS for classification that is computer vision-based. A sample of Auslan (Australian Sign Language (ASL)) signs is included in the dataset. Using top-notch position trackers, 27 examples of each of the 95 Auslan signs were recorded by a native signer. However, films captured Indian Sign Language (ISL) was filmed utilising numerous signs. Three individuals submitted 26 letters (A–Z) and 10 numbers (0–9). Few background removal techniques were performed on the samples and various skin color segmentation process was done. Third, a brand-new and reliable model for classifying ISL motions has been suggested utilising the mutation denoising autoencoder (MDAE). ISL and ASL datasets are publicly available and have been used to measure the effectiveness of the proposed system. In terms of measures like precision, recall, F1-Score, and accuracy, the proposed system is contrasted with well-known computer vision algorithms like the Support Vector Machine (SVM), Convolution Neural Network (CNN), Enhanced Convolution Neural Network (ECNN), and DAE. Python has been used to implement and test these categorisation techniques.

A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier