Transfer learning approaches in deep learning for Indian sign language classification

Sign language is mandatory and mostly used by huge number of individuals (particularly disabled) in society and it is another way of communication; The form of the hand, the contour of the movement, and the posture of the hand, face, and body parts are all distinct in each sign language; as a result, visual sign language identification is a challenging area of computer vision research; various academics have suggested several models in recent years that have been greatly improved using deep learning approaches. This study looked at the vision-based sign language recognition models suggested utilizing deep learning approaches in the last five years. Although the suggested model’s general trend reveals that sign language recognition accuracy has substantially increased, specific problems still have to be overcome. There are various and multiple languages in the world, people from different languages, have their alphabets and sign to communicate with each other. Many language speakers have developed a way to communicate with disabled people like Indian sign Language, French sign language several more; In this research paper, a novel Pashto sign language has been worked on. The most spoken language in southern Afghanistan is the Pashto language, and it has its grammar and history; it has approximately 40 million population in the world. The aim is to make a system by which disabled people of this language can communicate with the society. The system will take the signs and gestures of disabled people to change them into the Alphabet of Pashto language. 2500 dataset images are prepared and CNN model i s a ppl i e d to recognize the hand gestures and have executed the model with an accuracy of 98%.

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

Sign Language is a way of communication used by hearing-impaired people. As many normal people don’t know the Sign Language it is difficult to understand by them to understand thus, increasing the communication gap between normal and hearing-impaired people. To overcome this problem many systems,using Machine Learning(ML), Artificial Intelligence(AI), Deep Learning(DL), computer vision methods, were developed and being developed. These systems are used to translate Indian Sign Language(ISL) to understandable text, facilitating the communication between normal and hearing-impaired people. This study briefly explains various type of Indian Sign Language(ISL) recognition systems.

Sign language is a universal way of communication for challenged people with speaking and hearing limitations. Multiple mediums are accessible to translate or to acknowledge sign language and convert them to text. However, the text to signing conversion systems has been rarely developed; this is often thanks to the scarcity of any sign language dictionary. Our project aims at creating a system that consists of a module that initially transforms voice input to English text and which parses the sentence, then to which Indian sign language grammar rules are applied. This is done by eliminating stop words from the reordered sentence. Indian Sign Language (ISL) does not sustain the inflections of the word. Hence, stemming is applied to vary over the words to their root/ stem class. All words of the sentence are then checked against the labels in the dictionary containing videos representing each of the words. The present systems are limited to only straight conversion of words into ISL, whereas the proposed system is innovative, as our system aims to rework these sentences into ISL as per grammar in the real domain.

Objective: Development of Indian Sign Language video dictionary is essential in the today’s world of computerization. Though a lot of human video sign language dictionaries are available, we aim to develop the Indian Sign Language dictionary using synthetic animation which uses the computer generated cartoon rather than real human. Methods/Statistical Analysis: Sign Language cannot be spoken or written unlike other languages like English, Punjabi, Hindi, etc. The most commonly used words in Indian Sign Language are categorized and then these words are converted into the sign language writing notation (HamNoSys - Hamburg Notation System). This HamNoSys notation is then converted into SiGML (Signing Gesture Markup Language) using which the synthetic animation (using a computer generated cartoon) of the sign is generated. Findings: The synthetic animations are better as compared to human videos in terms of memory consumption, standardization, and flexibility. Synthetic animations can be modified as per the requirement whereas the human videos cannot be modified. The only drawback that seem is, these synthetic animations may lack the natural non-manual component of sign. Applications/Improvements: The synthetic dictionary created in this work can be used for translation system in which spoken or written sentence can be converted into the sign language animation. The dictionary created can be used to education to hard of hearing people. Display boards can be created for displaying the important messages in Indian sign language at the public gathering.

Sign languages are crucial in promoting independent living for deaf and hard-of-hearing individuals. They offer a means of communication that facilitates access to various aspects of daily life and supports autonomy. The deaf community in India, estimated at over 18 million individuals, faces significant communication barriers due to the limited use of Indian sign language (ISL) by the general population. This chapter explores the potential of machine learning (ML) for Indian sign language detection as a bridge between the deaf and hearing communities. The authors delve into existing research on sign language recognition (SLR) systems, analyze the challenges specific to ISL detection, and propose future directions leveraging advancements in ML. By examining the sociolinguistic landscape and the potential impact of ML-powered ISL detection, this chapter aims to contribute to a more inclusive society.

Sign language is a form of communication where people use bodily gestures, particularly those of hands and arms. This method of communication is put into motion when spoken communication is unattainable or disfavored. There are very few people who can translate sign language and readily understand them. It would be convenient for the hearing-impaired to have a platform where their sign language could be translated easily. Hence, through this study, with the help of artificial neural networks, we wish to compare how various widely implemented deep learning architectures respond to faultless translation of Indian sign language for the native audience. This research would streamline the development of software tools that can accurately predict or translate ISL. For the purpose of understanding the method of training the machine and exploring our model’s performance without any optimizations, a Convolutional Neural Network architecture was implemented. Over the course of our research, there have been several Pre-trained Transfer Learning Models implemented that have yielded promising results. The research aims to contrast how various convolutional neural networks perform while translating Indian Sign Actions on a custom dataset that factors in illumination, angles, and different backgrounds to provide a balanced and distinctive set of images. The goal of this study is to make clear comparisons between the various deep learning frameworks. Hence, a fresh Indian sign language dataset is introduced. Since every dataset in the field of deep learning has special properties that may be utilized for the betterment of the existing models, the development of a fresh dataset could be viewed as a development in the field. The optimum model for our task: classification of these gestures is found to be ResNet-50 (Accuracy = 98.25% and F1-score = 99.34%), and the least favorable was InceptionNet V3 (Accuracy = 66.75%, and F1-score = 70.89%).

Sign language simplifies communication for the deaf community.A kind of communication known as sign language uses visual sign patterns to convey feelings.When a deaf community wants to convey their ideas, speech, and hearing with sighted people, there is a communication gap.At the moment, two communities use human interpreters, which can be expensive and inconvenient.It might be challenging to distinguish across sign languages since their alphabets differ from one another.American Sign Language (ASL) has a very different alphabet than Indian Sign Language or Italian Sign Language (ISL).Thus, regional differences in sign language exist.We use convolutional neural network for capturing various image features and for feature extraction.The model is able to generalize on users and surroundings and the accuracy is high compared to other methods.

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

The development of Human-Computer Interaction (HCI) will not only empower the disabled with access to technology but also help to build tools for assisting the disable and their caregivers. Some disabled people are deaf, they use signs to communicate. Signs are emblematic forms which convey a specific meaning. These emblems construct the sign language, which is a non-verbal language used by the deaf community. This research attempts to build a Bangla Sign language recognition system that can recognize signs of both hands. Hands vary in shapes and sizes, as well as signs vary in orientations and motions. Accurate feature extraction is necessary for such systems. For such purposes, deep learning approaches can prove to be effective for the classification and feature engineering of images. In the beginning, an integrated CNN-LSTM model is proposed for building a Sign language Recognition System, a Bangla Sign Language (BSL) dataset consisting of Bangla lexical signs is considered. This dataset consists of 13,400 images comprising thirty-six classes of Bangla lexical signs. The model is trained using a CNN-LSTM model. This model produces a training accuracy of 90% and a testing accuracy of 88.5%. The proposed model is compared to CNN and other CNN state of the art models, namely VGG16, VGG9 and MobileNet. The CNN model and other CNN state undergo the problem of overfitting as their training accuracy is greater than the testing accuracy.

The Sign Language Recognition System (SLRS) represents an advanced technology designed to improve accessibility in communication within the deaf community, replacing the conventional method of relying on human translators. However, the current SLR systems for Indian Sign Language (ISL) suffer from inefficiency and a lack of robustness due to their reliance on a single classifier. To address these limitations, this research paper introduces a robust and efficient SLRS named Hybrid Deep Ensemble Network (HD-ENet1) for recognizing ISL words. The proposed HD-ENet1 incorporates a technique for selecting keyframes based on histogram differences which effectively eliminates redundant frames. To enhance system performance and ensure robustness, the method leverages a weighted average ensemble approach, utilizing three distinct models: LSTM, ConvLSTM, and BiLSTM networks as base classifiers. In order to determine the optimal combination of weights, the HD-ENet1 integrates the Hill Climbing (HC) optimization technique. The performance of the HD-ENet1 is measured with repeated k-fold cross validation technique where the proposed system demonstrated an average classification accuracy of 97.32% on the Indian Sign Language (ISL) dataset and 98.78% on the benchmark LSA64 dataset. A comparative analysis is conducted with several SLRS, revealing that the suggested approach consistently outperformed the existing approaches.

Sign Language is one of the prime applications of facial expression recognition system. It is an easy task for a human being to identify the expression of a person in a particular image sequence but not many efficient fully automated systems are available to perform this task. Lot of work has been done to automate the recognition system for many sign languages like Turkish Sign Language (TSL) [2], American Sign Language (ASL) [3], Chinese Sign Language (CSL) [4] etc. But rare effort has been done for any of the Indian Sign Language (ISL). Therefore, in this paper, a large video data set of Indian Sign Language (ISL) is prepared consisting of some of the expressions. Further, an automatic facial expression recognition system is developed to identify the expressions performed in these videos. Also same algorithm is run on the Turkish sign language data set provided by Aran et al. [2] and a comparison is provided between these two dataset.

The deaf and hard-of-hearing community often experiences communication barriers due to people who are not well-versed in the use of sign language.[1] This project would address this problem by developing an all-embracing machine learning (ML) model which would be able to interpret and translate the hand movements of American Sign Language (ASL) and Indian Sign Language (ISL) into corresponding spoken or written language in real time.[2] In using data from cameras, the system is designed to precisely predict and translate gestures both ASL and ISL. The solution integrates Natural Language Processing for context-aware translations and hence lets the system interpret the surrounding context for more meaningful and accurate implications. Moreover, the final project will explore the utilization of advanced Gesture recognition technologies, which might include the deep learning models and recurrent neural networks, shall enhance the system's accuracy as well as adaptability.[3] Data input multimodality will enhance gesture recognition: examples include skeletal tracking and hand shape analysis. User-friendly interface; it will be made possible for the users to interact with customization options of settings, preferences, including but not limited to language voice options.[6] Moreover, the project includes the potential for integration with other assistive technologies, such as speech-to-text devices and AR equipment among others, to provide a comprehensive communication assistant. At the end of it all, this project would empower the deaf and hard-of-hearing community by ensuring effective communication between the hearing population and the deaf and hard-of-hearing, social inclusion and access. The solution will be plentiful in schools, the workplaces, and public services, so all will be provided with the potentially useful tool of increasing accessibility and understanding.

Indian Sign Language (ISL) is the primary visual communication medium for approximately 63 million deaf and hard-of-hearing individuals in India, yet computational resources for ISL processing remain severely limited compared to spoken languages. This paper presents a comprehensive framework for developing an AI-powered mobile application capable of bidirectional translation between English/Hindi audio-visual content and Indian Sign Language. The system integrates three critical components: large-scale dataset aggregation from sources including iSign benchmark (118,000 video-sentence pairs) and ISLTranslate (31,222 sentence pairs), advanced natural language processing pipelines for grammatical transformation, and computer vision techniques for sign generation and recognition. We discuss the architectural design of such systems, the processing pipeline from audio input through NLP tokenization, lemmatization, and part-of-speech tagging to final ISL video output, and present evaluation methodologies using Dynamic Time Warping and machine translation metrics. The paper incorporates findings from recent research on ISL datasets, linguistic properties unique to sign languages, and practical implementation considerations for real-world deployment on mobile platforms. Index Terms—Indian Sign Language, bidirectional translation, natural language processing, computer vision, mobile application, accessibility technology, sign language datasets, neural machine translation.

Barriers in expressive and receptive communication for individuals with hearing and speech impairments remain a significant challenge to achieving equitable social interaction and digital accessibility. These limitations restrict their participation in everyday conversations, education, and professional environments, emphasizing the urgent need for intelligent assistive communication technologies. This research introduces an AI-driven real-time Indian Sign Language (ISL) recognition framework that integrates advanced computer vision and deep learning techniques to translate hand gestures into textual outputs. The primary objective of this study is to design a lightweight, accurate, and real-time sign language translator suitable for Divyangjan, ensuring inclusivity in educational and social interactions. The proposed research employs Mediapipe-based hand landmark extraction to detect 21 key points in each gesture frame, followed by preprocessing and normalisation to create robust spatial representations. A hybrid Convolutional Neural Network processes these landmark vectors through a Bidirectional Long Short-Term Memory (CNN–BiLSTM) model that captures both spatial and temporal dependencies in gesture motion, allowing it to recognise static and dynamic gestures such as J and Z. The system was trained on a self-collected dataset of over 26,000 gesture images covering all 26 ISL alphabets. Experimental analysis demonstrates that the proposed model achieved 97.8 percent accuracy, 97.4 percent precision, 97.2 percent recall, 97.3 percent F1-score, and a validation loss of only 0.08, outperforming traditional classifiers such as Random Forest and SVM by a significant margin. The trained model performs robustly under varying lighting, background, and hand orientation conditions, ensuring high reliability for real-world deployment. The novelty of this study lies in the fusion of Mediapipe landmark extraction with a temporal deep learning framework for continuous ISL gesture translation within an interactive CustomTkinter GUI. This human-centric, computationally efficient design enables accessible, real-time communication for the deaf and hard-of-hearing community, contributing to socially assistive AI systems that promote digital inclusivity and empowerment for Divyangjan.