Sign Language Recognition Using Convolutional Neural Networks and Vgg19

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

The rapid spread of the novel corona virus disease (COVID-19) has disrupted the traditional clinical services all over the world. Hospitals and healthcare centers have taken extreme care to minimize the risk of exposure to the virus by restricting the visitors and relatives of the patients. The dramatic changes happened in the healthcare norms have made it hard for the deaf patients to communicate and receive appropriate care. This paper reports a work on automatic sign language recognition that can mitigate the communication barrier between the deaf patients and the healthcare workers in India. Since hand gestures are the most expressive components of a sign language vocabulary, a novel dataset of dynamic hand gestures for the Indian sign language (ISL) words commonly used for emergency communication by deaf COVID-19 positive patients is proposed. A hybrid model of deep convolutional long short-term memory network has been utilized for the recognition of the proposed hand gestures and achieved an average accuracy of 83.36%. The model performance has been further validated on an alternative ISL dataset as well as a benchmarking hand gesture dataset and obtained average accuracies of 97% and 99.34pm 0.66%, respectively.

A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier

Sign language is used by people having speaking and hearing disabilities. It generally has a set of words, where each word is represented by one or more hand gestures in sequence and may contain facial expressions. In order to address the interpretation/translation from sign language to English Language, we present our sign recognition approach for Indian sign language which aims to provide a method for interpreting signs in Indian sign language to words in English language translation. The approach is to have a vision based system in which the sequence of images representing a word in ISL is translated to equivalent English word. The translation would be done by means of Deep learning algorithms namely convolutional neural nets and recurrent neural nets. The system will be analyzing sequence of images, hence CNNs will analyze each image and their sequence is analyzed by LSTM (which is an implementation of RNN). We divided dataset into training dataset and testing dataset, which obtained 73.60% accuracy. The image distributions are kept fairly different in training and testing datasets.

Sign language recognition plays a crucial role in supporting communication between deaf communities and hearing individuals. In particular, Vietnamese Sign Language (VSL) recognition remains a challenging task due to the complexity of hand gestures and limited available datasets. However, most existing approaches address static gestures and dynamic sign phrases as separate recognition problems, often employing different feature representations and independent processing pipelines. This fragmentation increases system complexity and limits scalability for real-time sign language applications. This study proposes a unified hand-landmark-based deep learning framework for recognizing both static and dynamic VSL gestures within a single integrated system. The system begins by detecting hand landmarks through the MediaPipe hand tracking pipeline. Based on temporal motion analysis of landmark sequences, a gesture routing mechanism automatically determines whether the input corresponds to a static gesture or a dynamic sign phrase. Static gestures are classified using a convolutional neural network (CNN), while dynamic gestures are processed using a long short-term memory (LSTM) network to capture temporal dependencies. Experiments were conducted on a VSL dataset consisting of 23 static hand signs and 4 dynamic sign phrases collected from multiple participants with variations in hand shape and gesture execution style. The system performance is evaluated using accuracy, precision, recall, and F1-score. Experimental results demonstrate that the proposed framework achieves an average recognition accuracy of 92% for static gestures and 88.4% for dynamic sign phrases, outperforming traditional machine learning baselines. The proposed system provides a practical and efficient solution for VSL recognition and has potential applications in real-time assistive communication systems.

Sign language identification and recognition technique is composed by the gesture images detection and the hand gestures recognition. Gesture images detection is to locate the image part of palm and fingers from the captured pictures, and rotating them to the appropriate gesture posture. Both of them are the important pre-processing for sign language identification and recognition. Lose them, the correctness rate of the sign language recognition algorithms will be dropped down to an unacceptable level. This paper presents novel processing algorithms for the gesture images detection and recognition. In the detection process, it rotates an askew gesture to right position, and to delete the elbow and forearm parts from the captured pictures. In the recognition process, it includes two phases with the model construction and the sign language identification. In the model construction phase, the static hand gesture of sign language is constructed by the Gaussian mixture model, and the unknown gesture image is identified by Gaussian model match. Based on this presented static sign language detection and recognition algorithms, the correct recognition rate is about 94% in average.

Barriers in expressive and receptive communication for individuals with hearing and speech impairments remain a significant challenge to achieving equitable social interaction and digital accessibility. These limitations restrict their participation in everyday conversations, education, and professional environments, emphasizing the urgent need for intelligent assistive communication technologies. This research introduces an AI-driven real-time Indian Sign Language (ISL) recognition framework that integrates advanced computer vision and deep learning techniques to translate hand gestures into textual outputs. The primary objective of this study is to design a lightweight, accurate, and real-time sign language translator suitable for Divyangjan, ensuring inclusivity in educational and social interactions. The proposed research employs Mediapipe-based hand landmark extraction to detect 21 key points in each gesture frame, followed by preprocessing and normalisation to create robust spatial representations. A hybrid Convolutional Neural Network processes these landmark vectors through a Bidirectional Long Short-Term Memory (CNN–BiLSTM) model that captures both spatial and temporal dependencies in gesture motion, allowing it to recognise static and dynamic gestures such as J and Z. The system was trained on a self-collected dataset of over 26,000 gesture images covering all 26 ISL alphabets. Experimental analysis demonstrates that the proposed model achieved 97.8 percent accuracy, 97.4 percent precision, 97.2 percent recall, 97.3 percent F1-score, and a validation loss of only 0.08, outperforming traditional classifiers such as Random Forest and SVM by a significant margin. The trained model performs robustly under varying lighting, background, and hand orientation conditions, ensuring high reliability for real-world deployment. The novelty of this study lies in the fusion of Mediapipe landmark extraction with a temporal deep learning framework for continuous ISL gesture translation within an interactive CustomTkinter GUI. This human-centric, computationally efficient design enables accessible, real-time communication for the deaf and hard-of-hearing community, contributing to socially assistive AI systems that promote digital inclusivity and empowerment for Divyangjan.

A hand sign for a disabled patient is a useful tool for human-to-human communication that can be used for a variety of purposes. They are frequently utilised by speech-impaired people around the world for communication since they are a natural form of engagement. In actuality, this group makes up around 1% of the Indian population. To enable easy communication among the signer and non-signer communities, an effective sign language recognition system (SLRS) can identify sign language motions. This research proposes a deep learning-based SLRS for classification that is computer vision-based. A sample of Auslan (Australian Sign Language (ASL)) signs is included in the dataset. Using top-notch position trackers, 27 examples of each of the 95 Auslan signs were recorded by a native signer. However, films captured Indian Sign Language (ISL) was filmed utilising numerous signs. Three individuals submitted 26 letters (A–Z) and 10 numbers (0–9). Few background removal techniques were performed on the samples and various skin color segmentation process was done. Third, a brand-new and reliable model for classifying ISL motions has been suggested utilising the mutation denoising autoencoder (MDAE). ISL and ASL datasets are publicly available and have been used to measure the effectiveness of the proposed system. In terms of measures like precision, recall, F1-Score, and accuracy, the proposed system is contrasted with well-known computer vision algorithms like the Support Vector Machine (SVM), Convolution Neural Network (CNN), Enhanced Convolution Neural Network (ECNN), and DAE. Python has been used to implement and test these categorisation techniques.

Sign language is mandatory and mostly used by huge number of individuals (particularly disabled) in society and it is another way of communication; The form of the hand, the contour of the movement, and the posture of the hand, face, and body parts are all distinct in each sign language; as a result, visual sign language identification is a challenging area of computer vision research; various academics have suggested several models in recent years that have been greatly improved using deep learning approaches. This study looked at the vision-based sign language recognition models suggested utilizing deep learning approaches in the last five years. Although the suggested model’s general trend reveals that sign language recognition accuracy has substantially increased, specific problems still have to be overcome. There are various and multiple languages in the world, people from different languages, have their alphabets and sign to communicate with each other. Many language speakers have developed a way to communicate with disabled people like Indian sign Language, French sign language several more; In this research paper, a novel Pashto sign language has been worked on. The most spoken language in southern Afghanistan is the Pashto language, and it has its grammar and history; it has approximately 40 million population in the world. The aim is to make a system by which disabled people of this language can communicate with the society. The system will take the signs and gestures of disabled people to change them into the Alphabet of Pashto language. 2500 dataset images are prepared and CNN model i s a ppl i e d to recognize the hand gestures and have executed the model with an accuracy of 98%.

Communication between hearing-impaired and hearing individuals remains a persistent global challenge, primarily due to the limited understanding of sign language among the general population. This communication gap frequently leads to social exclusion, dependency on human interpreters, and significant difficulties in critical environments such as healthcare, education, and public services. To address this, the current study proposes an innovative Artificial Intelligence (AI)-based real-time sign language recognition and communication support system. Designed to enable inclusive, two-way interaction, the system utilizes advanced computer vision techniques to capture and preprocess hand gestures. Machine learning models, specifically Convolutional Neural Networks (CNNs), interpret these gestures to generate corresponding text and speech output. Conversely, speech input from hearing users is converted into text and animated visual sign representations to ensure seamless bidirectional communication. Furthermore, the application integrates an SOS emergency alert module and an interactive learning interface for basic sign language. By reducing the dependency on human translators and facilitating natural interaction, this proposed system aims to significantly enhance accessibility, user independence, and social inclusion for the hearing-impaired community. Keywords: Sign Language Recognition, Artificial Intelligence, Computer Vision, Bidirectional Communication, Convolutional Neural Networks (CNN), Accessibility, Deep Learning.

The move to standardize Indian Sign Language has created an opportunity for researchers to focus on solving local problems, to increase its reach. In this paper, a survey and assessment of the techniques applied to the recognition and conversion of Indian Sign Language are performed. An overview of the techniques used in sign language recognition for Indian Sign Language is provided to understand the status of research in this field. Following this, a comparison of techniques aimed at rendering a more detailed picture of the research results is presented. The challenges faced by researchers, the limitations of current techniques, and the need for improved research in this area are highlighted. With the intent of spurring more in-depth research, key areas within the approaches and techniques in need of improvement are summarized.

Sign language is a form of communication where people use bodily gestures, particularly those of hands and arms. This method of communication is put into motion when spoken communication is unattainable or disfavored. There are very few people who can translate sign language and readily understand them. It would be convenient for the hearing-impaired to have a platform where their sign language could be translated easily. Hence, through this study, with the help of artificial neural networks, we wish to compare how various widely implemented deep learning architectures respond to faultless translation of Indian sign language for the native audience. This research would streamline the development of software tools that can accurately predict or translate ISL. For the purpose of understanding the method of training the machine and exploring our model’s performance without any optimizations, a Convolutional Neural Network architecture was implemented. Over the course of our research, there have been several Pre-trained Transfer Learning Models implemented that have yielded promising results. The research aims to contrast how various convolutional neural networks perform while translating Indian Sign Actions on a custom dataset that factors in illumination, angles, and different backgrounds to provide a balanced and distinctive set of images. The goal of this study is to make clear comparisons between the various deep learning frameworks. Hence, a fresh Indian sign language dataset is introduced. Since every dataset in the field of deep learning has special properties that may be utilized for the betterment of the existing models, the development of a fresh dataset could be viewed as a development in the field. The optimum model for our task: classification of these gestures is found to be ResNet-50 (Accuracy = 98.25% and F1-score = 99.34%), and the least favorable was InceptionNet V3 (Accuracy = 66.75%, and F1-score = 70.89%).

The purpose of the study is to provide a literature review of the work done on sign language in Pakistan and the world. This study also provides a framework of an already developed prototype to translate Pakistani sign language into speech and text while using convolutional neural networking (CNN) to facilitate unimpaired teachers to bridge the communication gap among the deaf learners and unimpaired teachers. Due to the lack of sign language teaching, unimpaired teachers face difficulty in communicating with impaired learners. This communication gap can be filled with the help of this translation tool. Research indicates that a prototype has been evolved that can translate the English textual content into sign language and highlighted that there is a need for translation tool which can translate the signs into English text. The current study will provide an architectural framework of the Pakistani sign language to English text translation tool that how different components of technology like deep learning, convolutional neural networking, python, tensor Flow, and NumPy, InceptionV3 and transfer learning, eSpeak text to speech help in the development of a translation tool prototype. Keywords : Pakistan sign language (PSL), sign language (SL), translation, deaf, unimpaired, convolutional neural networking (CNN). DOI : 10.7176/JEP/10-15-18 Publication date :May 31 st 2019