Static-gesture word recognition in Bangla sign language using convolutional neural network

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

Sign Language is argued as the first Language for hearing impaired people. It is the most physical and obvious way for the deaf and dumb people who have speech and hearing problems to convey themselves and general people. So, an interpreter is wanted whereas a general people needs to communicate with a deaf and dumb person. In respect to Bangladesh, 2.4 million people uses sign language but the works are extremely few for Bangladeshi Sign Language (BdSL). In this paper, we attempt to represent a BdSL recognition model which are constructed using of 50 sets of hand sign images. Bangla Sign alphabets are identified by resolving its shape and assimilating its structures that abstract each sign. In proposed model, we used multi-layered Convolutional Neural Network (CNN). CNNs are able to automate the method of structure formulation. Finally the model gained 92% accuracy on our dataset.

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

Abstract—People with hearing or speech impairments often face communication barriers in their day-to-day lives.The need for accessible communication tools for individuals with hearing or speech impairments continues to grow as digital communication and technology-driven interaction become central to everyday life. Indian Sign Language (ISL), being a primary mode of com- munication for many, requires automated systems that can inter- pret gestures accurately and efficiently. Motivated by this need, our project presents an AI-driven ISL Recognition and Transla- tion System that integrates Computer Vision, Deep Learning, and Natural Language Processing (NLP) within a unified web-based platform. The system employs MediaPipe to extract reliable hand landmarks and uses a Convolutional Neural Network (CNN) to learn and classify spatial gesture patterns from webcam streams or uploaded videos. For text-to-sign translation, the framework applies NLP techniques—including tokenization, lemmatization, and tense detection—to generate meaningful ISL animations. When a direct gesture animation is not available, the system automatically breaks words into alphabet-level signs to ensure complete translation. The preprocessing pipeline includes region- of-interest extraction and normalization to maintain consistent input quality. Implemented using a Django web framework with an intuitive user interface, the system supports educational, assistive, and accessibility-focused applications. Overall, this project delivers a reliable, user-centered solution that enhances communication and promotes digital inclusivity for the hearing and speech-impaired community Keywords: Indian Sign Language (ISL), Gesture Recognition, Convolutional Neural Network (CNN), MediaPipe, Natural Lan- guage Processing (NLP), Assistive Technology.

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

A hand sign for a disabled patient is a useful tool for human-to-human communication that can be used for a variety of purposes. They are frequently utilised by speech-impaired people around the world for communication since they are a natural form of engagement. In actuality, this group makes up around 1% of the Indian population. To enable easy communication among the signer and non-signer communities, an effective sign language recognition system (SLRS) can identify sign language motions. This research proposes a deep learning-based SLRS for classification that is computer vision-based. A sample of Auslan (Australian Sign Language (ASL)) signs is included in the dataset. Using top-notch position trackers, 27 examples of each of the 95 Auslan signs were recorded by a native signer. However, films captured Indian Sign Language (ISL) was filmed utilising numerous signs. Three individuals submitted 26 letters (A–Z) and 10 numbers (0–9). Few background removal techniques were performed on the samples and various skin color segmentation process was done. Third, a brand-new and reliable model for classifying ISL motions has been suggested utilising the mutation denoising autoencoder (MDAE). ISL and ASL datasets are publicly available and have been used to measure the effectiveness of the proposed system. In terms of measures like precision, recall, F1-Score, and accuracy, the proposed system is contrasted with well-known computer vision algorithms like the Support Vector Machine (SVM), Convolution Neural Network (CNN), Enhanced Convolution Neural Network (ECNN), and DAE. Python has been used to implement and test these categorisation techniques.

The purpose of the study is to provide a literature review of the work done on sign language in Pakistan and the world. This study also provides a framework of an already developed prototype to translate Pakistani sign language into speech and text while using convolutional neural networking (CNN) to facilitate unimpaired teachers to bridge the communication gap among the deaf learners and unimpaired teachers. Due to the lack of sign language teaching, unimpaired teachers face difficulty in communicating with impaired learners. This communication gap can be filled with the help of this translation tool. Research indicates that a prototype has been evolved that can translate the English textual content into sign language and highlighted that there is a need for translation tool which can translate the signs into English text. The current study will provide an architectural framework of the Pakistani sign language to English text translation tool that how different components of technology like deep learning, convolutional neural networking, python, tensor Flow, and NumPy, InceptionV3 and transfer learning, eSpeak text to speech help in the development of a translation tool prototype. Keywords : Pakistan sign language (PSL), sign language (SL), translation, deaf, unimpaired, convolutional neural networking (CNN). DOI : 10.7176/JEP/10-15-18 Publication date :May 31 st 2019

Barriers in expressive and receptive communication for individuals with hearing and speech impairments remain a significant challenge to achieving equitable social interaction and digital accessibility. These limitations restrict their participation in everyday conversations, education, and professional environments, emphasizing the urgent need for intelligent assistive communication technologies. This research introduces an AI-driven real-time Indian Sign Language (ISL) recognition framework that integrates advanced computer vision and deep learning techniques to translate hand gestures into textual outputs. The primary objective of this study is to design a lightweight, accurate, and real-time sign language translator suitable for Divyangjan, ensuring inclusivity in educational and social interactions. The proposed research employs Mediapipe-based hand landmark extraction to detect 21 key points in each gesture frame, followed by preprocessing and normalisation to create robust spatial representations. A hybrid Convolutional Neural Network processes these landmark vectors through a Bidirectional Long Short-Term Memory (CNN–BiLSTM) model that captures both spatial and temporal dependencies in gesture motion, allowing it to recognise static and dynamic gestures such as J and Z. The system was trained on a self-collected dataset of over 26,000 gesture images covering all 26 ISL alphabets. Experimental analysis demonstrates that the proposed model achieved 97.8 percent accuracy, 97.4 percent precision, 97.2 percent recall, 97.3 percent F1-score, and a validation loss of only 0.08, outperforming traditional classifiers such as Random Forest and SVM by a significant margin. The trained model performs robustly under varying lighting, background, and hand orientation conditions, ensuring high reliability for real-world deployment. The novelty of this study lies in the fusion of Mediapipe landmark extraction with a temporal deep learning framework for continuous ISL gesture translation within an interactive CustomTkinter GUI. This human-centric, computationally efficient design enables accessible, real-time communication for the deaf and hard-of-hearing community, contributing to socially assistive AI systems that promote digital inclusivity and empowerment for Divyangjan.

A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier

This paper aims to shed light on the common discourse markers in Sign Language, particularly Pakistani Sign Language. Pakistani Sign Language is overlooked and emerges as a non-standardized. These gestures/ discourse markers will be studied in comparison to American Sign Language (ASL). The main features will be analysed that call for a detailed analysis based on the use, the universality and application. This research aims to study if there is any influence of gender on these discourse markers. The Pakistan Sign Language Corpus will be collected and analysed with the assistance of Sign Language Interpreters. Furthermore, to see for any consistent language specific functions that these languages may share will also be accounted and noted. Keywords: Sign Language, Discourse markers, gender, American Sign Language, Pakistani Sign Language

The Text to Indian Sign Language (ISL) Converter is a web-based application designed to facilitate communication between hearing and deaf individuals by translating English text into corresponding ISL gestures. This system utilizes pre-recorded MP4 video animations of ISL alphabets, numbers, and common words. The tool processes user input, either through text or broken-down letters, and sequentially plays the corresponding ISL videos. A fallback mechanism is implemented to handle words not available in the dataset by spelling them out using ISL alphabet videos. The system was tested for robustness and usability, with results indicating that it successfully handles a variety of text inputs, maintains smooth playback continuity, and delivers an engaging, user-friendly interface. While the system does not yet integrate linguistic adjustments for ISL syntax or grammar, it provides a simple and effective solution for digital inclusion. Future work will involve expanding the dataset, incorporating natural language processing (NLP) for sentence restructuring, and collaborating with ISL experts to ensure accuracy. The project highlights the potential of browser-based tools in bridging communication gaps for the hearing-impaired community, with significant room for future enhancements

Sign language is a form of communication where people use bodily gestures, particularly those of hands and arms. This method of communication is put into motion when spoken communication is unattainable or disfavored. There are very few people who can translate sign language and readily understand them. It would be convenient for the hearing-impaired to have a platform where their sign language could be translated easily. Hence, through this study, with the help of artificial neural networks, we wish to compare how various widely implemented deep learning architectures respond to faultless translation of Indian sign language for the native audience. This research would streamline the development of software tools that can accurately predict or translate ISL. For the purpose of understanding the method of training the machine and exploring our model’s performance without any optimizations, a Convolutional Neural Network architecture was implemented. Over the course of our research, there have been several Pre-trained Transfer Learning Models implemented that have yielded promising results. The research aims to contrast how various convolutional neural networks perform while translating Indian Sign Actions on a custom dataset that factors in illumination, angles, and different backgrounds to provide a balanced and distinctive set of images. The goal of this study is to make clear comparisons between the various deep learning frameworks. Hence, a fresh Indian sign language dataset is introduced. Since every dataset in the field of deep learning has special properties that may be utilized for the betterment of the existing models, the development of a fresh dataset could be viewed as a development in the field. The optimum model for our task: classification of these gestures is found to be ResNet-50 (Accuracy = 98.25% and F1-score = 99.34%), and the least favorable was InceptionNet V3 (Accuracy = 66.75%, and F1-score = 70.89%).

Objective: Development of Indian Sign Language video dictionary is essential in the today’s world of computerization. Though a lot of human video sign language dictionaries are available, we aim to develop the Indian Sign Language dictionary using synthetic animation which uses the computer generated cartoon rather than real human. Methods/Statistical Analysis: Sign Language cannot be spoken or written unlike other languages like English, Punjabi, Hindi, etc. The most commonly used words in Indian Sign Language are categorized and then these words are converted into the sign language writing notation (HamNoSys - Hamburg Notation System). This HamNoSys notation is then converted into SiGML (Signing Gesture Markup Language) using which the synthetic animation (using a computer generated cartoon) of the sign is generated. Findings: The synthetic animations are better as compared to human videos in terms of memory consumption, standardization, and flexibility. Synthetic animations can be modified as per the requirement whereas the human videos cannot be modified. The only drawback that seem is, these synthetic animations may lack the natural non-manual component of sign. Applications/Improvements: The synthetic dictionary created in this work can be used for translation system in which spoken or written sentence can be converted into the sign language animation. The dictionary created can be used to education to hard of hearing people. Display boards can be created for displaying the important messages in Indian sign language at the public gathering.

Sign Language is the elementary communication media for Deaf & Mute (D&M) people. On the other hand, it seems too tenacious for the general people to understand this language. In order to tear out this communication barrier, a real-time automated translator is essential. Through this research, a computer vision-based approach has been developed for the recognition of Bangla Sign Language (BdSL) characters. In this work, a deep learning-based recognition model has been developed. Adaptive thresholding has been integrated with 2D Convolutional Neural Network (CNN) to construct this model. Proposed model has been trained to build this real-time automated translator through our own created dataset (dataset containing 3600 different images for 36 distinct characters). The proposed model has been trained and tested with 2880 (80%) training images and 720 (20%) testing images respectively. Thirty-six unique characters of Bangla Sign Language can be recognized through this model with significant accuracy. The model delivers validation accuracy of 99.72% and validation loss of 0.73%. A significant result has been achieved for the recognition and translation of Bangla Sign Language characters with this dataset over other existing Bangla Sign Language Recognition model.

Explainable federated learning for privacy-preserving bangla sign language detection