Predicting American Sign Language from Hand Gestures Using Image Processing and Deep Learning

With its development, artificial intelligence has formed the basis for many studies aimed at facilitating people's lives. More successful results have been tried to be obtained with the increasing data and developing equipment in these studies. It is seen that these developments in artificial intelligence are reflected in the studies related to sign language conversion. In this study, a data set belonging to the letters in the Turkish Sign Language Alphabet was created, and the classification process was carried out with both the deep learning model we created and VGG16, Inceptionv3, Resnet, and Mobilnet models, which are frequently used in image classification. In addition, an open-source data set containing the letters in the American Sign Language Alphabet was organized similar to the data set containing the letters in the Turkish Sign Language Alphabet we created, and Deep Learning models were used to classify the letters in the American Sign Language Alphabet by using this data set. Performance evaluations of the classifications made by Deep Learning Models using both data sets were made. With this study, the results obtained from training Deep Learning methods with different data sets were compared. In addition, it is thought that the study will be useful in determining both the data set and the deep learning method to be used for the studies on the recognition of Sign Language Letters.

The project "Transforming Expression into Language: American Sign Language Recognition using Image Processing" intends to employ image processing techniques to identify gestures in American Sign Language (ASL). The goal of this project is to develop an efficient system for translating ASL signs using visual cues generated from hand motions and gestures. The program intends to use image processing technology to bridge the communication gap between ASL users and nonusers. The use of image processing for ASL identification may improve communication in a range of situations while also increasing accessibility for the deaf community. The paper provides a realistic approach to real-time ASL motion interpretation using image processing techniques, thereby advancing the field of sign language recognition technology. KEYWORDS ASL Recognition, Image Processing Techniques, Convolutional Neural Networks (CNNs), Feature Extraction in Image Processing, Transforming Expressions into Text

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

The Electromyography (EMG) signal is a nonstationary bio-signal based on the measurement of the electrical activity of the muscles. EMG based recognition systems play an important role in many fields such as diagnosis of neuromuscular diseases, human-computer interactions, console games, sign language detection, virtual reality applications, and amputee device controls. In this study, a novel approach based on deep learning has been proposed to improve the accuracy rate in the prediction of hand movements. Firstly, 4-channel surface EMG (sEMG) signals have been measured while simulating 7 different hand gestures (Extension, Flexion, Open Hand, Punch, Radial Deviation, Rest, and Ulnar Deviation) from 30 participants. The obtained sEMG signals have been segmented into sections where each movement was found. Then, spectrogram images of the segmented sEMG signals have been created by means of ShortTime Fourier Transform (STFT). The created colored spectrogram images have trained with 50-layer Convolutional Neural Network (CNN) based on Residual Networks (ResNet) architecture. Owing to the proposed method, test accuracy of 99.59% and F1 Score of 99.57% have achieved for 7 different hand gesture classifications.

People around the world with speech and hearing impairment use a media of communication known as ‘Sign language’. In recent times, Sign language is omnipresent. However, there exists a challenge for people who do not know sign language, to communicate with people who can communicate exclusively using sign language. This gap can be bridged by using technologies of recent times to recognize gestures and design intuitive systems with deep learning. The aim of this paper is to recognise American Sign Language gestures dynamically and create an intuitive system which provides sign language translation to text and speech of various languages. The system uses Convolutional neural network, natural language processing, language translation and text-to-speech algorithms. It is capable of recognizing hand gestures dynamically and predicting the corresponding letters to form a desired sentence accurately.

Convolutional neural networks (CNN) have provided great advances for the task of sign language recognition (SLR). However, recurrent neural networks (RNN) in the form of long–short-term memory (LSTM) have become a means for providing solutions to problems involving sequential data. This research proposes the development of a sign language translation system that converts Panamanian Sign Language (PSL) signs into text in Spanish using an LSTM model that, among many things, makes it possible to work with non-static signs (as sequential data). The deep learning model presented focuses on action detection, in this case, the execution of the signs. This involves processing in a precise manner the frames in which a sign language gesture is made. The proposal is a holistic solution that considers, in addition to the seeking of the hands of the speaker, the face and pose determinants. These were added due to the fact that when communicating through sign languages, other visual characteristics matter beyond hand gestures. For the training of this system, a data set of 330 videos (of 30 frames each) for five possible classes (different signs considered) was created. The model was tested having an accuracy of 98.8%, making this a valuable base system for effective communication between PSL users and Spanish speakers. In conclusion, this work provides an improvement of the state of the art for PSL–Spanish translation by using the possibilities of translatable signs via deep learning.

Explainable federated learning for privacy-preserving bangla sign language detection

The predominant means of communication is speech; however, there are persons whose speaking or hearing abilities are impaired. Communication presents a significant barrier for persons with such disabilities. The primary objective of this work is to develop an efficient solution with machine learning for the speech and hearing impairment community with the help of AI. This study helps in the development of future technologies in the enhancement of communication accessibility. In the proposed work, a machine learning approach for the American Sign Language alphabet using image detection and processing techniques are utilized. The experiment was conducted using an American Sign Language recognition dataset obtained from Kaggle, which provides images of hand Gestures of the alphabet. The dataset was divided into training and testing subsets in the ratio of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$80: 20$</tex>, where 80 % of the data was used in training the model and the remaining 20 % was designated for testing the model. The dataset was normalized using denoising filters and proceeded with Canny edge detection to highlight sharp edge features. From the edge detected image, Histogram of Oriented Gradient features were extracted to detect the letter and shape information. These extracted features are then fed into the classifiers: Support Vector Machine, Random Forest, and KNearest Neighbors. The study was analyzed by experimental results, which show that the Support Vector Machine classifier achieved the highest accuracy of 93.33 %, closely followed by Random Forest and K-Nearest Neighbors. The analysis confirms that applying edge-based preprocessing along with the Histogram of Oriented Gradients feature extraction enables an effective classification of American Sign Language alphabets. This Comparative study provides a solid foundation for developing low-complexity, high-accuracy sign language recognition systems.

Dore Idioma, also known as a hand in Arabic and sign language in Albanian, will be a futuristic website for the deaf-dumb or deaf-mute people as it provides a solution for people who face difficulty speaking and hearing. According to various surveys and stats from different deaf-mute organizations, 63% [21] of the world's population is deaf-mute. To help such people, we devised the idea of Dore Idioma. The fundamental functionality of Dore Idioma is to convert sign language (hand gesture). As communication takes place between ordinary and deaf-mute people, therefore, it will be easy for everyone to communicate through Dore Idioma. It is creating a user-friendly application that helps the user to have an approach to the translator very quickly. We will use image labeling so that input is given via the user; the information is then compared with the pre-trained detection model, after which the detected sign is cross-referenced with the database, and the final output will be generated. Various other models/software has been created. Still, they were not efficient and user-friendly enough and implementing. Dore Idioma might help in practical and better communication amongst ordinary and deaf-mute people. The pre-trained model used to create Dore Idioma is MobileNet version 2 (Convolutional Neural Network). Our current model is working on an accuracy of 99% and a total loss of 0.38%.

Hand gestures play a vital role in sign language, serving as a primary means of non-verbal communication for individuals who are deaf or mute, enabling them to express themselves and interact with others. Although numerous sign language recognition systems have been developed, most are either costly or lack adaptability for real-world use. Recent advances in deep learning, particularly Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), have significantly improved gesture recognition by enabling systems to learn complex patterns from large datasets of images or video sequences. This paper introduces a novel algorithm called Sign Language Accessibility Enhancement Using Ensemble of Deep Learning Techniques for Hand Gesture Recognition in Deaf and Dumb Communication (SLAEDL-HGRDDC). The model combines multiple components into a robust framework to enhance recognition accuracy and system scalability. Initially, Gaussian Filtering (GF) is used to reduce noise and improve image quality. Then, Inception-ResNet-V2 is adopted to extract intricate spatial features from gesture images. The recognition process utilizes an ensemble of three deep learning classifiers – Long Short-Term Memory (LSTM), Gated Recurrent Units (GRUs), and Temporal Convolutional Networks (TCN). Finally, the Modified Al-Biruni Earth Radius Optimization (MBER) algorithm is employed for optimal hyperparameter tuning. Extensive simulations demonstrate that SLAEDL-HGRDDC outperforms traditional systems in various performance metrics.

Hand gestures are a form of non-verbal communication. Apart from traditional input devices, hand gestures are used for interaction with computers too. Communication via hand gestures finds many applications in the real world. Different people have hands with different shapes and orientations, which is termed as nonlinearity. The nonlinearity affects the performance of hand gesture models. A convolutional neural network (CNN) is an approach of neural networks, specifically known as deep learning. CNN is used to recognize and classify images. Sometimes, CNN could not correctly understand the hand gesture due to nonlinearity. Data augmentation helps CNN to understand the nonlinearity and complexity of images better. Data augmentation generates enormous data from lesser data, thus increasing the data adversity. Data augmentation uses various operations like zooming, rotating, shifting, shearing, and scaling to generate more data from the existing data. This article executes a CNN model using augmented data for recognition of static hand gestures. The dataset consists of 10 different hand gestures. The experimented CNN model has been trained using 10000 images and tested using 1000 images. The changes in the output of CNN with and without data augmentation have been highlighted. The CNN model employing data augmentation achieved an accuracy of 98.10%, whereas the CNN model excluding the data augmentation process attained an accuracy of 94.90% only.

This project introduces a real-time sign language detection system powered by Convolutional Neural Networks (CNNs), designed to aid individuals with hearing impairments in communication. Leveraging OpenCV for video capture and hand detection, along with custom modules for hand tracking and image classification, the system seamlessly integrates deep learning methodologies. A pre-trained CNN model, trained on a comprehensive dataset of sign language gestures, forms the core of the system, ensuring accurate classification in real-time. By capturing video frames from a webcam, detecting hands, and processing them through the CNN model, the system provides immediate feedback by overlaying predicted gestures onto the video stream. Through its effective implementation, this project underscores the potential of CNNs in facilitating accessibility and inclusivity, while also paving the way for future enhancements and applications in assistive technology and human-computer interaction. Top of Form Keywords— Real-time, Sign Language Detection, Convolutional Neural Networks (CNNs), Communication, Hearing Impairments, Deep Learning.

The purpose of the study is to provide a literature review of the work done on sign language in Pakistan and the world. This study also provides a framework of an already developed prototype to translate Pakistani sign language into speech and text while using convolutional neural networking (CNN) to facilitate unimpaired teachers to bridge the communication gap among the deaf learners and unimpaired teachers. Due to the lack of sign language teaching, unimpaired teachers face difficulty in communicating with impaired learners. This communication gap can be filled with the help of this translation tool. Research indicates that a prototype has been evolved that can translate the English textual content into sign language and highlighted that there is a need for translation tool which can translate the signs into English text. The current study will provide an architectural framework of the Pakistani sign language to English text translation tool that how different components of technology like deep learning, convolutional neural networking, python, tensor Flow, and NumPy, InceptionV3 and transfer learning, eSpeak text to speech help in the development of a translation tool prototype. Keywords : Pakistan sign language (PSL), sign language (SL), translation, deaf, unimpaired, convolutional neural networking (CNN). DOI : 10.7176/JEP/10-15-18 Publication date :May 31 st 2019

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

Historically, individuals with hearing impairments have faced neglect, lacking the necessary tools to facilitate effective communication. However, advancements in modern technology have paved the way for the development of various tools and software aimed at improving the quality of life for hearing-disabled individuals. This research paper presents a comprehensive study employing five distinct deep learning models to recognize hand gestures for the American Sign Language (ASL) alphabet. The primary objective of this study was to leverage contemporary technology to bridge the communication gap between hearing-impaired individuals and individuals with no hearing impairment. The models utilized in this research include AlexNet, ConvNeXt, EfficientNet, ResNet-50, and VisionTransformer were trained and tested using an extensive dataset comprising over 87,000 images of the ASL alphabet hand gestures. Numerous experiments were conducted, involving modifications to the architectural design parameters of the models to obtain maximum recognition accuracy. The experimental results of our study revealed that ResNet-50 achieved an exceptional accuracy rate of 99.98%, the highest among all models. EfficientNet attained an accuracy rate of 99.95%, ConvNeXt achieved 99.51% accuracy, AlexNet attained 99.50% accuracy, while VisionTransformer yielded the lowest accuracy of 88.59%.