Developing a Prototype to Translate Pakistan Sign Language into Text and Speech While Using Convolutional Neural Networking

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

Sign Language is argued as the first Language for hearing impaired people. It is the most physical and obvious way for the deaf and dumb people who have speech and hearing problems to convey themselves and general people. So, an interpreter is wanted whereas a general people needs to communicate with a deaf and dumb person. In respect to Bangladesh, 2.4 million people uses sign language but the works are extremely few for Bangladeshi Sign Language (BdSL). In this paper, we attempt to represent a BdSL recognition model which are constructed using of 50 sets of hand sign images. Bangla Sign alphabets are identified by resolving its shape and assimilating its structures that abstract each sign. In proposed model, we used multi-layered Convolutional Neural Network (CNN). CNNs are able to automate the method of structure formulation. Finally the model gained 92% accuracy on our dataset.

The purpose of the study is to provide a literature review of the work done on sign language (SL) around the world and in Pakistan and to develop a translation tool of speech and text to Pakistan Sign Language (PSL) with bilingual subtitles. Information and communication technology and tools development for teaching and learning purposes improve the learning process and facilitate both teachers and students. In Pakistan, unimpaired people face a lot of problems to communicate with deaf people due to the lack of SL understanding, learning resources, and interpreters. This problem is faced by the teachers who communicate with deaf students in the classrooms. The communication gap is filled with the development of a translation tool as Haseeb and Illyas concluded in their study that using this kind of tool, deaf people will have more opportunities to communicate with other members of society at every level. Different components of technology such as Python programming language, Natural Language Tool Kit, prerecorded PSL videos, Linux-based server, and databases are used to develop the prototype of PSL translation tool. This study provides a literature review to highlight the existing technological work done around the world and in Pakistan and also provides an architectural framework of the PSL translation tool which is developed by the researchers to facilitate the people who face difficulty to communicate with deaf people.

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

A hand sign for a disabled patient is a useful tool for human-to-human communication that can be used for a variety of purposes. They are frequently utilised by speech-impaired people around the world for communication since they are a natural form of engagement. In actuality, this group makes up around 1% of the Indian population. To enable easy communication among the signer and non-signer communities, an effective sign language recognition system (SLRS) can identify sign language motions. This research proposes a deep learning-based SLRS for classification that is computer vision-based. A sample of Auslan (Australian Sign Language (ASL)) signs is included in the dataset. Using top-notch position trackers, 27 examples of each of the 95 Auslan signs were recorded by a native signer. However, films captured Indian Sign Language (ISL) was filmed utilising numerous signs. Three individuals submitted 26 letters (A–Z) and 10 numbers (0–9). Few background removal techniques were performed on the samples and various skin color segmentation process was done. Third, a brand-new and reliable model for classifying ISL motions has been suggested utilising the mutation denoising autoencoder (MDAE). ISL and ASL datasets are publicly available and have been used to measure the effectiveness of the proposed system. In terms of measures like precision, recall, F1-Score, and accuracy, the proposed system is contrasted with well-known computer vision algorithms like the Support Vector Machine (SVM), Convolution Neural Network (CNN), Enhanced Convolution Neural Network (ECNN), and DAE. Python has been used to implement and test these categorisation techniques.

Sign language is the communication process of people with hearing impairments. For hearing-impaired communication in Bangladesh and parts of India, Bangla sign language (BSL) is the standard. While Bangla is one of the most widely spoken languages in the world, there is a scarcity of research in the field of BSL recognition. The few research works done so far focused on detecting BSL alphabets. To the best of our knowledge, no work on detecting BSL words has been conducted till now for the unavailability of BSL word dataset. In this research, a small static-gesture word dataset has been developed, and a deep learning-based method has been introduced that can detect BSL static-gesture words from images. The dataset, “BSLword” contains 30 static-gesture BSL words with 1200 images for training. The training is done using a multi-layered convolutional neural network with the Adam optimizer. OpenCV is used for image processing and TensorFlow is used to build the deep learning models. This system can recognize BSL static-gesture words with 92.50% accuracy on the word dataset.

Sign language is a form of communication where people use bodily gestures, particularly those of hands and arms. This method of communication is put into motion when spoken communication is unattainable or disfavored. There are very few people who can translate sign language and readily understand them. It would be convenient for the hearing-impaired to have a platform where their sign language could be translated easily. Hence, through this study, with the help of artificial neural networks, we wish to compare how various widely implemented deep learning architectures respond to faultless translation of Indian sign language for the native audience. This research would streamline the development of software tools that can accurately predict or translate ISL. For the purpose of understanding the method of training the machine and exploring our model’s performance without any optimizations, a Convolutional Neural Network architecture was implemented. Over the course of our research, there have been several Pre-trained Transfer Learning Models implemented that have yielded promising results. The research aims to contrast how various convolutional neural networks perform while translating Indian Sign Actions on a custom dataset that factors in illumination, angles, and different backgrounds to provide a balanced and distinctive set of images. The goal of this study is to make clear comparisons between the various deep learning frameworks. Hence, a fresh Indian sign language dataset is introduced. Since every dataset in the field of deep learning has special properties that may be utilized for the betterment of the existing models, the development of a fresh dataset could be viewed as a development in the field. The optimum model for our task: classification of these gestures is found to be ResNet-50 (Accuracy = 98.25% and F1-score = 99.34%), and the least favorable was InceptionNet V3 (Accuracy = 66.75%, and F1-score = 70.89%).

Sign language is an expressive visual language that uses the hand and face and is an important tool for deaf individuals to communicate. Standard means of recognition objectives cannot recognize complex gestures effectively, in turn limiting accessibility and transactions of communication effectively. This project seeks to close communication barriers by translating hand gestures into speech, text, and even serve as an output to virtual assistants. To date, prior research in signing recognition has focused primarily on identifiable hand gestures, often selecting a subset of Indian Sign Language (ISL) for classification. In this work, a deep learning method is proposed to perform static sign recognition through Convolutional Neural Networks (CNN) with the VGG19 architecture. The system is trained through a dataset of ISL gestures for predicting and classifying signs with high confidence. The predicted sign is converted to a text output, to then a speech output, stored as an audio file. In conclusion, the system provides accessibility through detection, enabling a user user experience for both signers/non-signers to communicate and add encryption with accessibility. The model produces a far more particular/accurate performance in sign language recognition, and follows through with promising assistive technology solutions

A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier

Deafness is one of the major health problems in Bangladesh, where almost 10% of the population falls under this category. The plight of deaf individuals to fit in with the mainstream population is significant. Unable to express themselves through sound, they have to rely on sign language to communicate.To create a more inclusive environment for this minority, we developed a model that recognizes Bengali characters and digits expressed using sign language in accordance with Bangla Sign Language (BdSL) guidelines. The model comprises two parts: a convolutional neural network that uses deep learning techniques, and an Extreme Gradient Boosting (XGBoost) classifier. Together, these components form the full architecture. The model’s execution was validated using the ‘Ishara-Lipi’ dataset, which is the first open-access digit and character dataset for BdSL. With the help of pre-processing techniques such as contrast-limited adaptive histogram equalization, using a more complex pre-trained CNN model like Inception-ResNet-v2, and optimizing the XGBoost model by using GridSearchCV, we achieved an accuracy of 86.67%, precision of 89%, recall and f1 score of 87%. Lastly, for digit recognition, we obtained an accuracy of 97.33%, precision of 98%, and recall and f1-score of 97%.

Sign language is used by people having speaking and hearing disabilities. It generally has a set of words, where each word is represented by one or more hand gestures in sequence and may contain facial expressions. In order to address the interpretation/translation from sign language to English Language, we present our sign recognition approach for Indian sign language which aims to provide a method for interpreting signs in Indian sign language to words in English language translation. The approach is to have a vision based system in which the sequence of images representing a word in ISL is translated to equivalent English word. The translation would be done by means of Deep learning algorithms namely convolutional neural nets and recurrent neural nets. The system will be analyzing sequence of images, hence CNNs will analyze each image and their sequence is analyzed by LSTM (which is an implementation of RNN). We divided dataset into training dataset and testing dataset, which obtained 73.60% accuracy. The image distributions are kept fairly different in training and testing datasets.

This paper aims to shed light on the common discourse markers in Sign Language, particularly Pakistani Sign Language. Pakistani Sign Language is overlooked and emerges as a non-standardized. These gestures/ discourse markers will be studied in comparison to American Sign Language (ASL). The main features will be analysed that call for a detailed analysis based on the use, the universality and application. This research aims to study if there is any influence of gender on these discourse markers. The Pakistan Sign Language Corpus will be collected and analysed with the assistance of Sign Language Interpreters. Furthermore, to see for any consistent language specific functions that these languages may share will also be accounted and noted. Keywords: Sign Language, Discourse markers, gender, American Sign Language, Pakistani Sign Language

The current research looks into using convolutional neural networks, also known as CNNs, to develop a hand sign understanding and conversion system. The study addresses the urgent need for a precise and effective way to close the communication gap that exists between the hearing and the deaf communities. The study examines how well CNNs can identify and translate hand sign gestures into text, giving sign language users a real-time means of communication. We use three steps in our methodology: gathering datasets, training models, and thorough assessment. The outcomes show encouraging hand sign recognition accuracy, creating new opportunities for inclusive communication. Sign language is one of the oldest and most natural forms of language for communication, but since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real-time method using neural networks for fingerspelling-based American sign language. In our method, the hand is first passed through a filter and after the filter is applied the hand is passed through a classifier which predicts the class of the hand gestures. Our method provides 95.7 % accuracy for the 26 letters of the alphabet. Keywords-Sign Language, ASL, Hearing disability, Convolutional Neural Network (CNN), Computer Vision, Machine Learning, Gesture recognition, Sign language recognition, Hue Saturation Value algorithm.

Hearing disabilities affect people all over the world. Nearly 360 million people have a disabling hearing loss. In the recent years, there has been a rapid increase in the number of deaf and dumb victims due to birth defects, accidents and oral diseases. Since deaf and dumb people cannot communicate with a hearing person, they have to depend on some sort of visual communication. Sign Language is not understood by the majority of hearing people. The objective of our project is to form a bridge between the hearing-impaired community and the hearing community; to further commence a two-way communication. We propose a two-way sign language translator that converts Voice to sign language and vice versa in real time. We process the video frame by frame using the OpenCV Library in Python. Furthermore, we use background subtraction method called Gaussian Mixture-based Background/Foreground Segmentation Algorithm on each frame to subtract the background. Contours of this processed image are then passed into a Deep Neural Net (DNN) to classify the frame to the corresponding written language equivalent words. For conversion from sign language to voice, we use basic Natural Language Processing and gtts to accurately conserve the grammar in the sign language. The enrolment rate and literacy among the hearing-impaired children is far below the average for the population at large. This technology will then help the normal schools better integrate the hearing-impaired community thus making education more accessible and cheaper for them. Speech Emotion Recognition (SER) is a vital aspect of human-computer interaction and emotional intelligence applications. This SER that employs Mel- frequency cepstral coefficients (MFCC) as the basis for feature extraction, followed by a Convolutional Neural Network (CNN) with a customized architecture for emotion classification. The methodology 2 involves transforming audio samples into MFCC images, enabling the application of CNNs typically used in computer vision tasks. This fusion of traditional audio analysis with deep learning techniques harnesses the power of MFCC's spectral representation and CNN's spatial pattern recognition. By modifying the CNN architecture, the model can better discern emotional cues in the MFCC images, enhancing SER performance. This strategy offers SER a viable path forward, with the ability to identify complex emotional expressions. In order to increase the accuracy of emotion recognition, it tackles the requirement for feature-rich representations and sophisticated modelling techniques. KEYWORDS: OpenCV, DNN, SER, CNN, MFCC