Indian Sign Language Recognition using Deep Learning

Communication is essential to humans because it allows the dissemination of knowledge and the formation of interpersonal connections. We communicate through speaking, facial expressions, hand gestures, reading, writing, and sketching, among other things. However, speaking is the most often utilized means of communication. People having speech and hearing disabilities can only communicate using hand gestures, making them extremely reliant on nonverbal modes of communication. Hearing-impaired persons can communicate via sign language. Globally, around 1 percent(5 million) of the Indian population falls into this group. ISL is a complete language with its own vocabulary, semantics, lexicon, and a variety of other distinctive linguistic features. In our work, we present the methods for Indian sign language recognition at the character and word levels. The Bag of Visual Words(BoVW) technique recognizes ISL at character level(A-Z, 0-9) with an accuracy of 99 percent. Indian Lexicon Sign Language Dataset - INCLUDE-50 dataset is used for word-level sign language recognition. Inception model, a deep Convolutional Neural Network(CNN) is used to train the spatial features and LSTM RNN(Recurrent Neural Network) is used to train the temporal features of the video. Using CNN predictions as input to RNN, we achieved an accuracy of 86.7 %. In order to optimize the training process, only 60 % of the dataset is trained using the Meta-Learning model along with LSTM RNN and obtained an accuracy of 84.4 %, thus reducing the training time by 70 % and reaching nearly as close accuracy as the previous pre-trained model.

A major challenge faced by our society is the difficulty individuals with disabilities encounter in expressing their emotions to others. People with disabilities often depend on sign (gesture) languages for communication. This project aims to develop a model capable of recognizing and translating sign language alphabets from hand gestures into text and audio. The primary objective is to enhance communication between individuals with hearing impairments and the broader society. We assessed our model's performance using a publicly accessible Indian Sign Language (ISL) dataset. For image classification, we employed Convolutional Neural Networks (CNNs) with the Inception V3 architecture. Hand gestures are captured through webcams, and our model identifies the corresponding alphabet for each gesture. This project seeks to overcome current challenges in Sign Language Recognition and aims to enhance its effectiveness and efficiency.

Sign Language is one of the prime applications of facial expression recognition system. It is an easy task for a human being to identify the expression of a person in a particular image sequence but not many efficient fully automated systems are available to perform this task. Lot of work has been done to automate the recognition system for many sign languages like Turkish Sign Language (TSL) [2], American Sign Language (ASL) [3], Chinese Sign Language (CSL) [4] etc. But rare effort has been done for any of the Indian Sign Language (ISL). Therefore, in this paper, a large video data set of Indian Sign Language (ISL) is prepared consisting of some of the expressions. Further, an automatic facial expression recognition system is developed to identify the expressions performed in these videos. Also same algorithm is run on the Turkish sign language data set provided by Aran et al. [2] and a comparison is provided between these two dataset.

Sign language is an expressive visual language that uses the hand and face and is an important tool for deaf individuals to communicate. Standard means of recognition objectives cannot recognize complex gestures effectively, in turn limiting accessibility and transactions of communication effectively. This project seeks to close communication barriers by translating hand gestures into speech, text, and even serve as an output to virtual assistants. To date, prior research in signing recognition has focused primarily on identifiable hand gestures, often selecting a subset of Indian Sign Language (ISL) for classification. In this work, a deep learning method is proposed to perform static sign recognition through Convolutional Neural Networks (CNN) with the VGG19 architecture. The system is trained through a dataset of ISL gestures for predicting and classifying signs with high confidence. The predicted sign is converted to a text output, to then a speech output, stored as an audio file. In conclusion, the system provides accessibility through detection, enabling a user user experience for both signers/non-signers to communicate and add encryption with accessibility. The model produces a far more particular/accurate performance in sign language recognition, and follows through with promising assistive technology solutions

The rapid spread of the novel corona virus disease (COVID-19) has disrupted the traditional clinical services all over the world. Hospitals and healthcare centers have taken extreme care to minimize the risk of exposure to the virus by restricting the visitors and relatives of the patients. The dramatic changes happened in the healthcare norms have made it hard for the deaf patients to communicate and receive appropriate care. This paper reports a work on automatic sign language recognition that can mitigate the communication barrier between the deaf patients and the healthcare workers in India. Since hand gestures are the most expressive components of a sign language vocabulary, a novel dataset of dynamic hand gestures for the Indian sign language (ISL) words commonly used for emergency communication by deaf COVID-19 positive patients is proposed. A hybrid model of deep convolutional long short-term memory network has been utilized for the recognition of the proposed hand gestures and achieved an average accuracy of 83.36%. The model performance has been further validated on an alternative ISL dataset as well as a benchmarking hand gesture dataset and obtained average accuracies of 97% and 99.34pm 0.66%, respectively.

As road traffic sign recognition is a crucial component for automatic driver assistance systems, it is a key problem in computer vision as well. Therefore, in this paper, we study on the problem of road traffic sign recognition utilising the computer vision technology. The main innovation of this paper is to propose an improved convolutional neural network, and then use it to tackle the road traffic sign recognition problem. Convolutional neural network can learn features from training data set, and a convolutional network contains alternating layers of convolution and pooling. Particularly, RGB traffic images are transformed to grey scale images, and then grey scale images are input to the improved convolutional neural network. Furthermore, the fixed layers are utilised to discover region of interests, and the learnable layers are used to extract features. In general, output information of the proposed two learnable layers are input to the classifier separately, and parameters of learnable layers and the classifier are trained at the same time. Finally, GTSDB data set is chosen to make performance evaluation, among which 600 images and 300 images are regarded as training and testing data set respectively. Experimental results demonstrate that the improved CNN-based traffic sign recognition performs better than the traditional CNN.

Objective: Development of Indian Sign Language video dictionary is essential in the today’s world of computerization. Though a lot of human video sign language dictionaries are available, we aim to develop the Indian Sign Language dictionary using synthetic animation which uses the computer generated cartoon rather than real human. Methods/Statistical Analysis: Sign Language cannot be spoken or written unlike other languages like English, Punjabi, Hindi, etc. The most commonly used words in Indian Sign Language are categorized and then these words are converted into the sign language writing notation (HamNoSys - Hamburg Notation System). This HamNoSys notation is then converted into SiGML (Signing Gesture Markup Language) using which the synthetic animation (using a computer generated cartoon) of the sign is generated. Findings: The synthetic animations are better as compared to human videos in terms of memory consumption, standardization, and flexibility. Synthetic animations can be modified as per the requirement whereas the human videos cannot be modified. The only drawback that seem is, these synthetic animations may lack the natural non-manual component of sign. Applications/Improvements: The synthetic dictionary created in this work can be used for translation system in which spoken or written sentence can be converted into the sign language animation. The dictionary created can be used to education to hard of hearing people. Display boards can be created for displaying the important messages in Indian sign language at the public gathering.

Retrospective cohort study. We aimed to develop and validate a convolutional neural network (CNN) model to distinguish between cervical ossification of posterior longitudinal ligament (OPLL) and multilevel degenerative spinal stenosis using Magnetic Resonance Imaging (MRI) and to compare the diagnostic ability with spine surgeons. Some artificial intelligence models have been applied in spinal image analysis and many of promising results were obtained; however, there was still no study attempted to develop a deep learning model in detecting cervical OPLL using MRI images. In this retrospective study, 272 cervical OPLL and 412 degenerative patients underwent surgical treatment were enrolled and divided into the training (513 cases) and test dataset (171 cases). CNN models applying ResNet architecture with 34, 50, and 101 layers of residual blocks were constructed and trained with the sagittal MRI images from the training dataset. To evaluate the performance of CNN, the receiver operating characteristic curves of 3 ResNet models were plotted and the area under the curve were calculated on the test dataset. The accuracy, sensitivity, and specificity of the diagnosis by the CNN were calculated and compared with 3 senior spine surgeons. The diagnostic accuracies of our ResNet34, ResNet50, and ResNet101 models were 92.98%, 95.32%, and 97.66%, respectively; the area under the curve of receiver operating characteristic curves of these models were 0.914, 0.942, and 0.971, respectively. The accuracies and specificities of ResNet50 and ResNet101 models were significantly higher than all spine surgeons; for the sensitivity, ResNet101 model achieved better values than that of the 2 surgeons. The performance of our ResNet model in differentiating cervical OPLL from degenerative spinal stenosis using MRI is promising, better results were achieved with more layers of residual blocks applied.

Sign language is a form of communication where people use bodily gestures, particularly those of hands and arms. This method of communication is put into motion when spoken communication is unattainable or disfavored. There are very few people who can translate sign language and readily understand them. It would be convenient for the hearing-impaired to have a platform where their sign language could be translated easily. Hence, through this study, with the help of artificial neural networks, we wish to compare how various widely implemented deep learning architectures respond to faultless translation of Indian sign language for the native audience. This research would streamline the development of software tools that can accurately predict or translate ISL. For the purpose of understanding the method of training the machine and exploring our model’s performance without any optimizations, a Convolutional Neural Network architecture was implemented. Over the course of our research, there have been several Pre-trained Transfer Learning Models implemented that have yielded promising results. The research aims to contrast how various convolutional neural networks perform while translating Indian Sign Actions on a custom dataset that factors in illumination, angles, and different backgrounds to provide a balanced and distinctive set of images. The goal of this study is to make clear comparisons between the various deep learning frameworks. Hence, a fresh Indian sign language dataset is introduced. Since every dataset in the field of deep learning has special properties that may be utilized for the betterment of the existing models, the development of a fresh dataset could be viewed as a development in the field. The optimum model for our task: classification of these gestures is found to be ResNet-50 (Accuracy = 98.25% and F1-score = 99.34%), and the least favorable was InceptionNet V3 (Accuracy = 66.75%, and F1-score = 70.89%).

A hand sign for a disabled patient is a useful tool for human-to-human communication that can be used for a variety of purposes. They are frequently utilised by speech-impaired people around the world for communication since they are a natural form of engagement. In actuality, this group makes up around 1% of the Indian population. To enable easy communication among the signer and non-signer communities, an effective sign language recognition system (SLRS) can identify sign language motions. This research proposes a deep learning-based SLRS for classification that is computer vision-based. A sample of Auslan (Australian Sign Language (ASL)) signs is included in the dataset. Using top-notch position trackers, 27 examples of each of the 95 Auslan signs were recorded by a native signer. However, films captured Indian Sign Language (ISL) was filmed utilising numerous signs. Three individuals submitted 26 letters (A–Z) and 10 numbers (0–9). Few background removal techniques were performed on the samples and various skin color segmentation process was done. Third, a brand-new and reliable model for classifying ISL motions has been suggested utilising the mutation denoising autoencoder (MDAE). ISL and ASL datasets are publicly available and have been used to measure the effectiveness of the proposed system. In terms of measures like precision, recall, F1-Score, and accuracy, the proposed system is contrasted with well-known computer vision algorithms like the Support Vector Machine (SVM), Convolution Neural Network (CNN), Enhanced Convolution Neural Network (ECNN), and DAE. Python has been used to implement and test these categorisation techniques.

Identifying organs/tissue and pathology on radiological and microscopic images can be performed using convolutional neural networks (CNN). However, there are scant studies on applying CNN to post-mortem gross images of visceral organs. This proof-of-concept study used 537 gross post-mortem images of dissected brain, heart, lung, liver, spleen, and kidney, which were randomly divided into a training and teaching datasets for the pre-trained CNN Xception. The CNN was trained using the training dataset and subsequently tested on the testing dataset. The overall accuracies were >95% percent for both training and testing datasets and have an F1 score of >0.95 for all dissected organs. This study showed that small datasets of post-mortem images can be classified with a very high accuracy using a pre-trained CNN. This novel area has the potential for future application in data mining, education and teaching, case review, research, quality assurance, auditing purposes, and identifying pathology.

The purpose of the study is to provide a literature review of the work done on sign language in Pakistan and the world. This study also provides a framework of an already developed prototype to translate Pakistani sign language into speech and text while using convolutional neural networking (CNN) to facilitate unimpaired teachers to bridge the communication gap among the deaf learners and unimpaired teachers. Due to the lack of sign language teaching, unimpaired teachers face difficulty in communicating with impaired learners. This communication gap can be filled with the help of this translation tool. Research indicates that a prototype has been evolved that can translate the English textual content into sign language and highlighted that there is a need for translation tool which can translate the signs into English text. The current study will provide an architectural framework of the Pakistani sign language to English text translation tool that how different components of technology like deep learning, convolutional neural networking, python, tensor Flow, and NumPy, InceptionV3 and transfer learning, eSpeak text to speech help in the development of a translation tool prototype. Keywords : Pakistan sign language (PSL), sign language (SL), translation, deaf, unimpaired, convolutional neural networking (CNN). DOI : 10.7176/JEP/10-15-18 Publication date :May 31 st 2019

A major objective of this project was to apply GE’s commercial machine learning and data analytics toolsets to large-scale, real-world, anonymized Phasor Measurement Unit (PMU) datasets in order to extract signatures, correlated and/or causal factors, and precursor patterns associated with significant power system phenomena. The project had a particular emphasis on extraction of insights relevant to asset health monitoring, real-time load modeling and cybersecurity monitoring. Additionally, the team was directed to undertake a comprehensive data quality analysis for the provided datasets and encouraged to estimate the ‘machine-learning readiness’ of the datasets by documenting any major obstacles to the application of commercial machine learning algorithms. To accomplish the aforementioned objectives, the project team’s work centered around the identification of key event signatures and application of the identified event signatures for event detection and event classification. The industry-validated, semi-supervised machine learning strategy employed for event signature identification involved several major tasks, including data-preprocessing, generation of an overabundance of features, normal data identification, normality modeling, and event signature identification through a methodical, quantitative ranking of features in order of relevance to each studied event type. Throughout the project, data quality issues and mitigation techniques were investigated. In this report, insights are provided regarding the readiness of the provided synchrophasor datasets for application of machine learning and data analytics. The methodologies employed for this technical strategy are summarized in this report. With regards to data preprocessing and feature generation, the provided Training and Test Datasets were ingested into GE’s big data environment. Subsequently, the team applied bad data cleansing and data imputation scripts, event detection scripts, and application programming interfaces (APIs) to the datasets for convenient data access. The project team completed development and validation of dozens of physics-based, statistics-based and transformation-based feature functions used for the extraction of over 60 synchrophasor features. Using a new parallel feature generation technology developed on this project, over 60 features have been rapidly generated for the full two years’ worth of Training and Test Dataset data associated with both the Eastern and Western interconnects. Even accommodating for temporal down-sampling inherent to the feature extraction procedure, this parallel feature generation activity resulted in a massive feature set with a storage requirement approximately equal to that of the raw training dataset itself. With regards to normal data identification and normality modeling, a normality model was built using the feature data extracted from the Training Dataset and iteratively refined subsequent to incremental adjustments and expansions of the Training Dataset feature data. With respect to event characterization and signature identification, an event signature identification pipeline was developed and used in conjunction with the normality model to identify over 15 event signatures for key event categories within the Training Dataset. The identified event signatures were used to characterize hundreds of key events in terms of relative severity, duration, and location of the event. An investigation was undertaken to identify correlated and causal factors involved in transformer events. A separate investigation into temporal trends in ring-down analysis results was undertaken to determine possible associations between system dynamics and various other factors such as loading, season or year. To validate the identified event signatures, additional work was undertaken to develop signature-based anomaly detection and classification tools suitable for convenient application to the synchrophasor datasets. The anomaly detection and classification tools, suitable for online application, were then applied to the entirety of the Eastern Interconnect Training and Test Datasets. Performance of the event detection and classification tools was evaluated upon receipt of the Test Dataset event logs (i.e., the labels for events contained in the Test Dataset), and promising results were obtained despite several challenges (documented herein) associated with application of supervised or semi-supervised machine learning methods to large-scale, anonymized datasets. Finally, the detection and classification tools were used to detect, classify, and characterize thousands of new events not included in the original event logs provided by the DOE within both the Training and Test Datasets.

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

Speech is the major way of human communication, but when it is limited, humans move to tactile kinaesthetic communication. People with speech-hearing impairments use sign language as an example of such adaptations. The deaf community uses Indian sign language (ISL) throughout India. In India, 250 licensed sign language interpreters are serving a deaf population of 1.8 to 7 million individuals. ISL interpreters are badly needed at institutes and places where persons with hearing impairments communicate. An Indian sign language picture database for English alphabets is established in this project. To prepare it for training, several pre-processing techniques were used. The effectiveness of deep learning neural networks is frequently influenced by the quantity of data available. As a result, data augmentation, a strategy for adding more and diverse samples to train datasets, was used to boost the effectiveness and outcomes of machine learning models. Our model is trained in CNN models utilizing transfer learning methodologies, with an accuracy of 95% for vgg16 and an accuracy of 92% for the inception model. More study on this research, as well as real-time implementation, has the potential to better connect people with hearing loss to society.