Dynamic Kannada Sign Language Recognition on Resource Constrained Devices

The Shape of Air: American Sign Language as Narrative Prosthesis in 21st Century North American Media

Sign Language is a meansof communication between audibly challenged people. To provide an interface between the audibly challenged community and the rest of the world we need Sign Language translators. A sign language recognition system computerizes the work of a sign language trEvery Sign Language Recognition (SLR) System is trained to recognizespecific sets of signs and they correspondingly output the sign in the required format. These SLR systems are built with powerful image processing techniques. The sign language recognition systems are capable of recognizing a specific set of signing gestures and output the corresponding text/audio. Most of these systems involve the techniques of detection, segmentation, tracking, gesture recognition and classification. This paperproposes a design for a SLR System.

Sign Language Recognition (SLR) has become an essential area of research due to its potential to promote understanding between the deaf and hearing communities through communication. This paper provides an in-depth study of various methodologies and models employed in SLR, focusing on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). We analyze their application to datasets from various sign languages, such as Arabic Sign Language (ArSL), American Sign Language (ASL), and British Sign Language (BSL), and explore how these models improve the recognition of dynamic, multi-dimensional hand gestures. This research not only advances the understanding of deep learning applications in sign language recognition but also addresses critical challenges in data processing and real-time applications, paving the way for inclusive technologies in informatics and human-computer interaction. Despite the progress in applying deep learning techniques to SLR, several challenges remain, particularly in dataset limitations, handling large vocabularies, and ensuring consistent performance across diverse environments and signers. The paper also investigates the broader applications of SLR, such as virtual reality, healthcare, education, and accessibility, and discusses the integration of SLR with human-computer interaction systems. Furthermore, it highlights current limitations in the field, such as difficulties with video data handling, the need for standard datasets, and issues related to training computational models. Finally, the paper outlines future research directions, including developing more robust SLR systems that can function effectively in uncontrolled environments, improving data collection methodologies, and creating real-time, user-friendly applications to assist the community of deaf and hard-of-hearing individuals.

ABSTRACTSign language is a communication tool for deaf and dumb people that includes known signs or body gestures to transfer meanings. It uses shapes, directions, movements of hands, and also facial expressions. A sign not only transmits a word but also conveys a tone. Many of the deaf people are not only able to speak, but also not able to write or read a language, so developing sign language translation or in other words sign language recognition (SLR) system can be very vital in their life. The SLR is extremely desired because of its capability to overcome the obstacles between deaf and normal people. It is one of the most important research fields in the human computer interaction studies. Hence, this paper presents an overview of the recent main research works with the vision-based SLR system, and the existing recognition techniques are discussed. Next, we focus on video-based SLR system and perform continuous SLR within video sequences.

Speech impairment affects an individual’s ability to communicate. People affected by this problem, suffer by the communication barrier with the normal people. Hence, to express their emotions and thoughts people will use sign language for their communication and. Although sign language is common in world, there may be a difficulty for people who do not have knowledge about sign language to communicate with speech impaired people who knows the sign language. Now there is a prominent growth in the field of computer vision and deep learning, which has been tremendous development in the fields of sign and motion recognition. The proposed model focusses on overcoming the challenges in verbal communication between non-sign language speakers and sign language speakers by building a deep learning model to recognize alphabets of American sign language. The model was trained on the dataset collected from Kaggle website which consisted of 26 American Sign Language alphabets. The model achieved 99.3% mean average precision in recognition of sign and average probability value for test image achieved was 0.99. The deep learning method used to build the model achieved more accuracy than the previous solutions.

Losing the ability to speak exerts psychological and social impacts on the affected people due to the lack of proper communication. Thus, Sign Language (SL) is considered a boon to people with hearing and speech impairment. SL has developed as a handy mean of communication that form the core of local deaf cultures. It is a visual–spatial language based on positional and visual components, such as the shape of fingers and hands, their location and orientation as well as arm and body movements. The problem is that SL is not understood by everyone, forming a communication gap between the mute and the able people. Multiple and systematic scholarly interventions that vary according to context have been implemented to overcome disability-related difficulties. Sign language recognition (SLR) systems based on sensory gloves are significant innovations that aim to procure data on the shape or movement of the human hand to bridge this communication gap, as the proposed system. The proposed model is a glove equipped with five flex sensors, interfacing with a control unit fixed on the arm, translating American Sign Language (ASL) and Arabic Sign Language (ArSL) to both text and speech, displayed on a simple Graphical User Interface (GUI). The proposed system aims to provide an affordable and user friendly SL translator system, working on the basis of Machine Learning (ML). However, it adapts to each person’s hand instead of using a generic data set. The system achieved 95% recognition rate with static gestures and up to 88% with dynamic gestures.

Contents: Preface. Part I: Reminiscences. L. Fant, Two Memorable Meals With Ursula and Ed. R.M. Battison, American Sign Language Linguistics 1970-1980: Memoir of a Renaissance. B. Bragg, A Fish Story for Ursula and Other Remembrances. Part II: Historical and Comparative Analyses of Sign Languages. J. Woodward, Sign Languages and Sign Language Families in Thailand and Vietnam. D. McKee, G. Kennedy, Lexical Comparisons of Signs From American, Australian, British, and New Zealand Sign Languages. H. Lane, R.C. Pillard, M. French, Origins of the American Deaf-World: Assimilating and Differentiating Societies and Their Relation to Genetic Patterning. Part III: Language in the Visual-Spatial Modality. E.L. Newport, T. Supalla, Sign Language Research at the Millennium. P. Siple, Attentional Resources and Working Memory: A New Framework for the Study of the Impact of Deafness on Cognition. M. Wilson, K. Emmorey, When Does Modality Matter? Evidence From ASL on the Nature of Working Memory. P.B. Braem, T. Bram, A Pilot Study of the Expressive Gestures Used by Classical Orchestra Conductors. N. Frishberg, An Interpreter Creates the Space. Part IV: Linguistic Analysis of Sign Languages. S.D. Fischer, More Than Just Handwaving: The Mutual Contributions of Sign Language and Linguistics. R.B. Wilbur, Phonological and Prosodic Layering of Nonmanuals in American Sign Language. R.C. Loew, C.T. Akamatsu, M. Lanaville, A Two-Handed Manual Alphabet in the United States. E. Pizzuto, V. Volterra, Iconicity and Transparency in Sign Languages: A Cross-Linguistic Cross-Cultural View. M. Kuntze, Codeswitching in ASL and Written English Language Contact. S.K. Liddell, Indicating Verbs and Pronouns: Pointing Away From Agreement. Part V: Language Acquisition. J.G. Bettger, Viewing Deaf Children in a New Way: Implications of Bellugi and Klima's Research for Education. R.P. Meier, Shared Motoric Factors in the Acquisition of Sign and Speech. A. Abrahamsen, Explorations of Enhanced Gestural Input to Children in the Bimodal Period. D. Lillo-Martin, Early and Late in Language Acquisition: Aspects of the Syntax and Acquisition of Wh-Questions in American Sign Language. J. Reilly, Bringing Affective Expression Into the Service of Language: Acquiring Perspective Marking in Narratives. C.A. Padden, V.L. Hanson, Search for the Missing Link: The Development of Skilled Reading in Deaf Children. L.A. Petitto, On the Biological Foundations of Human Language. Part VI: The Neural Organization of Sign Language. A.R. Damasio, H. Damasio, Language and the Brain. D.P. Corina, Some Observations Regarding Paraphasia in American Sign Language. H. Poizner, D. Brentari, M.E. Tyrone, J. Kegl, The Structure of Language as Motor Behavior: Clues From Signers With Parkinson's Disease. V.A. Fromkin, On the Uniqueness of Language.

Speech impairment limits a person’s capacity for oral and auditory communication. Improvements in communication between the deaf and the general public can be progressed by a real-time sign language detector. Recent studies have contributed to make progress in motion and gesture identification processes using Deep Learning (DL) methods and computer vision. But the development of static and dynamic sign language recognition (SLR) models is still a challenging area of research. The difficulty is in obtaining an appropriate model that addresses the challenges of continuous signs that are independent of the signer. Different signers’ speeds, durations, and many other factors make it challenging to create a model with high accuracy and continuity. This study mainly focused on SLR using a modified DL and hybrid optimization approach. Notably, spatial and geometric-based features are extracted via the Visual Geometry Group 16 (VGG16), and motion features are extracted using the optical flow approach. A new DL model, CNNSa-LSTM, is a combination of a Convolutional Neural Network (CNN), Self-Attention (SA), and Long-Short-Term Memory (LSTM) to identify sign language. This model is developed for feature extraction by combining CNNs for spatial analysis with SA mechanisms for focusing on relevant features, while LSTM effectively models temporal dependencies. The proposed CNNSa-LSTM model enhances performance in tasks involving complex, sequential data, such as sign language processing. Besides, a Hybrid Optimizer (HO) is proposed using the Hippopotamus Optimization Algorithm (HOA) and the Pathfinder Algorithm (PFA). The proposed model has been implemented in Python, and it has been evaluated over the existing models in terms of accuracy (98.7%), sensitivity (98.2%), precision (98.5%), Word Error Rate (WER) (0.131), Sign Error Rate (SER) (0.114), and Normalized Discounted Cumulative Gain (NDCG) (98%) as well. The proposed model has recorded the highest accuracy of 98.7%.

Sign language recognition devices are effective approaches to breaking the communication barrier between signers and non-signers and exploring human-machine interactions. Wearable gloves have been developed for gesture recognition and virtual reality applications by employing flexible sensors for motion detection and machine learning for data analysis. However, most existing wearable devices present limited sign language translating capacity due to the sensors' design and distribution. Here, we propose a cost-effective dual-hand soft fiber optic glove system consisting of multimode soft liquid-core fiber optic sensors, gyroscopes, wireless printed circuit boards, and batteries for sign language translation. In combination with different deep learning techniques and recognition strategies, the glove system can recognize static gestures and dynamic gestures of American Sign Language, and deduce the meaning of sentences by the sequence of gestures. The soft glove system exhibits a broad sign language range (10 numbers, 26 alphabets, 18 words, and 5 sentences meaning prediction), and high recognition accuracy (98.6% for static gestures, 95% for dynamic gestures). The results also present the recognizing capacity for high-correlated gestures (e.g., “M” and “N”). Finally, we demonstrate its application for controlling the motion of a virtual character through 7 discrete commands in the VR interface.

An optical modeless Sign Language Recognition (SLR) system is presented. The system uses the HAusdorf-Voronoi NETwork (HAVNET), an artificial neural network designed for 2D binary pattern recognition. It uses adaptation of the Hausdorff distance to determine the similarity between an input pattern and a learned representation. A detailed review of the architecture, the learning equations, and the recognition equations for the HAVNET network are presented. Competitive learning has been implemented in training the network using a nearest-neighbor technique. The SLR system is applied to the optical recognition of 24 static symbols from the American Sign Language convention. The SLR system represents the target images in a 80 X 80 pixel format. The implemented HAVNET network classifies the inputs into categories representing each of the symbols, using an output layer of 24 nodes. The network is trained with 5 different formats for each symbol and is tested with all 24 symbols in 15 new formats. Results from the SLR system without competitive training show shape identification problems, when distinguishing symbols with similar shapes. Implementation of competitive learning in the HAVNET neural network improved recognition accuracy on this task to 89%. The hand gestures are identified through a window search algorithm. Feature recognition is obtained from edge enhancement by applying a Laplacian filter and thresholding, which provides robustness to pose, color and background variations.

With its development, artificial intelligence has formed the basis for many studies aimed at facilitating people's lives. More successful results have been tried to be obtained with the increasing data and developing equipment in these studies. It is seen that these developments in artificial intelligence are reflected in the studies related to sign language conversion. In this study, a data set belonging to the letters in the Turkish Sign Language Alphabet was created, and the classification process was carried out with both the deep learning model we created and VGG16, Inceptionv3, Resnet, and Mobilnet models, which are frequently used in image classification. In addition, an open-source data set containing the letters in the American Sign Language Alphabet was organized similar to the data set containing the letters in the Turkish Sign Language Alphabet we created, and Deep Learning models were used to classify the letters in the American Sign Language Alphabet by using this data set. Performance evaluations of the classifications made by Deep Learning Models using both data sets were made. With this study, the results obtained from training Deep Learning methods with different data sets were compared. In addition, it is thought that the study will be useful in determining both the data set and the deep learning method to be used for the studies on the recognition of Sign Language Letters.

Sign language plays a crucial role as a communication tool for both the deaf and hard-of- hearing communities, enabling them to engage and interact effectively within their own community as well as with others.However, communication barriers arise when individuals unfamiliar with sign language engage with those who rely on it, underscoring the need for inclusive solutions. Real-time sign language interpretation systems, leveraging machine learning and computer vision technologies, present a promising approach to bridging this gap. These systems convert sign language gestures into spoken or written language by utilizing gesture recognition algorithms, neural networks, and natural language processing. By analyzing hand movements, facial expressions, and body language, the systems provide accurate, context-aware translations of various sign languages, such as American Sign Language (ASL), with minimal delay. This enables seamless, natural interactions, making such technologies essential for fostering inclusive communication in diverse settings. Key Words: Sign language recognition, real- time interpretation, machine learning, com- puter vision, gesture recognition, neural networks, natural language processing, communication barriers, inclusivity, American Sign Language (ASL).

PurposeTo conduct a head-to-head comparison between deep learning (DL) and radiomics models across institutions for predicting microvascular invasion (MVI) in hepatocellular carcinoma (HCC) and to investigate the model robustness and generalizability through rigorous internal and external validation.MethodsThis retrospective study included 2304 preoperative images of 576 HCC lesions from two centers, with MVI status determined by postoperative histopathology. We developed DL and radiomics models for predicting the presence of MVI using B-mode ultrasound, contrast-enhanced ultrasound (CEUS) at the arterial, portal, and delayed phases, and a combined modality (B + CEUS). For radiomics, we constructed models with enlarged vs. original regions of interest (ROIs). A cross-validation approach was performed by training models on one center’s dataset and validating the other, and vice versa. This allowed assessment of the validity of different ultrasound modalities and the cross-center robustness of the models. The optimal model combined with alpha-fetoprotein (AFP) was also validated. The head-to-head comparison was based on the area under the receiver operating characteristic curve (AUC).ResultsThirteen DL models and 25 radiomics models using different ultrasound modalities were constructed and compared. B + CEUS was the optimal modality for both DL and radiomics models. The DL model achieved AUCs of 0.802–0.818 internally and 0.667–0.688 externally across the two centers, whereas radiomics achieved AUCs of 0.749–0.869 internally and 0.646–0.697 externally. The radiomics models showed overall improvement with enlarged ROIs (P < 0.05 for both CEUS and B + CEUS modalities). The DL models showed good cross-institutional robustness (P > 0.05 for all modalities, 1.6–2.1% differences in AUC for the optimal modality), whereas the radiomics models had relatively limited robustness across the two centers (12% drop-off in AUC for the optimal modality). Adding AFP improved the DL models (P < 0.05 externally) and well maintained the robustness, but did not benefit the radiomics model (P > 0.05).ConclusionCross-institutional validation indicated that DL demonstrated better robustness than radiomics for preoperative MVI prediction in patients with HCC, representing a promising solution to non-standardized ultrasound examination procedures.

Sign language recognition (SLR) system is helpful for individuals who can't talk or hear. One can easily communicate with this kind of people using SLR system. In this paper we are focusing on SLR approach where we have apply our proposed algorithm to recognize hand gestures of routine words like Morning, Night, Monday, Tuesday and so forth. Total 10 routine words we have recognize using our proposed technique. For recognize hand pose we have utilized skin color detection algorithm. Further we have applied correlation-coefficient algorithm to distinguish closeness features which are further continue in Neuro-fuzzy(NF) classification algorithm to recognize words. Presented algorithm tried on Matlab. With taken of 10 routine words our proposed framework of techniques got 92% correction rate.

This article is dedicated to analyzing advanced approaches in temporal modeling and real-time gesture recognition within sign language recognition (SLR) systems. Sign glosses are expressed through the spatio-temporal characteristics of visual information, which requires the use of sequence-processing models for their automatic recognition. The study primarily evaluates the effectiveness of three key models: Long Short-Term Memory (LSTM) networks, Temporal Convolutional Networks (TCN), and Transformer-based architectures. The article also examines methods applied for real-time analysis of sign glosses, including: Sliding window segmentation of video streams; Self-attention mechanisms for identifying dependencies between gestures; Gloss mapping algorithms for linking sign movements to linguistic units; Ontological integration techniques for enhancing semantic accuracy. Practical results indicate that combining temporal modeling with semantic analysis and contextual verification algorithms ensures continuous and high-accuracy recognition of sign movements. In particular, multimodal systems (video + sensor + gloss) utilizing Transformer-based approaches achieved superior performance in real-time conversion of continuous sign gloss streams into text. The findings of this study hold practical significance for the development of smart assistive devices for automatic sign language translation, interactive interfaces for hearing-impaired users, and specialized SLR platforms for educational and instructional purposes.