- Research Article
500
- 10.1016/j.eswa.2020.113794
Sign Language Recognition: A Deep Survey
- Aug 06, 2020
- Expert Systems with Applications
- Razieh Rastgoo + 2 more +2
Sign Language Recognition: A Deep Survey
Continuous Sign Language Recognition (CSLR) is the process of understanding the combination of hand gestures, facial expressions and body pose in real-time, in conjunction with the grammatical and vocabulary structural elements that allow cross-cultural communication with people who have difficulty hearing and/or communicating verbally. This study aims to develop an automatic system capable of detecting the region of interest on the signer's body and classifying each sign into its appropriate category, which will be combined with the natural language (NL). To achieve these objectives, a novel approach is proposed with four main contributions, which can be summarized as the independence of the signer's position in the frame, the collection of an Algerian Sign Language dataset containing 46 dynamic signs (2083 videos), achieving 97.75% accuracy for dynamic sign recognition and 97.37% for static sign recognition and an NLP model to predict the next words in the sentence, thus speeding up communication.
Sign Language Recognition: A Deep Survey
Sign Language Recognition: A Deep Survey
Continuous Chinese Sign Language Recognition Using Millimeter-Wave Radar with DeepSeek Semantic Enhancement
Chinese sign language recognition plays a critical role in supporting communication for the hearing-impaired. Recently, Millimeter-wave radar has attracted growing attention in gesture recognition due to its advantages of non-contact sensing and privacy preservation. By extracting micro-Doppler signatures, Millimeter-wave radar enables gesture recognition without reliance on visual or wearable inputs. However, most existing radar-based methods focus on isolated gestures and lack the ability to segment gesture boundaries, making them unsuitable for continuous Chinese sign language recognition. Moreover, their outputs are often limited to disjointed word-level predictions, hindering semantic fluency. To address these limitations, we propose a novel radar-based continuous Chinese sign language recognition framework that uses time-frequency singularity analysis to segment gestures from radar echo signals. In response to the challenges of low spatial resolution and high noise in radar-based gesture recognition, we propose a hybrid model combining Swin Transformer and BiLSTM to capture both spatial and temporal features, enabling robust performance in continuous sign language recognition. To enhance sentence-level fluency and reduce confusion between similar gestures, we incorporate the large-scale language model DeepSeek for semantic correction and natural language refinement. Experimental results demonstrate that our method achieves 96.8% average frame-level classification accuracy and reduces word error rate to 5.6%, significantly outperforms traditional methods. The results demonstrate the superiority and effectiveness of the proposed approach.
Read moreImproving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal
Deep-learning-based continuous sign language recognition (CSLR) models typically consist of a visual module, a sequential module, and an alignment module. However, the effectiveness of training such CSLR backbones is hindered by limited training samples, rendering the use of a single connectionist temporal classification loss insufficient. To address this limitation, we propose three auxiliary tasks to enhance CSLR backbones. First, we enhance the visual module, which is particularly sensitive to the challenges posed by limited training samples, from the perspective of consistency. Specifically, since sign languages primarily rely on signers’ facial expressions and hand movements to convey information, we develop a keypoint-guided spatial attention module that directs the visual module to focus on informative regions, thereby ensuring spatial attention consistency. Furthermore, recognizing that the output features of both the visual and sequential modules represent the same sentence, we leverage this prior knowledge to better exploit the power of the backbone. We impose a sentence embedding consistency constraint between the visual and sequential modules, enhancing the representation power of both features. The resulting CSLR model, referred to as consistency-enhanced CSLR, demonstrates superior performance on signer-dependent datasets, where all signers appear during both training and testing. To enhance its robustness for the signer-independent setting, we propose a signer removal module based on feature disentanglement, effectively eliminating signer-specific information from the backbone. To validate the effectiveness of the proposed auxiliary tasks, we conduct extensive ablation studies. Notably, utilizing a transformer-based backbone, our model achieves state-of-the-art or competitive performance on five benchmarks, including PHOENIX-2014, PHOENIX-2014-T, PHOENIX-2014-SI, CSL, and CSL-Daily. Code and models are available at https://github.com/2000ZRL/LCSA_C2SLR_SRM.
Read moreEntropy-Regularized Joint CTC–Attention Learning for Low-Resource Continuous Sign Language Recognition
<title>Abstract</title> Continuous Sign Language Recognition (CSLR) seeks to transcribe unsegmented sign language videos into gloss sequences without frame-level supervision, presenting persistent challenges in temporal alignment, long-range dependency modeling, and reliable sequence-level generalization. While recent advances have achieved strong performance in high-resource languages, Kurdish Sign Language (KrdSL) remains largely unexplored due to the absence of sentence-level benchmarks. To address this gap, we introduce <bold>KrdSL-1400</bold> , the first continuous Kurdish Sign Language dataset, comprising 1,400 annotated video sequences covering 40 linguistically structured sentences performed by seven native signers, providing a standardized benchmark for low-resource CSLR. We propose a <bold>hybrid spatio-temporal CSLR framework</bold> that combines deep convolutional visual encoding with sequence-aware temporal modeling and a <bold>multi-task joint CTC–attention decoding strategy</bold> explicitly designed to address alignment uncertainty. The CTC objective enforces monotonic alignment, while an <bold>entropy-regularized multi-head attention mechanism</bold> dynamically emphasizes linguistically salient temporal segments, enabling robust sequence prediction without reliance on pose estimation or handcrafted features. Training dynamics exhibit stable and consistent convergence, with closely aligned training and validation WER curves indicating strong generalization. Quantitative evaluation shows that a CTC baseline achieves a WER of <bold>13.5%</bold> , which is reduced to <bold>10.5%</bold> using single-head attention, while the proposed model attains the best performance with a WER of <bold>9.5%</bold> , corresponding to an approximate <bold>30% relative improvement</bold> . <bold>Cross-dataset evaluation</bold> on the large-scale <bold>PHOENIX-2014-T</bold> benchmark further demonstrates generalization, achieving a WER of <bold>13.7%</bold> and outperforming recent attention-based and transformer-based CSLR approaches.
Read moreSwin-MSTP: Swin transformer with multi-scale temporal perception for continuous sign language recognition
Swin-MSTP: Swin transformer with multi-scale temporal perception for continuous sign language recognition
Continuous Sign Language Recognition Using MediaPipe
According to WHO, more than 5% of the world’s population suffer from hearing loss. They estimate that by 2050, the figure will reach over 700 million people. Therefore, the need for a sign language translation technique is essential. However, that has not been an effective approach to recognize dynamic sign language in real-time. This paper proposed the new method to continuous translate sign language recognition in real-time using Convolutional Neural Network and Long-Short term Memory. With the help of extracting coordinates of body joints from MediaPipe, it reaches an overall accuracy up to 99.5%.
Read moreTransition movement models for large vocabulary continuous sign language recognition
The major challenges that sign language recognition (SLR) now faces are developing methods that solve large vocabulary continuous sign problems. In this paper, large vocabulary continuous SLR based on transition movement models is proposed. The proposed method employs the temporal clustering algorithm to cluster a large amount of transition movements, and then the corresponding training algorithm is also presented for automatically segmenting and training these transition movement models. The clustered models can improve the generalization of transition movement models, and are very suitable for large vocabulary continuous SLR. At last, the estimated transition movement models, together with sign models, are viewed as candidate models of the Viterbi search algorithm for recognizing continuous sign language. Experiments show that continuous SLR based on transition movement models has good performance over a large vocabulary of 5113 signs.
Read moreGloss Prior Guided Visual Feature Learning for Continuous Sign Language Recognition.
Continuous sign language recognition (CSLR) is to recognize the glosses in a sign language video. Enhancing the generalization ability of CSLR's visual feature extractor is a worthy area of investigation. In this paper, we model glosses as priors that help to learn more generalizable visual features. Specifically, the signer-invariant gloss feature is extracted by a pre-trained gloss BERT model. Then we design a gloss prior guidance network (GPGN). It contains a novel parallel densely-connected temporal feature extraction (PDC-TFE) module for multi-resolution visual feature extraction. The PDC-TFE captures the complex temporal patterns of the glosses. The pre-trained gloss feature guides the visual feature learning through a cross-modality matching loss. We propose to formulate the cross-modality feature matching into a regularized optimal transport problem, it can be efficiently solved by a variant of the Sinkhorn algorithm. The GPGN parameters are learned by optimizing a weighted sum of the cross-modality matching loss and CTC loss. The experiment results on German and Chinese sign language benchmarks demonstrate that the proposed GPGN achieves competitive performance. The ablation study verifies the effectiveness of several critical components of the GPGN. Furthermore, the proposed pre-trained gloss BERT model and cross-modality matching can be seamlessly integrated into other RGB-cue-based CSLR methods as plug-and-play formulations to enhance the generalization ability of the visual feature extractor.
Read moreSign Language Recognition: A Comprehensive Review of Traditional and Deep Learning Approaches, Datasets, and Challenges
The Deaf are a large social group in society. Their unique way of communicating through sign language is often confined within their community due to limited understanding by individuals outside of this demographic. This is where sign language recognition (SLR) comes in to help normal people understand the meaning of sign language. In recent years, new methods of sign language recognition have been developed and achieved good results, so it is necessary to make a summary. This review mainly focuses on the introduction of sign language recognition techniques based on algorithms especially in recent years, including the recognition models based on traditional methods and deep learning approaches, sign language datasets, challenges and future directions in SLR. To make the method structure clearer, this article explains and compares the basic principles of different methods from the perspectives of feature extraction and temporal modelling. We hope that this review will provide some reference and help for future research in sign language recognition.
Read moreA lightweight network-based sign language robot with facial mirroring and speech system
A lightweight network-based sign language robot with facial mirroring and speech system
A multi-modal fusion framework for continuous sign language recognition based on multi-layer self-attention mechanism
Some of the existing continuous sign language recognition (CSLR) methods require alignment. However, this is time-consuming, and breaks the continuity of the frame sequence, and also affects the subsequent process of CSLR. In this paper, we propose a multi-modal network framework for CSLR based on a multi-layer self-attention mechanism. We propose a 3D convolution residual neural network (CR3D) and a multi-layer self-attention network (ML-SAN) for the feature extraction stage. The CR3D obtains the short-term spatiotemporal features of the RGB and optical flow image streams, whereas the ML-SAN uses a bi-gated recurrent unit (BGRU) to model the long-term sequence relationship and a multi-layer self-attention mechanism to learn the internal relationships between sign language sequences. For the performance optimization stage, we propose a cross-modal spatial mapping loss function, which improves the precision of CSLR by studying the spatial similarity between the video and text domains. Experiments were conducted on two test datasets: the RWTH-PHOENIX-Weather multi-signer dataset, and a Chinese SL (CSL) dataset. The results show that the proposed method can obtain state-of-the-art recognition performance on the two datasets, with word error rate (WER) value of 24.4% and accuracy value of 14.42%, respectively.
Read moreVision-Based Sign Language Classification: A Directional Review
ABSTRACTSign language is a communication tool for deaf and dumb people that includes known signs or body gestures to transfer meanings. It uses shapes, directions, movements of hands, and also facial expressions. A sign not only transmits a word but also conveys a tone. Many of the deaf people are not only able to speak, but also not able to write or read a language, so developing sign language translation or in other words sign language recognition (SLR) system can be very vital in their life. The SLR is extremely desired because of its capability to overcome the obstacles between deaf and normal people. It is one of the most important research fields in the human computer interaction studies. Hence, this paper presents an overview of the recent main research works with the vision-based SLR system, and the existing recognition techniques are discussed. Next, we focus on video-based SLR system and perform continuous SLR within video sequences.
Read moreA dual-stream deep learning framework for continuous sign language recognition to enhance communication accessibility in the Ha'il region.
Human-Computer Interaction (HCI) depends critically on hand gesture recognition, which also provides the basis of sign language communication for those with hearing problems. In automatic sign language identification systems, however, the contribution of non-manual elements like head posture and facial emotions has sometimes been disregarded. In order to fully capture the diversity of sign language modalities and to enhance the quality of life of deaf people in the region of Ha'il, we present in this study a dual-stream convolutional neural network (TS-CNN) that separately models hand motions and head positions. We provide fusion techniques that improve feature correlation and representation in order to guarantee efficient cooperation between the two streams. Moreover, we use a Feature Enhancement Module (FEM) to enhance gloss-level prediction accuracy and fine temporal alignment. Two benchmark datasets, RWTH-PHOENIX-Weather 2014 and CSL Split II, are used to assess the TS-CNN framework and show a notable increase in recognition performance. Competitive Word Error Rates (WER) obtained by TS-CNN from experimental data validate its performance in simulating the complicated multimodal character of continuous sign language recognition.
Read moreSpatial-Temporal Graph Transformer for Skeleton-Based Sign Language Recognition
For continuous sign language recognition (CSLR), the skeleton sequence is insusceptible to environmental variances and achieves much attention. Previous studies mainly employ hand-craft features or the spatial-temporal graph convolution networks for skeleton modality and neglect the importance of capturing the information between distant nodes and the long-term context in CSLR. To learn more robust spatial-temporal features for CSLR, we propose a Spatial-Temporal Graph Transformer (STGT) model for skeleton-based CSLR. With the self-attention mechanism, the human skeleton graph is treated as a fully connected graph, and the relationship between distant nodes can be established directly in the spatial dimension. In the temporal dimension, the long-term context can be learned easily due to the characteristic of the transformer. Moreover, we propose graph positional embedding and graph multi-head self-attention to help the STGT distinguish the meanings of different nodes. We conduct the ablation study on the action recognition dataset to validate the effectiveness and analyze the advantages of our method. The experimental results on two CSLR datasets demonstrate the superiority of the STGT on skeleton-based CSLR.
Read moreA Continuous Sign Language Recognition Method Based on Temporal Modeling and Semantic Alignment
Sign language is a primary means of communication for the deaf community. Continuous sign language recognition (CSLR) remains challenging due to its large vocabulary, flexible grammar, and co-articulation effects. This paper proposes an end-to-end framework that integrates temporal modeling with semantic alignment. The method employs MediaPipe to extract and normalize 42 hand keypoints per frame. A multi-constraint Transformer encoder, incorporating Word Existence and Word Count constraints, enhances temporal dependency modeling. A semantic alignment module bridges the gap between visual and textual representations, and a CTC-BERT bidirectional decoder enhances sentence coherence. Experiments on Chinese continuous sign language datasets achieve a Word Error Rate (WER) below 20%, demonstrating the method's high recognition accuracy and strong semantic consistency.
Read more