• Home
  • Search
  • A multi-modal fusion framework for continuous sign language recognition based on multi-layer self-attention mechanism
  • Cite Icon8
  • https://doi.org/10.3233/jifs-211697Copy DOI Icon

A multi-modal fusion framework for continuous sign language recognition based on multi-layer self-attention mechanism

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Some of the existing continuous sign language recognition (CSLR) methods require alignment. However, this is time-consuming, and breaks the continuity of the frame sequence, and also affects the subsequent process of CSLR. In this paper, we propose a multi-modal network framework for CSLR based on a multi-layer self-attention mechanism. We propose a 3D convolution residual neural network (CR3D) and a multi-layer self-attention network (ML-SAN) for the feature extraction stage. The CR3D obtains the short-term spatiotemporal features of the RGB and optical flow image streams, whereas the ML-SAN uses a bi-gated recurrent unit (BGRU) to model the long-term sequence relationship and a multi-layer self-attention mechanism to learn the internal relationships between sign language sequences. For the performance optimization stage, we propose a cross-modal spatial mapping loss function, which improves the precision of CSLR by studying the spatial similarity between the video and text domains. Experiments were conducted on two test datasets: the RWTH-PHOENIX-Weather multi-signer dataset, and a Chinese SL (CSL) dataset. The results show that the proposed method can obtain state-of-the-art recognition performance on the two datasets, with word error rate (WER) value of 24.4% and accuracy value of 14.42%, respectively.

Similar Papers
  • Research Article

Continuous Chinese Sign Language Recognition Using Millimeter-Wave Radar with DeepSeek Semantic Enhancement

  • Jan 01, 2025
  • IEEE Sensors Journal
  • Lei Zhang +5
  • Research Article

A dual-stream deep learning framework for continuous sign language recognition to enhance communication accessibility in the Ha'il region.

  • Feb 03, 2026
  • Scientific reports
  • Haifa Harrouch +3
  • Research Article
  • Citations16

Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal

  • Mar 08, 2024
  • ACM Transactions on Multimedia Computing, Communications, and Applications
  • Ronglai Zuo +1
  • Supplementary Content

Entropy-Regularized Joint CTC–Attention Learning for Low-Resource Continuous Sign Language Recognition

  • Feb 09, 2026
  • Research Square
  • Hanan A Taher +1
  • Research Article
  • Citations19

Swin-MSTP: Swin transformer with multi-scale temporal perception for continuous sign language recognition

  • Feb 01, 2025
  • Neurocomputing
  • Sarah Alyami +1
  • Research Article
  • Citations12

Gloss Prior Guided Visual Feature Learning for Continuous Sign Language Recognition.

  • Jan 01, 2024
  • IEEE transactions on image processing : a publication of the IEEE Signal Processing Society
  • Leming Guo +5
  • Conference Article

A Continuous Sign Language Recognition Method Based on Temporal Modeling and Semantic Alignment

  • Nov 14, 2025
  • J Li +5
  • Book Chapter
  • Citations1

Spatial-Temporal Graph Transformer for Skeleton-Based Sign Language Recognition

  • Jan 01, 2023
  • Zhengye Xiao +4
  • Research Article
  • Citations500

Sign Language Recognition: A Deep Survey

  • Aug 06, 2020
  • Expert Systems with Applications
  • Razieh Rastgoo +2
  • Conference Article
  • Citations3

Continuous Sign Language Recognition Using MediaPipe

  • Oct 19, 2023
  • Khang B Tran +2
  • Book Chapter

Conversational Algerian Sign Language Recognition Using the Deep Learning Approach

  • May 29, 2025
  • Lila Meddeber +1
  • Conference Article
  • Citations89

Transition movement models for large vocabulary continuous sign language recognition

  • Jan 01, 2004
  • Wen Gao +3
  • Conference Article
  • Citations59

A continuous Chinese sign language recognition system

  • Jan 01, 2000
  • Jiyong Ma +3
  • Conference Article
  • Citations22

A new system for Chinese sign language recognition

  • Jul 01, 2015
  • Jihai Zhang +2
  • Conference Article

Continuous Sign Language Recognition and Translation Using Deep Learning Approach

  • Mar 04, 2025
  • Rakesh Kumar M +2
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.