• Home
  • Search
  • Enhanced sign language translation using three stream multimodal occlusion resilient transformer
  • https://doi.org/10.1007/s10791-025-09772-2Copy DOI Icon

Enhanced sign language translation using three stream multimodal occlusion resilient transformer

Show More
  • Abstract
  • Literature Map
  • References
  • Similar Papers
Abstract

The proposed work addresses Continuous Sign Language Recognition and Translation (CSLRT) by introducing the Three-Stream Multimodal Occlusion-Resilient Transformer (T-MORT) model. T-MORT leverages three parallel encoder transformers to extract complementary features from visual, gesture, and emotion cues in sign language videos. These multi-modal representations are fused using a late fusion strategy, which preserves modality-specific information and enhances robustness to occlusions. A dual-decoder framework is employed: one decoder autoregressively generates gloss sequences to provide linguistic priors, while the second produces the final spoken language translation. To enhance efficiency, a dynamic keypoint extraction-based frame sampling algorithm, Spatio-Temporal Keypoint-Guided Sampling (STKGS), adaptively selects frames, reducing computational overhead. Extensive experiments on the PHOENIX14T dataset demonstrate state-of-the-art performance, surpassing existing CSLRT benchmarks. The proposed T-MORT model yields a BLEU-4 score of 25.07 and a ROUGE-L of 49.12.

Similar Papers
  • Conference Article
  • Citations3

Sign Language Translation Techniques Using Artificial Intelligence for the Hearing Impaired Community in Sri Lanka: A Review

  • Nov 23, 2023
  • Malith Priyankara +2
  • Research Article
  • Citations104

A survey on Sign Language machine translation

  • Oct 13, 2022
  • Expert Systems with Applications
  • Adrián Núñez-Marcos +2
  • Conference Article
  • Citations9

Dynamic Cross-Feature Fusion for American Sign Language Translation

  • Dec 15, 2021
  • Tejaswini Ananthanarayana +5
  • Research Article

Retrieval-Augmented Sign Language Translation

  • Oct 31, 2025
  • ACM Transactions on Multimedia Computing Communications and Applications
  • Huijie Yao +4
  • Research Article
  • Citations29

Increasing adaptability of a speech into sign language translation system

  • Sep 05, 2012
  • Expert Systems with Applications
  • Verónica López-Ludeña +4
  • Conference Article
  • Citations43

Time Series Neural Networks for Real Time Sign Language Translation

  • Dec 01, 2018
  • Sujay S Kumar +3
  • Research Article

Deep Understanding of Sign Language for Sign to Subtitle Alignment

  • Jan 01, 2026
  • IEEE Transactions on Multimedia
  • Youngjoon Jang +3
  • Research Article

MVDT: Multiview Distillation Transformer for View‐Invariant Sign Language Translation

  • Jan 01, 2025
  • IET Computer Vision
  • Zhong Guan +4
  • Research Article
  • Citations5

Dynamic sign language translating system using deep learning and natural language processing

  • Apr 28, 2021
  • Turkish Journal of Computer and Mathematics Education (TURCOMAT)
  • Aishwarya Kulkarni
  • Research Article

Enhanced Sign Language Translation Using Vision Transformers and Adaptive Representation

  • May 21, 2025
  • INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT
  • Abishek J
  • PDF
  • Research Article
  • Citations20

Sign language interpretation using machine learning and artificial intelligence

  • Nov 18, 2024
  • Neural Computing and Applications
  • Fatma M Najib
  • Book Chapter
  • Citations1

TSPNet-HF: A Hand/Face TSPNet Method for Sign Language Translation

  • Jan 01, 2022
  • Péricles B C Miranda +6
  • Research Article

Development of Sign Language Translator based on Gesture-to-Word Conversion

  • Dec 30, 2025
  • International Journal of Emerging Research in Science Engineering and Management
  • V Gopi +5
  • Conference Article

Spatio-temporal Sign Language Representation and Translation

  • Jan 01, 2022
  • Yasser Hamidullah +2
  • Research Article
  • Citations5

Development of Sign Language Application PESAN KULIAH With Material Substance Modification Based on Student Characteristics with Hearing Impaired

  • Jan 30, 2023
  • Journal of ICSAR
  • Utomo Utomo +3
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.