• Home
  • Search
  • Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation.
  • Cite Icon3
  • https://doi.org/10.1109/tbme.2025.3610284Copy DOI Icon

Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation.

Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

Individuals with hearing impairments face challenges in their ability to comprehend speech, particularly in noisy environments. This study explores the effectiveness of audio-visual speech enhancement (AVSE) in improving the intelligibility of vocoded speech in cochlear implant (CI) simulations. We propose a speech enhancement framework called Self-Supervised Learning-based AVSE (SSL-AVSE), which uses visual cues such as lip and mouth movements along with corresponding speech. Features are extracted using the AV-HuBERT model and refined through a bidirectional LSTM. Experiments were conducted using the Taiwan Mandarin speech with video (TMSV) dataset. Objective evaluations showed improvements in PESQ from 1.43 to 1.67 and in STOI from 0.70 to 0.74. NCM scores increased by up to 87.2% over the noisy baseline. Subjective listening tests further demonstrated maximum gains of 45.2% in speech quality and 51.9% in word intelligibility. SSL-AVSE consistently outperforms audio-only speech enhancement (AOSE) and conventional AVSE baselines. Listening tests with statistically significant results confirm its effectiveness. In addition to its strong performance, SSL-AVSE demonstrates cross-lingual generalization: although it was pretrained on English data, it performs effectively on Mandarin speech. This finding highlights the robustness of the features extracted by a pretrained foundation model and their applicability across languages. To the best of our knowledge, no prior work has explored the application of AVSE to CI simulations. This study provides the first evidence that incorporating visual information can significantly improve the intelligibility of vocoded speech in CI scenarios.

Similar Papers
  • Research Article
  • Citations1

Audio-visual speech enhancement in noisy environments using emotion-based contextual cues.

  • Jan 01, 2026
  • The Journal of the Acoustical Society of America
  • Tassadaq Hussain +8
  • Research Article

End-to-end audio-visual learning for cochlear implant sound coding simulations in noisy environments.

  • Jan 01, 2026
  • JASA express letters
  • Meng-Ping Lin +3
  • Conference Article
  • Citations23

On Training Targets and Objective Functions for Deep-learning-based Audio-visual Speech Enhancement

  • Nov 15, 2018
  • Daniel Michelsanti +3
  • PDF
  • Research Article
  • Citations16

Efficient Audio-Visual Speech Enhancement Using Deep U-Net With Early Fusion of Audio and Video Information and RNN Attention Blocks

  • Jan 01, 2021
  • IEEE Access
  • Jung-Wook Hwang +2
  • Research Article
  • Citations13

Audiovisual Speech Processing in Relationship to Phonological and Vocabulary Skills in First Graders.

  • Nov 04, 2021
  • Journal of Speech, Language, and Hearing Research
  • Liesbeth Gijbels +3
  • PDF
  • Research Article
  • Citations14

Deep Learning-Based Speech Enhancement With a Loss Trading Off the Speech Distortion and the Noise Residue for Cochlear Implants

  • Nov 08, 2021
  • Frontiers in Medicine
  • Yuyong Kang +2
  • Research Article
  • Citations43

Ideal time-frequency masking algorithms lead to different speech intelligibility and quality in normal-hearing and cochlear implant listeners.

  • Aug 26, 2014
  • IEEE transactions on bio-medical engineering
  • Raphael Koning +2
  • PDF
  • Dissertation

Mutual information based speech intelligibility prediction and its application to hearing aid fitting

  • May 13, 2022
  • Ganlong Wang
  • PDF
  • Research Article
  • Citations15

Candidacy for Cochlear Implantation in Prelingual Profoundly Deaf Adult Patients

  • Mar 28, 2022
  • Journal of Clinical Medicine
  • Ghizlene Lahlou +7
  • Conference Article
  • Citations16

An End-to-End Deep Learning Speech Coding and Denoising Strategy for Cochlear Implants

  • May 23, 2022
  • Tom Gajecki +1
  • Research Article
  • Citations54

Factors Affecting Daily Cochlear Implant Use in Children: Datalogging Evidence.

  • Nov 01, 2016
  • Journal of the American Academy of Audiology
  • Vijayalakshmi Easwar +3
  • Research Article
  • Citations28

Streaming of vowel sequences based on fundamental frequency in a cochlear-implant simulation.

  • Nov 01, 2008
  • The Journal of the Acoustical Society of America
  • Etienne Gaudrain +3
  • Research Article
  • Citations4

Breaking News: Verbotonal rehabilitation: Are we doing enough?

  • Jan 01, 2012
  • The Hearing Journal
  • Carl W Asp +2
  • Research Article
  • Citations108

Gender categorization is abnormal in cochlear implant users.

  • Aug 30, 2014
  • Journal of the Association for Research in Otolaryngology
  • Christina D Fuller +6
  • Research Article

Visual benefit in bimodal training with highly distorted speech sound

  • Jan 01, 2012
  • Seeing and Perceiving
  • Mika Sato +4
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.