Target Speaker Extraction under Noisy Underdetermined Conditions Using Conditional Variational Autoencoder, Global Style Token, and Neural Postfilter

Combined speech enhancement and auditory modelling for robust distributed speech recognition

Objective: To compare speech perception obtained with different time compression rates in teenagers that do or do not use personal listening devices (PLDs).Design: Teenagers in a high school were recruited to complete questionnaires reporting their recreational noise exposure using PLDs. The dose of individual recreational noise exposure was calculated. The individuals with the most and least doses of recreational noise were selected and grouped into PLD users and non-PLD users. Normal rate and time-compressed (60% and 70%) speech recognition in quiet and noisy conditions was measured.Study sample: PLD user and non-PLD user group each included 20 participants.Results: ANOVA analysis showed that the effect of group, background, compression rate, and interactions between any two factors are significant. Post hoc analysis showed that the speech recognition scores with normal rate in quiet and noise and those obtained from time-compressed speech in the quiet condition were not significantly different between PLD users and non-PLD users. However, differences in the time-compressed speech recognition scores (60% and 70%) in noisy conditions between the two groups were statistically significant.Conclusions: The fast-speed speech recognition in noise decreased significantly in PLD users compared with that in non-PLD users selected by extreme entertainment exposure.

Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for cross-modal self-supervision. This article (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes an audio-only self-supervision approach for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining can outperform fully supervised training and is especially useful to prevent overfitting on smaller sized datasets. We evaluate our learned audio representations for discrete emotion recognition, continuous affect recognition and automatic speech recognition. We outperform existing self-supervised methods for all tested downstream tasks. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition.

<b>Purpose:</b> The aim of the study was to assess whether sequential cochlear implantation (CI) with a prolonged interimplant interval (<i>M </i>= 15.2 years) between the first and second CIs benefited speech recognition and health-related quality of life.<b>Method: </b>This prospective study included 14 prelingually deafened participants who received their second CI after a prolonged interimplant interval (<i>M</i> = 15.2 years). Additionally, speech recognition ability over a 12-month period of bilateral implant use was investigated. The results of the speech recognition test in both quiet and noisy conditions were statistically analyzed for each CI alone and both CIs together. Nijmegen Cochlear Implant Questionnaire scores were also collected at activation and at 12 months after activation.<b>Results:</b> Improvements in speech recognition ability were observed following the use of the first implant alone and with the use of both implants together; however, progress was much slower with the use of the second implant alone, following its introduction. Furthermore, a significant difference in the trajectory of speech recognition ability was observed between the first and the second implanted ear. According to Nijmegen Cochlear Implant Questionnaire scores, all participants benefitted from bilateral CI after 12 months.<b>Conclusions: </b>Prolonged interimplant intervals resulted in asymmetrical speech recognition abilities. A significant improvement in the speech recognition scores was observed with the first implanted ear, and much slower progress was observed with the second implanted ear. However, the “poorer” second implanted ear could provide a considerable beneficial effect on the improved speech recognition and health-related quality of life with the bilateral CI.<br><b>Supplemental Material S1. </b>Arcsine transform; comparison of recognition scores with three modalities over time; predictors of diverge time point.<br>Zhang, Z., Wei, C., Zhang, Y., Zeng, Z., Cao, K., &amp; Liu, Y. (2020). Sequential bilateral cochlear implantation with prolonged time intervals. <i>Journal of Speech, Language, and Hearing Research</i>. https://doi.org/10.1044/2020_JSLHR-20-00140

This paper presents a hybrid Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) approach for Automatic Speech Recognition (ASR) using deep learning techniques on the Aurora-2 dataset. The dataset includes both clean and multi-condition modes, encompassing four noise scenarios : subway, babble, car, and exhibition hall, each evaluated at different signal-to-noise ratios (SNRs), and clean condition, and the results are compared with those from the ASC-10 dataset and the ESC-10 dataset. The problem addressed is the need for robust ASR models that perform well in both clean and noisy environments. The aim of utilizing the CNN-LSTM architecture is to enhance the recognition performance by combining the strengths of CNNs and LSTMs, rather than relying on either CNNs or LSTMs alone. Experimental results demonstrate that the combined CNN-LSTM model achieves superior classification performance, in clean environments on the Aurora2 dataset, attaining an accuracy of 97.96%, surpassing the individual CNN and LSTM models, which achieved 97.21% and 96.06%, respectively. In noisy conditions, the hybrid model also outperforms the standalone models, with an accuracy of 90.72%, compared to 90.12% for CNN and 86.12% for LSTM. These findings indicate that the CNN-LSTM model is more effective in handling various noise conditions and improving overall ASR accuracy.

A statistically designed Automatic Speech Recognition (ASR) system extracts features from speech signals using feature extraction methods, links the extracted features with the expected phonetics of the hypothesis using acoustic models, uses language model to add prior information about the structure of the target language. For many years, Mel-frequency Cepstral coefficients (MFCC), n-gram, and Hidden Markov Model (HMM) approaches have been used predominantly for feature extraction, language modeling and acoustic modeling, respectively. However, performance degradation of MFCC in noisy conditions and inaccuracy of HMMs while handling large vocabularies have made researchers to propose more efficient methods. The proposed work uses noise robust method Gammatone Frequency Cepstral Coefficients(GFCC) for feature extraction, trigram language modeling, and HMM-Gaussian mixture model (GMM) based acoustic modeling to implement a continuous Hindi language ASR system. Also, it applies Differential Evolution (DE) technique to refine the GFCC features and discriminative techniques to enhance performance of the acoustic model. The performance of the implemented system has been evaluated by using different feature extraction methods, variants of n-gram language modeling techniques and different discriminative techniques in clean as well as noisy conditions. Initially, the results reveal that DE optimized GFCC with HMM-Gaussian Mixture Model (GMM) acoustic modeling performs better than MFCC, PLP and MF-PLP feature extraction methods. Secondly, the experimental results show that the Minimum Phone Error (MPE) outperforms Maximum Mutual Information (MMI) and Maximum Likelihood Estimation (MLE) and trigram based language modeling gives more accurate results than unigram and bigram language modeling. Finally, it has been concluded that the continuous Hindi language ASR system implemented using DE refined GFFC feature extraction method with MPE discriminative training technique and trigram based language modeling gives better accuracy in clean as-well-as noisy environments.

This study investigated the effect of sound therapy combined with drug therapy (SDT) on gap detection threshold and speech recognition scores in patients with sudden sensorineural hearing loss (SSNHL). Patients with SSNHL were grouped randomly into SDT and drug therapy (DT) groups. All patients received standard drug treatment and patients in the SDT group additionally received sound stimulation for the affected ears for 6days. Pure tone audiogram, speech recognition scores at normal and time-compressed rates under quiet and noisy conditions, and the gap detection threshold of the SDT and DT groups before treatment and on day6 and 30 after treatment were compared. There were 20 patients in the SDT group and 24 in the DT group. The pure tone thresholds of affected ears were significantly lower in the SDT group on day6 after treatment than those in the DT group at 125 and 250Hz. Significantly lower gap detection thresholds and higher speech recognition scores under noisy conditions were observed at the normal and time-compressed rates in the SDT group than those in the DT group on day6 and 30 after treatment. Significant correlations were observed between the gap thresholds and speech recognition scores in a noisy environment at normal and time-compressed rates on day6 and 30. SDT may improve the recovery of hearing abilities, such as the gap in noise thresholds and speech recognition in noise, in the case of SSNHL. ChiCTR-IOR-17012262.

Decoding speaker's intent is a crucial part of spoken language understanding (SLU). The presence of noise or errors in the text transcriptions, in real life scenarios make the task more challenging. In this paper, we address the spoken language intent detection under noisy conditions imposed by automatic speech recognition (ASR) systems. We propose to employ confusion2vec word feature representation to compensate for the errors made by ASR and to increase the robustness of the SLU system. The confusion2vec, motivated from human speech production and perception, models acoustic relationships between words in addition to the semantic and syntactic relations of words in human language. We hypothesize that ASR often makes errors relating to acoustically similar words, and the confusion2vec with inherent model of acoustic relationships between words is able to compensate for the errors. We demonstrate through experiments on the ATIS benchmark dataset, the robustness of the proposed model to achieve state-of-the-art results under noisy ASR conditions. Our system reduces classification error rate (CER) by 20.84% and improves robustness by 37.48% (lower CER degradation) relative to the previous state-of-the-art going from clean to noisy transcripts. Improvements are also demonstrated when training the intent detection models on noisy transcripts.

IntroductionSpeech comprehension involves context-based lexical predictions for efficient semantic integration. This study investigated how noise affects the predictability effect on event-related potentials (ERPs) such as the N400 and late positive component (LPC) in speech comprehension.MethodsTwenty-seven listeners were asked to comprehend sentences in clear and noisy conditions (hereinafter referred to as “clear speech” and “noisy speech,” respectively) that ended with a high-or low-predictability word during electroencephalogram (EEG) recordings.ResultsThe study results regarding clear speech showed the predictability effect on the N400, wherein low-predictability words elicited a larger N400 amplitude than did high-predictability words in the centroparietal and frontocentral regions. Noisy speech showed a reduced and delayed predictability effect on the N400 in the centroparietal regions. Additionally, noisy speech showed a predictability effect on the LPC in the centroparietal regions.DiscussionThese findings suggest that listeners achieve comprehension outcomes through different neural mechanisms according to listening conditions. Noisy speech may be comprehended with a second-pass process that possibly functions to recover the phonological form of degraded speech through phonetic reanalysis or repair, thus compensating for decreased predictive efficiency.

The relevance of this study is driven by the increasing volume of online meetings and public discussions in digital formats, creating a demand for automated tools to analyze group communication. Traditional manual coding and transcription methods are highly labor-intensive and subjective, limiting large-scale research on communication patterns. The aim of this research is to develop and validate a comprehensive system for automated analysis of communicative behavior that integrates modern speaker diarization technologies, automatic speech recognition, and statistical analysis to provide a detailed picture of group dynamics in public discussions. Methods. The system is implemented using a microservice architecture with Python 3.10+, FastAPI, and React. Speaker diarization is performed using the pyannote.audio algorithm, which combines convolutional encoders with pre-trained WavLM models. Automatic speech recognition is carried out using transformer architectures (Whisper, AssemblyAI, Conformer). Communicative behavior analysis includes calculation of activity statistics, network analysis of interactions, and assessment of communication style. Results. The developed system successfully integrates speaker diarization with 0.5-second precision, automatic transcription, and multidimensional analysis of communication patterns. The modular architecture ensures flexibility for adaptation to various application domains. The system generates detailed timestamps of participant activity, visualizes speaking time distribution, and provides comprehensive analytics to improve decision-making processes. Prospects. Further development of the system includes integration of multimodal analysis considering non-verbal communication, improvement of stability in noisy conditions, domain adaptation for specific sectors, and implementation of real-time analysis of live discussions. The system opens new opportunities for studying group dynamics in corporate, educational, and governmental sectors.Received 2026-02-20Accepted 2026-03-27

On the distribution of Mel-filtered log-spectrum of speech in additive noise

An investigation of neural uncertainty estimation for target speaker extraction equipped RNN transducer

Performance of Automatic Speech Recognition (ASR) systems drops dramatically when transcribing conversations recorded in noisy conditions. Speech analytics suffer from this poor automatic transcription quality. To tackle this difficulty, a solution consists in mapping transcriptions into a space of hidden topics. This abstract representation allows to substantiate the drawbacks of the ASR process. The well-known and commonly used one is the topic-based representation from a Latent Dirichlet Allocation (LDA). Several studies demonstrate the effectiveness and reliability of this high-level representation. During the LDA learning process, distribution of words into each topic is estimated automatically. Nonetheless, in the context of a classification task, no consideration is made for the targeted classes. Thus, if the targeted application is to find out the main theme related to a dialogue, this information should be taken into consideration. In this paper, we propose to compare a classical topic-based representation of a dialogue, with a new one based not only on the dialogue content itself (words), but also on the theme related to the dialogue. This original representation is based on the author-topic (AT) model. The effectiveness of the proposed representation is evaluated on a classification task from automatic dialogue transcriptions between an agent and a customer of the Paris Transportation Company. Experiments confirmed that this author-topic model approach outperforms by far the classical topic representation, with a substantial gain of more than 7% in terms of correctly labeled conversations.

- Over the last decade, Automatic Speech Recognition (ASR)—have advanced substantially. The field has undergone a fundamental transformation with the introduction of end-to-end models, and its development has been further accelerated by recent developments in attention-based techniques and transfer learning on large- This paper compares state-of-the-art techniques in detail and provides a thorough review of research done in Hindi ASR since 2010. It examines modern methods for both monolingual and multilingual systems with an emphasis on deep learning models. This study examines multiple models on publicly available speech datasets to evaluate their performance for practical implementation. It also discusses open-source ASR research findings, challenges, and future directions, especially in mitigating data dependency, improving generalizability across low-resource languages, handling speaker variability, and operating in noisy conditions.

Natural Language Understanding (NLU) has been a major contributor to the innovation of voice assistant technologies, enabling more human-like interaction between human beings and machines. Between 2015 and 2024, significant advances have been made in improving the accuracy, contextuality, and flexibility of NLU systems. Earliest breakthroughs were focused on speech recognition; however, further research was focused on intent recognition, contextualization, and personalization. Despite such breakthroughs, there are still many research gaps, particularly in noisy situations, speech ambiguity, and scalability for multilingual usage. One of the most significant challenges is enhancing the robustness of NLU systems in noisy and heterogeneous conditions where standard models fail. Even though noise reduction methods have enhanced recognition performance, voice assistants continue to be impeded in actual, dynamic environments. Furthermore, natural language vagueness and multi-turn conversation continues to be challenging to handle, and further enhancement in dialogue management systems to enable better conversation flow and context retention is necessary. The multi-lingual ability has also improved, as seen in the improvements in transfer learning and cross-lingual models that improve performance in other linguistic settings. However, low-resource languages remain poorly supported, which indicates the demand for more universal models that can handle such languages with ease. Secondly, the greater emphasis on personalization brings the challenge of maintaining privacy and fairness in voice assistant systems, which calls for further research into ethical AI practices. The review identifies these research gaps and proposes that future progress should be made in further advancing noise-robust models, handling multi-turn dialogue intricacies, enhancing multilingual support, and personalizing while protecting privacy and reducing bias. Closing these gaps will make a substantial difference in the success of NLU in voice assistants, resulting in more accurate, context-sensitive, and user-focused systems