GhostVec: Directly Extracting Speaker Embedding from End-to-End Speech Recognition Model Using Adversarial Examples

The automatic speech recognition (ASR) system based on deep neural network is easy to be attacked by an adversarial example due to the vulnerability of neural network, which is a hot topic in recent years. The adversarial example does harm to the ASR system, especially if the common-dependent ASR goes wrong, it will lead to serious consequences. To improve the robustness and security of the ASR system, the defense method against adversarial examples must be proposed. Based on this idea, we propose an algorithm of devastation and detection on adversarial examples which can attack the current advanced ASR system. We choose advanced text-dependent and command-dependent ASR system as our target system. Generating adversarial examples by the OPT on text-dependent ASR and the GA-based algorithm on command-dependent ASR. The main idea of our method is input transformation of the adversarial examples. Different random intensities and kinds of noise are added to the adversarial examples to devastate the perturbation previously added to the normal examples. From the experimental results, the method performs well. For the devastation of examples, the original speech similarity before and after adding noise can reach 99.68%, the similarity of the adversarial examples can reach 0%, and the detection rate of the adversarial examples can reach 94%.

Combined speech enhancement and auditory modelling for robust distributed speech recognition

In this study, we investigate whether noise-augmented training can concurrently improve adversarial robustness in automatic speech recognition (ASR) systems. We conduct a comparative analysis of the adversarial robustness of four different ASR architectures, each trained under three different augmentation conditions: (1) background noise, speed variations, and reverberations; (2) speed variations only; (3) no data augmentation. We then evaluate the robustness of all resulting models against attacks with white-box or black-box adversarial examples. Our results demonstrate that noise augmentation not only enhances model performance on noisy speech but also improves the model’s robustness to adversarial attacks. • Comparative analysis: We examine for state-of-the-art automatic speech recognition (ASR) architectures under three different training conditions and evaluate their robustness with respect to white- and black-box adversarial attacks. • Practical implications: Our work has direct practical applications, as it shows that noise-augmented training, which is much cheeper than adversarial training, already helps in reducing the risks for adversarial attacks. This has not been explored in the context of automatic speech recognition yet and, thus, is a relevant contribution. • Contribution to security and privacy: Our paper provides necessary and useful insights to improve robustness against adversarial attacks.

In this paper, the improvements in the recently implemented Kannada speech recognition system is demonstrated in detail. The Kannada automatic speech recognition (ASR) system consists of ASR models which are created by using Kaldi, IVRS call flow and weather and agricultural commodity prices information databases. The task specific speech data used in the recently developed spoken dialogue system had high level of different background noises. The different types of noises present in collected speech data had an adverse effect on the on line and off line speech recognition performances. Therefore, to improve the speech recognition accuracy in Kannada ASR system, a noise reduction algorithm is developed which is a fusion of spectral subtraction with voice activity detection (SS-VAD) and minimum mean square error spectrum power estimator based on zero crossing (MMSE-SPZC) estimator. The noise elimination algorithm is added in the system before the feature extraction part. An alternative ASR models are created using subspace Gaussian mixture models (SGMM) and deep neural network (DNN) modeling techniques. The experimental results show that, the fusion of noise elimination technique and SGMM/DNN based modeling gives a better relative improvement of 7.68% accuracy compared to the recently developed GMM-HMM based ASR system. The least word error rate (WER) acoustic models could be used in spoken dialogue system. The developed spoken query system is tested from Karnataka farmers under uncontrolled environment.

Automatic Speech Recognition (ASR) systems convert spoken words into text and play a critical role in voice assistants, transcription services, and other accessibility tools. However, their reliance on machine learning models makes them susceptible to adversarial attacks. By adding small, often imperceptible perturbations to the input audio, or by convolving it with a carefully designed room impulse response, attackers can cause ASR systems to produce incorrect or even malicious transcripts. In this work, we propose an ensemble detection method for the efficient identification of targeted adversarial attacks in ASR systems. Specifically, we introduce MELo-FEST, which calculates the minimum energy in the low-frequency band of an audio signal to detect convolutive adversarial attacks. We then combine two spectrogram-based detection methods with a noise-adding approach to form an ensemble detector capable of identifying both additive and convolutive targeted adversarial attacks. Through extensive experiments, we demonstrate that our proposed ensemble detector can accurately identify adversarial audio generated by both non-adaptive and adaptive CW, PGD, and AdvReverb attacks in Wav2Vec2 and Whisper ASR systems, achieving an F1 score of at least 0.97. Moreover, our method performs both speech recognition and adversarial detection for each input audio sample in an average of under 0.13 seconds for Wav2Vec2 and 0.29 seconds for Whisper, making it well-suited for real-time applications. Additionally, we find that Whisper is more vulnerable than Wav2Vec2 to both non-adaptive and adaptive targeted adversarial attacks.

We report insights from translating Spanish conversational telephone speech into English text by cascading an automatic speech recognition (ASR) system with a statistical machine translation (SMT) system. The key new insight is that the informal register of conversational speech is a greater challenge for ASR than for SMT: the BLEU score for translating the reference transcript is 64%, but drops to 32% for translating automatic transcripts, whose word error rate (WER) is 40%. Several strategies are examined to mitigate the impact of ASR errors on the SMT output: (i) providing the ASR lattice, instead of the 1-best output, as input to the SMT system, (ii) training the SMT system on Spanish ASR output paired with English text, instead of Spanish reference transcripts, and (iii) improving the core ASR system. Each leads to consistent and complementary improvements in the SMT output. Compared to translating the 1-best output of an ASR system with 40% WER using an SMT system trained on Spanish reference transcripts, translating the output lattice of a better ASR system with 35% WER using an SMT system trained on ASR output improves BLEU from 32% to 38%.

This paper describes open source software (available at https://github.com/robotology/natural-speech) to build automatic speech recognition (ASR) systems and run them within the YARP platform. The toolkit is designed (i) to allow non-ASR experts to easily create their own ASR system and run it on iCub and (ii) to build deep learning-based models specifically addressing the main challenges an ASR system faces in the context of verbal human–iCub interactions. The toolkit mostly consists of Python, C++ code and shell scripts integrated in YARP. As additional contribution, a second codebase (written in Matlab) is provided for more expert ASR users who want to experiment with bio-inspired and developmental learning-inspired ASR systems. Specifically, we provide code for two distinct kinds of speech recognition: “articulatory” and “unsupervised” speech recognition. The first is largely inspired by influential neurobiological theories of speech perception which assume speech perception to be mediated by brain motor cortex activities. Our articulatory systems have been shown to outperform strong deep learning-based baselines. The second type of recognition systems, the “unsupervised” systems, do not use any supervised information (contrary to most ASR systems, including our articulatory systems). To some extent, they mimic an infant who has to discover the basic speech units of a language by herself. In addition, we provide resources consisting of pre-trained deep learning models for ASR, and a 2.5-h speech dataset of spoken commands, the VoCub dataset, which can be used to adapt an ASR system to the typical acoustic environments in which iCub operates.

Challenges of Automatic Speech Recognition for medical interviews - research for Polish language

In this study, a real-time automatic speech recognition (ASR) system based on the Kaldi ASR toolkit, with low-latency and customizable models, without any internet connection, was developed. The proposed ASR system includes a voice activity detection (VAD) module and an audio transmitter as a front-end speech processing and a decoder for the received audio signals. The ASR system was evaluated in terms of ASR accuracy and speech processing speed. Consequently, the ASR system achieved high ASR accuracy on the CSJ (Corpus of Spontaneous Japanese) test set with super low-latency.

These days, automatic speech recognition (ASR) systems achieve higher and higher accuracy rates. The score drops significantly, in case when the ASR system is being used with a non-native speaker of the language to be recognized. The main reason is specific pronunciation and accent features. A limited volume of labeled non-native speech datasets makes it difficult to train new ASR systems for non-native speakers.

This paper describes a comparison between hybrid and end-to-end Automatic Speech Recognition (ASR) systems, which were evaluated on the IberSpeech-RTVE 2020 Speech-to-Text Transcription Challenge. Deep Neural Networks (DNNs) are becoming the most promising technology for ASR at present. In the last few years, traditional hybrid models have been evaluated and compared to other end-to-end ASR systems in terms of accuracy and efficiency. We contribute two different approaches: a hybrid ASR system based on a DNN-HMM and two state-of-the-art end-to-end ASR systems, based on Lattice-Free Maximum Mutual Information (LF-MMI). To address the high difficulty in the speech-to-text transcription of recordings with different speaking styles and acoustic conditions from TV studios to live recordings, data augmentation and Domain Adversarial Training (DAT) techniques were studied. Multi-condition data augmentation applied to our hybrid DNN-HMM demonstrated WER improvements in noisy scenarios (about 10% relatively). In contrast, the results obtained using an end-to-end PyChain-based ASR system were far from our expectations. Nevertheless, we found that when including DAT techniques, a relative WER improvement of 2.87% was obtained as compared to the PyChain-based system.

Various adversarial audio attacks have recently been developed to fool automatic speech recognition (ASR) systems. We here propose a defense against such attacks based on the uncertainty introduced by dropout in neural networks. We show that our defense is able to detect attacks created through optimized perturbations and frequency masking on a state-of-the-art end-to-end ASR system. Furthermore, the defense can be made robust against attacks that are immune to noise reduction. We test our defense on Mozilla's CommonVoice dataset, the UrbanSound dataset, and an excerpt of the LibriSpeech dataset, showing that it achieves high detection accuracy in a wide range of scenarios.

<sec> <title>BACKGROUND</title> Code-switching between Egyptian Arabic (ARZ) and English is ubiquitous in clinical settings across the Arab world, yet no dedicated benchmark exists for evaluating Automatic Speech Recognition (ASR) systems under these conditions. Existing Arabic ASR benchmarks evaluate models on Modern Standard Arabic or single-dialect speech without medical vocabulary or code-switching. </sec> <sec> <title>OBJECTIVE</title> To introduce the Clinera ASR Benchmark, the first benchmark targeting medical ARZ-EN code-switched speech, and to evaluate commercial and open-source ASR systems on accurate recognition of medical terminology using both standard and novel medical-term-aware metrics </sec> <sec> <title>METHODS</title> We curated 683 utterances of Egyptian Arabic-English medical speech with dense medical-term annotations. We evaluated six ASR systems: the proprietary ElevenLabs Scribe v1, domain-adapted ArAZN-Whisper-S (244M parameters), Whisper-Large-V3, SeamlessM4T v2, Wav2Vec2 Large AR, and Whisper-Medium. We computed standard ASR metrics (WER, CER, MER) and proposed novel medical-term metrics (MT-Precision, MT-Recall, MT-F1, MT-wF1 weighted by inverse document frequency). Statistical comparisons used bootstrap confidence intervals and Wilcoxon signed-rank tests with Bonferroni correction. </sec> <sec> <title>RESULTS</title> ElevenLabs Scribe v1 achieved the lowest WER (27.8%) and highest MT-wF1 (60.5%), outperforming all open models by more than six times its size on medical-term recognition. General-purpose multilingual models achieved higher WER while almost entirely failing to recognize medical terms. Among open models, ArAZN-Whisper-S dominated on medical-term efficiency (26.0 MT-wF1 per 100M parameters). Even the best commercial system produced 11 fatal or high-risk medical errors affecting 1.6% of the corpus. </sec> <sec> <title>CONCLUSIONS</title> Current ASR systems exhibit a critical gap between general transcription accuracy and medical-term recognition in code-switched Arabic-English speech. Our benchmark and novel MT-metrics provide the first standardized evaluation framework for this clinically important setting, revealing that even top-performing systems pose patient safety risks through medical terminology errors. </sec>

Current automatic speech recognition (ASR) systems achieve over 90–95% accuracy, depending on the methodology applied and datasets used. However, the level of accuracy decreases significantly when the same ASR system is used by a non-native speaker of the language to be recognized. At the same time, the volume of labeled datasets of non-native speech samples is extremely limited both in size and in the number of existing languages. This problem makes it difficult to train or build sufficiently accurate ASR systems targeted at non-native speakers, which, consequently, calls for a different approach that would make use of vast amounts of large unlabeled datasets. In this paper, we address this issue by employing dual supervised learning (DSL) and reinforcement learning with policy gradient methodology. We tested DSL in a warm-start approach, with two models trained beforehand, and in a semi warm-start approach with only one of the two models pre-trained. The experiments were conducted on English language pronounced by Japanese and Polish speakers. The results of our experiments show that creating ASR systems with DSL can achieve an accuracy comparable to traditional methods, while simultaneously making use of unlabeled data, which obviously is much cheaper to obtain and comes in larger sizes.

The purpose of this article is to assess speech processing for listeners with simulated age-related hearing loss (ARHL) and to investigate whether the observed performance can be replicated using an automatic speech recognition (ASR) system. The long-term goal of this research is to develop a system that will assist audiologists/hearing-aid dispensers in the fine-tuning of hearing aids. Sixty young participants with normal hearing listened to speech materials mimicking the perceptual consequences of ARHL at different levels of severity. Two intelligibility tests (repetition of words and sentences) and 1 comprehension test (responding to oral commands by moving virtual objects) were administered. Several language models were developed and used by the ASR system in order to fit human performances. Strong significant positive correlations were observed between human and ASR scores, with coefficients up to .99. However, the spectral smearing used to simulate losses in frequency selectivity caused larger declines in ASR performance than in human performance. Both intelligibility and comprehension scores for listeners with simulated ARHL are highly correlated with the performances of an ASR-based system. In the future, it needs to be determined if the ASR system is similarly successful in predicting speech processing in noise and by older people with ARHL.