Speech-Based Real-Time Subtitling Services

Combined speech enhancement and auditory modelling for robust distributed speech recognition

BackgroundRecent advances in natural language processing and artificial intelligence have led to widespread adoption of speech recognition technologies. In consumer health applications, speech recognition is usually applied to support interactions with conversational agents for data collection, decision support, and patient monitoring. However, little is known about the use of speech recognition in consumer health applications and few studies have evaluated the efficacy of conversational agents in the hands of consumers. In other consumer-facing tools, cognitive load has been observed to be an important factor affecting the use of speech recognition technologies in tasks involving problem solving and recall. Users find it more difficult to think and speak at the same time when compared to typing, pointing, and clicking. However, the effects of speech recognition on cognitive load when performing health tasks has not yet been explored.ObjectiveThe aim of this study was to evaluate the use of speech recognition for documentation in consumer digital health tasks involving problem solving and recall.MethodsFifty university staff and students were recruited to undertake four documentation tasks with a simulated conversational agent in a computer laboratory. The tasks varied in complexity determined by the amount of problem solving and recall required (simple and complex) and the input modality (speech recognition vs keyboard and mouse). Cognitive load, task completion time, error rate, and usability were measured.ResultsCompared to using a keyboard and mouse, speech recognition significantly increased the cognitive load for complex tasks (Z=–4.08, P<.001) and simple tasks (Z=–2.24, P=.03). Complex tasks took significantly longer to complete (Z=–2.52, P=.01) and speech recognition was found to be overall less usable than a keyboard and mouse (Z=–3.30, P=.001). However, there was no effect on errors.ConclusionsUse of a keyboard and mouse was preferable to speech recognition for complex tasks involving problem solving and recall. Further studies using a broader variety of consumer digital health tasks of varying complexity are needed to investigate the contexts in which use of speech recognition is most appropriate. The effects of cognitive load on task performance and its significance also need to be investigated.

Physical workload during use of speech recognition and traditional computer input devices

This article describes the findings from a small-scale research project involving 14 test subjects. Experiments were conducted with student nurses and their use of speech recognition technologies. Speech and keyboard were compared as input methods. A hypothesis is put forward and questionnaires and experiments were devised to test the hypothesis. The experiments involved a form of software monitoring, which is discussed. Results from the data analysis of the experiments are included but not the results from the analysis of the questionnaires. During the discussion, summary data from the questionnaires are included. Issues such as usability, error rates and hardware problems are discussed. A brief conclusion considers the future of the keyboard as the predominant computer interface.

To conduct a replication study to validate previously identified significant risks and inefficiencies associated with the use of speech recognition (SR) for documentation within an electronic health record (EHR) system. Thirty-five emergency department clinicians undertook randomly allocated clinical documentation tasks using keyboard and mouse (KBM) or SR using a commercial EHR system. The experiment design, setting, and tasks (E2) replicated an earlier study (E1), while technical integration issues that may have led to poorer SR performance were addressed. Complex tasks were significantly slower to complete using SR (16.94%) than KBM (KBM: 191.9 s, SR: 224.4 s; p = 0.009; CI, 11.9-48.3), replicating task completion times observed in the earlier experiment. Errors (non-typographical) were significantly higher with SR compared with KBM for both simple (KBM: 3, SR: 84; p < 0.001; CI, 1.5-2.5) and complex tasks (KBM: 23, SR: 53; p = 0.001; CI, 0.5-1.0), again replicating earlier results (E1: 170, E2: 163; p = 0.660; CI, 0.0-0.0). Typographical errors were reduced significantly in the new study (E1: 465, E2: 150; p < 0.001; CI, 2.0-3.0). The results of this study replicate those reported earlier. The use of SR for clinical documentation within an EHR system appears to be consistently associated with decreased time efficiencies and increased errors. Modifications implemented to optimize SR integration in the EHR seem to have resulted in minor improvements that did not fundamentally change overall results. This replication study adds further evidence for the poor performance of SR-assisted clinical documentation within an EHR. Replication studies remain rare in informatics literature, especially where study results are unexpected or have significant implication; such studies are clearly needed to avoid overdependence on the results of a single study.

In this paper, the improvements in the recently implemented Kannada speech recognition system is demonstrated in detail. The Kannada automatic speech recognition (ASR) system consists of ASR models which are created by using Kaldi, IVRS call flow and weather and agricultural commodity prices information databases. The task specific speech data used in the recently developed spoken dialogue system had high level of different background noises. The different types of noises present in collected speech data had an adverse effect on the on line and off line speech recognition performances. Therefore, to improve the speech recognition accuracy in Kannada ASR system, a noise reduction algorithm is developed which is a fusion of spectral subtraction with voice activity detection (SS-VAD) and minimum mean square error spectrum power estimator based on zero crossing (MMSE-SPZC) estimator. The noise elimination algorithm is added in the system before the feature extraction part. An alternative ASR models are created using subspace Gaussian mixture models (SGMM) and deep neural network (DNN) modeling techniques. The experimental results show that, the fusion of noise elimination technique and SGMM/DNN based modeling gives a better relative improvement of 7.68% accuracy compared to the recently developed GMM-HMM based ASR system. The least word error rate (WER) acoustic models could be used in spoken dialogue system. The developed spoken query system is tested from Karnataka farmers under uncontrolled environment.

The automatic speech recognition (ASR) system based on deep neural network is easy to be attacked by an adversarial example due to the vulnerability of neural network, which is a hot topic in recent years. The adversarial example does harm to the ASR system, especially if the common-dependent ASR goes wrong, it will lead to serious consequences. To improve the robustness and security of the ASR system, the defense method against adversarial examples must be proposed. Based on this idea, we propose an algorithm of devastation and detection on adversarial examples which can attack the current advanced ASR system. We choose advanced text-dependent and command-dependent ASR system as our target system. Generating adversarial examples by the OPT on text-dependent ASR and the GA-based algorithm on command-dependent ASR. The main idea of our method is input transformation of the adversarial examples. Different random intensities and kinds of noise are added to the adversarial examples to devastate the perturbation previously added to the normal examples. From the experimental results, the method performs well. For the devastation of examples, the original speech similarity before and after adding noise can reach 99.68%, the similarity of the adversarial examples can reach 0%, and the detection rate of the adversarial examples can reach 94%.

Multi-microphone speech recognition integrating beamforming, robust feature extraction, and advanced DNN/RNN backend

In speech recognition, segmentation involves partitioning a continuous audio signal containing speech into smaller units or segments, such as words, phonemes, or syllables. This process is paramount in speech recognition systems, as it delineates the boundaries between distinct speech elements, facilitating subsequent analysis and processing. Segmentation accuracy significantly impacts speech recognition systems' overall precision and performance, enabling more precise identification and processing of individual speech units. Moreover, proper segmentation empowers the automatic speech recognition (ASR) system to distinguish between different syllables or words effectively, leading to more efficient speech recognition outcomes. This research paper investigates the importance of vowel detection for syllable segmentation in speech recognition, particularly in Arabic speech, such as the Quran, where changes in each syllable can alter the meaning. While existing techniques only consider pronunciation by different readers, this study employs onset detection to account for the presence of Arabic vowels. Specifically, the study focuses on detecting the onset of the recitation of Surah Al-Fatihah's fourth verse using 50 data sets in the syllable detection testing process. The results indicate that syllable detection performs excellently on syllables with /a/ and /i/ vowels. However, syllables with /u/ vowels produce results below 70%. The study suggests that the onset-based method is ideal for syllables with the presence of /a/and /i/ vowels, demonstrating the importance of considering Arabic vowel letters in speech recognition.

The evolution of robust speech recognition systems that maintain a high level of recognition accuracy in difficult and dynamically-varying acoustical environments is becoming increasingly important as speech recognition technology becomes a more integral part of mobile applications. In distributed speech recognition (DSR) architecture the recogniser's front-end is located in the terminal and is connected over a data network to a remote back-end recognition server. The terminal performs the feature parameter extraction, or the front-end of the speech recognition system. These features are transmitted over a data channel to the remote back-end recogniser. DSR provides particular benefits for the applications of mobile devices such as improved recognition performance compared to using the voice channel and ubiquitous access from different networks with a guaranteed level of recognition performance. A feature extraction algorithm integrated into the DSR system is required to operate in real-time as well as with the lowest possible computational costs.

Currently, the use of speech recognition is increaseingly in many applications such as mobile device interaction, interactive voice response system, voice search, voice dictation and voice identification. The heart of such applications is speech features needed to represent input signals. However, in real applications, speech signals are sampled with various sampling rates. The different sampling rates of input speech lead to the different features. This makes the speech recognition rate dropping. Therefore, this paper proposes an independent resolution descriptor based on fractal codes obtained by fractal encoding and decoding processes. The encoding process extracts fractal codes from partitioned speech signals, whereas the decoding process reconstructs independent resolution speech signals from the fractal codes. This method can effectively reconstruct speech signals at any sampling rates, especially at a higher sampling rate, which is a grand challenge. The proposed method is evaluated the performance by testing with AN4 corpus of CMU Sphinx speech recognition engine. The experimental results show that the proposed method can improve the accuracy of speech recognition, even if the sampling rate of testing speeches differs from that of training speeches.

This paper presents an analysis-by-synthesis approach for acoustic model adaptation. Using artificial speech data for speech recognition systems adaptation, has the potential to address the problem of data sparseness, to avoid speech recordings in real conditions and to provide the capability of performing large number of development cycles for Automatic Speech Recognition (ASR) systems in shorter time. The proposed adaptation framework uses unified ASR and synthesis system to produce artificial adaptation speech signals. In order to confirm the usability of the proposed approach, several experiments were performed where the artificial speech data was coded-decoded by different speech and waveform coders and the acoustic model used for synthesis was adapted for each coder. The recognition results show that the proposed method could be used successfully in the process of speech recognition systems performance assessment and improvement, not only for coded speech effects evaluation and adaptation, but also for other environment conditions.

This study focuses on the speech interaction experience of elderly users and proposes a speech recognition architecture, Con-DFSMN, which integrates a Convolution-augmented Transformer (Conformer) and Deep Feedforward Sequential Memory Network (DFSMN). The aim is to enhance recognition accuracy and system stability in variable and complex speech environments. The Additive Margin-Circle Loss (AM-Circle Loss) compound loss function is introduced, which effectively enhances the ability to distinguish class boundaries. This enables the model to exhibit excellent generalization performance when processing speech with high intra-class fluctuation and ambiguous boundaries. For the Tsinghua Chinese 30-hour Speech Corpus (THCHS-30) and DemantiaBank datasets, experimental results show that Con-DFSMN achieves accuracies of 95.3% and 91.8% in short speech recognition, respectively, representing a 13.1% improvement over traditional Transformer models. Con-DFSMN also maintains advantages in long speech recognition, with maximum accuracies of 90.8% and 87.3%. During training, Con-DFSMN demonstrates faster convergence speed and more stable loss decline, advancing 15 epochs earlier than Deepspeech2 and improving by approximately 25% compared to Transformer. The results verify the effectiveness and robustness of the model, demonstrating its adaptability to the speech features of the elderly, laying a foundation for the development of intelligent elderly speech interaction systems.

Speech Recognition is frequently cited as a potential remedy to distraction resulting from drivers' operation of in-vehicle devices. This position typically assumes that the introduction of speech recognition will result in reduced cognitive workload and improved driving performance. Past research neither fully supports nor fully discounts this assumption. However, it is difficult to compare many of these studies, due to differences in device operation tasks, the pacing of those tasks, speech recognition system performance, and system interface designs. In an effort to directly address the effect of voice recognition on driver distraction, the present authors developed a capability to manipulate the performance characteristics of a speech recognition system through a Wizard of Oz speech recognition system and installed this system in a simulated driving environment. The sensitivity of the simulated driving environment and speech recognition accuracy manipulation were evaluated in an initial study comparing driver cognitive workload and driving performance during self-paced simulated operation of a personal digital assistant (PDA) during no PDA use, manual control of the PDA, and speech control of the PDA. In the Speech PDA condition, speech recognition accuracy was varied between drivers. Analysis of drivers' emergency braking response times and rated cognitive workload revealed significantly lower cognitive demand and better performance in the No PDA condition when compared to the Manual PDA condition. The Speech PDA condition resulted in response times and rated cognitive workload levels that were between the No PDA and Manual PDA conditions, but not significantly different from either of these conditions. Further analysis of emergency braking performance revealed a non-significant trend towards better performance in conjunction with higher speech recognition accuracy levels. The potential for reducing driver distraction through the careful development and evaluation of speech recognition systems is discussed.

The accuracy of automatic speech recognition (ASR) systems is generally evaluated using corpora of grammatically sound read speech or natural spontaneous speech. This prohibits an accurate estimation of the performance of the acoustic modeling part of ASR because the language modeling performance is inherently integrated in the overall performance metric. In this work, ASR and human speech recognition (HSR) accuracies are compared for null grammar sentences in different signal-to-noise ratios and vocabulary sizes-1000, 2000, 4000, and 8000. The results shed light on differences between ASR and HSR in relative significance of bottom-up word recognition and context awareness.