Canonical state models for automatic speech recognition

Subspace Gaussian mixture based language modeling for large vocabulary continuous speech recognition

In this paper, the improvements in the recently implemented Kannada speech recognition system is demonstrated in detail. The Kannada automatic speech recognition (ASR) system consists of ASR models which are created by using Kaldi, IVRS call flow and weather and agricultural commodity prices information databases. The task specific speech data used in the recently developed spoken dialogue system had high level of different background noises. The different types of noises present in collected speech data had an adverse effect on the on line and off line speech recognition performances. Therefore, to improve the speech recognition accuracy in Kannada ASR system, a noise reduction algorithm is developed which is a fusion of spectral subtraction with voice activity detection (SS-VAD) and minimum mean square error spectrum power estimator based on zero crossing (MMSE-SPZC) estimator. The noise elimination algorithm is added in the system before the feature extraction part. An alternative ASR models are created using subspace Gaussian mixture models (SGMM) and deep neural network (DNN) modeling techniques. The experimental results show that, the fusion of noise elimination technique and SGMM/DNN based modeling gives a better relative improvement of 7.68% accuracy compared to the recently developed GMM-HMM based ASR system. The least word error rate (WER) acoustic models could be used in spoken dialogue system. The developed spoken query system is tested from Karnataka farmers under uncontrolled environment.

This research introduces a comprehensive strategy to enhance the performance of an existing automatic speech recognition (ASR) model, which has been previously documented in published articles. The study sets out to achieve several objectives. Firstly, it concentrates on updating the ASR model by retraining it with new datasets. This involves integrating samples from the latest Common Voice corpus release and data collected independently via the armspeech.com web application. Another key focus lies in optimizing the ASR model for near-real-time processing, intending to improve its speed and efficiency. The proposed adjustments to the model’s architecture aim to balance accuracy and processing speed, which is essential for applications requiring prompt speech recognition. Furthermore, the research explores the integration of Transformer models into the post-processing pipeline to introduce punctuation and capitalization into the ASR output. This step not only enhances the linguistic quality of transcriptions but also improves their readability and usability. In tandem with these advancements, the research presents a systematic approach to gathering, annotating, and storing datasets specifically tailored for punctuation and capitalization tasks. The methodology outlines the acquisition and organization of a dataset conducive to training Transformer models for these linguistic tasks. This comprehensive approach, which encompasses dataset enrichment, architectural modifications, and post-processing enhancements, aims to elevate the ASR model’s accuracy, speed, and linguistic refinement, with a particular focus on addressing the intricacies of the Armenian language. The research contributes valuable insights into the optimization of ASR systems, tackling both language-specific challenges and broader issues related to linguistic post-processing.

Although over 300M around the world speak Bangla, scant work has been done in improving Bangla voice-to-text transcription due to Bangla being a low-resource language. However, with the introduction of the Bengali Common Voice 9.0 speech dataset, Automatic Speech Recognition (ASR) models can now be significantly improved. With 399hrs of speech recordings, Bengali Common Voice is the largest and most diversified open-source Bengali speech corpus in the world. In this paper, we outperform the State-of-the-Art (SOTA) pretrained Bengali ASR models by finetuning a pretrained wav2vec2 model on the common voice dataset. We also demonstrate how to significantly improve the performance of an ASR model by adding an n-gram language model as a post-processor. Finally, we do some experiments and hyperparameter tuning to generate a robust Bangla ASR model that is better than the existing ASR models.

The subspace Gaussian mixture model (SGMM) has been exploited for cross-lingual speech recognition. The general motivation is that the subspace parameters can be estimated on multiple source languages and then transferred to the target language. In this work, we investigate an extension to SGMM, referred to as subspace mixture model (SMM), in which subspace parameters on the target language are casted as a linear mixture of the subspaces derived from source languages. This approach reduces the number of SGMM model parameters, while retaining the flexibility of subspace learning on the target language. Experiments show that the proposed SMM method outperforms SGMM significantly when the target language has limited training data.

Error correction techniques have been used to refine the output sentences from automatic speech recognition (ASR) models and achieve a lower word error rate (WER). Previous works usually adopt end-to-end models and has strong dependency on Pseudo Paired Data and Original Paired Data. But when only pre-training on Pseudo Paired Data, previous models have negative effect on correction. While fine-tuning on Original Paired Data, the source side data must be transcribed by a well-trained ASR model, which takes a lot of time and not universal. In this paper, we propose UCorrect, an unsupervised Detector-Generator-Selector framework for ASR Error Correction. UCorrect has no dependency on the training data mentioned before. The whole procedure is first to detect whether the character is erroneous, then to generate some candidate characters and finally to select the most confident one to replace the error character. Experiments on the public AISHELL-1 dataset and WenetSpeech dataset show the effectiveness of UCorrect for ASR error correction: 1) it achieves significant WER reduction, achieves 6.83% even without fine-tuning and 14.29% after fine-tuning; 2) it outperforms the popular NAR correction models by a large margin with a competitive low latency; and 3) it is an universal method, as it reduces all WERs of the ASR model with different decoding strategies and reduces all WERs of ASR models trained on different scale datasets.

Pretrained multilingual End-to-End (E2E) Automatic Speech Recognition (ASR) models demonstrate remarkable capabilities but struggle with domain-specific terminologies and underrepresented dialects. Fine-tuning requires expensive paired audio-transcript data, creating barriers to practical adaptation. This paper investigates language model (LM) integration via shallow fusion as an efficient, text-only adaptation method. We introduce Confidence Gated Fusion (CGF), a novel approach that dynamically determines the LM weight during decoding based on ASR model uncertainty, eliminating the expensive validation-set-dependent grid search required by traditional shallow fusion. We validate our approach using OpenAI’s Whisper across multiple model sizes on Arabic ASR, evaluating Modern Standard Arabic (MSA), Egyptian dialect (EGY), and a specialized judiciary domain. Integrating a domain-specific LM achieved up to 40.92% relative WER reduction on dialectal speech and 32.96% on the judiciary domain. Our CGF method achieved comparable performance to tuned baselines while requiring no hyperparameter optimization, with particular advantages in specialized domains (28.92% relative WER reduction on judiciary) and on smaller models where static weights often lack robustness.

In the context of huge volume of data generated in the last couple of years and the need of processing audio stream in real time data processing flows, this paper aims to explore, identify and experiment Automatic Speech Recognition (ASR) models in the context of low latency applications depending on different contexts using specific data sets. Our work aims to explore whether if such models can be deployed in the real time stream audio workflow and for that we compare multiple size of the Whisper model for multiple data sets and analyze the results. Additionally, we tested the models for English and Romanian language. In the future, for the multi-language context it is highly important to analyze using model dedicated fine-tuned for specific language and compare with the results of multi-language models. This paper contributes to the field by providing evidence, comparison, statistics, level of accuracy in order to use ASR models in the context of low latency applications, exploring multiple versions of Whispers models and help researcher in dimensioning the environments for real time infrastructure and extend streaming capabilities with Artificial Intelligence models. Equally, the paper can used for IT professional in running critical systems in different industries to emphasize the techniques to be used in running ASR model in production.

This paper investigates the issues that are associated with applying speech production models to automatic speech recognition (ASR). Here the applicability of articulatory representations to ASR is considered independently of the role of articulatory representations in speech perception. While the question of whether it is necessary or even possible for human listeners to recover the state of the articulators during the process of perceiving speech is an important one, it is not considered here. Hence, the authors refrain from posing completely new paradigms for ASR which more closely parallel the relationship between speech production and human speech understanding. Instead, work aimed at integrating speech production models into existing ASR formalisms is described.

This article describes a density ratio approach to integrating external Language Models (LMs) into end-to-end models for Automatic Speech Recognition (ASR). Applied to a Recurrent Neural Network Transducer (RNN-T) ASR model trained on a given domain, a matched in-domain RNN-LM, and a target domain RNN-LM, the proposed method uses Bayes' Rule to define RNN-T posteriors for the target domain, in a manner directly analogous to the classic hybrid model for ASR based on Deep Neural Networks (DNNs) or LSTMs in the Hidden Markov Model (HMM) framework (Bourlard & Morgan, 1994). The proposed approach is evaluated in cross-domain and limited-data scenarios, for which a significant amount of target domain text data is used for LM training, but only limited (or no) {audio, transcript} training data pairs are used to train the RNN-T. Specifically, an RNN-T model trained on paired audio & transcript data from YouTube is evaluated for its ability to generalize to Voice Search data. The Density Ratio method was found to consistently outperform the dominant approach to LM and end-to-end ASR integration, Shallow Fusion.

This article concerns comparative studies on the Automatic Speech Recognition (ASR) model incorporated with the Large Language Model (LLM) used for medical interviews. The proposed solution is tested on polish language benchmarks and dataset with medical interviews. The latest ASR technologies are based on convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. Most of them work as end-to-end solutions. The presented approach in the case of the Whisper model shows a two-stage solution with End-To-End ASR and LLM working together in a pipeline. The ASR output is an input for LLM. The LLM is a component by which the output from ASR is corrected and improved. Comparative studies for automatic recognition of the Polish language between modern End-To-End deep learning architectures and the ASR hybrid model were performed. The medical interview tests were performed with two state-of-the-art ASR models: OpenAI Whisper incorporated with LLM and Scribe ElevenLabs. Additionally, the results were compared with five more end-to-end models (QuartzNet, FastConformer, Wav2Vec 2.0 XLSR and ESPnet Model Zoo) on Mozilla Common Voice and VoxPopuli databases. Tests were conducted for clean audio signal, signal with bandwidth limitation, and degraded. The tested models were evaluated on the basis of Word Error Rate (WER) and Character Error Rate (CER). The results show that the Whisper model performs by far the best among the open-source models. ElevenLabs Scribe model, on the other hand, performs best for Polish on both general benchmark and medical data.

This paper concerns cross-lingual acoustic modeling in the case when there are limited target language resources. We build on an approach in which a subspace Gaussian mixture model (SGMM) is adapted to the target language by reusing the globally shared parameters estimated from out-of-language training data. In current cross-lingual systems, these parameters are fixed when training the target system, which can give rise to a mismatch between the source and target systems. We investigate a maximum a posteriori (MAP) adaptation approach to alleviate the potential mismatch. In particular, we focus on the adaptation of phonetic subspace parameters using a matrix variate Gaussian prior distribution. Experiments on the GlobalPhone corpus using the MAP adaptation approach results in word error rate reductions, compared with the cross-lingual base-line systems and systems updated using maximum likelihood, for training conditions with 1 hour and 5 hours of target language data.

While we have seen significant advances in automatic summarization for text, research on speech summarization is still limited. In this work, we address the challenge of automatically generating teasers for TED talks. In the first step, we create a corpus for automatic summarization of TED and TEDx talks consisting of the talks' recording, their transcripts and their descriptions. The corpus is used to build a speech summarization system for the task. We adapt and combine pre-trained models for automatic speech recognition (ASR) and text summarization using the collected data. This initial work shows that is more important to adapt the summarization model to the ASR transcripts than to adapt the ASR model to the talks.

Since 1972, the Speech Recognition Group at IBM's Thomas J. Watson Research Center has worked on a system for the automatic recognition of natural continuous speech. The goal of our research is to produce a voice-activated text-processing system. We have recently worked on a 5,000-word isolated-speech system for recognition of business correspondence and interoffice memoranda.Although speech recognition is performed with beguiling ease by human beings, the task of transferring this ability to computers has proven quite difficult. Our approach to the problem has been to develop general statistical techniques that allow the recognition system to organize itself when presented with a large sample of speech called training data. In essence, our system refines itself from some crudely specified initial configuration so as to take advantage of regularities present in a speech sample.Suppose that a talker wishes to convey to a listener the string of words W1, W2, ..., Wn. Each of the words has a pronunciation, in accordance with which the talker will guide his/her speech apparatus through an intricate cascade of gestures, with the result that a series of sound pressure waves will emanate. A microphone placed within the influence of these waves will respond to them in a fashion that is characteristic of the word sequence W1, W2, ..., Wn.Let the response of the microphone be denoted by the sequence A1, ..., Am. We can regard A1, ..., Am as a garbled version of the intended message W1, ..., Wn. The problem of recovering a message from a garbled transmission is the province of communication theory. The problem may be stated as follows: for a garbled transmission A (=A1, ...Am), determine the message W (=W1, ..., Wn) that is most probable given A. That is, find W such that Pr(W|A) is as large as possible.From Bayes's rule, we have Pr(W|A) = {Pr(W|A)}Pr(W)}Pr(A). Since we are interested in maximizing this for a particular fixed A, we can ignore the denominator on the right. The first term in the numerator constitutes a probabilistic characterization of the transmission channel. It tells us what A's to expect when the talker intends a particular word string W. The second term in the numerator is a probabilistic characterization of the word strings that the talker desires to convey. Pr(W) embodies all rules of grammar (ungrammatical word strings are less probable than grammatical ones) as well as all semantic constraints (meaningless word strings are less probable than meaningful ones).It is, of course, impossible to compute these probabilities exactly. We have, however, developed methods for obtaining useful approximations to them. We refer to the approximation of Pr(A|W) as a statistical model for the speech production process and to the approximation of Pr(W) as a statistical model for the language. A discussion of these models constitutes the body of the talk.

In this letter, a spoken dialogue system is demonstrated to disseminate the agricultural commodity price information in Kannada language/dialects. The spoken dialogue system mainly includes interactive voice response system (IVRS), automatic speech recognition (ASR) models and price information database. The IVRS call flow is used for speech data collection from the farmers across various dialect regions of Karnataka. The collected speech data is transcribed and validated using Indic transliteration tool (IT3: UTF-8). The dictionary and corresponding phoneme set is created for Kannada language. The ASR acoustic models are created using the speech recognition toolkit called Kaldi. The 85% and 15% of transcribed and validated speech corpus is used for Kaldi system training and decoding respectively. The agricultural marketing network (AGMARKNET) website is used as the price information database for different agricultural commodities. The postgresql database is used for managing the crawled data. The word error rates (WERs) of ASR models for districts, mandis, commodities and mixed data are discussed and the spoken dialogue system is developed and tested from the real farmers to access the real time agricultural commodity price information in language/dialects of Kannada.