Discriminative Training for Automatic Speech Recognition: Modeling, Criteria, Optimization, Implementation, and Performance

Semi-supervised and unsupervised discriminative language model training for automatic speech recognition

Improvement Comparison of Different Lattice-based Discriminative Training Methods in Chinese-monolingual and Chinese-English-bilingual Speech Recognition

The minimum classification error (MCE) framework for discriminative training is a simple and general formalism for directly optimizing recognition accuracy in pattern recognition problems. The framework applies directly to the optimization of hidden Markov models (HMMs) used for speech recognition problems. However, few if any studies have reported results for the application of MCE training to large-vocabulary, continuous-speech recognition tasks. This article reports significant gains in recognition performance and model compactness as a result of discriminative training based on MCE training applied to HMMs, in the context of three challenging large-vocabulary (up to 100 k word) speech recognition tasks: the Corpus of Spontaneous Japanese lecture speech transcription task, a telephone-based name recognition task, and the MIT Jupiter telephone-based conversational weather information task. On these tasks, starting from maximum likelihood (ML) baselines, MCE training yielded relative reductions in word error ranging from 7% to 20%. Furthermore, this paper evaluates the use of different methods for optimizing the MCE criterion function, as well as the use of precomputed recognition lattices to speed up training. An overview of the MCE framework is given, with an emphasis on practical implementation issues

In this paper, we propose a bilevel joint unsupervised and supervised training (BL-JUST) framework for automatic speech recognition. Compared to the conventional pre-training and fine-tuning strategy which is a disconnected two-stage process, BL-JUST tries to optimize an acoustic model such that it simultaneously minimizes both the unsupervised and supervised loss functions. Because BL-JUST seeks matched local optima of both loss functions, acoustic representations learned by the acoustic model strike a good balance between being generic and task-specific. We solve the BL-JUST problem using penalty-based bilevel gradient descent and evaluate the trained deep neural network acoustic models on various datasets with a variety of architectures and loss functions. We show that BL-JUST can outperform the widely-used pre-training and fine-tuning strategy and some other popular semi-supervised techniques.

Lattice segmentation and minimum Bayes risk discriminative training for large vocabulary continuous speech recognition

Discriminative training has been a leading factor for improving automatic speech recognition (ASR) performance over the last decade. The traditional discriminative training, however, has been aimed to minimize empirical error rates on training sets, which may not be well generalized to test sets. Many attempts have been made recently to incorporate the principle of large margin (PLM) into the training of hidden Markov models (HMMs) in ASR to improve the generalization abilities. Significant error rate reduction on the test sets has been observed on both small vocabulary and large vocabulary continuous ASR tasks using large-margin discriminative training (LMDT) techniques. In this paper, we introduce the PLM, define the concept of margin in the HMMs, and survey a number of popular LMDT algorithms proposed and developed recently. Specifically, we review and compare the large-margin minimum classification error (LM-MCE) estimation, soft-margin estimation (SME), large margin estimation (LME), large relative margin estimation (LRME), and large margin training (LMT) with a focus on the insights, the training criteria, the optimization techniques used, and the strengths and weaknesses of these different approaches. We suggest future research directions in our conclusion of this paper.

While dysarthric speech recognition can be a convenient interface for dysarthric speakers, it is hard to collect enough speech data to overcome the underestimation problem of acoustic models. In addition, there are lots of pronunciation variations in the collected database due to the paralysis of the articulator of dysarthric speakers. Thus, a discriminative training method is proposed for improving the performance of such resource-limited dysarthric speech recognition. The proposed method is applied to subspace Gaussian mixture modeling by incorporating pronunciation variations into a conventional minimum phone error discriminative training method.

Minimum classification error (MCE) based discriminative methods have been extensively studied and successfully applied to automatic speech recognition, speaker recognition, and utterance verification. Our goal is to modify the embedded string model based MCE algorithm to handle the case of a large number of cross-syllable trip hones used in a large vocabulary recognition system. A selective strategy about MCE based discriminative training method, in particular for Mandarin speech syllable loop recognition, is introduced. We use a syllable loop recognition task to evaluate the performance of the acoustic model of an established large vocabulary continuous speech recognition system. The basic idea is that since decoding errors occur in parts of the models in whole decoded sentences, it is reasonable to adjust the parameters of the wrong models. As a result, a weighted MCE formulation is derived, which can provide more effective convergence and about a 10% error rate reduction for a large training set. Our experiments show that, although the performance of the recognition system is improved, some original correct recognition results are misrecognized after discriminative training, and a divide and conquer strategy is proposed as a solution. Thus, the acoustic feature space is divided into two or more sub-spaces according to the discriminative training procedure. By combining these two methods, we obtain over 14.5% error rate reduction in syllable loop recognition.

Speech recognition can be defined as the process of converting voice signals into the ranks of the word, by applying a specific algorithm that is implemented in a computer program. The research of speech recognition in Indonesia is relatively limited. This paper has studied methods of feature extraction which is the best among the Linear Predictive Coding (LPC) and Mel Frequency Cepstral Coefficients (MFCC) for speech recognition in Indonesian language. This is important because the method can produce a high accuracy for a particular language does not necessarily produce the same accuracy for other languages, considering every language has different characteristics. Thus this research hopefully can help further accelerate the use of automatic speech recognition for Indonesian language. There are two main processes in speech recognition, feature extraction and recognition. The method used for comparison feature extraction in this study is the LPC and MFCC, while the method of recognition using Hidden Markov Model (HMM). The test results showed that the LPC method is better than MFCC in Indonesian language speech recognition.

For the past two decades, research in speech recognition has been intensively carried out worldwide, spurred on by advances in signal processing, algorithms, architectures, and hardware. Speech recognition systems have been developed for a wide variety of applications, ranging from small vocabulary keyword recognition over dial-up telephone lines, to medium size vocabulary voice interactive command and control systems on personal computers, to large vocabulary speech dictation, spontaneous speech understanding, and limited-domain speech translation. In this paper we review some of the key advances in several areas of automatic speech recognition. We also illustrate, by examples, how these key advances can be used for continuous speech recognition of Mandarin. Finally we elaborate the requirements in designing successful real-world applications and address technical challenges that need to be harnessed in order to reach the ultimate goal of providing an easy-to-use, natural, and flexible voice interface between people and machines.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

End-to-end automatic speech recognition (ASR) has become a popular alternative to traditional module-based systems, simplifying the model-building process with a single deep neural network architecture. However, the training of end-to-end ASR systems is generally data-hungry: a large amount of labeled data (speech-text pairs) is necessary to learn direct speech-to-text conversion effectively. To make the training less dependent on labeled data, pseudo-labeling, a semi-supervised learning approach, has been successfully introduced to end-to-end ASR, where a seed model is self-trained with pseudo-labels generated from unlabeled (speech-only) data. Here, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">momentum pseudo-labeling</i> (MPL), a simple yet effective strategy for semi-supervised ASR. MPL consists of a pair of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">online</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">offline</i> models that interact and learn from each other, inspired by the mean teacher method. The online model is trained to predict pseudo-labels generated on the fly by the offline model. The offline model maintains an exponential moving average of the online model parameters. The interaction between the two models allows better ASR training on unlabeled data by continuously improving the quality of pseudo-labels. We apply MPL to a connectionist temporal classification-based model and evaluate it on various semi-supervised scenarios with varying amounts of data or domain mismatch. The results demonstrate that MPL significantly improves the seed model by stabilizing the training on unlabeled data. Moreover, we present additional techniques, e.g., the use of Conformer and an external language model, to further enhance MPL, which leads to better performance than other semi-supervised methods based on pseudo-labeling.

Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if they are trained by recordings of single talkers. This paper proposes a multi-speaker ASR method that incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set was extracted as numeric vectors, and all of the embedding vectors were stacked to construct a total speaker profile matrix. The speaker profile matrix from the training dataset enables finding embedding vectors that are close to the speakers of the input recordings in the test conditions, and it helps to recognize the individual speakers’ voices mixed in the input. Furthermore, the proposed method efficiently reuses the decoder from the existing speaker-independent ASR model, eliminating the need for retraining the entire system. Various speaker embedding methods such as i-vector, d-vector, and x-vector were adopted, and the experimental results show 0.33% and 0.95% absolute (3.9% and 11.5% relative) improvements without and with the speaker profile in the word error rate (WER).

This paper describes how we can combine our previously proposed fast extended Baum-Welch algorithm and generalized discriminative feature transformation to achieve single pass discriminative training, which we only process the data once. Compared to the state of the art training procedure, which uses feature space maximum mutual information (fMMI) and boosted maximum mutual information (BMMI), our proposed training procedure can achieve around 80% of the improvement available from discriminative training. We also show that if we are allowed to process the data twice, it is possible to achieve almost all of the improvement. We evaluate different training procedures on various large scale tasks using Iraqi and modern standard Arabic speech recognition systems.

Open Access Journal of Neurology & Neurosurgery has been developed as an opportunity for the free and open international exchange of information regarding the progress in the clinical and neurosciences. OAJNN publishes original research, reviews, mini reviews, case reports