Using minimum classification error training in dimensionality reduction

Discriminative training aims at constructing a classifier that is of small scale but has high classification power. One type, Minimum Classification Error (MCE) training, has been used widely in pattern recognition, especially in the speech recognition field. In parallel with this, Relevance Vector Machine (RVM) has attracted many researchers' interest, based on its potential for alleviating the scalability problem of Support Vector Machine (SVM). It has been reported that RVM achieves high classification accuracy with a limited amount of classifier parameters, i.e., relevance vectors. Comparison studies between MCE training and SVM have been done, but not so much between MCE training and RVM. Motivated by this, we conduct theoretical and experimental comparisons of MCE training and RVM. Results show that MCE training is better suited to the development of small-scale but highly discriminative classifiers than its counterpart.

The minimum classification error (MCE) framework for discriminative training is a simple and general formalism for directly optimizing recognition accuracy in pattern recognition problems. The framework applies directly to the optimization of hidden Markov models (HMMs) used for speech recognition problems. However, few if any studies have reported results for the application of MCE training to large-vocabulary, continuous-speech recognition tasks. This article reports significant gains in recognition performance and model compactness as a result of discriminative training based on MCE training applied to HMMs, in the context of three challenging large-vocabulary (up to 100 k word) speech recognition tasks: the Corpus of Spontaneous Japanese lecture speech transcription task, a telephone-based name recognition task, and the MIT Jupiter telephone-based conversational weather information task. On these tasks, starting from maximum likelihood (ML) baselines, MCE training yielded relative reductions in word error ranging from 7% to 20%. Furthermore, this paper evaluates the use of different methods for optimizing the MCE criterion function, as well as the use of precomputed recognition lattices to speed up training. An overview of the MCE framework is given, with an emphasis on practical implementation issues

In our previous work, we have proposed a speaker modeling technique using spectral and pitch features for text-independent speaker identification based on Multi-Space Probability Distribution Gaussian Mixture Models (MSD-GMMs). We have presented a maximum likelihood (ML) estimation procedure for the MSD-GMM parameters and demonstrated its high recognition performance. In this paper, we describe an minimum classification error (MCE) training procedure for the MSDGMM speaker models. MCE training is also applied to automatically estimate mixture-dependent stream weights for spectral and pitch streams. The MCE-based MSD-GMM speaker models are evaluated for a text-independent speaker identification task. Experimental results show that MCE training of the MSD-GMM parameters significantly reduces identification errors and system performance is further improved by appropriately weighting spectral and pitch streams using MCE training.

This paper considers both the garbage modelling and the discriminative training of a speech recognition system consisting of a multilayer perceptron (MLP) network and hidden Markov models (HMMs). The string level training of the system is based on Minimum Classification Error (MCE) algorithm using especially the ideas recently proposed by Reichl et al. In order to improve the MCE training scheme, they defined a derivative for MCE cost function putting heavier weight to the misclassified utterances than the traditional translated sigmoidal. In this paper, we give a definition of a new cost function, whose derivative has similar properties as the one proposed by Reichl et al. The new and the sigmoidal cost function are compared with the string level (MCE) algorithm. The string recognition rates show that both methods achieve equal performances, but the convergence of the MCE algorithm is a bit faster with the new cost function. Moreover, the performance of two garbage models, the nth best and an interpolative garbage model, are also compared in this paper. The latter one is an approximation of the former one requiring less operations, but achieving comparable performance. The recognition system achieved 93.32% accuracy for test set at best. The test set contained 29188 Finnish digit strings from two environments.

Keyword spotting can be formulated as a non-uniform error automatic speech recognition (ASR) problem. It has been demonstrated [1] that this new formulation with the nonuniform MCE training technique can lead to improved system performance in keyword spotting applications. In this paper, we demonstrate that deep neural networks (DNNs) can be successfully trained on the non-uniform minimum classification error (MCE) criterion which weighs the errors on keywords much more significantly than those on non-keywords in an ASR task. The integration with a DNN-HMM system enables modeling of multi-frame distributions, which conventional systems find difficult to accomplish. To further improve the performance, more confusable data is generated by boosting the likelihood of the sentences that have more errors. The keyword spotting system is implemented within a weighted finite state transducer (WFST) framework and the DNN is optimized using standard backpropagation and stochastic gradient decent. We evaluate the performance of the proposed framework on a large vocabulary spontaneous conversational telephone speech dataset (Switchboard-1 Release 2). The proposed approach achieves an absolute figure of merit improvement of 3.65% over the baseline system.

Linear discriminant analysis (LDA) has long been used to derive data-driven temporal filters in order to improve the robustness of speech features used in speech recognition. In this paper, we proposed the use of new optimization criteria of principal component analysis (PCA) and the minimum classification error (MCE) for constructing the temporal filters. Detailed comparative performance analysis for the features obtained using the three optimization criteria, LDA, PCA, and MCE, with various types of noise and a wide range of SNR values is presented. It was found that the new criteria lead to superior performance over the original MFCC features, just as LDA-derived filters can. In addition, the newly proposed MCE-derived filters can often do better than the LDA-derived filters. Also, it is shown that further performance improvements are achievable if any of these LDA/PCA/MCE-derived filters are integrated with the conventional approach of cepstral mean and variance normalization (CMVN). The performance improvements obtained in recognition experiments are further supported by analyses conducted using two different distance measures.

In this paper, we present the theoretical framework of minimum classification error (MCE) training of generalized linear classifiers for text classification. We show that many important text classifiers, either probabilistic or non-probabilistic, can be unified under this framework, and the proposed MCE classifier training approach can be applied to improve the classifier performance. In addition, we describe an effective MCE classifier training algorithm that uses AdaBoost to generate alternative initial classifiers, as opposed to combining multiple classifiers as it is typically used. This method is applied to MCE classifier training to overcome local minimums in optimal classifier parameter search, utilizing the fact that the family of generalized linear classifiers is closed under AdaBoost. Moreover, we extend the loss function in MCE training to incorporate training sample prior distributions to compensate the imbalanced training data distribution in each category. Experimental studies are performed on the text classification tasks, and the significant classification error reductions of 25% - 55% are observed.

A new look at discriminative training for hidden Markov models

Neural network tree (NNTree) is a hybrid model for machine learning. So far, we have proposed an efficient algorithm for inducing NNTrees, and verified through experiments that NNTrees are efficient and effective for solving different pattern recognition problems. However, for problems like text categorization, induction of NNTrees can be very computationally expensive. To solve this problem, we have tried to induce NNTrees after dimensionality reduction. Specifically, we have studied the linear discriminant analysis (LDA) based approach, the principal component analysis (PCA) based approach, and the direct centroid (DC) based approach. Results show that DC is simple but not effective; and LDA performs better but the computational cost for finding the transformation matrix is very high. To solve the problem more efficiently, we propose in this paper the discriminant multiple centroid (DMC) approach. Actually, DMC is a two-stage approach, in which all data are first mapped to a lower dimensional space based on the centroids, and the LDA is then conducted in the mapped space. Experimental results obtained for three public text datasets show that in all cases DMC is much faster than LDA without significant degradation.

Improved PCA + LDA Applies to Gastric Cancer Image Classification Process

Vector quantization (VQ) has been explored in the past as a means of reducing likelihood computation in speech recognizers which use hidden Markov models (HMMs) containing Gaussian output densities. Although this approach has proved successful, there is an extent beyond which further reduction in likelihood computation substantially degrades the recognition accuracy. Since the components of the VQ frontend are typically designed after model training is complete, this degradation can be attributed to the fact that VQ and HMM parameters are not jointly estimated. In order to restore the accuracy of a recognizer using VQ to aggressively reduce computation, joint estimation is necessary. We propose a technique which couples VQ frontend design with minimum classification error training. We demonstrate on a large vocabulary subword task that in certain cases, our joint training algorithm can reduce the string error rate by 79% compared to that of VQ mixture selection alone.

In this paper we compare three frameworks for discriminative training of continuous-density hidden Markov models (CD-HMMs). Specifically, we compare two popular frameworks, based on conditional maximum likelihood (CML) and minimum classification error (MCE), to a new framework based on margin maximization. Unlike CML and MCE, our formulation of large margin training explicitly penalizes incorrect decodings by an amount proportional to the number of mislabeled hidden states. It also leads to a convex optimization over the parameter space of CD-HMMs, thus avoiding the problem of spurious local minima. We used discriminatively trained CD-HMMs from all three frameworks to build phonetic recognizers on the TIMIT speech corpus. The different recognizers employed exactly the same acoustic front end and hidden state space, thus enabling us to isolate the effect of different cost functions, parameterizations, and numerical optimizations. Experimentally, we find that our framework for large margin training yields significantly lower error rates than both CML and MCE training.

In recent years, statistical modeling approaches have steadily gained in popularity in the field of information retrieval. This article presents an HMM/N-gram-based retrieval approach for Mandarin spoken documents. The underlying characteristics and the various structures of this approach were extensively investigated and analyzed. The retrieval capabilities were verified by tests with word- and syllable-level indexing features and comparisons to the conventional vector-space model approach. To further improve the discrimination capabilities of the HMMs, both the expectation-maximization (EM) and minimum classification error (MCE) training algorithms were introduced in training. Fusion of information via indexing word- and syllable-level features was also investigated. The spoken document retrieval experiments were performed on the Topic Detection and Tracking Corpora (TDT-2 and TDT-3). Very encouraging retrieval performance was obtained.

Combined speech enhancement and auditory modelling for robust distributed speech recognition

Minimum classification error (MCE) based discriminative methods have been extensively studied and successfully applied to automatic speech recognition, speaker recognition, and utterance verification. Our goal is to modify the embedded string model based MCE algorithm to handle the case of a large number of cross-syllable trip hones used in a large vocabulary recognition system. A selective strategy about MCE based discriminative training method, in particular for Mandarin speech syllable loop recognition, is introduced. We use a syllable loop recognition task to evaluate the performance of the acoustic model of an established large vocabulary continuous speech recognition system. The basic idea is that since decoding errors occur in parts of the models in whole decoded sentences, it is reasonable to adjust the parameters of the wrong models. As a result, a weighted MCE formulation is derived, which can provide more effective convergence and about a 10% error rate reduction for a large training set. Our experiments show that, although the performance of the recognition system is improved, some original correct recognition results are misrecognized after discriminative training, and a divide and conquer strategy is proposed as a solution. Thus, the acoustic feature space is divided into two or more sub-spaces according to the discriminative training procedure. By combining these two methods, we obtain over 14.5% error rate reduction in syllable loop recognition.