A minimum classification error (MCE) framework for generalized linear classifier in machine learning for text categorization/retrieval

Discriminative training aims at constructing a classifier that is of small scale but has high classification power. One type, Minimum Classification Error (MCE) training, has been used widely in pattern recognition, especially in the speech recognition field. In parallel with this, Relevance Vector Machine (RVM) has attracted many researchers' interest, based on its potential for alleviating the scalability problem of Support Vector Machine (SVM). It has been reported that RVM achieves high classification accuracy with a limited amount of classifier parameters, i.e., relevance vectors. Comparison studies between MCE training and SVM have been done, but not so much between MCE training and RVM. Motivated by this, we conduct theoretical and experimental comparisons of MCE training and RVM. Results show that MCE training is better suited to the development of small-scale but highly discriminative classifiers than its counterpart.

In our previous work, we have proposed a speaker modeling technique using spectral and pitch features for text-independent speaker identification based on Multi-Space Probability Distribution Gaussian Mixture Models (MSD-GMMs). We have presented a maximum likelihood (ML) estimation procedure for the MSD-GMM parameters and demonstrated its high recognition performance. In this paper, we describe an minimum classification error (MCE) training procedure for the MSDGMM speaker models. MCE training is also applied to automatically estimate mixture-dependent stream weights for spectral and pitch streams. The MCE-based MSD-GMM speaker models are evaluated for a text-independent speaker identification task. Experimental results show that MCE training of the MSD-GMM parameters significantly reduces identification errors and system performance is further improved by appropriately weighting spectral and pitch streams using MCE training.

The minimum classification error (MCE) framework for discriminative training is a simple and general formalism for directly optimizing recognition accuracy in pattern recognition problems. The framework applies directly to the optimization of hidden Markov models (HMMs) used for speech recognition problems. However, few if any studies have reported results for the application of MCE training to large-vocabulary, continuous-speech recognition tasks. This article reports significant gains in recognition performance and model compactness as a result of discriminative training based on MCE training applied to HMMs, in the context of three challenging large-vocabulary (up to 100 k word) speech recognition tasks: the Corpus of Spontaneous Japanese lecture speech transcription task, a telephone-based name recognition task, and the MIT Jupiter telephone-based conversational weather information task. On these tasks, starting from maximum likelihood (ML) baselines, MCE training yielded relative reductions in word error ranging from 7% to 20%. Furthermore, this paper evaluates the use of different methods for optimizing the MCE criterion function, as well as the use of precomputed recognition lattices to speed up training. An overview of the MCE framework is given, with an emphasis on practical implementation issues

This paper considers both the garbage modelling and the discriminative training of a speech recognition system consisting of a multilayer perceptron (MLP) network and hidden Markov models (HMMs). The string level training of the system is based on Minimum Classification Error (MCE) algorithm using especially the ideas recently proposed by Reichl et al. In order to improve the MCE training scheme, they defined a derivative for MCE cost function putting heavier weight to the misclassified utterances than the traditional translated sigmoidal. In this paper, we give a definition of a new cost function, whose derivative has similar properties as the one proposed by Reichl et al. The new and the sigmoidal cost function are compared with the string level (MCE) algorithm. The string recognition rates show that both methods achieve equal performances, but the convergence of the MCE algorithm is a bit faster with the new cost function. Moreover, the performance of two garbage models, the nth best and an interpolative garbage model, are also compared in this paper. The latter one is an approximation of the former one requiring less operations, but achieving comparable performance. The recognition system achieved 93.32% accuracy for test set at best. The test set contained 29188 Finnish digit strings from two environments.

Keyword spotting can be formulated as a non-uniform error automatic speech recognition (ASR) problem. It has been demonstrated [1] that this new formulation with the nonuniform MCE training technique can lead to improved system performance in keyword spotting applications. In this paper, we demonstrate that deep neural networks (DNNs) can be successfully trained on the non-uniform minimum classification error (MCE) criterion which weighs the errors on keywords much more significantly than those on non-keywords in an ASR task. The integration with a DNN-HMM system enables modeling of multi-frame distributions, which conventional systems find difficult to accomplish. To further improve the performance, more confusable data is generated by boosting the likelihood of the sentences that have more errors. The keyword spotting system is implemented within a weighted finite state transducer (WFST) framework and the DNN is optimized using standard backpropagation and stochastic gradient decent. We evaluate the performance of the proposed framework on a large vocabulary spontaneous conversational telephone speech dataset (Switchboard-1 Release 2). The proposed approach achieves an absolute figure of merit improvement of 3.65% over the baseline system.

A new look at discriminative training for hidden Markov models

In this paper we compare three frameworks for discriminative training of continuous-density hidden Markov models (CD-HMMs). Specifically, we compare two popular frameworks, based on conditional maximum likelihood (CML) and minimum classification error (MCE), to a new framework based on margin maximization. Unlike CML and MCE, our formulation of large margin training explicitly penalizes incorrect decodings by an amount proportional to the number of mislabeled hidden states. It also leads to a convex optimization over the parameter space of CD-HMMs, thus avoiding the problem of spurious local minima. We used discriminatively trained CD-HMMs from all three frameworks to build phonetic recognizers on the TIMIT speech corpus. The different recognizers employed exactly the same acoustic front end and hidden state space, thus enabling us to isolate the effect of different cost functions, parameterizations, and numerical optimizations. Experimentally, we find that our framework for large margin training yields significantly lower error rates than both CML and MCE training.

In this paper, we present an approach of detecting speech presence for which the decision rule is based on a combination of multiple features using a sigmoid function. A minimum classification error (MCE) training is used to update the weights adjustment for the combination. The features, consisting of three parameters: the ratio of ZCR, the spectral energy, and spectral entropy, are combined linearly with weights derived from the sub-band domain. First, the Bark-scale wavelet decomposition (BSWD) is used to split the input speech into 24 critical sub-bands. Next, the feature parameters are derived from the selected frequency sub-band to form robust voice feature parameters. In order to discard the seriously corrupted frequency sub-band, a strategy of adaptive frequency sub-band extraction (AFSE) dependant on the sub-band SNR is then applied to only the frequency sub-band used. Finally, these three feature parameters, which only consider the useful sub-band, are combined through a sigmoid type function incorporating optimal weights based on MSE training to detect either a speech present frame or a speech absent frame. Experimental results show that the performance of the proposed algorithm is superior to the standard methods such as G.729B and AMR2.

Dimensionality reduction is an important problem in pattern recognition. In a speech recognition system, the size of the feature set is normally large in the order of 40. Therefore, it is necessary to reduce the dimensionality of the feature space for efficient and effective speech recognition. Two popular methods to reduce the dimensionality of the feature space are linear discriminat analysis (LDA) and principal component analysis (PCA). This paper uses the minimum error classification (MCE) training algorithm for dimensionality reduction and presents an alternative MCE training algorithm that performs better on testing data than the conventional MCE training algorithm. The effects of the initial value of the transformation matrix on the performance of MCE have also been studied.

In the past several years, we've been developing a high performance OCR engine for machine printed Chinese/ English documents. We have reported previously (1) how to use character modeling techniques based on MCE (minimum classification error) training to achieve the high recognition accuracy, and (2) how to use confidence-guided progressive search and fast match techniques to achieve the high recognition efficiency. In this paper, we present two more techniques that help reduce search errors and improve the robustness of our character recognizer. They are (1) to use MCE-trained character-pair models to avoid error-prone character-level segmentation for some trouble cases, and (2) to perform a MCE-based negative training to improve the rejection capability of the recognition models on the hypothesized garbage images during recognition process. The efficacy of the proposed techniques is confirmed by experiments in a benchmark test.

A derivation of minimum classification error from the theoretical classification risk using Parzen estimation

Minimum classification error (MCE) based discriminative methods have been extensively studied and successfully applied to automatic speech recognition, speaker recognition, and utterance verification. Our goal is to modify the embedded string model based MCE algorithm to handle the case of a large number of cross-syllable trip hones used in a large vocabulary recognition system. A selective strategy about MCE based discriminative training method, in particular for Mandarin speech syllable loop recognition, is introduced. We use a syllable loop recognition task to evaluate the performance of the acoustic model of an established large vocabulary continuous speech recognition system. The basic idea is that since decoding errors occur in parts of the models in whole decoded sentences, it is reasonable to adjust the parameters of the wrong models. As a result, a weighted MCE formulation is derived, which can provide more effective convergence and about a 10% error rate reduction for a large training set. Our experiments show that, although the performance of the recognition system is improved, some original correct recognition results are misrecognized after discriminative training, and a divide and conquer strategy is proposed as a solution. Thus, the acoustic feature space is divided into two or more sub-spaces according to the discriminative training procedure. By combining these two methods, we obtain over 14.5% error rate reduction in syllable loop recognition.

In recent years, statistical modeling approaches have steadily gained in popularity in the field of information retrieval. This article presents an HMM/N-gram-based retrieval approach for Mandarin spoken documents. The underlying characteristics and the various structures of this approach were extensively investigated and analyzed. The retrieval capabilities were verified by tests with word- and syllable-level indexing features and comparisons to the conventional vector-space model approach. To further improve the discrimination capabilities of the HMMs, both the expectation-maximization (EM) and minimum classification error (MCE) training algorithms were introduced in training. Fusion of information via indexing word- and syllable-level features was also investigated. The spoken document retrieval experiments were performed on the Topic Detection and Tracking Corpora (TDT-2 and TDT-3). Very encouraging retrieval performance was obtained.

In this work, we present a complete framework of discriminative training using non-uniform criteria for keyword spotting, adaptive boosted non-uniform minimum classification error (MCE) for keyword spotting on spontaneous speech. To further boost the spotting performance and tackle the potential issue of over-training in the non-uniform MCE proposed in our prior work, we make two improvements to the fundamental MCE optimization procedure. Furthermore, motivated by AdaBoost, we introduce an adaptive scheme to embed error cost functions together with model combinations during the decoding stage. The proposed framework is comprehensively validated on two challenging large-scale spontaneous conversational telephone speech (CTS) tasks in different languages (English and Mandarin) and the experimental results show it can achieve significant and consistent figure of merit (FOM) gains over both ML and discriminatively trained systems.

In this paper, a framework of minimum classification error (MCE) model adaptation for continuous density HMMs is proposed based on the approach of super string model. We show that the error rate minimization in the proposed approach can be formulated into maximizing a special ratio of two positive functions, and from that a general growth transform algorithm is derived for MCE based model adaptation. This algorithm departs from the generalized probability descent (GPD) algorithm, and it is well suited for model adaptation with a small amount of training data. The proposed approach is applied to linear regression based variance adaptation, and the close form solution for variance adaptation using MCE linear regression (MCELR) is derived. The MCELR approach is evaluated on large vocabulary speech recognition tasks. The relative performance gain is more than doubled on the standard (WSJ Spoke 3) database, comparing to maximum likelihood linear regression (MLLR) based variance adaptation for the same amount of adaptation data.