Comparison between Minimum Classification Error training and Relevance Vector Machine

In our previous work, we have proposed a speaker modeling technique using spectral and pitch features for text-independent speaker identification based on Multi-Space Probability Distribution Gaussian Mixture Models (MSD-GMMs). We have presented a maximum likelihood (ML) estimation procedure for the MSD-GMM parameters and demonstrated its high recognition performance. In this paper, we describe an minimum classification error (MCE) training procedure for the MSDGMM speaker models. MCE training is also applied to automatically estimate mixture-dependent stream weights for spectral and pitch streams. The MCE-based MSD-GMM speaker models are evaluated for a text-independent speaker identification task. Experimental results show that MCE training of the MSD-GMM parameters significantly reduces identification errors and system performance is further improved by appropriately weighting spectral and pitch streams using MCE training.

The minimum classification error (MCE) framework for discriminative training is a simple and general formalism for directly optimizing recognition accuracy in pattern recognition problems. The framework applies directly to the optimization of hidden Markov models (HMMs) used for speech recognition problems. However, few if any studies have reported results for the application of MCE training to large-vocabulary, continuous-speech recognition tasks. This article reports significant gains in recognition performance and model compactness as a result of discriminative training based on MCE training applied to HMMs, in the context of three challenging large-vocabulary (up to 100 k word) speech recognition tasks: the Corpus of Spontaneous Japanese lecture speech transcription task, a telephone-based name recognition task, and the MIT Jupiter telephone-based conversational weather information task. On these tasks, starting from maximum likelihood (ML) baselines, MCE training yielded relative reductions in word error ranging from 7% to 20%. Furthermore, this paper evaluates the use of different methods for optimizing the MCE criterion function, as well as the use of precomputed recognition lattices to speed up training. An overview of the MCE framework is given, with an emphasis on practical implementation issues

In this paper, we present the theoretical framework of minimum classification error (MCE) training of generalized linear classifiers for text classification. We show that many important text classifiers, either probabilistic or non-probabilistic, can be unified under this framework, and the proposed MCE classifier training approach can be applied to improve the classifier performance. In addition, we describe an effective MCE classifier training algorithm that uses AdaBoost to generate alternative initial classifiers, as opposed to combining multiple classifiers as it is typically used. This method is applied to MCE classifier training to overcome local minimums in optimal classifier parameter search, utilizing the fact that the family of generalized linear classifiers is closed under AdaBoost. Moreover, we extend the loss function in MCE training to incorporate training sample prior distributions to compensate the imbalanced training data distribution in each category. Experimental studies are performed on the text classification tasks, and the significant classification error reductions of 25% - 55% are observed.

This paper considers both the garbage modelling and the discriminative training of a speech recognition system consisting of a multilayer perceptron (MLP) network and hidden Markov models (HMMs). The string level training of the system is based on Minimum Classification Error (MCE) algorithm using especially the ideas recently proposed by Reichl et al. In order to improve the MCE training scheme, they defined a derivative for MCE cost function putting heavier weight to the misclassified utterances than the traditional translated sigmoidal. In this paper, we give a definition of a new cost function, whose derivative has similar properties as the one proposed by Reichl et al. The new and the sigmoidal cost function are compared with the string level (MCE) algorithm. The string recognition rates show that both methods achieve equal performances, but the convergence of the MCE algorithm is a bit faster with the new cost function. Moreover, the performance of two garbage models, the nth best and an interpolative garbage model, are also compared in this paper. The latter one is an approximation of the former one requiring less operations, but achieving comparable performance. The recognition system achieved 93.32% accuracy for test set at best. The test set contained 29188 Finnish digit strings from two environments.

Dimensionality reduction is an important problem in pattern recognition. In a speech recognition system, the size of the feature set is normally large in the order of 40. Therefore, it is necessary to reduce the dimensionality of the feature space for efficient and effective speech recognition. Two popular methods to reduce the dimensionality of the feature space are linear discriminat analysis (LDA) and principal component analysis (PCA). This paper uses the minimum error classification (MCE) training algorithm for dimensionality reduction and presents an alternative MCE training algorithm that performs better on testing data than the conventional MCE training algorithm. The effects of the initial value of the transformation matrix on the performance of MCE have also been studied.

Keyword spotting can be formulated as a non-uniform error automatic speech recognition (ASR) problem. It has been demonstrated [1] that this new formulation with the nonuniform MCE training technique can lead to improved system performance in keyword spotting applications. In this paper, we demonstrate that deep neural networks (DNNs) can be successfully trained on the non-uniform minimum classification error (MCE) criterion which weighs the errors on keywords much more significantly than those on non-keywords in an ASR task. The integration with a DNN-HMM system enables modeling of multi-frame distributions, which conventional systems find difficult to accomplish. To further improve the performance, more confusable data is generated by boosting the likelihood of the sentences that have more errors. The keyword spotting system is implemented within a weighted finite state transducer (WFST) framework and the DNN is optimized using standard backpropagation and stochastic gradient decent. We evaluate the performance of the proposed framework on a large vocabulary spontaneous conversational telephone speech dataset (Switchboard-1 Release 2). The proposed approach achieves an absolute figure of merit improvement of 3.65% over the baseline system.

A new look at discriminative training for hidden Markov models

In this paper we compare three frameworks for discriminative training of continuous-density hidden Markov models (CD-HMMs). Specifically, we compare two popular frameworks, based on conditional maximum likelihood (CML) and minimum classification error (MCE), to a new framework based on margin maximization. Unlike CML and MCE, our formulation of large margin training explicitly penalizes incorrect decodings by an amount proportional to the number of mislabeled hidden states. It also leads to a convex optimization over the parameter space of CD-HMMs, thus avoiding the problem of spurious local minima. We used discriminatively trained CD-HMMs from all three frameworks to build phonetic recognizers on the TIMIT speech corpus. The different recognizers employed exactly the same acoustic front end and hidden state space, thus enabling us to isolate the effect of different cost functions, parameterizations, and numerical optimizations. Experimentally, we find that our framework for large margin training yields significantly lower error rates than both CML and MCE training.

We present a probabilistic ranking-driven classifier for the detection of video semantic concept, such as airplane, building, etc. Most existing concept detection systems utilize Support Vector Machines (SVM) to perform the detection and ranking of retrieved video shots. However, the margin maximization principle of SVM does not perform ranking optimization but merely classification error minimization. To tackle this problem, we exploit the sparse Bayesian kernel model, namely the relevance vector machine (RVM), as the classifier for semantic concept detection. Based on automatic relevance determination principle, RVM outputs the posterior probabilistic prediction of the semantic concepts. This inference output is optimal for ranking the target video shots, according to the Probabilistic Ranking Principle. The probability output of RVM on individual uni-modal features also facilitates probabilistic fusion of multi-modal evidences to minimize Bayes risk. We demonstrate both theoretically and empirically that RVM outperforms SVM for video semantic concept detection. The testings on TRECVID 07 dataset show that RVM produces statically significant improvements in MAP scores over the SVM-based methods.

In the past several years, we've been developing a high performance OCR engine for machine printed Chinese/ English documents. We have reported previously (1) how to use character modeling techniques based on MCE (minimum classification error) training to achieve the high recognition accuracy, and (2) how to use confidence-guided progressive search and fast match techniques to achieve the high recognition efficiency. In this paper, we present two more techniques that help reduce search errors and improve the robustness of our character recognizer. They are (1) to use MCE-trained character-pair models to avoid error-prone character-level segmentation for some trouble cases, and (2) to perform a MCE-based negative training to improve the rejection capability of the recognition models on the hypothesized garbage images during recognition process. The efficacy of the proposed techniques is confirmed by experiments in a benchmark test.

In this paper, we present an approach of detecting speech presence for which the decision rule is based on a combination of multiple features using a sigmoid function. A minimum classification error (MCE) training is used to update the weights adjustment for the combination. The features, consisting of three parameters: the ratio of ZCR, the spectral energy, and spectral entropy, are combined linearly with weights derived from the sub-band domain. First, the Bark-scale wavelet decomposition (BSWD) is used to split the input speech into 24 critical sub-bands. Next, the feature parameters are derived from the selected frequency sub-band to form robust voice feature parameters. In order to discard the seriously corrupted frequency sub-band, a strategy of adaptive frequency sub-band extraction (AFSE) dependant on the sub-band SNR is then applied to only the frequency sub-band used. Finally, these three feature parameters, which only consider the useful sub-band, are combined through a sigmoid type function incorporating optimal weights based on MSE training to detect either a speech present frame or a speech absent frame. Experimental results show that the performance of the proposed algorithm is superior to the standard methods such as G.729B and AMR2.

In recent years, statistical modeling approaches have steadily gained in popularity in the field of information retrieval. This article presents an HMM/N-gram-based retrieval approach for Mandarin spoken documents. The underlying characteristics and the various structures of this approach were extensively investigated and analyzed. The retrieval capabilities were verified by tests with word- and syllable-level indexing features and comparisons to the conventional vector-space model approach. To further improve the discrimination capabilities of the HMMs, both the expectation-maximization (EM) and minimum classification error (MCE) training algorithms were introduced in training. Fusion of information via indexing word- and syllable-level features was also investigated. The spoken document retrieval experiments were performed on the Topic Detection and Tracking Corpora (TDT-2 and TDT-3). Very encouraging retrieval performance was obtained.

Vector quantization (VQ) has been explored in the past as a means of reducing likelihood computation in speech recognizers which use hidden Markov models (HMMs) containing Gaussian output densities. Although this approach has proved successful, there is an extent beyond which further reduction in likelihood computation substantially degrades the recognition accuracy. Since the components of the VQ frontend are typically designed after model training is complete, this degradation can be attributed to the fact that VQ and HMM parameters are not jointly estimated. In order to restore the accuracy of a recognizer using VQ to aggressively reduce computation, joint estimation is necessary. We propose a technique which couples VQ frontend design with minimum classification error training. We demonstrate on a large vocabulary subword task that in certain cases, our joint training algorithm can reduce the string error rate by 79% compared to that of VQ mixture selection alone.

This article adopts three soft computing techniques including support vector machine (SVM), least square support vector machine (LSSVM) and relevance vector machine (RVM) for prediction of status of epimetemorphic rock slope. The input variables of SVM, LSSVM and RVM are bulk density, height, inclination, cohesion and internal friction angle. There are 53 datasets which have been used to develop the SVM, LSSVM and RVM models. The developed SVM, LSSVM and RVM give equations for prediction of status of epimetemorphic rock slope. The performance of SVM, LSSVM and RVM is 100%. A comparative study has been presented between the developed SVM, LSSVM and RVM. The results confirm that the developed SVM, LSSVM and RVM are effective tools for prediction of status of epimetemorphic rock slope.

Application of support vector machines and relevance vector machines in predicting uniaxial compressive strength of volcanic rocks