Template-Based Continuous Speech Recognition

This paper describes the combination of two recent machine learning techniques for acoustic modeling in speech recognition: deep neural networks (DNNs) and graph-based semi-supervised learning (SSL). While DNNs have been shown to be powerful supervised classifiers and have achieved considerable success in speech recognition, graph-based SSL can exploit valuable complementary information derived from the manifold structure of the unlabeled test data. Previous work on graph-based SSL in acoustic modeling has been limited to frame-level classification tasks and has not been compared to, or integrated with, state-of-the-art DNN/HMM recognizers. This paper represents the first integration of graph-based SSL with DNN based speech recognition and analyzes its effect on word recognition performance. The approach is evaluated on two small vocabulary speech recognition tasks and shows a significant improvement in HMM state classification accuracy as well as a consistent reduction in word error rate over a state-of-the-art DNN/HMM baseline.

In this dissertation, a novel approach of integrating template matching with statistical modeling is proposed to improve continuous speech recognition. Hidden Markov Modeling (HMMs) has been the dominant approach in statistical speech recognition since it provides a principled way of jointly modeling speech spectral variations and time dynamics. However, HMMs have the shortcoming of assuming the observations being independent within each state, which makes it ineffective in modeling the details of speech temporal evolutions that are important for characterizing nonstationary speech sounds. Template-based methods make comparisons between a test pattern and the templates derived from training data, and therefore they are able to capture speech dynamics and time correlation of speech frames better than HMM based methods. However, template matching requires large memory space and computational time since feature vectors of training data need to be stored in computer memory for access at the recognition stage, which is difficult in large vocabulary continuous speech recognition (LVCSR). Our proposed approach takes advantages of both statistical modeling and template matching, which overcomes the weakness of conventional template-based method and is feasible for LVCSR. We use multiple Gaussian Mixture Model (GMM) indices to represent each frame of speech templates, and define the template unit to be context-dependent phone segments (triphone context). We also use phonetic decision trees borrowed from those commonly used in HMMs to tie triphone templates and predict triphones unseen in training data. Two local distances, log likelihood ratio (LLR) and Kullback-Leibler (KL) divergence, are proposed for dynamic time warping (DTW) based template matching. In order to reduce computational complexity and storage space, we propose methods of minimum distance template selection (MDTS) and maximum log-likelihood template selection (MLTS), and investigate a template compression method on top of template selection to further improve recognition performance. The template based methods were used to rescore lattices generated by baseline HMMs on the tasks of TIMIT continuous phone recognition and teleheath LVCSR and experimental results demonstrated that the proposed approach of integrating template matching with statistical modeling significantly improved recognition performances over the HMM baselines. The template selection methods also provided significant recognition accuracy improvements over the HMM baseline while largely reducing the computation and storage complexities. When all templates or MDTS were used, using the LLR local distance obtained better recognition performance than the KL divergence local distance. For MLTS and template compression, KL divergence local distance provided better performance than the LLR local distance, and the template compression method made further improvements over KL based MLTS. Since the templates were constructed based on the GMM indices extracted from HMM baselines, we also validated the effectiveness of the proposed template methods based on enhanced HMM baselines. Experimental results showed that LLR based all template method was able to consistently improve TIMIT phone recognition accuracies based on four enhanced HMM baselines. Prosodic features such as duration, energy, and pitch can reflect longer span information of speech than conventional single frame vectors but they have commonly been ignored by HMMs. Template based methods provide possibilities to conveniently integrate prosodic features into speech recognition, which has not been well studied in the past. In this dissertation, we investigate combining template based methods with the speech prosodic features of duration, energy and pitch to further improve speech recognition accuracy. The scores of prosodic information were computed by a GMM based method and a non-parametric method, and the prosodic scores were combined with the acoustic scores in triphone template matching. Experimental results obtained on the telehealth task showed that prosodic information had positive effects on vowel sound recognition.

Hidden Markov models (HMM) with Gaussian mixture observation densities are the dominant approach in speech recognition. These systems typically use a representational model for acoustic modeling which can often be prone to overfitting and does not translate to improved discrimination. We propose a new paradigm centered on principles of structural risk minimization using a discriminative framework for speech recognition based on support vector machines (SVMs). SVMs have the ability to simultaneously optimize the representational and discriminative ability of the acoustic classifiers. We have developed the first SVM-based large vocabulary speech recognition system that improves performance over traditional HMM-based systems. This hybrid system achieves a state-of-the-art word error rate of 10.6% on a continuous alphadigit task—a 10% improvement relative to an HMM system. On SWITCHBOARD, a large vocabulary task, the system improves performance over a traditional HMM system from 41.6% word error rate to 40.6%. This dissertation discusses several practical issues that arise when SVMs are incorporated into the hybrid system.

Speech to Text was one of speech recognition applications which speech signal was processed, recognized and converted into a textual representation. Hidden Markov Model (HMM) was the widely used method in speech recognition. However, the level of accuracy using HMM was strongly influenced by the optimalization of extraction process and modellling methods. Hence in this research, the use of genetic algorithm (GA) method to optimize the Ergodic HMM was tested. In Hybrid HMM-GA, GA was used to optimize the Baum-welch method in the training process. It was useful to improve the accuracy of the recognition result which is produced by the HMM parameters that generate the low accuracy when the HMM are tested. Based on the research, the percentage increases the level of accuracy of 20% to 41%. Proved that the combination of GA in HMM method can gives more optimal results when compared with the HMM system that not combine with any method.

BACKGROUND A hidden Markov model (HMM) is used for exon prediction on DNA of genes Plasmodium falciparum that has a model structure based on exon region structure in coding sequence (CDS). The objective research was to develop a new structure model to predict exon on DNA-genes of Plasmodium falciparum based on CDS structure using the HMM system. METHODS Model design in CDS, between two exon regions can be found one intron region and the model state number is used for its region. Its state number is used by separating start codon from first exon region and stop codon from the last exon region up to 9. The Viterbi algorithm and the backward-forward method for transition as well as emission states are used for training process. Furthermore, Viterbi and Baum-Welch algorithms are used for the testing process. The correlation coefficient (CC) was used as performance indicator, as the ratio of the estimated state in the output and the original state in the input of the model. RESULTS The simulation results has shown that the CC values depend on the given of the backward-forward transition state values randomly. The model with state number 9 showed the highest average of CC values of 0.7289 for Viterbi algorithm, and is 0.7166 for Baum-Welch algorithm. However, the lowest average of CC values has been found for the model with state number five. Its values are 0.6735 by using Viterbi algorithm and 0.6661 by using Baum-Welch algorithm. CONCLUSION The new structure model based on HMM system was valid to predict exon on DNA-genes of Plasmodium falciparum.

Programs currently available for gene prediction from within genomic DNA are far from being powerful enough to elucidate the gene structure completely. We develop a hidden Markov model (HMM) to represent the degeneracy features of splicing junction donor sites in eucaryotic genes. The HMM system is fully trained using an expectation maximization algorithm and the system performance is evaluated using the 10-way cross-validation method. Experimental results show that our HMM system can correctly classify more than 95% of the candidate sequences into the right categories. More than 91% of the true donor sites and 97% of the false donor sites in the test data are classified correctly. These results are very promising, considering that only the local information in DNA is used. This model will be a very important component of effective and accurate gene structure detection system currently being developed in our lab.

This paper describes a data-driven technique for optimizing the acoustic models for speech recognition systems that target commercial applications over telephones. Frame-averaged foreground log-likelihoods (foreground scores) correlate to recognition errors. These scores are used together with gender to optimize data weighting for the acoustic model. This process is interpreted as increasing the priors and associated parameters for poorly modeled data. The score-based optimization leads to about 7% fewer semantic errors on a live evaluation set collected after the last data used to estimate the acoustic model.

This paper proposes a new model for a noise-robust Automatic Speech Recognition (ASR) based on parallel branch Hidden Markov Model (HMM) structure with a novel approach for robust speech recognition. Automatic Speech Recognition applications such as voice command and control, audio indexing, speech-to-speech translation, do not usually work well in noisy environments. In this paper, we present the characteristics of a novel model by exploring vibrocervigraphic and electromyographic ASR methods and some other effective approaches to achieve the best results. By employing the proposed model, we obtain the word error rate, available bandwidths with cutoff frequencies, word recognition rate, etc. This paper includes advanced front-end processing with less computational requirements and a statistical modeling for large-vocabulary myoelectric speech. Therefore parameters estimation of ASR system like Mel-Frequency Cepstral Coefficients (MFCC) are investigated to create the statistically optimized model.

One of the biggest difficultiesin automatic speech recognition (ASR) is how to deal with variations of speech signals caused by non-linguistic information, such as age, gender, etc. Various methods have been proposed to compensate for the variations and one of them is speech structure [1]. Speech structure, which extracts only contrastive features and discards absolute features, is proved to be transform-invariant mathematically and to be very robust with the non-linguistic variations experimentally [2]. Although the conventional speech structure extracts local and distant contrastive features, it did not extract dynamic features explicitly which are supposed to exist in the contrastive features. In this paper, we reformulate speech structure based on trajectory HMM and derive trajectory structure (TSR), in which dynamic and contrastive features can be defined and used in ASR. We carry out an experiment of n-best rescoring of isolated word recognition using trajectory structure and obtain 28.5% relative decrease in word error rate.

Effective hidden Markov models for detecting splicing junction sites in DNA sequences

Up till recently, state-of-the-art, large vocabulary, continuous speech recognition (CSR) had employed hidden Markov modeling (HMM) to model speech sounds. In an attempt to improve over HMM we developed a hybrid system that integrates HMM technology with neural networks. We present the concept of a Segmental Neural Net (SNN) for phonetic modeling in CSR. By taking into account all the frames of a phonetic segment simultaneously, the SNN overcomes the well-known conditional-independence limitation of HMMs. We have developed a novel hybrid SNN/HMM system that combines the advantages of SNNs and HMMs using a multiple hypothesis (or N-best) paradigm. In this system, we generate likely phonetic segmentations from the HMM N-best list of word sequences, which are scored by the SNN. The HMM and SNN scores are then combined to optimize performance. In several speaker-independent, 1000-word CSR tests, the error rate for the hybrid system dropped 20% from that of a state-of-the-art HMM system alone.

Multi-Pattern Viterbi Algorithm for joint decoding of multiple speech patterns

A hybrid support vector machine (SVM) and hidden Markov model (HMM) approach is proposed for designing continuous speech recognition systems. Using novel properties of SVMs and combining them with HMMs one can obtain models that map easily to hardware and lead to more modular and scalable design. The overall architecture of the proposed system is based on the MAP (maximum a posteriori) framework which offers a direct, feedforward recognition model. The SVMs generate smooth estimates of local transition probabilities in the HMM, conditioned on the acoustic inputs. The transition probabilities are then used to estimate the global posterior probabilities of HMM state sequences. A parallel architecture that implements a simple speech recognition model in real-time is presented.

The current state-of-the-art in large-vocabulary, continuous speech recognition is based on the use of hidden Markov models (HMM). In an attempt to improve over HMM performance, the authors developed a hybrid system that combines the advantages of neural networks and HMM using a multiple hypothesis (or N-best) paradigm. The connectionist component of the system, the segmental neural net (SNN), models all the frames of a phonetic segment simultaneously, thus overcoming the well-known conditional-independence limitation of the HMM. They describe the hybrid system and discuss various aspects of SNN modeling, including network architectures, training algorithms and context modeling. Finally, they evaluate the hybrid system by performing several speaker-independent experiments with the DARPA Resource Management (RM) corpus, and demonstrate that the hybrid system shows a consistent improvement in performance over the baseline HMM system. >

Among speech researchers, it is widely believed that Hidden Markov Models (HMMs) are the most successful modelling approaches for acoustic events in speech recognition. However, common assumptions limit the classification abilities of HMMs and these can been relaxed by introducing neural networks in the HMM framework. With today's advances in VLSI technology, artificial neural networks (ANNs) can be integrated into a single chip offering adequate circuit complexity required to attain both a high recognition accuracy and an improved learning time. Analogue implementations are considered due to the high processing speeds. The relative performance of different speech coding parameters for use with two different ANN architectures that lend themselves to analogue hardware implementations are investigated. In this case, the dynamic ranges of the different coefficients need to be taken into consideration since they will affect the performance of the analogue chip due to the scaling of the coefficients to voltage signals. The hardware requirements for implementing the two architectures are then discussed.