Automatic context induction for tone model integration in mandarin speech recognition

Studies the problem of phonetic modeling for continuous Mandarin speech recognition by providing a systematic performance comparison for systems based on following primitive speech units: syllables, demi-syllables (initial and final), context-independent phones, left-or-right context-dependent phones (diphones) and left-and-right context-dependent phones (triphones). In our speaker-dependent continuous speech recognition experiments, a generalized triphone system has achieved the best performance of all. Our best system is in contrast to most other Mandarin speech recognition systems, which have been based on demi-syllable units.

The choice of basic modeling unit in building acoustic model for a continuous Mandarin speech recognition task is a very important issue [1]. Unlike traditional phoneme or Initial/Finals (IFs) units based acoustic modeling methods, which usually suffer from the limitations of less accuracy in modeling intrasyllable variations and long scale temporal dependencies, in this paper, a practicable syllable based approach is presented. In contrast with IFs, syllable can implicitly model the intrasyllable variations in good accuracy. Also, by carefully choosing context modeling schemes and parameter tying methods, syllable based acoustic model can capture longer temporal variations while keeping the complexity of model well controlled. Meanwhile, considering the data unbalanced problem, multiple sized unit model based approaches are also implemented in this research. The experiment result shows the acoustic model based on the presented syllable based approach is effective in improving the performance of the Chinese continuous speech recognition.

Statistical confusability between different acoustic models is important to character substitution error rate in large vocabulary continuous speech recognition. In this paper, we take factors of gender and speaking styles into consideration in Mandarin speech recognition. We modeled phonemes in different speaking styles, including read speech of female, male, and spontaneous dialogue. Then Minimum Gaussian Distances between Chinese Initial/Final model pairs are given and average phoneme distances are calculated which denote the pronunciation varieties. The effect of different style to average phonemic distance is studied and relative articulation is given for three databases. Qualitative relationship between phone size and error rate in recognition is analytical researched, showing that for a particular phoneme, pronunciation variety is one of reasons for misidentification in recognizing process, which provides us a novel mind to reduce substitution errors.

In this dissertation, a novel approach of integrating template matching with statistical modeling is proposed to improve continuous speech recognition. Hidden Markov Modeling (HMMs) has been the dominant approach in statistical speech recognition since it provides a principled way of jointly modeling speech spectral variations and time dynamics. However, HMMs have the shortcoming of assuming the observations being independent within each state, which makes it ineffective in modeling the details of speech temporal evolutions that are important for characterizing nonstationary speech sounds. Template-based methods make comparisons between a test pattern and the templates derived from training data, and therefore they are able to capture speech dynamics and time correlation of speech frames better than HMM based methods. However, template matching requires large memory space and computational time since feature vectors of training data need to be stored in computer memory for access at the recognition stage, which is difficult in large vocabulary continuous speech recognition (LVCSR). Our proposed approach takes advantages of both statistical modeling and template matching, which overcomes the weakness of conventional template-based method and is feasible for LVCSR. We use multiple Gaussian Mixture Model (GMM) indices to represent each frame of speech templates, and define the template unit to be context-dependent phone segments (triphone context). We also use phonetic decision trees borrowed from those commonly used in HMMs to tie triphone templates and predict triphones unseen in training data. Two local distances, log likelihood ratio (LLR) and Kullback-Leibler (KL) divergence, are proposed for dynamic time warping (DTW) based template matching. In order to reduce computational complexity and storage space, we propose methods of minimum distance template selection (MDTS) and maximum log-likelihood template selection (MLTS), and investigate a template compression method on top of template selection to further improve recognition performance. The template based methods were used to rescore lattices generated by baseline HMMs on the tasks of TIMIT continuous phone recognition and teleheath LVCSR and experimental results demonstrated that the proposed approach of integrating template matching with statistical modeling significantly improved recognition performances over the HMM baselines. The template selection methods also provided significant recognition accuracy improvements over the HMM baseline while largely reducing the computation and storage complexities. When all templates or MDTS were used, using the LLR local distance obtained better recognition performance than the KL divergence local distance. For MLTS and template compression, KL divergence local distance provided better performance than the LLR local distance, and the template compression method made further improvements over KL based MLTS. Since the templates were constructed based on the GMM indices extracted from HMM baselines, we also validated the effectiveness of the proposed template methods based on enhanced HMM baselines. Experimental results showed that LLR based all template method was able to consistently improve TIMIT phone recognition accuracies based on four enhanced HMM baselines. Prosodic features such as duration, energy, and pitch can reflect longer span information of speech than conventional single frame vectors but they have commonly been ignored by HMMs. Template based methods provide possibilities to conveniently integrate prosodic features into speech recognition, which has not been well studied in the past. In this dissertation, we investigate combining template based methods with the speech prosodic features of duration, energy and pitch to further improve speech recognition accuracy. The scores of prosodic information were computed by a GMM based method and a non-parametric method, and the prosodic scores were combined with the acoustic scores in triphone template matching. Experimental results obtained on the telehealth task showed that prosodic information had positive effects on vowel sound recognition.

Recurrent neural network language models (RNNLMs) have been successfully applied in a variety of language processing applications ranging from speech recognition to machine translation. They can fight the curse of dimensionality by learning a distributed representation (word vector). The components of these vectors measure the co-occurrence of the word with context features over a corpus. However, RNNLMs ignore the fact that the meaning of word can vary substantially in different contexts (e.g., for polysemous words). In this paper, we investigate part-of-speech information to address this issue to some extent on the basis of information about the meaning of a word they could provide. Experimental results on Mandarin speech recognition task show that a significant character error reduction of 1.18% absolute (7.72% relative) was obtained when using recurrent neural network language model with part-of-speech.

Minimum classification error (MCE) based discriminative methods have been extensively studied and successfully applied to automatic speech recognition, speaker recognition, and utterance verification. Our goal is to modify the embedded string model based MCE algorithm to handle the case of a large number of cross-syllable trip hones used in a large vocabulary recognition system. A selective strategy about MCE based discriminative training method, in particular for Mandarin speech syllable loop recognition, is introduced. We use a syllable loop recognition task to evaluate the performance of the acoustic model of an established large vocabulary continuous speech recognition system. The basic idea is that since decoding errors occur in parts of the models in whole decoded sentences, it is reasonable to adjust the parameters of the wrong models. As a result, a weighted MCE formulation is derived, which can provide more effective convergence and about a 10% error rate reduction for a large training set. Our experiments show that, although the performance of the recognition system is improved, some original correct recognition results are misrecognized after discriminative training, and a divide and conquer strategy is proposed as a solution. Thus, the acoustic feature space is divided into two or more sub-spaces according to the discriminative training procedure. By combining these two methods, we obtain over 14.5% error rate reduction in syllable loop recognition.

Ngram models are simple in language modeling and have been successfully used in speech recognition and other tasks. However, they can only capture the short distance context dependency within an n-words window where currently the largest practical n for a natural language is three while much of the context dependency in a natural language occurs beyond a three words window. In order to incorporate this kind of long distance context dependency in the ngram model of our Mandarin speech recognition system, this paper proposes a novel MI-Ngram modeling approach. This new MI-Ngram model consists of two components: a normal ngram model and a novel MI model. The ngram model captures the short distance context dependency within an n-words window while the MI model captures the context dependency between the word pairs over a long distance by using the concept of mutual information. That is, the MI-Ngram model incorporates the word occurrences beyond the scope of the normal ngram model. It is found that MI-Ngram modeling has much better performance than the normal word ngram modeling. Experimentation shows that about 20% of errors can be corrected by using a MI-Trigram model compared with the pure word trigram model.

For patients with relatively good low-frequency hearing and relatively poor high-frequency hearing, who met the pre-implant criteria for combined electric and acoustic stimulation (EAS), our aims were to i) assess deficits in low-frequency auditory function, ii) to identify measures which might be sensitive to changes resulting from the insertion of an intracochlear electrode array, and iii) to quantify the relationship between measures of auditory function and performance on tasks of speech and melody recognition. Measures of frequency selectivity, temporal resolution, and nonlinear cochlear function, along with measures of word, sentence, consonant, vowel, and melody recognition, were obtained from 5 normal-hearing and 17 hearing-impaired listeners. The hearing-impaired listeners had auditory thresholds at 500 Hz, ranging from 20 to 60 dB HL, and thresholds at 1 kHz, ranging from 60 to 100 dB HL. Nonlinear cochlear function was either reduced or absent. Frequency selectivity at 500 Hz was significantly reduced but still present in most patients. Temporal resolution, when measured at low modulation frequencies, was normal. Speech recognition in a modulated background revealed significantly poorer performance than normal. Speech and melody recognition varied over a large range. No measure of auditory function was correlated significantly with speech recognition. However, frequency selectivity was related to melody recognition. (1) Patients who qualify for EAS surgery have a wide range of speech and melody recognition abilities. (2) A number of the psychophysical measures tested may prove more sensitive than the audiogram in determining the degree of damage inflicted by the intracochlear electrode array. (3) Speech recognition was not correlated with any of the measures of auditory function.

Combined speech enhancement and auditory modelling for robust distributed speech recognition

Speech production knowledge has been used to enhance the phonetic representation and the performance of automatic speech recognition (ASR) systems successfully. Representations of speech production make simple explanations for many phenomena observed in speech. These phenomena can not be easily analyzed from either acoustic signal or phonetic transcription alone. One of the most important aspects of speech production knowledge is the use of articulatory knowledge, which describes the smooth and continuous movements in the vocal tract. In this paper, we present a new articulatory model to provide available information for rescoring the speech recognition lattice hypothesis. The articulatory model consists of a feature front-end, which computes a voicing feature based on a spectral harmonics correlation (SHC) function, and a back-end based on the combination of deep neural networks (DNNs) and hidden Markov models (HMMs). The voicing features are incorporated with standard Mel frequency cepstral coefficients (MFCCs) using heteroscedastic linear discriminant analysis (HLDA) to compensate the speech recognition accuracy rates. Moreover, the advantages of two different models are taken into account by the algorithm, which retains deep learning properties of DNNs, while modeling the articulatory context powerfully through HMMs. Mandarin speech recognition experiments show the proposed method achieves significant improvements in speech recognition performance over the system using MFCCs alone.

The article discusses the system of dialogue control manipulation robots. The analysis of the basic methods of automatic speech recognition, speech understanding, dialogue management, voice response synthesis in dialogue systems has been carried out. Three types of dialogue management are considered as "system initiative", "user initiative" and "combined initiative". A system of object-oriented dialog control of a robot based on the theory of finite state machines with using a deep neural network is proposed. The main difference of the proposed system lies in the separate implementation of the dialogue process and robot’s actions, which is close to the pace of natural dialogue control. This method of constructing a dialogue control robot allows system to automatically correct the result of speech recognition, robot’s actions based on tasks. The necessity of correcting the result of speech recognition and robot’s actions may be caused by the users’ accent, working environment noise or incorrect voice commands. The process of correcting speech recognition results and robot’s actions consists of three stages, respectively, in a special mode and a general mode. The special mode allows users to directly control the manipulator by voice commands. The general mode extends the capabilities of users, allowing them to get additional information in real time. At the first stage, continuous speech recognition is built by using a deep neural network, taking into account the accents and speech speeds of various users. Continuous speech recognition is a real-time voice to text conversion. At the second stage, the correction of the speech recognition result by managing the dialogue based on the theory of finite automata. At the third stage, the actions of the robot are corrected depending on the operating state of the robot and the dialogue management process. In order to realize a natural dialogue between users and robots, the problem is solved in creating a small database of possible dialogues and using various training data. In the experiments, the dialogue system is used to control the KUKA manipulator (KRC4 control) to put the desired block in the specified location, implemented in the Python environment using the RoboDK software. The processes and results of experiments confirming the operability of the interactive robot control system are given. A fairly high accuracy (92 %) and an automatic speech recognition rate close to the rate of natural speech were obtained.

Hidden Markov Models (HMMs) is one of the sophisticated methods for ASR system. In this work, we propose an Automatic Speech-Speaker Recognition (ASSR) system based on syllable models with HMM for speech expert on a FPGA platform. HMM is used to train the acoustic model for each phoneme evaluated on two databases, i.e. THCHS-30 (Tsinghua Chinese 30 hour database) and CMU ARCTIC Databases. The binary halved clustering (BHC) method which used binary-halved splitting to generate speaker models for low complexity requirement on speaker expert. The experiments are conducted on two types of platforms, including PC and Xilinx Spartan-6 FPGA. The experimental results indicate that the proposed work can achieve 90.8% accuracy rate at Mandarin speech recognition and 86.6% accuracy rate on English speech recognition, respectively. In addition, the work also evaluates OOV (Out of Vocabulary) which reaches 88.7% of detection rate on Mandarin and 84.9% of OOV detection rate on English as well. The speaker recognition rate also reaches to 81.3% and OOS (Out of Speaker) detection rate is 80.8%, respectively.

The authors present an NN-IIR (neural network/infinite impulse response filter) system for phoneme recognition in continuous speech based on the idea of modeling the recognition process by state evolution and interpretation equations. This work gives a solution to temporal information representation in phoneme recognition using neural networks and recursive filters, yielding better recognition results for continuous speech. This recognition system has two promising properties, i.e., capabilities for dealing with sequential properties and for interpreting speech signals by means of a training process. It was shown experimentally that the NN-IIR network obtained good performance for continuous speech recognition. Preliminary experiments with limited training data indicate that the NN-IIR provides good discrimination power for plosives, which are highly context-dependent.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The minimum classification error (MCE) framework for discriminative training is a simple and general formalism for directly optimizing recognition accuracy in pattern recognition problems. The framework applies directly to the optimization of hidden Markov models (HMMs) used for speech recognition problems. However, few if any studies have reported results for the application of MCE training to large-vocabulary, continuous-speech recognition tasks. This article reports significant gains in recognition performance and model compactness as a result of discriminative training based on MCE training applied to HMMs, in the context of three challenging large-vocabulary (up to 100 k word) speech recognition tasks: the Corpus of Spontaneous Japanese lecture speech transcription task, a telephone-based name recognition task, and the MIT Jupiter telephone-based conversational weather information task. On these tasks, starting from maximum likelihood (ML) baselines, MCE training yielded relative reductions in word error ranging from 7% to 20%. Furthermore, this paper evaluates the use of different methods for optimizing the MCE criterion function, as well as the use of precomputed recognition lattices to speed up training. An overview of the MCE framework is given, with an emphasis on practical implementation issues

Spectral representation of speech is complete when both the Fourier transform magnitude and phase spectra are specified. In conventional speech recognition systems, features are generally derived from the short-time magnitude spectrum. Although the importance of Fourier transform phase in speech perception has been realized, few attempts have been made to extract features from it. This is primarily because the resonances of the speech signal which manifest as transitions in the phase spectrum are completely masked by the wrapping of the phase spectrum. Hence, an alternative to processing the Fourier transform phase, for extracting speech features, is to process the group delay function which can be directly computed from the speech signal. The group delay function has been used in earlier efforts, to extract pitch and formant information from the speech signal. In all these efforts, no attempt was made to extract features from the speech signal and use them for speech recognition applications. This is primarily because the group delay function fails to capture the short-time spectral structure of speech owing to zeros that are close to the unit circle in the z-plane and also due to pitch periodicity effects. In this paper, the group delay function is modified to overcome these effects. Cepstral features are extracted from the modified group delay function and are called the modified group delay feature (MODGDF). The MODGDF is used for three speech recognition tasks namely, speaker, language, and continuous-speech recognition. Based on the results of feature and performance evaluation, the significance of the MODGDF as a new feature for speech recognition is discussed