Auditory Training in Speech Recognition and Hyperacuity

Discriminative training techniques have been shown to consistently outperform the maximum likelihood (ML) paradigm for acoustic model training in automatic speech recognition (ASR). Consequently, today's discriminative training methods are fundamental components of state-of-the-art systems and are a major line of research in speech recognition. This article gives a comprehensive overview of discriminative training methods for acoustic model training in the context of ASR. The article covers all related aspects of discriminative training for speech recognition, i.e., specific training criteria and their relation, statistical modeling, different parameter optimization approaches, efficient implementation of discriminative training, and a performance overview.

Sequence-Level Unsupervised Training in Speech Recognition: A Theoretical Study

Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if they are trained by recordings of single talkers. This paper proposes a multi-speaker ASR method that incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set was extracted as numeric vectors, and all of the embedding vectors were stacked to construct a total speaker profile matrix. The speaker profile matrix from the training dataset enables finding embedding vectors that are close to the speakers of the input recordings in the test conditions, and it helps to recognize the individual speakers’ voices mixed in the input. Furthermore, the proposed method efficiently reuses the decoder from the existing speaker-independent ASR model, eliminating the need for retraining the entire system. Various speaker embedding methods such as i-vector, d-vector, and x-vector were adopted, and the experimental results show 0.33% and 0.95% absolute (3.9% and 11.5% relative) improvements without and with the speaker profile in the word error rate (WER).

While auditory training in quiet has been shown to improve cochlear implant (CI) users' speech understanding in quiet, it is unclear whether training in noise will benefit speech understanding in noise. The present study investigated whether auditory training could improve CI users' speech recognition in noise and whether training with familiar stimuli in an easy listening task (closed-set digit recognition) would improve recognition of unfamiliar stimuli in a more difficult task (open-set sentence recognition). CI users' speech understanding in noise was assessed before, during, and after auditory training with a closed-set recognition task (digits identification) in speech babble. Before training was begun, recognition of digits, Hearing in Noise Test (HINT) sentences, and IEEE sentences presented in steady speech-shaped noise or multitalker speech babble was repeatedly measured to establish a stable estimate of baseline performance. After completing baseline measures, participants trained at home on their personal computers using custom software for approximately 30 mins/day, 5 days/wk, for 4 wks, for a total of 10 hrs of training. Participants were trained only to identify random sequences of three digits presented in speech babble, using a closed-set task. During training, the signal-to-noise ratio was adjusted according to subject performance; auditory and visual feedback was provided. Recognition of digits, HINT sentences, and IEEE sentences in steady noise and speech babble was remeasured after the second and fourth week of training. Training was stopped after the fourth week, and subjects returned to the laboratory 1 mo later for follow-up testing to see whether any training benefits had been retained. Mean results showed that the digit training in babble significantly improved digit recognition in babble (which was trained) and in steady noise (which was not trained). The training benefit generalized to improved HINT and IEEE sentence recognition in both types of noise. Training benefits were largely retained in follow-up measures made 1 mo after training was stopped. The results demonstrated that auditory training in noise significantly improved CI users' speech performance in noise, and that training with simple stimuli using an easy closed-set listening task improved performance with difficult stimuli and a difficult open-set listening task.

Semi-supervised and unsupervised discriminative language model training for automatic speech recognition

Multivariate Gaussian based speech compensation or mapping has been developed to reduce the mismatch between training and deployment conditions for robust speech recognition. The acoustic mapping procedure can be formulated as a feature space adaptation where a noisy input signal is transformed by a multivariate Gaussian network. We propose a novel algorithm to update the network parameters based on minimizing the Kullback-Leibler distance between the core recognizer's acoustic model and transformed features. It is designed to achieve optimal overall system performance rather than MMSE on a specific feature domain. An online stochastic gradient descent learning rule is derived. We evaluate the performance of the new algorithm using a JRTk broadcast news system on a distance-talking speech corpus and compare its performance with that of previous MMSE based approaches. The experiments show the KL based approach is more effective for a large vocabulary continuous speech recognition (LVCSR) system.

This paper investigates a kind of lightly supervised acoustic model training method for Mandarin continues speech recognition system. The speech materials with rough transcription, which provide some light supervision for acoustic model training, are available in various forms these days. In this work, the quality problem of this kind of data is classified into two types: the first is non-speech and low-quality speech in the corpora, while the second is the transcription errors. A framework is proposed to tackle these two types separately: the speech recognition with transcription-relevant language model is adopted to remove the first type, while with general language model to provide candidate transcription errors which are checked by the final automatic verification process. The performance of proposed framework was evaluated from two aspects: the data quality has significantly improved, and the speech recognition results show that a 21.88% relative CER reduction was obtained.

The minimum classification error (MCE) framework for discriminative training is a simple and general formalism for directly optimizing recognition accuracy in pattern recognition problems. The framework applies directly to the optimization of hidden Markov models (HMMs) used for speech recognition problems. However, few if any studies have reported results for the application of MCE training to large-vocabulary, continuous-speech recognition tasks. This article reports significant gains in recognition performance and model compactness as a result of discriminative training based on MCE training applied to HMMs, in the context of three challenging large-vocabulary (up to 100 k word) speech recognition tasks: the Corpus of Spontaneous Japanese lecture speech transcription task, a telephone-based name recognition task, and the MIT Jupiter telephone-based conversational weather information task. On these tasks, starting from maximum likelihood (ML) baselines, MCE training yielded relative reductions in word error ranging from 7% to 20%. Furthermore, this paper evaluates the use of different methods for optimizing the MCE criterion function, as well as the use of precomputed recognition lattices to speed up training. An overview of the MCE framework is given, with an emphasis on practical implementation issues

Bayesian network structures and inference techniques for automatic speech recognition

In this paper, we propose a novel technique for direct recognition of multiple speech streams given the single channel of mixed speech, without first separating them. Our technique is based on permutation invariant training (PIT) for automatic speech recognition (ASR). In PIT-ASR, we compute the average cross entropy (CE) over all frames in the whole utterance for each possible output-target assignment, pick the one with the minimum CE, and optimize for that assignment. PIT-ASR forces all the frames of the same speaker to be aligned with the same output layer. This strategy elegantly solves the label permutation problem and speaker tracing problem in one shot. Our experiments on artificially mixed AMI data showed that the proposed approach is very promising.

Spoken accents severely degrade the performance of automatic speech recognition (ASR) systems. Domain adversarial training (DAT) is widely adopted for generating domain-invariant features to reduce the influence of accents. However, the generated features trained by DAT are still maintaining some accent discrimination information, limiting the ASR performance. In addition, the features generated by DAT of each accent have different degrees of residual accent discriminant information. In this paper, we propose an adaptive attention network with DAT to further eliminate the influence of retaining accent information in features generated by DAT. We employ the adaptive attention module to transform the encoder output to a more general representation. Experiments on the AESRC2020 dataset show that the proposed method can achieve satisfactory performance improvements on seen and unseen accent when the correct accent information is still preserved in the output of the encoder.

We present the OC16-CE80 Chinese-English mixlingual speech database which was released as a main resource for training, development and test for the Chinese-English mixlingual speech recognition (MixASR-CHEN) challenge on O-COCOSDA 2016. This database consists of 80 hours of speech signals recorded from more than 1,400 speakers, where the utterances are in Chinese but each involves one or several English words. Based on the database and another two free data resources (THCHS30 and the CMU dictionary), a speech recognition (ASR) baseline was constructed with the deep neural network-hidden Markov model (DNN-HMM) hybrid system. We then report the baseline results following the MixASR-CHEN evaluation rules and demonstrate that OC16-CE80 is a reasonable data resource for mixlingual research.

Improving distant speech recognition is a crucial step towards flexible human-machine interfaces. Current technology, however, still exhibits a lack of robustness, especially when adverse acoustic conditions are met. Despite the significant progress made in the last years on both speech enhancement and speech recognition, one potential limitation of state-of-the-art technology lies in composing modules that are not well matched because they are not trained jointly.

A new training procedure has been developed to improve speech recognition performance when the talker is under stress and training tokens cannot be obtained in the recognition environment. Training tokens are obtained from the desired talker speaking both normally and with different talking styles (e.g., loud, soft, fast). The recognizer is then trained with all tokens and used under stress. This technique was tested with a, 35 aircraft-word vocabulary, 9 talker, 11 340 token, stress/style database. Two stress conditions included (1) workload stress created with a perceptual-motor critical-tracking task (2) intense noise presented through earphones at a 90 dB SPL level. A continuous-distribution speaker-dependent hidden Markov model recognizer was trained normally (five normally spoken tokens) and with multi-style training (one token each from normal, fast, clear, loud, and question-pitch talking styles). Error rates were 36% in noise and 12% under workload stress with normal training. Multi-style training reduced error rates substantially to 13% in noise and 9% under workload stress. Multi-style training also reduced the error rate for normally spoken words. [Supported by DARPA.]

Iterative estimation procedures that minimize empirical risk based on general loss functions such as the Levenshtein distance have been derived as extensions of the Extended Baum Welch algorithm. While reducing expected loss on training data is a desirable training criterion, the se algorithms can be difficult to apply. They are unlike MMI estimation in that they require an explicit listing of the hy potheses to be considered and in complex problems such lists tend to be prohibitively large. To overcome this difficulty, modeling techniques originally developed to improve search efficiency in Minimum Bayes Risk decoding can be used to transform these estimation algorithms so that exact update, risk minimization procedures can be used for complex recognition problems. Experimental results in two large vocabulary speech recognition tasks show improvements over conventionally trained MMIE models.