GRAPHICAL MODEL REPRESENTATIONS OF WORD LATTICES

Optimising dynamic graphical models for video content analysis

Operational and financial performance of Italian airport companies: A dynamic graphical model

This work extends and improves a recently introduced (Dec. 2007) dynamic Bayesian network (DBN) based audio-visual automatic speech recognition (AV-ASR) system. That system models the audio and visual components of speech as being composed of the same sub-word units when, in fact, this is not psycholinguistically true. We extend the system to model the audio and visual streams as being composed of separate, yet related, sub-word units. We also introduce a novel stream weighting structure incorporated into the model itself. In doing so, our system makes improvements in word error rate (WER) and overall recognition accuracy in a large vocabulary continuous speech recognition task (LVCSR). The ldquobestrdquo performing proposed system attains a WER of 66.71%whereas the ldquobestrdquo baseline system performs at a WER of 64.30%. The proposed system also improves accuracy to 45.95% from 39.40%.

Speech recognition systems have used the concept of states as a way to decompose words into sub-word units for decades. As the number of such states now reaches the number of words used to train acoustic models, it is interesting to consider approaches that relax the assumption that words are made of states. We present here an alternative construction, where words are projected into a continuous embedding space where words that sound alike are nearby in the Euclidean sense. We show how embeddings can still allow to score words that were not in the training dictionary. Initial experiments using a lattice rescoring approach and model combination on a large realistic dataset show improvements in word error rate.

State-level minimum Bayes risk (sMBR) training has become the de facto standard for sequence-level training of speech recognition acoustic models. It has an elegant formulation using the expectation semiring, and gives large improvements in word error rate (WER) over models trained solely using cross-entropy (CE) or connectionist temporal classification (CTC). sMBR training optimizes the expected number of frames at which the reference and hypothesized acoustic states differ. It may be preferable to optimize the expected WER, but WER does not interact well with the expectation semiring, and previous approaches based on computing expected WER exactly involve expanding the lattices used during training. In this paper we show how to perform optimization of the expected WER by sampling paths from the lattices used during conventional sMBR training. The gradient of the expected WER is itself an expectation, and so may be approximated using Monte Carlo sampling. We show experimentally that optimizing WER during acoustic model training gives 5% relative improvement in WER over a well-tuned sMBR baseline on a 2-channel query recognition task (Google Home).

Dynamic graphical models aim to describe the time-varying dependency structure of multiple time-series. In this article we review research focusing on the formulation and estimation of such models. The bulk of work in graphical structurelearning problems has focused in the stationary i.i.d setting, we present a brief overview of this work before introducing some dynamic extensions. In particular we focuson two classes of dynamic graphical model; continuous (smooth) models which are estimated via localised kernels, and piecewise models utilising regularisation based estimation. We give an overview of theoretical and empirical results regarding these models, before demonstrating their qualitative difference in the context of a real-world financial time-series dataset. We conclude with a discussion of the state of the field and future research directions.

Automatic Speech Recognition requires large amounts of training data to achieve good results. As this hand-labeling is both slow and expensive, the utilization of untranscribed speech has been explored for the task. This generally involves training a teacher-model on the available transcribed speech, and training a student model either directly on the latent representations of the teacher, or on the decoded output. We propose to combine these two approaches, and rescore the predictions of the teacher based on the decoded output. The probability of a decoded sentence, and how it corresponds with the probability distribution output of the teacher, affects the rescoring. When training our student model and evaluating using our proposed method, we find it gives up to 8.6% relative improvement in character error rate, and 5.4% relative improvement in word error rate over our strongest baseline.

Many features used in speech recognition tasks are hand-crafted and are not always related to the objective at hand, that is minimizing word error rate. Recently, we showed that replacing a perceptually motivated mel-filter bank with a filter bank layer that is learned jointly with the rest of a deep neural network was promising. In this paper, we extend filter learning to a speaker-adapted, state-of-the-art system. First, we incorporate delta learning into the filter learning framework. Second, we incorporate various speaker adaptation techniques, including VTLN warping and speaker identity features. On a 50-hour English Broadcast News task, we show that we can achieve a 5% relative improvement in word error rate (WER) using the filter and delta learning, compared to having a fixed set of filters and deltas. Furthermore, after speaker adaptation, we find that filter and delta learning allows for a 3% relative improvement in WER compared to a state-of-the-art CNN.

Adapting multilingual automatic speech recognition (MASR) models to support new languages is crucial for enhancing global communication accessibility. However, extending these models often leads to catastrophic forgetting, impairing their ability to accurately process previously learned languages. To address this, we present a mixture-of-experts (MoE) strategy that employs Low-Rank Adaptation (LoRA) experts, each dedicated to a specific language. Unlike previous approaches that rely on a small neural network for gating, we utilize the language identification (LID) outputs of the MASR model itself to more precisely activate the appropriate LoRA experts. Our method involves training two sets of LoRAs for LID and ASR, respectively: the LID LoRAs adapt the MASR model for LID across both existing and new languages, and the ASR LoRAs consist of multiple language-specific LoRA experts for ASR. During inference, we first employ the LID LoRAs to identify the language, followed by the activation of the corresponding ASR LoRAs based on this identification. Unfortunately, this approach means LID errors also cause incorrect ASR LoRAs activation. We perform language-wise beam search to allow self-correction for such mistakes. When applied to the Whisper model and integrated with ten new languages from the Common Voice dataset, our approach achieves up to 13.8% and 10.4% relative improvements in word error rate (WER) for new and previously learned languages, respectively. This method effectively reduces catastrophic forgetting with less than 3% additional computation overhead compared to the standard LoRA implementation.

A method of domain adaptation for clustered language models is developed. It is based on a previously developed clustering algorithm (Ueberla, 1994), but with a modified optimisation criterion. The results are shown to be slightly superior to the previously published 'Fillup' method (Besling and Meier, 1995), which can be used to adapt standard n-gram models. However, the improvement both methods give compared to models built from scratch on the adaptation data is quite small (less than 11% relative improvement in word error rate). This suggests that both methods are still unsatisfactory from a practical point of view.

We propose a neural network architecture, which operates on the raw speech signal, where the first layer contains a series of 1D time-domain filters. The output of this layer is fed to the second layer, which is a bank of 2D-convolution filters that capture the spectro-temporal modulations in the speech signal. The outputs of these two layers are concatenated, normalized and then fed to a feed-forward neural network to predict the senone posteriors, which are used for ASR decoding. During the training of the neural network, we have employed different strategies, where the 1D and 2D filters are initialized with (a) Gabor filters and (b) random values and the filter coefficients are either (a) allowed to be updated along with the other affine transform parameters of the network or (b) fixed during training. ASR experiments are conducted on 160 hours of Tamil speech data and the proposed architecture gives an absolute improvement in word error rate (WER) of 1.35% and 1.21% with respect to the neural network models trained on mel-frequency cepstral coefficients and log-filterbank energy features, respectively. We have also compared the performances of various strategies for filter initialization and training and reported the WERs.

We describe two methods that aim at normalizing acoustic vectors at the filterbank level such that the test data distribution matches the training data distribution. They enhance the histogram normalization technique proposed earlier by taking care of the variable silence fraction for each speaker, and by rotating the feature space. We report a number of recognition tests under minor (different microphones in training and test, telephone data) and major (office vs. car recordings) mismatch conditions. Both methods give superior performance to the basic histogram normalization approach. The overall improvements in word error rate (WER) range between 6% and 85% relative.

Recently we have explored the use of a Gaussian mixture model (GMM) based global transformer for artificial bandwidth extension (ABWE) for improving the automatic recognition of children's speech in mismatched condition. As the spectral characteristic of the speech varies significantly from one sound class to another so the global transformation would be sub-optimal for that purpose. Motivated by that in this work, we explore the use of class specific GMM based ABWE transformers for the bandwidth extension of the narrowband speech. For the deriving the class specific ABWE transformers an existing unsupervised hidden Markov model (HMM) based method is used. Further for contrast purpose an supervised class specific GMM based ABWE transformers are also explored. The unsupervised and supervised class specific ABWE approaches have resulted in 21.30% and 26.37% relative improvement in word error rate on digit recognition task. The effectiveness of class specific ABWE is also explored in terms of the mutual information between narrowband and the extended higherband speech as well as a group of other speech quality measures.

We present a new architecture and a training strategy for an adaptive mixture of experts with applications to domain robust language modeling. The proposed model is designed to benefit from the scenario where the training data are available in diverse domains as is the case for YouTube speech recognition. The two core components of our model are an ensemble of parallel long short-term memory (LSTM) expert layers for each domain and another LSTM based network which generates state dependent mixture weights for combining expert LSTM states by linear interpolation. The resulting model is a recurrent adaptive mixture model (RADMM) of domain experts. We train our model on 4.4B words from YouTube speech recognition data. We report results on the YouTube speech recognition test set. Compared with a background LSTM model, we obtain up to 12% relative improvement in perplexity and an improvement in word error rate from 12.3% to 12.1 % while using a lattice rescoring with strong pruning.

Automatic speech recognition (ASR) for pathologic speech remains a major challenge due to high variability in articulation, phonation, and prosody distortions. In this work, we propose a pathology-aware speech encoder based on BEST-RQ pre-training, which incorporates 46k hours of speech, including pathologic and atypical speech. We continue pre-training for domain adaptation and experiment with etiology-specific codebooks. We achieve a 13.2% relative word error rate (WER) improvement using the pathology-aware speech encoder with etiology-specific continued pre-training. Additionally, we examine the impact of incorporating synthetic and out-of-domain (OOD) data to further enhance ASR performance. Synthetic data reduces WER by up to 8.7%, while OOD data improves WER by 12.2%. Finally, we introduce a semantic similaritybased data augmentation technique to optimize data selection, achieving a WER improvement of up to 9.7% while minimizing the need for additional training data.