Dynamic Bayesian Networks for Language Modeling

We propose a method of dynamically registering out-of-vocabulary (OOV) words by assigning the pronunciations of these words to pre-inserted OOV tokens, editing the pronunciations of the tokens. To do this, we add OOV tokens to an additional, partial copy of our corpus, either randomly or to part-of-speech (POS) tags in the selected utterances, when training the language model (LM) for speech recognition. This results in an LM containing OOV tokens, to which we can assign pronunciations. We also investigate the impact of acoustic complexity and the “natural” occurrence frequency of OOV words on the recognition of registered OOV words. The proposed OOV word registration method is evaluated using two modern automatic speech recognition (ASR) systems, Julius and Kaldi, using DNN-HMM acoustic models and N-gram language models (plus an additional evaluation using RNN re-scoring with Kaldi). Our experimental results show that when using the proposed OOV registration method, modern ASR systems can recognize OOV words without re-training the language model, that the acoustic complexity of OOV words affects OOV recognition, and that differences between the “natural” and the assigned occurrence frequencies of OOV words have little impact on the final recognition results.

In this paper, we describe the work developed in the creation of syllable-based language model for continuous speech recognition system for Myanmar language. Speech recognition systems contain language model as the part of prediction word order sequence. In English and other languages, speech recognition system can use word-based language model. Since Myanmar is monosyllabic and syllable-timed language, the syllable-based language model is more suitable in speech recognition system. At the first step, this paper explains the structure of traditional (Phrase based) language model and describes how to normalized Myanmar sentences and building of normalized Myanmar language model. In the second step, the evaluation of the two language models is expressed. Finally, the comparison of these two language models with the perplexity values is described. The syllable-based Myanmar language model has lower perplexity value than the traditional language model. This language model can support the creation of word-based Myanmar language model.

The unreliability evaluation of a system including dependencies involving the state of components or the failure events, can be performed by modelling the system as a dynamic fault tree (DFT). The combinatorial technique used to solve standard Fault Trees is not suitable for the analysis of a DFT. The conversion into a dynamic Bayesian network (DBN) is a way to analyze a DFT. This paper presents a software tool allowing the automatic analysis of a DFT exploiting its conversion to a DBN. First, the architecture of the tool is described, together with the rules implemented in the tool, to convert dynamic gates in DBNs. Then, the tool is tested on a case of system: its DFT model and the corresponding DBN are provided and analyzed by means of the tool. The obtained unreliability results are compared with those returned by other tools, in order to verify their correctness. Moreover, the use of DBNs allows to compute further results on the model, such as diagnostic and sensitivity indices

We propose a dynamic differential Bayesian networks (DDBNs) and nonparametric regression model. This model is an extended model of traditional dynamic Bayesian networks (DBNs), which can incorporate temporal information in a natural way and directly handle real-valued data obtained from microarrays without any transformation. In addition, it can cope with differential information between gene expression levels, without any loss to the traditional advantage, i.e., the capability of estimating non-linear relationships between genes. We apply DDBNs to analyze simulated data and real data, i.e., Saccharomyces cerevisiae cell cycle gene expression data. We have confirmed the effectiveness of our approach in the sense that some edges have been successfully detected only by DDBNs, not by DBNs.

Coordination among cortical neurons is believed to be a key element in mediating many high-level cortical processes such as perception, attention, learning, and memory formation. Inferring the structure of the neural circuitry underlying this coordination is important to characterize the highly nonlinear, time-varying interactions between cortical neurons in the presence of complex stimuli. In this work, we investigate the applicability of dynamic Bayesian networks (DBNs) in inferring the effective connectivity between spiking cortical neurons from their observed spike trains. We demonstrate that DBNs can infer the underlying nonlinear and time-varying causal interactions between these neurons and can discriminate between mono- and polysynaptic links between them under certain constraints governing their putative connectivity. We analyzed conditionally Poisson spike train data mimicking spiking activity of cortical networks of small and moderately large size. The performance was assessed and compared to other methods under systematic variations of the network structure to mimic a wide range of responses typically observed in the cortex. Results demonstrate the utility of DBN in inferring the effective connectivity in cortical networks.

Speech-recognition systems must often decide between competing ways of breaking up the acoustic input into strings of words. Since the possible strings may be acoustically similar, a language model is required; given a word string, the model returns its linguistic probability. Several Markov language models are discussed. A novel kind of language model which reflects short-term patterns of word use by means of a cache component (analogous to cache memory in hardware terminology) is presented. The model also contains a 3g-gram component of the traditional type. The combined model and a pure 3g-gram model were tested on samples drawn from the Lancaster-Oslo/Bergen (LOB) corpus of English text. The relative performance of the two models is examined, and suggestions for the future improvements are made. >

Recurrent neural network language models (RNNLM) have become an increasingly popular choice for state-of-the-art speech recognition systems. RNNLMs are normally trained by minimizing the cross entropy (CE) using the stochastic gradient descent (SGD) algorithm. The SGD method only uses first-order derivatives and no higher order gradient information is used to consider the correlation between model parameters. It is unable to fully capture the curvature of the error cost function. This can lead to slow convergence in model training. In this paper, a limited-memory Broyden Fletcher Goldfarb Shannon (L-BFGS) based second order optimization technique is proposed for RNNLMs. This method efficiently approximates the matrix-vector product between the inverse Hessian and gradient vector via a recursion over past gradients with a compact memory requirement. Consistent perplexity and error rate reductions are obtained over the SGD method on two speech recognition tasks: Switchboard English and Babel Cantonese. A faster convergence and speed up in RNNLM training time was also obtained. Index Terms: recurrent neural network, language model, second order optimization, limited-memory BFGS, speech recognition

In our paper we propose new technique for language modelling of highly inflectional languages such as Czech, Russian an other Slavic languages. Our aim is to alleviate main problem encountered in these languages, which is enormous vocabulary growth caused by great number of different word forms derived from one word (lemma). We reduced the size of the vocabulary by decomposing words into stems and endings and storing these sub-word units (morphemes) in the vocabulary separately. Then we trained morpheme based language model on the decomposed corpus. This paper reports perplexities, OOV rates and some speech recognition results obtained with new language model.

Patients suffering from chronic conditions may need to make frequent decisions about the management of their condition in partnership with their health professionals. However, this may not be possible as appointments are not always scheduled according to necessity but instead at a fixed frequency. Remote monitoring technology has the potential to generate patient data but without intelligent systems capable of analysing the data and offering advice, more data just increase the person's dependency on clinical staff for its interpretation. Decision-support systems that can give people more autonomy in the management of their condition can therefore benefit both the affected person and clinicians. We propose the use of Dynamic Bayesian Networks built from expert knowledge to interpret data and support decision-making, offering advice to patients suffering from a chronic condition. We argue that expert knowledge is needed as well as data to build such a decision-support system as the data that would be required to use machine learning will never be available in the current clinical system with all treatment decisions made at appointments scheduled at fixed intervals. We illustrate the methodology using a case study in Gestational Diabetes.

Probability and statistics in computational linguistics, a brief review.- Three issues in modern language modeling.- Stochastic analysis of Structured Language Modeling.- Latent semantic language modeling for speech recognition.- Prosody modeling for automatic speech recognition and understanding.- Switching dynamic system models for speech articulation and acoustics.- Segmental HMMS: Modeling dynamics and underlying structure in speech.- Modelling graph-based observation spaces for segment-based speech recognition.- Towards robust and adaptive speech recognition models.- Graphical models and automatic speech recognition.- An introduction to Markov chain Monte Carlo methods.- Semiparametric filtering in speech processing.- List of workshop participants.

Topic tracking language model for speech recognition

Speech recognition in digital assistants such as Google Assistant can potentially benefit from the use of conversational context consisting of user queries and responses from the agent. We explore the use of recurrent, Long Short-Term Memory (LSTM), neural language models (LMs) to model the conversations in a digital assistant. Our proposed methods effectively capture the context of previous utterances in a conversation without modifying the underlying LSTM architecture. We demonstrate a 4% relative improvement in recognition performance on Google Assistant queries when using the LSTM LMs to rescore recognition lattices.

A study on language modeling for speech recognition

We propose algorithms to train production-quality n-gram language models using federated learning. Federated learning is a distributed computation platform that can be used to train global models for portable devices such as smart phones. Federated learning is especially relevant for applications handling privacy-sensitive data, such as virtual keyboards, because training is performed without the users’ data ever leaving their devices. While the principles of federated learning are fairly generic, its methodology assumes that the underlying models are neural networks. However, virtual keyboards are typically powered by n-gram language models for latency reasons. We propose to train a recurrent neural network language model using the decentralized FederatedAveraging algorithm and to approximate this federated model server-side with an n-gram model that can be deployed to devices for fast inference. Our technical contributions include ways of handling large vocabularies, algorithms to correct capitalization errors in user data, and efficient finite state transducer algorithms to convert word language models to word-piece language models and vice versa. The n-gram language models trained with federated learning are compared to n-grams trained with traditional server-based algorithms using A/B tests on tens of millions of users of a virtual keyboard. Results are presented for two languages, American English and Brazilian Portuguese. This work demonstrates that high-quality n-gram language models can be trained directly on client mobile devices without sensitive training data ever leaving the devices.

Code-switching is a commonly occurring phenomenon in many multilingual communities, wherein a speaker switches between languages within a single utterance. Conventional Word Error Rate (WER) is not sufficient for measuring the performance of code-mixed languages due to ambiguities in transcription, misspellings and borrowing of words from two different writing systems. These rendering errors artificially inflate the WER of an Automated Speech Recognition (ASR) system and complicate its evaluation. Furthermore, these errors make it harder to accurately evaluate modeling errors originating from code-switched language and acoustic models. In this work, we propose the use of a new metric, transliteration-optimized Word Error Rate (toWER) that smoothes out many of these irregularities by mapping all text to one writing system and demonstrate a correlation with the amount of code-switching present in a language. We also present a novel approach to acoustic and language modeling for bilingual code-switched Indic languages using the same transliteration approach to normalize the data for three types of language models, namely, a conventional n-gram language model, a maximum entropy based language model and a Long Short Term Memory (LSTM) language model, and a state-of-the-art Connectionist Temporal Classification (CTC) acoustic model. We demonstrate the robustness of the proposed approach on several Indic languages from Google Voice Search traffic with significant gains in ASR performance up to 10% relative over the state-of-the-art baseline.