Transliteration Based Approaches to Improve Code-Switched Speech Recognition Performance

This paper presents procedures used for development of a Croatian large vocabulary automatic speech recognition system (LVASR). The proposed acoustic model is based on context-dependent triphone hidden Markov models and Croatian phonetic rules. Different acoustic and language models, developed using a large collection of Croatian speech, are discussed and compared. The paper proposes the best feature vectors and acoustic modeling procedures using which lowest word error rates for Croatian speech are achieved. In addition, Croatian language modeling procedures are evaluated and adopted for speaker independent spontaneous speech recognition. Presented experiments and results show that the proposed approach for automatic speech recognition using context-dependent acoustic modeling based on Croatian phonetic rules and a parameter tying procedure can be used for efficient Croatian large vocabulary speech recognition with word error rates below 5%.

Nowadays automatic speech recognition (ASR) is an important task for machines. Several applications such as speech translation, virtual assistants and voice bot systems use ASR to understand human speech. Most of the research and available models are for widely used languages, such as English, German, French, Chinese and Spanish. This paper presents the Armenian speech recognition system. As a result of this research developed acoustic and language models for the Armenian language (modern ASR systems combine acoustic and language models to achieve higher accuracy). RNN-based Baidu’s Deep Speech deep neural network was used to train the acoustic model, and the KenLM toolkit was used to train the probabilistic language model. The acoustic model was trained and validated on ArmSpeech Armenian native speech corpus using transfer-learning and data augmentation techniques and tested on the Common Voice Armenian database. The language model was built based on the texts scraped from Armenian news websites. Final models are small in size and can be run and do real-time speech-to-text tasks on IoT devices. Testing on the Common Voice Armenian database the model gave 0.902565 WER and 0.305321 CER without the language model, and 0.552975 WER and 0.285904 CER with the language model. The paper aims to describe environment setup, data collection, acoustic and language models training processes, as well as final results and benchmarks.

Multi-microphone speech recognition integrating beamforming, robust feature extraction, and advanced DNN/RNN backend

The objective of this dissertation is to develop a new speech unit on acoustic modeling of the Thai language. The Thai syllables were studied in both acoustical and phonological properties. From the acoustical point of view, in the syllable structure, the final consonant is strongly influenced by the vowel duration. This relationship occurs only between the vowel and the final consonant. In contrase, the initial consonant is not affected by the duration of the vowel. Hence, the vowel and the final consonant are tightly tied while an inital consonant is loosely tied with the vowel in the syllable. From a phonological point of view, a syllable is composed of a pair of an onset and a rhyme unit. The onset consists of an initial consonant and its transition towards the following vowel. Along with the onset, the rhyme is composed of a vowel, a final consonant, and a tone. The onset-rhyme not only includes its contextual information, but also embeds the language modeling at the syllable level. Consequently, the decomposition of the syllable into an onset and rhyme is appropriate to the Thai language. The whole set of Thai syllables can be recognized by identifying onsets and rhymes. This research has objective to compare the efficiency of the units. Therefore a tone recognition system is not implemented in this research. To evaluate the effectiveness of the proposed acoustic model, various conventional speech units used in speech recognition systems have been investigated. Several experiments have been carried out to find the proper speech unit that can accurately create acoustic model and give a higher recognition rate. Results of recognition rates under different acoustic models are given and compared. The speech corpus used for training in this experiment was recorded from 9 male and 11 female speakers. This group of speakers also produced the speaker-dependent test set. In addition, the speaker-independent test set was produced from other 5 male and 5 female speakers. This speech corpus was designed to cover all onset-rhyme units in the Thai language. Experimental results show that the onset-rhyme model improves on the efficiency of other speech units. The onset-rhyme model improves on the accuracy of the baseline monophone model, the inter-syllable triphone model, and the context-dependent Initial-Final model by nearly 26.2%, 6.4%, and 4.2% for the speaker-dependent systems using only an acoustic model, and 29.7%, 6.0%, and 4.2% for the speaker-dependent systems using both acoustic and language model respectively. Using the language model, the onset accuracy is increased by around 16-21% for both SD and SI systems. In addition, the accuracy of the rhyme is substantially improved by nearly 45-47% for the SD and SI systems when the language model is applied. The results show that the onset-rhyme models attain a high recognition rate. Moreover, they also give more efficiency in terms of system complexity.

Conventional far-field automatic speech recognition (ASR) systems typically employ microphone array techniques for speech enhancement in order to improve robustness against noise or reverberation. However, such speech enhancement techniques do not always yield ASR accuracy improvement because the optimization criterion for speech enhancement is not directly relevant to the ASR objective. In this work, we develop new acoustic modeling techniques that optimize spatial filtering and long short-term memory (LSTM) layers from multi-channel (MC) input based on an ASR criterion directly. In contrast to conventional methods, we incorporate array processing knowledge into the acoustic model. Moreover, we initialize the network with beamformers' coefficients. We investigate effects of such MC neural networks through ASR experiments on the real-world far-field data where users are interacting with an ASR system in uncontrolled acoustic environments. We show that our MC acoustic model can reduce a word error rate (WER) by~16.5\% compared to a single channel ASR system with the traditional log-mel filter bank energy (LFBE) feature on average. Our result also shows that our network with the spatial filtering layer on two-channel input achieves a relative WER reduction of~9.5\% compared to conventional beamforming with seven microphones.

A statistically designed Automatic Speech Recognition (ASR) system extracts features from speech signals using feature extraction methods, links the extracted features with the expected phonetics of the hypothesis using acoustic models, uses language model to add prior information about the structure of the target language. For many years, Mel-frequency Cepstral coefficients (MFCC), n-gram, and Hidden Markov Model (HMM) approaches have been used predominantly for feature extraction, language modeling and acoustic modeling, respectively. However, performance degradation of MFCC in noisy conditions and inaccuracy of HMMs while handling large vocabularies have made researchers to propose more efficient methods. The proposed work uses noise robust method Gammatone Frequency Cepstral Coefficients(GFCC) for feature extraction, trigram language modeling, and HMM-Gaussian mixture model (GMM) based acoustic modeling to implement a continuous Hindi language ASR system. Also, it applies Differential Evolution (DE) technique to refine the GFCC features and discriminative techniques to enhance performance of the acoustic model. The performance of the implemented system has been evaluated by using different feature extraction methods, variants of n-gram language modeling techniques and different discriminative techniques in clean as well as noisy conditions. Initially, the results reveal that DE optimized GFCC with HMM-Gaussian Mixture Model (GMM) acoustic modeling performs better than MFCC, PLP and MF-PLP feature extraction methods. Secondly, the experimental results show that the Minimum Phone Error (MPE) outperforms Maximum Mutual Information (MMI) and Maximum Likelihood Estimation (MLE) and trigram based language modeling gives more accurate results than unigram and bigram language modeling. Finally, it has been concluded that the continuous Hindi language ASR system implemented using DE refined GFFC feature extraction method with MPE discriminative training technique and trigram based language modeling gives better accuracy in clean as-well-as noisy environments.

In the production of acoustic models for speech recognition applications, the use of Long Short Term Memory(LSTM) based Recurrent Neural Network(RNN) has begun to get better results than the use of Gaussian Mixture Model(GMM). The creation of GMM-based acoustic models is prolonging the deep learning process due to the need for aligned Hidden Markov Model(HMM). As a solution to this problem, another method to generate acoustic models is proposed that is based on Connectionist Temporal Classification(CTC). In this study, a CTC based model is created and the effect of different optimization techniques on the classification performance is compared. These tests were applied on Turkish speech datasets to determine the best optimization techniques to be used in speech recognition applications. Our evaluation results showed that GradientDescent, ProximalGradientDescent and RMSPROP produce better results than other algorithms.

In this paper we present the latest improvements to the Russian spontaneous speech recognition system developed in Speech Technology Center (STC). Significant word error rate (WER) reduction was obtained by applying hypothesis rescoring with sophisticated language models. These were the Recurrent Neural Network Language Model and regularized Long-Short Term Memory Language Model. For acoustic modeling we used the deep neural network (DNN) trained with speaker-dependent bottleneck features, similar to our previous system. This DNN was combined with the deep Bidirectional Long Short-Term Memory acoustic model by the use of score fusion. The resulting system achieves WER of 16.4 %, with an absolute reduction of 8.7 % and relative reduction of 34.7 % compared to our previous system result on this test set.

One of the most remarkable advantages of speech recognition (SR) technology includes speech to text conversion (STT). This paper's focal point is to recognize Bangla words among various speakers and noisy environments and convert them into text. Mel frequency cepstral coefficient (MFCC) has been used for extracting words from the audio file. Using gated recurrent units (GRU) along with directional long short term memory (LSTM), the continuity of words and next word-level prediction in a sentence have been implemented. An existing dataset that is enriched by recorded speech has been used here. Keras-Tensorflow toolkit has been used as a software apparatus for training data. The highest train accuracy has been obtained from the GRU architecture in the acoustic model (AM) is 94.44% while the test accuracy rate is only 47% due to the small test dataset. On the other hand, test accuracy is maximum up to 45.81% with the LSTM model in the language model (LM). The proposed method signifies good training and testing accuracy rate based on applying GRU model to the AM, LSTM to the LM and the numbers of total recordings of each sentence.

We present a novel approach for improving the performance of an End-to-End speech recognition system for the Gujarati language. We follow a deep learning-based approach that includes Convolutional Neural Network, Bi-directional Long Short Term Memory layers, Dense layers, and Connectionist Temporal Classification as a loss function. To improve the performance of the system with the limited size of the dataset, we present a combined language model (Word-level language Model and Character-level language model)-based prefix decoding technique and Bidirectional Encoder Representations from Transformers-based post-processing technique. To gain key insights from our Automatic Speech Recognition (ASR) system, we used the inferences from the system and proposed different analysis methods. These insights help us in understanding and improving the ASR system as well as provide intuition into the language used for the ASR system. We have trained the model on the Microsoft Speech Corpus, and we observe a 5.87% decrease in Word Error Rate (WER) with respect to base-model WER.

Nowadays, automatic speech recognizers have become quite goodin recognizing well prepared fluent speech (e.g. news readings).However, the recognition of spontaneous speech is still problem-atic. Some important reasons for this are that spontaneous speechis usually less articulated and contains a lot of disfluencies. In thispaper, a new methodology for coping with disfluencies is presentedand evaluated. The basic idea is to detect disfluencies and to deter-mine the nature of these disfluencies prior to the recognition, and touse that information to control/modify the search. At present, themethodology has been elaborated for filled pauses (FP) and wordrepetitions (WR). It enables us to eliminate about one associated normal word error per disfluency without introducing a significantaugmentation of the computational load. 1. Introduction The most frequently occurring disfluencies in spontaneous speechare filled pauses (FPs). If one has a sufficiently large spontaneousspeech corpus with FP transcriptions available, one can create ded-icated acoustic FP models to supplement the traditional triphonemodel set. One can also introduce lexical FP models (pronuncia-tion variants in the lexicon). In [7], a combination of acoustic andlexical FP models yielded a reduction of the word error rate (WER)of 7.8 % relative for medical dictation (with respect to a baselinesystem with one FP model in the lexicon).Even when good FP models are available, FPs may still con-fuse the recognizer because they lead to word sequences (-grams)that are not observed in the kind of written material that is normallyused for language model (LM) development. Extending this mate-rialwithspontaneous speech transcriptionswould beanother meansof improving a spontaneous speech recognizer. However, the nec-essary resources to accomplish this for a general task are usuallynot available (and certainly not for Dutch). For a limited domainlike medical dictation, the situation is different, and experiments byPakhomov [4]showed thatthetechnique can work : the WERcouldbe reduced from 32.2 to 28.5 %.A simple way of dealing with FPs at the level of the LM is toignore the FP in the LM context of the next word. However, Stol-cke and Shriberg [8] come to the surprising conclusion that discard-ing FPs from the trigram context actually increases the perplexity.However, they looked at the speech stretches that were supplied totherecognizer and thatweredelimitedonthebasisofacoustic crite-ria. The FPs occurring at sentence boundaries often appeared in themiddle of such stretches. By only discarding sentence internal FPsthe perplexity did decrease, as expected. In [5] the speech stretchescorresponded to sentences and all FPs were sentence internal. Forthis material, the discarding strategy resulted in a 4 % decrease ofthe overall perplexity and a 30 % decrease of the perplexity of theword after an FP.The problems with the discarding of FPs in the LM are that thenotion of a sentence is not always clear in spontaneous speech, andthat even when it is, the sentence boundaries only become availableafter the recognition (with FPs) has been accomplished.Apart from FPs, there are other important disfluency types inspontaneous speech, such as the word repetition (WR). It is definedas an event consisting of a completed (not interrupted) word (W)and a repetition (R) of that word. Clearly, WRs do not require anynew acoustic or lexical models. However, just like FPs, they maydisturb the recognizer at the level of the language model.In this paper we investigate adaptive search methodologies thataim at improving the recognition of spontaneous speech in thevicinity of FPsand WRs without affecting the performance in otherregions. In section 2 we briefly outline the basic principles of thenew adaptive approach and we further elaborate and evaluate it insections 3 and 4.

The acoustic-to-word model based on the Connectionist Temporal Classification (CTC) criterion is a natural end-to-end (E2E) system directly targeting word as output unit. Two issues exist in the system: first, the current output of the CTC model relies on the current input and does not account for context weighted inputs. This is the hard alignment issue. Second, the word-based CTC model suffers from the out-of-vocabulary (OOV) issue. This means it can model only frequently occurring words while tagging the remaining words as OOV. Hence, such a model is limited in its capacity in recognizing only a fixed set of frequent words. In this study, we propose addressing these problems using a combination of attention mechanism and mixed-units. In particular, we introduce Attention CTC, Self-Attention CTC, Hybrid CTC, and Mixed-unit CTC. First, we blend attention modeling capabilities directly into the CTC network using Attention CTC and Self-Attention CTC. Second, to alleviate the OOV issue, we present Hybrid CTC which uses a word and letter CTC with shared hidden layers. The Hybrid CTC consults the letter CTC when the word CTC emits an OOV. Then, we propose a much better solution by training a Mixed-unit CTC which decomposes all the OOV words into sequences of frequent words and multi-letter units. Evaluated on a 3400 hours Microsoft Cortana voice assistant task, our final acoustic-to-word solution using attention and mixed-units achieves a relative reduction in word error rate (WER) over the vanilla word CTC by 12.09\%. Such an E2E model without using any language model (LM) or complex decoder also outperforms a traditional context-dependent (CD) phoneme CTC with strong LM and decoder by 6.79% relative.

Sequence-to-sequence attention-based models on subword units allow simple open-vocabulary end-to-end speech recognition. In this work, we show that such models can achieve competitive results on the Switchboard 300h and LibriSpeech 1000h tasks. In particular, we report the state-of-the-art word error rates (WER) of 3.54% on the dev-clean and 3.82% on the test-clean evaluation subsets of LibriSpeech. We introduce a new pretraining scheme by starting with a high time reduction factor and lowering it during training, which is crucial both for convergence and final performance. In some experiments, we also use an auxiliary CTC loss function to help the convergence. In addition, we train long short-term memory (LSTM) language models on subword units. By shallow fusion, we report up to 27% relative improvements in WER over the attention baseline without a language model.

We have constructed a new speech data corpus consisting of the utterances of 221 elderly Japanese people (average age: 79.2) with the aim of improving the accuracy of automatic speech recognition (ASR) for the elderly. ASR is a beneficial modality for people with impaired vision or limited hand movement, including the elderly. However, speech recognition systems using standard recognition models, especially acoustic models, have been unable to achieve satisfactory performance for the elderly. Thus, creating more accurate acoustic models of the speech of elderly users is essential for improving speech recognition for the elderly. Using our new corpus, which includes the speech of elderly people living in three regions of Japan, we conducted speech recognition experiments using a variety of DNN-HNN acoustic models. As training data for our acoustic models, we examined whether a standard adult Japanese speech corpus (JNAS), an elderly speech corpus (S-JNAS) or a spontaneous speech corpus (CSJ) was most suitable, and whether or not adaptation to the dialect of each region improved recognition results. We adapted each of our three acoustic models to all of our speech data, and then re-adapt them using speech from each region. Without adaptation, the best recognition results were obtained when using the S-JNAS trained acoustic models (total corpus: 21.85% Word Error Rate). However, after adaptation of our acoustic models to our entire corpus, the CSJ trained models achieved the lowest WERs (entire corpus: 17.42%). Moreover, after readaptation to each regional dialect, the CSJ trained acoustic model with adaptation to regional speech data showed tendencies of improved recognition rates. We plan to collect more utterances from all over Japan, so that our corpus can be used as a key resource for elderly speech recognition in Japanese. We also hope to achieve further improvement in recognition performance for elderly speech.

Japanese large-vocabulary continuous-speech recognition using a newspaper corpus and broadcast news