Multi-microphone speech recognition integrating beamforming, robust feature extraction, and advanced DNN/RNN backend

This study explores the feasibility of constructing a small-scale speech recognition system capable of competing with larger, modern automated speech recognition (ASR) systems in both performance and word error rate (WER). Our central hypothesis posits that a compact transformer-based ASR model can yield comparable results, specifically in terms of WER, to traditional ASR models while challenging contemporary ASR systems that boast significantly larger computational sizes. The aim is to extend ASR capabilities to under-resourced languages with limited corpora, catering to scenarios where practitioners face constraints in both data availability and computational resources. The model, comprising a compact convolutional neural network (CNN) and transformer architecture with 2.214 million parameters, challenges the conventional wisdom that large-scale transformer-based ASR systems are essential for achieving high accuracy. In comparison, contemporary ASR systems often deploy over 300 million parameters. Trained on a modest dataset of approximately 3000 h – significantly less than the 50,000 h used in larger systems – the proposed model leverages the Common Voice and LibriSpeech datasets. Evaluation on the LibriSpeech test-clean and test-other datasets produced character error rates (CERs) of 6.40% and 16.73% and WERs of 16.03% and 35.51% respectively. Comparisons with existing architectures showcase the efficiency of our model. A gated recurrent unit (GRU) architecture, albeit achieving lower error rates, incurred a computational cost 24 times larger than our proposed model. Large-scale transformer architectures, while achieving marginally lower WERs (2%–4% on LibriSpeech test-clean), require 200 times more parameters and 53,000 additional hours of training data. Modern large language models are used to improve the WERs, but require large computational resources. To further enhance performance, a small 4-g language model was integrated into our end-to-end ASR model, resulting in improved WERs. The overarching goal of this work is to provide a practical solution for practitioners dealing with limited datasets and computational resources, particularly in the context of under-resourced languages.

In this paper, the improvements in the recently implemented Kannada speech recognition system is demonstrated in detail. The Kannada automatic speech recognition (ASR) system consists of ASR models which are created by using Kaldi, IVRS call flow and weather and agricultural commodity prices information databases. The task specific speech data used in the recently developed spoken dialogue system had high level of different background noises. The different types of noises present in collected speech data had an adverse effect on the on line and off line speech recognition performances. Therefore, to improve the speech recognition accuracy in Kannada ASR system, a noise reduction algorithm is developed which is a fusion of spectral subtraction with voice activity detection (SS-VAD) and minimum mean square error spectrum power estimator based on zero crossing (MMSE-SPZC) estimator. The noise elimination algorithm is added in the system before the feature extraction part. An alternative ASR models are created using subspace Gaussian mixture models (SGMM) and deep neural network (DNN) modeling techniques. The experimental results show that, the fusion of noise elimination technique and SGMM/DNN based modeling gives a better relative improvement of 7.68% accuracy compared to the recently developed GMM-HMM based ASR system. The least word error rate (WER) acoustic models could be used in spoken dialogue system. The developed spoken query system is tested from Karnataka farmers under uncontrolled environment.

Code-switching is a commonly occurring phenomenon in many multilingual communities, wherein a speaker switches between languages within a single utterance. Conventional Word Error Rate (WER) is not sufficient for measuring the performance of code-mixed languages due to ambiguities in transcription, misspellings and borrowing of words from two different writing systems. These rendering errors artificially inflate the WER of an Automated Speech Recognition (ASR) system and complicate its evaluation. Furthermore, these errors make it harder to accurately evaluate modeling errors originating from code-switched language and acoustic models. In this work, we propose the use of a new metric, transliteration-optimized Word Error Rate (toWER) that smoothes out many of these irregularities by mapping all text to one writing system and demonstrate a correlation with the amount of code-switching present in a language. We also present a novel approach to acoustic and language modeling for bilingual code-switched Indic languages using the same transliteration approach to normalize the data for three types of language models, namely, a conventional n-gram language model, a maximum entropy based language model and a Long Short Term Memory (LSTM) language model, and a state-of-the-art Connectionist Temporal Classification (CTC) acoustic model. We demonstrate the robustness of the proposed approach on several Indic languages from Google Voice Search traffic with significant gains in ASR performance up to 10% relative over the state-of-the-art baseline.

Over the past few decades, research has focused heavily on automatic speech recognition (ASR). Although ASR for a few languages is close to reality; ASR for the low resource languages like Malayalam is still in its infancy. Here, in this work, the authors discuss the experiment conducted on accented data of Malayalam speech using two approaches. One approach uses the spectral features for modeling using deep convolutional neural network and the other uses the influential features of speech signals for modeling using LSTM-RNN approach. The proposed methodology comprises three distinct stages; dataset preparation, feature extraction, classification, and hence the construction of deep learning models that recognize the accent-based spoken sentences in the Malayalam language. Mel-frequency cepstral coefficient (MFCC) algorithm, short-term Fourier transform (STFT), and mel spectrogram methodologies are used for feature engineering and hence the features that represent the speech signals are used for constructing the accented ASR system for the Malayalam language using long short-term memory (LSTM) a recurrent neural network (RNN). The spectrogram dataset has been constructed for the speech dataset and used for constructing the ASR model with deep convolutional neural network (DCNN). The result of the experiment shows that LSTM-based RNN outperforms DCNN for the proposed dataset that has been constructed for the experiment in the natural recording environment.

This paper reports the development of a multi-genre Urdu Broadcast (BC) corpus and a Large Vocabulary Continuous Speech Recognition (LVCSR) system. BC speech corpus of 98 hours from 453 speakers is collected and annotated. For acoustic modeling, Time-delay Neural Network (TDNN) is developed with prior Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) training and alignments. For the language model, 3-gram, 4-gram and Recurrent Neural Network (RNN) based models are developed on a text corpus of 188 million words. The developed models are tested on 4.3 hours of unseen BC multi-genre speech dataset and the best Word Error Rate (WER) 18.59% is achieved using RNN based Language Model (LM). Moreover, a detailed word error analysis is carried out to compare the errors made by humans and the Automatic Speech Recognition (ASR) System. The results showed a similar behavior of word misrecognitions by both humans and ASR.

In this work, we propose recurrent deep neural networks (DNNs) for robust automatic speech recognition (ASR). Full recurrent connections are added to certain hidden layer of a conventional feedforward DNN and allow the model to capture the temporal dependency in deep representations. A new backpropagation through time (BPTT) algorithm is introduced to make the minibatch stochastic gradient descent (SGD) on the proposed recurrent DNNs more efficient and effective. We evaluate the proposed recurrent DNN architecture under the hybrid setup on both the 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nd</sup> CHiME challenge (track 2) and Aurora-4 tasks. Experimental results on the CHiME challenge data show that the proposed system can obtain consistent 7% relative WER improvements over the DNN systems, achieving state-of-the-art performance without front-end preprocessing, speaker adaptive training or multiple decoding passes. For the experiments on Aurora-4, the proposed system achieves 4% relative WER improvement over a strong DNN baseline system.

This study developed a Speech-to-Text (STT) system based on the Long Short-Term Memory (LSTM) model to recognize and convert speech in the Malang Walikan language into text. The Malang Walikan language has a unique linguistic structure in the form of word reversal, which poses a challenge in speech recognition. The data used consisted of 1,000 sentences collected from social media and direct recordings. The data was processed using Mel Frequency Cepstral Coefficients (MFCC) and then used to train the LSTM model.The system's performance was evaluated using the Word Error Rate (WER), Character Error Rate (CER), and Average Test Loss metrics. The best results obtained showed a WER value of 1.0 on a 699:300 data split, a CER of 0.78 on a 799:200 split, and an Average Test Loss of 11.0147 on a 299:700 split.The high Average Test Loss value indicates the model's difficulty in minimizing prediction errors, which may be caused by the model's mismatch with the data patterns or overfitting. To improve the model's performance, it is recommended to improve the quality of the training data, optimize the parameters, and apply regularization techniques.

In this paper we present the latest improvements to the Russian spontaneous speech recognition system developed in Speech Technology Center (STC). Significant word error rate (WER) reduction was obtained by applying hypothesis rescoring with sophisticated language models. These were the Recurrent Neural Network Language Model and regularized Long-Short Term Memory Language Model. For acoustic modeling we used the deep neural network (DNN) trained with speaker-dependent bottleneck features, similar to our previous system. This DNN was combined with the deep Bidirectional Long Short-Term Memory acoustic model by the use of score fusion. The resulting system achieves WER of 16.4 %, with an absolute reduction of 8.7 % and relative reduction of 34.7 % compared to our previous system result on this test set.

Modeling the speaker variability is a key challenge for automatic speech recognition (ASR) systems. In this paper, the learning hidden unit contributions (LHUC) based adaptation techniques with compact speaker dependent (SD) parameters are used to facilitate both speaker adaptive training (SAT) and unsupervised test-time speaker adaptation for end-to-end (E2E) lattice-free MMI (LF-MMI) models. An unsupervised model-based adaptation framework is proposed to estimate the SD parameters in E2E paradigm using LF-MMI and cross entropy (CE) criterions. Various regularization methods of the standard LHUC adaptation, e.g., the Bayesian LHUC (BLHUC) adaptation, are systematically investigated to mitigate the risk of overfitting, on E2E LF-MMI CNN-TDNN and CNN-TDNN- BLSTM models. Lattice-based confidence score estimation is used for adaptation data selection to reduce the supervision label uncertainty. Experiments on the 300-hour Switchboard task suggest that, applying BLHUC in the proposed unsupervised E2E adaptation framework to byte pair encoding (BPE) based E2E LF-MMI systems consistently outperformed the baseline systems by relative word error rate (WER) reductions up to 10.5% and 14.7% on the NIST Hub5’00 and RT03 evaluation sets, and achieved the best performance in WERs of 9.0% and 9.7%, respectively. These results are comparable to the results of state-of-the-art adapted LF-MMI hybrid systems and adapted Conformer-based E2E systems.

Recently, artificial intelligence and data science have witnessed dramatic progress and rapid growth, especially Automatic Speech Recognition (ASR) technology based on Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs). Consequently, new end-to-end Recurrent Neural Network (RNN) toolkits were developed with higher speed and accuracy that can often achieve a Word Error Rate (WER) below 10%. These toolkits can nowadays be deployed, for instance, within aircraft cockpits and Air Traffic Control (ATC) systems in order to identify aircraft and display recognized voice messages related to flight data, especially for airports not equipped with radar. Hence, the performance of air traffic controllers and pilots can ultimately be improved by reducing workload and stress and enforcing safety standards. Our experiment conducted at Tangier’s International Airport ATC aimed to build an ASR model that is able to recognize aircraft call signs in a fast and accurate way. The acoustic and linguistic models were trained on the Ibn Battouta Speech Corpus (IBSC), resulting in an unprecedented speech dataset with approved transcription that includes real weather aerodrome observation data and flight information with a call sign captured by an ADS-B receiver. All of these data were synchronized with voice recordings in a structured format. We calculated the WER to evaluate the model’s accuracy and compared different methods of dataset training for model building and adaptation. Despite the high interference in the VHF radio communication channel and fast-speaking conditions that increased the WER level to 20%, our standalone and low-cost ASR system with a trained RNN model, supported by the Deep Speech toolkit, was able to achieve call sign detection rate scores up to 96% in air traffic controller messages and 90% in pilot messages while displaying related flight information from ADS-B data using the Fuzzy string-matching algorithm.

In the field of end-to-end speech recognition technology based on deep learning, CLDNN (Convolutional Long Short-Term Memory Fully Connected Deep Neural Network) is a commonly used model structure. The fully connected LSTM (Long Short Term Memory) model is used in the traditional CLDNN structure to process the timing information in the speech signal, which is prone to over-fitting during the training process and affects the learning effect. Deeper models tend to perform better, but increasing the model depth by simply stacking the network layers can cause gradient disappearance, gradient explosion, and degeneration problems. Aiming at the above phenomena and problems, this paper proposes an improved CLDNN structure. It combines the residual network and ConvLSTM to establish the residual ConvLSTM model, and replaces the fully connected LSTM model in the traditional CLDNN structure. The model structure solves the problems of the traditional CLDNN model, and can increase the model depth by stacking residual ConvLSTM blocks without gradient disappearance, gradient explosion and degeneration problems, which makes the speech recognition system perform better. The experimental results show that the model structure has a word error rate (WER) decrease of more than 8% in both Chinese and English speech recognition tasks compared to the traditional CLDNN structure.

In the case of low resource language, there is still the requirement for developing more efficient Automatic Speech Recognition (ASR) systems. In the proposed work, the ASR system is developed for the Gujarati language publicly available dataset. The approach in this paper applies the combination of Mel-frequency Cepstral Coefficients (MFCC) with Constant Q Cepstral Coefficients (CQCC)-based integrated front-end feature extraction techniques. To implement the backend part of the system, hybrid acoustic model is applied. Two-dimensional Convolutional Neural Network (Conv2D) with Bi-directional Gated Recurrent Units-based (BiGRU) backend model is used as the model. To build the ASR system, Connectionist Temporal Classification (CTC) loss function, CTC and prefix-based greedy decoder are also used with the acoustic model. The proposed work shows that the joint MFCC and CQCC feature extraction techniques show the 10–19% improvement in Word Error Rate (WER) as compared to isolated delta-delta features with the available integrated model.

Conventional far-field automatic speech recognition (ASR) systems typically employ microphone array techniques for speech enhancement in order to improve robustness against noise or reverberation. However, such speech enhancement techniques do not always yield ASR accuracy improvement because the optimization criterion for speech enhancement is not directly relevant to the ASR objective. In this work, we develop new acoustic modeling techniques that optimize spatial filtering and long short-term memory (LSTM) layers from multi-channel (MC) input based on an ASR criterion directly. In contrast to conventional methods, we incorporate array processing knowledge into the acoustic model. Moreover, we initialize the network with beamformers' coefficients. We investigate effects of such MC neural networks through ASR experiments on the real-world far-field data where users are interacting with an ASR system in uncontrolled acoustic environments. We show that our MC acoustic model can reduce a word error rate (WER) by~16.5\% compared to a single channel ASR system with the traditional log-mel filter bank energy (LFBE) feature on average. Our result also shows that our network with the spatial filtering layer on two-channel input achieves a relative WER reduction of~9.5\% compared to conventional beamforming with seven microphones.

A statistically designed Automatic Speech Recognition (ASR) system extracts features from speech signals using feature extraction methods, links the extracted features with the expected phonetics of the hypothesis using acoustic models, uses language model to add prior information about the structure of the target language. For many years, Mel-frequency Cepstral coefficients (MFCC), n-gram, and Hidden Markov Model (HMM) approaches have been used predominantly for feature extraction, language modeling and acoustic modeling, respectively. However, performance degradation of MFCC in noisy conditions and inaccuracy of HMMs while handling large vocabularies have made researchers to propose more efficient methods. The proposed work uses noise robust method Gammatone Frequency Cepstral Coefficients(GFCC) for feature extraction, trigram language modeling, and HMM-Gaussian mixture model (GMM) based acoustic modeling to implement a continuous Hindi language ASR system. Also, it applies Differential Evolution (DE) technique to refine the GFCC features and discriminative techniques to enhance performance of the acoustic model. The performance of the implemented system has been evaluated by using different feature extraction methods, variants of n-gram language modeling techniques and different discriminative techniques in clean as well as noisy conditions. Initially, the results reveal that DE optimized GFCC with HMM-Gaussian Mixture Model (GMM) acoustic modeling performs better than MFCC, PLP and MF-PLP feature extraction methods. Secondly, the experimental results show that the Minimum Phone Error (MPE) outperforms Maximum Mutual Information (MMI) and Maximum Likelihood Estimation (MLE) and trigram based language modeling gives more accurate results than unigram and bigram language modeling. Finally, it has been concluded that the continuous Hindi language ASR system implemented using DE refined GFFC feature extraction method with MPE discriminative training technique and trigram based language modeling gives better accuracy in clean as-well-as noisy environments.

Regional languages play an important role in cultural preservation and strengthening ethnic identity. Bugis, one of the most widely spoken regional languages, can be studied through speech recognition technology. This study analyzes the performance of two Deep Learning models, Long Short-Term Memory (LSTM) and Bidirectional Long Short-Term Memory (BiLSTM), for converting Bugis speech into text. The dataset consists of 2,500 audio recordings from native speakers along with their transcripts. The preprocessing stage includes trimming, denoising, and feature extraction using Mel-Frequency Cepstral Coefficients (MFCC). Evaluation using Word Error Rate (WER) and Bilingual Evaluation Understudy (BLEU) shows that BiLSTM outperforms LSTM, achieving a WER of 4.88% and a BLEU of 86.82%, compared to LSTM’s WER of 90.33% and BLEU of 3.86%. These findings indicate that BiLSTM is more effective in capturing contextual information, resulting in more accurate transcriptions. This research contributes to the preservation of the Bugis language and provides insights into the application of Deep Learning in low-resource speech recognition. Future work will explore attention mechanisms and larger datasets to further improve model performance