Post-ASR Correction for Low-Resource Rajasthani Language

This article concerns comparative studies on the Automatic Speech Recognition (ASR) model incorporated with the Large Language Model (LLM) used for medical interviews. The proposed solution is tested on polish language benchmarks and dataset with medical interviews. The latest ASR technologies are based on convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. Most of them work as end-to-end solutions. The presented approach in the case of the Whisper model shows a two-stage solution with End-To-End ASR and LLM working together in a pipeline. The ASR output is an input for LLM. The LLM is a component by which the output from ASR is corrected and improved. Comparative studies for automatic recognition of the Polish language between modern End-To-End deep learning architectures and the ASR hybrid model were performed. The medical interview tests were performed with two state-of-the-art ASR models: OpenAI Whisper incorporated with LLM and Scribe ElevenLabs. Additionally, the results were compared with five more end-to-end models (QuartzNet, FastConformer, Wav2Vec 2.0 XLSR and ESPnet Model Zoo) on Mozilla Common Voice and VoxPopuli databases. Tests were conducted for clean audio signal, signal with bandwidth limitation, and degraded. The tested models were evaluated on the basis of Word Error Rate (WER) and Character Error Rate (CER). The results show that the Whisper model performs by far the best among the open-source models. ElevenLabs Scribe model, on the other hand, performs best for Polish on both general benchmark and medical data.

Development of a code-switched Hindi-Marathi dataset and transformer-based architecture for enhanced speech recognition using dynamic switching algorithms

This study explores the feasibility of constructing a small-scale speech recognition system capable of competing with larger, modern automated speech recognition (ASR) systems in both performance and word error rate (WER). Our central hypothesis posits that a compact transformer-based ASR model can yield comparable results, specifically in terms of WER, to traditional ASR models while challenging contemporary ASR systems that boast significantly larger computational sizes. The aim is to extend ASR capabilities to under-resourced languages with limited corpora, catering to scenarios where practitioners face constraints in both data availability and computational resources. The model, comprising a compact convolutional neural network (CNN) and transformer architecture with 2.214 million parameters, challenges the conventional wisdom that large-scale transformer-based ASR systems are essential for achieving high accuracy. In comparison, contemporary ASR systems often deploy over 300 million parameters. Trained on a modest dataset of approximately 3000 h – significantly less than the 50,000 h used in larger systems – the proposed model leverages the Common Voice and LibriSpeech datasets. Evaluation on the LibriSpeech test-clean and test-other datasets produced character error rates (CERs) of 6.40% and 16.73% and WERs of 16.03% and 35.51% respectively. Comparisons with existing architectures showcase the efficiency of our model. A gated recurrent unit (GRU) architecture, albeit achieving lower error rates, incurred a computational cost 24 times larger than our proposed model. Large-scale transformer architectures, while achieving marginally lower WERs (2%–4% on LibriSpeech test-clean), require 200 times more parameters and 53,000 additional hours of training data. Modern large language models are used to improve the WERs, but require large computational resources. To further enhance performance, a small 4-g language model was integrated into our end-to-end ASR model, resulting in improved WERs. The overarching goal of this work is to provide a practical solution for practitioners dealing with limited datasets and computational resources, particularly in the context of under-resourced languages.

This work presents a dialect-aware and noise-robust Kannada automatic speech recognition (ASR) system that bridges the gap between low-resource linguistic contexts and state-of-the-art deep learning models. We design a two-stage approach: (i) a scratch-built convolutional neural network (CNN)–Transformer hybrid trained on curated Kannada speech data with fast Fourier transform-based noise reduction and (ii) fine-tuning OpenAI’s Whisper-small model on a dialect-diverse corpus. The proposed pipeline integrates adaptive noise suppression, subword tokenization, and beam-search decoding to handle agglutinative morphology, speaker variation, and environmental noise. Extensive experiments were conducted on two datasets: the Few-shot Learning Evaluation of Universal Representations of Speech (FLEURS) multilingual Kannada subset and a curated 150-sample, Custom-collected Kannada speech dataset covering both formal and conversational speech. On the FLEURS test set (≈ 2.5 hours), our fine-tuned Whisper model achieves a word error rate (WER) of 0.15, a character error rate (CER) of 0.255, and a BLEU score of 0.912, representing a 53% relative reduction in WER and 36% reduction in CER compared to the scratch CNN–Transformer baseline. For the customized dataset, the fine-tuned Whisper model achieves a WER of 0.2311 and a CER of 0.0453, outperforming Google Speech-to-Text Application Programming Interface (API) by 16.8% (relative WER reduction) and surpassing the scratch transformer by over 70% in WER. We further evaluate robustness under dialectal variation and noisy recordings, providing detailed error analysis and computational efficiency metrics. To our knowledge, this is the first comprehensive evaluation of Whisper fine-tuning for Kannada, demonstrating its viability for real-time, edge-deployable applications in education, accessibility, and public administration.

Adapting multilingual automatic speech recognition (MASR) models to support new languages is crucial for enhancing global communication accessibility. However, extending these models often leads to catastrophic forgetting, impairing their ability to accurately process previously learned languages. To address this, we present a mixture-of-experts (MoE) strategy that employs Low-Rank Adaptation (LoRA) experts, each dedicated to a specific language. Unlike previous approaches that rely on a small neural network for gating, we utilize the language identification (LID) outputs of the MASR model itself to more precisely activate the appropriate LoRA experts. Our method involves training two sets of LoRAs for LID and ASR, respectively: the LID LoRAs adapt the MASR model for LID across both existing and new languages, and the ASR LoRAs consist of multiple language-specific LoRA experts for ASR. During inference, we first employ the LID LoRAs to identify the language, followed by the activation of the corresponding ASR LoRAs based on this identification. Unfortunately, this approach means LID errors also cause incorrect ASR LoRAs activation. We perform language-wise beam search to allow self-correction for such mistakes. When applied to the Whisper model and integrated with ten new languages from the Common Voice dataset, our approach achieves up to 13.8% and 10.4% relative improvements in word error rate (WER) for new and previously learned languages, respectively. This method effectively reduces catastrophic forgetting with less than 3% additional computation overhead compared to the standard LoRA implementation.

Speech foundation models trained at a massive scale, both in terms of model and data size, result in robust systems capable of performing multiple speech tasks, including automatic speech recognition (ASR).These models transcend language and domain barriers, yet effectively measuring their performance remains a challenge.Traditional metrics like word error rate (WER) and character error rate (CER) are commonly used to evaluate ASR performance but often fail to reflect transcription quality in critical contexts, particularly when detecting fabricated outputs.This phenomenon, known as hallucination, is especially concerning in high-stakes domains such as healthcare, legal, and aviation, where errors can have severe consequences.In our work, we address this gap by investigating hallucination in ASR models.We examine how factors such as distribution shifts, model size, and model architecture influence the hallucination error rate (HER), a metric we introduce to quantify hallucinations.Our analysis of over 20 ASR models reveals three key insights: (1) High WERs can mask low hallucination rates, while low WERs may conceal dangerous hallucinations.(2) Synthetic noise, both adversarial and common perturbations like white noise, pitch shift, and time stretching, increase HER.(3) Distribution shift correlates strongly with HER ( = 0.91).Our findings highlight the importance of incorporating HER alongside traditional metrics like WER to better assess ASR model performance, particularly in high-stakes domains.

The object of this study is a transformer-based ASR architecture trained using an Indonesian speech dataset consisting of audio recordings and corresponding transcripts. This study examines the development of an Automatic Speech Recognition (ASR) system for Indonesian, which is still classified as a low-resource language, particularly in terms of dataset availability and model performance. The problem addressed in this study is the limited performance of the standard transformer model in accurately recognizing Indonesian speech. To overcome this limitation, an encoder modification integrating convolutional and vision transformer (ViT) blocks was proposed and compared with the baseline model. The data were preprocessed through 16 kHz mono audio conversion, silence segmentation, pre-emphasis filtering, log-Mel spectrogram extraction, normalization, and subword tokenization using SentencePiece with byte pair encoding (BPE). The dataset was divided into training, validation, and testing sets with a ratio of 80:10:10, comprising 63,952, 7,994, and 7,994 samples, respectively. Model generalization was improved using the SpecAugment data augmentation technique. The experimental results show that the standard model achieves a word error rate (WER) of 0.162 and a character error rate (CER) of 0.121, while the modified model reduces the WER to 0.158 and the CER to 0.118. The significance of this finding lies in the improved feature representation produced by the modified encoder, where the convolutional block captures local acoustic patterns and the ViT block enhances global context modeling on the spectrogram. This complementary mechanism explains the reduction in errors at the word level, which is crucial for a reliable speech-to-text system. Therefore, the proposed model can be applied to real-time two-way communication in service robot applications

With the rapid development of the gaming industry and the increasing popularity of multiplayer online games, realtime voice communication has become a crucial tool for team collaboration and tactical exchanges. Automatic Speech Recognition (ASR) technology plays a vital role in modern gaming by converting voice commands into text, enabling efficient communication among players. However, existing general-purpose ASR systems face significant challenges in gaming scenarios due to the unique characteristics of in-game communication, such as short phrases, rapid speech, game-specific jargon, and environmental noise. These limitations often lead to frequent recognition errors, increasing communication costs and negatively impacting the overall gaming experience. Furthermore, the scarcity of domainspecific ASR training data exacerbates these issues, hindering system optimization. To address the challenges of ASR systems in gaming scenarios, this study proposes the GO-AEC (Gaming-Oriented ASR Error Correction) framework. The framework leverages the generative capabilities of large language models (LLMs) and employs Retrieval-Augmented Generation (RAG) techniques with a gamespecific knowledge base to better adapt to gaming environments. Additionally, we introduce a data augmentation strategy that combines LLMs with text-to-speech (TTS) techniques to enhance the diversity and robustness of game-specific datasets. The GO-AEC framework consists of three key modules: the data augmentation module, the N-best hypothesis-based LLM correction module, and the dynamic knowledge base module powered by RAG. Experimental results demonstrate that, compared to baseline methods, the proposed framework reduces the character error rate (CER) by 6.22 % and the sentence error rate (SER) by 29.71 %. These findings indicate that the GO-AEC framework effectively addresses the challenges of ASR error correction in gaming scenarios.

Multilingual Automatic Speech Recognition Systems survey. Multilingual speech recognition is a new era of speech and Natural language tool research to facilitate the integrates of many Monolingual system techniques that have been developed using recent methods to address the unique challenge of the multilingual field. What are techniques hidden Markov model (HMM), Gaussian Mixture Model (GMM), Deep Learning (DL) have been used so far and increasing database usage for Multilingual speech processing has become a topic of ongoing interest to a researcher for several years, and the field is now attracting increased attention due to strong drivers of change? Multilingual Speech Recognition. Speech technology is an emerging technology and monolingual automatic speech recognition has made progress in recent years, but many researchers are drawn to multilingual speech recognition. Ideally, speech from different languages should be transcribed by a multilingual speech identifier, which determines both the language used and the series of words spoken by the speaker.

In the retail industry, fast and responsive service is essential for maintaining customer satisfaction and loyalty. A key challenge faced by store owners is delayed responses to customer arrivals, leading to dissatisfaction and potential lost sales. This project develops an automatic notification system using Speech-to-Text technology and a Telegram bot to detect voice keywords and send real-time notifications to store owners. The system was developed using the prototype methodology, allowing for iterative testing and refinement to ensure it met user needs and functional requirements. It integrates the Deepgram API for accurate speech transcription, the Telegram Bot API for notifications, and a web interface for managing keywords and monitoring system status. To enhance efficiency, a Voice Activity Detection (VAD) module was added, ensuring that only human speech is processed, thereby reducing unnecessary processing. Experimental results showed that the system achieved 100% accuracy in quiet environments and 80% in noisy conditions. The system's response time was also impressive, with an average time of 3.72 seconds in quiet conditions and 3.8 seconds in noisy environments. Word Error Rate (WER) and Character Error Rate (CER) evaluations indicated perfect accuracy in quiet conditions (WER 0%, CER 0%) and slight errors in noisy conditions (WER 13.33%, CER 12.5%). Overall, the system effectively improved service speed and responsiveness, offering store owners a valuable tool for enhancing customer experience in retail environments.

Measuring automatic speech recognition (ASR) system quality is critical for creating user-satisfying voice-driven applications. Word Error Rate (WER) has been traditionally used to evaluate ASR system quality; however, it sometimes correlates poorly with user perception/judgement of transcription quality. This is because WER weighs every word equally and does not consider semantic correctness which has a higher impact on user perception. In this work, we propose evaluating ASR output hypotheses quality with SemDist that can measure semantic correctness by using the distance between the semantic vectors of the reference and hypothesis extracted from a pre-trained language model. Our experimental results of 71K and 36K user annotated ASR output quality show that SemDist achieves higher correlation with user perception than WER. We also show that SemDist has higher correlation with downstream Natural Language Understanding (NLU) tasks than WER.

This work explores the use of Large Language Models (LLMs) for post-OCR spelling correction in full sentences across Portuguese, French, and English. Using outputs from a state-of-the-art recognition model on the BRESSAY, RIMES, and IAM datasets, we evaluated two different zero-shot prompts. Closed LLMs, such as Gemini and GPT series, consistently outperform open-source models in reducing Character Error Rate (CER) and Word Error Rate (WER), while also offering faster inference. Despite the good accuracy of open models, their high computational demands hinder their practical use. Code is available at https://github.com/savi8sant8s/zero-shot-spelling-corrector.

Multilingual automatic speech recognition (ASR) systems mostly benefit low resource languages but suffer degradation in performance across several languages relative to their monolingual counterparts. Limited studies have focused on understanding the languages behaviour in the multilingual speech recognition setups. In this paper, a novel data-driven approach is proposed to investigate the cross-lingual acoustic-phonetic similarities. This technique measures the similarities between posterior distributions from various monolingual acoustic models against a target speech signal. Deep neural networks are trained as mapping networks to transform the distributions from different acoustic models into a directly comparable form. The analysis observes that the languages closeness can not be truly estimated by the volume of overlapping phonemes set. Entropy analysis of the proposed mapping networks exhibits that a language with lesser overlap can be more amenable to cross-lingual transfer, and hence more beneficial in the multilingual setup. Finally, the proposed posterior transformation approach is leveraged to fuse monolingual models for a target language. A relative improvement of ~8% over monolingual counterpart is achieved.

Alongside acoustic information, linguistic features based on speech transcripts have been proven useful in Speech Emotion Recognition (SER). However, due to the scarcity of emotion labelled data and the difficulty of recognizing emotional speech, it is hard to obtain reliable linguistic features and models in this research area. In this paper, we propose to fuse Automatic Speech Recognition (ASR) outputs into the pipeline for joint training SER. The relationship between ASR and SER is understudied, and it is unclear what and how ASR features benefit SER. By examining various ASR outputs and fusion methods, our experiments show that in joint ASR-SER training, incorporating both ASR hidden and text output using a hierarchical co-attention fusion approach improves the SER performance the most. On the IEMOCAP corpus, our approach achieves 63.4% weighted accuracy, which is close to the baseline results achieved by combining ground-truth transcripts. In addition, we also present novel word error rate analysis on IEMOCAP and layer-difference analysis of the Wav2vec 2.0 model to better understand the relationship between ASR and SER.

We report insights from translating Spanish conversational telephone speech into English text by cascading an automatic speech recognition (ASR) system with a statistical machine translation (SMT) system. The key new insight is that the informal register of conversational speech is a greater challenge for ASR than for SMT: the BLEU score for translating the reference transcript is 64%, but drops to 32% for translating automatic transcripts, whose word error rate (WER) is 40%. Several strategies are examined to mitigate the impact of ASR errors on the SMT output: (i) providing the ASR lattice, instead of the 1-best output, as input to the SMT system, (ii) training the SMT system on Spanish ASR output paired with English text, instead of Spanish reference transcripts, and (iii) improving the core ASR system. Each leads to consistent and complementary improvements in the SMT output. Compared to translating the 1-best output of an ASR system with 40% WER using an SMT system trained on Spanish reference transcripts, translating the output lattice of a better ASR system with 35% WER using an SMT system trained on ASR output improves BLEU from 32% to 38%.