Speech-Driven Visitor Notification System Using Telegram Bot and Voice Activity Detection for Real-Time Retail Applications

This paper compares the performance of five commercial speech recognition APIs under noisy environments, namely those provided by Amazon AWS, Microsoft Azure, Google, Kakao, and Naver. To this end, we used an open dataset for development and evaluation of multi-channel noise processing technology provided in AI Hub. We tested each API's performance with respect to the speaker's gender and location and the speech content, and measured their error rate using both word error rate (WER) and character error rate (CER). Except for the AWS API, the error rate was higher when tested with female's data than male's one, and when tested with the data recorded from the side than the front. The error rate was also relatively high when the test sentences contained proper nouns such as person's names and local names, and the shorter the sentences, the higher the error rate. Moreover, the Google API outperformed all the others in terms of both WER and CER, with 53% and 18% of error rate, respectively.

State-of-the-art multilingual Automatic Speech Recognition (ASR) models produce systematic errors when applied to low-resource languages like Rajasthani, for which they lack dedicated training data. This article addresses this challenge by introducing a post-ASR correction framework that leverages the complementary error patterns in the outputs (termed as views) from two distinct models: Whisper-large-v3 and MMS-1B-All. We propose a multi-view, 1 character-level sequence-to-sequence (Seq2Seq) model that uses a gated fusion mechanism to dynamically weigh information from the two ASR outputs. On a new benchmark created from the IndicTTS Rajasthani corpus, our gated model achieves a Character Error Rate (CER) of 7.86% and a Word Error Rate (WER) of 29.98%. This outperforms the best single-view baselines (8.01% CER and 30.33% WER), simple multi-view concatenation (8.21% CER and 30.05% WER), as well as Llama-3.2-3B and mBART-50-large, both fine-tuned on Whisper and MMS inputs. It also surpasses powerful Large Language Models (LLMs) like GPT-4o and Gemini 2.5 Pro in a zero-shot setting. This work establishes the first baseline for post-ASR correction in Rajasthani, demonstrating that a compact, specialized model is more effective than general-purpose LLMs for this targeted, low-resource task.

This work presents a dialect-aware and noise-robust Kannada automatic speech recognition (ASR) system that bridges the gap between low-resource linguistic contexts and state-of-the-art deep learning models. We design a two-stage approach: (i) a scratch-built convolutional neural network (CNN)–Transformer hybrid trained on curated Kannada speech data with fast Fourier transform-based noise reduction and (ii) fine-tuning OpenAI’s Whisper-small model on a dialect-diverse corpus. The proposed pipeline integrates adaptive noise suppression, subword tokenization, and beam-search decoding to handle agglutinative morphology, speaker variation, and environmental noise. Extensive experiments were conducted on two datasets: the Few-shot Learning Evaluation of Universal Representations of Speech (FLEURS) multilingual Kannada subset and a curated 150-sample, Custom-collected Kannada speech dataset covering both formal and conversational speech. On the FLEURS test set (≈ 2.5 hours), our fine-tuned Whisper model achieves a word error rate (WER) of 0.15, a character error rate (CER) of 0.255, and a BLEU score of 0.912, representing a 53% relative reduction in WER and 36% reduction in CER compared to the scratch CNN–Transformer baseline. For the customized dataset, the fine-tuned Whisper model achieves a WER of 0.2311 and a CER of 0.0453, outperforming Google Speech-to-Text Application Programming Interface (API) by 16.8% (relative WER reduction) and surpassing the scratch transformer by over 70% in WER. We further evaluate robustness under dialectal variation and noisy recordings, providing detailed error analysis and computational efficiency metrics. To our knowledge, this is the first comprehensive evaluation of Whisper fine-tuning for Kannada, demonstrating its viability for real-time, edge-deployable applications in education, accessibility, and public administration.

The ability to discriminate between talkers based on their fundamental (F0) and formant frequencies can facilitate speech comprehension in multi-talker environments. To date, voice discrimination (VD) of children and adults has only been tested in quiet conditions. This study examines the effect of speech-shaped noise on the use of F0 only, formants only, and the combined F0 + formant cues for VD. A total of 24 adults (18–35 years) and 16 children (7–10 years) underwent VD threshold assessments in quiet and noisy environments with the tested cues. Thresholds were obtained using a three-interval, three-alternative, two-down, one-up adaptive procedure. The results demonstrated that noise negatively impacted the utilization of formants for VD. Consequently, F0 became the lead cue for VD for the adults in noisy environments, whereas the formants were the more accessible cue for VD in quiet environments. For children, however, both cues were poorly utilized in noisy environments. The finding that robust cues such as formants are not readily available for VD in noisy conditions has significant clinical implications. Specifically, the reliance on F0 in noisy environments highlights the difficulties that children encounter in multi-talker environments due to their poor F0 discrimination and emphasizes the importance of maintaining F0 cues in speech-processing strategies tailored for hearing devices.

This paper presents a progress report on a relatively difficult ASR task on a spontaneous speech corpus - Mandarin Conversational Dialogue Corpus (MCDC). A DNN-based acoustic model is constructed based on the CLDNN structure with a large dataset that comprises two spontaneous-speech corpora and one read-speech corpus. The study uses a large text dataset formed by seven corpora to train an efficient general language model (LM). Two adapted LMs specially for spontaneous speech recognition are also constructed. Experimental results showed that the best performances of 26.3% in character error rate (CER) and 32.5% in word error rate (WER) were reached on MCDC. They represented 27.9% and 22.2% of relative CER and WER reductions as compared with the performances by the previous best HMM-based method. This confirms that the proposed method is promising in tackling on Mandarin spontaneous speech recognition.

This study explores the feasibility of constructing a small-scale speech recognition system capable of competing with larger, modern automated speech recognition (ASR) systems in both performance and word error rate (WER). Our central hypothesis posits that a compact transformer-based ASR model can yield comparable results, specifically in terms of WER, to traditional ASR models while challenging contemporary ASR systems that boast significantly larger computational sizes. The aim is to extend ASR capabilities to under-resourced languages with limited corpora, catering to scenarios where practitioners face constraints in both data availability and computational resources. The model, comprising a compact convolutional neural network (CNN) and transformer architecture with 2.214 million parameters, challenges the conventional wisdom that large-scale transformer-based ASR systems are essential for achieving high accuracy. In comparison, contemporary ASR systems often deploy over 300 million parameters. Trained on a modest dataset of approximately 3000 h – significantly less than the 50,000 h used in larger systems – the proposed model leverages the Common Voice and LibriSpeech datasets. Evaluation on the LibriSpeech test-clean and test-other datasets produced character error rates (CERs) of 6.40% and 16.73% and WERs of 16.03% and 35.51% respectively. Comparisons with existing architectures showcase the efficiency of our model. A gated recurrent unit (GRU) architecture, albeit achieving lower error rates, incurred a computational cost 24 times larger than our proposed model. Large-scale transformer architectures, while achieving marginally lower WERs (2%–4% on LibriSpeech test-clean), require 200 times more parameters and 53,000 additional hours of training data. Modern large language models are used to improve the WERs, but require large computational resources. To further enhance performance, a small 4-g language model was integrated into our end-to-end ASR model, resulting in improved WERs. The overarching goal of this work is to provide a practical solution for practitioners dealing with limited datasets and computational resources, particularly in the context of under-resourced languages.

Automatic speech recognition (ASR) is a vital technology that transforms spoken language into written text, facilitating effective accessibility and communication. Despite the ongoing development of deep learning approaches, speech recognition remains a formidable task, especially for languages with limited data resources, such as Tamil. This work presents the development of an ASR system by utilizing the real-time spontaneous Tamil speech data collected from various types of people's communications in public places. The corpus is trained by fine-tuning the pre-trained wav2vec2 XLSR model. This model captures the diverse acoustic features and patterns and even applied to multiple dialects, making it adaptable to real-world speech. The implemented model is evaluated on various noisy environments like markets, hospitals, shops, etc. In terms of various evaluation metrics such as word error rate (WER) and character error rate (CER), the designed model exhibits an optimal performance by achieving a lower error rate when compared to the baseline ASR models.

Speech foundation models trained at a massive scale, both in terms of model and data size, result in robust systems capable of performing multiple speech tasks, including automatic speech recognition (ASR).These models transcend language and domain barriers, yet effectively measuring their performance remains a challenge.Traditional metrics like word error rate (WER) and character error rate (CER) are commonly used to evaluate ASR performance but often fail to reflect transcription quality in critical contexts, particularly when detecting fabricated outputs.This phenomenon, known as hallucination, is especially concerning in high-stakes domains such as healthcare, legal, and aviation, where errors can have severe consequences.In our work, we address this gap by investigating hallucination in ASR models.We examine how factors such as distribution shifts, model size, and model architecture influence the hallucination error rate (HER), a metric we introduce to quantify hallucinations.Our analysis of over 20 ASR models reveals three key insights: (1) High WERs can mask low hallucination rates, while low WERs may conceal dangerous hallucinations.(2) Synthetic noise, both adversarial and common perturbations like white noise, pitch shift, and time stretching, increase HER.(3) Distribution shift correlates strongly with HER ( = 0.91).Our findings highlight the importance of incorporating HER alongside traditional metrics like WER to better assess ASR model performance, particularly in high-stakes domains.

Noun dropping and mis-translations occasionally occurs with Machine Translation (MT) output. These errors can cause communication problems between system users. Some of the MT architectures are able to incorporate bilingual noun lexica, which can improve the translation quality of sentences which include nouns. In this paper, we proposed an automatic method to enable a monolingual user to add new words to the lexicon. In the experiments, we compare the proposed method to three other methods. According to the experimental results, the proposed method gives the best performance in both point of view of Character Error Rate (CER) and Word Error Rate (WER). The improvement from using only a transliteration system is very large, about 13 points in CER and 32 points in WER.KeywordsMachine TranslationStatistical Machine TranslationComputational LinguisticsWord Error RateBilingual LexiconThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we introduce the system developed at the Institute for Infocomm Research (I2 R) for the ASpIRE (Automatic Speech recognition In Reverberant Environments) challenge. The main components of the system are a front-end processing system consisting of a distributed beam-forming algorithm, that performs adaptive weighting and channel elimination, a speech dereverberation approach using a maximum-kurtosis criteria, and a robust voice activity detection (VAD) module based on using the sub-harmonic ratio (SHR). The acoustic back-end consists of a multi-conditional Deep Neural Network (DNN) model that uses speaker adapted features combined with a decoding strategy that performs semi-supervised DNN model adaptation using weighted labels generated by the first-pass decoding output. On the single-microphone evaluation, our system achieved a word error rate (WER) of 44.8%. With the incorporation of beamforming on the multi-microphone evaluation, our system achieved an improvement in WER of over 6% to give the best evaluation result of 38.5%.

Background and ObjectivesThe harmful effects of frequent exposure to loud sounds through portable music players (PMPs) in combination with earphones have been suggested to result in a high prevalence of recreational noise-induced hearing loss among children, adolescents, and young adults. The present study aimed to evaluate the effects of an active noise control technology applied to earphones on the preferred listening levels (PLLs) while listening to music in the presence of background noise. Subjects and MethodsTwenty-three adults between 20 and 40 years with normal hearing were recruited for this study. PLLs for listening to pop-rock and classical music were measured in the participants’ ear canal with a commercially available PMP for four earphone/headphone configurations in quiet and noisy conditions. Ear canal insertion loss was measured in open ear conditions as well as earphone/headphone conditions. ResultsThe average PLL while using earphones and headphones exceeded 85 dBA corresponding to the sound level to induce hearing damage, but in the case of canal earphones with noise cancelling (NC), it was below 75 dBA, corresponding to potentially harmful levels. The background subway noise significantly increased the PLL measured in quiet conditions using any four earphone/headphone types except canal earphones with NC. Canal earphones with NC showed the lowest PLLs compared with participants’ average PLLs using the other three earphone/headphone types. ConclusionsTo minimize recreational noise exposure at the risk of PMP use, the use of earphones with NC is recommended in noisy environments.

The present study aimed to examine the involvement of listening effort among multilinguals in their first (L1) and second (L2) languages in quiet and noisy listening conditions and investigate how the presence of a constraining context within sentences influences listening effort. A group of 46 young adult Arabic (L1)-Hebrew (L2) multilinguals participated in a listening task. This task aimed to assess participants' perceptual performance and the effort they exert (as measured through pupillometry) while listening to single words and sentences presented in their L1 and L2, in quiet and noisy environments (signal to noise ratio = 0 dB). Listening in quiet was easier than in noise, supported by both perceptual and pupillometry results. Perceptually, multilinguals performed similarly and reached ceiling levels in both languages in quiet. However, under noisy conditions, perceptual accuracy was significantly lower in L2, especially when processing sentences. Critically, pupil dilation was larger and more prolonged when listening to L2 than L1 stimuli. This difference was observed even in the quiet condition. Contextual support resulted in better perceptual performance of high-predictability sentences compared with low-predictability sentences, but only in L1 under noisy conditions. In L2, pupillometry showed increased effort when listening to high-predictability sentences compared with low-predictability sentences, but this increased effort did not lead to better understanding. In fact, in noise, speech perception was lower in high-predictability L2 sentences compared with low-predictability ones. The findings underscore the importance of examining listening effort in multilingual speech processing and suggest that increased effort may be present in multilingual's L2 within clinical and educational settings.

Modern automatic vehicle number-plate recognition models require a large corpora of labelled examples as training dataset. Recent advances in data-generation methods have addressed this challenge by leveraging auto-encoders (Yang et al. in 2015 8th International Symposium on Computational Intelligence and Design (ISCID). pp. 465–469, 2015 [1]) and adversarial networks (Kukreja et al. in 2020 4th International Conference on Electronics, Communication and Aerospace Technology (ICECA). pp. 1190–1195, 2020 [2]); (Han et al. in Appl. Sci. 10:2780, 2020 [3]); Liu et al. in Proceedings of the 25th ACM International Conference on Multimedia (MM ’17). Association for Computing Machinery, New York, NY, USA, pp. 1618–1626, 2017 [4]) to model the latent space of vehicle number-plate images based on unlabeled real images. As per our knowledge, however, there is no systematic study assessing efficacy of such methods for Indian vehicle number plates. While the usage of deep learning models for generation of training data has been reported with success, it further limits the interpretability of the overall solution by adding another deep neural network in the ANPR system pipeline (Linardatos et al. in Entropy 23:18, 2021[5]). The objective of this work, therefore, is to address these challenges by reporting promising evidences from an ablation study focused on understanding the impact of a simple image-generation control parameter like text font over the performance of convolutional recurrent neural network-based (CRNN) Indian vehicle number-plate recognition model. The design of our study is guided by initial experiments that suggested that merely exposing CRNN to variations in vehicle number text font alone in the synthetic training data cannot provide generalization over real images as we observed 100% mean word-error rates (WERs) and mean character-error rates (CER) regardless of the size of training data. We have, therefore, maintained a uniform range of rotation, translation, and Gaussian blurring across all of our reported experiments in the ablation study. We have created a test dataset of 100 real images drawn from an ANPR deployment site camera and 350 images under unconstrained environment setting available from the Kaggle platform. We observed that the lack of text font variety severely affected the performance of CRNN, rendering a mean WER and CER of 100% on test data, whereas increasing the number of fonts to even 10 classes enhanced the performance to upto 10% mean WER and 1.62% mean CER on constrained dataset and around 50% mean WER and 15.5% mean CER on unconstrained dataset. Our results show that it is possible to achieve a substantial performance headstart using conventional synthetic image-generation methods alone.

Development of a code-switched Hindi-Marathi dataset and transformer-based architecture for enhanced speech recognition using dynamic switching algorithms

In this work, we tackle the Diacritic Restoration (DR) task for Arabic dialectal sentences using a multimodal approach that combines both textual and speech information.We propose a model that represents the text modality using an encoder extracted from our own pretrained model named CATT.The speech component is handled by the encoder module of the OpenAI Whisper base model.Our solution is designed following two integration strategies.The former consists of fusing the speech tokens with the input at an early stage, where the 1500 frames of the audio segment are averaged over 10 consecutive frames, resulting in 150 speech tokens.To ensure embedding compatibility, these averaged tokens are processed through a linear projection layer prior to merging them with the text tokens.Contextual encoding is guaranteed by the CATT encoder module.The latter strategy relies on cross-attention, where text and speech embeddings are fused.The cross-attention output is then fed to the CATT classification head for token-level diacritic prediction.To further improve model robustness, we randomly deactivate the speech input during training, allowing the model to perform well with or without speech.Our experiments show that the proposed approach achieves a word error rate (WER) of 0.25 and a character error rate (CER) of 0.9 on the development set.On the test set, our model achieved WER and CER scores of 0.55 and 0.13, respectively.