Salience-driven Contextual Priming of Speech Recognition for Human-Robot Interaction

Combined speech enhancement and auditory modelling for robust distributed speech recognition

Speech recognition in Human-Robot Interaction (HRI) fully relies on audio-based Automatic Speech Recognition. However, speech recognition that relies solely on audio faces significant challenges in noisy environments and may lead to poor performance in such environments. One approach to address this is to also use lipreading in combination with traditional speech recognition. Recent work has shown that audiovisual speech recognition (AVSR) can achieve a Word Error Rate (WER) of only 0.9% on the dataset LRS3. In this paper, we assess the potential of combining audio with lipreading on a social robot platform, Pepper, which has not yet been widely tested for AVSR. Given that prior research has focused on non-robotic domains, it remains unclear whether such models can generalize well to social robot environments. We systematically evaluate and compare the performance of established offline and real-time audiovisual models with their audio-only counterparts. The experiments were conducted in both a controlled laboratory setting and a dynamic and noisy public environment. We evaluated the data using WER and also measured the inference latency of real-time models via Real-Time Factor and Words Per Second rates. The results demonstrate real-time performance for audio-only speech recognition across all latency metrics and near real-time performance for models that combine audio with lipreading. We also explored factors that might influence the inference performance of these models to understand how much video contributes to the audio. This includes factors related to (1) environmental and temporal variations, (2) model behavior, and (3) implementation choices. Our findings indicate that for now the audio-only models outperform the audiovisual models on a social robot platform, in contrast to what has been reported in the benchmarked literature. We conclude that more work is still needed to benefit from lipreading in HRI.

Subspace Gaussian mixture based language modeling for large vocabulary continuous speech recognition

The primary goal of nonlinear frequency compression (NFC) and other frequency-lowering strategies is to increase the audibility of high-frequency sounds that are not otherwise audible with conventional hearing aid (HA) processing due to the degree of hearing loss, limited HA bandwidth, or a combination of both factors. The aim of the present study was to compare estimates of speech audibility processed by NFC with improvements in speech recognition for a group of children and adults with high-frequency hearing loss. Monosyllabic word recognition was measured in noise for 24 adults and 12 children with mild to severe sensorineural hearing loss. Stimuli were amplified based on each listener's audiogram with conventional processing (CP) with amplitude compression or with NFC and presented under headphones using a software-based HA simulator. A modification of the speech intelligibility index (SII) was used to estimate audibility of information in frequency-lowered bands. The mean improvement in SII was compared with the mean improvement in speech recognition. All but 2 listeners experienced improvements in speech recognition with NFC compared with CP, consistent with the small increase in audibility that was estimated using the modification of the SII. Children and adults had similar improvements in speech recognition with NFC. Word recognition with NFC was higher than CP for children and adults with mild to severe hearing loss. The average improvement in speech recognition with NFC (7%) was consistent with the modified SII, which indicated that listeners experienced an increase in audibility with NFC compared with CP. Further studies are necessary to determine whether changes in audibility with NFC are related to speech recognition with NFC for listeners with greater degrees of hearing loss, with a greater variety of compression settings, and using auditory training.

Salient entities are assumed to be more accessible in memory, which makes them more likely to be referred to first and to be referred to with an attenuated expression, such as a pronoun. It is less clear, however, how different types of salience interact in influencing referent accessibility. In this article, we address the question whether non-linguistic factors can affect accessibility in the presence of a linguistic context. We present two story completion experiments in which we investigated the effect of visual salience (foregrounding) in interaction with linguistic salience (subjecthood) of two story characters both on the choice of referent and on the choice of referring expression. In Experiment 1, linguistic salience was moderated by inducing a topic shift in the discourse context. In Experiment 2, contexts in which linguistic salience was unclear were compared to contexts in which one of the characters was highly linguistically salient. The results show that visual salience influences referent choice independently of linguistic salience, but that it does not have an effect on the choice of referring expression. This suggests that visual salience has an influence on the global interpretation of the scene, but does not directly affect the accessibility status of individual entities. This is compatible with a view of language production in which utterance planning is influenced by conceptual and discourse factors rather than by low-level perceptual factors.

In this paper, we evaluate the performance of Automatic speech recognition (ASR) using cepstral-based Smoothed and Thresholded Power Spectrum (STPS-LP) features with residual Temporal Convolution Networks (Res-TCN). The experiments were performed in comparison with cepstral-based LP features, where both features were obtained from AMR bit-stream parameters. In terms of Word Error Rate (WER%), results show that using the STPS-LP cepstral features, the performance of speech recognition was on average 6.22% better in comparison with the conventional LP cepstral features, depending on the type of environmental noise.

This article describes a density ratio approach to integrating external Language Models (LMs) into end-to-end models for Automatic Speech Recognition (ASR). Applied to a Recurrent Neural Network Transducer (RNN-T) ASR model trained on a given domain, a matched in-domain RNN-LM, and a target domain RNN-LM, the proposed method uses Bayes' Rule to define RNN-T posteriors for the target domain, in a manner directly analogous to the classic hybrid model for ASR based on Deep Neural Networks (DNNs) or LSTMs in the Hidden Markov Model (HMM) framework (Bourlard & Morgan, 1994). The proposed approach is evaluated in cross-domain and limited-data scenarios, for which a significant amount of target domain text data is used for LM training, but only limited (or no) {audio, transcript} training data pairs are used to train the RNN-T. Specifically, an RNN-T model trained on paired audio & transcript data from YouTube is evaluated for its ability to generalize to Voice Search data. The Density Ratio method was found to consistently outperform the dominant approach to LM and end-to-end ASR integration, Shallow Fusion.

Code-switching is a commonly occurring phenomenon in many multilingual communities, wherein a speaker switches between languages within a single utterance. Conventional Word Error Rate (WER) is not sufficient for measuring the performance of code-mixed languages due to ambiguities in transcription, misspellings and borrowing of words from two different writing systems. These rendering errors artificially inflate the WER of an Automated Speech Recognition (ASR) system and complicate its evaluation. Furthermore, these errors make it harder to accurately evaluate modeling errors originating from code-switched language and acoustic models. In this work, we propose the use of a new metric, transliteration-optimized Word Error Rate (toWER) that smoothes out many of these irregularities by mapping all text to one writing system and demonstrate a correlation with the amount of code-switching present in a language. We also present a novel approach to acoustic and language modeling for bilingual code-switched Indic languages using the same transliteration approach to normalize the data for three types of language models, namely, a conventional n-gram language model, a maximum entropy based language model and a Long Short Term Memory (LSTM) language model, and a state-of-the-art Connectionist Temporal Classification (CTC) acoustic model. We demonstrate the robustness of the proposed approach on several Indic languages from Google Voice Search traffic with significant gains in ASR performance up to 10% relative over the state-of-the-art baseline.

The use of prior situational/contextual knowledge about a given task can significantly improve automatic speech recognition (ASR) performance. This is typically done through adaptation of acoustic or language models if data is available or using knowledge-based rescoring. The main adaptation techniques, however, are either domain-specific, which makes them inadequate for other tasks, or static and offline, and therefore cannot deal with dynamic knowledge. To circumvent this problem, we propose a real-time system which dynamically integrates situational context into ASR. The context integration is done either post-recognition, in which case a weighted Levenshtein distance between the ASR hypotheses and the context information based on the ASR confidence scores is proposed to extract the most likely sequence of spoken words, or pre-recognition, where the search space is adjusted to the new situational knowledge through adaptation of the finite state machine modeling the spoken language. Experiments conducted on 3 hours of Air Traffic Control (ATC) data achieved a 51% reduction of the Command Error Rate (CmdER) which is used as evaluation metric in the ATC domain.

This research concentrates on the progression and present status of automatic speech recognition systems powered by deep neural networks. It discusses model architectures, training approaches, evaluating model efficacy, and recent advancements specific to deep neural networks applied in automatic speech recognition models. It considers the challenges faced in crafting these speech recognition models, such as data scarcity and the necessity for adaptability. Our exploration traces the evolution of automatic speech recognition through deep neural networks, presenting valuable insights aimed at propelling the domain of speech recognition for diverse applications, spanning from smart devices to healthcare. KEYWORDS - Automatic Speech Recognition, Deep Neural Networks, Language Modeling, Robustness to Noise, Speech Modelling

We propose a method of dynamically registering out-of-vocabulary (OOV) words by assigning the pronunciations of these words to pre-inserted OOV tokens, editing the pronunciations of the tokens. To do this, we add OOV tokens to an additional, partial copy of our corpus, either randomly or to part-of-speech (POS) tags in the selected utterances, when training the language model (LM) for speech recognition. This results in an LM containing OOV tokens, to which we can assign pronunciations. We also investigate the impact of acoustic complexity and the “natural” occurrence frequency of OOV words on the recognition of registered OOV words. The proposed OOV word registration method is evaluated using two modern automatic speech recognition (ASR) systems, Julius and Kaldi, using DNN-HMM acoustic models and N-gram language models (plus an additional evaluation using RNN re-scoring with Kaldi). Our experimental results show that when using the proposed OOV registration method, modern ASR systems can recognize OOV words without re-training the language model, that the acoustic complexity of OOV words affects OOV recognition, and that differences between the “natural” and the assigned occurrence frequencies of OOV words have little impact on the final recognition results.

Audio-visual speech recognition based on joint training with audio-visual speech enhancement for robust speech recognition

This paper describes the combination of two recent machine learning techniques for acoustic modeling in speech recognition: deep neural networks (DNNs) and graph-based semi-supervised learning (SSL). While DNNs have been shown to be powerful supervised classifiers and have achieved considerable success in speech recognition, graph-based SSL can exploit valuable complementary information derived from the manifold structure of the unlabeled test data. Previous work on graph-based SSL in acoustic modeling has been limited to frame-level classification tasks and has not been compared to, or integrated with, state-of-the-art DNN/HMM recognizers. This paper represents the first integration of graph-based SSL with DNN based speech recognition and analyzes its effect on word recognition performance. The approach is evaluated on two small vocabulary speech recognition tasks and shows a significant improvement in HMM state classification accuracy as well as a consistent reduction in word error rate over a state-of-the-art DNN/HMM baseline.

This paper presents the results of a study on modeling the highly inflective Slovenian language. We focus on creating a language model for a large vocabulary speech recognition system. A new data-driven method is proposed for the induction of inflectional morphology into language modeling. The research focus is on data sparsity, which results from the complex morphology of the language. The idea of using subword units is examined. An attempt is made to figure out the segmentation of words into two subword units: stems and endings. No prior knowledge of the language is used. The subword units should fit into the frameworks of the probabilistic language models. A morphologically correct decomposition of words is not being sought, but searching for a decomposition which yields the minimum entropy of the training corpus. This entropy is approximated by using N-gram models. Despite some seemingly over-simplified assumption, the subword models improve the applicability of the language models for a sparse training corpus. The experiments were performed using the VEČER newswire text corpus as a training corpus. The test set was taken from the SNABI speech database, because the final models were evaluated in speech recognition experiments on SNABI speech database. Two different subword-based models are proposed and examined experimentally. The experiments demonstrate that subword-based models, which considerably reduce OOV rate, improve speech recognition WER when compared with standard word-based models, even though they increase test set perplexity. Subword-based models with improved perplexity, but which reduce the OOV rate much less than the previous ones, do not improve speech recognition results.

Speech and language recognition is one of the most important requirements in daily life. Most people in the world use English as the highly preferable language for easy communication. As a result, English is the highest speaking language in the World. Speech is the easiest way of medium for communication. Hunan-human, human-machine, and machine-machine communication are very much dependent on extensive and robust speech recognition. There are various important features of speech. The accent is one of them. It is a unique feature of speech that varies according to region and dialect. Speech scientists are continuously working on the improvement of speech recognition techniques and performance. Various challenges are being faced day by day and new areas of research are opened to overcome those difficulties. Variation of accent is one of the biggest challenges in improving speech recognition performances. In this work, the native and non-native English accent has been considered. The classification task has been performed on the English speech of non-native speakers and native American English speakers. Three sets of male and female speaker data both for native and non-native speech have been collected from the Voice Cloning Toolkit corpus of the Centre for Speech Technology Research. Mel Frequency Cepstral Coefficient features were extracted from the audio signals and given as input parameters to the supervised machine learning models. Evaluation metrics have been generated, multiple results were produced and compared.