The use of subword linguistic modeling for multiple tasks in speech recognition

Combined speech enhancement and auditory modelling for robust distributed speech recognition

This paper describes the combination of two recent machine learning techniques for acoustic modeling in speech recognition: deep neural networks (DNNs) and graph-based semi-supervised learning (SSL). While DNNs have been shown to be powerful supervised classifiers and have achieved considerable success in speech recognition, graph-based SSL can exploit valuable complementary information derived from the manifold structure of the unlabeled test data. Previous work on graph-based SSL in acoustic modeling has been limited to frame-level classification tasks and has not been compared to, or integrated with, state-of-the-art DNN/HMM recognizers. This paper represents the first integration of graph-based SSL with DNN based speech recognition and analyzes its effect on word recognition performance. The approach is evaluated on two small vocabulary speech recognition tasks and shows a significant improvement in HMM state classification accuracy as well as a consistent reduction in word error rate over a state-of-the-art DNN/HMM baseline.

One of the sources of difficulty in speech recognition and understanding is the variability due to alternate pronunciations of words. To address the issue we have investigated the use of multiple-pronunciation models (MPMs) in the decoding stage of a speaker-independent speech understanding system. In this paper we address three important issues regarding MPMs: (a) Model construction: How can MPMs be built from available data without human intervention? (b) Model embedding: How should MPM construction interact with the training of the sub-word unit models on which they are based? (c) Utility: Do they help in speech recognition? Automatic, data-driven MPM construction is accomplished using a structural HMM induction algorithm. The resulting MPMs are trained jointlywith a multi-layer perceptron functioningas a phonetic likelihood estimator. The experiments reported here demonstrate that MPMs can significantly improve speech recognition results over standard single pronunciation models.

In recent years, with the rise of human-computer interaction and the successful application of end-to-end models in the field of speech recognition, the construction of end-to-end speech recognition models has received extensive attention. Relying on the multi-task learning method and the connection between language identification and speech recognition, we proposed an end-to-end Transformer model, which is a multilingual speech recognition model integrating language identification. The model takes the speech recognition task as the main task and the language identification task as the auxiliary task. In this paper, the validity of the model is verified by using the datasets of 13 languages in the 2021 Oriental Language Recognition challenge (OLR). The experimental results show that the model constructed in this paper has a relative improvement of 37.46% in the speech recognition task compared with the baseline system proposed by the OLR organizer. The accuracy of language identification reaches 89.70 %. The results can get the fifth place in the 2021 OLR constraint track of speech recognition equally.

For patients with relatively good low-frequency hearing and relatively poor high-frequency hearing, who met the pre-implant criteria for combined electric and acoustic stimulation (EAS), our aims were to i) assess deficits in low-frequency auditory function, ii) to identify measures which might be sensitive to changes resulting from the insertion of an intracochlear electrode array, and iii) to quantify the relationship between measures of auditory function and performance on tasks of speech and melody recognition. Measures of frequency selectivity, temporal resolution, and nonlinear cochlear function, along with measures of word, sentence, consonant, vowel, and melody recognition, were obtained from 5 normal-hearing and 17 hearing-impaired listeners. The hearing-impaired listeners had auditory thresholds at 500 Hz, ranging from 20 to 60 dB HL, and thresholds at 1 kHz, ranging from 60 to 100 dB HL. Nonlinear cochlear function was either reduced or absent. Frequency selectivity at 500 Hz was significantly reduced but still present in most patients. Temporal resolution, when measured at low modulation frequencies, was normal. Speech recognition in a modulated background revealed significantly poorer performance than normal. Speech and melody recognition varied over a large range. No measure of auditory function was correlated significantly with speech recognition. However, frequency selectivity was related to melody recognition. (1) Patients who qualify for EAS surgery have a wide range of speech and melody recognition abilities. (2) A number of the psychophysical measures tested may prove more sensitive than the audiogram in determining the degree of damage inflicted by the intracochlear electrode array. (3) Speech recognition was not correlated with any of the measures of auditory function.

The model of automatic speech recognition (ASR) has an overfitting problem in the task of low-resource speech recognition, due to the lack of training data. Different from conventional methodologies that rely on signal acoustic modality, we proposed to design a novel ASR model for the low-resource speech recognition task by fusing acoustic and visual modalities, standing from the perspective of the physiological mechanism of human pronunciation. In this paper, the progressive down-sampling (PD) and grouped self-attention mechanism (GASM) are applied to design the audio-visual speech recognition (AVSR) framework, called PD-GASM, for the low-resource speech recognition task. In particular, The PD algorithm reduces computational resource consumption during the training process. The GASM algorithm is used to help the trained model handle both short and long sequences; therefore, the proposed model can be more suitable for processing spoken dialogue sequences. Our proposed method has been evaluated based on the Tujia language audio-visual data corpus, miniLRW and miniLRS2 audio-visual data corpus. The experimental results show that the best character error rates can achieve 35.15%, 14.89%, and 16.33 %, respectively. Our research has significant implications for expanding the application of AVSR in the field of low-resource speech recognition.

In this paper, we introduce a document-specific context probabilistic latent semantic analysis (DCPLSA) model for speech recognition. This is an extension of a CPLSA model [1] where the probability of word is conditioned only on topics. The CPLSA model uses the bigram counts that are the number of appearances of the bigrams in the corpus. These counts are the sum of the bigram counts in different documents where they could appear to describe different topics. We encounter this problem in the CPLSA model and introduce the document-specific CPLSA model (DCPLSA) where the probability of a word is conditioned on both topic and document. We carried out experiments on a continuous speech recognition (CSR) task using the Wall Street Journal (WSJ) corpus and have seen that the proposed DCPLSA approach yields significant reduction in both perplexity and word error rate (WER) measurements over the other approaches used in the literature.

Deep neural networks (DNN) have recently shown impressive performance as acoustic models for large vocabulary continuous speech recognition (LVCSR) tasks. Typically, the frame shift of the output of neural networks is much shorter than the average length of the modeling units, so the posterior vectors of neighbouring frames are likely to be similar. The similarity, together with the better discrimination of neural networks than typical acoustic models, shows a possibility of removing frames of neural network outputs according to the distance of posterior vectors. Then, the computation costs of beam searching can be effectively reduced. Based on that, the paper introduces a novel variable-frame-rate decoding approach based on neural network computation that accelerates the beam searching for speech recognition with minor loss of accuracy. By computing the distances of posterior vectors and removing frames with a posterior vector similar to the previous frame, the approach can make use of redundant information between frames and do a much quicker beam searching. Experiments on LVCSR tasks show a 2.4-times speed up of decoding compared to the typical framewise decoding implementation.

We present a method for open vocabulary retrieval based on a spoken document retrieval (SDR) system using subword models. The present paper proposes a new approach to open vocabulary SDR system using subword models which do not require subword recognition. Instead, subword sequences are obtained from the phone sequence outputted containing an out of vocabulary (OOV) word, a speech recognizer outputs a word sequence whose phone sequence is considered to be similar to the OOV word. When OOV words are provided in a query, the proposed system is able to retrieve the target section by comparing the phone sequences of the query and the word sequence generated by the speech recognizer.

The minimum classification error (MCE) framework for discriminative training is a simple and general formalism for directly optimizing recognition accuracy in pattern recognition problems. The framework applies directly to the optimization of hidden Markov models (HMMs) used for speech recognition problems. However, few if any studies have reported results for the application of MCE training to large-vocabulary, continuous-speech recognition tasks. This article reports significant gains in recognition performance and model compactness as a result of discriminative training based on MCE training applied to HMMs, in the context of three challenging large-vocabulary (up to 100 k word) speech recognition tasks: the Corpus of Spontaneous Japanese lecture speech transcription task, a telephone-based name recognition task, and the MIT Jupiter telephone-based conversational weather information task. On these tasks, starting from maximum likelihood (ML) baselines, MCE training yielded relative reductions in word error ranging from 7% to 20%. Furthermore, this paper evaluates the use of different methods for optimizing the MCE criterion function, as well as the use of precomputed recognition lattices to speed up training. An overview of the MCE framework is given, with an emphasis on practical implementation issues

Phonemes and allophones are the basic speech units for acoustic modeling in the majority of contemporary HMM based speech recognizers. Grapheme-based acoustic sub-word units were applied to multi-lingual and cross-lingual acoustic modeling in many tasks. Grapheme and phoneme based mono-, cross- and bilingual speech recognition of Czech and Slovak in the small and medium vocabulary task has been studied in our previous work. In this article we compare grapheme and phoneme based approach to acoustic modeling and model unit selection in large vocabulary continuous speech recognition (LVCSR) task in Slovak. The main goal of our experimental work is to investigate a possibility to select an optimal set of sub-word units for Slovak LVCSR system.

Human speech recognition seems effortless, but so far it has been impossible to approach human performance by machines. Compared with human speech recognition (HSR), the error rates of state-of-the-art automatic speech recognition (ASR) systems are an order of magnitude larger (Lee, 2004; Moore, 2003; see also Scharenborg et al., 2005). This is true for many different speech recognition tasks in noise-free environments, but also (and especially) in noisy environments (Lippmann, 1997; Sroka & Braida, 2005; Wesker et al., 2005). The advantage for humans remains even in experiments that deprive humans from exploiting ‘semantic knowledge’ or ‘knowledge of the world’ that is not readily accessible for machines. It is well known that there are several recognition tasks in which machines outperform humans, such as the recognition of license plates or barcodes. Speech differs from license plates and bar codes in many respects, all of which help to make speech recognition by humans a fundamentally different skill. Probably the most important difference is that bar codes have been designed on purpose with machine recognition in mind, while speech as a medium for human-human communication has evolved over many millennia. Linguists have designed powerful tools for analyzing and describing speech, but we hardly begin to understand how humans process speech. Recent research suggests that conventional linguistic frameworks, which represent speech as a sequence of sounds, which in their turn can be represented by discrete symbols, fail to capture essential aspects of speech signals and, perhaps more importantly, of the neural processes involved in human speech understanding. All existing ASR systems are tributable to the beads-on-a-string representation (Ostendorf, 1999) invented by linguistics. But is quite possible –and some would say quite likely- that human speech understanding in not based on neural processes that map dynamically changing signals onto sequences of discrete symbols. Rather, it may well be that infants develop very different representations of speech during their language acquisition process. Language acquisition is a side effect of purposeful interaction between infants and their environment: infants learn to understand and respond to speech because it helps to fulfil a set of basic goals (Maslow, 1954; Wang, 2003). An extremely important need is being able to adapt to new situations (speakers, acoustic environments, words, etc.) Pattern recognisers, on the other hand, do not aim at the optimisation of ‘purposeful

Speech trigger and speech command recognition technologies have become pivotal human-machine interfaces for AIoT applications. Typically, a speech-related system consists of a mel-filter-based handcrafted feature extractor and an artificial neuron-based classifier. Recent advancements [1–3] have demonstrated promising results by training the feature extractor jointly with the neural network classifier. Moreover, the emergence of spiking neuron network training algorithms and hardware designs has showcased their competitive edge over traditional artificial neural networks. Consequently, the design and training of learnable audio feature extractors and spiking neural network classifiers are the key focuses of this thesis. The first focus of this thesis is the design of an ultra-low-power, learnable audio feature extractor to support end-to-end training. Conventional low-power approaches, such as extreme pruning and quantization in the audio feature extractor (AuFEx) design, often result in decreased accuracy. To address this, the thesis proposes a hardware-aware, end-to-end training framework to co-optimize the AuFEx design, achieving high performance with minimal hardware cost. Two types of AuFEx designs are explored: a time-domain (TD) filter-based learnable audio feature extractor (AuFEx-TD) and a frequency-domain (FD) filter-based learnable audio feature extractor (AuFEx-FD). Using this framework, the AuFEx-TD achieves high accuracy (97.8%) for one wake word detection, with low power consumption (496 nW) and low latency (375 μs). The optimized AuFEx-FD demonstrates noise-robust 10-keyword spotting accuracy (87.5%@5dB – 92.2%@20dB) with low power consumption (1.26 μW). Both AuFEx-TD and AuFEx-FD designs are implemented using UMC 40nm CMOS technology to validate the framework's effectiveness. For the speech trigger or wake word detection (WWD) task, power and latency are critical design considerations. This thesis presents a sub-μW spiking neural network accelerator with an 8 ms latency. The core WWD engine employs a spiking convolutional neural network (SCNN) model, leveraging sparse activation and addition-only operations within spiking neurons. The proposed SCNN model enhances the existing framewise incremental computation flow [4] by incorporating a spike processing unit (SPU), reducing system power and latency by 16.5% and 43.2%, respectively. Extensive quantization further reduces weight and activation precisions to 4-bit and 1-bit. Additionally, a power-gating mechanism, driven by an energy-based voice activity detection (VAD) module, further minimizes power consumption in random and sparse event (RSE) scenarios. Full chip simulation results show that the chip consumes only 110 nW, with a 2.15% false alarm rate and a 3.00% false reject rate in a 10% voice event stream test. It achieves state-of-the-art recognition accuracy of 99% and 96% for one and two wake word detection tasks, respectively. For speech command recognition or keyword spotting (KWS) tasks, robustness against noise and speaker accents is essential for user experience. This thesis explores both end-to-end training and on-device transfer learning to enhance KWS chip robustness with minimal power overhead. The proposed chip comprises three main blocks: the Learnable Audio Feature Extractor (AuFEx), the Spiking-DSCNN Core (SC), and the Transfer Learning Core (TLC). The AuFEx is an optimized AuFEx-FD, with the on-chip memory for storing FD-filter parameters replaced by an FD-filter parameter generator, reducing power consumption to 1.225 μW. The SC implements a spiking neural network (SNN) model using a Depthwise Separable Convolutional Neural Network (DSCNN) topology. Multi-level encoding in spiking neurons reduces simulation time, while the inherent activation sparsity of these neurons decreases power consumption and latency. The TLC supports both forward and backward propagation for a fully connected layer and softmax layer, enabling the KWS chip to adapt to environmental and user variations using minimal labeled data and weight updates. Fabricated in UMC 40nm CMOS technology, the chip achieves 10-keyword spotting accuracy ranging from 87.5% to 92.2% under background noise levels of 5dB to 20dB. Through on-device transfer learning, the chip can improve KWS accuracy on personalized datasets by 11.73%, using only a few examples per keyword and limited training epochs. The simulated power consumption of the chip in streaming inference mode is 10.26 μW at a 250 kHz clock frequency and 0.85V supply voltage. In conclusion, this thesis presents an end-to-end spiking neural network-based speech system for speech trigger (WWD) and speech command recognition (KWS) tasks. By integrating the AuFEx-TD and the SCNN accelerator, as described in the first and second parts of the thesis, a low-latency, sub-μW wake word detection system has been developed. In the final part, a noise-robust, personalized, few-μW-level speech command recognition ASIC is introduced, demonstrating the feasibility of high-performance, low-power AIoT speech interfaces.

Automatic speech and speaker recognition are traditionally treated as two independent tasks and are studied separately. The human brain in contrast deciphers the linguistic content, and the speaker traits from the speech in a collaborative manner. This key observation motivates the work presented in this paper. A collaborative joint training approach based on multitask recurrent neural network models is proposed, where the output of one task is backpropagated to the other tasks. This is a general framework for learning collaborative tasks and fits well with the goal of joint learning of automatic speech and speaker recognition. Through a comprehensive study, it is shown that the multitask recurrent neural net models deliver improved performance on both automatic speech and speaker recognition tasks as compared to single-task systems. The strength of such multitask collaborative learning is analyzed, and the impact of various training configurations is investigated.

Speech recognition in noise (SiN) requires explicit cognitive resources such as working memory capacity (WMC). However, existing evidence on the role of WMC in SiN, especially for young adults with normal hearing, is mixed. This study aimed to systematically evaluate the role of WMC when the degree of speech recognition varied as a consequence of adding background noise of varying intensity levels. Specifically, the objective was to determine whether or not there was an effect of noise level on the recruitment of WMC during speech recognition tasks. Fifty-three young adults (18–37 years) with normal hearing were recruited and their ability to recognize speech in the presence of background noise was measured using sentences and words. Speech-to-noise ratios (SNRs) were varied from +10 to −10 dB SNR in 5 dB steps to manipulate the degree of speech recognition. Listeners’ WMC was also measured using two working memory tasks. A significant positive association between WMC and SiN was found for sentence recognition in noise. However, no significant interaction between WMC and SNR was found for isolated word and sentence recognition tasks. WMC was not found to be a significant predictor of word recognition in noise. These results suggests that for young normal hearing listeners, the recruitment of WMC during sentence recognition, especially in non-informational fluctuating noise, is not modulated by the SNR .