Collaborative Joint Training With Multitask Recurrent Model for Speech and Speaker Recognition

ABSTRACT: Speech is one of the most natural and efficient forms of communication among humans, and it is becoming increasingly integral to human-computer interaction in modern applications such as virtual assistants, smart devices, and automated customer service systems. Among the various technologies that leverage speech, speaker recognition stands out as a biometric method that identifies or verifies individuals based on the unique characteristics of their voice. This approach has gained significant traction due to its non-intrusive nature, ease of integration, and applicability across various domains such as security, forensics, and personalised user experiences. In this paper, we introduce a novel speaker recognition system specifically designed to identify speakers based on utterances in the Marathi language, a linguistically rich and widely spoken regional language in India. The system utilises Mel-Frequency Cepstral Coefficients (MFCCS) to extract distinguishing vocal features that closely mimic the human auditory perception. MFCCS are particularly effective in capturing speech’s phonetic and acoustic properties, making them a preferred choice in speech and speaker recognition tasks. Vector quantisation (VQ) is applied to optimise the feature set and reduce computational complexity. VQ compresses the high-dimensional MFCC feature vectors into representative clusters without significantly compromising accuracy, thereby enhancing system efficiency. These features are then processed using hidden Markov Models (HMMS), which are adept at modelling temporal sequences and dynamic variations in speech. HMMS offer a statistical framework that effectively captures the sequential nature of speech patterns, leading to more reliable speaker modelling and recognition.

This paper examines the utility of audio-visual speech for the two related tasks of speech and speaker recognition. A study of the confusion that exists between speaker and speech elements was performed to show that principal component analysis (PCA) based visual speech is considerably better for the task of speaker recognition than for speech. Decision fusion speech and speaker recognition engines were also tested under various levels of acoustic degradation to find that the optimal fusion configuration for speaker recognition was substantially different than that for speech. These results highlight the problem of employing similar visual features for both speech and speaker recognition.

Speech and speaker recognition technology has made very significant progress in the past 50 years. The progress can be summarized by the following changes: (1) from template matching to corpus-base statistical modeling, e.g., HMM and n-grams, (2) from filter bank/spectral resonance to Cepstral features (Cepstrum + DCepstrum + DDCepstrum), (3) from heuristic time-normalization to DTW/DP matching, (4) from gdistanceh-based to likelihood-based methods, (5) from maximum likelihood to discriminative approach, e.g., MCE/GPD and MMI, (6) from isolated word to continuous speech recognition, (7) from small vocabulary to large vocabulary recognition, (8) from context-independent units to context-dependent units for recognition, (9) from clean speech to noisy/telephone speech recognition, (10) from single speaker to speaker-independent/adaptive recognition, (11) from monologue to dialogue/conversation recognition, (12) from read speech to spontaneous speech recognition, (13) from recognition to understanding, (14) from single-modality (audio signal only) to multi-modal (audio/visual) speech recognition, (15) from hardware recognizer to software recognizer, and (16) from no commercial application to many practical commercial applications. Most of these advances have taken place in both the fields of speech recognition and speaker recognition. The majority of technological changes have been directed toward the purpose of increasing robustness of recognition, including many other additional important techniques not noted above.

Combined speech enhancement and auditory modelling for robust distributed speech recognition

The spiking neural network (SNN) using leaky-integrated-and-fire (LIF) neurons has been commonly used in automatic speech recognition (ASR) tasks. However, the LIF neuron is still relatively simple compared to that in the biological brain. Further research on more types of neurons with different scales of neuronal dynamics is necessary. Here we introduce four types of neuronal dynamics to post-process the sequential patterns generated from the spiking transformer to get the complex dynamic neuron improved spiking transformer neural network (DyTr-SNN). We found that the DyTr-SNN could handle the non-toy automatic speech recognition task well, representing a lower phoneme error rate, lower computational cost, and higher robustness. These results indicate that the further cooperation of SNNs and neural dynamics at the neuron and network scales might have much in store for the future, especially on the ASR tasks.

Although the lower layers of a deep neural network learn features which are transferable across datasets, these layers are not transferable within the same dataset. That is, in general, freezing the trained feature extractor (the lower layers) and retraining the classifier (the upper layers) on the same dataset leads to worse performance. In this paper, for the first time, we show that the frozen classifier is transferable within the same dataset. We develop a novel top-down training method which can be viewed as an algorithm for searching for high-quality classifiers. We tested this method on automatic speech recognition (ASR) tasks and language modelling tasks. The proposed method consistently improves recurrent neural network ASR models on Wall Street Journal, self-attention ASR models on Switchboard, and AWD-LSTM language models on WikiText-2.

Although the field of automatic speaker or speech recognition has been extensively studied over the past decades, the lack of robustness has remained a major challenge. The missing data technique (MDT) is a promising approach. However, its performance depends on the correlation across frequency bands. This paper presents a new reconstruction method for feature enhancement based on the trait. In this paper, the degree of concentration across frequency bands is measured with principal component analysis (PCA). Through theoretical analysis and experimental results, it is found that the correlation of the feature vector extracted from the sub-band (SB) is much stronger than the ones extracted from the full-band (FB). Thus, rather than dealing with the spectral features as a whole, this paper splits full-band into sub-bands and then individually reconstructs spectral features extracted from each SB based on MDT. At the end, those constructed features from all sub-bands will be recombined to yield the conventional mel-frequency cepstral coefficient (MFCC) for recognition experiments. The 2-sub-band reconstruction approach is evaluated in speaker recognition system. The results show that the proposed approach outperforms full-band reconstruction in terms of recognition performance in all noise conditions. Finally, we particularly discuss the optimal selection of frequency division ways for the recognition task. When FB is divided into much more sub-bands, some of the correlations across frequency channels are lost. Consequently, efficient division ways need to be investigated to perform further recognition performance.

For communication, speech is one of the natural forms. A person’s voice contains various parameters that convey information such as emotions, gender, attitude, health, and identity. Determination of these parameters will help to further develop the technology into a reliable and consistent means of identification using speaker recognition. Speaker recognition technologies have wide application areas especially in authentication; surveillance and forensic speaker recognition. In addition, speaker recognition refers to the automated method of identifying or confirming the identity of an individual based on his/her voice. Speech recognition strips out the personal differences to detect the words. Speaker recognition typically disregards the language and meaning to detect the physical person behind the speech. Speech recognition is language-dependent, while Speaker recognition is independent of language. In essence, voice biometrics provides speaker recognition rather than speech recognition. The most accepted form of identification for a human is his/her speech signal. Principally the speaker recognition is the computing task of validating a user’s claimed identity 108using characteristics extracted from their voice. The speaker recognition process based on a speech signal is treated as one of the most exciting technologies of human recognition. For Speaker identification activities we mainly emphasize the physical features of signal. Speakers could be categorized as speaker identification and speaker verification. In speaker identification, the obtained features are compared with all the speaker’s features which are stored in a voice model database and in speaker verification the obtained features are only compared with the stored features of the speaker he/she claimed to be. In this chapter the general principles of speaker recognition, methodology, and applications are discussed.

Automatic recognition of speech by machines begins with extraction of meaningful features from the speech signal. Conventional features like the MFCC are derived from the Fourier transform magnitude spectrum, while totally ignoring the phase spectrum. The importance of the Modified group delay feature (MODGDF) derived from the Fourier transform phase spectrum for speaker and phoneme recognition has been presented in our previous efforts. In this paper we try to analyse the feature theoretically and provide justifications in terms of de-correlation, robustness to convolutional and white noise, cluster structures, separability in lower dimensional space, task independence and class separability. The results of speaker identification and continuous speech recognition using the MODGDF as the front end are also presented. Joint features derived from the MODGDF and MFCC gave significant improvements in recognition performance for both speaker and continuous speech recognition tasks. Using the analytical results in the first half of the paper and the results of performance evaluation in the second half, the MODGDF is proposed as an alternative spectral representation of speech.

Cepstral features and their derivatives have been successfully used in speaker recognitions tasks over the past two decades. Recently new features are emerging that make use of information that is not contained in cepstral features or their derivative, namely the phase spectra and instantaneous frequency. There are numerous approaches to capture instantaneous amplitude and frequency variations in the signal, resulting in an amplitude and frequency modulation (AM-FM) feature set. While the addition of modulation features has been shown to improve speech recognition systems based on cepstral features, their use in speaker recognition is new and has not been fully explored. This paper outlines some of these emerging features and their extraction methods for use in speaker recognition tasks. The paper also talks about the recently proposed Empirical Mode Decomposition (EMD) as an alternative method of extracting instantaneous frequency.

Results from digital and analog filter bank preprocessors are compared in order to establish the validity of analog processing for automatic speech recognition (ASR) systems. Three systems are evaluated using speaker and context independent phoneme recognition tasks. The three ASR systems are identical except for the preprocessing techniques used to derive three signal representations: extraction of (1) the digital mel-frequency spectrum, (2) the mel-frequency spectrum from commercial discrete bandpass filters and (3) the exponential spectrum from analog VLSI bandpass filter bank. The discrete analog system exhibits a 38% increase in recognition accuracy over the digital preprocessing technique. The digital and analog VLSI-based techniques perform comparably (within 3% of each other).

This paper describes the combination of two recent machine learning techniques for acoustic modeling in speech recognition: deep neural networks (DNNs) and graph-based semi-supervised learning (SSL). While DNNs have been shown to be powerful supervised classifiers and have achieved considerable success in speech recognition, graph-based SSL can exploit valuable complementary information derived from the manifold structure of the unlabeled test data. Previous work on graph-based SSL in acoustic modeling has been limited to frame-level classification tasks and has not been compared to, or integrated with, state-of-the-art DNN/HMM recognizers. This paper represents the first integration of graph-based SSL with DNN based speech recognition and analyzes its effect on word recognition performance. The approach is evaluated on two small vocabulary speech recognition tasks and shows a significant improvement in HMM state classification accuracy as well as a consistent reduction in word error rate over a state-of-the-art DNN/HMM baseline.

In recent years, there have a little studies on speaker and speech recognition in Tibetan, which are mainly based on traditional methods of probability statistics. With the development of deep learning, neural networks have been widely used in speaker and automatic speech recognition, which have achieved remarkable results. In this paper, we utilize end-to-end model to study speaker verification and speech recognition in Tibetan. This article uses the ResCNN network for Tibetan speaker verification. In speech recognition, we adopt the DFCNN-CTC structure, where connectionist temporal classification (CTC) directly outputs the probability of sequence prediction without external post-processing. We have made some improvements to the two models. Experiments show that the improved model reduces EER by 3% and WER by 18% in speaker verification and speech recognition, respectively.

In recent years, with the rise of human-computer interaction and the successful application of end-to-end models in the field of speech recognition, the construction of end-to-end speech recognition models has received extensive attention. Relying on the multi-task learning method and the connection between language identification and speech recognition, we proposed an end-to-end Transformer model, which is a multilingual speech recognition model integrating language identification. The model takes the speech recognition task as the main task and the language identification task as the auxiliary task. In this paper, the validity of the model is verified by using the datasets of 13 languages in the 2021 Oriental Language Recognition challenge (OLR). The experimental results show that the model constructed in this paper has a relative improvement of 37.46% in the speech recognition task compared with the baseline system proposed by the OLR organizer. The accuracy of language identification reaches 89.70 %. The results can get the fifth place in the 2021 OLR constraint track of speech recognition equally.

We present lipreading using recurrent neural prediction model. Lipreading copes with time-series data like speech recognition. Therefore, many traditional methods use Hidden Markov Model (HMM) as the classifier for lipreading. However, in recent years, a speech recognition method using Recurrent Neural Prediction Model (RNPM) is proposed, and good result is reported. It is expected that RNPM also gives the good result for lipreading, because lipreading has the similar properties with speech recognition. The effectiveness of the proposed method is confirmed by using 8 words captured from 5 persons. In addition, the comparison with HMM is performed. It is confirmed that the comparable performance is obtained.KeywordsFeature VectorHide Markov ModelImage SequenceSpeech RecognitionGaussian Mixture ModelThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.