CNN-Based Speaker Verification and Speech Recognition in Tibetan

Speech and speaker recognition technology has made very significant progress in the past 50 years. The progress can be summarized by the following changes: (1) from template matching to corpus-base statistical modeling, e.g., HMM and n-grams, (2) from filter bank/spectral resonance to Cepstral features (Cepstrum + DCepstrum + DDCepstrum), (3) from heuristic time-normalization to DTW/DP matching, (4) from gdistanceh-based to likelihood-based methods, (5) from maximum likelihood to discriminative approach, e.g., MCE/GPD and MMI, (6) from isolated word to continuous speech recognition, (7) from small vocabulary to large vocabulary recognition, (8) from context-independent units to context-dependent units for recognition, (9) from clean speech to noisy/telephone speech recognition, (10) from single speaker to speaker-independent/adaptive recognition, (11) from monologue to dialogue/conversation recognition, (12) from read speech to spontaneous speech recognition, (13) from recognition to understanding, (14) from single-modality (audio signal only) to multi-modal (audio/visual) speech recognition, (15) from hardware recognizer to software recognizer, and (16) from no commercial application to many practical commercial applications. Most of these advances have taken place in both the fields of speech recognition and speaker recognition. The majority of technological changes have been directed toward the purpose of increasing robustness of recognition, including many other additional important techniques not noted above.

We present a feature enhancement method that uses neural networks (NNs) to map the reverberant feature in a log-melspectral domain to its corresponding anechoic feature. The mapping is done by cascade NNs trained using Cascade2 algorithm with an implementation of segment-based normalization. Experiments using speaker identification (SID) and automatic speech recognition (ASR) systems were conducted to evaluate the method. The experiments of SID system was conducted by using our own simulated and real reverberant datasets, while the CENSREC-4 evaluation framework was used as the evaluation for the ASR system. The proposed method could remarkably improve the performance of both systems by using limited stereo data and low speaker-variant data as the training data. From the evaluation using SID, we reached 26.0% and 34.8% of error rate reduction (ERR) relative to the baseline by using simulated and real data, respectively, by using only one pair of utterances for matched condition cases. Then, by using combined dataset containing 15 pairs of utterances by one speaker from three positions in a room, we could reach 93.7% of average identification rate (three known and two unknown positions), which was 42.2% of ERR relative to the use of cepstral mean normalization (CMN). From the evaluation using ASR, by using 40 pairs of utterances as the NN training data, we could reach 78.4% of ERR relative to the baseline by using simulated utterances by five speakers. Moreover, we could reach 75.4% and 71.6% of ERR relative to the baseline by using real utterances by five speakers and one speaker, respectively.

Abstract— Audio data carries rich information in the form of speech, music, and environmental sounds, but its raw waveform is often complex and high-dimensional, making direct analysis difficult. Neural network architectures have emerged as powerful tools for extracting meaningful representations from audio signals, enabling efficient analysis and interpretation. Recurrent Neural Networks (RNNs), and Transformer-based models— for learning robust and discriminative features from audio. By automatically capturing temporal, spectral, and contextual patterns, these architectures significantly improve performance in tasks such as speech recognition, speaker identification, music classification, and environmental sound detection. The findings highlight the potential of neural networks to replace traditional handcrafted features, thereby advancing the development of scalable, accurate, and realtime audio processing applications. The rapid growth of audio data across domains such as speech, music, healthcare, and environmental monitoring has created a strong need for effective methods to extract meaningful representations from complex audio signals. Traditional approaches rely on handcrafted features like MFCCs and spectrogram descriptors, which often fail to capture the full temporal and spectral dynamics present in raw audio. Keywords: Neural Networks, Deep Learning, Audio Representation Learning, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformer Models, Feature Extraction, Speech Recognition, Speaker Identification, Sound Event Detection, Spectrogram Analysis, Audio Signal Processing

For communication, speech is one of the natural forms. A person’s voice contains various parameters that convey information such as emotions, gender, attitude, health, and identity. Determination of these parameters will help to further develop the technology into a reliable and consistent means of identification using speaker recognition. Speaker recognition technologies have wide application areas especially in authentication; surveillance and forensic speaker recognition. In addition, speaker recognition refers to the automated method of identifying or confirming the identity of an individual based on his/her voice. Speech recognition strips out the personal differences to detect the words. Speaker recognition typically disregards the language and meaning to detect the physical person behind the speech. Speech recognition is language-dependent, while Speaker recognition is independent of language. In essence, voice biometrics provides speaker recognition rather than speech recognition. The most accepted form of identification for a human is his/her speech signal. Principally the speaker recognition is the computing task of validating a user’s claimed identity 108using characteristics extracted from their voice. The speaker recognition process based on a speech signal is treated as one of the most exciting technologies of human recognition. For Speaker identification activities we mainly emphasize the physical features of signal. Speakers could be categorized as speaker identification and speaker verification. In speaker identification, the obtained features are compared with all the speaker’s features which are stored in a voice model database and in speaker verification the obtained features are only compared with the stored features of the speaker he/she claimed to be. In this chapter the general principles of speaker recognition, methodology, and applications are discussed.

The performance of automatic speech recognition (ASR) may be degraded when accented speech is recognized because the speech has some linguistic differences from standard speech. Conventional accented speech recognition studies have utilized the accent embedding method, in which the accent embedding features are directly fed into the ASR network. Although the method improves the performance of accented speech recognition, it has some restrictions, such as increasing the computational costs. This study proposes an efficient method of training the ASR model for accented speech in a domain adversarial way based on the Domain Adversarial Neural Network (DANN). The DANN plays a role as a domain adaptation in which the training data and test data have different distributions. Thus, our approach is expected to construct a reliable ASR model for accented speech by reducing the distribution differences between accented speech and standard speech. DANN has three sub-networks: the feature extractor, the domain classifier, and the label predictor. To adjust the DANN for accented speech recognition, we constructed these three sub-networks independently, considering the characteristics of accented speech. In particular, we used an end-to-end framework based on Connectionist Temporal Classification (CTC) to develop the label predictor, a very important module that directly affects ASR results. To verify the efficiency of the proposed approach, we conducted several experiments of accented speech recognition for four English accents including Australian, Canadian, British (England), and Indian accents. The experimental results showed that the proposed DANN-based model outperformed the baseline model for all accents, indicating that the end-to-end domain adversarial training effectively reduced the distribution differences between accented speech and standard speech.

Self-supervised learning (SSL) has shown promise in learning representations of audio that are useful for automatic speech recognition (ASR). But, training SSL models like wav2vec 2.0 requires a two-stage pipeline. In this paper we demonstrate a single-stage training of ASR models that can utilize both unlabeled and labeled data. During training, we alternately minimize two losses: an unsupervised masked Contrastive Predictive Coding (CPC) loss and the supervised audio-to-text alignment loss Connectionist Temporal Classification (CTC). We show that this joint training method directly optimizes performance for the downstream ASR task using unsupervised data while achieving similar word error rates to wav2vec 2.0 on the Librispeech 100-hours dataset. Finally, we postulate that solving the contrastive task is a regularization for the supervised CTC loss.

This study proposes an approach to voice identification based on neural networks (DNN) for i-Vector. Modern voice identification systems based on DNN use large amounts of labeled training data. Using the LRE i-Vector Machine Learning Challenge restricts access to ready-to-use i-Vector for learning and testing the voice identification system. This poses unique challenges in developing DNN-based voice identification systems, since optimized external interfaces and network architectures can no longer be used. We propose to use the training i-Vectors to train the initial DNN to identify the voice. Next, we present a novel strategy for using this initial DNN to strip the language labels of the inappropriate set from the development data. The final DNN for voice identification is trained using the original training data and the estimated out-of-set language data. We show that augmenting the training set with out-of- set labels leads to a significant improvement in voice identification performance. In this paper, we studied the possibility of using neural networks for speech identification. In particular, standard approaches to speech recognition were considered, the concept of an artificial neuron as an object used in speech identification was defined. A speech recognition option using a neural network was investigated, and steps were presented to perform this task. Accuracy using neural networks with limited learning data and a higher i-vector dimension is superior to others with a score of 92.1%. From this study, we can conclude that the size of the UBM and the dimension of the i-vector affect the accuracy of voice identification based on the i-vector.

Voice interfaces are an essential part of contemporary human-computer interaction and speech-to-text (STT) systems allow one to work hands-free and enhance the accessibility. Nevertheless, the precision of voice command recognition is a major issue that is yet to be addressed particularly in noisy settings or in language with complex linguistic features. Current approaches can have issues with the consistency of real-time transcription and are not contextually aware and cannot work with limited or agglutinative languages like Hindi. In order to overcome these shortcomings, Enhancing Voice Command with Connectionist Temporal Classification (EVC-CTC) is proposed in this study. It is a strong framework which makes use of the CTC to enhance sequence matching between speech input and textual output. EVC-CTC combines both context modeling and language specific features to increase recognition accuracy particularly in the presence of real-world variability. The suggested approach is deployed as a modular and scalable STT pipeline that facilitates voice assistants that run on web-based platforms. Continuous voice streams are processed with EVC-CTC to preserve continuity of the context and dynamically respond to the background noise and variations in speakers. Experimental analysis indicates that EVC-CTC is much better than baseline models in Word Error Rate (WER) and semantic accuracy, especially in Hindi voice commands.

The establishment of speech acoustic model system based on Long Short-Term Memory (LSTM) makes further improvements for the speech recognition. However, the connectionist temporal classification (CTC) training method performances more better in directly corresponding to the phoneme sequence or bound sequence of the speech. This paper combines CTC and LSTM to establish a power dispatching speech recognition model and compares the LSTM-CTC methods with traditional GMM-HMM methods, RNN-based speech recognition methods, and unidirectional LSTM networks through experiments. The results show that the speech recognition framework of LSTM-CTC has higher precision than other methods, and also has strong generalization ability. The LSTM-CTC methods can provide higher speech recognition accuracy and are more suitable for speech recognition in power dispatching as well.

End-to-end Automatic Speech Recognition (E2E ASR) significantly simplifies the training process of an ASR model. Connectionist Temporal Classification (CTC) is one of the most popular methods for E2E ASR training. Implicitly, CTC has a unique topology which is very useful for sequence modelling. However, we find that by changing to another topology, we can make it even more effective. In this paper, we propose a new CTC-like method, for E2E ASR training, by modifying the topology of original CTC, so that the well-known abuse of the blank label in CTC can be resolved theoretically. As we change the topology, a normalisation term is necessary, which makes the form of the final loss function similar to Maximum Mutual Information (MMI); we hence name our method MMI-CTC. In addition to maximising the posterior probability of the target sequence, the normalisation enables models to explicitly minimise the probability of competing hypothesis at the word sequence level. Our experimental results show that MMI-CTC is more efficient than CTC, and that the normalisation is essential for sequence training.

Many end-to-end, large vocabulary, continuous speech recognition systems are now able to achieve better speech recognition performance than conventional systems. Most of these approaches are based on bidirectional networks and sequence-to-sequence modeling however, so automatic speech recognition (ASR) systems using such techniques need to wait for an entire segment of voice input to be entered before they can begin processing the data, resulting in a lengthy time-lag, which can be a serious drawback in some applications. An obvious solution to this problem is to develop a speech recognition algorithm capable of processing streaming data. Therefore, in this paper we explore the possibility of a streaming, online, ASR system for Japanese using a model based on unidirectional LSTMs trained using connectionist temporal classification (CTC) criteria, with local attention. Such an approach has not been well investigated for use with Japanese, as most Japanese-language ASR systems employ bidirectional networks. The best result for our proposed system during experimental evaluation was a character error rate of 9.87%.

The author reviews major methods of applying the vector quantization (VQ) technique to speech and speaker recognition. These include speech recognition based on the combination of VQ and the DTW/HMM (dynamic time warping/hidden Markov model) technique. VQ-distortion-based recognition, learning VQ algorithms, speaker adaptation by VQ-codebook mapping, and VQ-distortion-based speaker recognition. It is concluded that not only has the VQ technique reduced the amount of computation and storage, but it has also created new ideas for solving various problems in speech/speaker recognition. >

Speaker Identification (SI) is a process of identifying a speaker automatically via a machine using the speaker’s voice. In SI, one speaker’s voice is compared with n- number of speakers’ templates within the reference database to find the best match among the potential speakers. Speakers are capable of changing their voice, though, such as their accent, which makes is more challenging to identify who is talking. In this paper, we extracted phonemes from a speaker’s voice recording and investigated the associated frequencies and amplitudes to be assist in identifying the person who is speaking. This paper demonstrates the importance of phonemes in both speech and voice recognition systems. The results demonstrate that we can use phonemes to help the machine identify a particular speaker, however, phonemes get better accuracy in speech recognition than speaker identification.

A model-based framework of classification error rate estimation is proposed for speech and speaker recognition. It aims at predicting the run-time performance of a hidden Markov model (HMM) based recognition system for a given task vocabulary and grammar without the need of running recognition experiments using a separate set of testing samples. This is highly desirable both in theory and in practice. However, the error rate expression in HMM-based speech recognition systems has no closed form solution due to the complexity of the multi-class comparison process and the need for dynamic time warping to handle various speech patterns. To alleviate the difficulty, we propose a one-dimensional model-based misclassification measure to evaluate the distance between a particular model of interest and a combination of many of its competing models. The error rate for a class characterized by the HMM is then the value of a smoothed zero-one error function given the misclassification measure. The overall error rate of the task vocabulary could then be computed as a function of all the available class error rates. The key here is to evaluate the misclassification measure in terms of the parameters of environmental-matched models without running recognition experiments, where the models are adapted by very limited data that could be just the testing utterance itself. In this paper, we show how the misclassification measure could be approximated by first computing the distance between two mixture Gaussian densities, then between two HMMs with mixture Gaussian state observation densities and finally between two sequences of HMMs. The misclassification measure is then converted into classification error rate. When comparing the error rate obtained in running actual experiments and that of the new framework, the proposed algorithm accurately estimates the classification error rate for many types of speech and speaker recognition problems. Based on the same framework, it is also demonstrated that the error rate of a recognition system in a noisy environment could also be predicted.

The Recurrent Neural Network Transducer (RNN-T) extends Connectionist Temporal Classification (CTC) by jointly modeling both input-output and output-output dependencies, and it has been successfully applied in end-to-end speech recognition. RNN-T adds language information by expanding its dimensions in this way, but the model training is difficult. And some paths of the model are unreasonable when decoding. Therefore, we present the Alignment RNN Transducer (ART) algorithm, which uses the forward-backward algorithm in CTC to find the best path alignment information as context-related information for training. We not only reduce the training dimensions but also adds context information to the model. And we still use CTC as the loss function to avoid decoding the impossible path in the RNN-T. We verify the algorithm on a Mandarin speech corpus AIShell-1, and it achieves a 13.65% CER, compared with the 16.79% CER given by the RNN-T model.