Addressing Text-Dependent Speaker Verification Using Singing Speech

In this paper, automatic speaker verification using whispered speech is explored. In the past, whispered speech has been shown to convey relevant speaker identity and gender information, nevertheless it is not clear how to efficiently use this information in speech-based biometric systems. This study compares the performance of three different speaker verification systems trained and tested under different scenarios and with two different feature representations. First, we show the benefits of using AM-FM based features as well as their effectiveness for i-vectors extraction. Second, for the classical mel-frequency cepstral coefficient (MFCC) features we show that gains of up to 40% could be achieved with the fusion of traditional Gaussian mixture model (GMM) based systems and more recent i-vector based ones, relative to using either system alone for normal speech. Additionally, for MFCC, fusion schemes were shown to be more robust to addition of whispered speech data during training or enrollment. Overall, AM-FM based features were shown to be more robust to varying training/testing conditions and to improve speaker verification performance for both normal and whispered speech by using the GMM based system alone.

In this paper, we present a Mixture Linear Prediction based approach for robust Gammatone Cepstral Coefficients extraction (MLPGCCs). The proposed method provides performance improvement of Automatic Speaker Verification (ASV) using i-vector and Gaussian Probabilistic Linear Discriminant Analysis GPLDA modeling under transmission channel noise. The performance of the extracted MLPGCCs was evaluated using the NIST 2008 database where a single channel microphone recorded conversational speech. The system is analyzed in the presence of different channel transmission noises such as Additive White Gaussian (AWGN) and Rayleigh fading at various Signals to Noise Ratio (SNR) levels. The evaluation results show that the MLPGCCs features are a promising way for the ASV task. Indeed, the speaker verification performance using the MLPGCCs proposed features is significantly improved compared to the conventional Gammatone Frequency Cepstral Coefficients (GFCCs) and Mel Frequency Cepstral Coefficients (MFCCs) features. For speech signals corrupted with AWGN noise at SNRs ranging from (-5 dB to 15 dB), we obtain a significant reduction of the Equal Error Rate (EER) ranging from 9.41% to 6.65% and 3.72% to 1.50%, compared with conventional MFCCs and GFCCs features respectively. In addition, when the test speech signals are corrupted with Rayleigh fading channel we achieve an EER reduction ranging from 23.63% to 7.8% and from 10.88% to 6.8% compared with conventional MFCCs and GFCCs, respectively. We also found that the combination of GFCCs and MLPGCCs gives the highest performance of speaker verification system. The best performance combination achieved is around EER from 0.43% to 0.59% and 1.92% to 3.88%.

The extraction of speaker-related features through speaker representations or utterance embeddings has been extensively studied for years. Convolutional Neural Networks (CNNs) are the most commonly used backbone network for feature extraction, and deep Residual Networks (ResNets), which incorporate residual connectivity into CNNs, achieve effective frame-level feature extraction inside the same input samples. ResNets have demonstrated impressive performance as speaker embedding extractors. With emerging this backbone and its advanced extensions, such as Res2Net, as dominant backbone networks in speaker recognition, and despite its widespread adoption, a significant gap in current speaker embeddings is the unmodified use of these successful backbones from the visual domain, without considering the specific adaptation requirements for the speech field. Tailoring the backbone to account for the unique characteristics of speech representations ensures effective learning in Automatic Speaker Verification (ASV) systems. This thesis concentrates on the backbone network, the fundamental and most critical component of Deep Neural Networks (DNNs), responsible for extracting features from input data and mapping them into distinct feature representations. One of the key questions addressed in the thesis is the following: Which backbone is most robust and yields the most generalized speaker representation, with the potential to be applied to many existing ASV systems?. Through a strategic evaluation, the study analyzed the performance of integrating different residual-based backbones into the most popular ResNet-based fundamental baselines, ResNet-34 and ECAPA-TDNN based models, with highly different model complexity and CNN stem representations. To this end, the proposed architectures were evaluated on the publicly available VoxCeleb dataset. This thesis introduces an innovative approach to speaker modeling by incorporating scale and cardinality dimensions for the first time. Experimental findings reveal that utilizing multi-scale and multi-branch aggregated residual transformations within the Res2NeXt backbone yields considerable performance improvements. Notably, 6s8g and 8s8g variants of Res2NeXt, prove particularly beneficial in improving the representational capability of learned features for speaker embedding. The consistent superiority, excellent compatibility, and robustness of Res2NeXt compared to the current popular ResNet and Res2Net backbones across various architectures and conditions demonstrate its broad applicability and effectiveness as a backbone in speaker modeling. To further evaluate the Robustness and Generalization (R&G) capabilities of the proposed speaker embeddings, the thesis expanded its scope to cross-domain applications, specifically Speech Emotion Recognition (SER), one of the most challenging downstream tasks. A comparative analysis of the proposed speaker embeddings was conducted using two benchmark datasets, IEMOCAP and CREMA-D. The overarching goal of the thesis is to contribute to the development of general-purpose and universal speech representations that generalize effectively across multiple speech-related tasks. By examining the alignment and divergence of embeddings for ASV and SER, the study seeks to answer a fundamental question: Can a single representation framework cater to both speaker verification and emotion recognition or are task-specific embeddings inevitable?. The results confirm that speaker embeddings derived from the Res2NeXt backbone are also highly effective for the SER task, maintaining their performance across diverse evaluation protocols and data variations. This demonstrates the generalizability of Res2NeXt embeddings in the cross-domain task. The study attributes their success to the ability of Res2NeXt-based models to capture rich speech features, including tone, pitch, and phonation patterns, during speaker recognition training—features that are also vital for emotion recognition.

We discuss the limitations of the i-vector representation of speech segments in speaker recognition and explain how Joint Factor Analysis (JFA) can serve as an alternative feature extractor in a variety of ways. Building on the work of Zhao and Dong, we implemented a variational Bayes treatment of JFA which accommodates adaptation of universal background models (UBMs) in a natural way. This allows us to experiment with several types of features for speaker recognition: speaker factors and diagonal factors in addition to i-vectors, extracted with and without UBM adaptation in each case. We found that, in text-independent speaker verification experiments on NIST data, extracting i-vectors with UBM adaptation led to a 10% reduction in equal error rates although performance did not improve consistently over the whole DET curve. We achieved a further 10% reduction (with a similar inconsistency) by using speaker factors extracted with UBM adaptation as features. In text-dependent speaker recognition experiments on RSR2015 data, we were able to achieve very good performance using a JFA model with diagonal factors but no speaker factors as a feature extractor. Contrary to standard practice, this JFA model was configured so as to model speakerphrase combinations (rather than speakers) and it was trained on utterances of very short duration (rather than whole recording sessions). We also present a variant of the length normalization trick inspired by uncertainty propagation which leads to substantial gains in performance over the whole DET curve.

Speaker recognition systems based on spectral features perform well in acoustically matched and noise-free conditions. Spectral features are unsuccessful to model information about the speaker at higher levels. Prosody which represents intonation, rhythm and stress of speech, better represents speaker characteristics at higher levels. Voice disguise is a common threat to automatic speaker identification systems. It is the action of transforming one's voice purposefully to imitate or just to conceal one's identity. This work attempts to study the effects of voice disguise on prosody based speaker verification system. For this a prosody based automatic speaker verification system is implemented using support vector machine (SVM). For the extraction of prosodic features, speech is segmented into phrase-like regions by detecting long pauses regions and further syllable-like segmentation is done at the valleys of short time energy contour. In order to segment fused syllables, vowel onset point (VOP) detection is performed in such regions. Then prosodic features are extracted and used for building SVM models. To study the effect of voice disguise, normal and masked speech were recorded simultaneously for a set of speakers in the lab environment. Prosodic features for both normal and masked speech were analyzed. Using normal speech, automatic speaker verification system is implemented and testing was done using both normal and masked speech. Through this study, it is identified that prosodic features derived from pitch and duration are almost robust against voice masking but energy features are slightly affected.

With the development of spoofing technologies, automatic speaker verification (ASV) systems have encountered serious challenges on security. In order to address this problem, many anti-spoofing countermeasures have been explored. There are two intuitive recipes to protect an ASV system from spoofing. The first one is to use a cascaded structure where spoofing detection is performed firstly and ASV is subsequently conducted only on the attempts which have passed the spoofing detection. The other one is to perform spoofing detection and ASV jointly. The discriminate reliably of the joint system has been proven to be more advantageous than cascaded systems with traditional methods, not only in accuracy, but also in convenience and computational efficiency. In this paper, we proposed a multi-task learning approach based on deep neural network to make a joint system of ASV and anti-spoofing. The performance of different acoustic features and structures of deep neural networks has been investigated on the ASVspoof 2017 version 2.0 dataset. The experimental results showed that the joint equal error rate (EER) of our approach was reduced by 0.55% compared to a joint system with Gaussian back-end fusion baseline.

The spoofing countermeasure (CM) systems in automatic speaker verification (ASV) are not typically used in isolation of each other. These systems can be combined, for example, into a cascaded system where CM produces first a decision whether the input is synthetic or bona fide speech. In case the CM decides it is a bona fide sample, then the ASV system will consider it for speaker verification. End users of the system are not interested in the performance of the individual sub-modules, but instead are interested in the performance of the combined system. Such combination can be evaluated with tandem detection cost function (t-DCF) measure, yet the individual components are trained separately from each other using their own performance metrics. In this work we study training the ASV and CM components together for a better t-DCF measure by using reinforcement learning. We demonstrate that such training procedure indeed is able to improve the performance of the combined system, and does so with more reliable results than with the standard supervised learning techniques we compare against.

This paper addresses the design and implementation of automatic speaker verification (ASV) systems. There is great interest in developing and increasing the performance of ASV applications, taking into account the advantages offered when compared to other biometrical methods. State-of-the-art speaker recognizers are based on statistical models such as GMM, HMM, SVM, ANN or hybrid models. This work reports experiments on prompted text speaker verification on a Romanian corpus. First, a Hidden Markov Model (HMM) based continuous speech recognizer is built at context independent, single mixture, monophone level. Then the client model and the world model are built using appropriate speech data. The text-prompted speaker verification system is based on sentence HMMs, constructed for the key text by concatenating corresponding acoustic models. In the verification stage, the normalized log-likelihood is derived as the difference between the log-likelihood obtained through Viterbi forced alignment of the client model and world model, respectively. Finally a procedure used to determine the verification performances is presented.

Any biometric recognizer is vulnerable to direct spoofing attacks and automatic speaker verification (ASV) is no exception; replay, synthesis and conversion attacks all provoke false acceptances unless countermeasures are used. We focus on voice conversion (VC) attacks. Most existing countermeasures use full knowledge of a particular VC system to detect spoofing. We study a potentially more universal approach involving generative modeling perspective. Specifically, we adopt standard ivector representation and probabilistic linear discriminant analysis (PLDA) back-end for joint operation of spoofing attack detector and ASV system. As a proof of concept, we study a vocoder-mismatched ASV and VC attack detection approach on the NIST 2006 speaker recognition evaluation corpus. We report stand-alone accuracy of both the ASV and countermeasure systems as well as their combination using score fusion and joint approach. The method holds promise. Index Terms: speaker recognition, spoofing, voice conversion attack, i-vector, joint verification and anti-spoofing

Most of the speaker recognition systems use system features for speaker recognition which are mostly spectral in nature. Recently, there has been significant work on using source features, viz., prosodies and pitch dynamics, glottal flow derivative, Linear Prediction (LP) residual and its phase, wavelet-domain representation of LP residual, etc for speaker recognition. In this paper, a new source-like feature set, viz., Teager Energy Operator (TEO) Phase is developed for speaker recognition. Proposed TEO Phase has several salient features as compared to that of LP residual phase, viz., less number of spurious peaks in Hilbert envelope plot, and better perceptual and sample correlation with speech signal. Furthermore, this avoids use of voiced/unvoiced detection, preprocessing techniques such as windowing and pre-emphasis and LP residual computation. Experiments have been carried out for speaker recognition task using TEO Phase and LP residual phase with polynomial classifier of 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nd</sup> order approximation. For speaker verification, a reduction in equal error rate (EER) by 2.62 % over LP residual phase is achieved for proposed feature set.

This article verifies the reliability of automatic speaker verification (ASV) systems on new synthesis methods based on deep neural networks. ASV systems are widely used and applied regarding secure and effective biometric authentication. On the other hand, the rapid deployment of ASV systems contributes to the increased attention of attackers with newer and more sophisticated spoofing methods. Until recently, speech synthesis of the reference speaker did not seriously compromise the latest ASV systems. This situation is changing with the deployment of deep neural networks into the synthesis process. Projects including WaveNet, Deep Voice, Voice Loop, and many others generate very natural and high-quality speech that may clone voice identity. We are slowly approaching an era where we will not be able to recognize a genuine voice from a synthesized one. Therefore, it is necessary to define the robustness of current ASV systems to new methods of voice cloning. In this article, well-known SVM and GMM as well as new CNN-based ASVs are applied and subjected to synthesized speech from Tacotron 2 with the WaveNet TTS system. The results of this work confirm our concerns regarding the reliability of ASV systems against synthesized speech.

Significance of analytic phase of speech signals in speaker verification

It is widely known that the quality of confidence measure is critical for speech applications. In this paper, we present our recent work on improving word confidence scores by calibrating them using a small set of calibration data when only the recognized word sequence and associated raw confidence scores are made available. The core of our technique is the maximum entropy model with distribution constraints which naturally and effectively make use of the word distribution, the raw confidence-score distribution, and the context information. We demonstrate the effectiveness of our approach by showing that it can achieve relative 38% mean square error (MSE), 39% negative normalized likelihood (NNLL), and 23% equal error rate (EER) reduction on a voice mail transcription data set and relative 35% MSE, 45% NNLL, and 35% EER reduction on a command and control data set.

The relation of speaker verification to other pattern-recognition problems in speech is discussed, especially the distinction between speaker verification and speaker identification. The prospects for automatic speaker verification, its settings and applications are outlined. The techniques, evaluations, and implementations of various proposed speaker recognition systems are reviewed with special emphasis on issues peculiar to speaker verification. Two large-scale operating systems using different analysis techniques and applied to different settings are described.

With the growing number of devices that use voice biometric system for verification purposes, it becomes mandatory to accurately verify the target speaker. Moreover, intruders use different spoofing attacks to fool an automatic speaker verification (ASV) system. Voice replay attack being the easiest to generate among other spoofing attacks is frequently employed in front of the ASV systems of these devices to get access to the system for malicious purposes. Thus, we need to develop robust systems that are capable of classifying the target speaker and differentiate between the bonafide and replay voice. To combat the vulnerabilities of ASV systems, we proposed an automatic speaker verification and replay attack detection (ASVRAD) system. For this purpose, we proposed a novel audio feature i.e., Glottal Flow Gammatone Cepstrum Coefficients (GLGTCC), an enhanced version of Glottal Flow Cepstrum Coefficients (GLFCC) for accurate classification of the bonafide speakers and voice replay attacks detection. Our GLGTCC features can reliably capture the speaker-specific attributes from their vocal excitation patterns in the audio. Later, we employed a multiclass support vector machine (SVM) to classify the bonafide speakers, and a binary SVM for classification of bonafide and replay audios. Experimental results of our system on the standard ASVspoof 2019 PA dataset achieved the lowest EER of 3.00&#x0025; for speaker verification and 18.38&#x0025; on voice replay attacks over the comparative methods.