Non-Parallel Training in Voice Conversion Using an Adaptive Restricted Boltzmann Machine

In speech-to-singing (STS) voice conversion, the source speech signals from a speaker are used to generate his/her singing voice. Such a process requires accurate detection of boundaries between phonemes and words in the speech signal. The computation and modification of analysis parameters of speech signals with respect to the target musical scores or singing templates, largely depend upon estimation of phoneme durations. In this paper, an improved dual alignment scheme for speech and singing voices in template-based STS (TSTS) systems is proposed. The subsequence dynamic time warping (subDTW) is employed to match source speech to singer's speech in the first pass of dual alignment. We assume that an accurate correspondence between singer's speech and target singing vocals has been established as part of the singing template development. Therefore, once the source speech is aligned with the singer's speech, it is automatically aligned with singing template, that we call the second pass of dual alignment. The proposed scheme delivers a relative reduction of 95.8% in word alignment error, over the baseline dynamic time warping (DTW) approach. Also, it provides a relative improvement of 38.7% in mean opinion scores of synthesized singing voices in subjective studies, over the same baseline. We demonstrate that the proposed dual alignment with the subDTW is effective in STS conversion applications.

In conventional voice conversion methods, some features of a speech signal’s spectrum envelope are first extracted. Then, these features are converted so as to best match a target speaker’s speech by designing and using a set of conversions. Ultimately, the spectrum envelope of the target speaker’s speech signal is reconstructed from the converted features. The spectrum envelope reconstructed from the converted features usually deviates from its natural form. This aberration from the natural form observed in cases such as over-smoothing, over-fitting, and widening of formants is partially caused by two factors: (1) there is an error in the reconstruction of spectrum envelope from the features, and (2) the set of features extracted from the spectrum envelope of the speech signal is not closed. A method is put forward to improve the naturalness of speech by means of \(\epsilon \)-closed sets of extended vectors in voice conversion systems. In this approach, \(\epsilon \)-closed sets to reconstruct the natural spectrum envelope of a signal in the synthesis phase are introduced. The elements of these sets are generated by forming a group of extended vectors of features and applying a quantization scheme on the features of a speech signal. The use of this method in speech synthesis leads to a noticeable reduction of error in spectrum reconstruction from the features. Furthermore, the final spectrum envelope extracted from voice conversions maintains its natural form and, consequently, the problems arising from the deviation of voice from its natural state are resolved. The above method can be generally used as one phase of speech synthesis. It is independent of the voice conversion technique used and its parallel or non-parallel training method, and can be applied to improve the naturalness of the generated speech signal in all common voice conversion methods. Moreover, this method can be used in other fields of speech processing like texts to speech systems and vocoders to improve the quality of the output signal in the synthesis step.

Voice conversion with SI-DNN and KL divergence based mapping without parallel training data

The ideal goal of voice conversion is to convert the source speaker’s speech to sound naturally like the target speaker while maintaining the linguistic content and the prosody of the source speech. However, current approaches are insufficient to achieve comprehensive source prosody transfer and target speaker timbre preservation in the converted speech, and the quality of the converted speech is also unsatisfied due to the mismatch between the acoustic model and the vocoder. In this paper, we leverage the recent advances in information perturbation and propose a fully end-to-end approach to conduct high-quality voice conversion. We first adopt information perturbation to remove speaker-related information in the source speech to disentangle speaker timbre and linguistic content and thus the linguistic information is subsequently modeled by a content encoder. To better transfer the prosody of the source speech to the target, we particularly introduce a speaker-related pitch encoder which can maintain the general pitch pattern of the source speaker while flexibly modifying the pitch intensity of the generated speech. Finally, one-shot voice conversion is set up through continuous speaker space modeling. Experimental results indicate that the proposed end-to-end approach significantly outperforms the state-of-the-art models in terms of intelligibility, naturalness, and speaker similarity.

Voice conversion (VC), as a voice style transfer technology, is becoming increasingly prevalent while raising serious concerns about its illegal use. Proactively tracing the origins of VC-generated speeches, i.e., speaker traceability, can prevent the misuse of VC, but unfortunately has not been extensively studied. In this paper, we are the first to investigate the speaker traceability for VC and propose a traceable VC framework named VoxTracer. Our VoxTracer is similar to but beyond the paradigm of audio watermarking. We first use unique speaker embedding to represent speaker identity. Then we design a VAE-Glow structure, in which the hiding process imperceptibly integrates the source speaker identity into the VC, and the tracing process accurately recovers the source speaker identity and even the source speech in spite of severe speech quality degradation. To address the speech mismatch between the hiding and tracing processes affected by different distortions, we also adopt an asynchronous training strategy to optimize the VAE-Glow models. The VoxTracer is versatile enough to be applied to arbitrary VC methods and popular audio coding standards. Extensive experiments demonstrate that the VoxTracer achieves not only high imperceptibility in hiding, but also nearly 100% tracing accuracy against various types of audio lossy compressions (AAC, MP3, Opus and SILK) with a broad range of bitrates (16 kbps - 128 kbps) even in a very short time duration (0.74s). Our source code is available at https://github.com/hongchengzhu/VoxTracer.

Sequence-to-sequence (seq2seq) voice conversion (VC) models have greater potential in converting electrolaryngeal (EL) speech to normal speech (EL2SP) compared to conventional VC models. However, EL2SP based on seq2seq VC requires a sufficiently large amount of parallel data for the model training and it suffers from significant performance degradation when the amount of training data is insufficient. To address this issue, we suggest a novel, two-stage strategy to optimize the performance on EL2SP based on seq2seq VC when a small amount of the parallel dataset is available. In contrast to utilizing high-quality data augmentations in previous studies, we first combine a large amount of imperfect synthetic parallel data of EL and normal speech, with the original dataset into VC training. Then, a second stage training is conducted with the original parallel dataset only. The results show that the proposed method progressively improves the performance of EL2SP based on seq2seq VC.

Voice Conversion (VC) is a task of converting speaker-dependent features of a source speaker's speech without changing the linguistic content. There are many successful VC systems, each trying to overcome some challenges. These challenges include the unavailability of parallel data and solving problems due to the language difference between the source and target speech. Also, one of these challenges is extending the VC system to cover a conversion across many source and target domains with minimal cost. Generative Adversarial Networks (GANs) are showing promising VC results. This work focuses on exploring many-to-many non-parallel GAN-based mono-lingual VC models (nine models that are highly cited), explains the used evaluation methods including objective and subjective methods (eight evaluation methods are presented), and comments on these models.

This paper proposes a model of speaker-normalized acoustic-toarticulatory mapping using statistical voice conversion. A mapping function from acoustic parameters to articulatory parameters is usually developed with a single speaker’s parallel data. Hence the constructed mapping model can work appropriately only for this specific speaker, and applying this model to other speakers degrades the performance of acoustic-to-articulatory mapping. In this paper, two models of speaker conversion and acoustic-to-articulatory mapping are implemented using Gaussian Mixture Models (GMM), and by integrating these two models, we propose two methods of speaker-normalized acoustic-to-articulatory mapping. One is concatenating these models sequentially, and the other integrates the two models into a unified model, where acoustic parameters of a speaker can be converted directly to articulatory parameters of another speaker. Experiments show that both methods can improve the mapping accuracy and that the latter method works better than the former method. Especially in the case of velar stop consonants, the mapping accuracy is higher by 0.6 mm. Index Terms: acoustic-to-articulatory mapping, Gaussian mixture model, voice conversion, speaker normalization

In this paper we deal with initial experiments on creating a new Czech voice conversion system. Voice conversion (VC) is a process which modifies the speech signal produced by one (source) speaker so that it sounds like another (target) speaker. Using VC technique a new voice for speech synthesizer can be prepared with no need to record a huge amount of new speech data. The transformation is determined using equal sentences from both speakers; these sentences are time-aligned using modified dynamic time warping algorithm. The conversion is divided into two stages corresponding to the source-filter model of speech production. Within this work we employ conversion function based on Gaussian mixture model for transforming the spectral envelope described by line spectral frequencies. Residua are converted using so called residual prediction techniques. Unlike in other similar research works, we predict residua not from the transformed spectral envelope, but directly from the source speech. Four versions of residual prediction are described and compared in this study. Objective evaluation of converted speech using performance metrics shows that our system is comparable with similar existing VC systems.KeywordsGaussian Mixture ModelDynamic Time WarpingResidual SignalSpeech DataSource SpeechThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we present a simple but novel framework to train a nonparallel many-to-many voice conversion (VC) model based on the encoder-decoder architecture. It is observed that an encoder-decoder text-to-speech (TTS) model and an encoder-decoder VC model have the same structure. Thus, we propose to pre-train a multi-speaker encoder-decoder TTS model and transfer knowledge from the TTS model to a VC model by (1) adopting the TTS acoustic decoder as the VC acoustic decoder, and (2) forcing the VC speech encoder to learn the same speaker-agnostic linguistic features from the TTS text encoder so as to achieve speaker disentanglement in the VC encoder output. We further control the conversion of the pitch contour from source speech to target speech, and condition the VC decoder on the converted pitch contour during inference. Subjective evaluation shows that our proposed model is able to handle VC between any speaker pairs in the training speech corpus of over 200 speakers with high naturalness and speaker similarity.

Speaker adaptation techniques provide a powerful solution to customise automatic speech recognition (ASR) systems for individual users. Practical application of unsupervised model-based speaker adaptation techniques to data intensive end-to-end ASR systems is hindered by the scarcity of speaker-level data and performance sensitivity to transcription errors. To address these issues, a set of compact and data efficient speaker-dependent (SD) parameter representations are used to facilitate both speaker adaptive training and test-time unsupervised speaker adaptation of state-of-the-art Conformer ASR systems. The sensitivity to supervision quality is reduced using a confidence score-based selection of the less erroneous subset of speaker-level adaptation data. Two lightweight confidence score estimation modules are proposed to produce more reliable confidence scores. The data sparsity issue, which is exacerbated by data selection, is addressed by modelling the SD parameter uncertainty using Bayesian learning. Experiments on the benchmark 300-hour Switchboard and the 233-hour AMI datasets suggest that the proposed confidence score-based adaptation schemes consistently outperformed the baseline speaker-independent (SI) Conformer model and conventional non-Bayesian, point estimate-based adaptation using no speaker data selection. Similar consistent performance improvements were retained after external Transformer and LSTM language model rescoring. In particular, on the 300-hour Switchboard corpus, statistically significant WER reductions of 1.0%, 1.3%, and 1.4% absolute (9.5%, 10.9%, and 11.3% relative) were obtained over the baseline SI Conformer on the NIST Hub5'00, RT02, and RT03 evaluation sets respectively. Similar WER reductions of 2.7% and 3.3% absolute (8.9% and 10.2% relative) were also obtained on the AMI development and evaluation sets.

An automatic speaker verification system aims to verify the speaker identity of a speech signal. However, a voice conversion system could manipulate a person’s speech signal to make it sound like another speaker’s voice and deceive the speaker verification system. Most countermeasures for voice conversion-based spoofing attacks are designed to discriminate bona fide speech from spoofed speech for speaker verification systems. In this paper, we investigate the problem of source speaker identification – inferring the identity of the source speaker given the voice converted speech. To perform source speaker identification, we simply add voice-converted speech data with the label of source speaker identity to the genuine speech dataset during speaker embedding network training. Experimental results show the feasibility of source speaker identification when training and testing with converted speeches from the same voice conversion model(s). In addition, our results demonstrate that having more converted utterances from various voice conversion model for training helps improve the source speaker identification performance on converted utterances from unseen voice conversion models.

This paper presents an alternative voice conversion technique using support vector machine (SVM)-regression as a tool which converts a source speaker’s voice to specific standard target speaker. The main objective of the work is to capture a nonlinear mapping function between the parameters for the acoustic features of the two speakers. Line spectral frequencies (LSFs) have been used as features to represent the vocal tract characteristics. We use kernel induced feature space with radial basis function network (RBFN) type SVM that uses gaussian kernel. The intonation characteristics (pitch contour) is modified using the baseline technique, i.e. gaussian normalization. The transformed LSFs along with the modified pitch contour are used to synthesize the speech signal for the desired target speaker. The target speaker’s speech signal is synthesized and evaluated using both the subjective and the listening tests. The results signify that the proposed model improves the voice conversion performance in terms of capturing the speaker’s identity as compared to our previous approach. In the previous approach we used feed forward neural network (FFNN) based model for vocal tract modification and codebook based method for pitch contour modification. However, the performance of the proposed system can further be improved by suitably modifying various user-defined parameters used in regression analysis and using more training LSF vectors in the training stage.

The purpose of a voice conversion (VC) system is to change the perceived speaker identity of a speech signal to sound as if a target speaker had spoken it. In this paper, we propose a residual prediction method for the spectral detail transformation component of a VC system. The algorithm described here is based on the LPC analysis/synthesis framework, and achieves residual prediction from LPC parameters during voiced speech. This step consists of a GMM based LPC parameter classifier and a LPC residual codebook. The predicted residual is then combined with all-pole LPC spectrum to synthesize speech signal. Several aspects of this residual prediction method, including the validation of the codebook and the performance of the coded speech are tested using objective measures. The converted speech is found nearly indistinguishable from the target speaker's individuality in informal listening tests.

We describe novel training methods based on sequence-to-sequence (seq2seq) voice conversion (VC) to address two practical issues in electrolaryngeal (EL)-speech-to-normalspeech conversion (EL2SP): 1) low-resource training data, and 2) the huge domain shift gap during transfer learning. Seq2seq VC is promising for EL2SP but suffers performance degradation without sufficiently high-quality, parallel training data. The common method utilizes transfer learning to address low-resource issue, following a direct pretraining–fine-tuning paradigm. However, in EL2SP, as huge domain shifts exist between upstream larger-scale normal corpora and the target EL2SP dataset, the common method cannot achieve effective transfer learning, limiting EL2SP performance. Therefore, we present training methods with multistage pretraining and fine-tuning techniques, particularly including an encoder adaptation training and a two-stage fine-tuning method, both leveraging low-quality synthetic data (SD), to improve transferability. During pretraining, aside from knowledge transfer from a more easily accessible TTS database into seq2seq VC pretraining, encoder adaptation training further minimizes the representation learning gap of the encoder in comprehending EL speech, facilitating smoother transfer for downstream EL2SP. Subsequently, the two-stage EL2SP finetuning finalizes a generalized and stable performance. Moreover, by effectively utilizing low-quality SD, our techniques relax training data demands and enhance practicality. Experimental results demonstrate our methods dramatically outperform a baseline using the common method regarding conversion quality and intelligibility. Comparative analyses confirm progressive performance gains with deeper system designs.