Reconstructing Dual Learning for Neural Voice Conversion Using Relatively Few Samples

Automatic speaker verification (ASV) faces domain shift caused by the mismatch of intrinsic and extrinsic factors, such as recording device and speaking style, in real-world applications, which leads to severe performance degradation. Since single-speaker multi-condition (SSMC) data is difficult to collect in practice, existing domain adaptation methods are hard to ensure the feature consistency of the same class but different domains. To this end, we propose a cross-domain data generation method to obtain a domain-invariant ASV system. Inspired by voice conversion (VC) task, a StarGAN based generative model first learns cross-domain mappings from SSMC data, and then generates missing domain data for all speakers, thus increasing the intra-class diversity of the training set. Considering the difference between ASV and VC task, we renovate the corresponding training objectives and network structure to make the adaptation task-specific. Evaluations on achieve a relative performance improvement of about 5-8% over the baseline in terms of minDCF and EER, outperforming the CNSRC winner’s system of the equivalent scale.

This paper tackles GAN optimization and stability issues in the context of voice conversion. First, to simplify the conversion task, we propose to use spectral envelopes as inputs. Second we propose two adversarial weight training paradigms, the generalized weighted GAN and the generator impact GAN, both aim at reducing the impact of the generator on the discriminator, so both can learn more gradually and efficiently during training. Applying an energy constraint to the cycleGAN paradigm considerably improved conversion quality. A subjective experiment conducted on a voice conversion task on the voice conversion challenge 2018 dataset shows first that despite a significantly reduced network complexity, the proposed method achieves state-of-the-art results, and second that the proposed weighted GAN methods outperform a previously proposed one.

An artificial neural network is one of the most important models for training features in a voice conversion task. Typically, Neural Networks (NNs) are not effective in processing low-dimensional F0 features, thus this causes that the performance of those methods based on neural networks for training Mel Cepstral Coefficients (MCC) are not outstanding. However, F0 can robustly represent various prosody signals (e.g., emotional prosody). In this study, we propose an effective method based on the NNs to train the normalized-segment-F0 features (NSF0) for emotional prosody conversion. Meanwhile, the proposed method adopts deep belief networks (DBNs) to train spectrum features for voice conversion. By using these approaches, the proposed method can change the spectrum and the prosody for the emotional voice at the same time. Moreover, the experimental results show that the proposed method outperforms other state-of-the-art methods for voice emotional conversion.

Variational auto-encoder (VAE) is an effective neural network architecture to disentangle a speech utterance into speaker identity and linguistic content latent embeddings, then generate an utterance for a target speaker from that of a source speaker. This is possible by concatenating the identity embedding of the target speaker and the content embedding of the source speaker uttering a desired sentence. In this work, we propose to improve VAE models with self-attention and structural regularization (RGSM). Specifically, we found a suitable location of VAE’s decoder to add a self-attention layer for incorporating non-local information in generating a converted utterance and hiding the source speaker’s identity. We applied relaxed groupwise splitting method (RGSM) to regularize network weights and remarkably enhance generalization performance. In experiments of zero-shot many-to-many voice conversion task on VCTK data set, with the self-attention layer and relaxed group-wise splitting method, our model achieves a gain of speaker classification accuracy on unseen speakers by 28.3% while slightly improved conversion voice quality in terms of MOSNet scores. Our encouraging findings point to future research on integrating more variety of attention structures in VAE framework while controlling model size and overfitting for advancing zero-shot many-to-many voice conversions <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> The work was partially supported by NSF grants DMS-1854434 and DMS-1952644 at UC Irvine.

A novel approach to pitch transposition is presented, which allows to preserve the spectral envelope or to modify it in a voice conversion task. Pitch-shifted output is synthesized directly from the input signal using the phase vocoder, without explicitly estimating the fundamental frequency, which is often prone to errors. Frequency band extension technique is proposed to address the problem of bandwidth reduction when scaling pitch down. Artificial neural network was applied to transform the signal's spectral envelope in the full voice conversion task. Listening tests showed that 55% of listeners preferred the quality of the pitch modification using the phase vocoder over the one offered by a parametric approach.

Voice conversion (VC) algorithms modify the speech of a particular speaker to resemble that of another speaker. Many existing virtual reality (VR) and augmented reality (AR) systems make it possible to change the appearance of users, and if VC is added, then users can also change their voice. State-of-the-art VC methods employ recurrent neural networks (RNNs), including long short-term memory (LSTM) networks, for generating converted speech. However, it is difficult for RNNs to perform parallel computations because the computations at each timestep depend on the results of a previous timestep, which prevents them from operating in real-time. In contrast, we propose a novel VC approach based on a dilated convolutional neural network (Dilated CNN), which is a deep neural network model that allows for parallel computation. We adapted the Dilated CNN model to perform convolutions in both the forward and reverse directions to ensure the learning is successful. In addition, to ensure the model can be parallelized during both the training and inference phases, we developed a model architecture that predicts all output values from the value of the input speech, and does not rely on predicted values for the next input. The results demonstrate that the proposed VC approach has a faster conversion rate relative to that of state-of-the-art methods, while improving speech quality a little and maintaining speaker similarity.

Thinking about cross-domain image-to-image translation problems, where an input image belonging to domain U is transformed into an output image belonging to another domain V. A series of typical tasks, such as style transformation, colorization, super-resolution, can be seen as cross-domain image-to-image translation tasks. Recent methods such as Conditional Generative Adversarial Networks (cGANs) make big progress in this field, but they require paired image data, which is hard to obtain. The DualGAN (Unsupervised Dual Learning for Image-to-Image Translation) architecture was proposed to solve the issue of lack of paired data. But the pixel-level reconstruction losses of DualGAN are simple. In this paper, we replace the pixel-level reconstruction losses with the perceptual reconstruction losses, and propose a more advanced framework for cross-domain image-to-image translation named perceptual-DualGAN. The perceptual reconstruction losses consist of feature reconstruction losses and style reconstruction losses, both of them are computed from pretrained loss networks. Experiments on multiple image translation tasks show that our framework almost performs superior to other methods. And the results of experiments illustrate that our framework can generate more realistic and more natural photos.

An improved fully convolutional network based on post-processing with global variance (GV) equalization and noise-aware training (PN-FCN) for speech enhancement model is proposed. It aims at reducing the complexity of the speech improvement system, and it solves overly smooth speech signal spectrogram problem and poor generalization capability. The PN-FCN is fed with the noisy speech samples augmented with an estimate of the noise. In this way, the PN-FCN uses additional online noise information to better predict the clean speech. Besides, PN-FCN uses the global variance information, which improve the subjective score in a voice conversion task. Finally, the proposed framework adopts FCN, and the number of parameters is one-seventh of deep neural network (DNN). Results of experiments on the Valentini-Botinhaos dataset demonstrate that the proposed framework achieves improvements in both denoising effect and model training speed.

This paper proposes a new voice conversion (VC) task from human speech to dog-like speech while preserving linguistic information as an example of human to non-human creature voice conversion (H2NH-VC) tasks. Although most VC studies deal with human to human VC, H2NH-VC aims to convert human speech into non-human creature-like speech. Non-parallel VC allows us to develop H2NH-VC, because we cannot collect a parallel dataset that non-human creatures speak human language. In this study, we propose to use dogs as an example of a non-human creature target domain and define the "speak like a dog" task. To clarify the possibilities and characteristics of the "speak like a dog" task, we conducted a comparative experiment using existing representative non-parallel VC methods in acoustic features (Mel-cepstral coefficients and Mel-spectrograms), network architectures (five different kernel-size settings), and training criteria (variational autoencoder (VAE)- based and generative adversarial network-based). Finally, the converted voices were evaluated using mean opinion scores: dog-likeness, sound quality and intelligibility, and character error rate (CER). The experiment showed that the employment of the Mel-spectrogram improved the dog-likeness of the converted speech, while it is challenging to preserve linguistic information. Challenges and limitations of the current VC methods for H2NH-VC are highlighted.

We have developed a bi-directional neural network as a massively parallel computing architecture for the design of semiconductor devices and circuits. We call this operation reverse modeling since the neural network trained to model a circuit is used in a reverse direction. A feedforward neural network can be used to model the behavior of a system. It can quickly predict the system response of given input parameters. We have extended the applications of neural networks beyond their traditional roles of black box models. Our approach begins with a neural network trained to model the response of the circuit to design parameters. The reverse modeling is then carried out by applying a modified backpropagation learning rule to the trained network. This changes the multi-layer, feedforward neural network from a uni-directional model into a bi-directional model. In the forward direction, the neural network model predicts the circuit property from given design parameters. In the reverse direction, design parameters are synthesized from desired circuit properties. We have demonstrated this reverse modeling approach by designing an RF amplifier. A neural network was trained to model the relations between the matching circuit elements of the amplifier and its output characteristics (output power and power gain). The result model has an average error of 1.7%. The trained model was then used to synthesize matching circuits for 10 sets of desired output characteristics. The neural network synthesized circuits were simulated using LIBRA to verify their performance. The simulated circuit performance was, in average, within 2.6% of the desired characteristics.

Non-parallel many-to-many voice conversion is a kind of style transfer task in speech. Recently, AutoVC has been applied in this field as a popular solution, as it can achieve distribution-matching style transfer by training only the re- construction loss. However, in order to strike a good balance between timbre disentanglement and sound quality, AutoVC requires imposing very strict constraints on the dimensionality of the latent representation. This constraint affects the quality of the converted speech while making it challenging to apply to other datasets directly. This paper proposes a new voice conversion framework that uses only one encoder to obtain timbre and content information by partitioning the latent space in the channel dimension. Furthermore, two different types of classifiers and two additional reconstruction losses are proposed to ensure that different parts of the latent space contain only separated content and timbre information, respectively. Experiments on the VCTK dataset show that the proposed model achieves state-of-the-art results in terms of the naturalness and similarity of converted speech. In addition, we experimentally show that for different division proportions of latent space, the content and timbre information will always be well separated.

In this paper, we propose a vector quantization (VQ) based one-shot voice conversion (VC) approach without any supervision on speaker label. We model the content embedding as a series of discrete codes and take the difference between quantize-before and quantize-after vector as the speaker embedding. We show that this approach has a strong ability to disentangle the content and speaker information with reconstruction loss only, and one-shot VC is thus achieved.

In this paper, we present a voice conversion (VC) method that does not use any parallel data while training the model. VC is a technique where only speaker-specific information in source speech is converted while keeping the phonological information unchanged. Most of the existing VC methods rely on parallel data—pairs of speech data from the source and target speakers uttering the same sentences. However, the use of parallel data in training causes several problems: 1) the data used for the training are limited to the predefined sentences, 2) the trained model is only applied to the speaker pair used in the training, and 3) mismatches in alignment may occur. Although it is, thus, fairly preferable in VC not to use parallel data, a nonparallel approach is considered difficult to learn. In our approach, we achieve nonparallel training based on a speaker adaptation technique and capturing latent phonological information. This approach assumes that speech signals are produced from a restricted Boltzmann machine-based probabilistic model, where phonological information and speaker-related information are defined explicitly. Speaker-independent and speaker-dependent parameters are simultaneously trained under speaker adaptive training. In the conversion stage, a given speech signal is decomposed into phonological and speaker-related information, the speaker-related information is replaced with that of the desired speaker, and then voice-converted speech is obtained by mixing the two. Our experimental results showed that our approach outperformed another nonparallel approach, and produced results similar to those of the popular conventional Gaussian mixture models-based method that used parallel data in subjective and objective criteria.

Existing Tagalog Text-to-speech (TTS) systems still have room for improvement, and although recent attempts at creating local TTS systems for Philippine spoken languages were able to generate synthesized speech, they still possess relatively low Mean Opinion Scores (MOS), ranging from 1.5 to 3.9 (out of 5), when it comes to naturalness and intelligibility. Improving speech prosody, the main factor for a speech's naturalness or individuality, has been made possible through voice conversion (VC). This project aims to implement a VC system for Tagalog synthesized speech, specifically using Cycle Generative Adversarial Networks (Cycle-GAN), a state-of-the-art neural network architecture used in non-parallel VC. Inter-gender and intra-gender VC were made for two types of inputs: Google's own Tagalog TTS and a locally sourced TTS system built from Mary TTS. Results show that Google TTS and its VC models perform better overall than Mary TTS and its VC models. Mel Cepstral Distortions (MCD) and F0: Root Mean Square Errors (F0:RMSE) vary across all models, reaching an MCD as low as 6.52 dB for Google TTS' intra-gender VC and an F0:RMSE as low as 16.92 Hz from Google TTS' inter-gender VC. Meanwhile, undergoing VC also caused a degradation in perceived speech quality as seen in a decrease in MOS across all VC models. Inter-gender VC for both TTS inputs were subjectively more preferred over intra-gender VC, reaching MOS values of 3.76 and 2.32 for Google TTS and Mary TTS inputs, respectively. Furthermore, it was also shown that male respondents were likely to rate higher opinion scores for intra-gender VC than female respondents, likely due to differences in hearing sensitivities.

Currently, there is a great need for mobile monitoring systems used to detect heart rate abnormalities in citizens working or living in hard-to-reach areas. Due to problems with transmitting data from such areas, it is desirable to carry out the entire information processing process at the study site. But, despite the limited hardware costs for the implementation of such systems, it is necessary to ensure high reliability of monitoring results. Purpose is to develop a structural diagram of a mobile heart rate analysis device, select the architecture of the neural network of the conversion and segmentation subsystem of the mobile device, determine the structure of the neural network of the conversion and segmentation subsystem, train the neural network of the conversion and segmentation subsystem, conduct software modeling of the neural network of the conversion and segmentation subsystem, implementation of a neural network conversion and segmentation subsystem based on field programmable gate arrays (FPGAs). A structural diagram of a mobile heart rate analysis device has been developed. The choice of the neural network architecture of the mobile device conversion and segmentation subsystem is justified. The structure of the neural network of the transformation and segmentation subsystem is determined. The neural network of the transformation and segmentation subsystem was trained. A software simulation of the neural network of the transformation and segmentation subsystem was carried out, confirming the high reliability of the proposed neural network structure. A neural network conversion and segmentation subsystem has been implemented in the hardware description language VHDL based on an FPGA. The developed structural diagram of the mobile device and the neural network structure of the conversion and segmentation subsystem can be used when implementing mobile heart rate monitoring means.