Nonparallel Emotional Speech Conversion

We propose USTNet, a novel deep learning approach designed for learning shape‐to‐shape translation from unpaired domains in an unsupervised manner. The core of our approach lies in disentangled representation learning that factors out the discriminative features of 3D shapes into content and style codes. Given input shapes from multiple domains, USTNet disentangles their representation into style codes that contain distinctive traits across domains and content codes that contain domain‐invariant traits. By fusing the style and content codes of the target and source shapes, our method enables us to synthesize new shapes that resemble the target style and retain the content features of source shapes. Based on the shared style space, our method facilitates shape interpolation by manipulating the style attributes from different domains. Furthermore, by extending the basic building blocks of our network from two‐class to multi‐class classification, we adapt USTNet to tackle multi‐domain shape‐to‐shape translation. Experimental results show that our approach can generate realistic and natural translated shapes and that our method leads to improved quantitative evaluation metric results compared to 3DSNet. Codes are available at https://Haoran226.github.io/USTNet.

Convolutional neural networks (CNNs) have presented their potential in filling large missing areas with plausible contents. To address the blurriness issue commonly existing in the CNN-based inpainting, a typical approach is to conduct texture refinement on the initially completed images by replacing the neural patch in the predicted region using the closest one in the known region. However, such a processing might introduce undesired content change in the predicted region, especially when the desired content does not exist in the known region. To avoid generating such incorrect content, in this paper, we propose a content inference and style imitation network (CISI-net), which explicitly separate the image data into content code and style code. The content inference is realized by performing inference in the latent space to infer the content code of the corrupted images similar to the one from the original images. It can produce more detailed content than a similar inference procedure in the pixel domain, due to the dimensional distribution of content being lower than that of the entire image. On the other hand, the style code is used to represent the rendering of content, which will be consistent over the entire image. The style code is then integrated with the inferred content code to generate the complete image. Experiments on multiple datasets including structural and natural images demonstrate that our proposed approach out-performs the existing ones in terms of content accuracy as well as texture details.

Recent advancements in machine learning comprise generative models such as autoencoders (AE) for learning and compressing 3D data to generate low-dimensional latent representations of 3D shapes. Learning latent representations that disentangle the underlying factors of variations in 3D shapes is an intuitive way to achieve generalization in generative models. However, it remains an open problem to learn a generative model of 3D shapes such that the latent variables are disentangled and represent different interpretable aspects of 3D shapes. In this paper, we propose Split-AE, which is an autoencoder-based architecture for partitioning the latent space into two sets, named as content and style codes. The content code represents global features of 3D shapes to differentiate between semantic categories of shapes, while style code represents distinct visual features to differentiate between shape categories having similar semantic meaning. We present qualitative and quantitative experiments to verify feature disentanglement using our Split-AE. Further, we demonstrate that, given a source shape as an initial shape and a target shape as a style reference, the trained Split-AE combines the content of a source and style of a target shape to generate a novel augmented shape, that possesses the distinct features of the target shape category yet maintains the similarity of the global features with the source shape. We conduct a qualitative study showing that the augmented shapes exhibit a realistic interpretable mixture of content and style features across different shape classes with similar semantic meaning.

Image-to-image translation methods have progressed from only considering the image-level information to integrating the global- and instance-level information. However, only the foreground instances are refined, and the background semantics are taken as an entire feature, which causes a substantial loss of the semantic information in the translation. Additionally, the insufficient quality of the translated semantic regions also leads to an unsatisfactory performance of the object recognition or visual odometry tasks in which the translated images/videos are further used. In this paper, we propose a novel generative adversarial network for panoptic-level image-to-image translation (PanopticGAN). The proposed method has three advantages: (1) the extracted panoptic perception (i.e., the foreground instances and background semantic regions) as content codes are aligned with the sampled panoptic style codes, which considers the panoptic-level information to avoid the semantic information loss, and the latent space of each object has a rich fusion of content and style codes to generate the higher-fidelity results; (2) a feature masking module is proposed to extract the representations within each object contour by masks for sharpening the object boundaries; (3) the improved fidelity of the translated semantic regions further contributes to enhancing the performance of the object recognition or visual odometry tasks that the translated images/videos are used in. In this paper, we also annotate a compact panoptic segmentation dataset for the thermal-to-color translation task. Extensive experiments are conducted to demonstrate the effectiveness of our PanopticGAN over the latest methods.

Pattern analysis of EEG responses to speech and voice: Influence of feature grouping

Voice conversion is a challenging task which transforms the voice characteristics of a source speaker to a target speaker without changing linguistic content. Recently, there have been many works on many-to-many Voice Conversion (VC) based on Variational Autoencoder (VAEs) achieving good results, however, these methods lack the ability to disentangle speaker identity and linguistic content to achieve good performance on unseen speaker scenarios. In this paper, we propose a new method based on feature disentanglement to tackle many to many voice conversion. The method has the capability to disentangle speaker identity and linguistic content from utterances, it can convert from many source speakers to many target speakers with a single autoencoder network. Moreover, it naturally deals with the unseen target speaker scenarios. We perform both objective and subjective evaluations to show the competitive performance of our proposed method compared with other state-of-the-art models in terms of naturalness and target speaker similarity.

Automatic speaker recognition using time alignment of spectrograms

Face-swapping deepfake poses significant risks, including privacy violations, misinformation, and defamation, amplified by the availability of pretrained models on open-source platforms. Proactive defense strategies aim to disrupt deepfake generation by modifying the original images to protect identity features. However, existing methods often introduce artifacts in facial images or rely on specific deepfake models, limiting their usability. To address these problems, we propose a style code orchestration in latent space (SCOL) method that obfuscates identity by fusing different identities in the latent space without requiring face recognition models. This study optimizes the generator to follow the original appearance while retaining the obfuscated identity via identity-preserving constraints. Further, appearance-dominant components in the latent code are aligned for visual consistency. An identity inversion attack is introduced using opposite style codes to improve the effectiveness of the defense. Experimental results demonstrate that SCOL robustly defends against various face-swapping deepfake methods, maintaining visual consistency.

Privacy is the major concern that comes to the user's mind before sharing their data. There are various methods proposed in literature for providing privacy to speech data. Previous works that have been done to protect the speaker identity were done for speech applications like automatic speech recognition (ASR), speech analysis, etc. For these applications the presence of speaker identity is not essential while processing. The objective of this work is to provide privacy to the task in which presence of speaker identity is essential at the time of processing. In this work, privacy is provided to the speaker identity information present in speech signals while performing automatic speaker verification (ASV) tasks. In order to achieve the same, this work proposes a prosody modification based approach. The proposed approach is able to conceal the speaker identity from human perception by changing the pitch of the speech utterances with a pitch modification factor of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\alpha\geq 1$</tex> But at the same time the ASV system provides consistent performance irrespective of the change in pitch (i.e. for <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\alpha\geq 1)$</tex> . The same evidence has been shown through experiments in TIMIT and IITG-MV databases. A subjective study has also performed to verify the extent of speaker anonymization with respect to humans. The subjective study evaluates the performance in terms of mean opinion score (MOS). The observed MOS signifies the ability of the proposed approach to conceal the speaker's identity.

Speech signal is the primary mode of communication amongst humans. It is also used for the communication between human and machine, i.e., human-machine interaction (IoTs), communication with a range of electronic devices, e.g., Internet of Things (IoTs), and even with virtual assistants (VAs) or online search engines, e.g., Google etc. The use of speech signals for the communication over mobile phones, and smart phones based virtual assistants such as Cortana, Siri, Alexa etc. is well known. Usually, this communication involves some communication medium, which is susceptible to noise. The speech or audio signal used for the communication may be adversely affected by the noise that is added unwantedly over the communication channel or medium. It not only degrades the quality of communication and intelligibility of the message or content, but also leads to poor experience. The quality of speech signal may be degraded significantly, due to changes in the characteristics of speech signal, during the process from transmission to reception and the signal conversions involved at both ends. Hence there is a need for algorithms, signal processing methods, acoustic features and classifiers etc. for noise detection and noise reduction during the speech signal communication process. In this paper, a comprehensive review of various speech signal databases, speech signal processing methods used, acoustic features extracted and the classifiers used in different studies is presented. A comprehensive and comparative view of results obtained in different studies, and the applications is also presented. This study should be helpful to all the researchers working in the domains of speech signal processing in noisy environments or degraded speech etc.

Multi-view intact space clustering

Regions of nonmodal phonation, which exhibit deviations from uniform glottal-pulse periods and amplitudes, occur often in speech and convey information about linguistic content, speaker identity, and vocal health. Some aspects of these deviations are random, including small perturbations, known as jitter and shimmer, as well as more significant aperiodicities. Other aspects are deterministic, including repeating patterns of fluctuations such as diplophonia and triplophonia. These deviations are often the source of misinterpretation of the spectrum. In this paper, we introduce a general signal-processing framework for interpreting the effects of both stochastic and deterministic aspects of nonmodality on the short-time spectrum. As an example, we show that the spectrum is sensitive to even small perturbations in the timing and amplitudes of glottal pulses. In addition, we illustrate important characteristics that can arise in the spectrum, including apparent shifting of the harmonics and the appearance of multiple pitches. For stochastic perturbations, we arrive at a formulation of the power-spectral density as the sum of a low-pass line spectrum and a high-pass noise floor. Our findings are relevant to a number of speech-processing areas including linear-prediction analysis, sinusoidal analysis-synthesis, spectrally derived features, and the analysis of disordered voices.

In existing speech coding systems, all quantizer codebooks are designed to suit the statistical and perceptual characteristics of speech signals of a population of speakers. However, an individual's speech signal does not exhibit, even over a long time, the entire range of characteristics of the population. With the advent of the personal communication systems, personal information might become available and be used to improve the rate-distortion performance of speech coders. We assess the potential gain of personal speech coding by designing codebooks for individual speakers. Spectral quantisation, excitation and pitch lag codebooks of existing CELP coders are redesigned. The gains appear to be modest, suggesting that we need to use a different coding framework, which can model personal characteristics explicitly. Amongst the components, the spectral quantizer seems to be most amenable to personalization.

The necessity to use very large datasets in order to train Generative Adversarial Networks (GANs) has limited their use in cases where the data at disposal are scarce or poorly labelled (e.g., in real life applications). Recently, meta-learning proved that it can help solving effectively few-shot classification problems, but its use in noise-to-image generation was only partially explored. In this paper, we took the first step into applying a meta-learning algorithm (Reptile), to the discriminator of a GAN and to a mapping network in order to optimize the random noise z to guide the generator network into producing images belonging to specific classes. By doing so, we prove that the latent space distribution is crucial for the generation of sharp samples when few training data are at disposal and also managed to generate samples of previously unseen classes just by optimizing the latent space without changing any parameter in the generator network. Finally, we show several experiments with two widely used datasets: MNIST and Omniglot.

Silent gestures consist of complex multi‐articulatory movements but are now primarily studied through categorical coding of the referential gesture content. The relation of categorical linguistic content with continuous kinematics is therefore poorly understood. Here, we reanalyzed the video data from a gestural evolution experiment (Motamedi, Schouwstra, Smith, Culbertson, & Kirby, 2019), which showed increases in the systematicity of gesture content over time. We applied computer vision techniques to quantify the kinematics of the original data. Our kinematic analyses demonstrated that gestures become more efficient and less complex in their kinematics over generations of learners. We further detect the systematicity of gesture form on the level of thegesture kinematic interrelations, which directly scales with the systematicity obtained on semantic coding of the gestures. Thus, from continuous kinematics alone, we can tap into linguistic aspects that were previously only approachable through categorical coding of meaning. Finally, going beyond issues of systematicity, we show how unique gesture kinematic dialects emerged over generations as isolated chains of participants gradually diverged over iterations from other chains. We, thereby, conclude that gestures can come to embody the linguistic system at the level of interrelationships between communicative tokens, which should calibrate our theories about form and linguistic content.