Demystifying Encoder–Decoder Neural Networks: Correncoder for Regression via Latent Spaces [Lecture Notes

A person’s gait is a behavioral trait that is uniquely associated with each individual and can be used to recognize the person. As information about the human gait can be captured by wearable devices, a few studies have led to the proposal of methods to process gait information for identification purposes. Despite recent advances in gait recognition, an open set gait recognition problem presents challenges to current approaches. To address the open set gait recognition problem, a system should be able to deal with unseen subjects who have not included in the training dataset. In this paper, we propose a system that learns a mapping from a multimodal time series collected using insole to a latent (embedding vector) space to address the open set gait recognition problem. The distance between two embedding vectors in the latent space corresponds to the similarity between two multimodal time series. Using the characteristics of the human gait pattern, multimodal time series are sliced into unit steps. The system maps unit steps to embedding vectors using an ensemble consisting of a convolutional neural network and a recurrent neural network. To recognize each individual, the system learns a decision function using a one-class support vector machine from a few embedding vectors of the person in the latent space, then the system determines whether an unknown unit step is recognized as belonging to a known individual. Our experiments demonstrate that the proposed framework recognizes individuals with high accuracy regardless they have been registered or not. If we could have an environment in which all people would be wearing the insole, the framework would be used for user verification widely.

Face-swapping technology has been widely used in people's life, and people also put forward higher requirements for it. Most of the current face-swapping methods are difficult to generate a high-definition face image. Through StyleGAN, we can generate high-definition face images. However, face-swapping with StyleGAN is still challenging. Firstly, we need to map the target image to the latent space of StyleGAN. Many tasks need to map the input image to a new latent space for face-swapping, because identity features are complex and challenging to map to specific latent space layers directly. So face-swapping is completed in the remapping process, which consumes excess computing resources for reconstruction. And the generated image is difficult to maintain the original image color, face attributes, background and other attributes. We propose a new method, which only edits the code of w+ latent space of StyleGAN to complete the face-swapping and generate high-definition face images. We propose the GAN inversion method to improve the effect of face swapping, which combines convolution networks' advantages in extracting texture features and the benefits of transformers in extracting structure features. In the latent space of StyleGAN, the low-level feature layer is dominated by structure information, and the high-level feature layer is overwhelmed by texture information. Furthermore, we propose latent space selection, through which the neural network can learn disentangled representations of identity information in the latent space. Finally, we improved the post-processing process of face swapping to keep the image's background. Our method can complete face-swapping by editing the w+ space. Thus, high-quality face image can be generated and a lot of computing resource is saved on image reconstruction. At the same time, our method can keep other attributes better in the face-swapping process.

Robot skill learning in latent space of a deep autoencoder neural network

Deep Neural Networks have been proved as the go-to approach in modeling data distribution in a latent space, especially in Neural Style Transfer (NST), which casts a specific style extracted from a source image to another target image by calibrating the style and content information in a latent space. While existing methods focuses on different ways to extract features that more precisely describe style or content information to improve existing NST pipelines, the latent space of the NST model has not been well-explored. In this paper, we show that different half-spaces in the latent space are actually associated with particular styles of a network’s generated images. The corresponding constraints of these half-spaces can be computed by using linear classifiers, e.g. a Support Vector Machines (SVM). Leveraging the understanding of the relation between half-spaces in the latent space and output style, we propose the Linear Modification for Latent Representations (LMLR), a method that effectively increases or decreases the level of stylizing in the output image for any given NST model. We empirically evaluate our method on several state-of-the-art NST models and show that LMLR can manipulate the level of stylizing in the output image.

Magnetic Resonance Imaging (MRI) is one of the most commonly used modalities in medical imaging, and one of the main limitations is its slow scanning and reconstruction speed. Recently, deep learning used for compressed sensing (CS) methods have been proposed to accelerate the acquisition by undersampling in the K-space and reconstruct images with neural networks. However, there are still some challenges remained: First, directly training networks based on L1/L2 distance to the target fully sampled images may lead to fuzzy reconstruction images because L1/L2 loss only enforces the overall image or patch similarity, but does not consider the local details such as anatomical sharpness. Second, Generative Adversarial Networks (GAN) can partially solve this problem. The undersampling image gets the latent space through the encoder, and the image is reconstructed by the decoder based on GAN loss, but it may generate unrealistic details by lacking of constraints in K-space domain. Third, most of the networks after training are fixed and have limited adaptation capability in the inference time, and the patient-specific information cannot be effectively used. To resolve these challenges, we proposed a new compressed sensing GAN reconstruction method, and there are two main contributions: (1) We proposed a encoder-decoder structure, which guided GAN optimization strategy data-consistency in latent space to improve the reconstruction quality such as preserving more local details and improving the anatomical sharpness while constraining GAN to follow the data distribution in the K-space to prevent the unrealistic details. (2) An online update strategy is used to find the best representation in the latent space for the underlying patient, and the reconstruct result can be further improved by incorporating the patient-specific information. Extensive experimental results show the effectiveness of our method. KeywordsMRI reconstructionData-consistencyLatent spaceOnline update

Deep neural networks are widely used in computer vision for image classification, segmentation and generation. They are also often criticised as “black boxes” because their decision-making process is often not interpretable by humans. However, learning explainable representations that explicitly disentangle the underlying mechanisms that structure observational data is still a challenge. To further explore the latent space and achieve generic processing, we propose a pipeline for discovering the explainable directions in the latent space of generative models. Since the latent space contains semantically meaningful directions and can be explained, we propose a pipeline to fully resolve the representation of the latent space. It consists of a Dirichlet encoder, conditional deterministic diffusion, a group-swap and a latent traversal module. We believe that this study provides an insight into the advancement of research explaining the disentanglement of neural networks in the community.

Most examinations of neural networks' learned latent spaces typically employ dimensionality reduction techniques such as t-distributed stochastic neighbor embedding (t-SNE) or uniform manifold approximation and projection (UMAP). These methods distort the local neighborhood in the visualization, making it hard to distinguish the structure of a subset of samples in the latent space. In response to this challenge, we introduce the k* distribution and its corresponding visualization technique. This method uses local neighborhood analysis to guarantee the preservation of the structure of sample distributions for individual classes within the subset of the learned latent space. This facilitates easy comparison of different k* distributions, enabling analysis of how various classes are processed by the same neural network. Our study reveals three distinct distributions of samples within the learned latent space subset: 1) fractured; 2) overlapped; and 3) clustered, providing a more profound understanding of the existing contemporary visualizations. Experiments show that the distribution of samples within the network's learned latent space significantly varies depending on the class. Furthermore, we illustrate that our analysis can be applied to explore the latent space of diverse neural network architectures, various layers within neural networks, transformations applied to input samples, and the distribution of training and testing data for neural networks. Thus, the k* distribution should aid in visualizing the structure inside neural networks and further foster their understanding.

Effective health indicator (HI) construction can help equipment managers detect the abnormal state of rotating machinery quickly. However, although the current deep learning-based HI construction methods have good life prediction value, most of them lose the ability to detect device anomalies and little work has been done on model interpretability. Therefore, an interpretable HI construction method based on semi-supervised autoencoder (AE) latent space variance maximization (SSALSVM) was proposed to monitor the health status of bearings. In order to fully excavate degradation features inside the device and make the model focus on the encoding process, a deep convolutional neural network (DCNN) is used as the encoding layer, while only a layer of fully-connected layer is used as the decoding layer. In addition, to enable the latent space to capture the device early degradation point (EDP) successfully, an auxiliary layer is added to the output of the encoder layer. Simultaneously, for improving the sensitivity of the indicator to capture equipment abnormal state and highlight the difference between equipment health state and degradation state, the constraint of variance maximization is added into the latent space. The model optimizing process was presented by observing the projected variance of the test set in latent space of each epoch model. The validity of the proposed HI was verified by comparison experiments on two datasets.

Discrete and combinatorial optimization can be notoriously difficult due to complex and rugged characteristics of the objective function. We address this challenge by mapping the search process to a continuous space using recurrent neural networks. Alongside with an evolutionary run, we learn three mappings: from the original search space to a continuous Cartesian latent space, from that latent space back to the search space, and from the latent space to the search objective. We elicit gradient from that last network and use it to perform moves in the latent space, and apply this Neuromemetic Evolutionary Optimization (NEO) to evolutionary synthesis of programs. Evaluation on a range of benchmarks suggests that NEO significantly outperforms conventional genetic programming.

Summary Full-waveform inversion (FWI) can capture high-resolution geophysical properties but tends to fall into local minima. We addressed this problem by reducing the degree of freedom in the optimization problem and estimating an initial velocity model from shot gathers; our method comprises three steps. First, we trained a variational autoencoder using synthetic velocity models and created a low-dimensional latent space that represents realistic velocity models. Second, we prepared another encoder model, which estimates the latent features of the corresponding velocity model from shot gathers. Finally, we implemented latent space optimization (LSO) from the initial guess selected by the encoder. In the experiment, LSO reproduced much more accurate and realistic velocity models than the naive FWI, even when the initial guess was considerably different from the ground truth. This implies that with our method the latent space becomes easier to optimize than the original velocity model space. Furthermore, we verified that LSO using an initial guess selected by the encoder could increase the estimation accuracy. Our method also succeeded in accurately estimating the Marmousi2 model, without using any part of the benchmark dataset to directly train the variational autoencoder.

The objective of the proposal is to analyze what latent space is within a Deep-Learning system and how its visualization is capable of triggering a meaning-effect concerning the epistemology of big data. The latent space is the mathematical space that maps what a Neural Network has learned from the training dataset. It is the result of the compression of the input data and the step before the Neural Network’s output, a step that usually remains invisible to the human eye, rendering effective the promise of a transparent effect of reality generally promoted by Artificial Intelligence technologies. Precisely in contrast with this promise, the visualization of this complex spatiality makes accessible, and therefore intelligible, the epistemic and rhetorical relations inscribed within datasets, intended as archives that gather information. To achieve my objective, I will consider an artistic project realized by multimedia artist and coder Jake Elwes, Zizi-Queering the Dataset (2019), a multi-channel video where different facial portraits are shown in a morphing loop that visualizes what a Generative Adversarial Network has learned from the re-training of a dataset containing portraits with another one containing facial images of drag and non-binary individuals. This artistic gesture has led to a series of epistemic issues concerning big data and their situated and ideological meaning.

There is a growing interest in the joint application of graph signal processing and neural networks (NNs) for learning problems involving complex, non-linear and/or non-Euclidean datasets. This paper proposes an overparametrized graph-aware NN architecture able to represent a non-linear mapping between two graph signals, each defined on a different graph. The considered architecture is based on two NNs and a common latent space. Specifically, we consider an overparametrized graph-aware NN encoder which maps the input graph signal to a latent space, followed by an overparametrized graph-aware NN decoder that transforms the latent representation to the output graph signal. The parameters of the two NNs are jointly tuned by applying the back-propagation algorithm with an early stopping procedure to prevent overfitting. The overall architecture can be interpreted as an overparametrized graph-aware encoder/decoder NN operating over two different graphs. A key element in the encoder (decoding) scheme is the consideration of a nested collection of parametric graph-aware (down-) up-sampling operators, whose design will be studied in detail. We show by numerical simulations that the proposed scheme outperforms the corresponding benchmark NN architectures, previously introduced in the literature.

BackgroundPresently, there is a wide variety of classification methods and deep neural network approaches in bioinformatics. Deep neural networks have proven their effectiveness for classification tasks, and have outperformed classical methods, but they suffer from a lack of interpretability. Therefore, these innovative methods are not appropriate for decision support systems in healthcare. Indeed, to allow clinicians to make informed and well thought out decisions, the algorithm should provide the main pieces of information used to compute the predicted diagnosis and/or prognosis, as well as a confidence score for this prediction.MethodsHerein, we used a new supervised autoencoder (SAE) approach for classification of clinical metabolomic data. This new method has the advantage of providing a confidence score for each prediction thanks to a softmax classifier and a meaningful latent space visualization and to include a new efficient feature selection method, with a structured constraint, which allows for biologically interpretable results.ResultsExperimental results on three metabolomics datasets of clinical samples illustrate the effectiveness of our SAE and its confidence score. The supervised autoencoder provides an accurate localization of the patients in the latent space, and an efficient confidence score. Experiments show that the SAE outperforms classical methods (PLS-DA, Random Forests, SVM, and neural networks (NN)). Furthermore, the metabolites selected by the SAE were found to be biologically relevant.ConclusionIn this paper, we describe a new efficient SAE method to support diagnostic or prognostic evaluation based on metabolomics analyses.

Quantitative and flexible 3D shape dataset augmentation via latent space embedding and deformation learning

Catalogs have been used for over a century for designing engineering systems. While catalogs are excellent repositories of engineering information, they are difficult to navigate, specifically to spot clusters, gaps, substitutes, and outliers. Inspired by Ashby charts for material selection, we propose here, a visual representation of engineering catalogs using neural networks. In particular, we employ variational autoencoders (VAEs) to project catalog data onto a lower-dimensional latent space. The latent space can then be visualized to explore the underlying structure of the catalog. Specifically, creators can use this visual representation to identify gaps and outliers in their data, while end users can benefit from this representation to compare catalogs from competitors, and to find substitutes. Contours can be super-imposed on the charts to enable selection based on user-defined attributes; these contours are generalization of design indices associated with Ashby charts. Various examples of catalogs across engineering disciplines, ranging from materials and bearings to motors and batteries are illustrated using the proposed method. Using these examples, we (1) study the impact of the latent space dimension on the representational error, (2) illustrate how designers can easily choose alternate configurations based on their design requirements, and (3) gaps in catalog offerings can be clearly identified, providing a stimulus for new product development.