Modality Conversion of Handwritten Patterns by Cross Variational Autoencoders

<p><strong>Deep generative models have greatly advanced the field of artificial intelligence by learning the distribution of unlabelled datasets. In this thesis, we aim to develop a novel variational autoencoder (VAE)-based deep generative model that learns meaningful representations for effective unsupervised clustering in the latent space of high-dimensional datasets.</strong></p><p>Existing VAE-based deep clustering algorithms assume a mixture distribution either for the variational posterior over the latent variables or for the prior, to approximate the distribution of unlabelled datasets. The proposed mixtures variational autoencoders (MVAE) model assumes a mixture distribution for both the prior and variational posterior over the latent variables within the VAE framework.</p><p>We apply the variational inference to construct the evidence lower bound (ELBO) of the marginal log-likelihood function for our proposed MVAE model and then derive the analytical form for the clustering assignment estimate, also called the posterior probability estimate. This estimate is associated with each component of the VAE framework and enables the optimization of two distinct ELBO functions based on soft and hard assignment approaches. The training procedure for MVAE involves the joint optimization of both clustering assignment probabilities and model parameters. We evaluate the clustering performance on the latent embeddings of a dataset using both the posterior probability estimate and the Gaussian mixture model (GMM) methods.</p><p>We proposed two variants of the expectation-maximization (EM) algorithm to optimize the parameters for the MVAE model. The MVAE (EM) algorithm provides two distinct, numerically stable solutions for the posterior probability estimate: MVAE(EM-V1), where the estimate is associated with only the prior distribution, and MVAE(EM-V2), where the estimate incorporates both the variational posterior and the prior distribution. Among the proposed models, MVAE(EM-V2) requires less computation time for training. Additionally, this model achieves superior clustering performance on most datasets compared to the baseline models.</p><p>We modified the MVAE(EM-V2) algorithm by including a coefficient (β) with the regularizer term. The trained regularized mixtures VAE model with a small regularization coefficient (β < 1) achieves good unsupervised clustering performance on the test datasets. We further propose a variant of the regularized mixtures VAE model in which the regularization coefficient follows an annealing schedule from β > 1 to β < 1. The scheduled regularized model demonstrates superior clustering accuracy across most benchmark datasets compared to state-of-the-art deep clustering algorithms.</p><p>In this thesis, we point out that assuming a mixture distribution for both the prior and variational posterior components over the latent variables within the VAE framework enhances unsupervised clustering in the latent space. Our proposed models outperform state-of-the-art deep clustering algorithms, including VADE and k-DVAE, as well as the standard VAE model, in cluster analysis of latent representations across most benchmark datasets. Additionally, the proposed models demonstrate reasonable reconstruction performance and generate realistic examples from the latent space.</p>

Generative models rely on the idea that data can be represented in terms of latent variables which are uncorrelated by definition. Lack of correlation among the latent variable support is important because it suggests that the latent-space manifold is simpler to understand and manipulate than the real-space representation. Many types of generative model are used in deep learning, e.g., variational autoencoders (VAEs) and generative adversarial networks (GANs). Based on the idea that the latent space behaves like a vector space Radford et al. (2015), we ask whether we can expand the latent space representation of our data elements in terms of an orthonormal basis set. Here we propose a method to build a set of linearly independent vectors in the latent space of a trained GAN, which we call quasi-eigenvectors. These quasi-eigenvectors have two key properties: i) They span the latent space, ii) A set of these quasi-eigenvectors map to each of the labeled features one-to-one. We show that in the case of the MNIST image data set, while the number of dimensions in latent space is large by design, 98% of the data in real space map to a sub-domain of latent space of dimensionality equal to the number of labels. We then show how the quasi-eigenvectors can be used for Latent Spectral Decomposition (LSD). We apply LSD to denoise MNIST images. Finally, using the quasi-eigenvectors, we construct rotation matrices in latent space which map to feature transformations in real space. Overall, from quasi-eigenvectors we gain insight regarding the latent space topology.

To address the significant temporal characteristics and the scarcity of labelled data in complex industrial processes, specifically targeting the latent temporal dynamics that are difficult to observe directly during process evolution, this paper proposes a latent temporal feature dual‐head coordination network (LTFDHCN). First, a posterior generation (PG) module is constructed, utilizing a bi‐directional long short‐term memory (Bi‐LSTM) network to extract bidirectional temporal context and infer the posterior distribution in the latent space. Second, a prior prediction (PP) module is designed to establish a dynamic temporal model based on the latent variable of the previous time step. This mechanism predicts the prior distribution for the current moment, breaking the traditional assumption in variational autoencoders (VAE) that latent variables follow a static prior, thereby capturing deeper latent temporal dynamics. The Kullback–Leibler (KL) divergence is employed to minimize the discrepancy between the posterior and dynamic prior distributions, enforcing the latent space to encode temporal dependencies. Furthermore, a dual‐head coordination (DHC) strategy is proposed. Leveraging a shared latent space and decoder backbone, the model simultaneously executes unsupervised data reconstruction and supervised quality prediction. This strategy effectively overcomes the label scarcity issue by explicitly mining the latent information embedded in unlabelled data. The model is optimized via a weighted objective function that integrates KL divergence, reconstruction and prediction loss. Simulation experiments on a debutanizer column and a thermal power generation process demonstrate that the proposed method outperforms baseline methods, including VAE, VAE‐LSTM, VRNN, and temporal convolutional networks (TCN)‐VAE, verifying its effectiveness and superiority in handling complex industrial data.

The recently developed variational autoencoders (VAEs) have proved to be an effective confluence of the rich representational power of neural networks with Bayesian methods. However, most work on VAEs use a rather simple prior over the latent variables such as standard normal distribution, thereby restricting its applications to relatively simple phenomena. In this work, we propose hierarchical non-parametric variational autoencoders, which combines tree-structured Bayesian nonparametric priors with VAEs, to enable infinite flexibility of the latent representation space. Both the neural parameters and Bayesian priors are learned jointly using tailored variational inference. The resulting model induces a hierarchical structure of latent semantic concepts underlying the data corpus, and infers accurate representations of data instances. We apply our model in video representation learning. Our method is able to discover highly interpretable activity hierarchies, and obtain improved clustering accuracy and generalization capacity based on the learned rich representations.

Effective health indicator (HI) construction can help equipment managers detect the abnormal state of rotating machinery quickly. However, although the current deep learning-based HI construction methods have good life prediction value, most of them lose the ability to detect device anomalies and little work has been done on model interpretability. Therefore, an interpretable HI construction method based on semi-supervised autoencoder (AE) latent space variance maximization (SSALSVM) was proposed to monitor the health status of bearings. In order to fully excavate degradation features inside the device and make the model focus on the encoding process, a deep convolutional neural network (DCNN) is used as the encoding layer, while only a layer of fully-connected layer is used as the decoding layer. In addition, to enable the latent space to capture the device early degradation point (EDP) successfully, an auxiliary layer is added to the output of the encoder layer. Simultaneously, for improving the sensitivity of the indicator to capture equipment abnormal state and highlight the difference between equipment health state and degradation state, the constraint of variance maximization is added into the latent space. The model optimizing process was presented by observing the projected variance of the test set in latent space of each epoch model. The validity of the proposed HI was verified by comparison experiments on two datasets.

Generative deep learning is becoming a widely used approach in geological modelling, especially in problems that involve optimization processes under uncertainty like history matching. The basic idea of reservoir modelling and history matching by generative deep learning is to train some kind of reservoir generator and provide optimization under different constraints. There are different approaches that have been developed to utilize various modifications of Generative Adversarial Networks or Variational Autoencoders to implement reservoir generators, but most of them have "conventional" convolution neural networks, hence all data have to be regular (rectangular) in structure. This should be recognized as a limitation. In this work, we introduce a novel approach of reservoir modelling with Variational Autoencoders based on graph convolutions as opposed to "conventional" convolutions. In our approach reservoir model is considered as a graph, i.e., not a structured data type. Graph convolutions can deal with these data types, and connection with Variational Autoencoders provides us with the capability to generate initially unstructured reservoirs. Variational Autoencoders (VAE) demonstrate the ability to implicitly parameterize geological representations into a latent space of reduced dimensionality and provide ways to uncertainty quantification and production profiling among various geological concepts. In the first part, we introduce the motivation to Graph Variational Autoencoders (GVAE) as opposed to conventional Generative Deep learning models. The rest of the paper is dedicated to experiments on a synthetic dataset with two different geological scenarios. We show that trained GVAE performs the generation of reservoir models with reliable geology. A latent space structure between Encoder and Decoder of GVAE is represented, and interpretation considering geology under investigation is performed. Finally, we show how latent space can help to estimate the uncertainty of production and an optimization workflow under well data constraints.

One of the main challenges that the industry faces when dealing with massive data for failure diagnosis is high dimensionality of such data. This can be tackled by dimensionality reduction method such as principal components analysis, which usually results in an improved fault diagnosis. Other available techniques include auto-encoders and its variants denoising auto-encoders and sparse auto-encoders. Most recently, variational auto-encoders are one of the most promising techniques for unsupervised learning with successful applications in image processing and speech recognition. Differently from other auto-encoder methods, variational auto-encoders use variational inference to generate a latent representation of the data and impose a distribution over the latent variables and the data itself. In this article, we propose a fully unsupervised deep variational auto-encoder-based approach for dimensionality reduction in fault diagnosis and explore the variational auto-encoder capabilities for such a task. This is achieved by comparing the latent representations provided by variational auto-encoders to the ones from principal components analysis as well as when no reduction is performed in ball bearings’ fault classification using vibration signals. To tackle massive sensor data, we propose different architectures for the variational auto-encoder’s encoder and decoder that are based on deep neural networks and deep convolutional neural networks. Experiments are also carried out by varying the data preprocessing methods for generating spectrograms and hand-engineering features as well as the use of raw vibration data, the architecture of the neural networks fault classifiers operating on the latent representations from variational auto-encoder and principal components analysis, the degree of data dimension reduction, and the size of the available labeled data used for training the fault classifiers. The results show that variational auto-encoders are a competent and promising tool for dimensionality reduction for use in fault diagnosis and worth further exploring their capabilities beyond vibration signals of ball bearing elements.

Inverse problems often involve matching observational data using a physical model that takes a large number of parameters as input. These problems tend to be under-constrained and require regularization to impose additional structure on the solution in parameter space. A central difficulty in regularization is turning a complex conceptual model of this additional structure into a functional mathematical form to be used in the inverse analysis. In this work we propose a method of regularization involving a machine learning technique known as a variational autoencoder (VAE). The VAE is trained to map a low-dimensional set of latent variables with a simple structure to the high-dimensional parameter space that has a complex structure. We train a VAE on unconditioned realizations of the parameters for a hydrological inverse problem. These unconditioned realizations neither rely on the observational data used to perform the inverse analysis nor require any forward runs of the physical model, thus making the computational cost of generating the training data minimal. The central benefit of this approach is that regularization is then performed on the latent variables from the VAE, which can be regularized simply. A second benefit of this approach is that the VAE reduces the number of variables in the optimization problem, thus making gradient-based optimization more computationally efficient when adjoint methods are unavailable. After performing regularization and optimization on the latent variables, the VAE then decodes the problem back to the original parameter space. Our approach constitutes a novel framework for regularization and optimization, readily applicable to a wide range of inverse problems. We call the approach RegAE.

Auto-Encoder (AE) based neural generative frameworks model the joint-distribution between the data and the latent space using an Encoder-Decoder pair, with regularization imposed in terms of a prior over the latent space. Despite their advantages, such as stability in training, efficient inference, the performance of AE based models has not reached the superior standards of the other generative models such as Generative Adversarial Networks (GANs). Motivated by this, we examine the effect of the latent prior on the generation quality of deterministic AE models in this paper. Specifically, we consider the class of Generative AE models with deterministic Encoder-Decoder pair (such as Wasserstein Auto-Encoder (WAE), Adversarial Auto-Encoder (AAE)), and show that having a fixed prior distribution, \textit{a priori}, oblivious to the dimensionality of the `true' latent space, will lead to the infeasibility of the optimization problem considered. As a remedy to the issue mentioned above, we introduce an additional state space in the form of flexibly learnable latent priors, in the optimization objective of WAE/AAE. Additionally, we employ a latent-space interpolation based smoothing scheme to address the non-smoothness that may arise from highly flexible priors. We show the efficacy of our proposed models, called FlexAE and FlexAE-SR, through several experiments on multiple datasets, and demonstrate that FlexAE-SR is the new state-of-the-art for the AE based generative models in terms of generation quality as measured by several metrics such as Fr\'echet Inception Distance, Precision/Recall score. Code for our paper is available at: \url{https://github.com/dair-iitd/FlexAE}

Purpose:There has been burgeoning interest in applying machine learning methods for predicting radiotherapy outcomes. However, the imbalanced ratio of a large number of variables to a limited sample size in radiation oncology constitutes a major challenge. Therefore, dimensionality reduction methods can be a key to success. The study investigates and contrasts the application of traditional machine learning methods and deep learning approaches for outcome modeling in radiotherapy. In particular, new joint architectures based on variational autoencoder (VAE) for dimensionality reduction are presented and their application is demonstrated for the prediction of lung radiation pneumonitis (RP) from a large-scale heterogeneous dataset.Methods:A large-scale heterogeneous dataset containing a pool of 230 variables including clinical factors (e.g., dose, KPS, stage) and biomarkers (e.g., single nucleotide polymorphisms (SNPs), cytokines and micro-RNAs) in a population of 106 non-small cell lung cancer (NSCLC) patients who received radiotherapy was used for modeling RP. 22 patients had grade 2 or higher RP. Four methods were investigated, including feature selection (case A) and feature extraction (case B) with traditional machine learning methods, a VAE-MLP joint architecture (case C) with deep learning and lastly, the combination of feature selection and joint-architecture (case D). For feature selection, Random forest (RF), Support Vector Machine (SVM) and multi-layer perceptron (MLP) were implemented to select relevant features. Specifically, each method was run for multiple times to rank features within several cross-validated (CV) resampled sets. A collection of ranking lists were then aggregated by top 5% and Kemeny graph methods to identify the final ranking for prediction. A synthetic minority oversampling technique (SMOTE) was applied to correct for class imbalance during this process. For deep learning, a VAE-MLP joint architecture where a VAE aimed for dimensionality reduction and an MLP aimed for classification was developed. In this architecture, reconstruction loss and prediction loss were combined into a single loss function to realize simultaneous training and weights were assigned to different classes to mitigate class imbalance. To evaluate the prediction performance and conduct comparisons, the area under receiver operating characteristic (ROC) curves (AUCs) were performed for nested CVs for both handcrafted feature selections and the deep learning approach. The significance of differences in AUCs was assessed using the DeLong test of U-statistics.Results:An MLP-based method using weight pruning (WP) feature selection yielded the best performance among the different hand-crafted feature selection methods (case A), reaching an AUC of 0.804 (95% CI: 0.761–0.823) with 29 top features. A VAE-MLP joint architecture (case C) achieved a comparable but slightly lower AUC of 0.781 (95% CI: 0.737–0.808) with the size of latent dimension being 2. The combination of handcrafted features (case A) and latent representation (case D) achieved a significant AUC improvement of 0.831 (95% CI: 0.805–0.863) with 22 features (p-value = 0.000642 compared with handcrafted features only (Case A) and p-value=0.000453 compared to VAE alone (Case C)) with an MLP classifier.Conclusion:The potential for combination of traditional machine learning methods and deep learning VAE techniques has been demonstrated for dealing with limited datasets in modeling radiotherapy toxicities. Specifically, latent variables from a VAE-MLP joint architecture are able to complement handcrafted features for the prediction of RP and improve prediction over either method alone.

Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex

Latent space Bayesian optimization (LSBO) combines generative models, typically variational autoencoders (VAE), with Bayesian optimization (BO), to generate de novo objects of interest. However, LSBO faces challenges due to the mismatch between the objectives of BO and VAE, resulting in poor exploration capabilities. In this article, we propose novel contributions to enhance LSBO efficiency and overcome this challenge. We first introduce the concept of latent consistency/inconsistency as a crucial problem in LSBO, arising from the VAE-BO mismatch. To address this, we propose the latent consistent aware-acquisition function (LCA-AF) that leverages consistent points in LSBO. Additionally, we present LCA-VAE, a novel VAE method that creates a latent space with increased consistent points through data augmentation in latent space and penalization of latent inconsistencies. Combining LCA-VAE and LCA-AF, we develop LCA-LSBO. Our approach achieves high sample efficiency and effective exploration, emphasizing the significance of addressing latent consistency through the novel incorporation of data augmentation in latent space within LCA-VAE in LSBO. We showcase the performance of our proposal via de novo image generation and de novo chemical design tasks.

BackgroundThe early diagnosis and monitoring of Alzheimer's disease (AD) presents a significant challenge due to its heterogeneous nature, which includes variability in cognitive symptoms, diagnostic test results, and progression rates. This study aims to enhance the understanding of AD progression by integrating neuroimaging metrics with demographic data using a novel machine learning technique.MethodWe used supervised Variational Autoencoders (VAEs), a generative AI method, to analyze high‐dimensional neuroimaging data for AD progression, incorporating age and gender as covariates. We used two non‐overlapping datasets: 257 samples from ADNI3 and 676 from ADNI2, analyzing 68 cortical and 48 subcortical grey matter volumes extracted from their MRI 3D T1 images. The VAE model aimed to minimize reconstruction error (MSE), Kullback‐Leibler divergence, and classification error, and to estimate latent variables for each subject. A Disease Progression Score (DPS) was calculated by encoding the extracted imaging features into a latent space and projecting test data onto a trajectory curve. The study used stratified sampling for robustness and assessed the model's performance using the area under the ROC curve (AUC), and correlated the mean DPS with cognitive assessment scores by applying Kendall’s Tau.ResultThe VAE model demonstrated excellent discriminative power in classifying AD progression stages, with ROC AUC values near 1, particularly when using all 116 features. Cortical volumes were more predictive than subcortical volumes. The mean DPS showed a statistically significant correlation with cognitive assessments (p<0.01), with Kendall's Tau values of 0.66 for CDR‐SB, ‐0.45 for MMSE, and ‐0.49 for MoCA, indicating its validity as a quantitative biomarker for cognitive decline in AD.ConclusionSupervised VAEs can effectively model the progression of Alzheimer’s disease brain atrophy on MRI and could potentially serve as imaging biomarkers in clinical trials. The strong correlation between the DPS and cognitive assessments highlights the potential of supervised VAEs to provide a quantifiable measure of AD severity, which would be useful for clinical assessments and for objectively monitoring disease progression. The study's methodology and findings contribute to computational neuroscience and offer a foundation for future research in early detection and personalized treatment strategies for Alzheimer's disease.

This paper evaluates the effectiveness of advanced Variational Autoencoder (VAE) models in overcoming latent space entanglement and insufficient disentanglement, common issues in traditional VAEs. Traditional VAEs often face challenges in separating distinct features within the latent space, leading to entangled representations that hinder interpretability and compression efficiency. The advanced VAE models examined in this study address these issues by enhancing disentanglement, which results in clearer separation of latent factors and more interpretable representations. However, this improvement in disentanglement may result in a trade-off with reconstruction quality. The article shows that, while these sophisticated models improve disentanglement, they may also have worse reconstruction quality than classic VAEs. The findings highlight the necessity of hyperparameter optimization in successfully navigating this trade-off. Future study should investigate novel model architectures and hyperparameter optimization strategies to optimize the balance of disentanglement and reconstruction quality. Overall, the research emphasizes the ability of advanced VAE models to generate more interpretable representations and the importance of careful tuning to resolve the inherent trade-offs.

Variational autoencoders have emerged as one of the most common approaches for automating molecular generation. We seek to learn a cross-domain latent space capturing chemical and biological information, simultaneously. To do so, we introduce the Penalized Variational Autoencoder which directly operates on SMILES, a linear string representation of molecules, with a weight penalty term in the decoder to address the imbalance in the character distribution of SMILES strings. We find that this greatly improves upon previous variational autoencoder approaches in the quality of the latent space and the generalization ability of the latent space to new chemistry. Next, we organize the latent space according to chemical and biological properties by jointly training the Penalized Variational Autoencoder with linear units. Extensive experiments on a range of tasks, including reconstruction, validity, and transferability demonstrates that the proposed methods here substantially outperform previous SMILES and graph-based methods, as well as introduces a new way to generate molecules from a set of desired properties, without prior knowledge of a chemical structure.