Penalized Variational Autoencoder for Molecular Design

<p><strong>Deep generative models have greatly advanced the field of artificial intelligence by learning the distribution of unlabelled datasets. In this thesis, we aim to develop a novel variational autoencoder (VAE)-based deep generative model that learns meaningful representations for effective unsupervised clustering in the latent space of high-dimensional datasets.</strong></p><p>Existing VAE-based deep clustering algorithms assume a mixture distribution either for the variational posterior over the latent variables or for the prior, to approximate the distribution of unlabelled datasets. The proposed mixtures variational autoencoders (MVAE) model assumes a mixture distribution for both the prior and variational posterior over the latent variables within the VAE framework.</p><p>We apply the variational inference to construct the evidence lower bound (ELBO) of the marginal log-likelihood function for our proposed MVAE model and then derive the analytical form for the clustering assignment estimate, also called the posterior probability estimate. This estimate is associated with each component of the VAE framework and enables the optimization of two distinct ELBO functions based on soft and hard assignment approaches. The training procedure for MVAE involves the joint optimization of both clustering assignment probabilities and model parameters. We evaluate the clustering performance on the latent embeddings of a dataset using both the posterior probability estimate and the Gaussian mixture model (GMM) methods.</p><p>We proposed two variants of the expectation-maximization (EM) algorithm to optimize the parameters for the MVAE model. The MVAE (EM) algorithm provides two distinct, numerically stable solutions for the posterior probability estimate: MVAE(EM-V1), where the estimate is associated with only the prior distribution, and MVAE(EM-V2), where the estimate incorporates both the variational posterior and the prior distribution. Among the proposed models, MVAE(EM-V2) requires less computation time for training. Additionally, this model achieves superior clustering performance on most datasets compared to the baseline models.</p><p>We modified the MVAE(EM-V2) algorithm by including a coefficient (β) with the regularizer term. The trained regularized mixtures VAE model with a small regularization coefficient (β < 1) achieves good unsupervised clustering performance on the test datasets. We further propose a variant of the regularized mixtures VAE model in which the regularization coefficient follows an annealing schedule from β > 1 to β < 1. The scheduled regularized model demonstrates superior clustering accuracy across most benchmark datasets compared to state-of-the-art deep clustering algorithms.</p><p>In this thesis, we point out that assuming a mixture distribution for both the prior and variational posterior components over the latent variables within the VAE framework enhances unsupervised clustering in the latent space. Our proposed models outperform state-of-the-art deep clustering algorithms, including VADE and k-DVAE, as well as the standard VAE model, in cluster analysis of latent representations across most benchmark datasets. Additionally, the proposed models demonstrate reasonable reconstruction performance and generate realistic examples from the latent space.</p>

Generative models rely on the idea that data can be represented in terms of latent variables which are uncorrelated by definition. Lack of correlation among the latent variable support is important because it suggests that the latent-space manifold is simpler to understand and manipulate than the real-space representation. Many types of generative model are used in deep learning, e.g., variational autoencoders (VAEs) and generative adversarial networks (GANs). Based on the idea that the latent space behaves like a vector space Radford et al. (2015), we ask whether we can expand the latent space representation of our data elements in terms of an orthonormal basis set. Here we propose a method to build a set of linearly independent vectors in the latent space of a trained GAN, which we call quasi-eigenvectors. These quasi-eigenvectors have two key properties: i) They span the latent space, ii) A set of these quasi-eigenvectors map to each of the labeled features one-to-one. We show that in the case of the MNIST image data set, while the number of dimensions in latent space is large by design, 98% of the data in real space map to a sub-domain of latent space of dimensionality equal to the number of labels. We then show how the quasi-eigenvectors can be used for Latent Spectral Decomposition (LSD). We apply LSD to denoise MNIST images. Finally, using the quasi-eigenvectors, we construct rotation matrices in latent space which map to feature transformations in real space. Overall, from quasi-eigenvectors we gain insight regarding the latent space topology.

Auto-Encoder (AE) based neural generative frameworks model the joint-distribution between the data and the latent space using an Encoder-Decoder pair, with regularization imposed in terms of a prior over the latent space. Despite their advantages, such as stability in training, efficient inference, the performance of AE based models has not reached the superior standards of the other generative models such as Generative Adversarial Networks (GANs). Motivated by this, we examine the effect of the latent prior on the generation quality of deterministic AE models in this paper. Specifically, we consider the class of Generative AE models with deterministic Encoder-Decoder pair (such as Wasserstein Auto-Encoder (WAE), Adversarial Auto-Encoder (AAE)), and show that having a fixed prior distribution, \textit{a priori}, oblivious to the dimensionality of the `true' latent space, will lead to the infeasibility of the optimization problem considered. As a remedy to the issue mentioned above, we introduce an additional state space in the form of flexibly learnable latent priors, in the optimization objective of WAE/AAE. Additionally, we employ a latent-space interpolation based smoothing scheme to address the non-smoothness that may arise from highly flexible priors. We show the efficacy of our proposed models, called FlexAE and FlexAE-SR, through several experiments on multiple datasets, and demonstrate that FlexAE-SR is the new state-of-the-art for the AE based generative models in terms of generation quality as measured by several metrics such as Fr\'echet Inception Distance, Precision/Recall score. Code for our paper is available at: \url{https://github.com/dair-iitd/FlexAE}

Variational autoencoders (VAEs) have been widely applied for text modeling. In practice, however, they are troubled by two challenges: information underrepresentation and posterior collapse. The former arises as only the last hidden state of LSTM encoder is transformed into the latent space, which is generally insufficient to summarize the data. The latter is a longstanding problem during the training of VAEs as the optimization is trapped to a disastrous local optimum. In this paper, we propose Discrete Auto-regressive Variational Attention Model (DAVAM) to address the challenges. Specifically, we introduce an auto-regressive variational attention approach to enrich the latent space by effectively capturing the semantic dependency from the input. We further design discrete latent space for the variational attention and mathematically show that our model is free from posterior collapse. Extensive experiments on language modeling tasks demonstrate the superiority of DAVAM against several VAE counterparts. Code will be released.

Generative deep learning is becoming a widely used approach in geological modelling, especially in problems that involve optimization processes under uncertainty like history matching. The basic idea of reservoir modelling and history matching by generative deep learning is to train some kind of reservoir generator and provide optimization under different constraints. There are different approaches that have been developed to utilize various modifications of Generative Adversarial Networks or Variational Autoencoders to implement reservoir generators, but most of them have "conventional" convolution neural networks, hence all data have to be regular (rectangular) in structure. This should be recognized as a limitation. In this work, we introduce a novel approach of reservoir modelling with Variational Autoencoders based on graph convolutions as opposed to "conventional" convolutions. In our approach reservoir model is considered as a graph, i.e., not a structured data type. Graph convolutions can deal with these data types, and connection with Variational Autoencoders provides us with the capability to generate initially unstructured reservoirs. Variational Autoencoders (VAE) demonstrate the ability to implicitly parameterize geological representations into a latent space of reduced dimensionality and provide ways to uncertainty quantification and production profiling among various geological concepts. In the first part, we introduce the motivation to Graph Variational Autoencoders (GVAE) as opposed to conventional Generative Deep learning models. The rest of the paper is dedicated to experiments on a synthetic dataset with two different geological scenarios. We show that trained GVAE performs the generation of reservoir models with reliable geology. A latent space structure between Encoder and Decoder of GVAE is represented, and interpretation considering geology under investigation is performed. Finally, we show how latent space can help to estimate the uncertainty of production and an optimization workflow under well data constraints.

Latent space Bayesian optimization (LSBO) combines generative models, typically variational autoencoders (VAE), with Bayesian optimization (BO), to generate de novo objects of interest. However, LSBO faces challenges due to the mismatch between the objectives of BO and VAE, resulting in poor exploration capabilities. In this article, we propose novel contributions to enhance LSBO efficiency and overcome this challenge. We first introduce the concept of latent consistency/inconsistency as a crucial problem in LSBO, arising from the VAE-BO mismatch. To address this, we propose the latent consistent aware-acquisition function (LCA-AF) that leverages consistent points in LSBO. Additionally, we present LCA-VAE, a novel VAE method that creates a latent space with increased consistent points through data augmentation in latent space and penalization of latent inconsistencies. Combining LCA-VAE and LCA-AF, we develop LCA-LSBO. Our approach achieves high sample efficiency and effective exploration, emphasizing the significance of addressing latent consistency through the novel incorporation of data augmentation in latent space within LCA-VAE in LSBO. We showcase the performance of our proposal via de novo image generation and de novo chemical design tasks.

The hardware design space is high-dimensional and discrete. Systematic and efficient exploration of this space has been a significant challenge. Central to this problem is the intractable search complexity that grows exponentially with the design choices and the discrete nature of the search space. This work investigates the feasibility of learning a meaningful low-dimensional continuous representation for hardware designs to reduce such complexity and facilitate the search process. We devise a variational autoencoder (VAE)-based design space exploration framework called VAESA, to encode the hardware design space in a compact and continuous representation. We show that black-box and gradient-based design space exploration algorithms can be applied to the latent space, and design points optimized in the latent space can be reconstructed to high-performance realistic hardware designs. Our experiments show that performing the design space search on the latent space consistently leads to the optimal design point under a fixed number of samples. In addition, the latent space can improve the sample efficiency of the original algorithm by 6.8$\times$ and can discover hardware designs that are up to 5% more efficient than the optimal design searched directly in the high-dimensional input space.

This paper evaluates the effectiveness of advanced Variational Autoencoder (VAE) models in overcoming latent space entanglement and insufficient disentanglement, common issues in traditional VAEs. Traditional VAEs often face challenges in separating distinct features within the latent space, leading to entangled representations that hinder interpretability and compression efficiency. The advanced VAE models examined in this study address these issues by enhancing disentanglement, which results in clearer separation of latent factors and more interpretable representations. However, this improvement in disentanglement may result in a trade-off with reconstruction quality. The article shows that, while these sophisticated models improve disentanglement, they may also have worse reconstruction quality than classic VAEs. The findings highlight the necessity of hyperparameter optimization in successfully navigating this trade-off. Future study should investigate novel model architectures and hyperparameter optimization strategies to optimize the balance of disentanglement and reconstruction quality. Overall, the research emphasizes the ability of advanced VAE models to generate more interpretable representations and the importance of careful tuning to resolve the inherent trade-offs.

To address the significant temporal characteristics and the scarcity of labelled data in complex industrial processes, specifically targeting the latent temporal dynamics that are difficult to observe directly during process evolution, this paper proposes a latent temporal feature dual‐head coordination network (LTFDHCN). First, a posterior generation (PG) module is constructed, utilizing a bi‐directional long short‐term memory (Bi‐LSTM) network to extract bidirectional temporal context and infer the posterior distribution in the latent space. Second, a prior prediction (PP) module is designed to establish a dynamic temporal model based on the latent variable of the previous time step. This mechanism predicts the prior distribution for the current moment, breaking the traditional assumption in variational autoencoders (VAE) that latent variables follow a static prior, thereby capturing deeper latent temporal dynamics. The Kullback–Leibler (KL) divergence is employed to minimize the discrepancy between the posterior and dynamic prior distributions, enforcing the latent space to encode temporal dependencies. Furthermore, a dual‐head coordination (DHC) strategy is proposed. Leveraging a shared latent space and decoder backbone, the model simultaneously executes unsupervised data reconstruction and supervised quality prediction. This strategy effectively overcomes the label scarcity issue by explicitly mining the latent information embedded in unlabelled data. The model is optimized via a weighted objective function that integrates KL divergence, reconstruction and prediction loss. Simulation experiments on a debutanizer column and a thermal power generation process demonstrate that the proposed method outperforms baseline methods, including VAE, VAE‐LSTM, VRNN, and temporal convolutional networks (TCN)‐VAE, verifying its effectiveness and superiority in handling complex industrial data.

Towards learning transferable embeddings for protein conformations using Variational Autoencoders

Generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) play an increasingly important role in medical image analysis. The latent spaces of these models often show semantically meaningful directions corresponding to human-interpretable image transformations. However, until now, their exploration for medical images has been limited due to the requirement of supervised data. Several methods for unsupervised discovery of interpretable directions in GAN latent spaces have shown interesting results on natural images. This work explores the potential of applying these techniques on medical images by training a GAN and a VAE on thoracic CT scans and using an unsupervised method to discover interpretable directions in the resulting latent space. We find several directions corresponding to non-trivial image transformations, such as rotation or breast size. Furthermore, the directions show that the generative models capture 3D structure despite being presented only with 2D data. The results show that unsupervised methods to discover interpretable directions in GANs generalize to VAEs and can be applied to medical images. This opens a wide array of future work using these methods in medical image analysis. The code and animations of the discovered directions are available online at https://github.com/julschoen/Latent-Space-Exploration-CT.KeywordsGenerative modelsUnsupervised learningInterpretabilityCT

Deep generative modeling for mechanistic-based learning and design of metamaterial systems

Deep clustering incorporates embedding into clustering in order to find a lower-dimensional space suitable for clustering tasks. Conventional deep clustering methods aim to obtain a single global embedding subspace (aka latent space) for all the data clusters. In contrast, in this article, we propose a deep multirepresentation learning (DML) framework for data clustering whereby each difficult-to-cluster data group is associated with its own distinct optimized latent space and all the easy-to-cluster data groups are associated with a general common latent space. Autoencoders (AEs) are employed for generating cluster-specific and general latent spaces. To specialize each AE in its associated data cluster(s), we propose a novel and effective loss function which consists of weighted reconstruction and clustering losses of the data points, where higher weights are assigned to the samples more probable to belong to the corresponding cluster(s). Experimental results on benchmark datasets demonstrate that the proposed DML framework and loss function outperform state-of-the-art clustering approaches. In addition, the results show that the DML method significantly outperforms the SOTA on imbalanced datasets as a result of assigning an individual latent space to the difficult clusters.

A variational autoencoder (VAE) is a generational deep learning model based on encoding a particular observation into a latent space and then decoding it while incorporating some random noise with the intuition of being able to generate slightly different forms of the input observation. Here, I propose a VAE that is able to generate denovo drug compounds by feeding in a known set of drug compounds against a particular biological target. To demonstrate the ability of the proposed VAE as a generative model, known drug molecules belonging to the class of Neuraminidase (NA) inhibitors were taken from the ZINC database and fed into the VAE model as one-hot encoded SMILES strings. Similarly, active NA inhibitors and decoy molecules together were also fed to compare efficiency. The generated molecules were then screened to remove impractical structures. Next, a drug-likeness (QED) score was computed for each candidate molecule and a cutoff of 0.5 was used to extract viable candidates. To ensure that the generated drug compounds were active NA inhibitors, a series of Artificial Neural Networks (ANNs) classifiers based on three different characterization techniques, namely chemical fingerprinting, molecular descriptions and graph convolutions, were developed to identify active NA inhibitors from decoy molecules. The feature candidate data were then fed into the three designed ANNs to obtain the final set of novel, viable and active NA inhibitors. Seventy-one new NA inhibitors were obtained after three runs of the VAE model under different parameterizations. The proposed VAE can hence be used to generate de-novo drug compounds for a wide variety of biological targets.

Summary 4D seismic history matching (4D SHM) uses 4D seismic data to calibrate reservoir models to reduce production forecast uncertainty and improve reservoir surveillance. 4D seismic becomes very valuable in field developments with sparse well configurations and high areal uncertainty, such as offshore and carbon sequestration projects. Unlike the production data, the conventional uncertainty and sensitivity analysis (SA) with 4D seismic data might return misleading results. Due to the smooth nature of 4D seismic data, it is highly likely that low-frequency signals dampen the impact of model parameters on high-frequency signals. Consequently, some significant parameters are wrongly excluded from the 4D SHM process. Our work aims to address this issue by localizing the SA of 4D seismic data. The idea is first to identify specific seismic signals on the seismic maps and then perform the SA only at the individual locations rather than the entire map. This way we overcome the dampening effects of low-frequency signals. Several approaches to localize the SA are utilized. In one approach, we defined sliding windows to scan the seismic maps and then executed an SA inside the windows at each location. Other localization approaches employ dimensionality reduction and feature extraction tools. We used principal component analysis (PCA) and advanced machine learning methods such as autoencoders (AEs) and variational autoencoders (VAEs) to transform the 4D seismic maps into a latent space. The information content (the 4D seismic signals) in the high-dimensionality 4D seismic maps is represented by a few features in the latent space. Implementing an SA for each feature in the latent space is equivalent to performing SA with the seismic signals in the original map. The localized SA scheme is coupled with the ensemble smoother with multiple data assimilation (ESMDA) algorithm to carry out 4D SHM. Three 4D SHM scenarios were defined as full parameterization with no SA, conventional SA using the entire map, and localized SA. We ran these scenarios for a complex synthetic reservoir model based on a real field in the North Sea to match 4D P-wave seismic impedance. The results confirmed the superiority of the localized SA scenario which returned the final ensemble with the lowest error and the best match among the three scenarios. It also turned out that the PCA, for this specific case, is the most suitable methodology to localize the SA.