De Novo Drug Design via Multi-Label Learning and Adversarial Autoencoder

Automated design synthesis has the potential to revolutionize the modern engineering design process and improve access to highly optimized and customized products across countless industries. Successfully adapting generative machine learning to design engineering may enable such automated design synthesis and is a research subject of great importance. We present a review and analysis of deep generative machine learning models in engineering design. Deep generative models (DGMs) typically leverage deep networks to learn from an input dataset and synthesize new designs. Recently, DGMs such as feedforward neural networks (NNs), generative adversarial networks (GANs), variational autoencoders (VAEs), and certain deep reinforcement learning (DRL) frameworks have shown promising results in design applications like structural optimization, materials design, and shape synthesis. The prevalence of DGMs in engineering design has skyrocketed since 2016. Anticipating the continued growth, we conduct a review of recent advances to benefit researchers interested in DGMs for design. We structure our review as an exposition of the algorithms, datasets, representation methods, and applications commonly used in the current literature. In particular, we discuss key works that have introduced new techniques and methods in DGMs, successfully applied DGMs to a design-related domain, or directly supported the development of DGMs through datasets or auxiliary methods. We further identify key challenges and limitations currently seen in DGMs across design fields, such as design creativity, handling constraints and objectives, and modeling both form and functional performance simultaneously. In our discussion, we identify possible solution pathways as key areas on which to target the future work.

The de novo design of molecular structures using deep learning generative models introduces an encouraging solution to drug discovery in the face of the continuously increased cost of new drug development. From the generation of original texts, images, and videos, to the scratching of novel molecular structures the creativity of deep learning generative models exhibits the height machine intelligence can achieve. The purpose of this paper is to review the latest advances in generative chemistry which relies on generative modeling to expedite the drug discovery process. This review starts with a brief history of artificial intelligence in drug discovery to outline this emerging paradigm. Commonly used chemical databases, molecular representations, and tools in cheminformatics and machine learning are covered as the infrastructure for generative chemistry. The detailed discussions on utilizing cutting-edge generative architectures, including recurrent neural network, variational autoencoder, adversarial autoencoder, and generative adversarial network for compound generation are focused. Challenges and future perspectives follow.

<p><strong>Deep generative models have greatly advanced the field of artificial intelligence by learning the distribution of unlabelled datasets. In this thesis, we aim to develop a novel variational autoencoder (VAE)-based deep generative model that learns meaningful representations for effective unsupervised clustering in the latent space of high-dimensional datasets.</strong></p><p>Existing VAE-based deep clustering algorithms assume a mixture distribution either for the variational posterior over the latent variables or for the prior, to approximate the distribution of unlabelled datasets. The proposed mixtures variational autoencoders (MVAE) model assumes a mixture distribution for both the prior and variational posterior over the latent variables within the VAE framework.</p><p>We apply the variational inference to construct the evidence lower bound (ELBO) of the marginal log-likelihood function for our proposed MVAE model and then derive the analytical form for the clustering assignment estimate, also called the posterior probability estimate. This estimate is associated with each component of the VAE framework and enables the optimization of two distinct ELBO functions based on soft and hard assignment approaches. The training procedure for MVAE involves the joint optimization of both clustering assignment probabilities and model parameters. We evaluate the clustering performance on the latent embeddings of a dataset using both the posterior probability estimate and the Gaussian mixture model (GMM) methods.</p><p>We proposed two variants of the expectation-maximization (EM) algorithm to optimize the parameters for the MVAE model. The MVAE (EM) algorithm provides two distinct, numerically stable solutions for the posterior probability estimate: MVAE(EM-V1), where the estimate is associated with only the prior distribution, and MVAE(EM-V2), where the estimate incorporates both the variational posterior and the prior distribution. Among the proposed models, MVAE(EM-V2) requires less computation time for training. Additionally, this model achieves superior clustering performance on most datasets compared to the baseline models.</p><p>We modified the MVAE(EM-V2) algorithm by including a coefficient (β) with the regularizer term. The trained regularized mixtures VAE model with a small regularization coefficient (β < 1) achieves good unsupervised clustering performance on the test datasets. We further propose a variant of the regularized mixtures VAE model in which the regularization coefficient follows an annealing schedule from β > 1 to β < 1. The scheduled regularized model demonstrates superior clustering accuracy across most benchmark datasets compared to state-of-the-art deep clustering algorithms.</p><p>In this thesis, we point out that assuming a mixture distribution for both the prior and variational posterior components over the latent variables within the VAE framework enhances unsupervised clustering in the latent space. Our proposed models outperform state-of-the-art deep clustering algorithms, including VADE and k-DVAE, as well as the standard VAE model, in cluster analysis of latent representations across most benchmark datasets. Additionally, the proposed models demonstrate reasonable reconstruction performance and generate realistic examples from the latent space.</p>

The design of a new therapeutic agent is a time-consuming and expensive process. The rise of machine intelligence provides a grand opportunity of expeditiously discovering novel drug candidates through smart search in the vast molecular structural space. In this paper, we propose a new approach called adversarial deep evolutionary learning (ADEL) to search for novel molecules in the latent space of an adversarial generative model and keep improving the latent representation space. In ADEL, a custom-made adversarial autoencoder (AAE) model is developed and trained under a deep evolutionary learning (DEL) process. This involves an initial training of the AAE model, followed by an integration of multi-objective evolutionary optimization in the continuous latent representation space of the AAE rather than the discrete structural space of molecules. By using the AAE, an arbitrary distribution can be provided to the training of AAE such that the latent representation space is set to that distribution. This allows for a starting latent space from which new samples can be produced. Throughout the process of learning, new samples of high-quality are generated after each iteration of training and then added back into the full dataset. Therefore, allowing for a more comprehensive procedure of understanding the data structure. This combination of evolving data and continuous learning not only enables improvement in the generative model, but the data as well. By comparing ADEL to the previous work in DEL, we see that ADEL can obtain better property distributions.

Auto-Encoder (AE) based neural generative frameworks model the joint-distribution between the data and the latent space using an Encoder-Decoder pair, with regularization imposed in terms of a prior over the latent space. Despite their advantages, such as stability in training, efficient inference, the performance of AE based models has not reached the superior standards of the other generative models such as Generative Adversarial Networks (GANs). Motivated by this, we examine the effect of the latent prior on the generation quality of deterministic AE models in this paper. Specifically, we consider the class of Generative AE models with deterministic Encoder-Decoder pair (such as Wasserstein Auto-Encoder (WAE), Adversarial Auto-Encoder (AAE)), and show that having a fixed prior distribution, \textit{a priori}, oblivious to the dimensionality of the `true' latent space, will lead to the infeasibility of the optimization problem considered. As a remedy to the issue mentioned above, we introduce an additional state space in the form of flexibly learnable latent priors, in the optimization objective of WAE/AAE. Additionally, we employ a latent-space interpolation based smoothing scheme to address the non-smoothness that may arise from highly flexible priors. We show the efficacy of our proposed models, called FlexAE and FlexAE-SR, through several experiments on multiple datasets, and demonstrate that FlexAE-SR is the new state-of-the-art for the AE based generative models in terms of generation quality as measured by several metrics such as Fr\'echet Inception Distance, Precision/Recall score. Code for our paper is available at: \url{https://github.com/dair-iitd/FlexAE}

Electrical resistivity tomography (ERT) is a geophysical method used to imaging the subsurface and widely used in hydrogeological studies due to its sensitivity to electrical resistivity, which is directly related to rock type, porosity, ionic strength of the pore fluids, and surface conductivity of geologic materials. The prediction of subsurface properties from the recorded data at the surface requires solving a challenging geophysical inversion problem. For near-surface characterization studies, this is often accomplished with deterministic electrical resistivity inversion methods. Deterministic geophysical inversion approaches linearize the problem around an initial solution, resulting in a single smooth representation of the subsurface. Deterministic models are unable to capture the natural variability of the subsurface. Moreover, a single solution does not yield enough information for accurate uncertainty assessment. Contrary to deterministic approaches, stochastic inversion methods predict multiple model realizations that fit similarly the recorded geophysical data and allow assessing uncertainties. Lately, deep learning algorithms based on deep generative models have been used to re-parametrize model and data spaces into low-dimensional domains and solve geophysical inverse problems in a more efficient way.We propose an ERT inversion methodology in which a deep convolutional variational autoencoder (VAE) network is trained with a set of electrical resistivity models generated using geostatistical simulation. After training the VAE, the latent space is perturbed and updated iteratively with adaptive stochastic sampling to generate electrical resistivity models by inputting the optimized latent vectors to the decoder part of the VAE. From the set of decoded models, we use a finite volume approximation of Poisson’s equation to compute synthetic apparent resistivity models. The misfit between predicted and observed apparent resistivity data is used to drive the convergence of the iterative procedure and condition the optimization of new models in the subsequent iterations.The proposed methodology is illustrated by applying it to both a two-dimensional synthetic case and to a two-dimensional profile obtained from an ERT survey carried out in an area located in the Southern region of Portugal. In both application examples, the predicted models generate synthetic geophysical data that match the observed one. We show the ability of the model to assess spatial uncertainty and compare the results obtained in the real data set against commercial deterministic ERT inversion methodology.The work presented herein is supported by the PRIMA programme under grant agreement No. 1923, project Innovative and Sustainable Groundwater Management in the Mediterranean (InTheMED). The PRIMA programme is supported by the European Union.

Deep generative models have been successfully applied to many applications. However, existing methods experience limitations when generating large images (the literature usually generates small images, e.g., \(32 \times 32\) or \(128 \times 128\)). In this paper, we propose a novel scheme using tensor super-resolution with adversarial generative nets (TSRGAN), to generate large high-quality images by exploring tensor structures. Essentially, the super resolution process of TSRGAN is based on tensor representation. First, we impose tensor structures for concise image representation, which is superior in capturing the pixel proximity information and the spatial patterns of elementary objects in images, over the vectorization preprocess in existing works. Secondly, we propose TSRGAN that integrates deep convolutional generative adversarial networks and tensor super-resolution in a cascading manner, to generate high-quality images from random distributions. More specifically, we design a tensor super-resolution process that consists of tensor dictionary learning and tensor coefficients learning. Finally, on three datasets, the proposed TSRGAN generates images with more realistic textures, compared with state-of-the-art adversarial autoencoders and super-resolution methods. The size of the generated images is increased by over 8.5 times, namely \(374\times 374\) in PASCAL2.

Deep generative modeling for mechanistic-based learning and design of metamaterial systems

We explore methods of producing adversarial examples on deep generative models such as the variational autoencoder (VAE) and the VAE-GAN. Deep learning architectures are known to be vulnerable to adversarial examples, but previous work has focused on the application of adversarial examples to classification tasks. Deep generative models have recently become popular due to their ability to model input data distributions and generate realistic examples from those distributions. We present three classes of attacks on the VAE and VAE-GAN architectures and demonstrate them against networks trained on MNIST, SVHN and CelebA. Our first attack leverages classification-based adversaries by attaching a classifier to the trained encoder of the target generative model, which can then be used to indirectly manipulate the latent representation. Our second attack directly uses the VAE loss function to generate a target reconstruction image from the adversarial example. Our third attack moves beyond relying on classification or the standard loss for the gradient and directly optimizes against differences in source and target latent representations. We also motivate why an attacker might be interested in deploying such techniques against a target generative network.

Deep generative models have gained much attention given their ability to generate data for applications as varied as healthcare to financial technology to surveillance, and many more - the most popular models being generative adversarial networks (GANs) and variational auto-encoders (VAEs). Yet, as with all machine learning models, ever is the concern over security breaches and privacy leaks and deep generative models are no exception. In fact, these models have advanced so rapidly in recent years that work on their security is still in its infancy. In an attempt to audit the current and future threats against these models, and to provide a roadmap for defense preparations in the short term, we prepared this comprehensive and specialized survey on the security and privacy preservation of GANs and VAEs. Our focus is on the inner connection between attacks and model architectures and, more specifically, on five components of deep generative models: the training data, the latent code, the generators/decoders of GANs/VAEs, the discriminators/encoders of GANs/VAEs, and the generated data. For each model, component and attack, we review the current research progress and identify the key challenges. The paper concludes with a discussion of possible future attacks and research directions in the field.

Creative rhythmic transformations of musical audio refer to automated methods for manipulation of temporally-relevant sounds in time. This paper presents a method for joint synthesis and rhythm transformation of drum sounds through the use of adversarial autoencoders (AAE). Users may navigate both the timbre and rhythm of drum patterns in audio recordings through expressive control over a low-dimensional latent space. The model is based on an AAE with Gaussian mixture latent distributions that introduce rhythmic pattern conditioning to represent a wide variety of drum performances. The AAE is trained on a dataset of bar-length segments of percussion recordings, along with their clustered rhythmic pattern labels. The decoder is conditioned during adversarial training for mixing of data-driven rhythmic and timbral properties. The system is trained with over 500000 bars from 5418 tracks in popular datasets covering various musical genres. In an evaluation using real percussion recordings, the reconstruction accuracy and latent space interpolation between drum performances are investigated for audio generation conditioned by target rhythmic patterns.

Localization in reverberant environments remains an open challenge. Recently, supervised learning approaches have demonstrated very promising results in addressing reverberation. However, even with large data volumes, the number of labels available for supervised learning in such environments is usually small. We propose to address this issue with a semi-supervised learning (SSL) approach, based on deep generative modeling. Our chosen deep generative model, the variational autoencoder (VAE), is trained to generate the phase of relative transfer functions (RTFs) between microphones. In parallel, a direction of arrival (DOA) classifier network based on RTF-phase is also trained. The joint generative and discriminative model, deemed VAE-SSL, is trained using labeled and unlabeled RTF-phase sequences. In learning to generate and classify the sequences, the VAE-SSL extracts the physical causes of the RTF-phase (i.e., source location) from distracting signal characteristics such as noise and speech activity. This facilitates effective end-to-end operation of the VAE-SSL, which requires minimal preprocessing of RTF-phase. VAE-SSL is compared with two signal processing-based approaches, steered response power with phase transform (SRP-PHAT) and MUltiple SIgnal Classification (MUSIC), as well as fully supervised CNNs. The approaches are compared using data from two real acoustic environments - one of which was recently obtained at Technical University of Denmark specifically for our study. We find that VAE-SSL can outperform the conventional approaches and the CNN in label-limited scenarios. Further, the trained VAE-SSL system can generate new RTF-phase samples which capture the physics of the acoustic environment. Thus, the generative modeling in VAE-SSL provides a means of interpreting the learned representations. To the best of our knowledge, this paper presents the first approach to modeling the physics of acoustic propagation using deep generative modeling.

Deep convolutional neural networks (CNNs) have proven highly effective for visual recognition, where learning a universal representation from activations of convolutional layer plays a fundamental problem. In this paper, we present Fisher Vector encoding with Variational Auto-Encoder (FV-VAE), a novel deep architecture that quantizes the local activations of convolutional layer in a deep generative model, by training them in an end-to-end manner. To incorporate FV encoding strategy into deep generative models, we introduce Variational Auto-Encoder model, which steers a variational inference and learning in a neural network which can be straightforwardly optimized using standard stochastic gradient method. Different from the FV characterized by conventional generative models (e.g., Gaussian Mixture Model) which parsimoniously fit a discrete mixture model to data distribution, the proposed FV-VAE is more flexible to represent the natural property of data for better generalization. Extensive experiments are conducted on three public datasets, i.e., UCF101, ActivityNet, and CUB-200-2011 in the context of video action recognition and fine-grained image classification, respectively. Superior results are reported when compared to state-of-the-art representations. Most remarkably, our proposed FV-VAE achieves to-date the best published accuracy of 94.2% on UCF101.

We applied convolutional versions of a "standard" autoencoder (CAE), a variational autoencoder (VAE) and an adversarial autoencoder (AAE) to two different publicly available datasets and compared their anomaly detection performances. We used the MNIST dataset [14] as a simple anomaly detection scenario. The CIFAR10 dataset [13] was used to examine the autoencoders in a more complex anomaly detection task. The anomaly detection performance of our different autoencoder types is compared in a qualitative and quantitative manner. The time needed for training the models is measured to capture their complexity. The simplest model demanding the simplest training, the CAE, computes results which are nearly as accurate and for some cases even better than results achieved by the VAE and AAE. We show that all three autoencoder types computed convincing anomaly detection results for the more simple-structured MNIST scenario. However, none of the autoencoder types proved to capture a good representation of the relevant features of the more complex CIFAR10 dataset, leading to moderately good anomaly detection performances.

Objectives/Scope Variational autoencoders as generative models have been widely adopted through large language models in the natural language processing domain in recent years. However, the relative novelty of this architecture has resulted in limited investigation within the production engineering field. This study proposes a novel approach towards pump working condition monitoring for sucker rod pumping systems based on derived downhole dynamometer cards utilizing Conditional Variational Autoencoders (CVAE). Methods, Procedures, Process A case study comprising 17 wells in Upper Austrian oil fields of RAG Austria AG has been conducted. Using a finite element method, downhole dynamometer cards reflecting the operating condition of the pump are derived from polished rod load cell measurements at the surface. Consequently, a CVAE network is trained in two distinct configurations: utilizing labeled data encompassing pre-defined pump working conditions and employing a dataset solely comprising a healthy pump state exclusively for anomaly detection. Following the application of dimensionality reduction techniques of the generated hidden vectors, the transformed representations are employed to evaluate model performance. Results, Observations, Conclusions This study introduces a novel SRP-CVAE framework for continuous pump condition monitoring, intertwining finite element method based mechanical modeling of the sucker-rod string for downhole card derivation with a deep probabilistic generative model for pump working condition classification. It has been found that within a benchmark setting, pump condition classification accuracies can be increased by adopting a hybrid supervised CVAE architecture for automatic feature extraction incorporating both a generative and discriminative objective. The respective superior performance within this setting can be partly attributed to its inherent multi-objective optimization framework, wherein the model simultaneously optimizes reconstruction fidelity and classification accuracy through shared latent representations. In the second classification setting with the objective of solely distinguishing between a healthy or faulty pump working condition, the model is trained on datasets exclusively reflecting the respective states. The results show that the proposed SRP-CVAE model demonstrates significant robustness towards accurately detecting abnormal pump working states and generalizes well for conditions characterized by data scarcity. This may be ascribed to the inherent regularization through the Kullback-Leibler divergence regularization term effectively penalizing overly complex latent representations that deviate substantially from the imposed prior. Finally, the proposed architecture allows for synthetic data generation or augmentation by sampling from the latent space, which has proven useful for sampling rare pump working conditions. Novel/Additive Information This paper will serve as a resource to any engineer or technician in the production engineering domain who is building models for anomaly detection using sensoric data. The proposed architecture can be readily adapted to diverse anomaly detection applications or used to enhance classification accuracy in related domains, thereby advancing health, safety and environment standards as well as facilitating risk mitigation in field operations.