Latent Embedded Graphs for Image and Shape Interpolation

Autoencoders and generative neural network models have recently gained popularity in fluid mechanics due to their spontaneity and low processing time instead of high fidelity CFD simulations. Auto encoders are used as model order reduction tools in applications of fluid mechanics by compressing input high-dimensional data using an encoder to map the input space into a lower-dimensional latent space. Whereas, generative models such as Variational Auto-encoders (VAEs) and Generative Adversarial Networks (GANs) are proving to be effective in generating solutions to chaotic models with high ‘randomness’ such as turbulent flows. In this study, forced isotropic turbulence flow is generated by parameterizing into some basic statistical characteristics. The models trained on pre-simulated data from dependencies on these characteristics and the flow generation is then affected by varying these parameters. The latent vectors pushed along the generator models like the decoders and generators contain independent entries which can be used to create different outputs with similar properties. The use of neural network-based architecture removes the need for dependency on the classical mesh-based Navier-Stoke equation estimation which is prominent in many CFD softwares.

Image interpolation is used widely in the computer vision. Holding edge information is main problem in the image interpolation. By using bilinear and bicubic B-spline interpolation methods, a novel image interpolation approach was proposed in this paper. Firstly, inverse distance weighted average method was used to reduce image’s noise. Secondly, edge detection operator was used to extract image's edges information. It can help us to select different interpolation methods in the image interpolation process. Finally, we selected bilinear interpolation approach at non-edge regions, and bicubic B-spline interpolation method was used near edges regions. Further more, control vertexes were computed from pixels with calculation formula which has been simplified in the B-spline interpolation process. Experiments showed the interpolated image by the proposed method had good vision results for it could hold image's edge information effectively.

Generative models rely on the idea that data can be represented in terms of latent variables which are uncorrelated by definition. Lack of correlation among the latent variable support is important because it suggests that the latent-space manifold is simpler to understand and manipulate than the real-space representation. Many types of generative model are used in deep learning, e.g., variational autoencoders (VAEs) and generative adversarial networks (GANs). Based on the idea that the latent space behaves like a vector space Radford et al. (2015), we ask whether we can expand the latent space representation of our data elements in terms of an orthonormal basis set. Here we propose a method to build a set of linearly independent vectors in the latent space of a trained GAN, which we call quasi-eigenvectors. These quasi-eigenvectors have two key properties: i) They span the latent space, ii) A set of these quasi-eigenvectors map to each of the labeled features one-to-one. We show that in the case of the MNIST image data set, while the number of dimensions in latent space is large by design, 98% of the data in real space map to a sub-domain of latent space of dimensionality equal to the number of labels. We then show how the quasi-eigenvectors can be used for Latent Spectral Decomposition (LSD). We apply LSD to denoise MNIST images. Finally, using the quasi-eigenvectors, we construct rotation matrices in latent space which map to feature transformations in real space. Overall, from quasi-eigenvectors we gain insight regarding the latent space topology.

Deep generative models are promising in detecting novel cyber-physical attacks, mitigating the vulnerability of Cyber-physical systems (CPSs) without relying on labeled information. Nonetheless, these generative models face challenges in identifying attack behaviors that closely resemble normal data, or deviate from the normal data distribution but are in close proximity to the manifold of the normal cluster in latent space. To tackle this problem, this article proposes a novel unsupervised dual variational generative adversarial model named MST-DVGAN, to perform anomaly detection in multivariate time series data for CPS security. The central concept is to enhance the model's discriminative capability by widening the distinction between reconstructed abnormal samples and their normal counterparts. Specifically, we propose an augmented module by imposing contrastive constraints on the reconstruction process to obtain a more compact embedding. Then, by exploiting the distribution property and modeling the normal patterns of multivariate time series, a variational autoencoder is introduced to force the generative adversarial network (GAN) to generate diverse samples. Furthermore, two augmented loss functions are designed to extract essential characteristics in a self-supervised manner through mutual guidance between the augmented samples and original samples. Finally, a specific feature center loss is introduced for the generator network to enhance its stability. Empirical experiments are conducted on three public datasets, namely SWAT, WADI and NSL_KDD. Comparing with the state-of-the-art methods, the evaluation results show that the proposed MTS-DVGAN is more stable and can achieve consistent performance improvement.

Auto-Encoder (AE) based neural generative frameworks model the joint-distribution between the data and the latent space using an Encoder-Decoder pair, with regularization imposed in terms of a prior over the latent space. Despite their advantages, such as stability in training, efficient inference, the performance of AE based models has not reached the superior standards of the other generative models such as Generative Adversarial Networks (GANs). Motivated by this, we examine the effect of the latent prior on the generation quality of deterministic AE models in this paper. Specifically, we consider the class of Generative AE models with deterministic Encoder-Decoder pair (such as Wasserstein Auto-Encoder (WAE), Adversarial Auto-Encoder (AAE)), and show that having a fixed prior distribution, \textit{a priori}, oblivious to the dimensionality of the `true' latent space, will lead to the infeasibility of the optimization problem considered. As a remedy to the issue mentioned above, we introduce an additional state space in the form of flexibly learnable latent priors, in the optimization objective of WAE/AAE. Additionally, we employ a latent-space interpolation based smoothing scheme to address the non-smoothness that may arise from highly flexible priors. We show the efficacy of our proposed models, called FlexAE and FlexAE-SR, through several experiments on multiple datasets, and demonstrate that FlexAE-SR is the new state-of-the-art for the AE based generative models in terms of generation quality as measured by several metrics such as Fr\'echet Inception Distance, Precision/Recall score. Code for our paper is available at: \url{https://github.com/dair-iitd/FlexAE}

Edge-based image interpolation often leads to an image with good quality because of the importance of sharp edges and smooth contours to the human vision. The wavelet-based image interpolation approach has good potential in producing interpolated images with high quality edges. Using the wavelet multiresolution analysis theory, such method is computationally demanding. In this paper, a new edge-based image interpolation method that uses symmetric biorthogonal wavelet transform is proposed. We form a list of ideal step edges and study the relationships between the wavelet approximation sub-image of each edge and its wavelet detail sub-images. Based on these relationships, a simple and efficient algorithm that predicts the edge information of high resolution images is proposed. For this method, the 9/7-M inverse wavelet transform is shown, experimentally, to yield better image interpolation performance compared to traditional image interpolation approaches

Generation of photo-realistic images, semantic editing and representation learning are only a few of many applications of high-resolution generative models. Recent progress in GANs have established them as an excellent choice for such tasks. However, since they do not provide an inference model, downstream tasks such as classification cannot be easily applied on real images using the GAN latent space. Despite numerous efforts to train an inference model or design an iterative method to invert a pre-trained generator, previous methods are dataset (e.g. human face images) and architecture (e.g. StyleGAN) specific. These methods are nontrivial to extend to novel datasets or architectures. We propose a general framework that is agnostic to architecture and datasets. Our key insight is that, by training the inference and the generative model together, we allow them to adapt to each other and to converge to a better quality model. Our InvGAN, short for Invertible GAN, successfully embeds real images in the latent space of a high quality generative model. This allows us to perform image inpainting, merging, interpolation and online data augmentation. We demonstrate this with extensive qualitative and quantitative experiments.

Traditional computational approaches to design chemical species are limited by the need to compute properties for a vast number of candidates, e.g., by discriminative modeling. Therefore, inverse design methods aim to start from the desired property and optimize a corresponding chemical structure. From a machine learning viewpoint, the inverse design problem can be addressed through so-called generative modeling. Mathematically, discriminative models are defined by learning the probability distribution function of properties given the molecular or material structure. In contrast, a generative model seeks to exploit the joint probability of a chemical species with target characteristics. The overarching idea of generative modeling is to implement a system that produces novel compounds that are expected to have a desired set of chemical features, effectively sidestepping issues found in the forward design process. In this contribution, we overview and critically analyze popular generative algorithms like generative adversarial networks, variational autoencoders, flow, and diffusion models. We highlight key differences between each of the models, provide insights into recent success stories, and discuss outstanding challenges for realizing generative modeling discovered solutions in chemical applications.

Bottleneck problems are an important class of optimization problems that have recently gained increasing attention in the domain of machine learning and information theory. They are widely used in generative models, fair machine learning algorithms, design of privacy-assuring mechanisms, and appear as information-theoretic performance bounds in various multi-user communication problems. In this work, we propose a general family of optimization problems, termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity-leakage-utility bottleneck (CLUB)</i> model, which (i) provides a unified theoretical framework that generalizes most of the state-of-the-art literature for the information-theoretic privacy models, (ii) establishes a new interpretation of the popular generative and discriminative models, (iii) constructs new insights for the generative compression models, and (iv) can be used to obtain fair generative models. We first formulate the CLUB model as a complexity-constrained privacy-utility optimization problem. We then connect it with the closely related bottleneck problems, namely information bottleneck (IB), privacy funnel (PF), deterministic IB (DIB), conditional entropy bottleneck (CEB), and conditional PF (CPF). We show that the CLUB model generalizes all these problems as well as most other information-theoretic privacy models. Then, we construct the deep variational CLUB (DVCLUB) models by employing neural networks to parameterize variational approximations of the associated information quantities. Building upon these information quantities, we present unified objectives of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">supervised</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unsupervised</i> DVCLUB models. Leveraging the DVCLUB model in an unsupervised setup, we then connect it with state-of-the-art generative models, such as variational auto-encoders (VAEs), generative adversarial networks (GANs), as well as the Wasserstein GAN (WGAN), Wasserstein auto-encoder (WAE), and adversarial auto-encoder (AAE) models through the optimal transport (OT) problem. We then show that the DVCLUB model can also be used in fair representation learning problems, where the goal is to mitigate the undesired bias during the training phase of a machine learning model. We conduct extensive quantitative experiments on colored-MNIST and CelebA datasets.

The design of a new therapeutic agent is a time-consuming and expensive process. The rise of machine intelligence provides a grand opportunity of expeditiously discovering novel drug candidates through smart search in the vast molecular structural space. In this paper, we propose a new approach called adversarial deep evolutionary learning (ADEL) to search for novel molecules in the latent space of an adversarial generative model and keep improving the latent representation space. In ADEL, a custom-made adversarial autoencoder (AAE) model is developed and trained under a deep evolutionary learning (DEL) process. This involves an initial training of the AAE model, followed by an integration of multi-objective evolutionary optimization in the continuous latent representation space of the AAE rather than the discrete structural space of molecules. By using the AAE, an arbitrary distribution can be provided to the training of AAE such that the latent representation space is set to that distribution. This allows for a starting latent space from which new samples can be produced. Throughout the process of learning, new samples of high-quality are generated after each iteration of training and then added back into the full dataset. Therefore, allowing for a more comprehensive procedure of understanding the data structure. This combination of evolving data and continuous learning not only enables improvement in the generative model, but the data as well. By comparing ADEL to the previous work in DEL, we see that ADEL can obtain better property distributions.

Generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) play an increasingly important role in medical image analysis. The latent spaces of these models often show semantically meaningful directions corresponding to human-interpretable image transformations. However, until now, their exploration for medical images has been limited due to the requirement of supervised data. Several methods for unsupervised discovery of interpretable directions in GAN latent spaces have shown interesting results on natural images. This work explores the potential of applying these techniques on medical images by training a GAN and a VAE on thoracic CT scans and using an unsupervised method to discover interpretable directions in the resulting latent space. We find several directions corresponding to non-trivial image transformations, such as rotation or breast size. Furthermore, the directions show that the generative models capture 3D structure despite being presented only with 2D data. The results show that unsupervised methods to discover interpretable directions in GANs generalize to VAEs and can be applied to medical images. This opens a wide array of future work using these methods in medical image analysis. The code and animations of the discovered directions are available online at https://github.com/julschoen/Latent-Space-Exploration-CT.KeywordsGenerative modelsUnsupervised learningInterpretabilityCT

Generative deep learning is becoming a widely used approach in geological modelling, especially in problems that involve optimization processes under uncertainty like history matching. The basic idea of reservoir modelling and history matching by generative deep learning is to train some kind of reservoir generator and provide optimization under different constraints. There are different approaches that have been developed to utilize various modifications of Generative Adversarial Networks or Variational Autoencoders to implement reservoir generators, but most of them have "conventional" convolution neural networks, hence all data have to be regular (rectangular) in structure. This should be recognized as a limitation. In this work, we introduce a novel approach of reservoir modelling with Variational Autoencoders based on graph convolutions as opposed to "conventional" convolutions. In our approach reservoir model is considered as a graph, i.e., not a structured data type. Graph convolutions can deal with these data types, and connection with Variational Autoencoders provides us with the capability to generate initially unstructured reservoirs. Variational Autoencoders (VAE) demonstrate the ability to implicitly parameterize geological representations into a latent space of reduced dimensionality and provide ways to uncertainty quantification and production profiling among various geological concepts. In the first part, we introduce the motivation to Graph Variational Autoencoders (GVAE) as opposed to conventional Generative Deep learning models. The rest of the paper is dedicated to experiments on a synthetic dataset with two different geological scenarios. We show that trained GVAE performs the generation of reservoir models with reliable geology. A latent space structure between Encoder and Decoder of GVAE is represented, and interpretation considering geology under investigation is performed. Finally, we show how latent space can help to estimate the uncertainty of production and an optimization workflow under well data constraints.

In the field of deep learning, the generative model did not attract much attention until GANs (generative adversarial networks) appeared. In 2014, Google’s Ian Goodfellow proposed a generative model called GANs. GANs use different structures and objective functions from the existing generative model. For example, GANs use two neural networks: a generator that creates a realistic image, and a discriminator that distinguishes whether the input is real or synthetic. If there are no problems in the training process, GANs can generate images that are difficult even for experts to distinguish in terms of authenticity. Currently, GANs are the most researched subject in the field of computer vision, which deals with the technology of image style translation, synthesis, and generation, and various models have been unveiled. The issues raised are also improving one by one. In image synthesis, BEGAN (Boundary Equilibrium Generative Adversarial Network), which outperforms the previously announced GANs, learns the latent space of the image, while balancing the generator and discriminator. Nonetheless, BEGAN also has a mode collapse wherein the generator generates only a few images or a single one. Although BEGAN-CS (Boundary Equilibrium Generative Adversarial Network with Constrained Space), which was improved in terms of loss function, was introduced, it did not solve the mode collapse. The discriminator structure of BEGAN-CS is AE (AutoEncoder), which cannot create a particularly useful or structured latent space. Compression performance is not good either. In this paper, this characteristic of AE is considered to be related to the occurrence of mode collapse. Thus, we used VAE (Variational AutoEncoder), which added statistical techniques to AE. As a result of the experiment, the proposed model did not cause mode collapse but converged to a better state than BEGAN-CS.

A variational autoencoder (VAE) is a generational deep learning model based on encoding a particular observation into a latent space and then decoding it while incorporating some random noise with the intuition of being able to generate slightly different forms of the input observation. Here, I propose a VAE that is able to generate denovo drug compounds by feeding in a known set of drug compounds against a particular biological target. To demonstrate the ability of the proposed VAE as a generative model, known drug molecules belonging to the class of Neuraminidase (NA) inhibitors were taken from the ZINC database and fed into the VAE model as one-hot encoded SMILES strings. Similarly, active NA inhibitors and decoy molecules together were also fed to compare efficiency. The generated molecules were then screened to remove impractical structures. Next, a drug-likeness (QED) score was computed for each candidate molecule and a cutoff of 0.5 was used to extract viable candidates. To ensure that the generated drug compounds were active NA inhibitors, a series of Artificial Neural Networks (ANNs) classifiers based on three different characterization techniques, namely chemical fingerprinting, molecular descriptions and graph convolutions, were developed to identify active NA inhibitors from decoy molecules. The feature candidate data were then fed into the three designed ANNs to obtain the final set of novel, viable and active NA inhibitors. Seventy-one new NA inhibitors were obtained after three runs of the VAE model under different parameterizations. The proposed VAE can hence be used to generate de-novo drug compounds for a wide variety of biological targets.

Generative models have recently shown the ability to realistically generate data and model the distribution accurately. However, joint modeling of an image with the attribute that it is labeled with requires learning a cross modal correspondence between image and attribute data. Though the information present in a set of images and its attributes possesses completely different statistical properties altogether, there exists an inherent correspondence that is challenging to capture. Various models have aimed at capturing this correspondence either through joint modeling of a variational autoencoder or through separate encoder networks that are then concatenated. We present an alternative by proposing a bridged variational autoencoder that allows for learning cross-modal correspondence by incorporating cross-modal hallucination losses in the latent space. In comparison to the existing methods, we have found that by using a bridge connection in latent space we not only obtain better generation results, but also obtain highly parameter-efficient model which provide 40% reduction in training parameters for bimodal dataset and nearly 70% reduction for trimodal dataset. We validate the proposed method through comparison with state of the art methods and benchmarking on standard datasets.