Cosmological N-body simulations: a challenge for scalable generative models

Automated design synthesis has the potential to revolutionize the modern engineering design process and improve access to highly optimized and customized products across countless industries. Successfully adapting generative machine learning to design engineering may enable such automated design synthesis and is a research subject of great importance. We present a review and analysis of deep generative machine learning models in engineering design. Deep generative models (DGMs) typically leverage deep networks to learn from an input dataset and synthesize new designs. Recently, DGMs such as feedforward neural networks (NNs), generative adversarial networks (GANs), variational autoencoders (VAEs), and certain deep reinforcement learning (DRL) frameworks have shown promising results in design applications like structural optimization, materials design, and shape synthesis. The prevalence of DGMs in engineering design has skyrocketed since 2016. Anticipating the continued growth, we conduct a review of recent advances to benefit researchers interested in DGMs for design. We structure our review as an exposition of the algorithms, datasets, representation methods, and applications commonly used in the current literature. In particular, we discuss key works that have introduced new techniques and methods in DGMs, successfully applied DGMs to a design-related domain, or directly supported the development of DGMs through datasets or auxiliary methods. We further identify key challenges and limitations currently seen in DGMs across design fields, such as design creativity, handling constraints and objectives, and modeling both form and functional performance simultaneously. In our discussion, we identify possible solution pathways as key areas on which to target the future work.

Surface defect inspection based on deep learning has demonstrated outstanding performance in improving detection accuracy and model generalization. However, the small scale of defect datasets always limits the application of deep models in industry. Generative models can obtain realistic samples in a very cheap way, which can effectively solve this problem and thus has received widespread attention in recent years. This paper provides a comprehensive analysis and summary of the current studies of surface defect inspection methods proposed between 2022 and 2024. First, according to the use of generative models, these methods are classified into four categories: Variational Auto-Encoders (VAEs), Generative Adversarial Networks (GANs), Diffusion Models (DMs), and multi-models. Second, the research status of surface defect inspection based on generative models in recent years is discussed from four aspects: sample generation, detection objective, inspection task, and learning model. Then, the public datasets and evaluation metrics that are commonly used for surface defect inspection are discussed, and a comparative evaluation of defect inspection methods based on generative models is provided. Finally, this study discusses the existing challenges for the defect inspection methods based on generative models, providing insights for future research.

Deep generative models are promising in detecting novel cyber-physical attacks, mitigating the vulnerability of Cyber-physical systems (CPSs) without relying on labeled information. Nonetheless, these generative models face challenges in identifying attack behaviors that closely resemble normal data, or deviate from the normal data distribution but are in close proximity to the manifold of the normal cluster in latent space. To tackle this problem, this article proposes a novel unsupervised dual variational generative adversarial model named MST-DVGAN, to perform anomaly detection in multivariate time series data for CPS security. The central concept is to enhance the model's discriminative capability by widening the distinction between reconstructed abnormal samples and their normal counterparts. Specifically, we propose an augmented module by imposing contrastive constraints on the reconstruction process to obtain a more compact embedding. Then, by exploiting the distribution property and modeling the normal patterns of multivariate time series, a variational autoencoder is introduced to force the generative adversarial network (GAN) to generate diverse samples. Furthermore, two augmented loss functions are designed to extract essential characteristics in a self-supervised manner through mutual guidance between the augmented samples and original samples. Finally, a specific feature center loss is introduced for the generator network to enhance its stability. Empirical experiments are conducted on three public datasets, namely SWAT, WADI and NSL_KDD. Comparing with the state-of-the-art methods, the evaluation results show that the proposed MTS-DVGAN is more stable and can achieve consistent performance improvement.

Deep generative models (GMs) have transformed the exploration of drug-like chemical space (CS) by generating novel molecules through complex, nontransparent processes, bypassing direct structural similarity. This review examines five key architectures for CS exploration: recurrent neural networks (RNNs), variational autoencoders (VAEs), generative adversarial networks (GANs), normalizing flows (NF), and Transformers. It discusses molecular representation choices, training strategies for focused CS exploration, evaluation criteria for CS coverage, and related challenges. Future directions include refining models, exploring new notations, improving benchmarks, and enhancing interpretability to better understand biologically relevant molecular properties.

The fusion of Federated Learning (FL) and deep generative models is transforming medical imaging by enabling privacy-preserving and data-efficient machine learning. Training large-scale deep models on radiological imaging data remains challenging due to data scarcity, heterogeneity, and strict privacy constraints that limit data sharing across institutions. FL addresses these challenges by employing collaborative model training that does not expose raw patient data, while generative models synthesize realistic medical images to alleviate scarcity and imbalance. The convergence of these two fields has given rise to federated generative models (FGMs), which extend Generative Adversarial Networks (GANs), Variational Autoencoders (VAE), and Diffusion models into distributed and privacy-preserving environments. FGMs have demonstrated significant potential in various medical imaging tasks, including data augmentation, image reconstruction, cross-modality conversion, and enhancing classification and segmentation models. Despite promising progress, the field remains in its early stages, facing open challenges in communication efficiency, scalability, data heterogeneity, and clinical reliability. This survey provides a comprehensive review of FGMs in medical imaging, analyzing their architectures, applications, and benchmark datasets, while highlighting key challenges and outlining future research directions to advance privacy-preserving generative modeling in healthcare.

In the field of deep learning, the generative model did not attract much attention until GANs (generative adversarial networks) appeared. In 2014, Google’s Ian Goodfellow proposed a generative model called GANs. GANs use different structures and objective functions from the existing generative model. For example, GANs use two neural networks: a generator that creates a realistic image, and a discriminator that distinguishes whether the input is real or synthetic. If there are no problems in the training process, GANs can generate images that are difficult even for experts to distinguish in terms of authenticity. Currently, GANs are the most researched subject in the field of computer vision, which deals with the technology of image style translation, synthesis, and generation, and various models have been unveiled. The issues raised are also improving one by one. In image synthesis, BEGAN (Boundary Equilibrium Generative Adversarial Network), which outperforms the previously announced GANs, learns the latent space of the image, while balancing the generator and discriminator. Nonetheless, BEGAN also has a mode collapse wherein the generator generates only a few images or a single one. Although BEGAN-CS (Boundary Equilibrium Generative Adversarial Network with Constrained Space), which was improved in terms of loss function, was introduced, it did not solve the mode collapse. The discriminator structure of BEGAN-CS is AE (AutoEncoder), which cannot create a particularly useful or structured latent space. Compression performance is not good either. In this paper, this characteristic of AE is considered to be related to the occurrence of mode collapse. Thus, we used VAE (Variational AutoEncoder), which added statistical techniques to AE. As a result of the experiment, the proposed model did not cause mode collapse but converged to a better state than BEGAN-CS.

Traditional computational approaches to design chemical species are limited by the need to compute properties for a vast number of candidates, e.g., by discriminative modeling. Therefore, inverse design methods aim to start from the desired property and optimize a corresponding chemical structure. From a machine learning viewpoint, the inverse design problem can be addressed through so-called generative modeling. Mathematically, discriminative models are defined by learning the probability distribution function of properties given the molecular or material structure. In contrast, a generative model seeks to exploit the joint probability of a chemical species with target characteristics. The overarching idea of generative modeling is to implement a system that produces novel compounds that are expected to have a desired set of chemical features, effectively sidestepping issues found in the forward design process. In this contribution, we overview and critically analyze popular generative algorithms like generative adversarial networks, variational autoencoders, flow, and diffusion models. We highlight key differences between each of the models, provide insights into recent success stories, and discuss outstanding challenges for realizing generative modeling discovered solutions in chemical applications.

Bottleneck problems are an important class of optimization problems that have recently gained increasing attention in the domain of machine learning and information theory. They are widely used in generative models, fair machine learning algorithms, design of privacy-assuring mechanisms, and appear as information-theoretic performance bounds in various multi-user communication problems. In this work, we propose a general family of optimization problems, termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity-leakage-utility bottleneck (CLUB)</i> model, which (i) provides a unified theoretical framework that generalizes most of the state-of-the-art literature for the information-theoretic privacy models, (ii) establishes a new interpretation of the popular generative and discriminative models, (iii) constructs new insights for the generative compression models, and (iv) can be used to obtain fair generative models. We first formulate the CLUB model as a complexity-constrained privacy-utility optimization problem. We then connect it with the closely related bottleneck problems, namely information bottleneck (IB), privacy funnel (PF), deterministic IB (DIB), conditional entropy bottleneck (CEB), and conditional PF (CPF). We show that the CLUB model generalizes all these problems as well as most other information-theoretic privacy models. Then, we construct the deep variational CLUB (DVCLUB) models by employing neural networks to parameterize variational approximations of the associated information quantities. Building upon these information quantities, we present unified objectives of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">supervised</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unsupervised</i> DVCLUB models. Leveraging the DVCLUB model in an unsupervised setup, we then connect it with state-of-the-art generative models, such as variational auto-encoders (VAEs), generative adversarial networks (GANs), as well as the Wasserstein GAN (WGAN), Wasserstein auto-encoder (WAE), and adversarial auto-encoder (AAE) models through the optimal transport (OT) problem. We then show that the DVCLUB model can also be used in fair representation learning problems, where the goal is to mitigate the undesired bias during the training phase of a machine learning model. We conduct extensive quantitative experiments on colored-MNIST and CelebA datasets.

Generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs) have been recently applied to various fields. However, the VAE and GAN models have blur and mode collapse problems, respectively. Here, the authors propose a novel generative model, self‐converging generative network (SCGN), to address the issues. Self‐converging means the convergence of latent vectors into themselves through being trained in pairs with training data, by which the SCGN can reconstruct all training data. In the authors’ model, the latent vectors and weights of the generator are alternately trained. Specifically, the latent vectors are trained to follow a normal distribution, using a loss function derived from the Kullback–Leibler divergence and a pixel‐wise loss. The weights of the generator are adjusted for the generator to produce training data by means of a pixel‐wise loss. As a result, their SCGN did not fall into the mode collapse, which occurs in GANs, and made clearer images than VAEs thanks to no use of sampling. Moreover, the SCGN successfully learned the manifold of the dataset in the extensive experiments with CelebA.

Localization in reverberant environments remains an open challenge. Recently, supervised learning approaches have demonstrated very promising results in addressing reverberation. However, even with large data volumes, the number of labels available for supervised learning in such environments is usually small. We propose to address this issue with a semi-supervised learning (SSL) approach, based on deep generative modeling. Our chosen deep generative model, the variational autoencoder (VAE), is trained to generate the phase of relative transfer functions (RTFs) between microphones. In parallel, a direction of arrival (DOA) classifier network based on RTF-phase is also trained. The joint generative and discriminative model, deemed VAE-SSL, is trained using labeled and unlabeled RTF-phase sequences. In learning to generate and classify the sequences, the VAE-SSL extracts the physical causes of the RTF-phase (i.e., source location) from distracting signal characteristics such as noise and speech activity. This facilitates effective end-to-end operation of the VAE-SSL, which requires minimal preprocessing of RTF-phase. VAE-SSL is compared with two signal processing-based approaches, steered response power with phase transform (SRP-PHAT) and MUltiple SIgnal Classification (MUSIC), as well as fully supervised CNNs. The approaches are compared using data from two real acoustic environments - one of which was recently obtained at Technical University of Denmark specifically for our study. We find that VAE-SSL can outperform the conventional approaches and the CNN in label-limited scenarios. Further, the trained VAE-SSL system can generate new RTF-phase samples which capture the physics of the acoustic environment. Thus, the generative modeling in VAE-SSL provides a means of interpreting the learned representations. To the best of our knowledge, this paper presents the first approach to modeling the physics of acoustic propagation using deep generative modeling.

A deep generative model for probabilistic energy forecasting in power systems: normalizing flows

Medical image reconstruction is typically an ill-posed inverse problem. In order to address such ill-posed problems, the prior distribution of the sought after object property is usually incorporated by means of some sparsity-promoting regularization. Recently, prior distributions for images estimated using generative adversarial networks (GANs) have shown great promise in regularizing some of these image reconstruction problems. In this work, we apply an image-adaptive GAN-based reconstruction method (IAGAN) to reconstruct high fidelity images from incomplete medical imaging data. It is observed that the IAGAN method can potentially recover fine structures in the object that are relevant for medical diagnosis but may be oversmoothed in reconstructions with traditional sparsity-promoting regularization.

This paper contributes to the field of deep generative learning applied to solar photovoltaic (PV) synthetic data generation problems by exploring Deep Generative Model (DGM) that combines Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN), i.e., VAEGAN. We build upon knowledge in the area of deep learning to incorporate our Hybrid Deep Neural Network (HDNN), combining convolutional and Long Short-Term Memory (LSTM) layers at the encoding level for producing robust latent representations and subsequently high-quality synthetic PV data samples. The major advantage of these approaches is that it allows the DGMs to perform better feature extraction as well as to capture the historical trends in data effectively. The simulations on actual data acquired from a real PV system demonstrate the effectiveness of the DGMs to produce high-quality samples for multiple seasons of the year.

Autoencoders and generative neural network models have recently gained popularity in fluid mechanics due to their spontaneity and low processing time instead of high fidelity CFD simulations. Auto encoders are used as model order reduction tools in applications of fluid mechanics by compressing input high-dimensional data using an encoder to map the input space into a lower-dimensional latent space. Whereas, generative models such as Variational Auto-encoders (VAEs) and Generative Adversarial Networks (GANs) are proving to be effective in generating solutions to chaotic models with high ‘randomness’ such as turbulent flows. In this study, forced isotropic turbulence flow is generated by parameterizing into some basic statistical characteristics. The models trained on pre-simulated data from dependencies on these characteristics and the flow generation is then affected by varying these parameters. The latent vectors pushed along the generator models like the decoders and generators contain independent entries which can be used to create different outputs with similar properties. The use of neural network-based architecture removes the need for dependency on the classical mesh-based Navier-Stoke equation estimation which is prominent in many CFD softwares.

Drug discovery is a multi-stage process, often beginning with the identification of active molecules from a high-throughput screen or machine learning model. Once structure activity relationship trends become well established, identifying new analogs with better properties is important. Synthesizing these new compounds is a logical next step, and is key to research groups that have a synthetic chemistry team or external collaborators. Generative machine learning models have become widely adopted to generate new molecules and explore molecular space, with the goal of discovering novel compounds with desires properties. These generative models have been composed from recurrent neural networks (RNNs), Variational Autoencoders (VAEs), and Generative Adversarial Networks (GANs) and are often combined with transfer learning or scoring of physicochemical properties to steer generative design. While these generative models have proven useful in generating new molecular libraries, often they are not capable of addressing a wide variety of potential problems, and often converge into similar molecular space when combined with a scoring function for desired properties. In addition, generated compounds are often not synthetically feasible, reducing their capabilities outside of virtual composition and limiting their usefulness in real-world scenarios. Here we introduce a suite of automated tools called MegaSyn representing 3 components: a new hill-climb algorithm which makes use of SMILES-based RNN generative models, analog generation software, and retrosynthetic analysis coupled with fragment analysis to score molecules for their synthetic feasibility. We now describe the development and testing of this suite of tools and propose how they might be used to optimize molecules or prioritize promising lead compounds using test case examples.