Generative Models for 3D Vision (Dagstuhl Seminar 25202)

Mechanical metamaterials - engineered materials whose behavior is defined by their internal structure rather than their base material - have received great at- tention for the unusual properties they can exhibit. However, while the forward problem of predicting the mechanical behavior of a given metamaterial can be reliably done through simulation or experiments, the inverse problem - designing metamaterials with specific properties - remains difficult, especially for heteroge- neous metamaterials composed of multiple unit cells. This dissertation explores the use of generative machine learning to address this inverse design problem. The first part of the work investigates how inverse and generative models can be conditioned to produce unit cells with prescribed mechanical properties and how they can be used to generate multiple valid solutions for the same target. This is examined using a curved-beam metamaterial, which allows a compact geometric representation without complex design restrictions. The effect these restrictions have on the generative models is investigated in the second part. For this, a number of generative models—Variational Autoencoders, Generative Adversarial Networks, and Diffusion Models—are evaluated on a kirigami metamaterial where dependencies between the cuts cause intricate design constraints. The final part covers the challenges that arise when generative models are applied to heteroge- neous metamaterials, in which multiple unit cells are combined to create spatially varying mechanical properties. The results show that the applicability of generative models for the design of metamaterials strongly depends on the choice of geometric parameterization. Generative models make assumptions about their input data that generally hold for images, but can be invalid for metamaterials when complex dependencies be- tween the input parameters exist. B-splines can provide a representation without those dependencies and are therefore well suited to be paired up with generative models. Furthermore, it is also important how these geometrical parameters are sampled to create the training data for the generative models, as this sampling also affects the distribution of the properties in the training data. Overall, while generative machine learning models prove effective for the inverse design of mechanical metamaterials their successful application requires careful parameterization and selection of the training data.

Traditional computational approaches to design chemical species are limited by the need to compute properties for a vast number of candidates, e.g., by discriminative modeling. Therefore, inverse design methods aim to start from the desired property and optimize a corresponding chemical structure. From a machine learning viewpoint, the inverse design problem can be addressed through so-called generative modeling. Mathematically, discriminative models are defined by learning the probability distribution function of properties given the molecular or material structure. In contrast, a generative model seeks to exploit the joint probability of a chemical species with target characteristics. The overarching idea of generative modeling is to implement a system that produces novel compounds that are expected to have a desired set of chemical features, effectively sidestepping issues found in the forward design process. In this contribution, we overview and critically analyze popular generative algorithms like generative adversarial networks, variational autoencoders, flow, and diffusion models. We highlight key differences between each of the models, provide insights into recent success stories, and discuss outstanding challenges for realizing generative modeling discovered solutions in chemical applications.

The informed sampler: A discriminative approach to Bayesian inference in generative computer vision models

To complement computationally expensive regional climate model (RCM) simulations, machine learning methods can predict the high-resolution RCM data from low-resolution global climate model (GCM) input. Instead of merely targeting the conditional mean of the RCM field given the GCM data, more recent works are based on generative adversarial networks or diffusion models and aim to learn the full conditional distribution. In this spirit, we present a novel generative model that relies on statistical tools from forecast evaluation. The model can sample several plausible RCM realisations and enables assessing their variability. To achieve this goal, we use a simple neural network architecture that predicts Fourier coefficients of the high-resolution fields for multiple variables jointly (temperature, precipitation, solar radiation and wind). The loss function of our model is a proper scoring rule that measures the discrepancy between the model’s predictive distribution and the RCM’s true distribution. The score is minimised if both distributions agree. Our generative model is trained on multiple GCM-RCM combinations from the Euro-Cordex project. Furthermore, we show how the framework can be augmented to perform a bias-correction task: With a modified loss function, it is possible to generate data from the observational distribution, for example resembling gridded E-OBS data. To summarise, our work presents a machine learning method that allows us to generate multivariate high-resolution climate data, and can be extended flexibly to include further variables or downscale and bias-correct future projections.

Generative AI models have emerged as transformative tools in the healthcare domain, offering significant advances in disease detection, diagnosis, prognosis, and treatment planning. Although substantial progress has been made in the generation of 2D medical images, the synthesis of complex medical videos remains an unexplored area. The available literature on the generation of synthetic medical videos is minimal, highlighting a significant gap in this emerging area of research. This paper reviews the literature related to biomedical video synthesis using diffusion models and generative adversarial networks. The review aims to consolidate all relevant literature and highlight the different publicly available datasets, performance matrices, and the challenges associated with the generation of medical videos, along with some potential mitigation strategies. The findings of this review reveal that key challenges, such as maintaining temporal consistency, addressing computational inefficiencies, and overcoming data scarcity, are interconnected issues. Addressing these issues collectively is essential for the development of accurate and robust generative models tailored for medical video synthesis. The proposed potential mitigation strategies for the limitations of generative models in this review serve as a foundational resource for future research, aiming to enhance the reliability and applicability of generative AI models in clinical settings. These advances have the potential to significantly impact the domains of connected healthcare and personalized medicine by enabling the generation of realistic, high-quality medical video data that can enhance the training of diagnostic algorithms, improve the robustness of AI-assisted video interpretation, simulate disease progression or regression for more precise treatment planning, and support the development of personalized medicine techniques through enriched longitudinal data analysis.

Bottleneck problems are an important class of optimization problems that have recently gained increasing attention in the domain of machine learning and information theory. They are widely used in generative models, fair machine learning algorithms, design of privacy-assuring mechanisms, and appear as information-theoretic performance bounds in various multi-user communication problems. In this work, we propose a general family of optimization problems, termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity-leakage-utility bottleneck (CLUB)</i> model, which (i) provides a unified theoretical framework that generalizes most of the state-of-the-art literature for the information-theoretic privacy models, (ii) establishes a new interpretation of the popular generative and discriminative models, (iii) constructs new insights for the generative compression models, and (iv) can be used to obtain fair generative models. We first formulate the CLUB model as a complexity-constrained privacy-utility optimization problem. We then connect it with the closely related bottleneck problems, namely information bottleneck (IB), privacy funnel (PF), deterministic IB (DIB), conditional entropy bottleneck (CEB), and conditional PF (CPF). We show that the CLUB model generalizes all these problems as well as most other information-theoretic privacy models. Then, we construct the deep variational CLUB (DVCLUB) models by employing neural networks to parameterize variational approximations of the associated information quantities. Building upon these information quantities, we present unified objectives of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">supervised</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unsupervised</i> DVCLUB models. Leveraging the DVCLUB model in an unsupervised setup, we then connect it with state-of-the-art generative models, such as variational auto-encoders (VAEs), generative adversarial networks (GANs), as well as the Wasserstein GAN (WGAN), Wasserstein auto-encoder (WAE), and adversarial auto-encoder (AAE) models through the optimal transport (OT) problem. We then show that the DVCLUB model can also be used in fair representation learning problems, where the goal is to mitigate the undesired bias during the training phase of a machine learning model. We conduct extensive quantitative experiments on colored-MNIST and CelebA datasets.

Abstract. When recording street scenes, the privacy of the people concerned must always be taken into account. Previous work has focused heavily on directly identifiable features such as faces or license plates. However, the visibility of indirectly identifiable objects can also limit the protection of citizens. This is especially true for cars. Regular recordings could be used to infer when a person is at home or where else they have been. The obvious solution would be to simply make cars unrecognizable. However, previous work has shown that this has a negative impact on downstream tasks such as traffic sign recognition. Therefore, in this work, we use generative models to synthetically modify cars. We compare models from the generative adversarial network (DP-GAN, OASIS) and diffusion model communities (Kolors, SDXL) to see which ones are best suited in terms of anonymity, image integrity, and performance. For the GAN-based models, we use an image-to-image translation approach to modify only image sections with cars, while for the diffusion models, we develop two methods that use text-guided image inpainting. We compare the developed methods using established metrics and perform a survey with test subjects. In terms of anonymity, all models achieve convincing results, while diffusion models that generate each car mask individually produce particularly realistic images. In return, GAN-based methods process images more than twice as fast creating a trade-off between image integrity and performance.

Surface defect inspection based on deep learning has demonstrated outstanding performance in improving detection accuracy and model generalization. However, the small scale of defect datasets always limits the application of deep models in industry. Generative models can obtain realistic samples in a very cheap way, which can effectively solve this problem and thus has received widespread attention in recent years. This paper provides a comprehensive analysis and summary of the current studies of surface defect inspection methods proposed between 2022 and 2024. First, according to the use of generative models, these methods are classified into four categories: Variational Auto-Encoders (VAEs), Generative Adversarial Networks (GANs), Diffusion Models (DMs), and multi-models. Second, the research status of surface defect inspection based on generative models in recent years is discussed from four aspects: sample generation, detection objective, inspection task, and learning model. Then, the public datasets and evaluation metrics that are commonly used for surface defect inspection are discussed, and a comparative evaluation of defect inspection methods based on generative models is provided. Finally, this study discusses the existing challenges for the defect inspection methods based on generative models, providing insights for future research.

Reliable synthesis and prediction of seismic waveforms play an important role in evaluating seismic hazards and designing earthquake-resilient structures. However, current methods, such as ground motion models and physics-based simulations, are often limited in fully capturing the complexity of seismic wave propagation, at higher frequencies (&gt;5 Hz). Some of these limitations can potentially be overcome through machine learning techniques. In earthquake engineering, machine learning models have been used for predicting peak ground accelerations and Fourier spectra responses. To model entire waveforms, extensive efforts to generate seismic waveforms have employed advanced machine learning techniques, such as generative models, with most previous approaches relying on generative adversarial networks (GANs). In contrast to these earlier models, this study presents an efficient and extensible generative framework to produce realistic high-frequency seismic waveforms, compared to GANs. Our approach encodes spectrograms of the waveform data into a lower-dimensional sub-manifold using an autoencoder, and a state-of-the-art diffusion model is subsequently trained to generate these latent embeddings. Conditioning is currently performed on key parameters: earthquake magnitude, recording distance, site conditions, and faulting style. The resulting generative model can synthesize waveforms with frequency content up to 50 Hz, from which several scalar ground motion statistics, such as peak ground motion amplitudes, spectral accelerations, or Arias intensity can be directly derived. We validate the quality of the generated waveforms using standard seismological benchmarks and performance metrics from image generation research. Our openly available model produces high-frequency waveforms that align with real data across a wide range of input parameters, including regions where observations are sparse, and accurately reproduces both median trends and variability of empirical ground motion statistics. Our generative waveform model can be potentially used to perform seismic hazard where broadband data are often required such as to train earthquake early warning model. Given the increasing number of generative waveform models, we emphasize that they should be openly accessible and included in community efforts for ground motion model evaluations.

Generative diffusion models are a well-established method for generating high-quality images. However, there are studies that show that diffusion models are less privacy-friendly than generative models, such as generative adversarial networks and a growing family of their modifications. The discovered vulnerabilities require in-depth study of various security aspects. This is especially important for sensitive areas such as medical image analysis tasks and their practical applications. The paper describes a method for detecting image patterns presented in generated images that can potentially be identified in real CT images of patients with pulmonary tuberculosis. The method includes the following main procedures: correlation of pairs of generated and real images to pre-select pairs that involve further analysis; calculation of correlation statistics using direct and inverse Fisher transforms; performing affine image registration and calculating pairwise similarity scores; nonlinear (elastic) image registration and recalculation of similarity scores to highlight the most similar/dissimilar image areas.

Large annotated datasets are required for training deep learning models, but in medical imaging data sharing is often complicated due to ethics, anonymization and data protection legislation. Generative AI models, such as generative adversarial networks (GANs) and diffusion models, can today produce very realistic synthetic images, and can potentially facilitate data sharing. However, in order to share synthetic medical images it must first be demonstrated that they can be used for training different networks with acceptable performance. Here, we therefore comprehensively evaluate four GANs (progressive GAN, StyleGAN 1–3) and a diffusion model for the task of brain tumor segmentation (using two segmentation networks, U-Net and a Swin transformer). Our results show that segmentation networks trained on synthetic images reach Dice scores that are 80%–90% of Dice scores when training with real images, but that memorization of the training images can be a problem for diffusion models if the original dataset is too small. Our conclusion is that sharing synthetic medical images is a viable option to sharing real images, but that further work is required. The trained generative models and the generated synthetic images are shared on AIDA data hub.

Autoencoders and generative neural network models have recently gained popularity in fluid mechanics due to their spontaneity and low processing time instead of high fidelity CFD simulations. Auto encoders are used as model order reduction tools in applications of fluid mechanics by compressing input high-dimensional data using an encoder to map the input space into a lower-dimensional latent space. Whereas, generative models such as Variational Auto-encoders (VAEs) and Generative Adversarial Networks (GANs) are proving to be effective in generating solutions to chaotic models with high ‘randomness’ such as turbulent flows. In this study, forced isotropic turbulence flow is generated by parameterizing into some basic statistical characteristics. The models trained on pre-simulated data from dependencies on these characteristics and the flow generation is then affected by varying these parameters. The latent vectors pushed along the generator models like the decoders and generators contain independent entries which can be used to create different outputs with similar properties. The use of neural network-based architecture removes the need for dependency on the classical mesh-based Navier-Stoke equation estimation which is prominent in many CFD softwares.

Drug discovery is a multi-stage process, often beginning with the identification of active molecules from a high-throughput screen or machine learning model. Once structure activity relationship trends become well established, identifying new analogs with better properties is important. Synthesizing these new compounds is a logical next step, and is key to research groups that have a synthetic chemistry team or external collaborators. Generative machine learning models have become widely adopted to generate new molecules and explore molecular space, with the goal of discovering novel compounds with desires properties. These generative models have been composed from recurrent neural networks (RNNs), Variational Autoencoders (VAEs), and Generative Adversarial Networks (GANs) and are often combined with transfer learning or scoring of physicochemical properties to steer generative design. While these generative models have proven useful in generating new molecular libraries, often they are not capable of addressing a wide variety of potential problems, and often converge into similar molecular space when combined with a scoring function for desired properties. In addition, generated compounds are often not synthetically feasible, reducing their capabilities outside of virtual composition and limiting their usefulness in real-world scenarios. Here we introduce a suite of automated tools called MegaSyn representing 3 components: a new hill-climb algorithm which makes use of SMILES-based RNN generative models, analog generation software, and retrosynthetic analysis coupled with fragment analysis to score molecules for their synthetic feasibility. We now describe the development and testing of this suite of tools and propose how they might be used to optimize molecules or prioritize promising lead compounds using test case examples.

In the field of deep learning, the generative model did not attract much attention until GANs (generative adversarial networks) appeared. In 2014, Google’s Ian Goodfellow proposed a generative model called GANs. GANs use different structures and objective functions from the existing generative model. For example, GANs use two neural networks: a generator that creates a realistic image, and a discriminator that distinguishes whether the input is real or synthetic. If there are no problems in the training process, GANs can generate images that are difficult even for experts to distinguish in terms of authenticity. Currently, GANs are the most researched subject in the field of computer vision, which deals with the technology of image style translation, synthesis, and generation, and various models have been unveiled. The issues raised are also improving one by one. In image synthesis, BEGAN (Boundary Equilibrium Generative Adversarial Network), which outperforms the previously announced GANs, learns the latent space of the image, while balancing the generator and discriminator. Nonetheless, BEGAN also has a mode collapse wherein the generator generates only a few images or a single one. Although BEGAN-CS (Boundary Equilibrium Generative Adversarial Network with Constrained Space), which was improved in terms of loss function, was introduced, it did not solve the mode collapse. The discriminator structure of BEGAN-CS is AE (AutoEncoder), which cannot create a particularly useful or structured latent space. Compression performance is not good either. In this paper, this characteristic of AE is considered to be related to the occurrence of mode collapse. Thus, we used VAE (Variational AutoEncoder), which added statistical techniques to AE. As a result of the experiment, the proposed model did not cause mode collapse but converged to a better state than BEGAN-CS.

Generative Adversarial Networks has shown impressive achievement in computer graphics applications, and it's now widely used worldwide. GAN is composed of generator and discriminator, which are crucial compositions of GAN. GAN can generate 3D models, graphics that are required in animated movies or video game characters. For example, a generative model could simply generate a new picture that looks like a specific type of animal. In the meantime, the discriminative model could distinguish the picture from a human being or an animal. GAN variants consist of progressive GAN as well as conditional GAN. Since GAN can be applied in so many different areas, in this essay, we are going to talk about how GAN is applied in image and computer vision. More specifically, face synthesis, image to image translation, and super resolution. Conclusively, GAN has a significant contribution to various areas, and it boosts the advancement in the domain of computer graphics.