Deep generative models meet statistical methods: a generalized framework for financial regime identification

Automated design synthesis has the potential to revolutionize the modern engineering design process and improve access to highly optimized and customized products across countless industries. Successfully adapting generative machine learning to design engineering may enable such automated design synthesis and is a research subject of great importance. We present a review and analysis of deep generative machine learning models in engineering design. Deep generative models (DGMs) typically leverage deep networks to learn from an input dataset and synthesize new designs. Recently, DGMs such as feedforward neural networks (NNs), generative adversarial networks (GANs), variational autoencoders (VAEs), and certain deep reinforcement learning (DRL) frameworks have shown promising results in design applications like structural optimization, materials design, and shape synthesis. The prevalence of DGMs in engineering design has skyrocketed since 2016. Anticipating the continued growth, we conduct a review of recent advances to benefit researchers interested in DGMs for design. We structure our review as an exposition of the algorithms, datasets, representation methods, and applications commonly used in the current literature. In particular, we discuss key works that have introduced new techniques and methods in DGMs, successfully applied DGMs to a design-related domain, or directly supported the development of DGMs through datasets or auxiliary methods. We further identify key challenges and limitations currently seen in DGMs across design fields, such as design creativity, handling constraints and objectives, and modeling both form and functional performance simultaneously. In our discussion, we identify possible solution pathways as key areas on which to target the future work.

The de novo molecule generation problem involves generating novel or modified molecular structures with desirable properties. Taking advantage of the great representation learning ability of deep learning models, deep generative models, which differ from discriminative models in their traditional machine learning approach, provide the possibility of generation of desirable molecules directly. Although deep generative models have been extensively discussed in the machine learning community, a specific investigation of the computational issues related to deep generative models for de novo molecule generation is needed. A concise and insightful discussion of recent advances in applying deep generative models for de novo molecule generation is presented, with particularly emphasizing the most important challenges for successful application of deep generative models in this specific area.This article is categorized under: Computer and Information Science > Chemoinformatics Computer and Information Science > Computer Algorithms and Programming

Early warning for learning performance requires to identify the maximum number of at-risk students as early as possible within a semester. However, educational data often suffer from the issue of data imbalanced, making it challenging to simultaneously achieve both high precision (accurate identification) and high recall (comprehensive coverage) in at-risk student detection. Deep generative models and oversampling models are effective methods to solve data imbalanced issues, which can improve classification performance. This paper proposes a method that combines the advantages of deep generative models and oversampling models to build a blending model for dealing with the imbalanced education data, which can effectively improve the precision, recall, f1-score and auc for online learning early warning. First, we compare baseline models to select the best classifier, then choose the highest-precision deep generative model and the highest-recall oversampling model to construct blending models, which are shown to improve early warning prediction metrics. Finally, interpretable models are used to analyze differences in at-risk student prediction between the blending model, deep generative model, and oversampling model. The proposed models are validated on both extremely imbalanced datasets and new semester datasets. Results show that: (1) Compared to the baseline model, both the base learners build by deep generative model and oversampling model can improve the evaluation metrics of the model, the deep generative base learners achieve higher precision than oversampling model, the oversampling base learners achieve higher recall than the deep generative base learners. (2) The blending model composed of deep generative base learner and oversampling base learner can further improve the f1-score and auc based on their individual strengths, the proposed blending model can also conduct effective early warning 3 units earlier than baseline models. (3) Compared to its base learners, blending model G-B-Blending changed the key variables for prediction, and the at-risk students identified by the blending model come from the union set of at-risk students identified by GAN+GB and B-SMOTE+GB individually. (4) The blending model proposed in this paper achieves better prediction results than the baseline on both extremely imbalanced datasets and new semesters datasets, it can identify more at-risk students more accurately at earlier units, allowing teachers to save more energy and time for teaching interventions. This research provides significant insights for dealing with imbalanced datasets by blending with deep generative model and oversampling model in education.

Synthetic data offers a promising avenue for addressing privacy, scarcity, and fairness challenges in healthcare datasets. However, there is limited evaluation of how different generation methods balance fidelity, utility, and fairness, particularly for underrepresented subgroups. This study addresses this gap by comparing representative generative modelling techniques, both probabilistic and deep approaches, that are popular in the research literature. We empirically evaluate BayesBoost, CTGAN, TVAE, CopulaGAN, and DECAF on two healthcare datasets containing numerical, binary, and categorical features. Each model’s performance is assessed along three axes: data fidelity, machine learning utility, and fairness, using Accuracy Parity, Equalised Odds, and Predictive Rate Parity. Results show that BayesBoost consistently achieved superior fidelity, utility, and fairness preservation, particularly when paired with Random Forest classifiers, achieving around 60 – 63 % higher downstream utility than GAN-based deep generative baselines (e.g., Random Forest accuracy up to 0.88 with BayesBoost versus 0.54 to − 0.55 for GAN-based methods). Deep generative models, while effective in capturing complex structures, often degraded fairness, especially for underrepresented groups, with equalised odds deviating by over 100 % from the ideal parity value of 1.0 in some settings. The Variational Autoencoder outperformed other deep generative models in fairness preservation, especially for equalised odds, although with some reduction in fidelity and utility. Overall, these findings suggest that synthetic data generation for healthcare must move beyond fidelity evaluations to explicitly assess fairness and subgroup impacts, with probabilistic models such as BayesBoost showing strong potential for ethical deployment, while deep generative models require further adaptation for fairness-sensitive applications.

Inverse design allows the design of molecules with desirable properties using property optimization. Deep generative models have recently been applied to tackle inverse design, as they possess the ability to optimize molecular properties directly through structure modification using gradients. While the ability to carry out direct property optimizations is promising, the use of generative deep learning models to solve practical problems requires large amounts of data and is very time-consuming. In this work, we propose STONED – a simple and efficient algorithm to perform interpolation and exploration in the chemical space, comparable to deep generative models. STONED bypasses the need for large amounts of data and training times by using string modifications in the SELFIES molecular representation. We achieve comparable performance on typical benchmarks without any training. We demonstrate applications in high-throughput virtual screening for the design of drugs, photovoltaics, and the construction of chemical paths, allowing for both property and structure-based interpolation in the chemical space. We anticipate our results to be a stepping stone for developing more sophisticated inverse design models and benchmarking tools, ultimately helping generative models achieve wide adoption.

This paper mainly applies deep generative models for short-term load forecasting on the enterprise electricity consumption dataset. After data cleaning on the electricity use dataset with the help of related weather data, we complement missing data and improve the data quality to better implement neural network generative prediction models. We build DeepAR and Wavenet as the representative of deep generative models. The main result is that deep generative models perform better compared with other baseline models, such as ARIMA, machine learning and baseline neural networks, no matter what accuracy metric and prediction horizon. Further improvement is to test in higher frequency electricity dataset with better quality.

The de novo design of molecular structures using deep learning generative models introduces an encouraging solution to drug discovery in the face of the continuously increased cost of new drug development. From the generation of original texts, images, and videos, to the scratching of novel molecular structures the creativity of deep learning generative models exhibits the height machine intelligence can achieve. The purpose of this paper is to review the latest advances in generative chemistry which relies on generative modeling to expedite the drug discovery process. This review starts with a brief history of artificial intelligence in drug discovery to outline this emerging paradigm. Commonly used chemical databases, molecular representations, and tools in cheminformatics and machine learning are covered as the infrastructure for generative chemistry. The detailed discussions on utilizing cutting-edge generative architectures, including recurrent neural network, variational autoencoder, adversarial autoencoder, and generative adversarial network for compound generation are focused. Challenges and future perspectives follow.

Localization in reverberant environments remains an open challenge. Recently, supervised learning approaches have demonstrated very promising results in addressing reverberation. However, even with large data volumes, the number of labels available for supervised learning in such environments is usually small. We propose to address this issue with a semi-supervised learning (SSL) approach, based on deep generative modeling. Our chosen deep generative model, the variational autoencoder (VAE), is trained to generate the phase of relative transfer functions (RTFs) between microphones. In parallel, a direction of arrival (DOA) classifier network based on RTF-phase is also trained. The joint generative and discriminative model, deemed VAE-SSL, is trained using labeled and unlabeled RTF-phase sequences. In learning to generate and classify the sequences, the VAE-SSL extracts the physical causes of the RTF-phase (i.e., source location) from distracting signal characteristics such as noise and speech activity. This facilitates effective end-to-end operation of the VAE-SSL, which requires minimal preprocessing of RTF-phase. VAE-SSL is compared with two signal processing-based approaches, steered response power with phase transform (SRP-PHAT) and MUltiple SIgnal Classification (MUSIC), as well as fully supervised CNNs. The approaches are compared using data from two real acoustic environments - one of which was recently obtained at Technical University of Denmark specifically for our study. We find that VAE-SSL can outperform the conventional approaches and the CNN in label-limited scenarios. Further, the trained VAE-SSL system can generate new RTF-phase samples which capture the physics of the acoustic environment. Thus, the generative modeling in VAE-SSL provides a means of interpreting the learned representations. To the best of our knowledge, this paper presents the first approach to modeling the physics of acoustic propagation using deep generative modeling.

We explore methods of producing adversarial examples on deep generative models such as the variational autoencoder (VAE) and the VAE-GAN. Deep learning architectures are known to be vulnerable to adversarial examples, but previous work has focused on the application of adversarial examples to classification tasks. Deep generative models have recently become popular due to their ability to model input data distributions and generate realistic examples from those distributions. We present three classes of attacks on the VAE and VAE-GAN architectures and demonstrate them against networks trained on MNIST, SVHN and CelebA. Our first attack leverages classification-based adversaries by attaching a classifier to the trained encoder of the target generative model, which can then be used to indirectly manipulate the latent representation. Our second attack directly uses the VAE loss function to generate a target reconstruction image from the adversarial example. Our third attack moves beyond relying on classification or the standard loss for the gradient and directly optimizes against differences in source and target latent representations. We also motivate why an attacker might be interested in deploying such techniques against a target generative network.

Asymmetric deep generative models

Generative models emerge as promising candidates for novel sequence-data driven approaches to protein design, and for the extraction of structural and functional information about proteins deeply hidden in rapidly growing sequence databases. Here we propose simple autoregressive models as highly accurate but computationally efficient generative sequence models. We show that they perform similarly to existing approaches based on Boltzmann machines or deep generative models, but at a substantially lower computational cost (by a factor between 102 and 103). Furthermore, the simple structure of our models has distinctive mathematical advantages, which translate into an improved applicability in sequence generation and evaluation. Within these models, we can easily estimate both the probability of a given sequence, and, using the model’s entropy, the size of the functional sequence space related to a specific protein family. In the example of response regulators, we find a huge number of ca. 1068 possible sequences, which nevertheless constitute only the astronomically small fraction 10−80 of all amino-acid sequences of the same length. These findings illustrate the potential and the difficulty in exploring sequence space via generative sequence models.

Quantile regression and conditional density estimation can reveal structure that is missed by mean regression, such as multimodality and skewness. In this article, we introduce a deep learning generative model for joint quantile estimation called Penalized Generative Quantile Regression (PGQR). Our approach simultaneously generates samples from many random quantile levels, allowing us to infer the conditional distribution of a response variable given a set of covariates. Our method employs a novel variability penalty to avoid the problem of vanishing variability, or memorization, in deep generative models. Further, we introduce a new family of partial monotonic neural networks (PMNN) to circumvent the problem of crossing quantile curves. A major benefit of PGQR is that it can be fit using a single optimization, thus, bypassing the need to repeatedly train the model at multiple quantile levels or use computationally expensive cross-validation to tune the penalty parameter. We illustrate the efficacy of PGQR through extensive simulation studies and analysis of real datasets. Code to implement our method is available at https://github.com/shijiew97/PGQR.

Image security in visual data is growing in importance due to the increasing use of digital images in many applications. This paper describes a novel image encrypting system using deep generative models with a self-attention mechanism to improve encrypting speed, decrypting speed, or both. Based on the CycleGAN models, the encrypting generator encrypts images while a dedicated decryption network decrypts them properly. The self-attention module improves the dispersibility of visual data to capture global dependencies among images at variable scales to achieve a highly secure image transformation with high reconstruction accuracy. Performance analysis has been done using two color image sets, brain MRI images to detect brain tumors, and images to detect skin cancer. Analysis indicates high security (Entropy = 7.9996, NPCR ≈ 99.99%), high reconstruction accuracy (SSIM ≈ 0.99, PSNR > 40 dB), and high resistance to differential (UACI ≈ 33.46) and occlusion attacks to images. This paper opens up novel avenues in applying deep models to visual cryptography schemes.

A new generalized framework for lung cancer detection and classification are introduced in this paper. Specifically, two types of deep models are presented. The first model is a generative model to capture the distribution of the important features in a set of small class-unbalanced collected CXR images. This generative model can be utilized to synthesize any number of CXR images for each class. For example, our generative model can generate images with tumors with different sizes and positions in the lung. Hence, the system can automatically convert the small unbalanced collected dataset to a larger balanced one. The second model is the ResNet50 that is trained using the large balanced dataset for cancer classification into benign and malignant. The proposed framework acquires 98.91% overall detection accuracy, 98.85% area under curve (AUC), 98.46% sensitivity, 97.72% precision, 97.89% F1 score. The classifier takes 1.2334 s on average to classify a single image using a machine with 13GB RAM.

The fusion of Federated Learning (FL) and deep generative models is transforming medical imaging by enabling privacy-preserving and data-efficient machine learning. Training large-scale deep models on radiological imaging data remains challenging due to data scarcity, heterogeneity, and strict privacy constraints that limit data sharing across institutions. FL addresses these challenges by employing collaborative model training that does not expose raw patient data, while generative models synthesize realistic medical images to alleviate scarcity and imbalance. The convergence of these two fields has given rise to federated generative models (FGMs), which extend Generative Adversarial Networks (GANs), Variational Autoencoders (VAE), and Diffusion models into distributed and privacy-preserving environments. FGMs have demonstrated significant potential in various medical imaging tasks, including data augmentation, image reconstruction, cross-modality conversion, and enhancing classification and segmentation models. Despite promising progress, the field remains in its early stages, facing open challenges in communication efficiency, scalability, data heterogeneity, and clinical reliability. This survey provides a comprehensive review of FGMs in medical imaging, analyzing their architectures, applications, and benchmark datasets, while highlighting key challenges and outlining future research directions to advance privacy-preserving generative modeling in healthcare.