Label-Noise Robust Deep Generative Model for Semi-Supervised Learning

Automated design synthesis has the potential to revolutionize the modern engineering design process and improve access to highly optimized and customized products across countless industries. Successfully adapting generative machine learning to design engineering may enable such automated design synthesis and is a research subject of great importance. We present a review and analysis of deep generative machine learning models in engineering design. Deep generative models (DGMs) typically leverage deep networks to learn from an input dataset and synthesize new designs. Recently, DGMs such as feedforward neural networks (NNs), generative adversarial networks (GANs), variational autoencoders (VAEs), and certain deep reinforcement learning (DRL) frameworks have shown promising results in design applications like structural optimization, materials design, and shape synthesis. The prevalence of DGMs in engineering design has skyrocketed since 2016. Anticipating the continued growth, we conduct a review of recent advances to benefit researchers interested in DGMs for design. We structure our review as an exposition of the algorithms, datasets, representation methods, and applications commonly used in the current literature. In particular, we discuss key works that have introduced new techniques and methods in DGMs, successfully applied DGMs to a design-related domain, or directly supported the development of DGMs through datasets or auxiliary methods. We further identify key challenges and limitations currently seen in DGMs across design fields, such as design creativity, handling constraints and objectives, and modeling both form and functional performance simultaneously. In our discussion, we identify possible solution pathways as key areas on which to target the future work.

Localization in reverberant environments remains an open challenge. Recently, supervised learning approaches have demonstrated very promising results in addressing reverberation. However, even with large data volumes, the number of labels available for supervised learning in such environments is usually small. We propose to address this issue with a semi-supervised learning (SSL) approach, based on deep generative modeling. Our chosen deep generative model, the variational autoencoder (VAE), is trained to generate the phase of relative transfer functions (RTFs) between microphones. In parallel, a direction of arrival (DOA) classifier network based on RTF-phase is also trained. The joint generative and discriminative model, deemed VAE-SSL, is trained using labeled and unlabeled RTF-phase sequences. In learning to generate and classify the sequences, the VAE-SSL extracts the physical causes of the RTF-phase (i.e., source location) from distracting signal characteristics such as noise and speech activity. This facilitates effective end-to-end operation of the VAE-SSL, which requires minimal preprocessing of RTF-phase. VAE-SSL is compared with two signal processing-based approaches, steered response power with phase transform (SRP-PHAT) and MUltiple SIgnal Classification (MUSIC), as well as fully supervised CNNs. The approaches are compared using data from two real acoustic environments - one of which was recently obtained at Technical University of Denmark specifically for our study. We find that VAE-SSL can outperform the conventional approaches and the CNN in label-limited scenarios. Further, the trained VAE-SSL system can generate new RTF-phase samples which capture the physics of the acoustic environment. Thus, the generative modeling in VAE-SSL provides a means of interpreting the learned representations. To the best of our knowledge, this paper presents the first approach to modeling the physics of acoustic propagation using deep generative modeling.

Medical image segmentation tasks hitherto have achieved excellent progresses with large-scale datasets, which empowers us to train potent deep convolutional neural networks (DCNNs). However, labeling such large-scale datasets is laborious and error-prone, which leads the noisy (or incorrect) labels to be an ubiquitous problem in the real-world scenarios. In addition, data collected from different sites usually exhibit significant data distribution shift (or domain shift). As a result, noisy label and domain shift become two common problems in medical imaging application scenarios, especially in medical image segmentation, which degrade the performance of deep learning models significantly. In this paper, we identify a novel problem hidden in medical image segmentation, which is unsupervised domain adaptation on noisy labeled data, and propose a novel algorithm named "Self-Cleansing Unsupervised Domain Adaptation" (S-CDUA) to address such issue. S-CUDA sets up a realistic scenario to solve the above problems simultaneously where training data (i.e., source domain) not only shows domain shift w.r.t. unsupervised test data (i.e., target domain) but also contains noisy labels. The key idea of S-CUDA is to learn noise-excluding and domain invariant knowledge from noisy supervised data, which will be applied on the highly corrupted data for label cleansing and further data-recycling, as well as on the test data with domain shift for supervised propagation. To this end, we propose a novel framework leveraging noisy-label learning and domain adaptation techniques to cleanse the noisy labels and learn from trustable clean samples, thus enabling robust adaptation and prediction on the target domain. Specifically, we train two peer adversarial networks to identify high-confidence clean data and exchange them in companions to eliminate the error accumulation problem and narrow the domain gap simultaneously. In the meantime, the high-confidence noisy data are detected and cleansed in order to reuse the contaminated training data. Therefore, our proposed method can not only cleanse the noisy labels in the training set but also take full advantage of the existing noisy data to update the parameters of the network. For evaluation, we conduct experiments on two popular datasets (REFUGE and Drishti-GS) for optic disc (OD) and optic cup (OC) segmentation, and on another public multi-vendor dataset for spinal cord gray matter (SCGM) segmentation. Experimental results show that our proposed method can cleanse noisy labels efficiently and obtain a model with better generalization performance at the same time, which outperforms previous state-of-the-art methods by large margin. Our code can be found at https://github.com/zzdxjtu/S-cuda.

Asymmetric deep generative models

In medical image analysis, the presence of noisy labels and imbalanced data poses significant challenges to the performance of deep learning models, particularly in critical diagnostic tasks. To address this issue, we propose DIER‐Net, a learning with noisy label framework designed to handle noisy labels in imbalanced medical datasets. Our approach introduces a debiased sample selection technique that effectively filters out noisy labels while preserving important minority class samples. Additionally, we employ intrinsic and extrinsic regularization strategies to enhance the model's robustness by leveraging both clean and noisy data. Our method is evaluated on two widely used medical image datasets: the ISIC melanoma classification and Kaggle histopathologic lymph node classification. The experimental results demonstrate that DIER‐Net consistently outperforms existing state‐of‐the‐art methods, particularly in settings with high levels of label noise, offering a robust solution for real‐world clinical applications where noisy and imbalanced data are common. DIER‐Net provides an effective approach to enhance the reliability of AI systems in medical imaging, contributing to more accurate and trustworthy diagnostic outcomes.

Automatic building extraction from aerial or satellite images is a dense pixel prediction task for many applications. It demands a large number of clean label data to train a deep neural network for building extraction. But it is labor expensive to collect such pixel-wise annotated data manually. Fortunately, the building footprint data of geographic information system (GIS) maps provide a cheap way of generating building label data, but these labels are imperfect due to misalignment between the GIS maps and images. In this letter, we consider the task of learning a deep neural network to label images pixel-wise from such noisy label data for building extraction. To this end, we propose a general label noise-adaptive (NA) neural network framework consisting of a base network followed by an additional probability transition modular (PTM) which is introduced to capture the relationship between the true label and the noisy label. The parameters of the PTM can be estimated as part of the training process of the whole network by the off-the-shelf backpropagation algorithm. We conduct experiments on real-world data set to demonstrate that our proposed PTM can better handle noisy labels and improve the performance of convolutional neural networks (CNNs) trained on the noisy label data generated by GIS maps for building extraction. The experimental results indicate that being armed with our proposed PTM for fully CNN, it provides a promising solution to reduce manual annotation effort for the labor-expensive object extraction tasks from remote sensing images.

Detecting frauds in computing platforms involves identifying malicious user activity sessions. Recently, deep learning models have been employed to design fraud detection approaches. Effective training of these deep learning models requires a large amount of well-annotated sessions. However, due to the cost of expert annotation, many organizations rely on heuristics to perform automated annotation, which leads to the noisy label learning problem. It is well known that the performance of deep learning models can easily degrade because of noisy or inaccurate labels. To tackle this challenge, we propose a supervised Contrastive Learning based Fraud Detection (CLFD) framework, which is designed to operate in the noisy label setting. CLFD employs an effective label corrector for correcting noisy labels and which is specifically designed for the fraud detection task. Then, by employing the corrected labels, it trains a fraud detector through supervised contrastive learning, and derives separable representations. We empirically evaluate our CLFD framework and other state-of-the-art baselines on benchmark datasets. Our CLFD framework demonstrates superior performance over state-of-the-art baselines.

Graph-based semi-supervised learning plays an important role in large scale image classification tasks. However, the problem becomes very challenging in the presence of noisy labels and outliers. Moreover, traditional robust semi-supervised learning solutions suffers from prohibitive computational burdens thus cannot be computed for streaming data. Motivated by that, we present a novel unified framework robust structure-aware semi-supervised learning called Unified RSSL (URSSL) for batch processing and recursive processing robust to both outliers and noisy labels. Particularly, URSSL applies joint semi-supervised dimensionality reduction with robust estimators and network sparse regularization simultaneously on the graph Laplacian matrix iteratively to preserve the intrinsic graph structure and ensure robustness to the compound noise. First, in order to relieve the influence from outliers, a novel semi-supervised robust dimensionality reduction is applied relying on robust estimators to suppress outliers. Meanwhile, to tackle noisy labels, the denoised graph similarity information is encoded into the network regularization. Moreover, by identifying strong relevance of dimensionality reduction and network regularization in the context of robust semi-supervised learning (RSSL), a two-step alternative optimization is derived to compute optimal solutions with guaranteed convergence. We further derive our framework to adapt to large scale semi-supervised learning particularly suitable for large scale image classification and demonstrate the model robustness under different adversarial attacks. For recursive processing, we rely on reparameterization to transform the formulation to unlock the challenging problem of robust streaming-based semi-supervised learning. Last but not least, we extend our solution into distributed solutions to resolve the challenging issue of distributed robust semi-supervised learning when images are captured by multiple cameras at different locations. Extensive experimental results demonstrate the promising performance of this framework when applied to multiple benchmark datasets with respect to state-of-the-art approaches for important applications in the areas of image classification and spam data analysis.

Early warning for learning performance requires to identify the maximum number of at-risk students as early as possible within a semester. However, educational data often suffer from the issue of data imbalanced, making it challenging to simultaneously achieve both high precision (accurate identification) and high recall (comprehensive coverage) in at-risk student detection. Deep generative models and oversampling models are effective methods to solve data imbalanced issues, which can improve classification performance. This paper proposes a method that combines the advantages of deep generative models and oversampling models to build a blending model for dealing with the imbalanced education data, which can effectively improve the precision, recall, f1-score and auc for online learning early warning. First, we compare baseline models to select the best classifier, then choose the highest-precision deep generative model and the highest-recall oversampling model to construct blending models, which are shown to improve early warning prediction metrics. Finally, interpretable models are used to analyze differences in at-risk student prediction between the blending model, deep generative model, and oversampling model. The proposed models are validated on both extremely imbalanced datasets and new semester datasets. Results show that: (1) Compared to the baseline model, both the base learners build by deep generative model and oversampling model can improve the evaluation metrics of the model, the deep generative base learners achieve higher precision than oversampling model, the oversampling base learners achieve higher recall than the deep generative base learners. (2) The blending model composed of deep generative base learner and oversampling base learner can further improve the f1-score and auc based on their individual strengths, the proposed blending model can also conduct effective early warning 3 units earlier than baseline models. (3) Compared to its base learners, blending model G-B-Blending changed the key variables for prediction, and the at-risk students identified by the blending model come from the union set of at-risk students identified by GAN+GB and B-SMOTE+GB individually. (4) The blending model proposed in this paper achieves better prediction results than the baseline on both extremely imbalanced datasets and new semesters datasets, it can identify more at-risk students more accurately at earlier units, allowing teachers to save more energy and time for teaching interventions. This research provides significant insights for dealing with imbalanced datasets by blending with deep generative model and oversampling model in education.

Noisy labels seriously degrade the performance of the deep learning models. Especially on segmentation tasks, labels are represented at pixel level, and therefore, it is easier to generate them as noisy labels. In this letter, we aim to cleanse the dataset which contains noisy labels using Kullback–Leibler (KL) divergence algorithm and noise robust loss function. Here, we regard the whole dataset as noisy labels and separate noisy labels into two different parts. One is the strong noisy labels where there are the most noisy labels, which is referred to as incorrect labels, and the other is The weak noisy labels where pixels are incorrect in some parts of the image. The KL algorithm is more efficient in removing the labels which contain the most noise. At the same time, we use noise robust model to remove the labels where pixels are inaccurate only in small parts of the image. Noise robust loss function is robust to noisy labels, and thus, we consider noisy labels when intersection over union (IoU) is lower than a constant threshold despite using noise robust loss function. The effectiveness of our proposed method is assessed using Kaggle’s TGS Salt Identification Challenge dataset. We have demonstrated that using our method, segmentation performance is increased without heavy noisy labels.

Generative models emerge as promising candidates for novel sequence-data driven approaches to protein design, and for the extraction of structural and functional information about proteins deeply hidden in rapidly growing sequence databases. Here we propose simple autoregressive models as highly accurate but computationally efficient generative sequence models. We show that they perform similarly to existing approaches based on Boltzmann machines or deep generative models, but at a substantially lower computational cost (by a factor between 102 and 103). Furthermore, the simple structure of our models has distinctive mathematical advantages, which translate into an improved applicability in sequence generation and evaluation. Within these models, we can easily estimate both the probability of a given sequence, and, using the model’s entropy, the size of the functional sequence space related to a specific protein family. In the example of response regulators, we find a huge number of ca. 1068 possible sequences, which nevertheless constitute only the astronomically small fraction 10−80 of all amino-acid sequences of the same length. These findings illustrate the potential and the difficulty in exploring sequence space via generative sequence models.

Quantile regression and conditional density estimation can reveal structure that is missed by mean regression, such as multimodality and skewness. In this article, we introduce a deep learning generative model for joint quantile estimation called Penalized Generative Quantile Regression (PGQR). Our approach simultaneously generates samples from many random quantile levels, allowing us to infer the conditional distribution of a response variable given a set of covariates. Our method employs a novel variability penalty to avoid the problem of vanishing variability, or memorization, in deep generative models. Further, we introduce a new family of partial monotonic neural networks (PMNN) to circumvent the problem of crossing quantile curves. A major benefit of PGQR is that it can be fit using a single optimization, thus, bypassing the need to repeatedly train the model at multiple quantile levels or use computationally expensive cross-validation to tune the penalty parameter. We illustrate the efficacy of PGQR through extensive simulation studies and analysis of real datasets. Code to implement our method is available at https://github.com/shijiew97/PGQR.

Inverse design allows the design of molecules with desirable properties using property optimization. Deep generative models have recently been applied to tackle inverse design, as they possess the ability to optimize molecular properties directly through structure modification using gradients. While the ability to carry out direct property optimizations is promising, the use of generative deep learning models to solve practical problems requires large amounts of data and is very time-consuming. In this work, we propose STONED – a simple and efficient algorithm to perform interpolation and exploration in the chemical space, comparable to deep generative models. STONED bypasses the need for large amounts of data and training times by using string modifications in the SELFIES molecular representation. We achieve comparable performance on typical benchmarks without any training. We demonstrate applications in high-throughput virtual screening for the design of drugs, photovoltaics, and the construction of chemical paths, allowing for both property and structure-based interpolation in the chemical space. We anticipate our results to be a stepping stone for developing more sophisticated inverse design models and benchmarking tools, ultimately helping generative models achieve wide adoption.

Droughts pose significant challenges for accurate monitoring due to their complex spatiotemporal characteristics. Data-driven machine learning (ML) models have shown promise in detecting extreme events when enough well-annotated data is available. However, droughts do not have a unique and precise definition, which leads to noise in human-annotated events and presents an imperfect learning scenario for deep learning models. This article introduces a 3-D convolutional neural network (CNN) designed to address the complex task of drought detection, considering spatiotemporal dependencies and learning with noisy and inaccurate labels. Motivated by the shortcomings of traditional drought indices, we leverage supervised learning with labeled events from multiple sources, capturing the shared conceptual space among diverse definitions of drought. In addition, we employ several strategies to mitigate the negative effect of noisy labels (NLs) during training, including a novel label correction (LC) method that relies on model outputs, enhancing the robustness and performance of the detection model. Our model significantly outperforms state-of-the-art drought indices when detecting events in Europe between 2003 and 2015, achieving an AUROC of 72.28%, an AUPRC of 7.67%, and an ECE of 16.20%. When applying the proposed LC method, these performances improve by +5%, +15%, and +59%, respectively. Both the proposed model and the robust learning methodology aim to advance drought detection by providing a comprehensive solution to label noise and conceptual variability.

Synthetic data offers a promising avenue for addressing privacy, scarcity, and fairness challenges in healthcare datasets. However, there is limited evaluation of how different generation methods balance fidelity, utility, and fairness, particularly for underrepresented subgroups. This study addresses this gap by comparing representative generative modelling techniques, both probabilistic and deep approaches, that are popular in the research literature. We empirically evaluate BayesBoost, CTGAN, TVAE, CopulaGAN, and DECAF on two healthcare datasets containing numerical, binary, and categorical features. Each model’s performance is assessed along three axes: data fidelity, machine learning utility, and fairness, using Accuracy Parity, Equalised Odds, and Predictive Rate Parity. Results show that BayesBoost consistently achieved superior fidelity, utility, and fairness preservation, particularly when paired with Random Forest classifiers, achieving around 60 – 63 % higher downstream utility than GAN-based deep generative baselines (e.g., Random Forest accuracy up to 0.88 with BayesBoost versus 0.54 to − 0.55 for GAN-based methods). Deep generative models, while effective in capturing complex structures, often degraded fairness, especially for underrepresented groups, with equalised odds deviating by over 100 % from the ideal parity value of 1.0 in some settings. The Variational Autoencoder outperformed other deep generative models in fairness preservation, especially for equalised odds, although with some reduction in fidelity and utility. Overall, these findings suggest that synthetic data generation for healthcare must move beyond fidelity evaluations to explicitly assess fairness and subgroup impacts, with probabilistic models such as BayesBoost showing strong potential for ethical deployment, while deep generative models require further adaptation for fairness-sensitive applications.