SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness

Contrastive learning of graphs under label noise

Automatic modulation classification (AMC) has a wide range of applications in both civilian and military fields, such as industrial Internet of Things (IIoT) security, communication spectrum management, and military electronic countermeasures. However, label mislabeling often occurs in practical scenarios, significantly impacting the performance and robustness of deep neural networks (DNNs). In this article, we propose a meta-learning guided label noise distillation method to enhance the robustness of AMC models against label noise or errors. Specifically, we propose a teacher-student heterogeneous network (TSHN) to discriminate and distill label noise. Following the notion that labels represent information, a teacher network, utilizing trusted few-shot labeled samples, reevaluates and corrects labels for a considerable number of untrusted labeled samples through meta-learning. By dividing and conquering untrusted labeled samples according to their confidence levels, the student network learns more effectively. Additionally, we propose a multiview signal (MVS) method to further enhance the performance of hard-to-classify categories with few-shot trusted labeled samples. Extensive experiments on the RadioML2016 and HisarMod2019.1 data sets demonstrate that our methods significantly improve accuracy and robustness in signal AMC across diverse label noise scenarios, including symmetric, asymmetric, and mixed label noise. For example, compared to the baseline convolutional neural network with the cross-entropy loss, our proposed TSHN achieves a remarkable 1.26% to 36.84% accuracy improvement under symmetric label noise and 0.12% to 38.59% accuracy improvement under mixed label noise. Moreover, TSHN exhibits greater robustness to varying label noise rates compared to existing methods.

In this paper we have used JoCoR, a fairly recent method for learning with label noise, that makes use of two neural networks with a joint loss function using an additional contrastive loss to increase the agreement between them. This method can be extended to more than two networks in a straightforward way. We have carried out experiments on the CIFAR-10 and CIFAR-100 datasets (contaminated by synthetic label noise) with this kind of extension using several contrastive losses. We have concluded that it makes a significant improvement if we use a third network, especially when we use Kullback-Leibler terms for all possible pairs of softmax outputs. Further extension also means some kind of improvement, but in the case of the CIFAR datasets, those were not so significant, maybe except the cases with lower ratio of label noise.

Federated learning (FL) is a privacy-preserving distributed learning paradigm that enables clients to jointly train a global model. In real-world FL implementations, client data could have label noise, and different clients could have vastly different label noise levels. Although there exist methods in centralized learning for tackling label noise, such methods do not perform well on heterogeneous label noise in FL settings, due to the typically smaller sizes of client datasets and data privacy requirements in FL. In this paper, we propose FedCorr, a general multi-stage framework to tackle heterogeneous label noise in FL, without making any assumptions on the noise models of local clients, while still maintaining client data privacy. In particular, (1) FedCorr dynamically identifies noisy clients by exploiting the dimensionalities of the model prediction subspaces independently measured on all clients, and then identifies incorrect labels on noisy clients based on persample losses. To deal with data heterogeneity and to increase training stability, we propose an adaptive local proximal regularization term that is based on estimated local noise levels. (2) We further finetune the global model on identified clean clients and correct the noisy labels for the remaining noisy clients after finetuning. (3) Finally, we apply the usual training on all clients to make full use of all local data. Experiments conducted on CIFAR-10/100 with federated synthetic label noise, and on a real-world noisy dataset, Clothing1M, demonstrate that FedCorr is robust to label noise and substantially outperforms the state-of-the-art methods at multiple noise levels.

The accuracy of deep learning classifiers trained using the cross entropy loss function suffers even when a fraction of training labels are wrong or input images are uninformative. Training images and labels for computational pathology are often noisy due to the difficulty in signal localization and certain disease classifications being subjective discretization of the underlying continuums of disease conditions. For training classifiers robust to input and label noise, we propose a modified and sample-specific version of generalized cross entropy loss. We take advantage of the bootstrapping properties in deep learning models in order to design loss functions that are aware of the difficulty of classifying individual samples, due to either the label noise or the lack of a strong visual signal. We carry out extensive experiments to validate our approach by comparing against the models trained using other loss functions. The superior performance of our methodology in all the experiments support the requirement of using robust loss functions for histology image classification.

Robustness to label noise is a critical property for weakly-supervised classifiers trained on massive datasets. Robustness to label noise is a critical property for weakly-supervised classifiers trained on massive datasets. In this paper, we first derive analytical bound for any given noise patterns. Based on the insights, we design TrustNet that first adversely learns the pattern of noise corruption, being it both symmetric or asymmetric, from a small set of trusted data. Then, TrustNet is trained via a robust loss function, which weights the given labels against the inferred labels from the learned noise pattern. The weight is adjusted based on model uncertainty across training epochs. We evaluate TrustNet on synthetic label noise for CIFAR-10 and CIFAR-100, and real-world data with label noise, i.e., Clothing1M. We compare against state-of-the-art methods demonstrating the strong robustness of TrustNet under a diverse set of noise patterns.

The deep model training procedure requires large-scale datasets of annotated data. Due to the difficulty of annotating a large number of samples, label noise caused by incorrect annotations is inevitable, resulting in low model performance and poor model generalization. To combat label noise, current methods usually select clean samples based on the small-loss criterion and use these samples for training. Due to some noisy samples similar to clean ones, these small-loss criterion-based methods are still affected by label noise. To address this issue, in this work, we propose Regroup Median Loss (RML) to reduce the probability of selecting noisy samples and correct losses of noisy samples. RML randomly selects samples with the same label as the training samples based on a new loss processing method. Then, we combine the stable mean loss and the robust median loss through a proposed regrouping strategy to obtain robust loss estimation for noisy samples. To further improve the model performance against label noise, we propose a new sample selection strategy and build a semi-supervised method based on RML. Compared to state-of-the-art methods, for both the traditionally trained and semi-supervised models, RML achieves a significant improvement on synthetic and complex real-world datasets. The source is at https://github.com/Feng-peng-Li/Regroup-Loss-Median-to-Combat-Label-Noise.

Model retraining and updating are essential processes in AI applications. However, during updates, there is a potential for performance degradation, in which the overall performance improves, but local performance deteriorates. This study proposes a backward compatibility metric that focuses on the compatibility of local predictive performance. The score of the proposed metric increases if the accuracy over the conditional distribution for each input is higher than before. Furthermore, we propose a model retraining method based on the proposed metric. Due to the use of the conditional distribution, our metric and retraining method are robust against label noises, while existing sample-based backward compatibility metrics are often affected by noise. We perform a theoretical analysis of our method and derive an upper bound for the generalization error. Numerical experiments demonstrate that our retraining method enhances compatibility while achieving equal or better trade-offs in overall performance compared to existing methods.

Availability of large annotated datasets in computer vision, speech understanding, or natural language processing is one of the main reasons for deep neural networks popularity. Unfortunately, such data can suffer from label noise, introduced by incorrectly labelled patterns. Since neural networks, as data-driven approaches, are strongly dependent on the quality of training data, the results of building deep neural structures with noisy examples can be unreliable. In this paper, we present preliminary experimental results on how two regularization techniques, namely dropout and batch normalization, influence vulnerability to incorrect labels. On popular MNIST and CIFAR-10 datasets we demonstrate that combination of these two approaches can be considered as a tool to improve network robustness to mislabelled training examples.

Collecting large-scale data with clean labels for supervised training of neural networks is practically challenging. Although noisy labels are usually cheap to acquire, existing methods suffer a lot from label noise. This paper targets at the challenge of robust training at high label noise regimes. The key insight to achieve this goal is to wisely leverage a small trusted set to estimate exemplar weights and pseudo labels for noisy data in order to reuse them for supervised training. We present a holistic framework to train deep neural networks in a way that is highly invulnerable to label noise. Our method sets the new state of the art on various types of label noise and achieves excellent performance on large-scale datasets with real-world label noise. For instance, on CIFAR100 with a $40\%$ uniform noise ratio and only 10 trusted labeled data per class, our method achieves $80.2{\pm}0.3\%$ classification accuracy, where the error rate is only $1.4\%$ higher than a neural network trained without label noise. Moreover, increasing the noise ratio to $80\%$, our method still maintains a high accuracy of $75.5{\pm}0.2\%$, compared to the previous best accuracy $48.2\%$. Source code available: https://github.com/google-research/google-research/tree/master/ieg

Reading digits from natural images is a challenging computer vision task central to a variety of emerging applications. However, the increased scalability and complexity of datasets or complex applications bring about inevitable label noise. Because the label noise in the scene digit recognition dataset is sequence-like, most existing methods cannot deal with label noise in scene digit recognition. We propose a novel sequence class-label noise filter called Confident Sequence Learning. Confident Sequence Learning consists of two critical parts: the sequence-like confidence segmentation algorithm and the Confident Learning method. The sequence-like confidence segmentation algorithms slice the sequence-like labels and the sequence-like predicted probabilities, reorganize them in the form of the independent stochastic process and the white noise process. The Confident Learning method estimates the joint distribution between observed labels and latent labels using the segmented labels and probabilities. The TRDG dataset and SVHN dataset experiments showed that the confident sequence learning could find label errors with high accuracy and significantly improve the VGG-Attn and the TPS-ResNet-Attn model’s performance in the presence of synthetic sequence class-label noise.

Multimodal medical image fusion enhances diagnostic interpretation by integrating anatomical and functional information into a single image. This work proposes an efficient hybrid framework, termed SVD-VGG Hybrid Fusion, unifying Singular Value Decomposition (SVD) for luminance decomposition and a lightweight VGG-based feature extractor for high-frequency enhancement. Synthetic Gaussian noise ([Formula: see text]) is added to MRI and Poisson noise to PET images to simulate representative acquisition degradations, while the SVD and VGG-based feature paths strengthen structural detail and functional contrast. Experiments were conducted on a single public brain dataset with image pairs resized to [Formula: see text] for fusion and [Formula: see text] for feature extraction. Quantitative evaluation using PSNR, SSIM, CC, and perceptual LPIPS indicates that the proposed method achieves consistent structural fidelity, perceptual quality, and color preservation while maintaining sub-second runtime per case. Although evaluated only on brain MRI-PET data and under synthetic noise conditions, the results suggest that the SVD-VGG hybrid design provides a noise-aware and color-preserving fusion strategy suitable for practical multimodal image fusion workflows.

Evaluation Of Recommendation System On A Movie Dataset

Deep neural networks have been found to be vulnerable to adversarial attacks. To address this challenge, this paper adopts the evolutionary multi-objective approach to the learning process, and manages to achieve a balance between learning accuracy and robustness against adversarial attacks. In addition, we propose to minimize the model complexity together with the adversarial training loss to defend against fast gradient signed method attacks. Our experimental results using two deep neural networks models, LeNet-5 and VGG-11, on MNIST and CIFAR-10 datasets, respectively, confirm that the proposed methods are effective in improving the robustness of deep learning models against adversarial attacks.

Mislabeled samples are ubiquitous in real-world datasets as rule-based or expert labeling is usually based on incorrect assumptions or subject to biased opinions. Neural networks can "memorize" these mislabeled samples and, as a result, exhibit poor generalization. This poses a critical issue in fault detection applications, where not only the training but also the validation datasets are prone to contain mislabeled samples. In this work, we propose a novel two-step framework for robust training with label noise. In the first step, we identify outliers (including the mislabeled samples) based on the update in the hypothesis space. In the second step, we propose different approaches to modifying the training data based on the identified outliers and a data augmentation technique. Contrary to previous approaches, we aim at finding a robust solution that is suitable for real-world applications, such as fault detection, where no clean, "noise-free" validation dataset is available. Under an approximate assumption about the upper limit of the label noise, we significantly improve the generalization ability of the model trained under massive label noise.