- Research Article
19
- 10.1016/j.ins.2022.10.109
The manifold regularized SVDD for noisy label detection
- Oct 31, 2022
- Information Sciences
- Xiaoying Wu + 2 more +2
The manifold regularized SVDD for noisy label detection
Machine learning (ML) is an approach driven by data, and as research in machine learning progresses, the issue of noisy labels has garnered widespread attention. Noisy labels can significantly reduce the accuracy of supervised classification models, making it important to address this problem. Therefore, it is a very meaningful task to detect as many noisy labels as possible from the big data. In this study, a new method is proposed for detecting noisy labels in datasets. This method leverages a deep pre-trained network to extract a feature set from the image data first which can extract more accurate data features. Then, a membership degree based on tightness into the support vector data description (SVDD) model named TF-SVDD is introduced to detect noisy data in the dataset. In order to simulate different types of label noise more accurately, we first assumed that the labels of the datasets used were all correct, and in addition constructed the noise set using two method: the density peak noise set and the random noise set. Experimental results demonstrate that the TF-SVDD can effectively detect noisy label data, surpassing traditional support vector data description algorithms and other methods in terms of outlier detection accuracy, with the average accuracy mostly exceeding 50%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}, and even reaching 80%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}. Furthermore, one novel measure called ‘confidence’ is employed to rectify noisy labels in the data. Following the correction of noisy labels, the accuracy of image classification experiences a significant improvement, with the average promotion ratio mostly exceeding 10%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}, and reaching 30%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}.
Loading PDF
The manifold regularized SVDD for noisy label detection
The manifold regularized SVDD for noisy label detection
Support Vector Data Description by using hyper-ellipse instead of hyper-sphere
—Support Vector Data Description (SVDD) describes data by using a hyper-sphere. In this paper, we propose an extended SVDD (ESVDD) which describes data by using a hyper-ellipse. Clearly, ESVDD can describe data better than SVDD in the input space. Both hyper-sphere and hyper-ellipse are very rigid for data description. The kernel ESVDD which will be proposed in this paper and the kernel SVDD enhance the ability of ESVDD and SVDD for data description, respectively. The formulation of SVDD/ESVDD contains a penalty term C which controls the tradeoff between the volume of hyper-sphere/hyper-ellipse and the training errors. We show that the ESVDD can control this tradeoff better than the SVDD. Keywords- Kernel; ESVDD; Data description; Hyper-ellipse. I. I NTRODUCTION The one-class classification problem is an interesting field in pattern recognition and machine learning researches. In this kind of classification, we assume the one class of data as the target class and the rest of data are classified as the outlier. One-class classification is particularly significant in applications where only a single class of data objects is applicable and easy to obtain. Objects from the other classes could be too difficult or expensive to be made available. So we would only describe the target class to separate it from the outlier class. The SVDD is a kind of one-class classification method based on Support Vector Machine [1, 2] which proposed by Tax. It tries to construct a boundary around the target data by enclosing the target data within a minimum hyper-sphere. Inspired by the support vector machines (SVMs), the SVDD decision boundary is described by a few target objects, known as support vectors (SVs). A more flexible boundary can be obtained with the introduction of kernel functions, by which data are mapped into a high-dimensional feature space. The most commonly used kernel function is Gaussian kernel. This method has attracted many researchers from the various fields. For example Liu et al. applied the SVDD techniques for novelty detection as part of the validation on an Intelligent Flight Control System (IFCS) [3]. Ji et al. discussed the SVDD application in gene expression data clustering [4]. Yu et al. used SVDD for image categorization from internet images [5]. Recently, some efforts have been expended to improve the SVDD method. Guo et al. proposed a simple post-processing method which tries to modify the SVDD boundary in order to achieve a tight data description [6]. As another example Cho apply the orthogonal filtering as a preprocessing step is executed before SVDD modeling to remove the unwanted variation of data [7]. In this paper, we propose an extended SVDD (ESVDD) which describes data by using a hyper-ellipse instead of a hyper-sphere. Clearly, ESVDD can
Read moreDeep learning with noisy labels: Exploring techniques and remedies in medical image analysis.
Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis.
BadLabel: A Robust Perspective on Evaluating and Enhancing Label-Noise Learning.
Label-noise learning (LNL) aims to increase the model's generalization given training data with noisy labels. To facilitate practical LNL algorithms, researchers have proposed different label noise types, ranging from class-conditional to instance-dependent noises. In this paper, we introduce a novel label noise type called BadLabel, which can significantly degrade the performance of existing LNL algorithms by a large margin. BadLabel is crafted based on the label-flipping attack against standard classification, where specific samples are selected and their labels are flipped to other labels so that the loss values of clean and noisy labels become indistinguishable. To address the challenge posed by BadLabel, we further propose a robust LNL method that perturbs the labels in an adversarial manner at each epoch to make the loss values of clean and noisy labels again distinguishable. Once we select a small set of (mostly) clean labeled data, we can apply the techniques of semi-supervised learning to train the model accurately. Empirically, our experimental results demonstrate that existing LNL algorithms are vulnerable to the newly introduced BadLabel noise type, while our proposed robust LNL method can effectively improve the generalization performance of the model under various types of label noise. The new dataset of noisy labels and the source codes of robust LNL algorithms are available at https://github.com/zjfheart/BadLabels.
Read moreEvidentialMix: Learning with Combined Open-set and Closed-set Noisy Labels
The efficacy of deep learning depends on large-scale data sets that have been carefully curated with reliable data acquisition and annotation processes. However, acquiring such large-scale data sets with precise annotations is very expensive and time-consuming, and the cheap alternatives often yield data sets that have noisy labels. The field has addressed this problem by focusing on training models under two types of label noise: 1) closed-set noise, where some training samples are incorrectly annotated to a training label other than their known true class; and 2) open-set noise, where the training set includes samples that possess a true class that is (strictly) not contained in the set of known training labels. In this work, we study a new variant of the noisy label problem that combines the open-set and closed-set noisy labels, and introduce a benchmark evaluation to assess the performance of training algorithms under this setup. We argue that such problem is more general and better reflects the noisy label scenarios in practice. Furthermore, we propose a novel algorithm, called EvidentialMix, that addresses this problem and compare its performance with the state-of-the-art methods for both closed-set and open-set noise on the proposed benchmark. Our results show that our method produces superior classification results and better feature representations than previous state-of-the-art methods. The code is available at https:/github.com/ragavsachdeva/EvidentialMix.
Read moreSupport vector data description model to map urban extent from National Polar-Orbiting Partnership Satellite–Visible Infrared Imaging Radiometer Suite nightlights and normalized difference vegetation index
We explored a one-class classifier, the support vector data description (SVDD), using the Suomi National Polar-Orbiting Partnership Satellite–Visible Infrared Imaging Radiometer Suite and normalized difference vegetation index to map the urban extent, which was tested in the Beijing and Tianjin city group area. The urban edge-pixels were selected as training samples for SVDD based on a profile-based sampling method combining nighttime light value histograms. The results showed that the overall accuracy of SVDD was similar to the support vector machine (SVM) model. However, kappa coefficients of SVDD for highly developed cities were superior to SVM, as producer and user accuracies of SVDD were almost equal to show high agreement of urban and nonurban areas. For metropolitan areas, such as Beijing and Tianjin, the urban extent generated by SVDD is closer to the reference data. The R 2 between the quantity of SVDD-estimated urban extent and population, 0.86, was higher than that obtained from SVM, 0.76, indicating that the estimated urban extent from the SVDD is more efficient for understanding the population development. The SVDD was further applied for three other representative metropolitans in China: Shanghai, Guangzhou, and Shenzhen to validate the SVDD’s performance, and similar results were achieved. The success of the SVDD-based urban extent extraction improves our ability to map urban extent at regional and national scales.
Read moreRobust Medical Image Classification From Noisy Labeled Data With Global and Local Representation Guided Co-Training.
Deep neural networks have achieved remarkable success in a wide variety of natural image and medical image computing tasks. However, these achievements indispensably rely on accurately annotated training data. If encountering some noisy-labeled images, the network training procedure would suffer from difficulties, leading to a sub-optimal classifier. This problem is even more severe in the medical image analysis field, as the annotation quality of medical images heavily relies on the expertise and experience of annotators. In this paper, we propose a novel collaborative training paradigm with global and local representation learning for robust medical image classification from noisy-labeled data to combat the lack of high quality annotated medical data. Specifically, we employ the self-ensemble model with a noisy label filter to efficiently select the clean and noisy samples. Then, the clean samples are trained by a collaborative training strategy to eliminate the disturbance from imperfect labeled samples. Notably, we further design a novel global and local representation learning scheme to implicitly regularize the networks to utilize noisy samples in a self-supervised manner. We evaluated our proposed robust learning strategy on four public medical image classification datasets with three types of label noise, i.e., random noise, computer-generated label noise, and inter-observer variability noise. Our method outperforms other learning from noisy label methods and we also conducted extensive experiments to analyze each component of our method.
Read moreImpervious surface extraction in imbalanced datasets: integrating partial results and multi-temporal information in an iterative one-class classifier
ABSTRACTAccurate urban land use/cover monitoring is an essential step towards a sustainable future. As a key part of the classification process, the characteristics of reference data can significantly affect classification accuracy and quality of produced maps. However, ideal reference data is not always readily available; users frequently have difficulty generating sufficient reference data for some classes given time, cost, data availability, expertise level, or other limitations. This study aims at dealing with this lack of sufficiently balanced reference data by presenting a modified hybrid one-class support vector data description (SVDD) model. The underlying hypothesis is that the lack of balanced reference data can be overcome through integration of partially extracted results and multi-temporal spectral information. The partially extracted results, defined as highly accurate classified pixels identified in previous algorithmic iterations, allow a gradual increase of the available training data. Furthermore, the method incorporates a voting system that integrates multi-temporal images using the SVDD algorithm. We applied this hybrid method to binary impervious classification of multi-temporal Landsat Thematic Mapper imagery from Central New York with imbalanced reference data. The proposed hybrid one-class SVDD model achieved a 5–6% improvement in overall accuracy and 0.05–0.09 in kappa than the typical one-class SVDD benchmark. While the method was tested on a single site (albeit with an unusually high reference dataset size of >870,000 pixels) we feel confident to suggest implementation of our methodology in other sites over the traditional method. This is because our approach automatically reverts to the traditional method when voting is inconsistent or there is a limited number of highly accurately classified pixels to assist future iterations. Future work could explore the quantity and temporal specificity (e.g. benefits of specific months) of the multi-temporal image selection and/or test other one-class classifiers.
Read moreA new support vector data description method for machinery fault diagnosis with unbalanced datasets
A new support vector data description method for machinery fault diagnosis with unbalanced datasets
SVDD-based weighted oversampling technique for imbalanced and overlapped dataset learning
SVDD-based weighted oversampling technique for imbalanced and overlapped dataset learning
Deep Learning Network Intensification for Preventing Noisy-Labeled Samples for Remote Sensing Classification
The deep-learning-network performance depends on the accuracy of the training samples. The training samples are commonly labeled by human visual investigation or inherited from historical land-cover or land-use maps, which usually contain label noise, depending on subjective knowledge and the time of the historical map. Helping the network to distinguish noisy labels during the training process is a prerequisite for applying the model for training across time and locations. This study proposes an antinoise framework, the Weight Loss Network (WLN), to achieve this goal. The WLN contains three main parts: (1) the segmentation subnetwork, which any state-of-the-art segmentation network can replace; (2) the attention subnetwork (λ); and (3) the class-balance coefficient (α). Four types of label noise (an insufficient label, redundant label, missing label and incorrect label) were simulated by dilate and erode processing to test the network’s antinoise ability. The segmentation task was set to extract buildings from the Inria Aerial Image Labeling Dataset, which includes Austin, Chicago, Kitsap County, Western Tyrol and Vienna. The network’s performance was evaluated by comparing it with the original U-Net model by adding noisy training samples with different noise rates and noise levels. The result shows that the proposed antinoise framework (WLN) can maintain high accuracy, while the accuracy of the U-Net model dropped. Specifically, after adding 50% of dilated-label samples at noise level 3, the U-Net model’s accuracy dropped by 12.7% for OA, 20.7% for the Mean Intersection over Union (MIOU) and 13.8% for Kappa scores. By contrast, the accuracy of the WLN dropped by 0.2% for OA, 0.3% for the MIOU and 0.8% for Kappa scores. For eroded-label samples at the same level, the accuracy of the U-Net model dropped by 8.4% for OA, 24.2% for the MIOU and 43.3% for Kappa scores, while the accuracy of the WLN dropped by 4.5% for OA, 4.7% for the MIOU and 0.5% for Kappa scores. This result shows that the antinoise framework proposed in this paper can help current segmentation models to avoid the impact of noisy training labels and has the potential to be trained by a larger remote sensing image set regardless of the inner label error.
Read moreSVDD boundary and DPC clustering technique-based oversampling approach for handling imbalanced and overlapped data
SVDD boundary and DPC clustering technique-based oversampling approach for handling imbalanced and overlapped data
Improved Deep Support Vector Data Description Model Using Feature Patching for Industrial Anomaly Detection
In industrial contexts, anomaly detection is crucial for ensuring quality control and maintaining operational efficiency in manufacturing processes. Leveraging high-level features extracted from ImageNet-trained networks and the robust capabilities of the Deep Support Vector Data Description (SVDD) model for anomaly detection, this paper proposes an improved Deep SVDD model, termed Feature-Patching SVDD (FPSVDD), designed for unsupervised anomaly detection in industrial applications. This model integrates a feature-patching technique with the Deep SVDD framework. Features are extracted from a pre-trained backbone network on ImageNet, and each extracted feature is split into multiple small patches of appropriate size. This approach effectively captures both macro-structural information and fine-grained local information from the extracted features, enhancing the model’s sensitivity to anomalies. The feature patches are then aggregated and concatenated for further training with the Deep SVDD model. Experimental results on both the MvTec AD and CIFAR-10 datasets demonstrate that our model outperforms current mainstream approaches and provides significant improvements in anomaly detection performance, which is vital for industrial quality assurance and defect detection in real-time manufacturing scenarios.
Read moreDistilling Effective Supervision From Severe Label Noise
Collecting large-scale data with clean labels for supervised training of neural networks is practically challenging. Although noisy labels are usually cheap to acquire, existing methods suffer a lot from label noise. This paper targets at the challenge of robust training at high label noise regimes. The key insight to achieve this goal is to wisely leverage a small trusted set to estimate exemplar weights and pseudo labels for noisy data in order to reuse them for supervised training. We present a holistic framework to train deep neural networks in a way that is highly invulnerable to label noise. Our method sets the new state of the art on various types of label noise and achieves excellent performance on large-scale datasets with real-world label noise. For instance, on CIFAR100 with a 40% uniform noise ratio and only 10 trusted labeled data per class, our method achieves 80.2% classification accuracy, where the error rate is only 1.4% higher than a neural network trained without label noise. Moreover, increasing the noise ratio to 80%, our method still maintains a high accuracy of 75.5%, compared to the previous best accuracy 48.2%.
Read moreMulti-category classification with label noise by robust binary loss
Multi-category classification with label noise by robust binary loss