Noise correction to improve data and model quality for crowdsourcing

In crowdsourcing scenarios, we can obtain each instance’s multiple noisy labels set from different crowd workers and then use a ground truth inference algorithm to infer its integrated label. Despite the effectiveness of ground truth inference algorithms, a certain level of noise still remains in the integrated labels. To reduce the impact of noise, many noise correction algorithms have been proposed in recent years. To the best of our knowledge, however, nearly all existing noise correction algorithms only exploit each instance’s own multiple noisy label sets but ignore the multiple noisy label sets of its neighbors. Here neighbors refer to the nearest instances found in the feature space based on the distance metric learning. In this article, we propose neighborhood weighted voting-based noise correction (NWVNC). In NWVNC, we at first take advantage of the multiple noisy label sets of each instance’s neighbors (including itself) to estimate the probability that it belongs to its integrated label. Then, we use the estimated probability to identify and filter noise instances and thus obtain a clean set and a noise set. Finally, we train three heterogeneous classifiers on the clean set and correct the noise instances by the consensus voting of three trained classifiers. The experimental results on 34 simulated and two real-world crowdsourced datasets show that NWVNC significantly outperforms all the other state-of-the-art noise correction algorithms used for comparison.

Instance difficulty-based noise correction for crowdsourcing

Label noise in medical image classification datasets significantly hampers the training of supervised deep learning methods, undermining their generalizability. The test performance of a model tends to decrease as the label noise rate increases. Over recent years, several methods have been proposed to mitigate the impact of label noise in medical image classification and enhance the robustness of the model. Predominantly, these works have employed CNN-based architectures as the backbone of their classifiers for feature extraction. However, in recent years, Vision Transformer (ViT)-based backbones have replaced CNNs, demonstrating improved performance and a greater ability to learn more generalizable features, especially when the dataset is large. Nevertheless, no prior work has rigorously investigated how transformer-based backbones handle the impact of label noise in medical image classification. In this paper, we investigate the architectural robustness of ViT against label noise and compare it to that of CNNs. We use two medical image classification datasets-COVID-DU-Ex, and NCT-CRC-HE-100K-both corrupted by injecting label noise at various rates. Additionally, we show that pretraining is crucial for ensuring ViT's improved robustness against label noise in supervised training.

Objective. Learning to classify cardiac abnormalities requires large and high-quality labeled datasets, which is a challenge in medical applications. Small datasets from various sources are often aggregated to meet this requirement, resulting in a final dataset prone to label noise due to inter- and intra-observer variability and different expertise. It is well known that label noise can affect the performance and generalizability of the trained models. In this work, we explore the impact of label noise and self-learning label correction on the classification of cardiac abnormalities on large heterogeneous datasets of electrocardiogram (ECG) signals. Approach. A state-of-the-art self-learning multi-class label correction method for image classification is adapted to learn a multi-label classifier for electrocardiogram signals. We evaluated our performance using 5-fold cross-validation on the publicly available PhysioNet/Computing in Cardiology (CinC) 2021 Challenge data, with full and reduced sets of leads. Due to the unknown label noise in the testing set, we tested our approach on the MNIST dataset. We investigated the performance under different levels of structured label noise for both datasets. Main results. Under high levels of noise, the cross-validation results of self-learning label correction show an improvement of approximately 3% in the challenge score for the PhysioNet/CinC 2021 Challenge dataset and an improvement in accuracy of 5% and reduction of the expected calibration error of 0.03 for the MNIST dataset. We demonstrate that self-learning label correction can be used to effectively deal with the presence of unknown label noise, also when using a reduced number of ECG leads.

Mislabeled training data is a challenge to face in order to build a robust classifier whether it is an ensemble or not. This work handles the mislabeling problem by exploiting four different ensemble margins for identifying, then eliminating or correcting the mislabeled training data. Our approach is based on class noise ordering and relies on the margin values of misclassified data. The effectiveness of our ordering-based class noise removal and correction methods is demonstrated in performing image classification. A comparative analysis is conducted with respect to the majority vote filter, a reference ensemble-based class noise filter.

Worker similarity-based noise correction for crowdsourcing

Class label noise is a data-level difficulty associated with training objects with incorrectly assigned labels. This problem may originate from poorly documented historic data, errors during data generation process or mistakes made by human experts. Inclusion of such examples during the training process will mislead the classifier by presenting a falsified class distribution and consequently lead to degradation of models' generalization abilities. This phenomenon becomes even more troublesome in multi-class scenarios that may be affected by highly complex intra-class noise. Decomposition strategies with binary classifiers were proven to alleviate this difficulty by using simplified binary subtasks that are less affected by the noise. In this paper we propose to extend this approach by using the one-class classification decomposition. In this scenario each class has assigned individual one-class classifier that aims at capturing its distinguishing characteristics. This allows to create a robust data description and then apply a dedicated classifier combination in order to reconstruct the original multi-class task. We further extend this concept by using fuzzy one-class classifiers that allow to associate membership values with each training objects. This allows us to reduce the influence of uncertain and potentially noisy samples on the shape of learned decision boundary. Experimental study backed-up with statistical analysis shows that fuzzy one-class classifier decomposition offers an excellent robustness to noise in multi-class classification.

The presence of a substantial number of noisy instances in a given dataset may adversely affect the hypothesis learnt from that data. Removing noisy instances prior to the construction of a classifier has been shown to improve the classification ability of a learner on new data. This paper introduces a novel technique for identifying observations with class noise in a dataset using frequent itemsets. For the given dataset, each instance is assigned a NoiseFactor, indicating a relative likelihood that it contains class noise. A frequent itemset is a set of instances with common attribute values which contains at least as many instances as a user-defined minimum support threshold. Consequently, the set of frequent itemsets contains information related to the structure and dependence between the attributes. Each frequent itemset is assigned a class, based on the proportion of instances within the itemset from each class. Instances that are contained in itemsets that have a large proportion of instances from the other class are identified as noisy. The technique proposed in this paper is analyzed in numerous case studies using real-world software measurement datasets with either inherent or injected noise. A comparison is provided with two well-known techniques for the identification of class noise: Classification Filter and Ensemble Filter. The results demonstrate that this new algorithm is very effective at identifying instances with class noise.

Research in machine learning, data mining and related areas has produced a wide variety of algorithms for cost-sensitive (CS) classification, where instead of maximizing the classification accuracy, minimizing the misclassification cost becomes the objective. However, these methods assume that training sets do not contain significant noise, which is rarely the case in real-world environments. In this paper, we systematically study the impacts of class noise on CS learning, and propose a cost-guided class noise handling algorithm to identify noise for effective CS learning. We call it cost-guided iterative classification filter (CICF), because it seamlessly integrates costs and an existing classification filter (C. Brodley and M. Friedl, 1999) for noise identification. Instead of putting equal weights to handle noise in all classes in existing efforts, CICF puts more emphasis on expensive classes, which makes it especially successful in dealing with datasets with a large cost-ratio. Experimental results and comparative studies from real-world datasets indicate that the existence of noise may seriously corrupt the performance of CS classifiers, and by adopting the proposed CICF algorithm, we can significantly reduce the misclassification cost of a CS classifier in noisy environments.

Traditional truth inference algorithms take multiple source labels as input and infer true labels for objects. Besides source labels, object features have been introduced in inference algorithms to achieve superior performance. A typical algorithm such as learning from crowds learns a classification model with the guide of inferred true labels where true labels are inferred from source labels. However, the main shortcoming exists in current algorithms and limits their inference performance: label noise. Since source labels from real-world data are noisy, a classifier is likely to be misguided to learn an imprecise decision boundary. In this paper, we propose a deep clustering-based aggregation model (DCAM) to overcome the shortcoming. DCAM introduces clustering for object features to form fine-grained clusters, where objects in the same cluster are supposed to have similar labels. DCAM exploits a cluster label distribution to represent the labeling information of all objects in the corresponding cluster to overcome the problem of label noise. To implement the idea of clustering-based truth inference, DCAM integrates source label generation and deep clustering in a unified framework by utilizing maximum a posteriori (MAP) estimation. Therefore, the proposed model is a novel approach for truth inference with object features. Experimental results on eight real-world inference tasks show that DCAM has a significant improvement of inference accuracy over the state-of-the-art truth inference algorithms. We further discuss the effect of cluster numbers, the quality of clustering, and illustrate the learned embeddings to support the effectiveness of DCAM.

Class noise detection based on software metrics and ROC curves

As crowdsourcing has cast a new solution to numerous tasks, truth inference, which deduces the accurate answer from massive noise labels (answers), has become quite an essential issue. However, existing proposals of truth inference only excel at limited tasks since they excessively depend on modeling either workers or labels with simple assumptions. In this paper, we propose BAT (Bipartite Attention-driven Truth) to flexibly infer the truth in various scenarios. The key behind BAT is to explore a comprehensive approach from the whole topology of crowdsourcing itself rather than any individual component. Specifically, BAT firstly characterizes the crowdsourcing as an attributed bipartite graph (ABG). Then it deploys a bipartite graph neural network (bi-GNN). The bi-GNN relies on a bipartite attention mechanism for exploiting the importance of different answers to compute the correct one. For verifying BAT, we compare BAT with other eight existing truth inference methods on real-world datasets from different domains (image, text, audio). The results show that BAT performs best on different crowdsourcing tasks.

Supervised classification systems used for land cover mapping require accurate reference databases. These reference data come generally from different sources such as field measurements, thematic maps, or aerial photographs. Due to misregistration, update delay, or land cover complexity, they may contain class label noise, i.e., a wrong label assignment. This study aims at evaluating the impact of mislabeled training data on classification performances for land cover mapping. Particularly, it addresses the random and systematic label noise problem for the classification of high resolution satellite image time series. Experiments are carried out on synthetic and real datasets with two traditional classifiers: Support Vector Machines (SVM) and Random Forests (RF). A synthetic dataset has been designed for this study, simulating vegetation profiles over one year. The real dataset is composed of Landsat-8 and SPOT-4 images acquired during one year in the south of France. The results show that both classifiers are little influenced for low random noise levels up to 25%–30%, but their performances drop down for higher noise levels. Different classification configurations are tested by increasing the number of classes, using different input feature vectors, and changing the number of training instances. Algorithm complexities are also analyzed. The RF classifier achieves high robustness to random and systematic label noise for all the tested configurations; whereas the SVM classifier is more sensitive to the kernel choice and to the input feature vectors. Finally, this work reveals that the cross-validation procedure is impacted by the presence of class label noise.

The problem of class noisy instances is omnipresent in different classification problems. However, most of research focuses on noise handling in binary classification problems and adaptations to multiclass learning. This paper aims to contextualize noise labels in the context of non-binary classification problems, including multiclass, multilabel, multitask, multi-instance ordinal and data stream classification. Practical considerations for analyzing noise under these classification problems, as well as trends, open-ended problems and future research directions are analyzed. We believe this paper could help expand research on class noise handling and help practitioners to better identify the particular aspects of noise in challenging classification scenarios.

The linguistic artificial intelligence teaching model can be assisted by the intelligent speech recognition model. The traditional speech recognition algorithm has certain problems, so it cannot effectively eliminate speech noise. Based on the advantages of the linguistics teaching model, this article combines the linguistics teaching model and the artificial intelligence model to build an artificial intelligence assisted teaching model that can be used for classroom teaching. Moreover, this study improves the traditional algorithm and constructs an artificial intelligence linguistics teaching model based on the improved algorithm. The filtering part of noise includes preliminary filtering of speech signals based on the short-term energy detection method, and further detection and recognition of preliminary filtering speech signals based on the artificial intelligence model detection method. After these two steps of filtering and recognition, the voice file is sent to the client for processing and control. In addition, this study set up a control experiment to analyze the performance of the model. The research results show that the algorithm in this paper has a certain effect.