Truth Inference With a Deep Clustering-Based Aggregation Model

Noise correction to improve data and model quality for crowdsourcing

Contrastive learning of graphs under label noise

As crowdsourcing has cast a new solution to numerous tasks, truth inference, which deduces the accurate answer from massive noise labels (answers), has become quite an essential issue. However, existing proposals of truth inference only excel at limited tasks since they excessively depend on modeling either workers or labels with simple assumptions. In this paper, we propose BAT (Bipartite Attention-driven Truth) to flexibly infer the truth in various scenarios. The key behind BAT is to explore a comprehensive approach from the whole topology of crowdsourcing itself rather than any individual component. Specifically, BAT firstly characterizes the crowdsourcing as an attributed bipartite graph (ABG). Then it deploys a bipartite graph neural network (bi-GNN). The bi-GNN relies on a bipartite attention mechanism for exploiting the importance of different answers to compute the correct one. For verifying BAT, we compare BAT with other eight existing truth inference methods on real-world datasets from different domains (image, text, audio). The results show that BAT performs best on different crowdsourcing tasks.

This work examines the truth inference problem in a distributed crowdsourcing scenario. Labeling tasks are outsourced to workers associated with different platforms, and truth inference is to be performed without sharing the workers’ individual responses with other platforms or the central coordinator. The reliability of the labels may vary over different workers and tasks, and is characterized by a Gaussian mixture model. A federated truth inference (FTI) algorithm is proposed based on a distributed implementation of the block expectation-maximization (EM) algorithm. The messages sent by each local platform to the central coordinator contain aggregates of workers’ responses, instead of individual labels. The convergence of the FTI algorithm can be verified theoretically. A communication-efficient variant of the FTI scheme is also proposed by allowing the distributed platforms to perform multiple local EM computations before updating the global estimates at the central coordinator in each iteration. The effectiveness of our proposed schemes is demonstrated using both synthetic and real-world datasets.

Deep face recognition has achieved rapid development but still suffers from occlusions, illumination and pose variations, especially for face identification. The success of deep learning models in face recognition lies in large-scale high quality face data with accurate labels. However, in real-world applications, the collected data may be mixed with severe label noise, which significantly degrades the generalization ability of deep models. To alleviate the impact of label noise on face recognition, inspired by curriculum learning, we propose a self-paced deep learning model (SPDL) by introducing a negative \(l_1\)-norm regularizer for face recognition with label noise. During training, SPDL automatically evaluates the cleanness of samples in each batch and chooses cleaner samples for training while abandons the noisy samples. To demonstrate the effectiveness of SPDL, we use deep convolution neural network architectures for the task of robust face recognition. Experimental results show that our SPDL achieves superior performance on LFW, MegaFace and YTF when there are different levels of label noise.

Constrained clustering that integrates knowledge in the form of constraints in a clustering process has been studied for more than two decades. Popular clustering algorithms such as K-means, spectral clustering and recent deep clustering already have their constrained versions, but they usually lack of expressiveness in the form of constraints. In this paper we consider prior knowledge expressing relations between some data points and their assignments to clusters in propositional logic and we show how a deep clustering framework can be extended to integrate this knowledge. To achieve this, we define an expert loss based on the weighted models of the logical formulas; the weights depend on the soft assignment of points to clusters dynamically computed by the deep learner. This loss is integrated in the deep clustering method. We show how it can be computed efficiently using Weighted Model Counting and decomposition techniques. This method has the advantages of both integrating general knowledge and being independent of the neural architecture. Indeed, we have integrated the expert loss into two well-known deep clustering algorithms (IDEC and SCAN). Experiments have been conducted to compare our systems IDEC-LK and SCAN-LK to state-of-the-art methods for pairwise and triplet constraints in terms of computational cost, clustering quality and constraint satisfaction. We show that IDEC-LK can achieve comparable results with these systems, which are tailored for these specific constraints. To show the flexibility of our approach to learn from high-level domain constraints, we have integrated implication constraints, and a new constraint, called span-limited constraint that limits the number of clusters a set of points can belong to. Some experiments are also performed showing that constraints on some points can be extrapolated to other similar points.

Robustness to label noise is a critical property for weakly-supervised classifiers trained on massive datasets. Robustness to label noise is a critical property for weakly-supervised classifiers trained on massive datasets. In this paper, we first derive analytical bound for any given noise patterns. Based on the insights, we design TrustNet that first adversely learns the pattern of noise corruption, being it both symmetric or asymmetric, from a small set of trusted data. Then, TrustNet is trained via a robust loss function, which weights the given labels against the inferred labels from the learned noise pattern. The weight is adjusted based on model uncertainty across training epochs. We evaluate TrustNet on synthetic label noise for CIFAR-10 and CIFAR-100, and real-world data with label noise, i.e., Clothing1M. We compare against state-of-the-art methods demonstrating the strong robustness of TrustNet under a diverse set of noise patterns.

In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.

Graph-based semisupervised learning is of great importance in many effective learning systems, particularly in agnostic settings where no parametric information or other prior knowledge about the data distribution is available. It leverages the graph structure to propagate labels from labeled nodes to unlabeled ones. Two separate stages are usually involved: constructing an affinity graph and propagating labels on the graph for transductive inference. It is suboptimal to manage them independently, as the correlation between the affinity graph and the labels would not be fully exploited. In this article, we integrate these two stages into one unified framework by formulating the graph construction as a regularized function estimation problem, similar to label propagation. We then propose an alternating diffusion process to solve them alternately, which allows us to learn the graph and unknown labels in an iterative fashion. With the proposed framework, we can construct a dynamic graph adapted to the given and predicted labels iteratively, resulting in more accurate and robust label propagation performance. Extensive experiments on synthetic data and various real-world data have demonstrated the superiority of the proposed method compared with other state-of-the-art methods.

Deep cross-modal hashing has become an essential tool for supervised multimodal search. These models tend to be optimized with large, curated multimodal datasets, where most labels have been manually verified. Unfortunately, in many scenarios, such accurate labeling may not be avail-able. In contrast, datasets with low-quality annotations may be acquired, which inevitably introduce numerous mis-takes or label noise and therefore degrade the search per-formance. To address the challenge, we present a general robust cross-modal hashing framework to correlate distinct modalities and combat noisy labels simultaneously. More specifically, we propose a proxy-based contrastive (PC) loss to mitigate the gap between different modalities and train networks for different modalities Jointly with small-loss samples that are selected with the PC loss and a mu-tual quantization loss. The small-loss sample selection from such Joint loss can help choose confident examples to guide the model training, and the mutual quantization loss can maximize the agreement between different modalities and is beneficial to improve the effectiveness of sample selection. Experiments on three widely-used multimodal datasets show that our method significantly outperforms existing state-of-the-arts.

Recently a category of tracking methods based on "tracking-by-detection" is widely used in visual tracking problem. Most of these methods update the classifier online using the samples generated by the tracker to handle the appearance changes. However, the self-updating scheme makes these methods suffer from drifting problem because of the incorrect labels of weak classifiers in training samples. In this paper, we split the class labels into true labels and noise labels and model them by sparse representation. A novel dynamic classifier selection method, robust to noisy training data, is proposed. Moreover, we apply the proposed classifier selection algorithm to visual tracking by integrating a part based online boosting framework. We have evaluated our proposed method on 12 challenging sequences involving severe occlusions, significant illumination changes and large pose variations. Both the qualitative and quantitative evaluations demonstrate that our approach tracks objects accurately and robustly and outperforms state-of-the-art trackers.

With the rapid growing of crowdsourcing systems, class labels for supervised learning can be easily obtained from crowdsourcing platforms. To deal with the problem that labels obtained from crowds are usually noisy due to imperfect reliability of non-expert workers, we let multiple workers provide labels for the same object. Then, true labels of the labeled object are estimated through ground truth inference algorithms. The inferred integrated labels are expected to be of high quality. In this paper, we propose a novel ground truth inference algorithm based on EM algorithm, which not only infers the true labels of the instances but also simultaneously estimates the reliability of each worker and the difficulty of each instance. Experimental results on seven real-world crowdsourcing datasets show that our proposed algorithm outperforms eight state-of-the art algorithms.

In this paper, a fast algorithm for computing inbreeding coefficients in large populations in which a list of ancestors for each animal is required is modified for more quick calculation. With the current algorithm, an efficient procedure to make the list of the ancestors is used, which is based on grouping the ancestors of each animal chronologically by the use of longest ancestral path. A simulation study is carried out to compare the computational properties of the presented algorithm with those of the original and another typical algorithms. It is found that the presented algorithm becomes considerably fast relative to the original one, as the average number of generations and the number of animals increase. It is also shown that the current algorithm takes less time than another typical algorithm compared, when less than 19 generations are evaluated. The results obtained clearly indicate that the current algorithm requires much less computer time and less memory than the original and another typical algorithms, respectively.

Facilitated by the powerful feature extraction ability of neural networks, deep clustering has achieved great success in analyzing high-dimensional and complex real-world data. The performance of deep clustering methods is affected by various factors such as network structures and learning objectives. However, as pointed out in this survey, the essence of deep clustering lies in the incorporation and utilization of prior knowledge, which is largely ignored by existing works. From pioneering deep clustering methods based on data structure assumptions to recent contrastive clustering methods based on data augmentation invariances, the development of deep clustering intrinsically corresponds to the evolution of prior knowledge. In this survey, we provide a comprehensive review of deep clustering methods by categorizing them into six types of prior knowledge. We find that in general the prior innovation follows two trends, namely, i) from mining to constructing, and ii) from internal to external. Besides, we provide a benchmark on five widely-used datasets and analyze the performance of methods with diverse priors. By providing a novel prior knowledge perspective, we hope this survey could provide some novel insights and inspire future research in the deep clustering community.

BoAu: Malicious traffic detection with noise labels based on boundary augmentation