Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning

Supervised deep learning has achieved tremendous success in many computer vision tasks, which however is prone to overfit noisy labels. To mitigate the undesirable influence of noisy labels, robust loss functions offer a feasible approach to achieve noise-tolerant learning. In this work, we systematically study the problem of noise-tolerant learning with respect to both classification and regression. Specifically, we propose a new class of loss function, namely asymmetric loss functions (ALFs), which are tailored to satisfy the Bayes-optimal condition and thus are robust to noisy labels. For classification, we investigate general theoretical properties of ALFs on categorical noisy labels, and introduce the asymmetry ratio to measure the asymmetry of a loss function. We extend several commonly-used loss functions, and establish the necessary and sufficient conditions to make them asymmetric and thus noise-tolerant. For regression, we extend the concept of noise-tolerant learning for image restoration with continuous noisy labels. We theoretically prove that lp loss ( ) is noise-tolerant for targets with the additive white Gaussian noise. For targets with general noise, we introduce two losses as surrogates of l0 loss that seeks the mode when clean pixels keep dominant. Experimental results demonstrate that ALFs can achieve better or comparative performance compared with the state-of-the-arts.

The convergence of Stochastic Gradient Descent (SGD) using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal that the convergence rate is O(1/T) for SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence.

In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.

On the convergence of gradient descent for robust functional linear regression

Noisy labels seriously degrade the performance of the deep learning models. Especially on segmentation tasks, labels are represented at pixel level, and therefore, it is easier to generate them as noisy labels. In this letter, we aim to cleanse the dataset which contains noisy labels using Kullback–Leibler (KL) divergence algorithm and noise robust loss function. Here, we regard the whole dataset as noisy labels and separate noisy labels into two different parts. One is the strong noisy labels where there are the most noisy labels, which is referred to as incorrect labels, and the other is The weak noisy labels where pixels are incorrect in some parts of the image. The KL algorithm is more efficient in removing the labels which contain the most noise. At the same time, we use noise robust model to remove the labels where pixels are inaccurate only in small parts of the image. Noise robust loss function is robust to noisy labels, and thus, we consider noisy labels when intersection over union (IoU) is lower than a constant threshold despite using noise robust loss function. The effectiveness of our proposed method is assessed using Kaggle’s TGS Salt Identification Challenge dataset. We have demonstrated that using our method, segmentation performance is increased without heavy noisy labels.

Deep-learning has proved in recent years to be a powerful tool for image analysis and is now widely used to segment both 2D and 3D medical images. Deep-learning segmentation frameworks rely not only on the choice of network architecture but also on the choice of loss function. When the segmentation process targets rare observations, a severe class imbalance is likely to occur between candidate labels, thus resulting in sub-optimal performance. In order to mitigate this issue, strategies such as the weighted cross-entropy function, the sensitivity function or the Dice loss function, have been proposed. In this work, we investigate the behavior of these loss functions and their sensitivity to learning rate tuning in the presence of different rates of label imbalance across 2D and 3D segmentation tasks. We also propose to use the class re-balancing properties of the Generalized Dice overlap, a known metric for segmentation assessment, as a robust and accurate deep-learning loss function for unbalanced tasks.

Chest radiography is the most common radiology examination for thoracic disease diagnosis, such as pneumonia. A tremendous number of chest X-rays prompt data-driven deep learning models in constructing computer-aided diagnosis systems for thoracic diseases. However, in realistic radiology practice, a deep learning-based model often suffers from performance degradation when trained on data with noisy labels possibly caused by different types of annotation biases. To this end, we present a novel stochastic neural ensemble learning (SNEL) framework for robust thoracic disease diagnosis using chest X-rays. The core idea of our method is to learn from noisy labels by constructing model ensembles and designing noise-robust loss functions. Specifically, we propose a fast neural ensemble method that collects parameters simultaneously across model instances and along optimization trajectories. Moreover, we propose a loss function that both optimizes a robust measure and characterizes a diversity measure of ensembles. We evaluated our proposed SNEL method on three publicly available hospital-scale chest X-ray datasets. The experimental results indicate that our method outperforms competing methods and demonstrate the effectiveness and robustness of our method in learning from noisy labels. Our code is available at https://github.com/hywang01/SNEL.

In this paper, we proposed a new loss function to train the conditional generative adversarial network (CGAN). CGANs use a condition to generate images. Adding a class condition to the discriminator helps improve the training process of GANs and has been widely used for CGAN. Therefore, many loss functions have been proposed for the discriminator to add class conditions to it. Most of them have the problem of adjusting weights. This paper presents a simple yet new loss function that uses class labels, but no adjusting is required. This loss function is based on WGAN-GP loss, and the discriminator has outputs of the same order (the reason for no adjusting). More specifically, the discriminator has K (the number of classes) outputs, and each of them is used to compute the distance between fake and real samples of one class. Another loss to enable the discriminator to classify is also proposed by applying SoftMax to the outputs and adding cross-entropy to our first loss. The proposed loss functions are applied to a CGAN for image-to-image translation (here stain transformation for pathological images). The performances of proposed losses with some state-of-the-art losses are compared using Histogram Intersection Score between generated images and target images. The accuracy of a classifier is also computed to measure the effect of stain transformation. Our first loss performs almost similar to the loss that achieved the best results. (Abstract)

Robust regularized extreme learning machine for regression using iteratively reweighted least squares

Adaptive Cross Entropy for ultrasmall object detection in Computed Tomography with noisy labels

The accuracy of information retrieval systems is often measured using complex loss functions such as the average precision (AP) or the normalized discounted cumulative gain (NDCG). Given a set of positive and negative samples, the parameters of a retrieval system can be estimated by minimizing these loss functions. However, the non-differentiability and non-decomposability of these loss functions does not allow for simple gradient based optimization algorithms. This issue is generally circumvented by either optimizing a structured hinge-loss upper bound to the loss function or by using asymptotic methods like the direct-loss minimization framework. Yet, the high computational complexity of loss-augmented inference, which is necessary for both the frameworks, prohibits its use in large training data sets. To alleviate this deficiency, we present a novel quicksort flavored algorithm for a large class of non-decomposable loss functions. We provide a complete characterization of the loss functions that are amenable to our algorithm, and show that it includes both AP and NDCG based loss functions. Furthermore, we prove that no comparison based algorithm can improve upon the computational complexity of our approach asymptotically. We demonstrate the effectiveness of our approach in the context of optimizing the structured hinge loss upper bound of AP and NDCG loss for learning models for a variety of vision tasks. We show that our approach provides significantly better results than simpler decomposable loss functions, while requiring a comparable training time.

This dissertation explores the critical role of loss functions in enhancing the predictive performance of deep machine learning models. Loss functions are an integral element of all the ongoing advances we witness daily in this domain. I design custom loss functions and their impacts on various machine learning tasks, particularly in computer vision. In the first stage of my research, I aim to improve the prediction performance of deep learning models by providing them with more precise feedback associated with task requirements. This led me to create the concept of assistive loss functions. My first proposed loss function, inspired by the Active Shape Model (ASM), is ASM Loss, designed to improve the detection of facial landmarks and head pose accuracy in facial images. Additionally, influenced by Knowledge Distillation (KD), I introduce KD-Loss, a loss designed specifically for regression tasks, focusing on detecting facial landmarks. Proceeding to the second phase of my research, I focus on regression problems. Building upon my prior work, I introduce Adaptive Coordinate-based Regression (ACR)-Loss, a context-aware loss function with face alignment implication. I provide a mechanism to dynamically measure how hard the localization of each landmark point is, and accordingly, ACR-Loss provides unique feedback to the model. Furthermore, I introduce Intensity-aware Categorical (IC)-Loss for detecting sagittal cervical spine landmark points in medical images. It effectively deploys characteristics from classification problems to enhance regression optimization. Both ACR-Loss and IC-Loss comprise contextual information, leading to a significant enhancement in localization precision. In the third phase, I turned my attention toward classification problems within the domain of deep machine learning. I introduce Adaptive Correlation-based (Ad-corre) Loss, designed to enhance the discriminative capabilities of deep learning models, with a specific focus on improving classification accuracy in automatic facial expression recognition. Furthermore, I introduce Informative (Info) Loss, a specialized loss function developed for medical applications, specifically aimed at distinguishing between Mild Cognitive Impairment and Normal Cognition groups based on transcriptions of interviews conducted as part of the I-CONECT study. Overall, I have proposed custom loss functions to improve the predictive performance in both regression and classification problems. Furthermore, I have introduced assistive loss functions, designed to enhance the performance of the main loss functions. The collective outcomes of this extensive exploration highlight the critical role of loss functions, adapting and optimizing them for various domains in deep machine learning. This research suggests and proposes a variety of methodologies for designing adaptive and context-aware loss functions within deep machine learning.

CNN-based deep architecture has been successfully applied to medical image semantic segmentation task because of its effective feature learning mechanism. However, due to the lack of semantic guidance, such supervised learning model may be susceptible to annotation noise. In order to address this problem, we propose a novel medical image segmentation algorithm based on automatic label error correction. Firstly, local visual saliency regions, namely the Local Visual Cues (LVCs), are captured from low-level feature channels. Then, a deformable spatial transformation module is integrated into our LVC-Net to build visual connections between the predictions and LVCs. By combining noisy labels with image LVCs, a novel loss function is proposed based on their intrinsic spatial relationship. Our method can effectively suppress the influence of label noise by utilizing potential visual guidance during the learning process, thereby generate better semantic segmentation results. Comparative experiment on hip x-ray image segmentation task demonstrate that our algorithm achieves significant improvement over state-of-the-arts in the presences of noisy label.

In many real-world machine learning classification applications, the model performance based on deep neural networks (DNNs) oftentimes suffers from label noise. Various methods have been proposed in the literature to address this issue, primarily by focusing on designing noise-tolerant loss functions, cleaning label noise, and correcting the objective loss. However, the noise-tolerant loss functions face challenges when the noise level increases. This article aims to reveal a convergence path of a trained model in the presence of label noise, and here, the convergence path depicts the evolution of a trained model over epochs. We first propose a theorem to demonstrate that any surrogate loss function can be used to learn DNNs from noisy labels. Next, theories on the general convergence path for the deep models under label noise are presented and verified through a series of experiments. In addition, we design an algorithm based on the proposed theorems that make efficient corrections on the noisy labels and achieve strong robustness in the DNN models. We designed several experiments using benchmark datasets to assess noise tolerance and verify the theorems presented in this article. The comprehensive experimental results firmly confirm our theoretical results and also clearly validate the effectiveness of our method under various levels of label noise.

In many applications, such as robotic perception, scene understanding, augmented reality, 3D reconstruction, and medical image analysis, depth from images is a fundamentally ill-posed problem. The success of depth estimation models relies on assembling a suitably large and diverse training dataset and on the selection of appropriate loss functions. It is critical for researchers in this field to be made aware of the wide range of publicly available depth datasets along with the properties of various loss functions that have been applied to depth estimation. Selection of the right training data combined with appropriate loss functions will accelerate new research and enable better comparison with state-of-the-art. Accordingly, this work offers a comprehensive review of available depth datasets as well as the loss functions that are applied in this problem domain. These depth datasets are categorised into five primary categories based on their application, namely (i) people detection and action recognition, (ii) faces and facial pose, (iii) perception-based navigation (i.e., street signs, roads), (iv) object and scene recognition, and (v) medical applications. The important characteristics and properties of each depth dataset are described and compared. A mixing strategy for depth datasets is presented in order to generalise model results across different environments and use cases. Furthermore, depth estimation loss functions that can help with training deep learning depth estimation models across different datasets are discussed. State-of-the-art deep learning-based depth estimation methods evaluations are presented for three of the most popular datasets. Finally, a discussion about challenges and future research along with recommendations for building comprehensive depth datasets will be presented as to help researchers in the selection of appropriate datasets and loss functions for evaluating their results and algorithms.