Boosting Robust Learning Via Leveraging Reusable Samples in Noisy Web Data

Classifying the sub-categories of an object from the same super-category (e.g., bird species and cars) in fine-grained visual classification (FGVC) highly relies on discriminative feature representation and accurate region localization. Existing approaches mainly focus on distilling information from high-level features. In this article, by contrast, we show that by integrating low-level information (e.g., color, edge junctions, texture patterns), performance can be improved with enhanced feature representation and accurately located discriminative regions. Our solution, named Attention Pyramid Convolutional Neural Network (AP-CNN), consists of 1) a dual pathway hierarchy structure with a top-down feature pathway and a bottom-up attention pathway, hence learning both high-level semantic and low-level detailed feature representation, and 2) an ROI-guided refinement strategy with ROI-guided dropblock and ROI-guided zoom-in operation, which refines features with discriminative local regions enhanced and background noises eliminated. The proposed AP-CNN can be trained end-to-end, without the need of any additional bounding box/part annotation. Extensive experiments on three popularly tested FGVC datasets (CUB-200-2011, Stanford Cars, and FGVC-Aircraft) demonstrate that our approach achieves state-of-the-art performance. Models and code are available at https://github.com/PRIS-CV/AP-CNN_Pytorch-master.

Labeling objects at the subordinate level typically requires expert knowledge, which is not always available from a random annotator. Accordingly, learning directly from web images for fine-grained visual classification (FGVC) has attracted broad attention. However, the existence of noise in web images is a huge obstacle for training robust deep neural networks. In this paper, we propose a novel approach to remove irrelevant samples from the real-world web images during training, and only utilize useful images for updating the networks. Thus, our network can alleviate the harmful effects caused by irrelevant noisy web images to achieve better performance. Extensive experiments on three commonly used fine-grained datasets demonstrate that our approach is much superior to state-of-the-art webly supervised methods. The data and source code of this work have been made anonymously available at: https://github.com/z337-408/WSNFGVC.

Fine-grained visual categorization (FGVC) is challenging due in part to the fact that it is often difficult to acquire an enough number of training samples. To employ large models for FGVC without suffering from overfitting, existing methods usually adopt a strategy of pre-training the models using a rich set of auxiliary data, followed by fine-tuning on the target FGVC task. However, the objective of pre-training does not take the target task into account, and consequently such obtained models are suboptimal for fine-tuning. To address this issue, we propose in this paper a new deep FGVC model termed MetaFGNet. Training of MetaFGNet is based on a novel regularized meta-learning objective, which aims to guide the learning of network parameters so that they are optimal for adapting to the target FGVC task. Based on MetaFGNet, we also propose a simple yet effective scheme for selecting more useful samples from the auxiliary data. Experiments on benchmark FGVC datasets show the efficacy of our proposed method.

Fine-grained visual classification (FGVC) involves classifying multiple subcat-egories within a unified major category, a task characterized by significant intra-class variability and minimal inter-class differences. Previous methods often rely on pre-trained visual models augmented with specialized modules, typically using large scale models that are challenging for industrial deployment. Moreover , image data often comes with auxiliary information (e.g., spatiotemporal priors, attributes, and text descriptions), offering opportunities to enhance FGVC accuracy. Here we propose a novel lightweight Transformer-based approach that incorporates additional auxiliary information to enhance classification accuracy. Our method introduces a simplified pixel-focused aggregation attention (AA) to achieve local and global feature fusion and improves it with a separable aggre-gation attention (SAA) to reduce model complexity. We also present the Extra Inside Padding (EIP) method for integrating auxiliary information with minimal additional parameters. Without pre-training, our model surpasses other lightweight neural networks on fine-grained datasets such as CUB-200-2011, demonstrating a significant improvement in accuracy. Our approach offers a promising direction for FGVC tasks, highlighting the effectiveness of integrating multimodal data for enhanced performance. Our source code is available at https://github.com/yang-zzy/SAA-EIP.

Collecting large-scale data with clean labels for supervised training of neural networks is practically challenging. Although noisy labels are usually cheap to acquire, existing methods suffer a lot from label noise. This paper targets at the challenge of robust training at high label noise regimes. The key insight to achieve this goal is to wisely leverage a small trusted set to estimate exemplar weights and pseudo labels for noisy data in order to reuse them for supervised training. We present a holistic framework to train deep neural networks in a way that is highly invulnerable to label noise. Our method sets the new state of the art on various types of label noise and achieves excellent performance on large-scale datasets with real-world label noise. For instance, on CIFAR100 with a $40\%$ uniform noise ratio and only 10 trusted labeled data per class, our method achieves $80.2{\pm}0.3\%$ classification accuracy, where the error rate is only $1.4\%$ higher than a neural network trained without label noise. Moreover, increasing the noise ratio to $80\%$, our method still maintains a high accuracy of $75.5{\pm}0.2\%$, compared to the previous best accuracy $48.2\%$. Source code available: https://github.com/google-research/google-research/tree/master/ieg

In this paper, we propose a two-stage CNN-based framework to learn color names from web images, aiming to predict color names for tiny image patches. To deal with the noisy labels widespread in web images, we propose a self-supervised CNN (SS-CNN) model in the first stage. The SS-CNN model is trained on image patches with their own color histograms as supervision information. Thus its outputs are able to reflect the color characteristics of images without the influence of the noisy labels. In the second stage, we finetune the SS-CNN model to learn the mapping from image patches to color names, where the patch labels are inherited from its father images. Besides, sample selection is imported iteratively in turns with the finetuning process, which helps filtering out some noisy samples and further improves the model accuracy. Our model shows high representation ability to colors and achieves better performance of color naming compared with the state-of-the-art methods.

Adversarially attack feature similarity for fine-grained visual classification

Datasets often include noisy labels, but learning from them is difficult. Since mislabeled examples usually have larger loss values in training, the small-loss trick is regarded as a standard metric to identify the clean example from the training set for better performance. Nonetheless, this proposal ignores that some clean but hard-to-learn examples also generate large losses. They could be misidentified by this criterion. In this paper, we propose a new metric called the Integrated Area Margin (IAM), which is superior to the traditional small-loss trick, particularly in recognizing the clean but hard-to-learn examples. According to the IAM, we further offer the Hyperspherical Margin Weighting (HMW) approach. It is a new sample weighting strategy that restructures the importance of each example. It should be highlighted that our approach is universal and can strengthen various methods in this field. Experiments on both benchmark and real-world datasets indicate that our HMW outperforms many state-of-the-art approaches in learning with noisy label tasks. Codes are available at https://github.com/Zhangshuojackpot/HMW.

Learning feature representation from discriminative local regions plays a key role in fine-grained visual classification. Employing attention mechanisms to extract part features has become a trend. However, there are two major limitations in these methods: First, they often focus on the most salient part while neglecting other inconspicuous but distinguishable parts. Second, they treat different part features in isolation while neglecting their relationships. To handle these limitations, we propose to locate multiple different distinguishable parts and explore their relationships in an explicit way. In this pursuit, we introduce two lightweight modules that can be easily plugged into existing convolutional neural networks. On one hand, we introduce a feature boosting and suppression module that boosts the most salient part of feature maps to obtain a part-specific representation and suppresses it to force the following network to mine other potential parts. On the other hand, we introduce a feature diversification module that learns semantically complementary information from the correlated part-specific representations. Our method does not need bounding boxes/part annotations and can be trained end-to-end. Extensive experimental results show that our method achieves state-of-the-art performances on several benchmark fine-grained datasets. Source code is available at https://github.com/chaomaer/FBSD.

Computer simulations of soft matter require a compromise to be made between the computational efficiency and the resolution of a model studied. Highly resolved models can give insights into the interactions between individual atoms in a soft material. But, since these atomistic models are computational expensive, they are limited to small length scales and short time scales, which makes it difficult to compare simulation results with the ones from experiments in the laboratory. On the other hand, continuum models enable the study of soft matter at larger length and longer time scales and are computationally less expensive, but they rather focus on macroscopic properties than on their atomistic origin. A possible way to bridge the gap between these scales is to perform simulations at an intermediate level of resolution. The problem, which exists at this mesoscopic scale, is the lack of accurate models. Hence, new ones have to be built. The process to construct mesoscopic models based on information from the atomistic scale is commonly referred to as bottom-up coarse graining. Bottom-up coarse graining describes the process of lowering the resolution of a atomistic model to make it applicable at larger length and time scales. The major goal of this Ph.D. thesis is to increase the knowledge on so called structure-based bottom-up coarse graining techniques.These methods enable the derivation of coarse grained (CG) models, which accurately reproduce the structure of an atomistic or fine grained (FG) model at the mesoscopic scale. The shortcomings of different structure-based methods are carefully analyzed and new approaches to overcome them are presented.

The methods for remote sensing image (RSI) scene classification based on deep convolutional neural networks (DCNNs) have achieved prominent success. However, confronted with adversarial examples obtained by adding imperceptible perturbations to clean images, the great vulnerability of DCNNs makes it worth exploring effective defense methods. To date, numerous countermeasures for adversarial examples have been proposed, but how to improve the defensive ability for unknown attacks still to be answered. To address this issue, in this article, we propose an effective defense framework specified for RSI scene classification, named perturbation-seeking generative adversarial networks (PSGANs). In brief, a new training framework is designed to train the classifier by introducing the examples generated during the image reconstruction process, in addition to clean examples and adversarial ones. These generated examples can be random kinds of unknown attacks during training and thus are utilized to eliminate the blind spots of a classifier. To assist the proposed training framework, a reconstruction method is developed. First, instead of modeling the distribution of clean examples, we model the distributions of the perturbations added in adversarial examples. Second, to make a tradeoff between the diversity of the reconstructed examples and the optimization of PSGAN, a scale factor named seeking radius is introduced to scale the generated perturbations before they are subtracted by the given adversarial examples. Comprehensive and extensive experimental results on three widely used benchmarks for RSI scene classification demonstrate the great effectiveness of PSGAN when faced with both known and unknown attacks. Our source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/xuxiangsun/PSGAN</uri> .

Indian mythology is a treasure trove of divine tales, yet a gap in understanding still exists between foreign tourists and the rich cultural heritage of Indian deities. To address the problem, this paper presents a deep learning-driven mobile application named “MythicVision” designed to help foreign tourists better understand India’s rich cultural heritage by recognizing and interpreting images of Indian mythological deities. At first, four state-of-the-art deep models have been trained and evaluated on a custom in-house dataset consists of 10,970 images of various Indian deities sourced from both natural scene and web images. Then, model-wise weights have been assigned by estimating the test accuracies obtained from test sets. In weight-centric decision mechanism, buckets of all possible classes of image object are updated by aggregating the corresponding model-weights if, the predicted class of the specific model matches any of the possible class. Finally, any possible output class with highest aggregated value is selected as final class of the image object. The whole framework is seamlessly integrated in an end-to-end web application for the ease of user convenience. Key features of “MythicVision” include model-wise weight computation and a weight-centric decision mechanism, which deliver more accurate results compared to traditional majority voting in multi-class image classification. The experimental findings demonstrate that developed framework produces an accuracy of 96% on in-house dataset. The designed MythicVision aims to recognize and classify real-time Indian deity images along with providing valuable information to the users about the deity. The developed web application along with source code and user guidelines have been publicly released in https://github.com/Adinp1213/MythicVision for academic, research and other non-commercial purposes.

Fine-grained visual classification is a challenging task because of intra-class variation and inter-class similarity. Most fine-grained models predominantly focus on discriminative region localization which can effectively solve the intra-class variation, but ignore global information and the problem of inter-class similarity which easily leads to overfitting on specific samples. To address these issues, we develop an end-to-end model based on selecting discriminative features for fine-grained visual classification without the help of part or bounding box annotations. In order to accurately select discriminative features, we integrate effective information from different receptive fields to enhance the quality of features, then the features of discriminative regions detected by anchors and the whole image’s feature are jointly processed for classification. Besides, we propose a new loss function to optimize the model to find discriminative regions and prevent overfitting in the particular sample, which can simultaneously solve the problems of intra-class variation and inter-class similarity. Comprehensive experiments show that the proposed approach is superior to the state-of-the-art methods on CUB-200-2011, Stanford Cars and FGVC-Aircraft datasets.

The deep model training procedure requires large-scale datasets of annotated data. Due to the difficulty of annotating a large number of samples, label noise caused by incorrect annotations is inevitable, resulting in low model performance and poor model generalization. To combat label noise, current methods usually select clean samples based on the small-loss criterion and use these samples for training. Due to some noisy samples similar to clean ones, these small-loss criterion-based methods are still affected by label noise. To address this issue, in this work, we propose Regroup Median Loss (RML) to reduce the probability of selecting noisy samples and correct losses of noisy samples. RML randomly selects samples with the same label as the training samples based on a new loss processing method. Then, we combine the stable mean loss and the robust median loss through a proposed regrouping strategy to obtain robust loss estimation for noisy samples. To further improve the model performance against label noise, we propose a new sample selection strategy and build a semi-supervised method based on RML. Compared to state-of-the-art methods, for both the traditionally trained and semi-supervised models, RML achieves a significant improvement on synthetic and complex real-world datasets. The source is at https://github.com/Feng-peng-Li/Regroup-Loss-Median-to-Combat-Label-Noise.

In this paper, a novel approach for discovering visual patterns associated with semantic concepts using web image resources is proposed.This approach can be used to improve the performance in image clustering and retrieval, image annotation, and other applications such as object recognition. Exploring the rich information in web images that represent semantic concepts as both visual content and text information, this research attempts to effectively learn intrinsic patterns related to semantic concepts. Because the quality of learning algorithms is strongly related to the selection of positive and negative samples, positive samples are first selected effectively, then negative samples are determined reliably based on the selected positive samples. Finally, a good quality visual model associated with a semantic concept is built through an unsupervised learning process. The proposed scheme is completely automatic,needing no human intervention,and is robust and reliable for generic images. Experimental results demonstrate the effectiveness of the proposed approach.