Query-guided feature mining for weakly supervised object detection

Supervised object detection and semantic segmentation require object or even pixel level annotations. When there exist image level labels only, it is challenging for weakly supervised algorithms to achieve accurate predictions. The accuracy achieved by top weakly supervised algorithms is still significantly lower than their fully supervised counterparts. In this paper, we propose a novel weakly supervised curriculum learning pipeline for multi-label object recognition, detection and semantic segmentation. In this pipeline, we first obtain intermediate object localization and pixel labeling results for the training images, and then use such results to train task-specific deep networks in a fully supervised manner. The entire process consists of four stages, including object localization in the training images, filtering and fusing object instances, pixel labeling for the training images, and task-specific network training. To obtain clean object instances in the training images, we propose a novel algorithm for filtering, fusing and classifying object instances collected from multiple solution mechanisms. In this algorithm, we incorporate both metric learning and density-based clustering to filter detected object instances. Experiments show that our weakly supervised pipeline achieves state-of-the-art results in multi-label image classification as well as weakly supervised object detection and very competitive results in weakly supervised semantic segmentation on MS-COCO, PASCAL VOC 2007 and PASCAL VOC 2012.

Object detection under imperfect data receives great attention recently. Weakly supervised object detection (WSOD) suffers from severe localization issues due to the lack of instance-level annotation, while semi-supervised object detection (SSOD) remains challenging led by the inter-image discrepancy between labeled and unlabeled data. In this study, we propose the Single Instance annotated Object Detection (SIOD), requiring only one instance annotation for each existing category in an image. Degraded from inter-task (WSOD) or inter-image (SSOD) discrepancies to the intra-image discrepancy, SIOD provides more reliable and rich prior knowledge for mining the rest of unlabeled instances and trades off the annotation cost and performance. Under the SIOD setting, we propose a simple yet effective framework, termed Dual-Mining (DMiner), which consists of a Similarity-based Pseudo Label Generating module (SPLG) and a Pixel-level Group Contrastive Learning module (PGCL). SPLG firstly mines latent instances from feature representation space to alleviate the annotation missing problem. To avoid being misled by inaccurate pseudo labels, we propose PGCL to boost the tolerance to false pseudo labels. Extensive experiments on MS COCO verify the feasibility of the SIOD setting and the superiority of the proposed method, which obtains consistent and significant improvements compared to baseline methods and achieves comparable results with fully supervised object detection (FSOD) methods with only 40% instances annotated. Code is available at https://github.com/solicucu/SIOD.

Object detection is a challenging task in remote sensing because objects only occupy a few pixels in the images, and the models are required to simultaneously learn object locations and detection. Even though the established approaches well perform for the objects of regular sizes, they achieve weak performance when analyzing small ones or getting stuck in the local minima (e.g., false object parts). Two possible issues stand in their way. First, the existing methods struggle to perform stably on the detection of small objects because of the complicated background. Second, most of the standard methods used handcrafted features and do not work well on the detection of objects parts that are missing. We here address the above issues and propose a new architecture with a multipatch feature pyramid network (MPFP-Net). Different from the current models that, during training, only pursue the most discriminative patches, in MPFP-Net, the patches are divided into class-affiliated subsets, in which the patches are related, and based on the primary loss function, a sequence of smooth loss functions is determined for the subsets to improve the model for collecting small object parts. To enhance the feature representation for patch selection, we introduce an effective method to regularize the residual values and make the fusion transition layers strictly norm-preserving. The network contains bottom-up and crosswise connections to fuse the features of different scales to achieve better accuracy compared to several state-of-the-art object detection models. Also, the developed architecture is more efficient than the baselines.

Existing fully supervised deep learning methods usually require a large number of training samples with abundant annotations for the model training, which is extremely expensive and labor-consuming. Therefore, in order to alleviate huge labeling costs, it is highly desirable to develop weakly supervised learning methods. Here, weakly supervised learning refers to that during the model training, the labels of the training data could be inexact, incomplete, or inaccurate. Typically, there are three types of weakly supervised learning scenarios: inexact supervision, incomplete supervision, and inaccurate supervision. In this thesis, we study all three types of weak supervision scenarios based on three different fundamental 2D and 3D recognition tasks including weakly supervised object detection (WSOD), few-shot image segmentation (FSS), and weakly supervised point cloud segmentation (WSPCS). Specifically, for WSOD, we only have the image-level annotations for the novel class images in the web domain, corresponding to inexact supervision. For FSS, we only have a few pixel-level labeled images (e.g., one or five images) for the novel classes, corresponding to incomplete supervision. For WSPCS, we consider partially labeled samples as weak annotations for the model training, i.e., only a few sparse points inside the whole scene are labeled and all other points are unlabeled, corresponding to incomplete supervision. Moreover, we observe one major limitation in existing consistency-based weakly supervised point cloud segmentation methods, i.e., unsatisfied pseudo labels due to the conventional confidence-based selection, which further leads to inaccurate supervision. For weakly supervised object detection, in Chapter 3, we propose a novel webly supervised object detection (WebSOD) method for novel class detection, which only requires the web images retrieved via the internet using class names as the keywords. Here, we only have the image-level annotations for web images during the model training. Our proposed method combines bottom-up and top-down cues. Within our approach, we introduce a bottom-up mechanism based on the well-trained fully supervised object detector (e.g., Faster RCNN) as an object region estimator for web images by recognizing the common objectiveness shared between base and novel classes. With the estimated regions on the web images, we then use the top-down attention cues as the guidance for region classification. Furthermore, we propose a residual feature refinement (RFR) block to tackle the domain mismatch between the web domain and the target domain. For few-shot image segmentation, currently, the state-of-the-art methods treat this task as a conditional foreground-background segmentation problem, assuming each class is independent. Different from existing methods, in Chapter 4, we introduce the concept of meta-class, which is the meta-information (e.g. certain middle-level features) shareable among all classes. To explicitly learn meta-class representations, we propose a novel Meta-class Memory-based few-shot segmentation method (MM-Net), where we introduce a set of learnable memory embeddings to memorize the meta-class information during the base class training and transfer it to novel classes during the inference stage. For weakly supervised point cloud segmentation, we only have a few sparse labeled points as well as a large number of unlabeled points. To exploit the unlabeled data, we design two different methods from the aspect of adversarial training and consistency training. Firstly, in Chapter 5, considering the smoothness-based methods have achieved promising progress, we advocate applying the consistency constraint under various perturbations to effectively regularize unlabeled 3D points. In particular, we propose a novel Dual Adaptive Transformations (DAT) model for weakly supervised point cloud segmentation, where the dual adaptive transformations are performed via an adversarial strategy at both point-level and region-level, aiming at enforcing the local and structural smoothness constraints on 3D point clouds. Secondly, in Chapter 6, we observe that the straightforward way of applying consistency constraints to weakly supervised point cloud segmentation has two major limitations: unsatisfied pseudo labels due to the conventional confidence-based selection and insufficient consistency constraints due to discarding unreliable pseudo labels. Therefore, we propose a novel Reliability-Adaptive Consistency Network (RAC-Net) to use both prediction confidence and model uncertainty to measure the reliability of pseudo labels and apply consistency training on all unlabeled points while with different consistency constraints for different points based on the reliability of corresponding pseudo labels.

Recent interest in geospatial artificial intelligence (GeoAI) has fostered a wide range of applications using artificial intelligence (AI), especially deep learning for geospatial problem solving. Major challenges, however, such as a lack of training data and ignorance of spatial principles and spatial effects in AI model design remain, significantly hindering the in-depth integration of AI with geospatial research. This article reports our work in developing a cutting-edge deep learning model that enables object detection, especially of natural features, in a weakly supervised manner. Our work has made three innovative contributions: First, we present a novel method of object detection using only weak labels. This is achieved by developing a spatially explicit model according to Tobler’s first law of geography to enable weakly supervised object detection. Second, we integrate the idea of an attention map into the deep learning–based object detection pipeline and develop a multistage training strategy to further boost detection performance. Third, we have successfully applied this model for the automated detection of Mars impact craters, the inspection of which often involved tremendous manual work prior to our solution. Our model is generalizable for detecting both natural and man-made features on the surface of the Earth and other planets. This research has made a major contribution to the enrichment of the theoretical and methodological body of knowledge of GeoAI.

Supervised object detection models are trained to recognize certain objects. These models are classified into two types: single-stage detectors and two-stage detectors. The single-stage detectors just need one pass through the model to anticipate all the bounding boxes, whereas the two-stage detectors require to first estimate the image portions where the object could be located. Due to their speed and simplicity, single-stage anchor-based models are used in many industrial settings. Training such models require bounding boxes that describe the spatial location of an object, which are usually drawn by an expert. However, the question remains, how much area should be considered when drawing the bounding boxes? In this paper, we demonstrate the effects that the size and placement of a rectangular bounding box can have on the performance of the anchor-based models. For this, we first perform experiments on a synthetically generated binary dataset and then on a real-world object detection dataset. Our results show that fixing the size of the bounding boxes can help in improving the performance of the model in the case of single class object detection (approximately 50% improvement in mAP@[.5:.95] for real world dataset). Furthermore, we also demonstrate how freely available tools can be combined for obtaining the best possible semi automated object labeling pipeline.

The deep generative adversarial networks (GAN) recently have been shown to be promising for different computer vision applications, like image editing, synthesizing high resolution images, generating videos, etc. These networks and the corresponding learning scheme can handle various visual space mappings. We approach GANs with a novel training method and learning objective, to discover multiple object instances for three cases: 1) synthesizing a picture of a specific object within a cluttered scene; 2) localizing different categories in images for weakly supervised object detection; and 3) improving object discovery in object detection pipelines. A crucial advantage of our method is that it learns a new deep similarity metric, to distinguish multiple objects in one image. We demonstrate that the network can act as an encoder-decoder generating parts of an image which contain an object, or as a modified deep CNN to represent images for object detection in supervised and weakly supervised scheme. Our ranking GAN offers a novel way to search through images for object specific patterns. We have conducted experiments for different scenarios and demonstrate the method performance for object synthesizing and weakly supervised object detection and classification using the MS-COCO and PASCAL VOC datasets.

For alleviating the human labor associated with annotating the training data for learning object detectors, recent research has focused on semi-supervised object detection (SSOD) and weakly supervised object detection (WSOD) approaches. In SSOD, instead of annotating all the instances in the whole training set, people only need to annotate the part of the training instances using bounding boxes. In WSOD, people need to annotate the image-level tags on all training images to indicate the object categories contained by the corresponding images since more detailed bounding box annotations are no longer needed. Along this line of research, this paper makes a further step to alleviate the human labor in annotating training data, leading to the problem of object detection with semi-annotated weak labels (ODSAWLs). Instead of labeling image-level tags on all training images, ODSAWL only needs the image-level tags for a small portion of the training images, and then, the object detectors can be learned from a small portion of the weakly-labeled training images and from the remaining unlabeled training images. To address such a challenging problem, this paper proposes a cross model co-training framework that collaborates an object localizer and a tag generator in an alternative optimization procedure. Specifically, during the learning procedure, these two (deep) models can transfer the needed knowledge (including labels and visual patterns) between each other. The whole learning procedure is accomplished in a few stages under the guidance of a progressive learning curriculum. To demonstrate the effectiveness of the proposed approach, we implement the comprehensive experiments on three benchmark datasets, where the obtained experimental results are quite encouraging. Notably, by using only about 15% weakly labeled training images, the proposed approach can effectively approach, or even outperform, the state-of-the-art WSOD methods.

A two-stage semi-supervised object detection method for SAR images with missing labels based on meta pseudo-labels

With the increasing popularity of Unmanned Aerial Vehicles (UAVs), the accuracy of detecting small objects in large-view images is also expected to increase. However, accurate small object detection is still a challenging problem. Currently, Image Pyramid Network, Feature Pyramid Network (FPN), rich training strategies and data augmentation are widely used to address this problem. To accurately detect small objects, the most important thing is to mine for more feature information. We propose Widened Residual Block (WRB) to break through the bottleneck of residual information gain to extract more feature information. The second is to emphasize or suppress features to prevent small objects from being overwhelmed by a broad background. We introduce an attention mechanism into PANet and propose Enhanced Attention PANet (EA-PANet), which consists of two parts: Context Attention Module (COAM) and Attention Enhancement Module (AEM). COAM outputs attention heatmaps with context, and AEM fuses features from the channel attention module (CAM) and COAM to avoid distraction from a vast background. In addition, we design a lightweight Decoupled Attention Head (DA-head) to dynamically compute important regions for specific tasks and achieve reliable predictions. Experiments show that our method outperforms state-of-the-art (SOTA) detectors. The source code for this work is available at https://github.com/liuxiaolei111/FindSmallObjects.

Object detection problems also expect fast and accurate location of target objects. But most approaches following convolutional neural networks (CNN) framework generate a large number of redundant candidate regions, wasting a large percent of computation time for candidate classification. Human visual system is able to quickly find the parts or features belonging to objects of interest. Hence, it is believed that by detecting salient regions associated with key classification features for specific objects can quickly provide rough location of objects to be detected. A feature mining method by adding an enforcement learning block on top of CNN framework is proposed. Experiment results showed that the proposed method is able to generate accurate task specific salient regions both for localization and classification. This study attempts to make object localization and classification in one shot based on salient region detection and puts forward a new idea for key feature mining and utilization.

Weakly supervised object detection (WSOD) aims to tackle the object detection problem using only labeled image categories as supervision. A common approach used in WSOD to deal with the lack of localization information is Multiple Instance Learning, and in recent years methods started adopting Multiple Instance Detection Networks (MIDN), which allows training in an end-to-end fashion. In general, these methods work by selecting the best instance from a pool of candidates and then aggregating other instances based on similarity. In this work, we claim that carefully selecting the aggregation criteria can considerably improve the accuracy of the learned detector. We start by proposing an additional refinement step to an existing approach (OICR), which we call refinement knowledge distillation. Then, we present an adaptive supervision aggregation function that dynamically changes the aggregation criteria for selecting boxes related to one of the ground-truth classes, background, or even ignored during the generation of each refinement module supervision. Experiments in Pascal VOC 2007 demonstrate that our Knowledge Distillation and smooth aggregation function significantly improves the performance of OICR in the weakly supervised object detection and weakly supervised object localization tasks. These improvements make the Boosted-OICR competitive again versus other state-of-the-art approaches.

We present a systematic search for Odd Radio Circles (ORCs) and other unusual radio morphologies using data from the first year of the Evolutionary Map of the Universe (EMU) survey. ORCs are rare, enigmatic objects characterised by edge-brightened rings of radio emission, often found in association with distant galaxies. To identify these objects, we employ a hybrid methodology combining supervised object detection techniques and visual inspection of radio source candidates. This approach leads to the discovery of five new ORCs and two additional candidate ORCs, expanding the known population of these objects. In addition to ORCs, we also identify 55 Galaxies with Large-scale Ambient Radio Emission (GLAREs), which feature irregular, rectangular, or circular shapes of diffuse radio emission mostly surrounding central host galaxies. These GLAREs may represent different evolutionary stages of ORCs and studying them could offer valuable insights into their evolutionary processes. We also highlight a subset of Starburst Radio Ring Galaxies, which are star-forming galaxies exhibiting edge-brightened radio rings surrounding their central star-forming regions. We emphasise the importance of multi-wavelength follow-up observations to better understand the physical properties, host galaxy characteristics, and evolutionary pathways of these radio sources.

Object detection under weakly supervised learning is a challenging issue. In remote sensing ship detection task, the weakly supervised learning method requires that the training set only has image-level class annotations. In the absence of location information, it is difficult to locate ships and extract features. Moreover, when the detector extracts more than one candidate region, the image-level annotation makes it difficult to determine whether the candidate regions are multiple ships or mixed with the detected background. To address these issues, this article analyzes the interaction between class information and location information, and proposes a weakly supervised detection method, PistonNet, based on Data-efficient image Transformers (DeiT). PistonNet proposes an artificial point, which is inserted into the feature map. Artificial point can suppress the background and enhance the object by interfering in the weight distribution of object and background in Self-Attention calculation. PistonNet also proposes joint confidence probability to improve detection accuracy. Experiments on the GF1-LRSD and NWPU VHR-10 dataset show that the proposed method boosts detection performance effectively. The main improvements of PistonNet as well as the contributions of this article are threefold. Firstly, PistonNet is a specific weakly supervised object detection (WSOD) method for single-class detection, which provides an innovative approach to classifying target and background. Secondly, PistonNet reaches the level of advanced supervised detectors on detection accuracy with fewer parameters. Finally, the objects’ locations detected by PistonNet are obtained by segmenting regions on the heatmap. PistonNet’s background suppression ability makes it free from dependence on segmentation threshold.

A weakly supervised object detection (WSOD) task is to train a detector with only image-level labels provided. Except for the training difficulty introduced by weaker annotations, the inherent complexity of the remote-sensing images (RSIs) also adds to the challenge. To boost the detector’s localization accuracy, we aim to exploit more semantic information contained in images and help improve the general robustness of the model. Noticing previous methods tend to focus on the most discriminative part of an object, we design a self-training-based network that leverages local semantic features. To this end, we develop a semantic-balanced localization module (SBLM) that distinguishes foreground from background and accurate proposals from incomplete ones, by leveraging a balance of region of interest (ROI) and its context information. Moreover, we find that the self-training strategy highly relies on the quality of pseudo-ground-truth boxes. Motivated by this possible lack of robustness, we design a comprehensive clustering module (CCM) and saliency-based proposal filtering (SPF) module that select pseudo-ground truth more comprehensively under supervision. To be more specific, CCM aims to reduce the arbitrariness during assigning pseudo-labels by considering multiple categorical vectors simultaneously. Salient object detection (SOD) is applied in the SPF module to help evaluate the quality of the chosen pseudo-ground-truth boxes. The detection performance is significantly boosted with the proposed method. Extensive experiments conducted on the NWPU VHR-10.v2 dataset and the DIOR dataset validate that the proposed model outperforms the previous state-of-the-art methods favorably with an mAP of 64.9% and 28.1%, respectively.