Multimodal Knowledge Distillation for Arbitrary-Oriented Object Detection in Aerial Images

Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark

Object detection in aerial images having many practical applications in real life is becoming popular along with the surging development of deep learning and UAVs (Unmanned Aerial Vehicles). However, adverse weather conditions such as rain, night, and fog might reduce the quality of input images and significantly affect the performance of many perfectly trained detectors (detectors are trained in clear weather conditions). Moreover, object detection in aerial images often seeks high accuracy, while well-known object detection methods use horizontal bounding boxes to represent the object's location. These issues raise the inconsistency between classification and bounding box regression. Understanding the need for practical solutions, we create two experimental Fog datasets based on the original DOTA dataset to provide the in-depth analysis of fog density on multiple well-established oriented object detectors, namely Gliding Vertex, R3Det and ReDet. Furthermore, our training resources achieve promising results, flexibly ensemble to other methods to enhance models' performance and adapt to many adverse weather conditions.

This paper presents a new training model for orientation invariant object detection in aerial images by extending a deep learning based RetinaNet which is a single-stage detector based on feature pyramid networks and focal loss for dense object detection. Unlike R3Det which applies feature refinement to handle rotating objects, we proposed further improvement to cope with the densely arranged and class imbalance problems in aerial imaging, on three aspects: 1) All training images are traversed in each iteration instead of only one image in each iteration in order to cover all possibilities; 2) The learning rate is reduced if losses are not reduced; and 3) The learning rate is reduced if losses are not changed. The proposed method was calibrated and validated by comprehensive for performance evaluation and benchmarking. The experiment results demonstrate the significant improvement in comparison with R3Dec approach on the same data set. In addition to the well-known public data set DOTA for benchmarking, a new data set is also established by considering the balance between the training set and testing set. The map of losses which dropped down smoothly without jitter and overfitting also illustrates the advantages of the proposed newmodel.

Object detection in aerial images is an active yet challenging task in computer vision because of the bird’s-eye view perspective, the highly complex backgrounds, and the variant appearances of objects. Especially when detecting densely packed objects in aerial images, methods relying on horizontal proposals for common object detection often introduce mismatches between the Region of Interests (RoIs) and objects. This leads to the common misalignment between the final object classification confidence and localization accuracy. In this paper, we propose a RoI Transformer to address these problems. The core idea of RoI Transformer is to apply spatial transformations on RoIs and learn the transformation parameters under the supervision of oriented bounding box (OBB) annotations. RoI Transformer is with lightweight and can be easily embedded into detectors for oriented object detection. Simply apply the RoI Transformer to light head RCNN has achieved state-of-the-art performances on two common and challenging aerial datasets, i.e., DOTA and HRSC2016, with a neglectable reduction to detection speed. Our RoI Transformer exceeds the deformable Position Sensitive RoI pooling when oriented bounding-box annotations are available. Extensive experiments have also validated the flexibility and effectiveness of our RoI Transformer.

Object detection in aerial images is challenging for at least two reasons: (1) most objects are small scale relative to high resolution aerial images; and (2) the object position distribution is nonuniform, making the detection inefficient. In this paper, a novel network, the coarse-grained density map network (CDMNet), is proposed to address these problems. Specifically, we format density maps into coarsegrained form and design a lightweight dual task density estimation network. The coarse-grained density map can not only describe the distribution of objects, but also cluster objects, quantify scale and reduce computing. In addition, we propose a cluster region generation algorithm guided by density maps to crop input images into multiple subregions, denoted clusters, where the objects are adjusted in a reasonable scale. Besides, we improved mosaic data augmentation to relieve foreground-background and category imbalance problems during detector training. Evaluated on two popular aerial datasets, VisDrone[29] and UAVDT[6], CDMNet has achieved significant accuracy improvement compared with previous state-of-the-art methods.

Object detection in aerial images plays a crucial role across diverse domains such as agriculture, environmental monitoring, and security. Aerial images present several challenges, including dense small objects, intricate backgrounds, and occlusions, necessitating robust detection algorithms. This paper addresses the critical need for accurate and efficient object detection in aerial images using a Transformer-based approach enhanced with specialized methodologies, termed DFS-DETR. The core framework leverages RT-DETR-R18, integrating the Cross Stage Partial Reparam Dilation-wise Residual Module (CSP-RDRM) to optimize feature extraction. Additionally, the introduction of the Detail-Sensitive Pyramid Network (DSPN) enhances sensitivity to local features, complemented by the Dynamic Scale Sequence Feature-Fusion Module (DSSFFM) for comprehensive multi-scale information integration. Moreover, Multi-Attention Add (MAA) is utilized to refine feature processing, which enhances the model’s capacity for understanding and representation by integrating various attention mechanisms. To improve bounding box regression, the model employs MPDIoU with normalized Wasserstein distance, which accelerates convergence. Evaluation across the VisDrone2019, AI-TOD, and NWPU VHR-10 datasets demonstrates significant improvements in the mean average precision (mAP) values: 24.1%, 24.0%, and 65.0%, respectively, surpassing RT-DETR-R18 by 2.3%, 4.8%, and 7.0%, respectively. Furthermore, the proposed method achieves real-time inference speeds. This approach can be deployed on drones to perform real-time ground detection.

Decision letter for "No-Extra Components Density Map Cropping Guided Object Detection in Aerial Images"

Applications of aerial imaging, especially based on unmanned aerial vehicles (UAVs) platform, rapidly explode in recent years. Meanwhile, vision-based sensing, e.g., detection and recognition, for UAVs becomes increasingly important. Objects in aerial images are usually of tiny size, hence occupying a limited area. Terminology speaking, the images are very sparse in spatial. However, existing work in aerial object detection commonly ignores this point. Conversely, we explore the availability of such a property in improving the detection performance of aerial images. Specifically, we propose a general method, train in dense and test in sparse (TDTS), to exploit sparsity in aerial object detection: 1) in the training stage, the possible positions of object are learned by training a fully convolutional network (called prophet head) and 2) in the testing stage, prophet head identifies the possible object locations to reduce redundant computation in classification and box prediction head by sparse convolution. By extensive experiments on the VisDrone2019-Det data set, we find that the sparsity can not only help to speed up inference but also to improve accuracy. Thus, we argue that the sparsity deserves more attention.

Rotated object detection in aerial images is still challenging due to arbitrary orientations, large scale and aspect ratio variations, and extreme density of objects. Existing state-of-the-art rotated object detection methods mainly rely on angle-based detectors. However, angle-based detectors can easily suffer from a long-standing boundary problem. To tackle this problem, we propose a purely angle-free framework for rotated object detection, called Point RCNN. Point RCNN is a two-stage detector including both PointRPN and PointReg which are angle-free. Given an input aerial image, first, the backbone-FPN extracts hierarchical features, then, the PointRPN module generates an accurate rotated region of interests (RRoIs) by converting the learned representative points of each rotated object using the MinAreaRect function of OpenCV. Motivated by RepPoints, we designed a coarse-to-fine process to regress and refine the representative points for more accurate RRoIs. Next, based on the learned RRoIs of PointRPN, the PointReg module learns to regress and refine the corner points of each RRoI to perform more accurate rotated object detection. Finally, the final rotated bounding box of each rotated object can be attained based on the learned four corner points. In addition, aerial images are often severely unbalanced in categories, and existing rotated object detection methods almost ignore this problem. To tackle the severely unbalanced dataset problem, we propose a balanced dataset strategy. We experimentally verified that re-sampling the images of the rare categories can stabilize the training procedure and further improve the detection performance. Specifically, the performance was improved from 80.37 mAP to 80.71 mAP in DOTA-v1.0. Without unnecessary elaboration, our Point RCNN method achieved new state-of-the-art detection performance on multiple large-scale aerial image datasets, including DOTA-v1.0, DOTA-v1.5, HRSC2016, and UCAS-AOD. Specifically, in DOTA-v1.0, our Point RCNN achieved better detection performance of 80.71 mAP. In DOTA-v1.5, Point RCNN achieved 79.31 mAP, which significantly improved the performance by 2.86 mAP (from ReDet’s 76.45 to our 79.31). In HRSC2016 and UCAS-AOD, our Point RCNN achieved higher performance of 90.53 mAP and 90.04 mAP, respectively.

In recent years, detection methods for generic object detection have achieved significant progress. However, due to the large number of small objects in aerial images, mainstream detectors struggle to achieve a satisfactory detection performance. The challenges of small object detection in aerial images are primarily twofold: (1) Insufficient feature representation: The limited visual information for small objects makes it difficult for models to learn discriminative feature representations. (2) Background confusion: Abundant background information introduces more noise and interference, causing the features of small objects to easily be confused with the background. To address these issues, we propose a Multi-Level Contextual and Semantic Information Aggregation Network (MCSA-Net). MCSA-Net includes three key components: a Spatial-Aware Feature Selection Module (SAFM), a Multi-Level Joint Feature Pyramid Network (MJFPN), and an Attention-Enhanced Head (AEHead). The SAFM employs a sequence of dilated convolutions to extract multi-scale local context features and combines a spatial selection mechanism to adaptively merge these features, thereby obtaining the critical local context required for the objects, which enriches the feature representation of small objects. The MJFPN introduces multi-level connections and weighted fusion to fully leverage the spatial detail features of small objects in feature fusion and enhances the fused features further through a feature aggregation network. Finally, the AEHead is constructed by incorporating a sparse attention mechanism into the detection head. The sparse attention mechanism efficiently models long-range dependencies by computing the attention between the most relevant regions in the image while suppressing background interference, thereby enhancing the model’s ability to perceive targets and effectively improving the detection performance. Extensive experiments on four datasets, VisDrone, UAVDT, MS COCO, and DOTA, demonstrate that the proposed MCSA-Net achieves an excellent detection performance, particularly in small object detection, surpassing several state-of-the-art methods.

Object detection in aerial images is more difficult than general object detection. The main reasons are as follows: (1) aerial image object pixels are small and difficult to detect; (2) aerial image objects are sparsely distributed, which increases the difficulty of detection. In this paper, we propose an adaptive Clustered Detection network based on Clustered Detection, which can better solve the above-mentioned problems. The whole process of image detection is as follows: (1) given an aerial image, the adaptive clustering sub-network will output the coordinates of several candidate clustering areas and the corresponding confidence, and also output a suggested candidate clustering area the number of N; (2) the fusion network merges and divides the candidate clustering area according to the coordinates, confidence and N of the candidate clustering area, and outputs N segmented pictures; (3) these segmented pictures are accurately detected, the original pictures are globally detected, and the results of the precise detection and the global detection are merged to obtain the final detection result. Compared with the Clustered Detection, our network can adaptively determine the number of clustering regions in the picture, thereby improving the detection efficiency and accuracy.

Detecting objects in aerial images is challenging for at least two reasons: (1) target objects like pedestrians are very small in pixels, making them hardly distinguished from surrounding background; and (2) targets are in general sparsely and non-uniformly distributed, making the detection very inefficient. In this paper, we address both issues inspired by observing that these targets are often clustered. In particular, we propose a Clustered Detection (ClusDet) network that unifies object clustering and detection in an end-to-end framework. The key components in ClusDet include a cluster proposal sub-network (CPNet), a scale estimation sub-network (ScaleNet), and a dedicated detection network (DetecNet). Given an input image, CPNet produces object cluster regions and ScaleNet estimates object scales for these regions. Then, each scale-normalized cluster region is fed into DetecNet for object detection. ClusDet has several advantages over previous solutions: (1) it greatly reduces the number of chips for final object detection and hence achieves high running time efficiency, (2) the cluster-based scale estimation is more accurate than previously used single-object based ones, hence effectively improves the detection for small objects, and (3) the final DetecNet is dedicated for clustered regions and implicitly models the prior context information so as to boost detection accuracy. The proposed method is tested on three popular aerial image datasets including VisDrone, UAVDT and DOTA. In all experiments, ClusDet achieves promising performance in comparison with state-of-the-art detectors.

Oriented object detection is a challenging task in aerial images since the objects in aerial images are displayed in arbitrary directions and are frequently densely packed. The mainstream detectors describe rotating objects using a five-parameter or eight-parameter representations, which have representation ambiguity for orientated object definition and border loss discontinuity in the regression process. We proposed an innovative representation method based on area ratio of parallelogram, called area ratio of parallelogram (ARP). Specifically, ARP regresses the minimum bounding rectangle of the oriented object and three area ratios. Three area ratios include the area ratio of a directed object to the smallest circumscribed rectangle and two parallelograms to the minimum circumscribed rectangle. It simplifies offset learning and eliminates the issue of angular periodicity or label point sequences for oriented objects. To further remedy the confusion issue of nearly horizontal objects, the area ratio between the object and its minimal circumscribed rectangle is employed to guide the selection of horizontal or oriented detection for each object. Moreover, the rotated efficient intersection over union loss with horizontal bounding box and three area ratios are designed to optimize the bounding box regression for rotating objects. Experimental results on remote sensing datasets, including HRSC2016, DOTA, and UCAS-AOD, show that the proposed method achieves superior detection performance than many state-of-the-art approaches.

Object detection in aerial images is widely applied in many applications. In recent years, faster region convolutional neural network shows a great improvement on object detecting in natural images. Considering the size and distribution characteristic of object in remote sensing images, the region proposal network (RPN) should be changed before being adopted. In this letter, a scale adaptive proposal network (SAPNet) is proposed to improve the accuracy of multiobject detection in remote sensing images. The SAPNet consists of multilayer RPNs which are designed to generate multiscale object proposals, and a final detection subnetwork in which fusion feature layer has been applied for better multiobject detection. Comparative experimental results show that the proposed SAPNet significantly improves the accuracy of multiobject detection.

Feature Adaptive Learning Detr for Small Object Detection in Aerial Images