Horizontal Box Supervised Oriented Object Detection Based on Edge Feature Self-Attention and Multitask Synthetic Visual Patterns

Compared with general object detection with horizontal bounding boxes in natural images, oriented object detection in remote sensing images is an active and challenging research topic as objects are usually displayed in arbitrary orientations. To model the variant orientations of oriented objects, general CNN-based methods usually adopt more parameters or well-designed modules, which are often complex and inefficient. To address this issue, the detector requires two key components to deal with: (i) generating oriented proposals in a light-weight network to achieve effective representation of arbitrarily oriented objects; (ii) extracting the rotation-invariant feature map in both spatial and orientation dimensions. In this paper, we propose a novel, lightweight rotated region proposal network to produce arbitrary-oriented proposals by sliding two vertexes only on adjacent sides and adopt a simple yet effective representation to describe oriented objects. This may decrease the complexity of modeling orientation information. Meanwhile, we adopt the rotation-equivariant backbone to generate the feature map with explicit orientation channel information and utilize the spatial and orientation modules to obtain completely rotation-invariant features in both dimensions. Without tricks, extensive experiments performed on three challenging datasets DOTA-v1.0, DOTA-v1.5 and HRSC2016 demonstrate that our proposed method can reach state-of-the-art accuracy while reducing the model size by 40% in comparison with the previous best method.

Object detection is a challenging task in the field of remote sensing applications due to the complex backgrounds and uncertain orientation of targets. Compared with the horizontal bounding box, the oriented bounding box can provide orientation information while retaining the true size. Most existing oriented object detection methods are based on Faster-RCNN and the other one-stage methods that can achieve real-time speed but have shortcomings in localization and detection accuracy. To further enhance the performance of one-stage methods, we propose an oriented object detection framework that is based on the single shot detector, namely, single shot anchor refinement network (S²ARN). The S²ARN obtains the accurate detection results by performing two consecutive regressions. More precisely, the multilevel features of the backbone are used to regress the coordinate offsets between the predefined rotated anchors and the ground-truth boxes to generate the refined anchors. The classification and regression subnetworks assigned to the output features are used to perform the second regression to determine the class labels and further adjust the location of the refined anchors. In addition, receptive field amplification modules (RFAMs) are inserted to enlarge the receptive field and extract more discriminative features. Furthermore, in the anchor matching step, angle-related Intersection over Union (ArIoU) is used to calculate the Intersection over Union (IoU) score instead of the traditional method. Benefiting from the multiple regressions and the insensitivity of the ArIoU score to the angle deviation, the angle sampling interval of the rotated anchor can be reduced. The experimental results for the two public datasets, HRSC2016 and UCAS-AOD, demonstrate the effectiveness of the proposed network.

Oriented object detection is a challenging task in aerial images since the objects in aerial images are displayed in arbitrary directions and are frequently densely packed. The mainstream detectors describe rotating objects using a five-parameter or eight-parameter representations, which have representation ambiguity for orientated object definition and border loss discontinuity in the regression process. We proposed an innovative representation method based on area ratio of parallelogram, called area ratio of parallelogram (ARP). Specifically, ARP regresses the minimum bounding rectangle of the oriented object and three area ratios. Three area ratios include the area ratio of a directed object to the smallest circumscribed rectangle and two parallelograms to the minimum circumscribed rectangle. It simplifies offset learning and eliminates the issue of angular periodicity or label point sequences for oriented objects. To further remedy the confusion issue of nearly horizontal objects, the area ratio between the object and its minimal circumscribed rectangle is employed to guide the selection of horizontal or oriented detection for each object. Moreover, the rotated efficient intersection over union loss with horizontal bounding box and three area ratios are designed to optimize the bounding box regression for rotating objects. Experimental results on remote sensing datasets, including HRSC2016, DOTA, and UCAS-AOD, show that the proposed method achieves superior detection performance than many state-of-the-art approaches.

The development of technological devices, such as satellites and drones, has made it easier to collect images and videos from above. With rich and diverse data sources, the problem of detecting objects in aerial images is formed to serve real-life situations: rescue missions, social scenarios, security needs, and more. This issue attracts a lot of attention from researchers in computer vision because of its high applicability in real life. However, object detection in images collected from drones faces many significant challenges due to the complex factors of real-life environments. To overcome these difficulties, many studies focus on developing effective models for detecting and tracking objects from UAV systems. In this paper, our primary emphasis is comprehensive survey of methodologies pertaining to horizontal and oriented object detection. We analyzed and compared existing approaches such as CNN-based vs. Transformer-based, Anchor-based vs. Anchor-free. Moreover, we also learned about ways to represent objects using bounding boxes in oriented object detection. After researching many practical and innovative techniques in different approaches, we have carefully selected 15 methods for horizontal object detection and 18 for oriented object detection that were recently researched and developed to conduct classifications, surveys and assessments. We also provide a taxonomy mind map to assist readers in gaining an overview of the methods we surveyed. Additional, we examine 17 aerial image datasets collected from platforms and in various locations worldwide. These are typical datasets used to serve two primary tasks: Horizontal Object Detection and Oriented Object Detection. Then, we compared the experimental resultsof the previously introduced methods on four typical datasets: VisDrone, UAVDT, DOTAv1.0, and HRSC2016. The above datasets present diversity in the number of classes and the complexity of real-life conditions. Furthermore, they have been tested with various methods and shown remarkable results. For the reasons mentioned, we have enough basis to confirm that the comparison results are objective so that other research groups can make accurate comments and assessments on experimental methods on each dataset. We believe that updating and synthesizing the latest research will provide complete and comprehensive information, helping researchers to have more survey material for the development of the field of object detection in the future.

Object detection in aerial images having many practical applications in real life is becoming popular along with the surging development of deep learning and UAVs (Unmanned Aerial Vehicles). However, adverse weather conditions such as rain, night, and fog might reduce the quality of input images and significantly affect the performance of many perfectly trained detectors (detectors are trained in clear weather conditions). Moreover, object detection in aerial images often seeks high accuracy, while well-known object detection methods use horizontal bounding boxes to represent the object's location. These issues raise the inconsistency between classification and bounding box regression. Understanding the need for practical solutions, we create two experimental Fog datasets based on the original DOTA dataset to provide the in-depth analysis of fog density on multiple well-established oriented object detectors, namely Gliding Vertex, R3Det and ReDet. Furthermore, our training resources achieve promising results, flexibly ensemble to other methods to enhance models' performance and adapt to many adverse weather conditions.

With the continuous progress of remote sensing image object detection tasks in recent years, researchers in this field have gradually shifted the focus of their research from horizontal object detection to the study of object detection in arbitrary directions. It is worth noting that some properties are different from horizontal object detection during oriented object detection that researchers have yet to notice much. This article presents the design of a straightforward and efficient arbitrary-oriented detection system, leveraging the inherent properties of the orientation task, including the rotation angle and box aspect ratio. In the detection of low aspect ratio objects, the angle is of little importance to the orientation bounding box, and it is even difficult to define the angle information in extreme categories. Conversely, in the detection of objects with high aspect ratios, the angle information plays a crucial role and can have a decisive impact on the quality of the detection results. By exploiting the aspect ratio of different targets, this letter proposes a ratio-balanced angle loss that allows the model to make a better trade-off between low-aspect ratio objects and high-aspect ratio objects. The rotation angle of each oriented object, which we naturally embed into a two-dimensional Euclidean space for regression, thus avoids an overly redundant design and preserving the topological properties of the circular space. The performance of the UCAS-AOD, HRSC2016, and DLR-3K datasets show that the proposed model in this paper achieves a leading level in terms of both accuracy and speed.

Object detection for aerial images has achieved remarkable progress in recent years. Nevertheless, most exiting studies do not differentiate oriented object detection from horizontal detection. Certain schemes ignore the ambiguity of oriented object representation and leverage label assignment designed for horizontal object detection directly. Consequently, it leads to unstable training and causes performance degradation, because high-quality samples surrounding the oriented bounding boxes can not be leveraged effectively. To address this problem, we propose a gliding Free, orientation Free, and anchor Free Network (Free <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\rm ^{3}$</tex-math></inline-formula> Net) with high-efficiency for oriented object detection. Specifically, we propose an unambiguous oriented object representation scheme, named FreeGliding, by gliding the projection points of samples on each edge of horizontal bounding boxes. It makes the detection largely free from representation ambiguity and multi-task dependency. To overcome the restrictions of label assignment, we put forward a novel Loss-aware Outer Sample Selection (LOSS) scheme, which takes into consideration spatial information and localization capability to retain high-quality samples surrounding the objects. Moreover, we introduce an Oriented Feature Fusion (OFF) scheme to tackle feature alignment by adjusting the receptive field and fusing oriented features dynamically. Experimental results on two large-scale remote sensing datasets HRSC2016 and DOTA demonstrate that Free <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\rm ^{3}$</tex-math></inline-formula> Net outperforms the state-of-the-art schemes with a large margin. We hope our work can inspire rethinking the design of anchor-free detectors, and serve as a strong baseline for oriented object detection.

In recent years, oriented object detection has garnered significant attention due to its broad applicability and the high performance of recent models. Due to the high resolution of remote-sensing images, object detection models typically require substantial computational resources and time. To address this problem, this paper introduces sparse attention module to the oriented object detector that improves the efficiency by selective calculating attention on foreground features. We have refined the key components to make the pipeline more suitable for remote sensing images and oriented object detection. Based on the improvements in this paper, our model achieves significant performance improvements of 2% mAP50 on DIOR-R and 1.2% mAP50 on DOTAv1 compared to baseline ARS-DETR, as well as a 10.9% improvement in FPS and a 24.2% reduction in FLOPs.

Object detection has a great significance to remote sensing image recognition. Conventional object detection methods by using horizontal bounding box have shown good performance in general images. However, because of looking down perspective of remote sensing images, rotation bounding box is more suitable and precise for object detection. Although rotation bounding box has been researched by some works for remote sensing images, most of them are still two stages. In this paper, by improvement of YOLO architecture, we propose an one-stage object detection method to predict rotation bounding box for remote sensing images. Firstly, in the data preprocessing stage, an image cropping operation is utilized instead of zooming operation to generate suitable input data for network. Meanwhile, in order to avoid an object from being divided, an overlapping band is set during cropping. Secondly, a novel rotation bounding box representation method is introduced, and a corresponding loss function is designed in training process. Experiments on the Dota dataset demonstrate that our method outperforms state-of-the-art rotation objects detection methods, in terms of mAP, our method achieves 73.50%mAP, which is higher than methods based on rotation bounding box by 2.0%.

In the field of remote sensing object detection, the annotation work of rotated bounding boxes is extremely tedious and consumes a large amount of human and time costs. To effectively alleviate this problem, this paper proposes a novel weakly supervised rotated object detection method specifically designed for remote sensing scenarios. This method only requires horizontal bounding box annotations for training, which significantly reduces the annotation requirements. The core of this method is a feature fusion network that integrates a multi-scale receptive field module and a Global Attention Mechanism (GAM), aiming to enhance the model’s feature representation ability for multi-scale and multi-directional objects. We design a dual-branch learning framework: a weakly supervised branch associates horizontal box annotations with rotated box predictions through geometric constraints to learn and generate preliminary rotated detection results; a self-supervised branch applies rotation transformations to the original image and uses consistency loss to constrain the prediction outputs of the two branches, thereby providing a reliable supervision signal for the estimation of the object’s orientation angle. This design ingeniously solves the core challenge of angle learning ambiguity in the weakly supervised setting. Extensive experiments on two mainstream remote sensing detection benchmarks, DOTA and DIOR-R, show that the proposed method performs excellently. It not only significantly outperforms other weakly supervised comparison methods but also is comparable to or more competitive than many fully supervised methods that require full rotated annotations. At the same time, this method also shows great advantages in memory efficiency and inference speed. The results show that the method proposed in this paper provides an effective way to achieve efficient and low-cost remote sensing rotated object detection.

Withcontinuous advancements in remote sensing technology, object detection in remote sensing images has emerged as a critical research direction in maritime surveillance, port management, and national defense. Among these applications, ship detection is a key task. Due to the fact that ships in images typically exhibit arbitrary rotations, multi-scale distributions, and complex backgrounds, conventional detection methods based on horizontal or rotated bounding boxes often fail to adequately capture the fine-grained information of the targets, thereby compromising detection accuracy. This paper proposes the Dense Representative Points-Guided Rotated-Ship Detection (DenseRRSD) method. The proposed approach represents ship objects using dense representative points (RepPoints) to effectively capture local semantic information, thereby avoiding the background noise issues associated with traditional rectangular bounding box representations. To further enhance detection accuracy, an edge region sampling strategy is devised to uniformly sample RepPoints from critical ship parts, and a Weighted Residual Feature Pyramid Network (WRFPN) is introduced to efficiently fuse the multi-scale features through residual connections and learnable weights. In addition, a Weighted Chamfer Loss (WCL) combined with a staged localization loss strategy is employed to progressively refine localization from coarse to fine stages. Experimental results on both the HRSC2016 dataset and the newly constructed DOTA-SHIP dataset demonstrate that DenseRRSD achieves state-of-the-art detection accuracy, with mean Average Precision (mAP) scores of 91.2% and 83.2%, respectively, significantly outperforming existing methods. These results verify the effectiveness and robustness of the proposed approach in rotated-ship detection under diverse conditions.

Infrared imaging is based on thermal radiation and does not rely on visible light, allowing for it to operate normally at night and in low-light conditions. This characteristic is beneficial for regulatory authorities to monitor ships. Existing infrared image segmentation methods face challenges such as the absence of color information, blurred edges, weak high-frequency details, and low contrast due to the imaging principles. Consequently, the segmentation accuracy for small-sized ship targets and edges is low, influenced by the indistinct features of infrared images and the weak difference between the background and targets. To address these issues, this paper proposes an infrared image ship segmentation algorithm called the Infrared Image Edge-Enhanced Segmentation Network (IERNet) to extract ship temperature information. By using pseudo-color infrared images, the sensitivity to edges is enhanced, improving the edge features of ships in infrared images. The Sobel operator is used to obtain edge feature maps, and the Convolutional Block Attention Module (CBAM) extracts key feature information. In the Fusion Unit, edge features guide the extraction of infrared ship features in the backbone network, resulting in feature maps rich in edge information. Finally, a specialized loss function with edge weights supervises the fusion features. An eXtreme Gradient Boosting (XGBoost) machine learning model is then established to predict the ship image brightness temperature threshold, using engine brightness threshold, water area brightness threshold, boundary brightness threshold, and temperature gradient as predictive elements. In terms of image segmentation, our algorithm achieves a segmentation performance of 89.17% mIoU. Regarding the XGBoost model’s performance, it achieves high goodness of fit and small error values on both the training and testing sets, demonstrating its good performance in predicting ship temperature. The model achieves over 70% goodness of fit, and the RMSE values for both models are 3.472, indicating minimal errors. Statistical analysis reveals that the proportion of ship temperature differences predicted by the XGBoost model exceeding 2 is less than 0.020%. The proposed temperature detection method offers higher accuracy and versatility, contributing to more efficient detection of abnormal ship temperatures at night.

In aerial scenes, oriented object detection is sensitive to the orientation of objects, which makes the formulation of orientation-aware object representation become a critical problem. Existing methods mostly adopt rectangle anchor or discrete points as object representation, which may lead to the feature aliasing between overlapping objects and ignore the orientation information of objects. To solve these issues, we propose a novel anchor-free oriented object detection network named DRDet, which adopts Dual-angle Rotated Lines (DRL) as object representation. Different from other object representations, DRL can adaptively rotate and extend to the boundary of the object according to its orientation and shape, which explicitly introduces the orientation information into the formulation of object representation. And it can adaptively cope with the geometric deformation of objects. Based on the dual-angle rotated lines, we design an Orientation-guided Feature Encoder (OFE) to encode discriminant object feature along each rotated line, respectively. Instead of encoding rectangle feature, the OFE module adopts line features for orientation-guided feature encoding, which can alleviate the feature aliasing between neighboring objects or background. To further enhance the flexibility of dual-angle rotated lines, we design a Dual-angle Decoder (DD) that predicts two angle offsets according to the orientation-guided feature and converts the angle offsets and regression offsets into dual-angle rotated line representation, which can help to guide the adaptive rotation of each rotated line, respectively. Our proposed method achieves consistent improvement on both DOTA and HRSC2016 datasets. Extensive experimental results verify the effectiveness of our method in oriented object detection.

It is challenging to accurately detect flexible objects with arbitrary orientation from monitoring images in power grid maintenance and inspection sites. This is because these images exhibit a significant imbalance between the foreground and background, which can lead to low detection accuracy when using a horizontal bounding box (HBB) as the detector in general object detection algorithms. Existing multi-oriented detection algorithms that use irregular polygons as the detector can improve accuracy to some extent, but their accuracy is limited due to boundary problems during the training process. This paper proposes a rotation-adaptive YOLOv5 (R_YOLOv5) with a rotated bounding box (RBB) to detect flexible objects with arbitrary orientation, effectively addressing the above issues and achieving high accuracy. Firstly, a long-side representation method is used to add the degree of freedom (DOF) for bounding boxes, enabling accurate detection of flexible objects with large spans, deformable shapes, and small foreground-to-background ratios. Furthermore, the further boundary problem induced by the proposed bounding box strategy is overcome by using classification discretization and symmetric function mapping methods. Finally, the loss function is optimized to ensure training convergence for the new bounding box. To meet various practical requirements, we propose four models with different scales based on YOLOv5, namely R_YOLOv5s, R_YOLOv5m, R_YOLOv5l, and R_YOLOv5x. Experimental results demonstrate that these four models achieve mean average precision (mAP) values of 0.712, 0.731, 0.736, and 0.745 on the DOTA-v1.5 dataset and 0.579, 0.629, 0.689, and 0.713 on our self-built FO dataset, exhibiting higher recognition accuracy and a stronger generalization ability. Among them, R_YOLOv5x achieves a mAP that is about 6.84% higher than ReDet on the DOTAv-1.5 dataset and at least 2% higher than the original YOLOv5 model on the FO dataset.

Small and cluttered objects are common in real-world which are challenging for detection. The difficulty is further pronounced when the objects are rotated, as traditional detectors often routinely locate the objects in horizontal bounding box such that the region of interest is contaminated with background or nearby interleaved objects. In this paper, we first innovatively introduce the idea of denoising to object detection. Instance-level denoising on the feature map is performed to enhance the detection to small and cluttered objects. To handle the rotation variation, we also add a novel IoU constant factor to the smooth L1 loss to address the long standing boundary problem, which to our analysis, is mainly caused by the periodicity of angular (PoA) and exchangeability of edges (EoE). By combing these two features, our proposed detector is termed as SCRDet++. Extensive experiments are performed on large aerial images public datasets DOTA, DIOR, UCAS-AOD as well as natural image dataset COCO, scene text dataset ICDAR2015, small traffic light dataset BSTLD and our released S 2 TLD by this paper. The results show the effectiveness of our approach. The released dataset S 2 TLD is made public available, which contains 5,786 images with 14,130 traffic light instances across five categories.