Deep Learning-Based Object Detection Techniques for Remote Sensing Images: A Survey

Most current researches primarily focus on improving experimental accuracy using large models, often neglecting the deployment challenges. There is a growing need for lightweight algorithms in certain remote sensing devices. Moreover, remote sensing images (RSIs) often contain numerous small, densely distributed targets, which pose significant detection challenges. To address these issues, we have improved the YOLOv8s network and developed a lightweight remote sensing object detection (RSOD) network based on multiscale features fusion and context information (MFFCI–YOLOv8). This network combines multiscale feature fusion and contextual information to accurately detect objects in RSIs. First, we introduce the lightweight CSP bottleneck with attention module, which utilizes partial convolution calculation and SimAM attention mechanisms to decrease the number of parameters and computational complexity while enhancing feature extraction capabilities. Second, we design the gate spatial pyramid pooling fast module to enhance the model's perception of scale and contextual information, thus improving the detection of small objects. Last, we employ the multiscale fusion lightweight neck module for more efficient multiscale feature fusion, preventing the loss of small objects. Compared to YOLOv8s, our overall model reduces the number of parameters by 7.7% and FLOPs by 11.9%. We validated the accuracy of MFFCI–YOLOv8 on two remote sensing datasets, NWPU VHR-10 and VisDrone. The experimental results demonstrate that our model offers a low computational cost and high detection accuracy compared to other RSOD models and other YOLO models.

Recent studies in remote sensing object detection have made excellent progress and shown promising performance. However, most current detectors only explore rotation-invariant feature extraction but disregard the valuable spatial and semantic prior knowledge in remote sensing images (RSIs), which limits the detection performance when encountering blurred or heavy occluded objects. To address this issue, we propose a mask-reconstruction relation learning (MRRL) framework to learn such prior knowledge among objects and a consistency-reasoning transformer over relation proposals (CTRP) to recognize objects with limited visual features via consistency reasoning. Specifically, MRRL framework applies random mask to some objects in the training dataset and performs masked objects reconstruction to guide the network to learn the distribution consistency of objects. CTRP is the core component of the MRRL framework, which models the interaction between spatial and semantic priors, and uses easy detected objects to reason hard detected objects. The trained CTRP can be integrated into the existing detector to improve the ability of object detection with limited visual features in RSIs. Extensive experiments on widely-used datasets for two distinct tasks, namely remote sensing object detection task and occluded object detection task, demonstrate the effectiveness of the proposed method. Source code is available at https://github.com/sunpeng96/CTRP_mmrotate.

Content-based remote sensing (RS) image retrieval (CBRSIR) is a critical way to organize high-resolution RS (HRRS) images in the current big data era. The increasing volume of HRRS images from different satellites and sensors leads to more attention to the cross-source CSRSIR (CS-CBRSIR) problem. Due to the data drift, one crucial problem in CS-CBRSIR is the modality discrepancy. Most existing methods focus on finding a common feature space for various HRRS images to address this issue. In this space, their similarity relations can be measured directly to obtain the cross-source retrieval results straight. This way is feasible and reasonable, however, the specific information corresponding to HRRS images from different sources is always ignored, limiting retrieval performance. To overcome this limitation, we develop a new model for CS-CBRSIR in this paper named dual modality collaborative learning (DMCL). To fully explore the specific information from diverse HRRS images, DMCL first introduces ResNet50 as the feature extractor. Then, a common space mutual learning module is developed to map the specific features into a common space. Here, the modality discrepancy is reduced from the aspects of features and their distributions. Finally, to supplement the specific knowledge to the common features, we develop modality transformation and the dual-modality feature learning modules. Their function is to transmit the specific knowledge from different sources mutually and fuse the specific and common features adaptively. The comprehensive experiments are conducted on a public dataset. Compared with many existing methods, the behavior of our DMCL is stronger. These encouraging results for a public dataset indicate that the proposed DMCL is useful in CS-CBRSIR tasks.

With the development of remote sensing technology, remote sensing object detection has been widely applied in various fields, but it still faces some thorny challenges, such as the following: 1) the complexity of object scale changes in remote sensing images makes it difficult to improve the performance of small object detection and 2) remote sensing images have complex backgrounds and densely arranged small and weak objects, which pose a serious problem of feature interference. To alleviate these challenges, we propose an end-to-end adaptive feature separation network called AFSNet, which includes a scale-aware module (SAM) and a class-aware module (CAM). The SAM mainly enables feature maps of different resolutions to detect objects of different scales. Shallow feature maps mainly suppress the features of large objects they contain to focus on small object detection, while deep feature maps increase the detailed features of large objects they contain to focus on large object detection. The CAM is mainly used to distinguish the features in the feature map by category, separating the features of different categories into different channels, thus mitigating the problem of inter class feature interference, and blocking background interference. The effectiveness of this article has been proven on the NWPU VHR-10, IPIU-M, DIOR, and DOTA2.0 datasets. It can be widely applied in civilian, military, and other fields. Through experimental verification, our AFSNet achieved 97.70% mAP on the NWPU VHR-10 dataset, 78.9% mAP on the DIOR dataset, and 58.22% mAP on the DOTA2.0 dataset. Our code is available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Xidian-AIGroup190726/AFSNet</uri>.

The presence of shadows in high-resolution (HR) remote-sensing images reduces object detection accuracy. To address this problem, in this paper, we proposed a deep neural network algorithm for shadow detection by using the AISD and SSAD remote-sensing shadow image datasets. To improve the ability to extract spatial information from feature maps, we developed a cross-spatial attention module that focuses on semantic information in the horizontal and vertical directions at each position point on the remote-sensing image. This module overcomes the limitations of existing technologies in accurately judging small areas and suspected shadow areas and in missing or incorrectly detected shadow areas. In addition, to improve the ability to extract shadow features and the accuracy of shadow detection in remote-sensing images, we developed a channel attention module that assigns more attention to channels that conform to the shadow color characteristics. The network architecture comprises an encoder – decoder structure, with ResNeXt50 used as the backbone for the encoder and a multi resolution parallel fusion (MRPF) designed for the decoder; cross-spatial and channel attention were incorporated into the decoder unit. Experimental results demonstrated the superior performance of the proposed algorithm, with an F1 score of 92.6% for the shadow category on the test set, thus, outperforming other algorithms and making the proposed method an effective solution for shadow detection in HR remote-sensing images.

Object detection plays a crucial role in scene understanding and has extensive practical applications. In the field of remote sensing object detection, both detection accuracy and robustness are of significant concern. Existing methods heavily rely on sophisticated adversarial training strategies that tend to improve robustness at the expense of accuracy. However, detection robustness is not always indicative of improved accuracy. Therefore, in this paper, we research how to enhance robustness, while still preserving high accuracy, or even improve both simultaneously, with simple vanilla adversarial training or even in the absence thereof. In pursuit of a solution, we first conduct an exploratory investigation by shifting our attention from adversarial training, referred to as adversarial fine-tuning, to adversarial pretraining. Specifically, we propose a novel pretraining paradigm, namely structured adversarial self-supervised (SASS) pretraining, to strengthen both clean accuracy and adversarial robustness for object detection in remote sensing images. At a high level, SASS pretraining aims to unify adversarial learning and self-supervised learning into pretraining and encode structured knowledge into pretrained representations for powerful transferability to downstream detection. Moreover, to fully explore the inherent robustness of vision Transformers and facilitate their pretraining efficiency, by leveraging the recent masked image modeling (MIM) as the pretext task, we further instantiate SASS pretraining into a concise end-to-end framework, named structured adversarial MIM (SA-MIM). SA-MIM consists of two pivotal components, structured adversarial attack and structured MIM (S-MIM). The former establishes structured adversaries for the context of adversarial pretraining, while the latter introduces a structured local-sampling global-masking strategy to adapt to hierarchical encoder architectures. Comprehensive experiments on three different datasets have demonstrated the significant superiority of the proposed pretraining paradigm over previous counterparts for remote sensing object detection. More importantly, regardless of with or without adversarial fine-tuning, it enables simultaneous improvements on detection accuracy and robustness as expected, promisingly alleviating the dependence on complicated adversarial fine-tuning.

Applications of remote sensing images in both defense and civilian sectors have spurred substantial research interest. In the field of remote sensing, object detection confronts challenges such as complex backgrounds, scale diversity, and the presence of dense small objects. To address these issues, we propose an improved deep learning-based model, the Global Multi-scale Fusion Self-calibration Network, which is expected to contribute to alleviating the challenges. It consists of three main components: the hierarchical feature aggregation backbone, which uses improved modules such as the receptive field context-aware feature extraction module, the global information acquisition module, and the simple parameter-free attention module to extract key features and minimize the background interference. To couple multi-scale features, we enhanced the fusing component and designed the multi-scale enhanced pyramid structure integrating the proposed new modules. During the detection phase, especially when focusing on small object detection, we designed a novel convolutional attention feature fusion head. This head is constructed to integrate local and global branches for feature extraction by leveraging channel shuffling and multi-head attention mechanisms for efficient and accurate detection. Experiments on the Detection in Optical Remote Sensing Images (DIOR), Northwestern Polytechnical University Very High-Resolution‐10 (NWPU VHR‐10), remote sensing object detection (RSOD), and DOTAv1.0 data sets show that our method achieves mAP50(mean average precision at 50% intersection over union) of 69.7%, 91.3%, 94.2%, and 70.0%, respectively, outperforming existing comparative methods. The proposed network is expected to provide new perspectives for remote sensing tasks and possible solutions for relevant applications in the image domain.

Object detection in remote sensing images is a major challenge because of the complex background and diverse appearance of remote sensing images. At the same time, targets may have different sizes and densely distributed targets in the image. To address these challenges, we propose an enhanced YOLOv8 model that integrates three key components: dynamic convolution (DyConv), which replaces the ordinary convolutional layer in the C2F module in the backbone network, and adaptively adjusts the convolution filter according to the input features to enhance the model’s ability to deal with objects of different scales and appearance. Dual level Routing Attention (BRA) was used to process the high-level features of the backbone network, suppress irrelevant background information, emphasize effective features and establish semantic correlation. Asymptotic Feature Pyramid Network (AFPN) improves multi-scale feature fusion to ensure the effective fusion of small object details in the low level and high-level semantic information. Through experiments on the Remote Sensing Object Detection (RSOD) dataset, our improved YOLOv8 model shows significant performance improvement over the original model, achieving 65.4% of the mAP50-95 score, an improvement of 3.3%. Compared with the mainstream single-stage, two-stage and DETR models, our proposed model achieves the improvement of detection accuracy while ensuring computational efficiency.

Object detection possesses extremely significant applications in the field of optical remote sensing images. A great many works have achieved remarkable results in this task. However, some common problems, such as scale, illumination, and image quality, are still unresolved. Inspired by the mechanism of cascade attention eagle-eye fovea, we propose a new attention mechanism network named the eagle-eye fovea network (EFNet) which contains two foveae for remote sensing object detection. The EFNet consists of two eagle-eye fovea modules: front central fovea (FCF) and rear central fovea (RCF). The FCF is mainly used to learn the candidate object knowledge based on the channel attention and the spatial attention, while the RCF mainly aims to predict the refined objects with two subnetworks without anchors. Three remote sensing object-detection datasets, namely DIOR, HRRSD, and AIBD, are utilized in the comparative experiments. The best results of the proposed EFNet are obtained on the HRRSD with a 0.622 AP score and a 0.907 AP50 score. The experimental results demonstrate the effectiveness of the proposed EFNet for both multi-category datasets and single category datasets.

With the wide application of remote sensing images (RSIs) in military and civil fields, remote sensing object detection (RSOD) has gradually become a hot research direction. However, we observe two main challenges for remote sensing object detection, namely the complicated background and the small objects issues. Given the different appearances of generic objects and remote sensing objects, the detection algorithms designed for the former usually cannot perform well for the latter. We propose a novel global to local scale-aware detection network (GLSANet) for remote sensing object detection, aiming to solve the above mentioned two challenges. Firstly, we design a global semantic information interaction module (GSIIM) to excavate and reinforce the high-level semantic information in the deep feature map, which alleviates the obstacles of complex background on foreground objects. Secondly, we optimize the feature pyramid network to improve the performance of multiscale object detection in RSIs. Finally, a local attention pyramid (LAP) is introduced to highlight the feature representation of small objects gradually while suppressing the background and noise in the shallower feature maps. Extensive experiments on three public datasets demonstrate that the proposed method achieves superior performance compared with the state-of-the-art detectors, especially on small object detection dataset. Specifically, our algorithm reaches 94.57% mAP on NWPU VHR-10 dataset, 95.93% mAP on RSOD dataset and 77.9% mAP on DIOR dataset, respectively.

The condition along the high-speed railway is very important for the safe operation. In the paper, man-made objects along the high-speed railway are used as detecting targets. The paper analyses the existing forms of hidden dangers which are caused by man-made objects along the high-speed railway and what harms they could bring to operation of the high-speed railway. The method of detecting man-made objects along the high-speed railway based on high resolution remote sensing images is studied using remote sensing image classification. According to the established method which is suitable for detecting man-made objects along the high-speed railway, firstly detecting and extracting man-made objects along the high-speed railway, and improving classification accuracy by GIS vector data. Then, using the method to detect man-made objects along the Xuzhou section of Beijing-Shanghai high-speed railway based on GF-3. The results show that there is hidden danger of the plastic house, which proves the effectiveness and practical significance of the method.

Oil is an important resource for the development of modern society. Accurate detection of oil wells is of great significance to the investigation of oil exploitation status and the formulation of an exploitation plan. However, detecting small objects in large-scale and high-resolution remote sensing images, such as oil wells, is a challenging task due to the problems of large number, limited pixels, and complex background. In order to overcome this problem, first, we create our own oil well dataset to conduct experiments given the lack of a public dataset. Second, we provide a comparative assessment of two state-of-the-art object detection algorithms, SSD and YOLO v4, for oil well detection in our image dataset. The results show that both of them have good performance, but YOLO v4 has better accuracy in oil well detection because of its better feature extraction capability for small objects. In view of the fact that small objects are currently difficult to be detected in large-scale and high-resolution remote sensing images, this article proposes an improved algorithm based on YOLO v4 with sliding slices and discarding edges. The algorithm effectively solves the problems of repeated detection and inaccurate positioning of oil well detection in large-scale and high-resolution remote sensing images, and the accuracy of detection result increases considerably. In summary, this study investigates an appropriate algorithm for oil well detection, improves the algorithm, and achieves an excellent effect on a large-scale and high-resolution satellite image. It provides a new idea for small objects detection in large-scale and high-resolution remote sensing images.

With the wide applications of remote sensing technology in engineering, the demands of efficient object detection algorithms for remote sensing images have also been significantly increased in recent years. Traditional detection methods have the shortcomings of low accuracy and poor robustness, which is difficult to be applied to remote sensing images with complex background and varieties of objects. Recently, the deep convolutional neural networks have already shown great advances in object detection and outperformed many traditional methods. In this work, we study the performance of a region proposal-based method, Mask R-CNN, for detecting airplane and ship in remote sensing images. Specifically, we add an FPN module to improve the accuracy of small objects detection, and a mask branch is used to describe the shape of objects. In addition, we used a series of data augmentation strategies during training for meeting the CNN’s requirement of training samples diversity. The experimental results show that our model has superior performance in object detection of remote sensing images.

At present, it is challenging to extract landslides from high-resolution remote-sensing images using deep learning. Because landslides are very complex, the accuracy of traditional extraction methods is low. To improve the efficiency and accuracy of landslide extraction, a new model is proposed based on the U-Net model to automatically extract landslides from remote-sensing images: L-Unet. The main innovations are as follows: (1) A multi-scale feature-fusion (MFF) module is added at the end of the U-Net encoding network to improve the model’s ability to extract multi-scale landslide information. (2) A residual attention network is added to the U-Net model to deepen the network and improve the model’s ability to represent landslide features. (3) The bilinear interpolation algorithm in the decoding network of the U-Net model is replaced by data-dependent upsampling (DUpsampling) to improve the quality of the feature maps. Experimental results showed that the precision, recall, MIoU and F1 values of the L-Unet model are 4.15%, 2.65%, 4.82% and 3.37% higher than that of the baseline U-Net model, respectively. It was proven that the new model can extract landslides accurately and effectively.

Semantic segmentation is a fundamental task in remote sensing image understanding. Recently, Deep Convolutional Neural Networks (DCNNs) have considerably improved the performance of the semantic segmentation of natural scenes. However, it is still challenging for very high-resolution remote sensing images. Due to the large and complex scenes as well as the influence of illumination and imaging angle, it is particularly difficult for the existing methods to accurately obtain the category of pixels at object boundaries—the so-called boundary blur. We propose a framework called Boundary-Aware Semi-Supervised Semantic Segmentation Network (BAS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> Net), which obtains more accurate segmentation results without additional annotation workload, especially at the object boundaries. The Channel-weighted Multi-scale Feature (CMF) module balances semantic and spatial information, and the Boundary Attention Module (BAM) weights the features with rich semantic boundary information to alleviate the boundary blur. Additionally, to decrease the amount of difficult and tedious manual labeling of remote sensing images, a discriminator network infers pseudolabels from unlabeled images to assist semisupervised learning, and further improves the performance of the segmentation network. To validate the effectiveness of the proposed framework, extensive experiments have been performed on both the ISPRS Vaihingen dataset, and the novel remote sensing dataset AIR-SEG with more categories, and complex boundaries. The results demonstrate a significant improvement of accuracy especially on boundaries and for small objects.