Boosting Small Ship Detection in Optical Remote Sensing Images via Image Super-Resolution

Fine-grained remote sensing ship detection is crucial in a variety of fields, such as ship safety, marine environmental protection, and maritime traffic management. Despite recent progress, current research suffers from the following three major challenges: insufficient features representation, conflicts in shared features, and inappropriate anchor labeling strategy, which significantly impede accurate fine-grained ship detection. To address these issues, we propose FineShipNet as a solution. Specifically, we first propose a novel blend synchronization module, which aims to facilitate the coutilization of semantic information from top-level and bottom-level features and minimize information redundancy. Subsequently, the blend feature maps are fed into a novel polarized feature focusing module, which decouples the features used in classification and regression to create task-specific discriminating features maps. Meanwhile, we adopt the adaptive harmony anchor labeling and propose a novel metric, harmony score, to choose high-quality anchors that can effectively capture the discriminating features of the target. Extensive experiments on four fine-grained remote sensing ship datasets (HRSC2016, DOSR, FGSD2021, and ShipRSImageNet) demonstrate that our FineShipNet outperforms current state-of-the-art object detection methods, achieving superior performance with mean average precision scores of 81.3%, 68.5%, 85.7%, and 63.9%, respectively.

In recent years, many ship detection algorithms based on convolutional neural networks (CNNs) have been proposed to improve the performance of ship detection. However, with the increase in model complexity and size, it is challenging to deploy these models to resource-constrained edge platforms. In this letter, a low coupling algorithm that belongs to anchor-free methods is proposed for ship detection to reduce the model complexity and still obtain a competitive performance. The proposed low coupling network (LCNet) is easy to deploy and contributes to speeding up the inference and improving memory utilization. In addition, we propose a model compression process consisting of the quantization-aware training (QAT) method and a structural pruning method based on Taylor expansion, which can effectively reduce the model size according to hardware resource constraints. Comparative experimental results demonstrate that LCNet outperforms the state-of-the-art ship detection and natural object detection algorithms, with a 95.27% mAP and 88.91% F1 score on the HRSC2016 dataset. Our proposed model compression method also achieves a compression ratio of at least 80% with a negligible loss of performance.

Ship detection in optical remote sensing images poses challenges such as complex scenes, large target scale differences, and imperfect fine-grained classification. Deep learning methods like YOLO have provided effective solutions. With emerging detection approaches like Transformers achieving new breakthroughs in accuracy, this study proposes an enhanced detection model by incorporating Transformer into YOLO network. The model replaces the Backbone of YOLOv5s with SwinTransformer-Tiny and refines the Loss function to Wise-IoU, tailored for remote sensing images and ship targets. Experimental verification on self-curated single-category dataset LEVIR-ship and multi-category dataset HRSC-4 shows that the comprehensive metric mAP50 values have increased from 93.8% and 96.2% to 95.2% and 96.5%, respectively. Both precision (P) and recall (R) values have significantly improved. There is also some enhancement in fine-grained classification metrics on the HRSC-4 dataset. The newly improved model demonstrates good performance.

Salient object detection in optical remote sensing images (ORSI-SOD) is a task that detects the most prominent context in optical remote sensing images (ORSIs). ORSI-SOD is challenging due to the diverse sizes and shapes of the targets, their irregular distribution in the scene, and the occlusion caused by surrounding environments. Recently, many deep learning-based models have demonstrated promising performance in ORSI-SOD. However, there still remains considerable potential for addressing these challenges in ORSI-SOD. In this article, we propose a novel architecture, LSHNet. We propose a dual-branch architecture consisting of an edge encoder that leverages structure features using edges as the structure-prior and an image encoder that extracts context features from the image. We propose three modules. Image-structure fusion module (ISFM) integrates the two-stream features extracted from dual branch encoders through intrapatch and internal-patch attention mechanisms to utilize diverse receptive fields. Local-global feature fusion module (LGFM) transfers global features representing the target to local feature maps to discriminate the region of the targets from background clutters. The semantic cues updating module (SCUM) updates the representative features of the target from high-level to low-level. By integrating hierarchical information effectively, global features extracted from multilayers can be rectified. We experiment with the three main evaluated datasets in ORSI-SOD: ORSSD, EORSSD, and ORSI-4199. We demonstrate the promising results on the three datasets and analyze the effectiveness of the proposed modules in the ablation study.

Decision letter for "Speed-Oriented Lightweight Salient Object Detection in Optical Remote Sensing Images"

In the literature of object detection in optical remote sensing images, a popular pipeline is first modifying an off-the-shelf deep neural network, then initializing the modified network by pretrained weights on a source data set, and finally fine-tuning the network on a target data set. The procedure works well in practice but might not make full use of underlying knowledge implied by pretrained weights. In this article, we propose a novel method, referred to as Fisher regularization, for efficient knowledge transferring. Based on Bayes’ theorem, the method stores underlying knowledge into a Fisher information matrix and fine-tunes parameters based on the knowledge. The proposed method would not introduce extra parameters and is less sensitive to hyperparameters than classical weight decay. Experiments on NWPUVHR-10 and DOTA data sets show that the proposed method is effective and works well with different object detectors.

Recently masked autoencoder (MAE) has achieved great success in visual representation learning and delivered promising potential in many downstream vision tasks. However, due to the lack of saliency supervision signal in original MAE, almost no saliency information can be learned from masked image reconstruction process. Therefore, salient object detection (SOD) can hardly benefit from MAE pretraining. To address this issue, we integrate SOD model and saliency supervision in MAE and propose a simple and effective framework DHMMAE utilizing MAE with dynamic hybrid masking ratios to pretrain SOD model. Specifically, we treat MAE as an online data augmenter to generate endless pseudo images for SOD model to predict saliency maps, where saliency supervision is employed to ensure the model can learn robust saliency prior knowledge from the reconstructed images. Besides, we propose a simple and novel network EANet driven by DHMMAE pretraining for SOD in optical remote sensing images (ORSIs). Two key modules are designed for EANet to further enhance the performance: enhanced diverse feature aggregation module (EDFAM) and adjacent-context shuffle spatial attention module (ASSAM). EDFAM aggregates diverse features via three different types of convolution layer and enhances them by convolution block attention module. ASSAM captures spatial location information of salient objects by employing channel shuffle operation and weighted spatial attention mechanism on the fused adjacent context. Experiments on three ORSI-SOD datasets demonstrate that our proposed method outperforms the cutting-edge methods. Code is available at: https://github.com/Voruarn/EANet.

Existing salient object detection methods for optical remote sensing images still face certain limitations due to complex background variations, significant scale discrepancies among targets, severe background interference, and diverse topological structures. On the one hand, the feature transmission process often neglects the constraints and complementary effects of high-level features on low-level features, leading to insufficient feature interaction and weakened model representation. On the other hand, decoder architectures generally rely on simple cascaded structures, which fail to adequately exploit and utilize contextual information. To address these challenges, this study proposes a Hierarchical Semantic Interaction Module to enhance salient object detection performance in optical remote sensing scenarios. The module introduces foreground content modeling and a hierarchical semantic interaction mechanism within a multi-scale feature space, reinforcing the synergy and complementarity among features at different levels. This effectively highlights multi-scale and multi-type salient regions in complex backgrounds. Extensive experiments on multiple optical remote sensing datasets demonstrate the effectiveness of the proposed method. Specifically, on the EORSSD dataset, our full model integrating both CA and PA modules improves the max F-measure from 0.8826 to 0.9100 (↑2.74%), increases maxE from 0.9603 to 0.9727 (↑1.24%), and enhances the S-measure from 0.9026 to 0.9295 (↑2.69%) compared with the baseline. These results clearly demonstrate the effectiveness of the proposed modules and verify the robustness and strong generalization capability of our method in complex remote sensing scenarios.

Ship detection and classification is critical for national maritime security and national defense. Although some SAR (Synthetic Aperture Radar) image-based ship detection approaches have been proposed and used, they are not able to satisfy the requirement of real-world applications as the number of SAR sensors is limited, the resolution is low, and the revisit cycle is long. As massive optical remote sensing images of high resolution are available, ship detection and classification on theses images is becoming a promising technique, and has attracted great attention on applications including maritime security and traffic control. Some digital image processing methods have been proposed to detect ships in optical remote sensing images, but most of them face difficulty in terms of accuracy, performance and complexity. Recently, an autoencoder-based deep neural network with extreme learning machine was proposed, but it cannot meet the requirement of real-world applications as it only works with simple and small-scaled data sets. Therefore, in this paper, we propose a novel ship detection and classification approach which utilizes deep convolutional neural network (CNN) as the ship classifier. The performance of our proposed ship detection and classification approach was evaluated on a set of images downloaded from Google Earth at the resolution 0.5m. 99% detection accuracy and 95% classification accuracy were achieved. In model training, 75× speedup is achieved on 1 Nvidia Titanx GPU.

Detecting rotated ships is difficult in optical remote sensing images due to the challenges of complex scenes. Existing advanced rotated ship detectors are typically anchor-based algorithms that require plenty of predefined anchors. However, the use of anchors brings three critical problems: 1) a large number of anchors bring a huge amount of calculation; 2) the attributes (e.g., size and aspect ratios) of anchors are designed via <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ad hoc</i> heuristics; and 3) only a tiny fraction of anchors that overlap with ground-truth bounding boxes of ships tightly can be considered as positive samples, which causes an extreme imbalance between positive and negative samples. As a result, the detection accuracy will be influenced seriously when the design of anchors is not suitable. To address the above problems, this article proposes a novel anchor-free rotated ship detection framework, called SKNet, which detects rotated ships as keypoints in optical remote sensing images. In SKNet, a ship target is modeled as its center keypoint and morphological sizes, including the width, height, and rotation angle. Accordingly, we design two customized modules: orthogonal pooling and soft-rotate-nonmaximum suppression (NMS), where the former is to improve the prediction accuracy of the center keypoint and the morphological size, and the latter is to effectively remove redundant rotated ship detection results. Extensive experiments are conducted to demonstrate the effectiveness of SKNet on three optical remote sensing image data sets: HRSC2016, DOTA-ship, and HPDM-OSOD, which is collected by ourselves and published in this article. Empirical studies show that SKNet achieves state-of-the-art detection performance while being time-efficient. Overall, SKNet achieves the best speed–accuracy tradeoff.

The recognition of sea ice is of great significance for reflecting climate change and ensuring the safety of ship navigation. Recently, many deep-learning-based methods have been proposed and applied to segment and recognize sea ice regions. However, there are huge differences in sea ice size and irregular edge profiles, which bring challenges to the existing sea ice recognition. In this article, a global-local Transformer network, called SeaIceNet, is proposed for sea ice recognition in optical remote sensing images. In SeaIceNet, a dual global-attention head (DGAH) is proposed to capture global information. On this basis, a global-local feature fusion (GLFF) mechanism is designed to fuse global structural correlation features and local spatial detail features. Furthermore, a detail-guided decoder is developed to retain more high-resolution detail information during feature reconstruction for improving the performance of sea ice recognition. Extensive experiments on several sea ice datasets demonstrated that the proposed SeaIceNet has better performance than the existing methods in multiple evaluation indicators. Moreover, it excels in addressing challenges associated with sea ice recognition in optical remote sensing images, including the difficulty in accurately identifying irregular frozen ponds in complex environments, the broken and unclear boundaries between sea and thin ice that hinder precise segmentation, and the loss of high-resolution spatial details during model learning that complicates refinement.

Applications of remote sensing images in both defense and civilian sectors have spurred substantial research interest. In the field of remote sensing, object detection confronts challenges such as complex backgrounds, scale diversity, and the presence of dense small objects. To address these issues, we propose an improved deep learning-based model, the Global Multi-scale Fusion Self-calibration Network, which is expected to contribute to alleviating the challenges. It consists of three main components: the hierarchical feature aggregation backbone, which uses improved modules such as the receptive field context-aware feature extraction module, the global information acquisition module, and the simple parameter-free attention module to extract key features and minimize the background interference. To couple multi-scale features, we enhanced the fusing component and designed the multi-scale enhanced pyramid structure integrating the proposed new modules. During the detection phase, especially when focusing on small object detection, we designed a novel convolutional attention feature fusion head. This head is constructed to integrate local and global branches for feature extraction by leveraging channel shuffling and multi-head attention mechanisms for efficient and accurate detection. Experiments on the Detection in Optical Remote Sensing Images (DIOR), Northwestern Polytechnical University Very High-Resolution‐10 (NWPU VHR‐10), remote sensing object detection (RSOD), and DOTAv1.0 data sets show that our method achieves mAP50(mean average precision at 50% intersection over union) of 69.7%, 91.3%, 94.2%, and 70.0%, respectively, outperforming existing comparative methods. The proposed network is expected to provide new perspectives for remote sensing tasks and possible solutions for relevant applications in the image domain.

Corn is an important food crop worldwide, and its yield is directly related to Chinese food security. Accurate remote sensing extraction of corn can realize the rational application of land resources, which is of great significance to the sustainable development of modern agriculture. In the field of large-scale crop remote sensing classification, single-period optical remote sensing images often cannot achieve high-precision classification. To improve classification accuracy, multiple time series image combinations have gradually been adopted. However, due to the influence of cloudy and rainy weather, it is often difficult to obtain complete time series optical images. Synthetic aperture radar (SAR) data are imaged by microwaves, which have strong penetrating power and are not affected by clouds. A critical way to solve this problem is to use SAR images to compensate for the lack of optical images and obtain a complete time series image in the corn-growing season. However, SAR images have limited wavelengths and cannot provide important wavelengths, such as visible light bands and near-infrared information. To solve this problem, this study took Zhaodong City, a vital corn-planting base in China, as the research area; took GF-6/GF-3 and Sentinel-1/Sentinel-2 as remote sensing data sources; designed12 classification scenarios; analyzed the best classification period and the best time series combination of corn classification; studied the influence of SAR images on the classification results of time series images; and compared the classification differences between GF-6/GF-3 and Sentinel-1/Sentinel-2. The results show that the classification accuracy of time series combinations is much higher than that of single-period images. The polarization characteristics of SAR images can improve the classification accuracy with time series images. The classification accuracy of GF series images from China is obviously higher than that of Sentinel series images. The research performed in this paper can provide a reference for agricultural classification by using remote sensing data.

Detecting oriented objects in optical remote sensing images has been consistently challenging due to difficulties in bounding boxes localization. The cascaded regression framework, widely employed for high-quality bounding box refinement, has demonstrated effectiveness in this domain. However, our experiments reveal a discontinuity issue in bounding box optimization in cascaded regression framework. As a result, performance gain is not guaranteed across all stages in this framework. In this paper, we propose a Distribution Discriminative Detector(DDDet) to address the above issues and enhance the optimization of bounding boxes in oriented object detection. Specifically, a novel Conditional Anchor Refinement Framework(CARF) is designed to improve cascaded regression structure. CARF distinguishes bounding boxes with different distributions, adaptively optimizing them within the well-assigned regressors. Subsequently, the Aligned Convolution Module(ACM) is integrated into each regressor, facilitating the continuous alignment between features and refined anchors. Furthermore, the Geometry-guided Training Sample Selection(GTSS) method is incorporated into CARF to assign labels based on object shape priors. Experimental results show that DDDet obtains state-of-the-art performance on mainstream datasets for oriented object detection in remote sensing image, which demonstrates the effectiveness of the proposed method. Our method surpasses many current single-stage detectors, two-stage detectors, and refine-stage detectors, achieving the mAP of 79.41% on DOTA dataset, and 44.15% on FAIR1M dataset.

ABSTRACTThe amplitude of internal waves is an important parameter for the remote-sensing detection. However, there is no analytic method developed for the amplitude inversion of internal waves based on optical remote-sensing images. In this article, the deep-learning model is introduced to inverse the amplitude of internal waves based on a large number of optical remote-sensing images. The peak-to-peak distance and 15 types of texture characteristic parameters of images are computed, and the relationship between the amplitude of internal waves and the characteristic parameters of remote-sensing images are also investigated. In addition, based on the correlation difference between these parameters and the amplitude, three types of deep-learning model are established by selecting different parameters as input variables. Results show that the inverted amplitude of internal waves from our models shows good agreement with the in-situ data of internal waves, and the average errors between them are about 6.7%, 4.9%, and 3.6%, respectively, which indicates that the deep-learning model is effective for amplitude inversion of internal waves based on optical remote-sensing images.