Distilling Knowledge From Super-Resolution for Efficient Remote Sensing Salient Object Detection

Salient object detection (SOD) task for optical remote sensing images (RSIs) plays an important role in many remote sensing applications. Most of the existing methods train their networks depending on a large amount of pixel-wise datasets. However, such expensive and time-consuming training setting prevents the approaches becoming flexible and scalable solutions. To this end, we explore efficient SOD for optical RSIs based on easily accessible weak supervision source. In this work, we propose a novel end-to-end progressive attention-based feature recovery framework with scribble supervision. Specifically, to better locate challenging salient objects in optical RSIs, an object position module (OPM) is proposed to capture and enhance the long-range semantic dependence of objects’ position information, which depends on the complementary attention mechanism. And to restore the entire salient objects, a context refinement module (CRM) is proposed, which extract local contextual information for better propagating high-level semantics to low-level details. Moreover, to improve the adaptability of the network to the changing scenarios of optical RSIs, we propose a salient region correcting (SRC) mechanism to help the predicted salient regions rectify their saliency values by constraining the saliency relationship between predictions from different augmentation models. In addition, due to the lack of dataset for weakly supervised SOD for optical RSIs, we relabeled an existing large-scale optical RSIs dataset with scribbles, namely EORSSD-S. Experimental results on benchmark datasets demonstrate that the proposed method can outperform other weakly supervised SOD methods. And the proposed method even outperformed some fully supervised methods. https://github.com/melonless/PAFR.

Aggregating transformers and CNNs for salient object detection in optical remote sensing images

Recently masked autoencoder (MAE) has achieved great success in visual representation learning and delivered promising potential in many downstream vision tasks. However, due to the lack of saliency supervision signal in original MAE, almost no saliency information can be learned from masked image reconstruction process. Therefore, salient object detection (SOD) can hardly benefit from MAE pretraining. To address this issue, we integrate SOD model and saliency supervision in MAE and propose a simple and effective framework DHMMAE utilizing MAE with dynamic hybrid masking ratios to pretrain SOD model. Specifically, we treat MAE as an online data augmenter to generate endless pseudo images for SOD model to predict saliency maps, where saliency supervision is employed to ensure the model can learn robust saliency prior knowledge from the reconstructed images. Besides, we propose a simple and novel network EANet driven by DHMMAE pretraining for SOD in optical remote sensing images (ORSIs). Two key modules are designed for EANet to further enhance the performance: enhanced diverse feature aggregation module (EDFAM) and adjacent-context shuffle spatial attention module (ASSAM). EDFAM aggregates diverse features via three different types of convolution layer and enhances them by convolution block attention module. ASSAM captures spatial location information of salient objects by employing channel shuffle operation and weighted spatial attention mechanism on the fused adjacent context. Experiments on three ORSI-SOD datasets demonstrate that our proposed method outperforms the cutting-edge methods. Code is available at: https://github.com/Voruarn/EANet.

The optical remote sensing images (RSIs) show various spatial resolutions and cluttered background, where salient objects with different scales, types, and orientations are presented in diverse RSI scenes. Therefore, it is inappropriate to directly extend cutting-edge saliency detection methods for conventional RGB images to optical RSIs. Besides, the existing saliency models targeting RSIs often render imperfect saliency maps, where some of them are with coarse boundary details. To solve this problem, this article attempts to introduce the edge information to precisely detect salient objects in RSIs. Accordingly, we propose an edge-aware multiscale feature integration network (EMFI-Net) for salient object detection by conducting multiscale feature integration under the explicit and implicit assistance of salient edge cues. Specifically, our network contains two parts including the encoder and decoder. First, the encoder extracts multiscale deep features from three RSIs with different resolutions, where the high-level deep semantic features from three RSIs are integrated using a cascaded feature fusion module. Second, the encoder explicitly enriches the multiscale deep features by integrating the salient edge cues extracted by a salient edge extraction module. Meanwhile, we also implicitly deploy an edge-aware constraint to the supervision of the saliency map prediction by introducing a hybrid loss function. Finally, the decoder integrates the enriched multiscale deep features in a coarse-to-fine way, yielding a high-quality saliency map. The experiments conducted on two public optical RSI datasets clearly prove the effectiveness and superiority of the proposed EMFI-Net against the state-of-the-art saliency models.

Nowadays, ship detection in sea-sky background is not only useful in maritime visual surveillance, but also helpful in maritime search and rescue. Since ships are salient objects in infrared images with sea-sky background, we present a novel and effective algorithm based on saliency for ship detection in this situation. Our algorithm adopts global saliency, local saliency and background prior to generate saliency maps. Ships are finally segmented in saliency maps. Our algorithm is compared with four classic salient object detection algorithms. And experimental results show our algorithm outperforms the other four algorithms in qualitative and quantitative terms. Keyword list: ship detection ; salient object detection ; infrared image ; sea - sky background . 1. INTRODUCTION Since ships are important vehicles for sea transportation, ship detection in sea-sky background is extensively used in maritime visual surveillance and reconnaissance, avoidance of ship collision accidents, maritime search and rescue, etc. Infrared imaging systems can almost work in any environment and all-weather conditions with its visual capacity in darkness and frog, which makes it a very suitable tool in this application. In recent years, ship detection in infrared images with sea-sky background attracts increasing attention. Many methods are proposed in this field, including image segmentation methods [1-2] and morphological methods [3]. However, due to complex sea-sky environment and large variation of ship sizes, these methods cannot solve the issue completely. And since ships are often salient objects in infrared images with sea-sky background, some researchers have recently tried salient object detection ideas used in color images [4-6]. And the paper is focused on using salient object detection methods to solve the issue. Salient objects are objects which stand out in the image and attract most people ¶VLPPHGLDWHDWWHQWLRQ7KHVWDQGDUGway to detect salient ob MHFWVLVILUVWO\XVLQJVDOLHQF\GHWHFWLRQPHWKRGVWRJHQHUDWHVDOLHQF\PDSVLQZKLFKDSL[HO¶VJUH\value is proportional to its degree of saliency and then segmenting salient objects in saliency maps. Many salient detection methods can be divided into three categories: local contrast methods, global rarity methods and other methods. In local contrast methods, regions which have high contrast with local neighbouring regions are regarded as salient regions. Itti et al. [7] decompose an image into several feature maps . The saliency in each feature map is measured by local center-surround difference in several scales and saliency of different feature maps are integrated to a final saliency

Existing salient object detection methods for optical remote sensing images still face certain limitations due to complex background variations, significant scale discrepancies among targets, severe background interference, and diverse topological structures. On the one hand, the feature transmission process often neglects the constraints and complementary effects of high-level features on low-level features, leading to insufficient feature interaction and weakened model representation. On the other hand, decoder architectures generally rely on simple cascaded structures, which fail to adequately exploit and utilize contextual information. To address these challenges, this study proposes a Hierarchical Semantic Interaction Module to enhance salient object detection performance in optical remote sensing scenarios. The module introduces foreground content modeling and a hierarchical semantic interaction mechanism within a multi-scale feature space, reinforcing the synergy and complementarity among features at different levels. This effectively highlights multi-scale and multi-type salient regions in complex backgrounds. Extensive experiments on multiple optical remote sensing datasets demonstrate the effectiveness of the proposed method. Specifically, on the EORSSD dataset, our full model integrating both CA and PA modules improves the max F-measure from 0.8826 to 0.9100 (↑2.74%), increases maxE from 0.9603 to 0.9727 (↑1.24%), and enhances the S-measure from 0.9026 to 0.9295 (↑2.69%) compared with the baseline. These results clearly demonstrate the effectiveness of the proposed modules and verify the robustness and strong generalization capability of our method in complex remote sensing scenarios.

Salient object detection in optical remote sensing images (ORSI-SOD) is a task that detects the most prominent context in optical remote sensing images (ORSIs). ORSI-SOD is challenging due to the diverse sizes and shapes of the targets, their irregular distribution in the scene, and the occlusion caused by surrounding environments. Recently, many deep learning-based models have demonstrated promising performance in ORSI-SOD. However, there still remains considerable potential for addressing these challenges in ORSI-SOD. In this article, we propose a novel architecture, LSHNet. We propose a dual-branch architecture consisting of an edge encoder that leverages structure features using edges as the structure-prior and an image encoder that extracts context features from the image. We propose three modules. Image-structure fusion module (ISFM) integrates the two-stream features extracted from dual branch encoders through intrapatch and internal-patch attention mechanisms to utilize diverse receptive fields. Local-global feature fusion module (LGFM) transfers global features representing the target to local feature maps to discriminate the region of the targets from background clutters. The semantic cues updating module (SCUM) updates the representative features of the target from high-level to low-level. By integrating hierarchical information effectively, global features extracted from multilayers can be rectified. We experiment with the three main evaluated datasets in ORSI-SOD: ORSSD, EORSSD, and ORSI-4199. We demonstrate the promising results on the three datasets and analyze the effectiveness of the proposed modules in the ablation study.

Existing methods for burned area segmentation (BAS) in optical remote sensing images (ORSIs) mainly adopt convolution neural network (CNN) as the backbone, which has limited receptive filed and suffers from long-term dependencies problem. To address this issue, we propose a novel salient object detection (SOD) network (DBINet) based on dual interactive convolution-transformer backbone for BAS in ORSIs. DBINet combines the benefits of CNN and transformer: CNN is good at extracting local spatial information while transformer does well in modeling long-term dependencies. The core component of DBINet is three newly designed modules: dual-feature fusion module (DFM), Convertor and a novel decoder. Specifically, DFM is proposed to bridge two different backbones. Convertor is designed to fuse the multi-scale coarse features from the main encoder and produce the fined feature for the decoder. The decoder has a multi-level feature aggregating process and a self-refining process, which restores the resolution and generates the prediction results. Experiments on three datasets demonstrate that our DBINet outperforms the state-of-the-art methods, and achieves the best S-measure on three datasets: 0.847, 0.888 and 0.883. Code is available at: https://github.com/Voruarn/DBINet.

Applications of remote sensing images in both defense and civilian sectors have spurred substantial research interest. In the field of remote sensing, object detection confronts challenges such as complex backgrounds, scale diversity, and the presence of dense small objects. To address these issues, we propose an improved deep learning-based model, the Global Multi-scale Fusion Self-calibration Network, which is expected to contribute to alleviating the challenges. It consists of three main components: the hierarchical feature aggregation backbone, which uses improved modules such as the receptive field context-aware feature extraction module, the global information acquisition module, and the simple parameter-free attention module to extract key features and minimize the background interference. To couple multi-scale features, we enhanced the fusing component and designed the multi-scale enhanced pyramid structure integrating the proposed new modules. During the detection phase, especially when focusing on small object detection, we designed a novel convolutional attention feature fusion head. This head is constructed to integrate local and global branches for feature extraction by leveraging channel shuffling and multi-head attention mechanisms for efficient and accurate detection. Experiments on the Detection in Optical Remote Sensing Images (DIOR), Northwestern Polytechnical University Very High-Resolution‐10 (NWPU VHR‐10), remote sensing object detection (RSOD), and DOTAv1.0 data sets show that our method achieves mAP50(mean average precision at 50% intersection over union) of 69.7%, 91.3%, 94.2%, and 70.0%, respectively, outperforming existing comparative methods. The proposed network is expected to provide new perspectives for remote sensing tasks and possible solutions for relevant applications in the image domain.

There are two main areas in saliency detection: fixation prediction and salient object detection. Researchers always use bottom-up low-level features and straightforward rules for salient object detection. However these rules are unreliable in common scenes. As a result, most salient object detection technologies can gain a fine performance in limited situation. While in a more complex scene (such as PASCAL-S dataset), they may miss the really wanted salient objects. Fixation prediction is a technology of predicting the regions which can attract human's attention. Most fixation prediction technologies are based on the discoveries of neuroscience and cognitive psychology. Their rules for fixation prediction more accord with human's sensory mechanism. Thus, fixation map can provide a more reliable salient priori. Based on above reasons, we propose a novel salient object detection method with fixation priori. We refine fixation map to gain a reliable priori, and conduct salient object detection based on the priori. We perform experiment on two datasets, and find that our method outperforms the comparative state-of-art salient object detection technologies.

The application of optical remote sensing images (ORSIs) is prevalent in many fields. Accordingly, ORSI-oriented salient object detection (SOD) has attracted more attention in recent years. However, yet many previously proposed methods present appealing performance in natural scene images (NSIs), they are difficult to be directly extended to remote sensing images due to the more complex scenes, such as blended backgrounds and diversiform topological shapes. Most specifically designed models often fail to achieve satisfactory results due to the weak usage of edge information and the ignorance of attention loss. Besides, computational inefficiency often causes poor applicability. To solve these problems, we propose a new model, namely, Bidimensional Attention and Full-stage Semantic Guidance Network (BAFS-Net), containing an edge guidance branch and a mainstream detection branch. Concretely, edge guidance generates boundary information, in which supervision with border labels is imposed to highlight the salient regions and plays a complementary role on the main branch. The mainstream detection branch involves two important components, i.e., bidimensional attention modules (BAMs) and semantic-guided fusion modules (SGFMs). Between these two, BAM uniformly assembles channel and spatial attention in an efficient and rational manner, addressing the open issue of dimensionwisely attention computation. SGFM hammers at the fusion of high-level features and low-level features. Moreover, the semantic maps are employed to interact with SGFM in full stages. Our approach surpasses most state-of-the-art RSI-SOD methods proposed in recent years, with respect to the accuracy, parameter size, computational cost, and floating point operations per second (FLOPS). The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ZhengJianwei2/BAFS-Net</uri> .

Salient object detection is evaluated using binary ground truth (GT) with the labels being salient object class and background. In this study, the authors corroborate based on three subjective experiments on a novel image dataset that objects in natural images are inherently perceived to have varying levels of importance. The authors' dataset, named SalMoN (saliency in multi-object natural images), has 588 images containing multiple objects. The subjective experiments performed record spontaneous attention and perception through eye fixation duration, point clicking and rectangle drawing. As object saliency in a multi-object image is inherently multi-level, they propose that salient object detection must be evaluated for the capability to detect all multi-level salient objects apart from the salient object class detection capability. For this purpose, they generate multi-level maps as GT corresponding to all the dataset images using the results of the subjective experiments, with the labels being multi-level salient objects and background. They then propose the use of mean absolute error, Kendall's rank correlation and average area under precision–recall curve to evaluate existing salient object detection methods on their multi-level saliency GT dataset. Approaches that represent saliency detection on images as local-global hierarchical processing of a graph perform well in their dataset.

Salient object detection has drawn much attention recently. The extracted salient objects can be used for tasks including recognition, segmentation and retrieval. Most existing models for saliency detection can only be applied to the high contrast and normal contrast scenes. They are not robust enough to handle objects in low contrast images. Due to low lightness, low Signal to Noise Ratio (SNR) and low appearance information, saliency detection in low contrast scenes remains a challenge. To solve this problem, this paper proposes a saliency detection model based on the feature extraction, in which optimal features have been learned in different contrast environment to improve the effectiveness of salient object detection. Compared with other existing saliency models, the proposed method has strong adaptability for different scenes, and gives superior performance in detecting the salient object especially in low contrast images.

Although great progress has been made in Salient Object Detection (SOD) in Optical Remote Sensing Images (ORSIs), it still faces critical challenges, particularly in dealing with irregular topological structures and complex contextual relationships. To address these issues, we propose a Progressive Interaction and Saliency-guided Enhancement Network (PISENet). Specifically, a Progressive Interaction Encoder (PIE) is proposed, which adopts a dual-path heterogeneous fusion architecture and a hierarchical progressive interaction mechanism to capture global irregular topological structures and local fine-grained image details. Meanwhile, it mitigates semantic gaps across multi-scale features and achieves effective cross-level feature fusion. Subsequently, a Global Context Enhancement Module (GCEM) is designed, which incorporates non-local blocks to enhance spatial correlations among features. A parallel multi-branch structure is further utilized to capture multi-level contextual information ranging from local details to long-range semantics, thereby strengthening the modeling of global context. Finally, a Multi-scale Progressive Attention Enhancement Decoder (MPAED) is devised, which adopts a saliency-guided attention mechanism to jointly model spatial and channel-wise dependencies, enhance responses in salient regions and boundaries, and progressively decode and aggregate deep semantic and shallow detailed features. Extensive experiments on three benchmark datasets demonstrate that our method achieves significant superiority over state-of-the-art approaches.

We provide a comprehensive evaluation of salient object detection (SOD) models. Our analysis identifies a serious design bias of existing SOD datasets which assumes that each image contains at least one clearly outstanding salient object in low clutter. The design bias has led to a saturated high performance for state-of-the-art SOD models when evaluated on existing datasets. The models, however, still perform far from being satisfactory when applied to real-world daily scenes. Based on our analyses, we first identify 7 crucial aspects that a comprehensive and balanced dataset should fulfill. Then, we propose a new high quality dataset and update the previous saliency benchmark. Specifically, our SOC (Salient Objects in Clutter) dataset, includes images with salient and non-salient objects from daily object categories. Beyond object category annotations, each salient image is accompanied by attributes that reflect common challenges in real-world scenes. Finally, we report attribute-based performance assessment on our dataset.