Progressive Attention-Based Feature Recovery With Scribble Supervision for Saliency Detection in Optical Remote Sensing Image

Aggregating transformers and CNNs for salient object detection in optical remote sensing images

Recently masked autoencoder (MAE) has achieved great success in visual representation learning and delivered promising potential in many downstream vision tasks. However, due to the lack of saliency supervision signal in original MAE, almost no saliency information can be learned from masked image reconstruction process. Therefore, salient object detection (SOD) can hardly benefit from MAE pretraining. To address this issue, we integrate SOD model and saliency supervision in MAE and propose a simple and effective framework DHMMAE utilizing MAE with dynamic hybrid masking ratios to pretrain SOD model. Specifically, we treat MAE as an online data augmenter to generate endless pseudo images for SOD model to predict saliency maps, where saliency supervision is employed to ensure the model can learn robust saliency prior knowledge from the reconstructed images. Besides, we propose a simple and novel network EANet driven by DHMMAE pretraining for SOD in optical remote sensing images (ORSIs). Two key modules are designed for EANet to further enhance the performance: enhanced diverse feature aggregation module (EDFAM) and adjacent-context shuffle spatial attention module (ASSAM). EDFAM aggregates diverse features via three different types of convolution layer and enhances them by convolution block attention module. ASSAM captures spatial location information of salient objects by employing channel shuffle operation and weighted spatial attention mechanism on the fused adjacent context. Experiments on three ORSI-SOD datasets demonstrate that our proposed method outperforms the cutting-edge methods. Code is available at: https://github.com/Voruarn/EANet.

Current state-of-the-art remote sensing salient object detectors always require high-resolution spatial context to ensure excellent performance, which incurs enormous computation costs and hinders real-time efficiency. In this work, we propose a universal super-resolution assisted learning (SRAL) framework to boost performance and accelerate the inference efficiency of existing approaches. To this end, we propose to reduce the spatial resolution of the input remote sensing images (RSIs), which is model-agnostic, and can be applied to existing algorithms without extra computation cost. Specifically, a transposed saliency detection decoder (TSDD) is designed to upsample interim features progressively. On top of it, an auxiliary super-resolution decoder (ASRD) is proposed to build a multitask learning (MTL) framework to investigate an efficient complementary paradigm of saliency detection and super-resolution. Furthermore, a novel task-fusion guidance module (TFGM) is proposed to effectively distill domain knowledge from the super-resolution auxiliary task to the salient object detection task in optical RSIs. The presented ASRD and TFGM can be omitted in the inference phase without any extra computational budget. Extensive experiments on three datasets show that the presented SRAL with 224×224 input is superior to more than 20 algorithms. Moreover, it can be successfully generalized to existing typical networks with significant accuracy improvements in a parameter-free manner. Codes and models are available at https://github.com/lyf0801/SRAL.

Salient object detection in optical remote sensing images (ORSI-SOD) is a task that detects the most prominent context in optical remote sensing images (ORSIs). ORSI-SOD is challenging due to the diverse sizes and shapes of the targets, their irregular distribution in the scene, and the occlusion caused by surrounding environments. Recently, many deep learning-based models have demonstrated promising performance in ORSI-SOD. However, there still remains considerable potential for addressing these challenges in ORSI-SOD. In this article, we propose a novel architecture, LSHNet. We propose a dual-branch architecture consisting of an edge encoder that leverages structure features using edges as the structure-prior and an image encoder that extracts context features from the image. We propose three modules. Image-structure fusion module (ISFM) integrates the two-stream features extracted from dual branch encoders through intrapatch and internal-patch attention mechanisms to utilize diverse receptive fields. Local-global feature fusion module (LGFM) transfers global features representing the target to local feature maps to discriminate the region of the targets from background clutters. The semantic cues updating module (SCUM) updates the representative features of the target from high-level to low-level. By integrating hierarchical information effectively, global features extracted from multilayers can be rectified. We experiment with the three main evaluated datasets in ORSI-SOD: ORSSD, EORSSD, and ORSI-4199. We demonstrate the promising results on the three datasets and analyze the effectiveness of the proposed modules in the ablation study.

Existing salient object detection methods for optical remote sensing images still face certain limitations due to complex background variations, significant scale discrepancies among targets, severe background interference, and diverse topological structures. On the one hand, the feature transmission process often neglects the constraints and complementary effects of high-level features on low-level features, leading to insufficient feature interaction and weakened model representation. On the other hand, decoder architectures generally rely on simple cascaded structures, which fail to adequately exploit and utilize contextual information. To address these challenges, this study proposes a Hierarchical Semantic Interaction Module to enhance salient object detection performance in optical remote sensing scenarios. The module introduces foreground content modeling and a hierarchical semantic interaction mechanism within a multi-scale feature space, reinforcing the synergy and complementarity among features at different levels. This effectively highlights multi-scale and multi-type salient regions in complex backgrounds. Extensive experiments on multiple optical remote sensing datasets demonstrate the effectiveness of the proposed method. Specifically, on the EORSSD dataset, our full model integrating both CA and PA modules improves the max F-measure from 0.8826 to 0.9100 (↑2.74%), increases maxE from 0.9603 to 0.9727 (↑1.24%), and enhances the S-measure from 0.9026 to 0.9295 (↑2.69%) compared with the baseline. These results clearly demonstrate the effectiveness of the proposed modules and verify the robustness and strong generalization capability of our method in complex remote sensing scenarios.

The optical remote sensing images (RSIs) show various spatial resolutions and cluttered background, where salient objects with different scales, types, and orientations are presented in diverse RSI scenes. Therefore, it is inappropriate to directly extend cutting-edge saliency detection methods for conventional RGB images to optical RSIs. Besides, the existing saliency models targeting RSIs often render imperfect saliency maps, where some of them are with coarse boundary details. To solve this problem, this article attempts to introduce the edge information to precisely detect salient objects in RSIs. Accordingly, we propose an edge-aware multiscale feature integration network (EMFI-Net) for salient object detection by conducting multiscale feature integration under the explicit and implicit assistance of salient edge cues. Specifically, our network contains two parts including the encoder and decoder. First, the encoder extracts multiscale deep features from three RSIs with different resolutions, where the high-level deep semantic features from three RSIs are integrated using a cascaded feature fusion module. Second, the encoder explicitly enriches the multiscale deep features by integrating the salient edge cues extracted by a salient edge extraction module. Meanwhile, we also implicitly deploy an edge-aware constraint to the supervision of the saliency map prediction by introducing a hybrid loss function. Finally, the decoder integrates the enriched multiscale deep features in a coarse-to-fine way, yielding a high-quality saliency map. The experiments conducted on two public optical RSI datasets clearly prove the effectiveness and superiority of the proposed EMFI-Net against the state-of-the-art saliency models.

Although great progress has been made in Salient Object Detection (SOD) in Optical Remote Sensing Images (ORSIs), it still faces critical challenges, particularly in dealing with irregular topological structures and complex contextual relationships. To address these issues, we propose a Progressive Interaction and Saliency-guided Enhancement Network (PISENet). Specifically, a Progressive Interaction Encoder (PIE) is proposed, which adopts a dual-path heterogeneous fusion architecture and a hierarchical progressive interaction mechanism to capture global irregular topological structures and local fine-grained image details. Meanwhile, it mitigates semantic gaps across multi-scale features and achieves effective cross-level feature fusion. Subsequently, a Global Context Enhancement Module (GCEM) is designed, which incorporates non-local blocks to enhance spatial correlations among features. A parallel multi-branch structure is further utilized to capture multi-level contextual information ranging from local details to long-range semantics, thereby strengthening the modeling of global context. Finally, a Multi-scale Progressive Attention Enhancement Decoder (MPAED) is devised, which adopts a saliency-guided attention mechanism to jointly model spatial and channel-wise dependencies, enhance responses in salient regions and boundaries, and progressively decode and aggregate deep semantic and shallow detailed features. Extensive experiments on three benchmark datasets demonstrate that our method achieves significant superiority over state-of-the-art approaches.

Existing methods for burned area segmentation (BAS) in optical remote sensing images (ORSIs) mainly adopt convolution neural network (CNN) as the backbone, which has limited receptive filed and suffers from long-term dependencies problem. To address this issue, we propose a novel salient object detection (SOD) network (DBINet) based on dual interactive convolution-transformer backbone for BAS in ORSIs. DBINet combines the benefits of CNN and transformer: CNN is good at extracting local spatial information while transformer does well in modeling long-term dependencies. The core component of DBINet is three newly designed modules: dual-feature fusion module (DFM), Convertor and a novel decoder. Specifically, DFM is proposed to bridge two different backbones. Convertor is designed to fuse the multi-scale coarse features from the main encoder and produce the fined feature for the decoder. The decoder has a multi-level feature aggregating process and a self-refining process, which restores the resolution and generates the prediction results. Experiments on three datasets demonstrate that our DBINet outperforms the state-of-the-art methods, and achieves the best S-measure on three datasets: 0.847, 0.888 and 0.883. Code is available at: https://github.com/Voruarn/DBINet.

Nowadays, ship detection in sea-sky background is not only useful in maritime visual surveillance, but also helpful in maritime search and rescue. Since ships are salient objects in infrared images with sea-sky background, we present a novel and effective algorithm based on saliency for ship detection in this situation. Our algorithm adopts global saliency, local saliency and background prior to generate saliency maps. Ships are finally segmented in saliency maps. Our algorithm is compared with four classic salient object detection algorithms. And experimental results show our algorithm outperforms the other four algorithms in qualitative and quantitative terms. Keyword list: ship detection ; salient object detection ; infrared image ; sea - sky background . 1. INTRODUCTION Since ships are important vehicles for sea transportation, ship detection in sea-sky background is extensively used in maritime visual surveillance and reconnaissance, avoidance of ship collision accidents, maritime search and rescue, etc. Infrared imaging systems can almost work in any environment and all-weather conditions with its visual capacity in darkness and frog, which makes it a very suitable tool in this application. In recent years, ship detection in infrared images with sea-sky background attracts increasing attention. Many methods are proposed in this field, including image segmentation methods [1-2] and morphological methods [3]. However, due to complex sea-sky environment and large variation of ship sizes, these methods cannot solve the issue completely. And since ships are often salient objects in infrared images with sea-sky background, some researchers have recently tried salient object detection ideas used in color images [4-6]. And the paper is focused on using salient object detection methods to solve the issue. Salient objects are objects which stand out in the image and attract most people ¶VLPPHGLDWHDWWHQWLRQ7KHVWDQGDUGway to detect salient ob MHFWVLVILUVWO\XVLQJVDOLHQF\GHWHFWLRQPHWKRGVWRJHQHUDWHVDOLHQF\PDSVLQZKLFKDSL[HO¶VJUH\value is proportional to its degree of saliency and then segmenting salient objects in saliency maps. Many salient detection methods can be divided into three categories: local contrast methods, global rarity methods and other methods. In local contrast methods, regions which have high contrast with local neighbouring regions are regarded as salient regions. Itti et al. [7] decompose an image into several feature maps . The saliency in each feature map is measured by local center-surround difference in several scales and saliency of different feature maps are integrated to a final saliency

This paper studies audio-visual salient object detection. The task of salient object detection is to detect and mark the objects that are most concerned by people in the visual scene. Traditionally, visual salient object detection uses only images or video frames to detect salient objects, without modeling human multi-modal perception which includes the interaction between vision and hearing. Therefore, in order to improve the visual salient object detection, we incorporate audio modality into the traditional visual salient object detection task by applying a two-stream audio-visual deep learning network. To this end, we also build an audio-visual salient object detection dataset called AVSOD based on the existing dataset. To verify the effectiveness of audio modality in salient object detection, we compare the experimental performance of the deep learning model with and without audio modality. The experimental results demonstrate that audio modality has a good supplementary effect on the task of visual salient object detection, and also verified the effectiveness of the proposed dataset.

The application of optical remote sensing images (ORSIs) is prevalent in many fields. Accordingly, ORSI-oriented salient object detection (SOD) has attracted more attention in recent years. However, yet many previously proposed methods present appealing performance in natural scene images (NSIs), they are difficult to be directly extended to remote sensing images due to the more complex scenes, such as blended backgrounds and diversiform topological shapes. Most specifically designed models often fail to achieve satisfactory results due to the weak usage of edge information and the ignorance of attention loss. Besides, computational inefficiency often causes poor applicability. To solve these problems, we propose a new model, namely, Bidimensional Attention and Full-stage Semantic Guidance Network (BAFS-Net), containing an edge guidance branch and a mainstream detection branch. Concretely, edge guidance generates boundary information, in which supervision with border labels is imposed to highlight the salient regions and plays a complementary role on the main branch. The mainstream detection branch involves two important components, i.e., bidimensional attention modules (BAMs) and semantic-guided fusion modules (SGFMs). Between these two, BAM uniformly assembles channel and spatial attention in an efficient and rational manner, addressing the open issue of dimensionwisely attention computation. SGFM hammers at the fusion of high-level features and low-level features. Moreover, the semantic maps are employed to interact with SGFM in full stages. Our approach surpasses most state-of-the-art RSI-SOD methods proposed in recent years, with respect to the accuracy, parameter size, computational cost, and floating point operations per second (FLOPS). The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ZhengJianwei2/BAFS-Net</uri> .

The main aim of optical remote sensing image (ORSI) object detection is to ascertain the area as well as class of all targets in given images. Nowadays, object detection methods established from deep learning are progressively applied to the analysis of the information from ORSI. But because of the compound background as well as extensive object scale shown in ORSI, the difficulty of object detection has increased. There are still some challenges to be solved. First, when the two bounding boxes don't overlap, current indexes like intersection over union (IoU) cannot identify the distance between them. Secondly, the existing methods using convolutional neural network (CNN) could not take advantage of the characteristics of multi-level features so that these methods are not very good at recognizing small-sized objects. To deal with these above problems, this paper comes up with a ORSI object detection method established on multi-level feature fusion as well as improved boundary box regression compared to IoU. Firstly, a new measure called generalized intersection over union (GIoU) is practiced. That metric could measure the similarity betwixt two bounding boxes, regardless of whether they overlap or not. At the same time, we also directly use GIoU as loss. Lastly, a multi-level feature fusion structure is practiced and it is also integrated into the existing convolutional neural network. In this way, our method could take advantage of the multi-level features. Quantitative comparison of the addressed method with the baseline approach on large-scale dataset named Dior are conducted. The comparison with the most advanced method shows that our method has reached the most advanced performance.

Despite Convolutional Neural Networks (CNNs) based methods have been successful in detecting salient objects, their underlying mechanism that decides the salient intensity of each image part separately cannot avoid inconsistency of parts within the same salient object. This would ultimately result in an incomplete shape of the detected salient object. To solve this problem, we dig into part-object relationships and take the unprecedented attempt to employ these relationships endowed by the Capsule Network (CapsNet) for salient object detection. The entire salient object detection system is built directly on a Two-Stream Part-Object Assignment Network (TSPOANet) consisting of three algorithmic steps. In the first step, the learned deep feature maps of the input image are transformed to a group of primary capsules. In the second step, we feed the primary capsules into two identical streams, within each of which low-level capsules (parts) will be assigned to their familiar high-level capsules (object) via a locally connected routing. In the final step, the two streams are integrated in the form of a fully connected layer, where the relevant parts can be clustered together to form a complete salient object. Experimental results demonstrate the superiority of the proposed salient object detection network over the state-of-the-art methods.&#13;\n

We provide a comprehensive evaluation of salient object detection (SOD) models. Our analysis identifies a serious design bias of existing SOD datasets which assumes that each image contains at least one clearly outstanding salient object in low clutter. The design bias has led to a saturated high performance for state-of-the-art SOD models when evaluated on existing datasets. The models, however, still perform far from being satisfactory when applied to real-world daily scenes. Based on our analyses, we first identify 7 crucial aspects that a comprehensive and balanced dataset should fulfill. Then, we propose a new high quality dataset and update the previous saliency benchmark. Specifically, our SOC (Salient Objects in Clutter) dataset, includes images with salient and non-salient objects from daily object categories. Beyond object category annotations, each salient image is accompanied by attributes that reflect common challenges in real-world scenes. Finally, we report attribute-based performance assessment on our dataset.

Decision letter for "Speed-Oriented Lightweight Salient Object Detection in Optical Remote Sensing Images"