RGB-D salient object detection based on BC2 FNet network

Salient object detection, which simulates human visual perception in locating the most significant object(s) in a scene, has been widely applied to various computer vision tasks. Now, the advent of depth sensors means that depth maps can easily be captured; this additional spatial information can boost the performance of salient object detection. Although various RGB-D based salient object detection models with promising performance have been proposed over the past several years, an in-depth understanding of these models and the challenges in this field remains lacking. In this paper, we provide a comprehensive survey of RGB-D based salient object detection models from various perspectives, and review related benchmark datasets in detail. Further, as light fields can also provide depth maps, we review salient object detection models and popular benchmark datasets from this domain too. Moreover, to investigate the ability of existing models to detect salient objects, we have carried out a comprehensive attribute-based evaluation of several representative RGB-D based salient object detection models. Finally, we discuss several challenges and open directions of RGB-D based salient object detection for future research. All collected models, benchmark datasets, datasets constructed for attribute-based evaluation, and related code are publicly available at https://github.com/taozh2017/RGBD-SODsurvey.

A pooling-based feature pyramid network for salient object detection

Neurobiology researches suggest that the motion information attracts more attention of human visual system than other low-level features such as brightness, color and texture. Consequently, video saliency detection methods not only consider the spatial saliency caused by the underlying features of images, but also the motion information in temporal domain. In this study, we proposes a model of video salient object detection based on a two-pathway framework that the spatio-temporal contrast guides the search for salient targets. Firstly, along the non-selective pathway, which is computed with the intra-frame and inter-frame maps of the color contrast and motion contrast, combining with the previous saliency map, to represent the prior information of the possible target locations. In contrast, the low-level features such as brightness, color and motion features are extracted in the selective pathway to search target accurately. Finally, the Bayesian inference is used to further obtain the optimal results. Experimental results show that our algorithm improves the performance of salient object detection on video compared to the representative method of Contour Guided Visual Search.

Improving RGB-D salient object detection by addressing inconsistent saliency problems

Salient Object Detection (SOD) in RGB-D images plays a crucial role in the field of computer vision, with its central aim being to identify and segment the most visually striking objects within a scene. However, optimizing the fusion of multi-modal and multi-scale features to enhance detection performance remains a challenge. To address this issue, we propose a network model based on semantic localization and multi-scale fusion (SLMSF-Net), specifically designed for RGB-D SOD. Firstly, we designed a Deep Attention Module (DAM), which extracts valuable depth feature information from both channel and spatial perspectives and efficiently merges it with RGB features. Subsequently, a Semantic Localization Module (SLM) is introduced to enhance the top-level modality fusion features, enabling the precise localization of salient objects. Finally, a Multi-Scale Fusion Module (MSF) is employed to perform inverse decoding on the modality fusion features, thus restoring the detailed information of the objects and generating high-precision saliency maps. Our approach has been validated across six RGB-D salient object detection datasets. The experimental results indicate an improvement of 0.20~1.80%, 0.09~1.46%, 0.19~1.05%, and 0.0002~0.0062, respectively in maxF, maxE, S, and MAE metrics, compared to the best competing methods (AFNet, DCMF, and C2DFNet).

PATNet: Patch-to-pixel attention-aware transformer network for RGB-D and RGB-T salient object detection

Attention and boundary guided salient object detection

Exploiting long-range semantic contexts and geometric information is crucial to infer salient objects from RGB and depth features. However, existing methods mainly focus on excavating local features within fixed regions by continuously feeding forward networks. In this article, we introduce Dynamic Message Propagation (DMP) to dynamically learn context information within more flexible regions. We integrate DMP into a Siamese-based network to process the RGB image and depth map separately and design a multi-level feature fusion module to explore cross-level information between refined RGB and depth features. Extensive experiments show clear improvements of our method over 17 methods on six benchmark datasets for RGB-D salient object detection (SOD). Additionally, our method outperforms its competitors for the video SOD task. Code is available at https://github.com/chenbaian-cs/DMPNet .

Perceptual localization and focus refinement network for RGB-D salient object detection

Mobile devices usually mount a depth sensor to resolve ill-posed problems, like salient object detection on cluttered background. The main barrier of exploring RGBD data is to handle the information from two different modalities. To cope with this problem, in this paper, we propose a boundary-aware cross-modal fusion network for RGBD salient object detection. In particular, to enhance the fusion of color and depth features, we present a cross-modal feature sampling module to balance the contribution of the RGB and depth features based on the statistics of their channel values. In addition, in our multi-scale dense fusion network architecture, we not only incorporate edge-sensitive losses to preserve the boundary of the detected salient region, but also refine its structure by merging the estimated saliency maps of different scales. We accomplish the multi-scale saliency map merging using two alternative methods which produce refined saliency maps via per-pixel weighted combination and an encoder-decoder network. Extensive experimental evaluations demonstrate that our proposed framework can achieve the state-of-the-art performance on several public RGBD-based datasets.

We present an effective method to progressively integrate and refine the cross-modality complementarities for RGB-D salient object detection (SOD). The proposed network mainly solves two challenging issues: 1) how to effectively integrate the complementary information from RGB image and its corresponding depth map, and 2) how to adaptively select more saliency-related features. First, we propose a cross-modality feature modulation (cmFM) module to enhance feature representations by taking the depth features as prior, which models the complementary relations of RGB-D data. Second, we propose an adaptive feature selection (AFS) module to select saliency-related features and suppress the inferior ones. The AFS module exploits multi-modality spatial feature fusion with the self-modality and cross-modality interdependencies of channel features are considered. Third, we employ a saliency-guided position-edge attention (sg-PEA) module to encourage our network to focus more on saliency-related regions. The above modules as a whole, called cmMS block, facilitates the refinement of saliency features in a coarse-to-fine fashion. Coupled with a bottom-up inference, the refined saliency features enable accurate and edge-preserving SOD. Extensive experiments demonstrate that our network outperforms state-of-the-art saliency detectors on six popular RGB-D SOD benchmarks.

Under ideal environmental conditions, RGB-based deep convolutional neural networks can achieve high performance for salient object detection (SOD). In scenes with cluttered backgrounds and many objects, depth maps have been combined with RGB images to better distinguish spatial positions and structures during SOD, achieving high accuracy. However, under low-light and uneven lighting conditions, RGB and depth information may be insufficient for detection. Thermal images are insensitive to lighting and weather conditions, being able to capture important objects even during nighttime. By combining thermal images and RGB images, we propose an effective and consistent feature fusion network (ECFFNet) for RGB-T SOD. In ECFFNet, an effective cross-modality fusion module fully fuses features of corresponding sizes from the RGB and thermal modalities. Then, a bilateral reversal fusion module performs bilateral fusion of foreground and background information, enabling the full extraction of salient object boundaries. Finally, a multilevel consistent fusion module combines features across different levels to obtain complementary information. Comprehensive experiments on three RGB-T SOD datasets show that the proposed ECFFNet outperforms 12 state-of-the-art methods under different evaluation indicators.

Salient object detection (SOD) has been attracting a lot of interest, and recently many computational models have been developed. Here, the authors formulate a SOD model, in which saliency map is computed as a combination of the colour, its distribution‐based saliency and orientation saliency. Similar to traditional SODs, the proposed method is based on super‐pixel segmentation and super‐pixel utilises both colour and its distribution‐based saliency to generate a coarse saliency map. However, distinct from traditional SODs, authors further use orientation contrast to optimise the coarse saliency map to obtain an improved saliency map. Authors’ contributions are twofold. First, the authors combine colour uniqueness and its distribution with local orientation information (LOI) used in Itti's model to effectively improve profiles of salient regions. Second, a reciprocal function is defined to substitute the Gabor function used in LOI, and the authors have proved that the substitution could detect relatively homogeneous and uniform regions at the boundary of salient object, whereas it is what the traditional models lack. Authors’ approach significantly outperforms state‐of‐the‐art methods on four benchmark datasets, while the authors demonstrate that the proposed method runs as fast as most existing algorithms.

Most existing RGB-D salient object detection (SOD) methods extract features of both modalities in parallel or adopt depth features as supplementary information for unidirectional interaction from depth modality to RGB modality in the encoder stage. These methods ignore the influence of low-quality depth maps, and there is still room for improvement in effectively fusing RGB features and depth features. To address the above problems, this paper proposes a Feature Interaction Network (FINet), which performs bi-directional interaction through feature interaction module (FIM) in the encoder stage. The feature interaction module is divided into two parts: depth enhancement module (DEM) filters the noise in the depth features through the attention mechanism; and cross enhancement module (CEM) effectively interacts RGB features and depth features. In addition, this paper proposes a two-stage cross-modal fusion strategy: high-level fusion adopts the semantic information of high level for coarse localization of salient regions, and low-level fusion makes full use of the detailed information of low level through boundary fusion, and then we progressively refine high-level and low-level cross-modal features to obtain the final saliency prediction map. Extensive experiments show that the proposed model achieves better performance than eight state-of-the-art models on five standard datasets.

The performance of the salient object detection of strip surface defects has been promoted largely by deep learning based models. However, due to the complexity of strip surface defects, the existing models perform poorly in the challenging scenes such as noise disturbance, and low contrast between defect regions and background. Meanwhile, the detection results of existing models often suffer from coarse boundary details. Therefore, we propose a novel saliency model, namely an Edge-aware Multi-level Interactive Network, to detect the defects from the strip steel surface. Concretely, our model adopts the U-shape architecture where the two crucial points are the interactive feature integration and the edge-guided saliency fusion. Firstly, except the skip connection that combines the same stage of encoder and decoder, we deploy another connection, where the features from adjacent levels of encoder are transferred to the same stage of decoder. By this way, we are able to provide an effective fusion of multi-level deep features, yielding a well depiction for defects. Secondly, to give well-defined boundaries for prediction results, we add the edge extraction branch after each decoder block, where the progressive feature aggregation endows the edge with precise details and complete object cues. Meanwhile, together with the edge extraction branches, we deploy the saliency prediction branch at each decoder stage. After that, coupled with the fine edge information, we fuse all outputs of saliency prediction branches into the final saliency map, where the edge cue steers the saliency result to pay more attention to the boundary details. Following this way, we can provide a high-quality saliency map which can accurately locate and segment the defects. Extensive experiments are performed on the public dataset, and the results prove the effectiveness and robustness of our model which consistently outperforms the state-of-the-art models.