Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks.

We provide a comprehensive evaluation of salient object detection (SOD) models. Our analysis identifies a serious design bias of existing SOD datasets which assumes that each image contains at least one clearly outstanding salient object in low clutter. The design bias has led to a saturated high performance for state-of-the-art SOD models when evaluated on existing datasets. The models, however, still perform far from being satisfactory when applied to real-world daily scenes. Based on our analyses, we first identify 7 crucial aspects that a comprehensive and balanced dataset should fulfill. Then, we propose a new high quality dataset and update the previous saliency benchmark. Specifically, our SOC (Salient Objects in Clutter) dataset, includes images with salient and non-salient objects from daily object categories. Beyond object category annotations, each salient image is accompanied by attributes that reflect common challenges in real-world scenes. Finally, we report attribute-based performance assessment on our dataset.

Nowadays, ship detection in sea-sky background is not only useful in maritime visual surveillance, but also helpful in maritime search and rescue. Since ships are salient objects in infrared images with sea-sky background, we present a novel and effective algorithm based on saliency for ship detection in this situation. Our algorithm adopts global saliency, local saliency and background prior to generate saliency maps. Ships are finally segmented in saliency maps. Our algorithm is compared with four classic salient object detection algorithms. And experimental results show our algorithm outperforms the other four algorithms in qualitative and quantitative terms. Keyword list: ship detection ; salient object detection ; infrared image ; sea - sky background . 1. INTRODUCTION Since ships are important vehicles for sea transportation, ship detection in sea-sky background is extensively used in maritime visual surveillance and reconnaissance, avoidance of ship collision accidents, maritime search and rescue, etc. Infrared imaging systems can almost work in any environment and all-weather conditions with its visual capacity in darkness and frog, which makes it a very suitable tool in this application. In recent years, ship detection in infrared images with sea-sky background attracts increasing attention. Many methods are proposed in this field, including image segmentation methods [1-2] and morphological methods [3]. However, due to complex sea-sky environment and large variation of ship sizes, these methods cannot solve the issue completely. And since ships are often salient objects in infrared images with sea-sky background, some researchers have recently tried salient object detection ideas used in color images [4-6]. And the paper is focused on using salient object detection methods to solve the issue. Salient objects are objects which stand out in the image and attract most people ¶VLPPHGLDWHDWWHQWLRQ7KHVWDQGDUGway to detect salient ob MHFWVLVILUVWO\XVLQJVDOLHQF\GHWHFWLRQPHWKRGVWRJHQHUDWHVDOLHQF\PDSVLQZKLFKDSL[HO¶VJUH\value is proportional to its degree of saliency and then segmenting salient objects in saliency maps. Many salient detection methods can be divided into three categories: local contrast methods, global rarity methods and other methods. In local contrast methods, regions which have high contrast with local neighbouring regions are regarded as salient regions. Itti et al. [7] decompose an image into several feature maps . The saliency in each feature map is measured by local center-surround difference in several scales and saliency of different feature maps are integrated to a final saliency

Due to the rapid development of deep learning, salient object detection has achieved promising results in recent years. However, how to construct a powerful saliency detection network to generate saliency maps with completely highlighted salient objects and effectively suppressed background noise still remains an open and challenging problem. In this paper, we propose a novel trifurcated cascaded refinement network (TCRNet) to explore multi-level feature fusion and global information representation. Specifically, all the side-output features from the backbone network are divided into three branches (i.e., global information, deep information and shallow information) according to feature similarities, and then the global contexts, semantic knowledge and fined details respectively generated by the three branches are integrated in a cascaded refinement way. This refinement strategy aims to guide the feature learning of important regions and suppress background noises. Moreover, we propose a cross-level feature fusion (CFF) module to fuse multi-level similar features within a same branch by exploring their interrelationship, thus capturing the important semantic information and structure details. To this end, the complementarity of similar features is utilized to refine feature at each level by sharing an encoded spatial weight map. Besides, we design a context-aware feature learning (CFL) module to capture global contexts in a single layer by taking account of both single-feature representation and multi-feature interaction, thus providing important location information of salient objects. Extensive experiments on five benchmarks are conducted to demonstrate that the proposed network TCRNet performs favorably against 20 state-of-the-art salient object detection methods on five benchmark datasets, indicating the effectiveness and superiority of our method.

MFENet: Multitype fusion and enhancement network for detecting salient objects in RGB-T images

Real-world scenes always exhibit objects with clutter backgrounds, posing great challenges for deep salient object detection models. In this paper, we propose salient object detection by engaging two saliency cues, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the part-whole hierarchies and contrast cues, resulting in a PWHCNet. Specifically, two branches, which consists of a Dynamic Grouping Capsules (DGC) branch and a DenseHRNet branch, are put in place to learn the part-whole hierarchies and contrast cues, respectively. Moreover, to help highlight the whole salient object in complex scenes, a Background Suppression (BS) module is proposed to guide the shallow features of DenseHRNet with the aid of the part-whole relational cues captured by DGC. Subsequently, these two saliency cues are integrated via a Self-Channel and Mutual-Spatial (SCMS) attention mechanism. Experimental results on five benchmarks demonstrate that the proposed PWHCNet achieves state-of-the-art performance while obtaining the whole salient objects with fine details.

Salient object detection (SOD) has been attracting a lot of interest, and recently many computational models have been developed. Here, the authors formulate a SOD model, in which saliency map is computed as a combination of the colour, its distribution‐based saliency and orientation saliency. Similar to traditional SODs, the proposed method is based on super‐pixel segmentation and super‐pixel utilises both colour and its distribution‐based saliency to generate a coarse saliency map. However, distinct from traditional SODs, authors further use orientation contrast to optimise the coarse saliency map to obtain an improved saliency map. Authors’ contributions are twofold. First, the authors combine colour uniqueness and its distribution with local orientation information (LOI) used in Itti's model to effectively improve profiles of salient regions. Second, a reciprocal function is defined to substitute the Gabor function used in LOI, and the authors have proved that the substitution could detect relatively homogeneous and uniform regions at the boundary of salient object, whereas it is what the traditional models lack. Authors’ approach significantly outperforms state‐of‐the‐art methods on four benchmark datasets, while the authors demonstrate that the proposed method runs as fast as most existing algorithms.

Salient object detection aims to detect the most attractive objects in images, which has been widely used as a fundamental of various multimedia applications. In this paper, we propose a novel salient object detection method for RGB-D images based on evolution strategy. Firstly, we independently generate two saliency maps on color channel and depth channel of a given RGB-D image based on its super-pixels representation. Then, we fuse the two saliency maps with refinement to provide an initial saliency map with high precision. Finally, we utilize cellular automata to iteratively propagate saliency on the initial saliency map and generate the final detection result with complete salient objects. The proposed method is evaluated on two public RGB-D datasets, and the experimental results show that our method outperforms the state-of-the-art methods.

Despite Convolutional Neural Networks (CNNs) based methods have been successful in detecting salient objects, their underlying mechanism that decides the salient intensity of each image part separately cannot avoid inconsistency of parts within the same salient object. This would ultimately result in an incomplete shape of the detected salient object. To solve this problem, we dig into part-object relationships and take the unprecedented attempt to employ these relationships endowed by the Capsule Network (CapsNet) for salient object detection. The entire salient object detection system is built directly on a Two-Stream Part-Object Assignment Network (TSPOANet) consisting of three algorithmic steps. In the first step, the learned deep feature maps of the input image are transformed to a group of primary capsules. In the second step, we feed the primary capsules into two identical streams, within each of which low-level capsules (parts) will be assigned to their familiar high-level capsules (object) via a locally connected routing. In the final step, the two streams are integrated in the form of a fully connected layer, where the relevant parts can be clustered together to form a complete salient object. Experimental results demonstrate the superiority of the proposed salient object detection network over the state-of-the-art methods.&#13;\n

<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> salient object detection has recently received much attention for 3D scene perception owing to its omnidirectional field of view (FoV). The capability of recognizing salient objects of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> images remains technically challenging due to severe spherical distortion. In this paper, we develop a hierarchical uncertainty-aware <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> image salient object detection methodology that explicitly explores the geometric and spatial complementary coherence of Tangent projection (TP) and Equirectangular projection (ERP) by a collaborative learning strategy. Concretely, to mitigate spherical distortion, we first intend to learn saliency-related features from less-distorted tangent images, in which a deformation-aware attention block is introduced to mitigate the geometric distortion caused by projecting a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> image onto a 2D plane. However, the discrepancies among tangent images pose a new challenge to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> image salient object detection. To tackle this issue and achieve accurate localization for salient objects of all sizes, we design a spatial-frequency saliency feature aggregation module to leverage fast Fourier convolution to capture global contextual information from ERP images, such that obtaining more representative saliency features. Moreover, a hierarchical uncertainty-aware bi-projection consistency learning module with strong local-global information embedding capabilities is constructed, which learns the geometric and spatial correlations between tangent images and ERP images via a collaborative learning strategy. Ultimately, salient object maps are produced for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> images on the basis of the merged saliency features driven by the uncertainty. Extensive experiments show that our developed method improves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\mathrm{F}}_\beta ^{\sigma }$</tex-math></inline-formula> by an average of 31.67% compared to twenty existing advanced methods on the publicly available 360-SOD dataset.

Salient object detection, which simulates human visual perception in locating the most significant object(s) in a scene, has been widely applied to various computer vision tasks. Now, the advent of depth sensors means that depth maps can easily be captured; this additional spatial information can boost the performance of salient object detection. Although various RGB-D based salient object detection models with promising performance have been proposed over the past several years, an in-depth understanding of these models and the challenges in this field remains lacking. In this paper, we provide a comprehensive survey of RGB-D based salient object detection models from various perspectives, and review related benchmark datasets in detail. Further, as light fields can also provide depth maps, we review salient object detection models and popular benchmark datasets from this domain too. Moreover, to investigate the ability of existing models to detect salient objects, we have carried out a comprehensive attribute-based evaluation of several representative RGB-D based salient object detection models. Finally, we discuss several challenges and open directions of RGB-D based salient object detection for future research. All collected models, benchmark datasets, datasets constructed for attribute-based evaluation, and related code are publicly available at https://github.com/taozh2017/RGBD-SODsurvey.

Accurate localization of the salient object from an image is a difficult problem when the saliency map is noisy and incomplete. A fast approach to detect salient objects from images is proposed in this paper. To well balance the size of the object and the saliency it contains, the salient object detection is first formulated with the maximum saliency density on the saliency map. To obtain the global optimal solution, a branch-and-bound search algorithm is developed to speed up the detection process. Without any prior knowledge provided, the proposed method can effectively and efficiently detect salient objects from images. Extensive results on different types of saliency maps with a public dataset of five thousand images show the advantages of our approach as compared to some state-of-the-art methods.

Salient object detection is evaluated using binary ground truth (GT) with the labels being salient object class and background. In this study, the authors corroborate based on three subjective experiments on a novel image dataset that objects in natural images are inherently perceived to have varying levels of importance. The authors' dataset, named SalMoN (saliency in multi-object natural images), has 588 images containing multiple objects. The subjective experiments performed record spontaneous attention and perception through eye fixation duration, point clicking and rectangle drawing. As object saliency in a multi-object image is inherently multi-level, they propose that salient object detection must be evaluated for the capability to detect all multi-level salient objects apart from the salient object class detection capability. For this purpose, they generate multi-level maps as GT corresponding to all the dataset images using the results of the subjective experiments, with the labels being multi-level salient objects and background. They then propose the use of mean absolute error, Kendall's rank correlation and average area under precision–recall curve to evaluate existing salient object detection methods on their multi-level saliency GT dataset. Approaches that represent saliency detection on images as local-global hierarchical processing of a graph perform well in their dataset.

In the face of complex scene images, the introduction of depth information can greatly improve the performance of salient object detection. However, up-sampling and down-sampling operations in neural networks maybe blur the boundaries of objects in the saliency map, thereby reducing the performance of salient object detection. Aiming at this problem, a boundary-driven cross-modal and cross-layer fusion network (BC2FNet) for RGB-D salient object detection is proposed in this paper, which preserves the boundary of the object by adding the guidance of boundary information to the cross-modal and cross-layer fusion, respectively. Firstly, a boundary generation module is designed to extract two kinds of boundary information from low-level features of RGB and depth modalities, respectively. Secondly, a boundary-driven feature selection module is designed, which is dedicated to simultaneously focusing on important feature information and preserving boundary details in the process of RGB and depth modality fusion. Finally, a boundary-driven cross-layer fusion module is proposed which simultaneously adds two kinds of boundary information in the process of up-sampling fusion on adjacent layers. By embedding this module into the top-down information fusion flow, the predicted saliency map can contain accurate objects and sharp boundaries. Simulation results on five standard RGB-D data sets show that the proposed model can achieve better performance.

PATNet: Patch-to-pixel attention-aware transformer network for RGB-D and RGB-T salient object detection

Salient object detection is a fundamental problem in both pattern recognition and image processing tasks. Previous salient object detection algorithms usually involve various features based on priors/assumptions about the properties of the objects. Inspired by the effectiveness of recently developed feature learning, we propose a novel deep salient object detection (DSOD) model using the deep residual network (ResNet 152-layers) for saliency computation. In particular, we model the image saliency from both local and global perspectives. In the local feature estimation stage, we detect local saliency by using a deep residual network (ResNet-L) which learns local region features to determine the saliency value of each pixel. In the global feature extraction stage, another deep residual network (ResNet-G) is trained to predict the saliency score of each image based on the global features. The final saliency map is generated by a conditional random field (CRF) to combining the local and global-level saliency map. Our DSOD model is capable of uniformly highlighting the objects-of-interest from complex background while well preserving object details. Quantitative and qualitative experiments on three benchmark datasets demonstrate that our DSOD method outperforms state-of-the-art methods in the salient object detection.