Multi-Projection Fusion and Refinement Network for Salient Object Detection in 360° Omnidirectional Image.

A pooling-based feature pyramid network for salient object detection

Given that existing salient object detection methods cannot effectively predict the fine contours of salient objects when extracting local or global contexts and features, we propose a novel contour self-compensated network (CSCNet) to generate a more accurate saliency map with complete contour. Unlike the common binary saliency detection, we reconstruct the salient object detection problem into a multi-classification problem of the background, the salient object, and the salient contour, where the salient contour is used as the third label for ground truth. Meanwhile, the image and its superpixel map are concatenated as the input of our network to add more edge information. Also, a penalty loss is proposed to restrict the spatial relationship between the background, objects, and their contours. Experimentally, we evaluate the proposed CSCNet on six benchmark datasets in both accuracy and efficiency and evaluate the attribute-based performance on the SOC dataset. Compared with 13 state-of-the-art algorithms, our CSCNet can detect salient objects more accurately and completely without adding too many convolutional layers and parameters.

Attention and boundary guided salient object detection

Salient object detection (SOD) has achieved remarkable performance in well‐illuminated scenarios. However, in inevitable low‐light environments, SOD faces inherent challenges such as insufficient illumination, noise interference, and contrast degradation, which severely compromise the perceptual accuracy of visual models. To address these issues, we propose a novel Edge‐Guided Dual‐Stream Collaborative Network (EGDSNet), which enhances the integrity of salient objects through multi‐level feature interaction between edge detection and segmentation branches. Specifically, EGDSNet adopts a dual‐branch architecture: the edge detection branch integrates a Local Context Refinement (LCR) module and a Cross‐Modal Edge Refinement (CER) module to suppress noise while reinforcing boundary responses of targets. The segmentation branch employs a Global Feature Fusion (GFF) module and an Edge‐Guided Feature Enhancement (EFE) module to achieve complementary integration of global semantics and local details. The two branches interact via an edge‐guided attention mechanism, enabling saliency predictions to maintain structural integrity while attaining precise boundary localization. Additionally, the Residual Channel Attention (RCA) module is leveraged to further suppress noise and extract structural details from low‐level features. Comprehensive experiments on the LLI and SOD‐LL datasets demonstrate that our model outperforms state‐of‐the‐art methods in both quantitative metrics and qualitative visualizations, validating its superiority in low‐light SOD tasks.

Many existing salient object detection methods are dedicated to fusing features from different levels of a pre-trained convolutional neural network (CNN). However, these methods can easily lead to internal discontinuities within the salient objects because of unreasonable feature fusion strategies and short-range dependencies resulting from common convolution and pooling operations. In this paper, we propose a novel non-local duplicate pooling (NLDP) network to overcome these internal discontinuities. NLDP begins by removing the first few convolutional layers of a classic CNN, which have small receptive fields and require large amounts of calculation. A novel duplicate pooling module (DPM) is then used to generate richer and more detailed saliency maps. This is achieved by constructing a double-pathway that can integrating partial feature maps. Within the DPM, a non-local module (NLM) is used to obtain long-range dependencies. This enhances the internal continuities between the saliency maps. Comprehensive experiments conducted on six benchmark datasets have confirmed the increased effectiveness and detection speed of our method in relation to other salient object detection methods.

<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> salient object detection has recently received much attention for 3D scene perception owing to its omnidirectional field of view (FoV). The capability of recognizing salient objects of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> images remains technically challenging due to severe spherical distortion. In this paper, we develop a hierarchical uncertainty-aware <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> image salient object detection methodology that explicitly explores the geometric and spatial complementary coherence of Tangent projection (TP) and Equirectangular projection (ERP) by a collaborative learning strategy. Concretely, to mitigate spherical distortion, we first intend to learn saliency-related features from less-distorted tangent images, in which a deformation-aware attention block is introduced to mitigate the geometric distortion caused by projecting a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> image onto a 2D plane. However, the discrepancies among tangent images pose a new challenge to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> image salient object detection. To tackle this issue and achieve accurate localization for salient objects of all sizes, we design a spatial-frequency saliency feature aggregation module to leverage fast Fourier convolution to capture global contextual information from ERP images, such that obtaining more representative saliency features. Moreover, a hierarchical uncertainty-aware bi-projection consistency learning module with strong local-global information embedding capabilities is constructed, which learns the geometric and spatial correlations between tangent images and ERP images via a collaborative learning strategy. Ultimately, salient object maps are produced for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> images on the basis of the merged saliency features driven by the uncertainty. Extensive experiments show that our developed method improves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\mathrm{F}}_\beta ^{\sigma }$</tex-math></inline-formula> by an average of 31.67% compared to twenty existing advanced methods on the publicly available 360-SOD dataset.

Perceptual localization and focus refinement network for RGB-D salient object detection

Recent progress in salient object detection (SOD) mainly depends on the Atrous Spatial Pyramid Pooling (ASPP) module for multi-scale learning. Intuitively, different input images, different pixels, and different network layers may have different preferences for various feature scales. However, ASPP treats all feature scales as equally important by a simple sum operation. To this end, we propose Attentive Atrous Spatial Pyramid Pooling (A2SPP) by adding a new Cubic Information-Embedding Attention (CIEA) module at each branch of ASPP. In this way, each position in the 3D feature map can automatically learn the feature scales it prefers. Specifically, CIEA consists of Spatial-Embedding Channel Attention (SECA) and Channel-Embedding Spatial Attention (CESA). Instead of the previous direct squeeze and ignoring of one dimension when computing the attention for the other dimension, SECA/CESA attempts to embed spatial/channel information into channel/spatial attention, respectively. In addition, CIEA learns SECA and CESA for each 3D position simultaneously rather than previous separate computation of channel and spatial attention for each 2D position. Incorporating A2SPP and CIEA, the proposed A2SPPNet performs favorably against previous state-of-the-art SOD methods.

Despite Convolutional Neural Networks (CNNs) based methods have been successful in detecting salient objects, their underlying mechanism that decides the salient intensity of each image part separately cannot avoid inconsistency of parts within the same salient object. This would ultimately result in an incomplete shape of the detected salient object. To solve this problem, we dig into part-object relationships and take the unprecedented attempt to employ these relationships endowed by the Capsule Network (CapsNet) for salient object detection. The entire salient object detection system is built directly on a Two-Stream Part-Object Assignment Network (TSPOANet) consisting of three algorithmic steps. In the first step, the learned deep feature maps of the input image are transformed to a group of primary capsules. In the second step, we feed the primary capsules into two identical streams, within each of which low-level capsules (parts) will be assigned to their familiar high-level capsules (object) via a locally connected routing. In the final step, the two streams are integrated in the form of a fully connected layer, where the relevant parts can be clustered together to form a complete salient object. Experimental results demonstrate the superiority of the proposed salient object detection network over the state-of-the-art methods.&#13;\n

There are several formats to describe the omnidirectional images. Among them, equirectangular projection (ERP), represented as 2D image, is the most widely used format. There exist many outstanding methods capable of well predicting the saliency maps for the conventional 2D images. But these works cannot be directly extended to predict the saliency map of the ERP image, since the content on ERP is not for direct display. Instead, the viewport image on demand is generated after converting the ERP image to the sphere domain, followed by rectilinear projection. In this paper, we propose a model to predict the saliency maps of the ERP images using existing saliency predictors for the 2D image. Some pre-processing and post-processing are used to manage the problem mentioned above. In particular, a smoothing based optimization is realized on the sphere domain. A public dataset of omnidirectional images is used to perform all the experiments and competitive results are achieved.

Omnidirectional images have attracted significant attention in recent years due to the rapid development of virtual reality technologies. Equirectangular projection (ERP), a naive form to store and transfer omnidirectional images, however, is challenging for existing two-dimensional (2D) image super-resolution (SR) methods due to its inhomogeneous distributed sampling density and distortion across latitude. In this paper, we make one of the first attempts to design a spherical pseudo-cylindrical representation, which not only allows pixels at different latitudes to adaptively adopt the best distinct sampling density but also is model-agnostic to most off-the-shelf SR methods, enhancing their performances. Specifically, we start by upsampling each latitude of the input ERP image and design a computationally tractable optimization algorithm to adaptively obtain a (sub)-optimal sampling density for each latitude of the ERP image. Addressing the distortion of ERP, we introduce a new viewport-based training loss based on the original 3D sphere format of the omnidirectional image, which inherently lacks distortion. Finally, we present a simple yet effective recursive progressive omnidirectional SR network to showcase the feasibility of our idea. The experimental results on public datasets demonstrate the effectiveness of the proposed method as well as the consistently superior performance of our method over most state-of-the-art methods both quantitatively and qualitatively.

Salient object detection network with multi-scale feature refinement and boundary feedback

MFENet: Multitype fusion and enhancement network for detecting salient objects in RGB-T images

Salient object detection in complex scenes and environments is a challenging research topic. Most works focus on RGB-based salient object detection, which limits its performance of real-life applications when confronted with adverse conditions such as dark environments and complex backgrounds. Taking advantage of RGB and thermal infrared(RGBT) images becomes a new research direction for detecting salient objects in complex scenes, since the thermal infrared spectrum provides the complementary information and has been used in many computer vision tasks. However, current research for RGBT salient object detection is limited by the lack of a large-scale dataset and comprehensive benchmark. This work contributes such a RGBT image dataset named VT5000, including 5000 spatially aligned RGBT image pairs with ground truth annotations. VT5000 has 11 challenges collected in different scenes and environments for exploring the robustness of algorithms. With this dataset, we propose a powerful baseline approach, which extracts multilevel features of each modality and aggregates these features of all modalities with the attention mechanism for accurate RGBT salient object detection. To further solve the problem of blur boundaries of salient objects, we also use an edge loss to refine the boundaries. Extensive experiments show that the proposed baseline approach outperforms the state-of-the-art methods on VT5000 dataset and other two public datasets. In addition, we carry out a comprehensive analysis of different algorithms of RGBT salient object detection on VT5000 dataset, and then make several valuable conclusions and provide some potential research directions for RGBT salient object detection. Our new VT5000 dataset is made publicly available at https://github.com/lz118/RGBT-Salient-Object-Detection.

Beauty and Personal care product retrieval has attracted more and more attention due to its wide application value. However, due to the diversity of data and the complexity of image background, this task is very challenging. In this paper, we propose a multi-feature fusion method based on salient object detection to improve retrieval performance. The key of our method is to extract the foreground objects of the query set by using the salient object detection network, so as to eliminate the background interference. Then the foreground target images and dataset are put into the multi-classification networks to extract multiple fusion features for retrieval. We use the perfect-500k dataset for experiments, and the results show that our method is effective. Our method ranked 2st in the Grand Challenge of AI Meets Beauty in ACM Multimedia 2020 with a MAP score of 0.43729. We released our code on GitHub:github.com/R-M-Yan/ACMMM2020AIMeetBeauty.