Saliency detection in complex scenes

Perceptual localization and focus refinement network for RGB-D salient object detection

In this paper, a novel saliency detection algorithm is proposed to fuse both the background and foreground information while detecting salient objects in complex scenes. Firstly, we extract background seeds as well as their spatial information from image borders to construct a background-based saliency map. Then, an optimal contour closure is selected as the foreground region according to the first-stage saliency map. The optimal contour closure can provide a preferable description for salient object. We compute a foreground-based saliency map using the selected foreground region and integrate it with the background-based one. Finally, the unified saliency map is further refined to obtain a more accurate result. Experimental results show that the proposed algorithm can achieve favorable performance compared to the state-of-the-art ones.

Salient object detection in complex scenes and environments is a challenging research topic. Most works focus on RGB-based salient object detection, which limits its performance of real-life applications when confronted with adverse conditions such as dark environments and complex backgrounds. Taking advantage of RGB and thermal infrared(RGBT) images becomes a new research direction for detecting salient objects in complex scenes, since the thermal infrared spectrum provides the complementary information and has been used in many computer vision tasks. However, current research for RGBT salient object detection is limited by the lack of a large-scale dataset and comprehensive benchmark. This work contributes such a RGBT image dataset named VT5000, including 5000 spatially aligned RGBT image pairs with ground truth annotations. VT5000 has 11 challenges collected in different scenes and environments for exploring the robustness of algorithms. With this dataset, we propose a powerful baseline approach, which extracts multilevel features of each modality and aggregates these features of all modalities with the attention mechanism for accurate RGBT salient object detection. To further solve the problem of blur boundaries of salient objects, we also use an edge loss to refine the boundaries. Extensive experiments show that the proposed baseline approach outperforms the state-of-the-art methods on VT5000 dataset and other two public datasets. In addition, we carry out a comprehensive analysis of different algorithms of RGBT salient object detection on VT5000 dataset, and then make several valuable conclusions and provide some potential research directions for RGBT salient object detection. Our new VT5000 dataset is made publicly available at https://github.com/lz118/RGBT-Salient-Object-Detection.

Real-world scenes always exhibit objects with clutter backgrounds, posing great challenges for deep salient object detection models. In this paper, we propose salient object detection by engaging two saliency cues, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the part-whole hierarchies and contrast cues, resulting in a PWHCNet. Specifically, two branches, which consists of a Dynamic Grouping Capsules (DGC) branch and a DenseHRNet branch, are put in place to learn the part-whole hierarchies and contrast cues, respectively. Moreover, to help highlight the whole salient object in complex scenes, a Background Suppression (BS) module is proposed to guide the shallow features of DenseHRNet with the aid of the part-whole relational cues captured by DGC. Subsequently, these two saliency cues are integrated via a Self-Channel and Mutual-Spatial (SCMS) attention mechanism. Experimental results on five benchmarks demonstrate that the proposed PWHCNet achieves state-of-the-art performance while obtaining the whole salient objects with fine details.

Compared with RGB images, light field images contain more abundant visual information, which is helpful to accurately detect salient objects in complex scenes. However, most of the existing light field saliency detection methods use single light field data or do not fully consider the differences and complementarities between different light field data, resulting in insufficient multi-modal fusion. To address these issues, a multi-modal feature fusion network is proposed, which makes full use of the rich visual information in the light field images to realize the accurate saliency object detection. The proposed network consists of two parallel subnets, which are used to process the micro-lens image array and all-foucs image respectively. Then the light field refinement module is used to refine the feature map extracted from the micro-lens array stream, and finally the multi-modal feature fusion is realized by the light field attention module to predict saliency objects more accurately. In order to verify the effectiveness of proposed method, extensive comparison with several existing light field saliency detection algorithms is carried on both Lytro-Illum and LFSD datasets. Experimental results show that the proposed method is superior to others in all evaluation metrics on Lytro-Illum dataset, and has desired generalization abilities on LFSD dataset.

A pooling-based feature pyramid network for salient object detection

Salient object detection in video using deep non-local neural networks

Effective infrared small target detection is still challenging due to small target sizes and the clutter in the background. Unfortunately, many advanced methods do not perform well in preserving and detecting multi-scale objects in complex scenes. We propose an infrared small targets method to suppress the background and adapt the infrared small targets with different sizes. Based on the singular value analysis in the facet model, we propose a multi-derivative descriptor to enhance the targets and suppress various clutter in the dual derivative channels. In the first-order derivative channel, we design four facet kernels with different directions to enhance and preserve the isotropic small targets and suppress the block clutter. In the second-order derivative channel, we use the facet kernel to enhance the center pixels of targets and suppress the band clutter. In order to adapt to the targets with various sizes, we propose a constrained random walker technique, including an adaptive matching algorithm to extract the local regions of each candidate adaptively based on the constraint of size and shape. The experimental results demonstrate that the proposed method can accurately detect multi-scale small targets in complex scenes, resulting in better detection performance than the state-of-the-art methods.

Open-vocabulary object detection is an innovative computer vision task capable of widely recognizing and localizing various objects in images. Unlike traditional methods, it can handle diverse object categories and is suitable for real-time applications in dynamic environments. Existing methods typically achieve zero-shot detection capabilities by fusing images and text. However, when semantic discrepancies exist between text and images, biased prediction problems arise, diminishing the effectiveness of semantic guidance. To address these issues, we propose a universal open-vocabulary object detection method that leverages foundational models to provide fine-grained semantic guidance for the detection process. We design a multi-level adaptive scene perception algorithm that captures subtle features of target objects in complex scenes, enabling precise separation of background and foreground. Additionally, we introduce the Text-KAN (T-KAN) model, which integrates textual descriptions with image features. By employing learnable activation functions, it resolves dependencies on linear matrices, enhances text interpretability, corrects semantic biases, and achieves precise alignment between images and text at a fine-grained level. We comprehensively evaluate the performance of our proposed method on existing open-vocabulary benchmarks, conducting experiments on the COCO and LVIS datasets. The results demonstrate significant performance gains in detecting novel categories, highlighting the method’s strong generalization capabilities. This work provides valuable insights and references for advancing the field of open-vocabulary object detection.

Six degrees-of-freedom (6D) object pose estimation plays an important role in pattern recognition of fields such as robotics and augmented reality. However, there are issues with low accuracy and real-time performance of 6D object pose estimation in complex scenes. To address these challenges, in this article, RFF-PoseNet (a 6D object pose estimation network based on robust feature fusion) is proposed for complex scenes. Firstly, a more lightweight Ghost module is used to replace the convolutional blocks in the feature extraction network. Then, a pyramid pooling module is added to the semantic label branch of PoseCNN to fuse the features of different pooling layers and enhance the network’s ability to capture information about objects in complex scenes and the correlations between contextual information. Finally, a pose regression and optimization module is utilized to further improve object pose estimation in complex scenes. Simulation experiments conducted on the YCB-Video and Occlusion LineMOD datasets show that the RFF-PoseNet algorithm can strengthen the correlation of features between different levels and the recognition ability of unclear targets, thereby achieving excellent accuracy and real-time performance, as well as strong robustness.

This paper introduces a novel technique for the automatic identification of real world objects in complex scenes. The identification problem requires the comparison of assemblies of image regions with a previously stored view of a known prototype. Shape context representation and matching are employed for recovering point correspondences between the image and the prototype object. This work is inspired from the problem that, a number of ambient conditions such as partial object occlusion and contour distortion, may affect the performance of the matching process and consequently the identification result. In our approach, subtractive clustering is applied in a novel fashion to enable the identification of regions of interest on the complex scene, based on a density metric. In order to increase the robustness of the identifier to mismatches and reduce the computational cost of the process, a selection of the initial suspicious regions is performed. The performance of the identifier has been examined in a great range of image and prototype selections.

The concept of inspection robot is born in the strong & smart grid. By studying on the mechanism of human visual attention, this paper presents a method to detect the encroachments on transmission Lines from the perspective of the saLiency-based visual attention system. There are conspicuous diffidence between the encroachments and transmission Lines and it is easy for human to distinguish the encroachments from the transmission Lines because the encroachments were always the saLient objects in the view, consequently, it is a reasonable way to detect the encroachments based on the mechanism of visual attention. The experiment results show that this method can effectively detect and locate different types of encroachments in different natural environments.

Deep (machine) learning in recent years has significantly increased the predictive modeling strengths for applications in the areas of computer vision, speech recognition, and natural language processing. Being heavily data-driven, conventional general-purpose deep learning methods mainly rely on training data to learn task-specific data representations. In this thesis, we focus on leveraging novel learning algorithms in deep learning to tackle visual recognition challenges at the three key levels -- namely pixel, object, and image levels. Throughout the thesis, we demonstrate how each and every of our proposed methods can address a specific visual recognition problem. Firstly, we apply deep learning to improve the saliency detection performance of deep networks. A coarse saliency map of the input image's salient object is initially predicted by a convolutional-deconvolutional network (or equivalently a fully convolutional network). Following that, the coarse saliency map is sequentially refined by a separate network called Recurrent Attentional Convolutional-Deconvolutional Network (RACDNN) that shares its network weights across its sequential steps. At every sequential (temporal) step, the same RACDNN is applied to a newly selected/attended image sub-region and it performs saliency map refinement for that particular sub-region. Despite having many learnable network parameters and no access to additional training data, RADCNN can be trained well and it outperforms the "single-step" (coarse saliency detection) baseline and state-of-the-art saliency detection methods by large margins, on several saliency detection benchmark datasets. Secondly, we introduce auto-encoder weight transfer networks (AE-WTN+), a deep model for scaling up object detection. Conventional WTN transfers object class knowledge from the classification weights of a pretrained large-scale image classification network, to the classification weights of an object detection network. However, they suffer from under-fitting and are not able to preserve the rich object class information of novel classes. In this work, we propose a new WTN method -- AE-WTN+ which makes use of normalization techniques to ease network optimization. Inputs and intermediate features are normalized to improve trainability for reduced under-fitting. More importantly, a novel auto-encoding (reconstruction) loss is proposed to encourage information preservation of all classes in the weight predictions of AE-WTN+. This enables the predicted weights of AE-WTN+ to maintain the class discriminability and neighbourhood relationships contained in the pre-trained large-scale classification weights. Experiments on large-scale detection datasets validate the effectiveness of AE-WTN+ on both seen and novel classes. Thirdly, we improve the efficiency of cross-layer network connections for very deep convolutional network in the context of image classification. This gives rise to a new convolutional network architecture called Deluge Networks (DelugeNets) which pass the output (feature activations) of any preceding composite network layer as direct inputs to all succeeding composite layers. For the incoming feature activations of preceding composite layers, the succeeding composite layers first apply rather lightweight cross-layer depthwise convolutional weights on them for feature aggregation. The heavy lifting is carried out by the last convolutional weight layer in each preceding composite layer that performs cross-channel convolutions. The proposed technique enables unimpeded information flow among many layers in very deep DelugeNets with great efficiency. Experiments on small-scale and large-scale datasets show that DelugeNets perform competitively to state-of-the-art deep convolutional networks at lower computational and parameter costs. Fourthly, we propose Stochastic Downsampling (SDPoint), a method to overcome a major limitation suffered by conventional deep convolutional networks -- their computational costs during inference are fixed and cannot be changed. SDPoint downsamples the feature maps at a randomly selected point (layer index) in the network hierarchy during training. The many downsampling configurations known as SDPoint instances each entail a unique downsampling point and downsampling ratio. The same convolutional network weights are shared across the different SDPoint instances that downsample feature maps differently. During inference, the computational cost of a network trained with SDPoint can be instantaneously adjusted to fit a given computational budget, by manually selecting an appropriate SDPoint instance. Additionally, sharing weights between different feature map scales provides significant regularization benefits, making the networks trained with SDPoint less sensitive to scales and more competent. In image classification experiments, SDPoint demonstrates significant cost-accuracy and computational advantages, over independently-trained (non-weight-shared) baselines with various inference costs.

We are developing a helper robot that carries out tasks ordered by users through speech. The robot needs a vision system to recognize objects appearing in the orders. However, conventional vision systems cannot recognize objects in complex scenes. They may find many objects and cannot determine which is the target. This paper proposes a method of using a conversation with the user to solve this problem. The robot asks a question to which the user can easily answer and whose answer can efficiently reduce the number of candidate objects. It considers the characteristics of features used for object recognition such as the easiness for humans to specify them by word, generating a user-friendly and efficient sequence of questions. Experimental results show that the robot can detect target objects by asking the questions generated by the method.

Robust detection of moving objects in complex scenes is one of the most challenging issues in computer vision. In this paper, we present a novel texture-wise approach to segment moving objects with codebook and local binary patterns (LBP). In many moving segmentation algorithms, the information from limited frames before current image is used. Our approach models background over long time with small memory. Firstly, we construct codebook model which represents a compressed form of background model for long image sequences. A single Gaussian model of per-pixel is built to deal with illumination changes. By using the correlation and texture of spatially proximal pixels, local binary patterns background model is constructed. Finally current image is segmented into two parts, foreground and background, by comparing current image with background model. Experiments show that the proposed approach achieves promising results robustly in real videos.