Robust Background Template for Saliency Detection

Saliency detection is a technique to analyze image surroundings to extract relevant regions from the background. In this paper, we propose a simple and effective saliency detection method based on image sparse representation and color features combination. First, the input image is segmented into non-overlapping super-pixels, so as to perform the saliency detection at the region level to reduce computational complexity. Then, a background optimization selection scheme is used to construct an appropriate background template. Based on this, a primary saliency map is obtained by using image sparse representation. Next, through the linear combination of color coefficients we generate an improved saliency map with more prominent salient regions. Finally, the two saliency maps are integrated within Bayesian framework to obtain the final saliency map. Experimental results show that the proposed method has desirable detection performance in terms of detection accuracy and running time.

Saliency detection is a technique for automatically extracting regions of interest from the background and has been widely used in the computer vision field. This study proposes a simple and effective saliency detection method combining color contrast and hash fingerprint. In our solution, the input image is segmented into nonoverlapping superpixels, so as to perform the saliency detection at the region level to reduce computational complexity. A background optimization selection is used to construct an accurate background template. Based on this, a saliency map that highlights the whole salient region is obtained by estimating color contrast. Besides, another saliency map that enhances the salient region while restraining the background is also generated through hash fingerprint matching. Ultimately, the final saliency map can be obtained by fusing the two saliency maps. Comparing the performance with other methods, the proposed algorithm works better even in the presence of complex background or very large salient regions.

Vision-Language Pretrained (VLP) models have achieved impressive performance on multimodal tasks, including text-image retrieval, based on dense representations. Meanwhile, Learned Sparse Retrieval (LSR) has gained traction in text-only settings due to its interpretability and efficiency with fast term-based lookup via inverted indexes. Inspired by these advantages, recent work has extended LSR to the multimodal domain. However, these methods often rely on computationally expensive contrastive pre-training, or distillation from a frozen dense model, which limits the potential for mutual enhancement. To address these limitations, we propose a simple yet effective framework that enables bi-directional learning between dense and sparse representations through Self-Knowledge Distillation. This bi-directional learning is achieved using an integrated similarity score-a weighted sum of dense and sparse similarities-which serves as a shared teacher signal for both representations. To ensure efficiency, we fine-tune the final layer of the dense encoder and the sparse projection head, enabling easy adaptation of any existing VLP model. Experiments on MSCOCO and Flickr30k demonstrate that our sparse retriever not only outperforms existing sparse baselines, but also achieves performance comparable to-or even surpassing-its dense counterparts, while retaining the benefits of sparse models.

In recent years, the object-level saliency detection has attracted much research attention, due to its usefulness in many high-level tasks. Existing methods are mostly based on the contrast hypothesis, which regards the regions with high contrast in a certain context as salient objects. Although the contrast hypothesis is effective in many scenarios, it cannot handle some difficult cases. As a remedy to address the weakness of contrast hypothesis, we propose a novel compactness hypothesis, which assumes salient regions are more compact than background from the perspectives of both color layout and texture layout. Based on the compactness hypotheses, we implement an effective object-level saliency detection method. In the proposed method, we first construct a weak saliency map based on the compact hypotheses, then collect samples from the weak saliency map to train a dedicated classifier. This classifier is applied on each individual pixel of the input image to produce a confidence score. Finally, the confidence scores are used to form a saliency map. This process is carried out at different scales, and the corresponding results are integrated into the formation of the final saliency map. The proposed approach is evaluated on eight benchmark data sets, where it delivers the competitive performance compared with the state-of-the-art methods.

Dense feature representations are recently popular for human action recognition based on video. They achieve better performance than sparse local representations. However, dense representations usually lead to high-dimensional data which contain plenty of redundancy information. In order to solve this issue, a novel compact representation based on global Gist feature and 2DPCA-2DLDA is proposed. Firstly Global grid based Gist features are extracted from image sequences and described as Gist matrices. The Gist matrices reflect global structure distribution of local grids, and at the same time depict frequency characteristics of the local grids. Secondly, bilateral dimensionality reduction is developed to our proposed Gist matrices. The most discriminative spatial structure and frequency characteristic of local grids are separately selected with 2DPCA and 2DLDA. Finally, the derived structure and frequency properties are simultaneously embedded into a low dimensional compact global descriptor. Experimental results on UCF Sports dataset validate that the proposed method makes a good compromise between feature discrimination and dimensions. And the action recognition accuracy is promising.

Humans are accustomed to autonomously associating prior knowledge with the text in a query when answering questions. However, for machines lacking cognition and common sense, a query is merely a combination of some words. Although we can enrich the semantic information of the given query through language representation or query expansion (QE) , the information contained in the query is still insufficient. In this paper, we propose an effective passage retrieval method named query context expansion-based retrieval (QCER) for open-domain question answering (OpenQA) . QCER associates a query with domain information by adding contextual association information based on the pseudo-relevance feedback (PRF) . QCER uses a dense reader to select top-n expansion terms for QE. We implement QCER by appending reader predictions, theoretically present in candidate passages, as contextual information to the initial query to form the new query. QCER with sparse representations (BM25) can improve retrieval efficiency and accelerate query convergence so that the reader can find the desired answer using fewer relevant passages, e.g., 10 passages, as soon as possible. Moreover, QCER can be easily combined with dense passage retrieval (DPR) to achieve even better performance, as sparse and dense representations are often complementary. Remarkably, we demonstrate that QCER achieves state-of-the-art performance in three tasks, passage retrieval, passage reading, and passage reranking, on the Natural Questions (NQ) and TriviaQA (Trivia) datasets under an extractive QA setup.

Emotion detection from text has become a popular task due to the key role of emotions in human-machine interaction. Current approaches represent text as a sparse bag-of-words vector. In this work, we propose a new approach that utilizes pre-trained, dense word embedding representations. We introduce an ensemble approach combining both sparse and dense representations. Our experiments include five datasets for emotion detection from different domains and show an average improvement of 11.6% in macro average F1-score.

Automatic extraction of salient regions is beneficial for various computer vision applications, such as image segmentation and object recognition. The salient visual information across images is very useful and plays a significant role for the visually impaired in identifying tactile information. In this paper, we introduce a novel saliency cuts method using local adaptive thresholding to obtain four regions from a given saliency map. First, we produced four regions for image segmentation using a saliency map as an input image and local adaptive thresholding. Second, the four regions were used to initialize an iterative version of the GrabCuts algorithm and to produce a robust and high-quality binary mask with a full resolution. Finally, salient objects’ outer boundaries and inner edges were detected using the solution from our previous research. Experimental results showed that local adaptive thresholding using integral images can produce a more robust binary mask compared to the results from previous works that make use of global thresholding techniques for salient object segmentation. The proposed method can extract salient objects with a low-quality saliency map, achieving a promising performance compared to existing methods. The proposed method has advantages in extracting salient objects and generating simple, important edges from natural scene images efficiently for delivering visually salient information to the visually impaired.

At the core of any inference procedure, deep neural networks are dot product operations, which are the component that requires the highest computational resources. For instance, deep neural networks, such as VGG-16, require up to 15-G operations in order to perform the dot products present in a single forward pass, which results in significant energy consumption and thus limits their use in resource-limited environments, e.g., on embedded devices or smartphones. One common approach to reduce the complexity of the inference is to prune and quantize the weight matrices of the neural network. Usually, this results in matrices whose entropy values are low, as measured relative to the empirical probability mass distribution of its elements. In order to efficiently exploit such matrices, one usually relies on, inter alia, sparse matrix representations. However, most of these common matrix storage formats make strong statistical assumptions about the distribution of the elements; therefore, cannot efficiently represent the entire set of matrices that exhibit low-entropy statistics (thus, the entire set of compressed neural network weight matrices). In this paper, we address this issue and present new efficient representations for matrices with low-entropy statistics. Alike sparse matrix data structures, these formats exploit the statistical properties of the data in order to reduce the size and execution complexity. Moreover, we show that the proposed data structures can not only be regarded as a generalization of sparse formats but are also more energy and time efficient under practically relevant assumptions. Finally, we test the storage requirements and execution performance of the proposed formats on compressed neural networks and compare them to dense and sparse representations. We experimentally show that we are able to attain up to ×42 compression ratios, ×5 speed ups, and ×90 energy savings when we lossless convert the state-of-the-art networks, such as AlexNet, VGG-16, ResNet152, and DenseNet, into the new data structures and benchmark their respective dot product.

Stereoscopic perception is an important part of human visual system that allows the brain to perceive depth. However, depth information has not been well explored in existing saliency detection models. In this letter, a novel saliency detection method for stereoscopic images is proposed. Firstly, we propose a measure to evaluate the reliability of depth map, and use it to reduce the influence of poor depth map on saliency detection. Then, the input image is represented as a graph, and the depth information is introduced into graph construction. After that, a new definition of compactness using color and depth cues is put forward to compute the compactness saliency map. In order to compensate the detection errors of compactness saliency when the salient regions have similar appearances with background, foreground saliency map is calculated based on depth-refined foreground seeds selection mechanism and multiple cues contrast. Finally, these two saliency maps are integrated into a final saliency map through weighted-sum method according to their importance. Experiments on two publicly available stereo datasets demonstrate that the proposed method performs better than other 10 state-of-the-art approaches.

Nowadays, ship detection in sea-sky background is not only useful in maritime visual surveillance, but also helpful in maritime search and rescue. Since ships are salient objects in infrared images with sea-sky background, we present a novel and effective algorithm based on saliency for ship detection in this situation. Our algorithm adopts global saliency, local saliency and background prior to generate saliency maps. Ships are finally segmented in saliency maps. Our algorithm is compared with four classic salient object detection algorithms. And experimental results show our algorithm outperforms the other four algorithms in qualitative and quantitative terms. Keyword list: ship detection ; salient object detection ; infrared image ; sea - sky background . 1. INTRODUCTION Since ships are important vehicles for sea transportation, ship detection in sea-sky background is extensively used in maritime visual surveillance and reconnaissance, avoidance of ship collision accidents, maritime search and rescue, etc. Infrared imaging systems can almost work in any environment and all-weather conditions with its visual capacity in darkness and frog, which makes it a very suitable tool in this application. In recent years, ship detection in infrared images with sea-sky background attracts increasing attention. Many methods are proposed in this field, including image segmentation methods [1-2] and morphological methods [3]. However, due to complex sea-sky environment and large variation of ship sizes, these methods cannot solve the issue completely. And since ships are often salient objects in infrared images with sea-sky background, some researchers have recently tried salient object detection ideas used in color images [4-6]. And the paper is focused on using salient object detection methods to solve the issue. Salient objects are objects which stand out in the image and attract most people ¶VLPPHGLDWHDWWHQWLRQ7KHVWDQGDUGway to detect salient ob MHFWVLVILUVWO\XVLQJVDOLHQF\GHWHFWLRQPHWKRGVWRJHQHUDWHVDOLHQF\PDSVLQZKLFKDSL[HO¶VJUH\value is proportional to its degree of saliency and then segmenting salient objects in saliency maps. Many salient detection methods can be divided into three categories: local contrast methods, global rarity methods and other methods. In local contrast methods, regions which have high contrast with local neighbouring regions are regarded as salient regions. Itti et al. [7] decompose an image into several feature maps . The saliency in each feature map is measured by local center-surround difference in several scales and saliency of different feature maps are integrated to a final saliency

Over the last decade, it has become increasingly affordable to digitize 2D and 3D shape information using multiple modalities, such as (video) cameras, image-based reconstruction systems, laser-range scanners, or depth cameras. If these dense models can be processed and described in an efficient and informative way, they can be used in applications, such as ergonomic design, virtual shopping, scientific and medical visualization, realistic simulation, photo-realistic rendering, the design of natural user interfaces, and semantic scene understanding. Traditionally, the notion of shape has been studied either by analyzing projections of shapes in images or by analyzing a sparse set of marker positions on 3D shapes. Typical tasks in 2D shape analysis include segmenting objects in images and tracking objects across a sequence of images, and typical tasks in 3D shape analysis include reconstructing the three-dimensional object depth from input images and identifying corresponding points on different 3D models. The analysis and processing of shape data becomes especially challenging because of the increasing amount of data captured by sensors used to acquire shapes, and because modern applications such as natural user interfaces require real-time processing of the input shapes. Meeting these challenges requires models of shape analysis that are compact and informative, thereby allowing the development of algorithms that can process large datasets efficiently. To achieve these goals, interdisciplinary approaches are needed that use concepts from a variety of research areas, including numerical computing, differential geometry, deformable shape modeling, sparse data representation, and machine learning. On the algorithmic side, many shape analysis tasks are modeled using partial differential equations, which can be solved using tools from the field of numerical computing. The fields of differential geometry and deformable shape modeling have recently begun to influence shape analysis methods. Furthermore, tools from the field of sparse representations, which aim to describe input data using a compressible representation with respect to a set of carefully selected basis elements, have the potential to significantly reduce the amount of data that needs to be processed in shape analysis tasks. The related field of machine learning offers similar potential. This seminar brought together 28 researchers from North America and Europe engaged in recent and upcoming developments in shape analysis who view these challenges from different perspectives and who together discussed the pressing open problems and novel solutions to them.

In recent years, there has been an increased scope for assistive software and technologies, which help the visually impaired to perceive and recognize natural scene images. In this article, we propose a novel saliency cuts approach using local adaptive thresholding to obtain four regions from a given saliency map. The saliency cuts approach is an effective tool for salient object detection. First, we produce four regions for image segmentation using a saliency map as an input image and applying an automatic threshold operation. Second, the four regions are used to initialize an iterative version of the Grab Cut algorithm and to produce a robust and high-quality binary mask with a full resolution. Lastly, based on the binary mask and extracted salient object, outer boundaries and internal edges are detected by Canny edge detection method. Extensive experiments demonstrate that the proposed method correctly detects and extracts the main contents of the image sequences for delivering visually salient information to the visually impaired people compared to the results of existing salient object segmentation algorithms

In this paper, we propose a novel method for automatic object segmentation from natural scene images. This method is based on saliency map, mean shift segmentation and level set method. First, a histogram based contrast method is used to generate the saliency map of the input image. Second, the input image is segmented into clusters using mean shift. Based on the saliency map, segmented clusters are classified into background and foreground clusters. After that, an initial contour for level set method is determined by applying morphological erosion on foreground clusters. Finally, the level set method is used to evolve the initial contour and find the object of interest. The experimental results have shown that our proposed method is not only able to replace manual labeling of initial contour in level set method but also able to yield more accurate segmentation results than previous segmentation approaches.

Image super-resolution (SR) produces a high-resolution (HR) image using either a single or multiple low-resolution (LR) input image(s) of the same scene. Sparse representation techniques are effectively applied for image SR because of their high reconstruction accuracy. SR reconstruction is accomplished through learning of an overcomplete dictionary and then solving a series of regularization problems applied on each patch extracted from the input image. This paper demonstrates a sparse representation based coupled overcomplete dictionary training and SR procedure for LR multispectral images. The proposed work is also implemented using a multicore parallel processing technique to provide faster reconstruction. Experimental results show superiority of the proposed method over some others.