Weighted Pooling Based on Visual Saliency for Image Classification

Non-negativity and dependence constrained sparse coding for image classification

Recently support vector machines (SVM) using spatial pyramid matching (SPM) kernel have been highly successful in image classification applications. And linear spatial pyramid matching using sparse coding (ScSPM) scheme has been proposed to enhance the performance of SPM both in time and classification accuracy. In order to reduce the time complexity of dictionary construction process, sparse coding with affinity propagation method has been proposed in this paper. Because the dictionary used for sparse coding plays a key role in these methods, we also adopt supervised dictionary learning method to construct dictionary. The coding coefficients of each class have greater separability for SVM classification. Substantial experiments on Scene15 and CalTech101 image datasets have been conducted to investigate the performance of proposed approach in multi-class image classification; the results show that the approach can reach higher accuracy compared with ScSPM.

In the problem where there is not enough data to use Deep Learning, Bag-of-Visual-Words (BoVW) is still a good alternative for image classification. In BoVW model, many pooling methods are proposed to incorporate the spatial information of local feature into the image representation vector, but none of the methods devote to making each visual word have its own pooling regions. The practice of designing the same pooling regions for all the words restrains the discriminability of image representation, since the spatial distributions of the local features indexed by different visual words are not same. In this paper, we propose to make each visual word have its own pooling regions, and raise a simple yet effective method for learning pooling region. Concretely, a kind of small window named observation window is used to obtain its responses to each word over the whole image region. The pooling regions of each word are organized by a kind of tree structure, in which each node indicates a pooling region. For each word, its pooling regions are learned by constructing a tree with its labelled coordinate data. The labelled coordinate data consist of the coordinates of responses and image class labels. The effectiveness of our method is validated by observing if there is an obvious classification accuracy improvement after applying our method. Our experimental results on four small datasets (i.e., Scene-15, Caltech-101, Caltech-256 and Corel-10) show that, the classification accuracy is improved by about 1% to 2.5%. We experimentally demonstrate that the practice of making each word have its own pooling regions is beneficial to image classification task, which is the significance of our work.

Recently, spatial partitioning approaches such as spatial pyramid matching (SPM) are commonly used in image classification to collect the global and local features of the images. They divide the input image into small sub-regions (typically in a hierarchical manner) and generate a feature vector for each of them. Although the codes for the descriptors are assigned softly in modern image feature representation techniques, the codes must fall into only a single sub-region when forming the feature vector. In other words, the soft code assignment is used in the descriptor space but the codes are still “hard” voted from the view point of the image space. This paper proposes a spatial soft voting method, in which the existence of the codes are expressed by a Gaussian function and the maps of the existence are sampled to form a feature vector. The generated feature vectors are “soft” both in the descriptor space and the image space. In addition, extra computational cost as compared to SPM is negligibly small. The concept of the spatial soft voting is general and can be applied to most hard spatial partitioning approaches.KeywordsFeature VectorImage SpaceLocal DescriptorSparse CodeCode WordThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Compact and discriminative representation of Bag-of-Features

Fast low rank representation based spatial pyramid matching for image classification

This paper proposes a new approach for image classification by combining pyramid match kernel(PMK) with spatial pyramid. Unlike the conventional spatial pyramid matching (SPM) approach which only uses a single-resolution feature vector to represent an image, we use a multi-resolution feature vector to represent an image for SPM. We then calculate the match scores at each resolution of SPM representation and finally compute the matching between two images by applying the concept of PMK using the match scores obtained from the multiple resolutions. Our experimental results show that the proposed combined pyramid matching achieves a significant improvement on classification performance.

To improve the synthetic aperture radar (SAR) target discrimination performance under complex scenes, this letter presents a new SAR target discrimination method based on the bag-of-words model. The method contains three main stages. In the local feature extraction stage, the SAR-SIFT feature is extracted. In the feature coding stage, we improve the existing category-specific and shared dictionary learning (CSDL) and propose the sample-reweighted CSDL (SR-CSDL). The local features are sparsely coded using the codebook learned from SR-CSDL. In the feature pooling stage, spatial pyramid matching with max pooling is used to aggregate the local coding coefficients to generate the global feature for each chip image. Experimental results using the miniSAR data verify the effectiveness of the proposed method.

Image representation using local features and sparse coding (SC) plays a very important role in image classification when the dataset is fairly large. Despite of its worldwide popularity, there are still some improving space in classification efficiency and computational investment in training and coding phrase of SC. In this paper, we put forward a novel object categories recognition method from two aspects. First, the contextual relevance between image patches are fully utilized by merging local feature of every sub-patch with its neighboring ones into strong context features to generate the multiple sparse representations, which are received by the SC and multi-scale max pooling SPM(Spatial Pyramid Matching), respectively. Second, while calculating the sparse coefficients of SC, we need to solve L1-regularized least square problem. Screening out the zero coefficients and discarding the corresponding inactive codewords before solving Lasso problem can remarkably speed up the optimization. The proposed method outperforms state-of-the-art performance in a large number of image categorization experiments on several benchmarks: the ground truth dataset (21 Land-Use database), the event dataset (UIUC-Sport dataset), and the object recognition dataset (Caltech101 dataset).

Image representation using local features and sparse coding (SC) plays a very important role in image classification when the dataset is fairly large. Despite of its worldwide popularity, there are still some improving space in classification efficiency and computational investment in training and coding phrase of SC. In this paper, we put forward a novel object categories recognition method from two aspects. First, the contextual relevance between image patches are fully utilized by merging local feature of every sub-patch with its neighboring ones into strong context features to generate the multiple sparse representations, which are received by the SC and multi-scale max pooling SPM(Spatial Pyramid Matching), respectively. Second, while calculating the sparse coefficients of SC, we need to solve L1-regularized least square problem. Screening out the zero coefficients and discarding the corresponding inactive codewords before solving Lasso problem can remarkably speed up the optimization. The proposed method outperforms state-of-the-art performancein a large number of image categorization experiments on several benchmarks: the ground truth dataset (21 Land-Use database), the event dataset (UIUC-Sport dataset), and the object recognition dataset (Caltech101 dataset).

Neighbour feature attention-based pooling

Recent research has shown the initial success of sparse coding (Sc) in solving many computer vision tasks. Motivated by the fact that kernel trick can capture the nonlinear similarity of features, which helps in finding a sparse representation of nonlinear features, we propose kernel sparse representation (KSR). Essentially, KSR is a sparse coding technique in a high dimensional feature space mapped by an implicit mapping function. We apply KSR to feature coding in image classification, face recognition, and kernel matrix approximation. More specifically, by incorporating KSR into spatial pyramid matching (SPM), we develop KSRSPM, which achieves a good performance for image classification. Moreover, KSR-based feature coding can be shown as a generalization of efficient match kernel and an extension of Sc-based SPM. We further show that our proposed KSR using a histogram intersection kernel (HIK) can be considered a soft assignment extension of HIK-based feature quantization in the feature coding process. Besides feature coding, comparing with sparse coding, KSR can learn more discriminative sparse codes and achieve higher accuracy for face recognition. Moreover, KSR can also be applied to kernel matrix approximation in large scale learning tasks, and it demonstrates its robustness to kernel matrix approximation, especially when a small fraction of the data is used. Extensive experimental results demonstrate promising results of KSR in image classification, face recognition, and kernel matrix approximation. All these applications prove the effectiveness of KSR in computer vision and machine learning tasks.

Spatial Pyramid Matching (SPM) and related methods have been found perform well in general image classification. A recently proposed improvement on this was LrrSPM, which used low rank representation of encode the SIFT descriptors, to achieve comparable recognition rates, with faster processing speeds. While image classification of this kind has been applied to many fields, an area where this has not been used is in industrial welding applications. This paper attempts to apply LrrSPM to welding image datasets, and show comparable classification results. It also proposes a cognitive architecture involving associative neural networks to perform classification on the images.

Most of the existing hyperspectral image change detection methods based on deep learning utilize convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short-term memory (LSTM). However, CNNs primarily focus on extracting spatial information from hyperspectral images, thereby failing to fully utilize channel and temporal information. In this paper, we propose a network called STSTNet that can jointly extract spatial, spectral, and temporal feature information. To achieve a richer feature representation space, we employ a spatial attention module to extract spatial feature information from feature maps, and a spectral transformer encoder structure to extract abundant spectral sequence information. Specifically, we design two temporal information extraction structures after each spatial information extraction structure or spectral information extraction structure to capture correlations and interactions within the hyperspectral images. Extensive experiments have been conducted on three hyperspectral datasets, and the experimental results demonstrate the network’s excellent performance.

This paper presents a novel approach, named bag-of-bags of words (BBoW), to address the problem of Content-Based Image Retrieval (CBIR) from image databases. The proposed bag-of-bags of words model extends the classical bag-of-words (BoW) model. An image is represented as a connected graph of local features on a regular grid. Then irregular partitions (subgraphs) of images are further built via Normalized Cuts. Each subgraph in the partition is then represented by its own signature. Compared to existing methods for image retrieval, such as Spatial Pyramid Matching (SPM), the BBoW model does not assume that similar parts of a scene always appear at the same location in images of the same category. The extension of the proposed model to pyramid gives rise to a method we name irregular pyramid matching (IPM). The experiments demonstrate the strength of our method for image retrieval when the partitions are stable across an image category. The statistical analysis of subgraphs is discussed in the paper.