Asymmetric propagation based batch mode active learning for image retrieval

Relevance feedback (RF) schemes have been widely designed to improve the performance of content-based image retrieval. Despite the success, it is not appropriate to require the user to label a large number of samples in RF. Collaborative image retrieval (CIR) aims to reduce the labeling efforts of the user by resorting to the auxiliary information. Support vector machine (SVM) active learning can select ambiguous samples as the most informative ones for the user to label with the help of the optimal hyperplane of SVM, and thus alleviate the labeling efforts of conventional RF. However, the optimal hyperplane of SVM is usually unstable and inaccurate with small-sized training data, and this is always the case in image retrieval since the user would not like to label a large number of feedback samples and cannot label each sample accurately all the time. In this paper, we propose a novel active learning method, i.e., geometric optimum experimental design (GOED), to select multiple representative samples in the database as the most informative ones for the user to label. Especially, GOED can alleviate the small-sized training data problem by leveraging the geometric structure of unlabeled samples in the reproducing kernel Hilbert space and thus further enhance the performance of image retrieval. Different from the conventional manifold regularization framework, the new method can effectively select the most informative samples for the user to label in image retrieval. By minimizing the expected average prediction variance on the test data, GOED has a clear geometric interpretation to select a set of the most representative samples in the database iteratively with the global optimum. Compared with the popular SVM active learning, our method is label-independent and can effectively avoid various potential problems caused by insufficient and inexactly labeled samples in RF, and is more appropriate and useful for image retrieval. Extensive experiments on both synthetic datasets and a real-world image database have been conducted to show the advantages of the proposed GOED for CIR.

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on certain criteria. In this article we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimize the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and two biomedical image databases demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. In addition, we present a joint optimization framework for performing both transfer and active learning simultaneously unlike the existing approaches of learning in two separate stages, that is, typically, transfer learning followed by active learning. We specifically minimize a common objective of reducing distribution difference between the domain adapted source, the queried and labeled samples and the rest of the unlabeled target domain data. Our empirical studies on two biomedical image databases and on a publicly available 20 Newsgroups dataset show that incorporation of uncertainty information and transfer learning further improves the performance of the proposed active learning based classifier. Our empirical studies also show that the proposed transfer-active method based on the joint optimization framework performs significantly better than a framework which implements transfer and active learning in two separate stages.

Support vector machine (SVM) active learning is one popular and successful technique for relevance feedback in content-based image retrieval (CBIR). Despite the success, conventional SVM active learning has two main drawbacks. First, the performance of SVM is usually limited by the number of labeled examples. It often suffers a poor performance for the small-sized labeled examples, which is the case in relevance feedback. Second, conventional approaches do not take into account the redundancy among examples, and could select multiple examples that are similar (or even identical). In this work, we propose a novel scheme for explicitly addressing the drawbacks. It first learns a kernel function from a mixture of labeled and unlabeled data, and therefore alleviates the problem of small-sized training data. The kernel will then be used for a batch mode active learning method to identify the most informative and diverse examples via a min-max framework. Two novel algorithms are proposed to solve the related combinatorial optimization: the first approach approximates the problem into a quadratic program, and the second solves the combinatorial optimization approximately by a greedy algorithm that exploits the merits of submodular functions. Extensive experiments with image retrieval using both natural photo images and medical images show that the proposed algorithms are significantly more effective than the state-of-the-art approaches. A demo is available at http://msm.cais.ntu.edu.sg/LSCBIR/.

Combining manifold ranking with active learning (MRAL for short) is one popular and successful technique for relevance feedback in content-based image retrieval (CBIR). Despite the success, conventional MRAL has two main drawbacks. First, the performance of manifold ranking is very sensitive to the scale parameter used for calculating the Laplacian matrix. Second, conventional MRAL does not take into account the redundancy among examples and thus could select multiple examples that are similar to each other. In this work, a novel MRAL framework is presented to address the drawbacks. Concretely, we first propose a self-tuning manifold ranking algorithm that can adaptively calculate the Laplacian matrix via a local scaling mechanism, and then develop a hybrid active learning algorithm by integrating three well-known selective sampling criteria, which is able to effectively and efficiently identify the most informative and diversified examples for the user to label. Experiments on 10,000 Corel images show that the proposed method is significantly more effective than some existing approaches.

Semantic segmentation is a task that heavily relies on the annotated data. The image annotation cost is very expensive. While active learning aims to select the most valuable samples by an iterative procedure. It can reduce the annotation cost and improve the performance of classification. In semantic segmentation, its more common to select a batch of instances instead of a single instance at each iteration. In this paper, we propose a novel batch mode active learning algorithm for semantic segmentation. Different from the previous active learning algorithms for the image classification, we first introduce a new selecting criterion : the image prior of nonlocal self-similarity. It can measure the interaction between the pixels of the image. We combine the informativeness and representativeness with the criterion of nonlocal self-similarity to complete the selection of images at each iteration. In addition, we also use the model uncertainty to measure the information of samples. The model uncertainty is captured by using Monte Carlo Dropout in the semantic segmentation model. In this work, we evaluate our method on the CamVid and PASCAL VOC 2012 datasets. The importance of the nonlocal self-similarity is also assessed. The experiments demonstrate that our algorithm outperforms current state-of-the-art active learning methods over the segmentation performance.

In content-base image retrieval, relevance feedback (RF) schemes based on support vector machine (SVM) have been widely used to narrow the semantic gap between low-level visual features and high-level human perception. However, the performance of image retrieval with SVM active learning is known to be poor when the training data is insufficient. In this paper, the problem is solved by incorporating the unlabelled images into the learning process. We proposed a semi-supervised active learning algorithm which uses not only labeled training samples but also unlabeled ones to build better models. In relevance feedback, active learning algorithm is often used to reduce the cost of labeling by selecting only the most informative data. In addition, we introduced a semi-supervised approach which employed Nearest-Neighbor technique to label the unlabeled sample with a certain degree of uncertainty in its class information. Using these samples, Fuzzy support vector machine (FSVM) which takes into account the fuzzy nature of some training samples during its training is trained. We compared our method with standard active SVM on a database of 10,000 images, the experiment results show that the efficiency of SVM active learning can be improved by incorporating unlabelled images, and thus improve the overall retrieval performance. DOI: http://dx.doi.org/10.11591/telkomnika.v11i7.2807

This paper presents a subspace SVM ensemble algorithm for adaptive relevance feedback (RF) learning. Our method deals with the case that user's relevance feedback examples are usually insufficient and overlapped together in feature space, which decreases the learning effectiveness of RF classifiers. To enhance classification efficiency in such case, multiple SVMs are learned by clustering-based training set partition, each of which fits its cluster-specific sample distribution and gives labeling regressions to test samples that fall within this cluster. To adapt features to sample distribution within each cluster, AdaBoost feature selection is conducted onto pyramid Haar of H&I bands in HSI space. In AdaBoost, we evaluate the feature discriminative ability by an entropy-based uncertainty criterion, based on which an Eigen feature subspace is constructed in cluster-specific SVM training. Finally, regression results of multiple SVMs are probabilistic assembled to give the final labeling prediction for test image. We compare our cluster-based cascade SVMs (CSS) RF method in COREL 5,000 database with: 1. Single SVM; 2. Active Learning SVM [5]; 3. Bootstrap Sampling SVM [7]. The superior experimental results demonstrate the efficiency of our algorithm.

Active learning can effectively reduce labelling effort for remote sensing image classification. In this paper, we propose a new active learning method for hyperspectral image classification. We consider batch mode active learning and relatively large amount of data which can be a problem when using current state of the art algorithm based on kernel machines. The active learning procedure is based on the uncertainty sampling strategy and a deep neural network. Stacked autoencoders trained on redundant spatial and spectral features and a few labeled training samples are used to initialize a deep neural network. Uncertainty for a given sample is measured by the difference between the largest two class outputs of the neural network. The less difference there is, the more uncertainty the sample has. Batch of samples with most uncertainty will be selected after label query and added into the training set. Then the neural network is retrained. And such active batch selection will iterate until the budget (the upper limit of label queries) is reached. Experimental results on Pavia university dataset showed that our method outperforms the current support vector machines (SVMs) based multiclass/level uncertainty (MCLU) method both in classification accuracy and generalization capability.

Active learning can effectively reduce labelling effort for remote sensing image classification. In this paper, we propose a new active learning method for hyperspectral image classification. We consider batch mode active learning and relatively large amount of data which can be a problem when using current state of the art algorithm based on kernel machines. The active learning procedure is based on the uncertainty sampling strategy and a deep neural network. Stacked autoencoders trained on redundant spatial and spectral features and a few labeled training samples are used to initialize a deep neural network. Uncertainty for a given sample is measured by the difference between the largest two class outputs of the neural network. The less difference there is, the more uncertainty the sample has. Batch of samples with most uncertainty will be selected after label query and added into the training set. Then the neural network is retrained. And such active batch selection will iterate until the budget (the upper limit of label queries) is reached. Experimental results on Pavia university dataset showed that our method outperforms the current support vector machines (SVMs) based multiclass/level uncertainty (MCLU) method both in classification accuracy and generalization capability.

In content-based image retrieval (CBIR), a set of low-level features are extracted from an image to represent its visual content. Retrieval is performed by image example where a query image is given as input by the user and an appropriate similarity measure is used to find the best matches in the corresponding feature space. This approach suffers from the fact that there is a large discrepancy between the low-level visual features that one can extract from an image and the semantic interpretation of the image’s content that a particular user may have in a given situation. That is, users seek semantic similarity, but we can only provide similarity based on low-level visual features extracted from the raw pixel data, a situation known as the semantic gap. The selection of an appropriate similarity measure is thus an important problem. Since visual content can be represented by different attributes, the combination and importance of each set of features varies according to the user’s semantic intent. Thus, the retrieval strategy should be adaptive so that it can accommodate the preferences of different users. Relevance feedback (RF) learning has been proposed as a technique aimed at reducing the semantic gap. It works by gathering semantic information from user interaction. Based on the user’s feedback on the retrieval results, the retrieval scheme is adjusted. By providing an image similarity measure under human perception, RF learning can be seen as a form of supervised learning that finds relations between high-level semantic interpretations and low-level visual properties. That is, the feedback obtained within a single query session is used to personalize the retrieval strategy and thus enhance retrieval performance. In this chapter we present an overview of CBIR and related work on RF learning. We also present our own previous work on a RF learning-based probabilistic region relevance learning algorithm for automatically estimating the importance of each region in an image based on the user’s semantic intent.

With the rapid growing volume of images in medical databases, development of efficient image retrieval systems to retrieve relevant or similar images to a query image has become an active research area. Despite many efforts to improve the performance of techniques for accurate image retrieval, its success in biomedicine thus far has been quite limited. This article presents an adaptive content-based image retrieval (CBIR) system for improving the performance of image retrieval in mammographic databases. In this work, the authors propose a new relevance feedback approach based on incremental learning with support vector machine (SVM) regression. Also, the authors present a new local perturbation method to further improve the performance of the proposed relevance feedback system. The approaches enable efficient online learning by adapting the current trained model to changes prompted by the user's relevance feedback, avoiding the burden of retraining the CBIR system. To demonstrate the proposed image retrieval system, the authors used two mammogram data sets: A set of 76 mammograms scored based on geometrical similarity and a larger set of 200 mammograms scored by expert radiologists based on pathological findings. The experimental results show that the proposed relevance feedback strategy improves the retrieval precision for both data sets while achieving high efficiency compared to offline SVM. For the data set of 200 mammograms, the authors obtained an average precision of 0.48 and an area under the precision-recall curve of 0.79. In addition, using the same database, the authors achieved a high pathology matching rate greater than 80% between the query and the top retrieved images after relevance feedback. Using mammographic databases, the results demonstrate that the proposed approach is more accurate than the model without using relevance feedback not only in image retrieval but also in pathology matching while maintaining its effectiveness for online relevance feedback applications.

Active learning for networked data that focuses on predicting the labels of other nodes accurately by knowing the labels of a small subset of nodes is attracting more and more researchers because it is very useful especially in cases, where labeled data are expensive to obtain. However, most existing research either only apply to networks with assortative community structure or focus on node attribute data with links or are designed for working in single mode that will work at a higher learning and query cost than batch active learning in general. In view of this, in this paper, we propose a batch mode active learning method which uses information-theoretic techniques and random walk to select which nodes to label. The proposed method requires only network topology as its input, does not need to know the number of blocks in advance, and makes no initial assumptions about how the blocks connect. We test our method on two different types of networks: assortative structure and diassortative structure, and then compare our method with a single mode active learning method that is similar to our method except for working in single mode and several simple batch mode active learning methods using information-theoretic techniques and simple heuristics, such as employing degree or betweenness centrality. The experimental results show that the proposed method in this paper significantly outperforms them.

Employing unlabeled data to improve the classification performance of SVM, and its application in audio event classification

Relevance feedback has emerged as a powerful tool to boost the retrieval performance in content-based image retrieval (CBIR). In the past, most research efforts in this field have focused on designing effective algorithms for traditional relevance feedback. Given that a CBIR system can collect and store users' relevance feedback information in a history log, an image retrieval system should be able to take advantage of the log data of users' feedback to enhance its retrieval performance. In this paper, we propose a unified framework for log-based relevance feedback that integrates the log of feedback data into the traditional relevance feedback schemes to learn effectively the correlation between low-level image features and high-level concepts. Given the error-prone nature of log data, we present a novel learning technique, named soft label support vector machine, to tackle the noisy data problem. Extensive experiments are designed and conducted to evaluate the proposed algorithms based on the COREL image data set. The promising experimental results validate the effectiveness of our log-based relevance feedback scheme empirically.

Relevance feedback has emerged as a powerful tool to boost the retrieval performance in content-based image retrieval (CBIR). In the past, most research efforts in this field have focused on designing effective algorithms for traditional relevance feedback. Given that a CBIR system can collect and store users' relevance feedback information in a history log, an image retrieval system should be able to take advantage of the log data of users' feedback to enhance its retrieval performance. In this paper, we propose a unified framework for log-based relevance feedback that integrates the log of feedback data into the traditional relevance feedback schemes to learn effectively the correlation between low-level image features and high-level concepts. Given the error-prone nature of log data, we present a novel learning technique, named soft label support vector machine, to tackle the noisy data problem. Extensive experiments are designed and conducted to evaluate the proposed algorithms based on the COREL image data set. The promising experimental results validate the effectiveness of our log-based relevance feedback scheme empirically.