Variants of evolutionary learning for interactive image retrieval

We evaluate the small sample size (SSS) performance of evolutionary algorithms (EAs) for relevance feedback (RF) in image retrieval. We focus on the requirement to learn the user’s information need based on a small — between 2 and 25 — number of positive and negative training images. Despite this being a fundamental requirement, none of the existing works dealing with EAs for RF systematically evaluates their SSS performance. To address this issue, we compare four variants of EAs for RF. Common for all variants is the hierarchical, region-based image similarity model, with region and feature weights as parameters. The difference between the variants is in the objective function of the EA used to adjust the model parameters. The objective functions include: (O-1) precision; (O-2) average rank; (O-3) ratio of within-class (i.e., positive images) and between-class (i.e., positive and negative images) scatter; and (O-4) combination of O-2 and O-3. We note that — unlike O-1 and O-2 — O-3 and O-4 are not used in any of the existing works dealing with EAs for RF. The four variants are evaluated on five test databases, containing 61,895 general-purpose images, in 619 semantic categories. Results of the evaluation reveal that variants with objective functions O-3 and O-4 consistently outperform those with O-1 and O-2. Furthermore, comparison with the representative of the existing RF methods shows that EAs are both effective and efficient approaches for SSS learning in region-based image retrieval.

In content-base image retrieval, relevance feedback (RF) schemes based on support vector machine (SVM) have been widely used to narrow the semantic gap between low-level visual features and high-level human perception. However, the performance of image retrieval with SVM active learning is known to be poor when the training data is insufficient. In this paper, the problem is solved by incorporating the unlabelled images into the learning process. We proposed a semi-supervised active learning algorithm which uses not only labeled training samples but also unlabeled ones to build better models. In relevance feedback, active learning algorithm is often used to reduce the cost of labeling by selecting only the most informative data. In addition, we introduced a semi-supervised approach which employed Nearest-Neighbor technique to label the unlabeled sample with a certain degree of uncertainty in its class information. Using these samples, Fuzzy support vector machine (FSVM) which takes into account the fuzzy nature of some training samples during its training is trained. We compared our method with standard active SVM on a database of 10,000 images, the experiment results show that the efficiency of SVM active learning can be improved by incorporating unlabelled images, and thus improve the overall retrieval performance. DOI: http://dx.doi.org/10.11591/telkomnika.v11i7.2807

An approach for content-based image retrieval with relevance feedback based on a structured multi-feature space is proposed. It uses a novel kernel for merging multiple feature subspaces into a complementary space. The kernel exploits nature of the data by assigning appropriate weights for each feature set. The weights are dynamically adapted to user preferences in a relevance feedback scenario.

This paper presents a content-based image retrieval (CBIR) system that incorporates real-valued multiple instance learning (MIL) into the user relevance feedback (RF) to learn the user's subjective visual concepts, especially where the user's most interested region and how to map the local feature vector of that region to the high-level concept pattern of the user. RF provides a way to obtain the subjectivity of the user's high-level visual concepts, and MIL enables the automatic learning of the user's high-level concepts. The user interacts with the CBIR system by relevance feedback in a way that the extent to which the image samples retrieved by the system are relevant to the user's intention is labeled. The system in turn applies the MIL method to find user's most interested image region from the feedback. A multilayer neural network that is trained progressively through the feedback and learning procedure is used to map the low-level image features to the high-level concepts.

In image retrieval, relevance feedback uses information, obtained interactively from the user, to understand the user's perceptions of a query image and to improve retrieval accuracy. We propose simultaneous relevant feature selection and classification using the samples provided by the user to improve retrieval accuracy. The classifier is defined by a separating hyperplane, while the sparse weight vector characterizing the hyperplane defines a small set of relevant features. This set of relevant features is used for classification and can be used for analysis at a later stage. Mutually exclusive sets of images are shown to the user at each iteration to obtain maximum information from the user. Experimental results show that our algorithm performs better than the feature relevance weighting and feature selection schemes and comparably with the classification scheme using SVMs, in terms of retrieval accuracy, and it has the advantage of being faster than the classification scheme using SVMs.

Region-based image retrieval and relevance feedback are two important methods to bridge the gap between the low-level visual features and the high-level semantic concepts in content-based image retrieval. In this paper, we address the issue of introducing the relevance feedback mechanism into the region-based image retrieval with online feature selection during each feedback round. Our contribution is two-fold. (1) A novel region-based image representation is proposed. Based on a generative model, a fuzzy codebook is extracted from the original region-based features, which represents the images in a uniform real-value feature space. (2) A feature selection criterion is developed and an effective relevance feedback algorithm is implemented in a boosting manner to simultaneously select the optimal features from the fuzzy codebook, and generate a strong ensemble classifier over the selected features. Experimental results show that the proposed scheme can substantially improve the retrieval performance of the region-based image retrieval system.

Relevance feedback is a powerful technique to enhance Content-Based Image Retrieval (CBIR) performance. It solicits the user's relevance judgments on the retrieved images returned by the CBIR systems. The user's labeling is then used to learn a classifier to distinguish between relevant and irrelevant images. However, the top returnedimages may not be the most informative ones. The challenge is thus to determine which unlabeled images would be the most informative (i.e., improve the classifier the most) if they were labeled and used as training samples. In this paper, we propose a novel active learning algorithm, called Laplacian Optimal Design (LOD), for relevance feedback image retrieval. Our algorithm is based on aregression model which minimizes the least square error on the measured (or, labeled) images and simultaneously preserves the local geometrical structure of the image space. Specifically, we assume that if two images are sufficiently close to each other, then their measurements (or, labels) are close as well. By constructing a nearest neighbor graph, the geometrical structure of the image space can be described by the graph Laplacian. We discuss how results from the field of optimal experimental design may be used to guide our selection of a subset of images, which gives us the most amount of information. Experimental results on Corel database suggest that theproposed approach achieves higher precision in relevance feedback image retrieval.

This paper describes the unsupervised-interactive learning method, using the self-organizing tree map (SOTM) architecture, for the automation of relevance feedback (RF) in content-based image retrieval. The SOTM is shown to exhibit good behavior in relevance classification; providing a possible solution to minimizing user interactions in both fully automatic and semiautomatic domains, while achieving high retrieval accuracy in the context of adaptive retrieval. Computer simulation shows this system is very effective when applied to compressed domain retrieval systems for texture retrieval and the JPEG photograph database applications.

This chapter studies the user relevance feedback in image retrieval. We take this problem as a standard two-class pattern classification problem aiming at refining the retrieval precision by learning through the user relevance feedback data. However, we have investigated the problem by noting two important unique characteristics of the problem: small sample collection and asymmetric sample distributions between positive and negative samples. We have developed a novel approach to empirical Bayesian learning to solve for this problem by explicitly exploiting the two unique characteristics, which is the methodology of BAyesian Learning in ASymmetric and Small sample collections, thus called BALAS. In BALAS different learning strategies are used for positive and negative sample collections, respectively, based on the two unique characteristics. By defining the relevancy confidence as the relevant posterior probability, we have developed an integrated ranking scheme in BALAS which complementarily combines the subjective relevancy confidence and the objective similarity measure to capture the overall retrieval semantics. The experimental evaluations have confirmed the rationale of the proposed ranking scheme, and have also demonstrated that BALAS is superior to an existing relevance feedback method in the current literature in capturing the overall retrieval semantics.

Combining manifold ranking with active learning (MRAL for short) is one popular and successful technique for relevance feedback in content-based image retrieval (CBIR). Despite the success, conventional MRAL has two main drawbacks. First, the performance of manifold ranking is very sensitive to the scale parameter used for calculating the Laplacian matrix. Second, conventional MRAL does not take into account the redundancy among examples and thus could select multiple examples that are similar to each other. In this work, a novel MRAL framework is presented to address the drawbacks. Concretely, we first propose a self-tuning manifold ranking algorithm that can adaptively calculate the Laplacian matrix via a local scaling mechanism, and then develop a hybrid active learning algorithm by integrating three well-known selective sampling criteria, which is able to effectively and efficiently identify the most informative and diversified examples for the user to label. Experiments on 10,000 Corel images show that the proposed method is significantly more effective than some existing approaches.

Relevance feedback is an important mechanism for narrowing the semantic gap in content-based image retrieval and the process involves the user labeling positive and negative images. Very often, it is some specific objects or regions in the positive feedback images that the user is really interested in rather than the entire image. This paper presents a hierarchical graphical model for automatically extracting objects and regions that the user is interested in from the positive images which in turn are used to derive features that better reflect the user’s feedback intentions for improving interactive image retrieval. The novel hierarchical graphical model embeds image formation prior, user intention prior and statistical prior in its edges and uses a max-flow/min-cut method to simultaneously segment all positive feedback images into user interested and user uninterested regions. An important innovation of the graphical model is the introduction of a layer of visual appearance prototype nodes to incorporate user intention and form bridges linking similar objects in different images. This architecture not only makes it possible to use all feedback images to obtain more robust user intention prior thus improving the object segmentation results and in turn enhancing the retrieval performance, but also greatly reduces the complexity of the graph and the computational cost. Experimental results are presented to demonstrate the effectiveness of the new method.

Many previous studies have shown that naturally occurring data cannot possibly fill up the high dimensional space uniformly, rather it must concentrate around lower dimensional structure. The typical supervised subspace learning algorithms to discover this low dimensional structure include Linear Discriminant Analysis (LDA). For LDA, the training data points are usually pre-given. However, in some real world applications like relevance feedback image retrieval, there is opportunity to interact with the user and actively select the training points for labeling. In this paper, we propose a novel active subspace learning algorithm which selects the most informative data points and uses them for learning an optimal subspace. Using techniques from experimental design, we discuss how to perform data selection in supervised or semi-supervised subspace learning by minimizing the expected error. Experiments on image retrieval show improvement over state-of-the-art methods.

With the rapid growing volume of images in medical databases, development of efficient image retrieval systems to retrieve relevant or similar images to a query image has become an active research area. Despite many efforts to improve the performance of techniques for accurate image retrieval, its success in biomedicine thus far has been quite limited. This article presents an adaptive content-based image retrieval (CBIR) system for improving the performance of image retrieval in mammographic databases. In this work, the authors propose a new relevance feedback approach based on incremental learning with support vector machine (SVM) regression. Also, the authors present a new local perturbation method to further improve the performance of the proposed relevance feedback system. The approaches enable efficient online learning by adapting the current trained model to changes prompted by the user's relevance feedback, avoiding the burden of retraining the CBIR system. To demonstrate the proposed image retrieval system, the authors used two mammogram data sets: A set of 76 mammograms scored based on geometrical similarity and a larger set of 200 mammograms scored by expert radiologists based on pathological findings. The experimental results show that the proposed relevance feedback strategy improves the retrieval precision for both data sets while achieving high efficiency compared to offline SVM. For the data set of 200 mammograms, the authors obtained an average precision of 0.48 and an area under the precision-recall curve of 0.79. In addition, using the same database, the authors achieved a high pathology matching rate greater than 80% between the query and the top retrieved images after relevance feedback. Using mammographic databases, the results demonstrate that the proposed approach is more accurate than the model without using relevance feedback not only in image retrieval but also in pathology matching while maintaining its effectiveness for online relevance feedback applications.

The traditional content-based image retrieval system can not meet the users’ needs due to the ‘semantic gap’ between low-level features and the richness of human semantics. In an attempt to narrow the ‘semantic gap’, an approach which combines region-based image retrieval (RBIR) and relevance feedback (RF) with semantic color and spatial location features is presented in this paper. This method segments images into homogeneous regions and gives name for each region. In addition, the importance of each region is also taken into account. The Earth Mover’s Distance (EMD) is employed to measure the image similarity. To further improve the efficiency and accuracy of retrieval system, relevance feedback techniques are introduced in this paper. Experimental results on 1000 images database demonstrate the effectiveness and the efficiency of our approach is encouraging.

Image retrieval is an active area of research, which is growing very rapidly. Indeed, stimulated by the rapid growth in storage capacity and processing speed, the number of images in electronic collections and the World Wide Web has considerably increased over the last few years. However, with this abundance of information, people are continuously looking for tools that help them find the image(s) they are looking for within a reasonable amount of time. These tools are image retrieval engines. When using an image retrieval engine, the user is continuously interacting with the machine. First, he1 uses the system’s interface to formulate a query that expresses his needs. Second, he provides feedback about the retrieved results at each search iteration. This allows the engine to provide more accurate results by using relevance feedback (RF) techniques. Third, he may be asked to assign a goodness score or weight to each image retrieved, which helps evaluating the system’s performance. In this chapter, we will review the main interactions between human and the machine in the context of image retrieval. We will address several issues, including: Query formulation: • How the user expresses his needs and what he is looking for • The different ways the query can be formulated: keywords-based, sentence-based, query by example image, query by sketch, query by feature values, composite queries, etc. • Query by region of interest (ROI) vs. global query. • Queries with positive example only vs. queries with both positive and negative examples. • Page zero problem: finding a good image to initiate a retrieval session. Relevance feedback: we will try to answer questions like: • Why do systems use relevance feedback? • How can the user express his needs during the relevance feedback process • How this information is exploited by the system to perform operations like feature selection or the identification of the sought image. 1 Note that the masculine gender has been used strictly to facilitate reading, and is to be understood to include the feminine. Advances in Human-Computer Interaction 216 • The different families of RF techniques. • Relevance feedback with retrieval memory, i.e., taking into account the value of old iteration queries when constructing the new one. • Whether it is useful for the system to create user profiles, and the challenges it has to face. • The number of RF iterations required to obtain satisfactory results. Viewing retrieval results: • Existing viewing techniques: 2D linear presentation, 3D-based presentation, etc. • Different ways the resulting images may be ordered and presented to the user: similarity-ordered, time-ordered, event-ordered, etc. Evaluation of the retrieval performance by the user: • How the user can express his satisfaction/dissatisfaction about the retrieved images • What about the ground truth in image retrieval evaluation? • System response time and its influence on user satisfaction. • The ease of use of the system’s interface. Other issues: • User’s needs: He may be looking for a specific image, for images that meet a given need (e.g. illustrate a concept) or simply browsing the collection looking for potentially “good” images.