Speeding up active relevance feedback with approximatekNN retrieval for hyperplane queries

In content-based image retrieval (CBIR), a set of low-level features are extracted from an image to represent its visual content. Retrieval is performed by image example where a query image is given as input by the user and an appropriate similarity measure is used to find the best matches in the corresponding feature space. This approach suffers from the fact that there is a large discrepancy between the low-level visual features that one can extract from an image and the semantic interpretation of the image’s content that a particular user may have in a given situation. That is, users seek semantic similarity, but we can only provide similarity based on low-level visual features extracted from the raw pixel data, a situation known as the semantic gap. The selection of an appropriate similarity measure is thus an important problem. Since visual content can be represented by different attributes, the combination and importance of each set of features varies according to the user’s semantic intent. Thus, the retrieval strategy should be adaptive so that it can accommodate the preferences of different users. Relevance feedback (RF) learning has been proposed as a technique aimed at reducing the semantic gap. It works by gathering semantic information from user interaction. Based on the user’s feedback on the retrieval results, the retrieval scheme is adjusted. By providing an image similarity measure under human perception, RF learning can be seen as a form of supervised learning that finds relations between high-level semantic interpretations and low-level visual properties. That is, the feedback obtained within a single query session is used to personalize the retrieval strategy and thus enhance retrieval performance. In this chapter we present an overview of CBIR and related work on RF learning. We also present our own previous work on a RF learning-based probabilistic region relevance learning algorithm for automatically estimating the importance of each region in an image based on the user’s semantic intent.

In order to reduce the “semantic gap” between the underlying features and the high-level semantics in the image retrieval, the feature re-weighted correlation feedback mechanism is applied. The basic idea is to reduce the weight of the irrelevant feature dimension in the feature space and increase the weight of the relevant feature dimension. The improved method of feature weight adjustment proposed in this paper is compared with another method. The accuracy of the algorithm is proved by the experiment as the criterion of the evaluation effect.

The traditional content-based image retrieval system can not meet the users’ needs due to the ‘semantic gap’ between low-level features and the richness of human semantics. In an attempt to narrow the ‘semantic gap’, an approach which combines region-based image retrieval (RBIR) and relevance feedback (RF) with semantic color and spatial location features is presented in this paper. This method segments images into homogeneous regions and gives name for each region. In addition, the importance of each region is also taken into account. The Earth Mover’s Distance (EMD) is employed to measure the image similarity. To further improve the efficiency and accuracy of retrieval system, relevance feedback techniques are introduced in this paper. Experimental results on 1000 images database demonstrate the effectiveness and the efficiency of our approach is encouraging.

With the rapid growing volume of images in medical databases, development of efficient image retrieval systems to retrieve relevant or similar images to a query image has become an active research area. Despite many efforts to improve the performance of techniques for accurate image retrieval, its success in biomedicine thus far has been quite limited. This article presents an adaptive content-based image retrieval (CBIR) system for improving the performance of image retrieval in mammographic databases. In this work, the authors propose a new relevance feedback approach based on incremental learning with support vector machine (SVM) regression. Also, the authors present a new local perturbation method to further improve the performance of the proposed relevance feedback system. The approaches enable efficient online learning by adapting the current trained model to changes prompted by the user's relevance feedback, avoiding the burden of retraining the CBIR system. To demonstrate the proposed image retrieval system, the authors used two mammogram data sets: A set of 76 mammograms scored based on geometrical similarity and a larger set of 200 mammograms scored by expert radiologists based on pathological findings. The experimental results show that the proposed relevance feedback strategy improves the retrieval precision for both data sets while achieving high efficiency compared to offline SVM. For the data set of 200 mammograms, the authors obtained an average precision of 0.48 and an area under the precision-recall curve of 0.79. In addition, using the same database, the authors achieved a high pathology matching rate greater than 80% between the query and the top retrieved images after relevance feedback. Using mammographic databases, the results demonstrate that the proposed approach is more accurate than the model without using relevance feedback not only in image retrieval but also in pathology matching while maintaining its effectiveness for online relevance feedback applications.

To overcome the semantic gap in the retrieval process, the user relevance feedback frame work is added to the current image retrieval system. Relevance feedback method iteratively refines and updates the retrieved result by learning the user-labeled examples to further improves the overall performance. In the proposed scheme feature descriptor derived from the EDBTC compressed data stream. Firstly, a color image is decomposed using EDBTC scheme to produce two new image representations, namely color quantizer and bitmap image by using Improved LBG-VQ (ILBG)algorithm. Two image feature descriptors called Color Histogram Feature (CHF), and Bit Pattern Feature(BPF) can be subsequently generated from the EDBTC color quantizer and its corresponding bitmap image respectively without performing the decoding process. The similarity degree between two images is simply measured with the similarity distance score of their feature descriptor. Second, we presented a relevance feedback (RF) framework for effective image retrieval by using a support vector machine (SVM). Effectiveness of EDBTC feature descriptor is quantitatively examined and compared in the RGB color space as well as in HSI color channel. Extensive experiments shows that there is a significant improvement in retrieval performance with the proposed method using SVMRF compared with the retrieval performance without RF.

Although relevance feedback (RF) has been extensively studied in the content-based image retrieval community, no commercial Web image search engines support RF because of scalability, efficiency, and effectiveness issues. In this paper, we propose a unified relevance feedback framework for Web image retrieval. Our framework shows advantage over traditional RF mechanisms in the following three aspects. First, during the RF process, both textual feature and visual feature are used in a sequential way. To seamlessly combine textual feature-based RF and visual feature-based RF, a query concept-dependent fusion strategy is automatically learned. Second, the textual feature-based RF mechanism employs an effective search result clustering (SRC) algorithm to obtain salient phrases, based on which we could construct an accurate and low-dimensional textual space for the resulting Web images. Thus, we could integrate RF into Web image retrieval in a practical way. Last, a new user interface (UI) is proposed to support implicit RF. On the one hand, unlike traditional RF UI which enforces users to make explicit judgment on the results, the new UI regards the users' click-through data as implicit relevance feedback in order to release burden from the users. On the other hand, unlike traditional RF UI which hardily substitutes subsequent results for previous ones, a recommendation scheme is used to help the users better understand the feedback process and to mitigate the possible waiting caused by RF. Experimental results on a database consisting of nearly three million Web images show that the proposed framework is wieldy, scalable, and effective.

Image retrieval is an active area of research, which is growing very rapidly. Indeed, stimulated by the rapid growth in storage capacity and processing speed, the number of images in electronic collections and the World Wide Web has considerably increased over the last few years. However, with this abundance of information, people are continuously looking for tools that help them find the image(s) they are looking for within a reasonable amount of time. These tools are image retrieval engines. When using an image retrieval engine, the user is continuously interacting with the machine. First, he1 uses the system’s interface to formulate a query that expresses his needs. Second, he provides feedback about the retrieved results at each search iteration. This allows the engine to provide more accurate results by using relevance feedback (RF) techniques. Third, he may be asked to assign a goodness score or weight to each image retrieved, which helps evaluating the system’s performance. In this chapter, we will review the main interactions between human and the machine in the context of image retrieval. We will address several issues, including: Query formulation: • How the user expresses his needs and what he is looking for • The different ways the query can be formulated: keywords-based, sentence-based, query by example image, query by sketch, query by feature values, composite queries, etc. • Query by region of interest (ROI) vs. global query. • Queries with positive example only vs. queries with both positive and negative examples. • Page zero problem: finding a good image to initiate a retrieval session. Relevance feedback: we will try to answer questions like: • Why do systems use relevance feedback? • How can the user express his needs during the relevance feedback process • How this information is exploited by the system to perform operations like feature selection or the identification of the sought image. 1 Note that the masculine gender has been used strictly to facilitate reading, and is to be understood to include the feminine. Advances in Human-Computer Interaction 216 • The different families of RF techniques. • Relevance feedback with retrieval memory, i.e., taking into account the value of old iteration queries when constructing the new one. • Whether it is useful for the system to create user profiles, and the challenges it has to face. • The number of RF iterations required to obtain satisfactory results. Viewing retrieval results: • Existing viewing techniques: 2D linear presentation, 3D-based presentation, etc. • Different ways the resulting images may be ordered and presented to the user: similarity-ordered, time-ordered, event-ordered, etc. Evaluation of the retrieval performance by the user: • How the user can express his satisfaction/dissatisfaction about the retrieved images • What about the ground truth in image retrieval evaluation? • System response time and its influence on user satisfaction. • The ease of use of the system’s interface. Other issues: • User’s needs: He may be looking for a specific image, for images that meet a given need (e.g. illustrate a concept) or simply browsing the collection looking for potentially “good” images.

In order to improve the retrieval accuracy of image retrieval, semantic-based image retrieval has become popular in the recent years. It is because this kind of retrieval method could narrow “semantic gap” between the visual features and the high-level semantic features. However, most of the existing methods of the semantic-based image retrieval are limited to fixed number of semantic features. A dynamic hierarchical semantic network method is proposed in this paper to overcome this limitation. The proposed dynamic hierarchical semantic network is constructed by relevance feedback. The number of semantic features can be dynamically changed according to user feedbacks. Moreover, the semantic features are allowed to have different levels of abstraction. In addition, the proposed method also integrates low-level visual feature-based image retrieval style, which could full use of the advantages of visual feature-based image retrieval and semantic-based image retrieval. Experimental results show that the proposed method achieves higher retrieval accuracy than fixed number of semantic feature method and only one level semantic feature method.

We systematically compare four variants of evolutionary algorithms (EAs) for relevance feedback (RF) in image retrieval. We focus on the small sample size (SSS) performance, i.e., the requirement to learn the user’s information need based on a small – between 2 and 25 – number of positive and negative training images. Despite this being a fundamental requirement, none of the existing works dealing with EAs for RF in image retrieval addresses it. Common for the four variants is the hierarchical, region-based image similarity model, with region and feature weights as parameters. The difference between the variants is in the objective function of the EA used to adjust the model parameters. The objective functions include: (O-1) precision; (O-2) average rank; (O-3) ratio of within-class (i.e., positive images) and between-class (i.e., positive and negative images) scatter; and (O-4) combination of O-2 and O-3. We note that – unlike O-1 and O-2 – O-3 and O-4 are not used in any of the existing works dealing with EAs for RF. The four variants are evaluated on five test databases, containing 61,895 general-purpose images, in 619 semantic categories. Results of the evaluation reveal that variants with objective functions O-3 and O-4 consistently outperform those with O-1 and O-2. Furthermore, comparison with the representative of the existing RF methods shows that EAs are both effective and efficient approaches for SSS learning in region-based image retrieval.

Content Based Image Retrieval (CBIR) depends on the visual content of an image for obtaining similar images. Research on CBIR faces a challenge of semantic gap between low level features and high level visual content. The optimal solution must reduce the computational complexity and computational time in retrieval process. Relevance Feedback is a solution to reduce the semantic gap. A new framework for content based image retrieval is proposed which is Bipartite Reiterative algorithm. This gives an approach of iteration to relevance feedback mechanism. It consists of two levels which are basic retrieval scheme and reiterative feedback scheme. Positive images are obtained from the user as part of the basic retrieval scheme. New feedback algorithm is developed based on weights assigned to visual attributes. The weights are inversely proportional to the visual attribute distance between query image and the positive images given by the user. Iterative feedback query can be performed after the user feedback. The query will be progressed according to the user feedback. The algorithm reduces the computational complexity and computational time in image retrieval.

We evaluate the small sample size (SSS) performance of evolutionary algorithms (EAs) for relevance feedback (RF) in image retrieval. We focus on the requirement to learn the user’s information need based on a small — between 2 and 25 — number of positive and negative training images. Despite this being a fundamental requirement, none of the existing works dealing with EAs for RF systematically evaluates their SSS performance. To address this issue, we compare four variants of EAs for RF. Common for all variants is the hierarchical, region-based image similarity model, with region and feature weights as parameters. The difference between the variants is in the objective function of the EA used to adjust the model parameters. The objective functions include: (O-1) precision; (O-2) average rank; (O-3) ratio of within-class (i.e., positive images) and between-class (i.e., positive and negative images) scatter; and (O-4) combination of O-2 and O-3. We note that — unlike O-1 and O-2 — O-3 and O-4 are not used in any of the existing works dealing with EAs for RF. The four variants are evaluated on five test databases, containing 61,895 general-purpose images, in 619 semantic categories. Results of the evaluation reveal that variants with objective functions O-3 and O-4 consistently outperform those with O-1 and O-2. Furthermore, comparison with the representative of the existing RF methods shows that EAs are both effective and efficient approaches for SSS learning in region-based image retrieval.

Image retrieval with relevant feedback on large and high-dimensional image databases is a challenging task. In this paper, we propose an image retrieval method, called BCFIR (Binary Codes for Fast Image Retrieval). BCFIR utilizes sparse discriminant analysis to select the most important original feature set, and solve the small class problem in the relevance feedback. Besides, to increase the retrieval performance on large-scale image databases, in addition to BCFIR mapping real-valued features to short binary codes, it also applies a bagging learning strategy to improve the ability general capabilities of autoencoders. In addition, our proposed method also takes advantage of both labeled and unlabeled samples to improve the retrieval precision. The experimental results on three databases demonstrate that the proposed method obtains competitive precision compared with other state-of-the-art image retrieval methods.

Relevance feedback (RF) schemes have been widely designed to improve the performance of content-based image retrieval. Despite the success, it is not appropriate to require the user to label a large number of samples in RF. Collaborative image retrieval (CIR) aims to reduce the labeling efforts of the user by resorting to the auxiliary information. Support vector machine (SVM) active learning can select ambiguous samples as the most informative ones for the user to label with the help of the optimal hyperplane of SVM, and thus alleviate the labeling efforts of conventional RF. However, the optimal hyperplane of SVM is usually unstable and inaccurate with small-sized training data, and this is always the case in image retrieval since the user would not like to label a large number of feedback samples and cannot label each sample accurately all the time. In this paper, we propose a novel active learning method, i.e., geometric optimum experimental design (GOED), to select multiple representative samples in the database as the most informative ones for the user to label. Especially, GOED can alleviate the small-sized training data problem by leveraging the geometric structure of unlabeled samples in the reproducing kernel Hilbert space and thus further enhance the performance of image retrieval. Different from the conventional manifold regularization framework, the new method can effectively select the most informative samples for the user to label in image retrieval. By minimizing the expected average prediction variance on the test data, GOED has a clear geometric interpretation to select a set of the most representative samples in the database iteratively with the global optimum. Compared with the popular SVM active learning, our method is label-independent and can effectively avoid various potential problems caused by insufficient and inexactly labeled samples in RF, and is more appropriate and useful for image retrieval. Extensive experiments on both synthetic datasets and a real-world image database have been conducted to show the advantages of the proposed GOED for CIR.

Relevance feedback has emerged as a powerful tool to boost the retrieval performance in content-based image retrieval (CBIR). In the past, most research efforts in this field have focused on designing effective algorithms for traditional relevance feedback. Given that a CBIR system can collect and store users' relevance feedback information in a history log, an image retrieval system should be able to take advantage of the log data of users' feedback to enhance its retrieval performance. In this paper, we propose a unified framework for log-based relevance feedback that integrates the log of feedback data into the traditional relevance feedback schemes to learn effectively the correlation between low-level image features and high-level concepts. Given the error-prone nature of log data, we present a novel learning technique, named soft label support vector machine, to tackle the noisy data problem. Extensive experiments are designed and conducted to evaluate the proposed algorithms based on the COREL image data set. The promising experimental results validate the effectiveness of our log-based relevance feedback scheme empirically.

Relevance feedback has emerged as a powerful tool to boost the retrieval performance in content-based image retrieval (CBIR). In the past, most research efforts in this field have focused on designing effective algorithms for traditional relevance feedback. Given that a CBIR system can collect and store users' relevance feedback information in a history log, an image retrieval system should be able to take advantage of the log data of users' feedback to enhance its retrieval performance. In this paper, we propose a unified framework for log-based relevance feedback that integrates the log of feedback data into the traditional relevance feedback schemes to learn effectively the correlation between low-level image features and high-level concepts. Given the error-prone nature of log data, we present a novel learning technique, named soft label support vector machine, to tackle the noisy data problem. Extensive experiments are designed and conducted to evaluate the proposed algorithms based on the COREL image data set. The promising experimental results validate the effectiveness of our log-based relevance feedback scheme empirically.