Integration of spatial relationships in visual language model for scene retrieval

One of the main challenges in medical image retrieval is the increasing volume of image data, which render it difficult for domain experts to find relevant information from large data sets. Effective and efficient medical image retrieval systems are required to better manage medical image information. Text‐based image retrieval (TBIR) was very successful in retrieving images with textual descriptions. Several TBIR approaches rely on models based on bag‐of‐words approaches, in which the image retrieval problem turns into one of standard text‐based information retrieval; where the meanings and values of specific medical entities in the text and metadata are ignored in the image representation and retrieval process. However, we believe that TBIR should extract specific medical entities and terms and then exploit these elements to achieve better image retrieval results. Therefore, we propose a novel reranking method based on medical‐image‐dependent features. These features are manually selected by a medical expert from imaging modalities and medical terminology. First, we represent queries and images using only medical‐image‐dependent features such as image modality and image scale. Second, we exploit the defined features in a new reranking method for medical image retrieval. Our motivation is the large influence of image modality in medical image retrieval and its impact on image‐relevance scores. To evaluate our approach, we performed a series of experiments on the medical ImageCLEF data sets from 2009 to 2013. The BM25 model, a language model, and an image‐relevance feedback model are used as baselines to evaluate our approach. The experimental results show that compared to the BM25 model, the proposed model significantly enhances image retrieval performance. We also compared our approach with other state‐of‐the‐art approaches and show that our approach performs comparably to those of the top three runs in the official ImageCLEF competition.

This paper presents a novel approach, named bag-of-bags of words (BBoW), to address the problem of Content-Based Image Retrieval (CBIR) from image databases. The proposed bag-of-bags of words model extends the classical bag-of-words (BoW) model. An image is represented as a connected graph of local features on a regular grid. Then irregular partitions (subgraphs) of images are further built via Normalized Cuts. Each subgraph in the partition is then represented by its own signature. Compared to existing methods for image retrieval, such as Spatial Pyramid Matching (SPM), the BBoW model does not assume that similar parts of a scene always appear at the same location in images of the same category. The extension of the proposed model to pyramid gives rise to a method we name irregular pyramid matching (IPM). The experiments demonstrate the strength of our method for image retrieval when the partitions are stable across an image category. The statistical analysis of subgraphs is discussed in the paper.

Traditionally, many information retrieval models assume that terms occur in documents independently. Although these models have already shown good performance, the word independency assumption seems to be unrealistic from a natural language point of view, which considers that terms are related to each other. Therefore, such an assumption leads to two well‐known problems in information retrieval (IR), namely, polysemy, or term mismatch, and synonymy. In language models, these issues have been addressed by considering dependencies such as bigrams, phrasal‐concepts, or word relationships, but such models are estimated using simple n‐grams or concept counting. In this paper, we address polysemy and synonymy mismatch with a concept‐based language modeling approach that combines ontological concepts from external resources with frequently found collocations from the document collection. In addition, the concept‐based model is enriched with subconcepts and semantic relationships through a semantic smoothing technique so as to perform semantic matching. Experiments carried out on TREC collections show that our model achieves significant improvements over a single word‐based model and the Markov Random Field model (using a Markov classifier).

In this paper, we propose a novel dependency language modeling approach for information retrieval. The approach extends the existing language modeling approach by relaxing the independence assumption. Our goal is to build a language model in which various word relationships can be integrated. In this work, we integrate two types of relationship extracted from WordNet and co-occurrence relationships respectively. The integrated model has been tested on several TREC collections. The results show that our model achieves substantial and significant improvements with respect to the models without these relationships. These results clearly show the benefit of integrating word relationships into language models for IR.

Text contained in an image carries high-level semantics that can be exploited to achieve richer image understanding. In particular, the mere presence of text provides strong guiding content that should be employed to tackle a diversity of computer vision tasks such as image retrieval, fine-grained classification, and visual question answering. In this paper, we address the problem of fine-grained classification and image retrieval by leveraging textual information along with visual cues to comprehend the existing intrinsic relation between the two modalities. The novelty of the proposed model consists of the usage of a PHOC descriptor to construct a bag of textual words along with a Fisher Vector Encoding that captures the morphology of text. This approach provides a stronger multimodal representation for this task and as our experiments demonstrate, it achieves state-of-the-art results on two different tasks, fine-grained classification and image retrieval. The code of this model will be publicly available at1.

In this article, we propose a novel system for feature selection, which is one of the key problems in content-based image indexing and retrieval as well as various other research fields such as pattern classification and genomic data analysis. The proposed system aims at enhancing semantic image retrieval results, decreasing retrieval process complexity, and improving the overall system usability for end-users of multimedia search engines. Three feature selection criteria and a decision method construct the feature selection system. Two novel feature selection criteria based on inner-cluster and intercluster relations are proposed in the article. A majority voting-based method is adapted for efficient selection of features and feature combinations. The performance of the proposed criteria is assessed over a large image database and a number of features, and is compared against competing techniques from the literature. Experiments show that the proposed feature selection system improves semantic performance results in image retrieval systems.

Previous research has shown that passage-level evidence can bring added benefits to document retrieval when documents are long or span different subject areas. Recent developments in language modeling approach to IR provided a new effective alternative to traditional retrieval models. These two streams of research motivate us to examine the use of passages in a language model framework. This paper reports on experiments using passages in a simple language model and a relevance model, and compares the results with document-based retrieval. Results from the INQUERY search engine, which is not based on a language modeling approach, are also given for comparison. Test data include two heterogeneous and one homogeneous document collections. Our experiments show that passage retrieval is feasible in the language modeling context, and more importantly, it can provide more reliable performance than retrieval based on full documents.

As the number and diversity of distributed Web databases on the Internet exponentially increase, it is difficult for user to know which databases are appropriate to search. Given database language models that describe the content of each database, database selection services can provide assistance in locating databases relevant to the information needs of users. In this paper, we propose a database selection approach based on statistical language modeling. The basic idea behind the approach is that, for databases that are categorized into a topic hierarchy, individual language models are estimated at different search stages, and then the databases are ranked by the similarity to the query according to the estimated language model. Two-stage smoothed language models are presented to circumvent inaccuracy due to word sparseness. Experimental results demonstrate that such a language modeling approach is competitive with current state-of-the-art database selection approaches.

Dimensionality reduction (DR) of image features plays an important role in image retrieval and classification tasks. Recently, two types of methods have been proposed to improve both the accuracy and efficiency for the dimensionality reduction problem. One uses Non-negative matrix factorization (NMF) to describe the image distribution on the space of base matrix. Another one for dimension reduction trains a subspace projection matrix to project original data space into some low-dimensional subspaces which have deep architecture, so that the low-dimensional codes would be learned. At the same time, the graph based similarity learning algorithm which tries to exploit contextual information for improving the effectiveness of image rankings is also proposed for image class and retrieval problem. In this paper, after above two methods mentioned are utilized to reduce the high-dimensional features of images respectively, we learn the graph based similarity for the image classification problem. This paper compares the proposed approach with other approaches on an image database.

Hidden Markov models (HMMs) have been widely used in various fields, including image categorization and retrieval. Most of the existing methods train HMMs by low-level features of image blocks; however, the blockbased features can not reflect high-level semantic concepts well. This paper proposes a new method to train HMMs by region-based features, which can be obtained after image segmentation. Our work can be characterized by two key properties: (1) Region-based HMM is adopted to achieve better categorization performance, for the region-based features accord with the human perception better. (2) Multi-layer semantic representation (MSR) is introduced to couple with region-based HMM in a long-term relevance feedback framework for image retrieval. The experimental results demonstrate the effectiveness of our proposal in both aspects of categorization and retrieval.

Scene Text Deblurring in Non-stationary Video Sequences

In this paper, we propose a new color edge model and color edge histogram descriptor for contents-based image retrieval. The edge descriptor proposed by MPEG-7 standard is a representative approach for the contents-based image retrieval using the edge histogram that is derived from a gray-level image. This paper introduces a novel method that extracts color edge information from spectral color images rather than monochrome images and a new color edge histogram descriptor for the contents-based image retrieval. The proposed color edge model is obtained in two phases. In the first phase, we characterize the R,G,B channel components as a linear map (the differential) and impose a statistical interpretation on this map by considering the behavior of the map as it applied to unit normed vectors in the second phase. As a result, the constructed edge model will be expressed in a statistical fashion and will provide a mechanism to determine the possibility of the edge existence. The color edge histogram based on the direction of the color edge model is subsequently applied to the contents-based e-catalogue image retrieval. For the evaluation, the results of image retrieval using the proposed method are compared with those of image retrieval using the edge descriptor by MPEG-7 and other approaches. The experimental result supports the efficiency of the proposed method.KeywordsColor ImageImage RetrievalQuery ImageColor HistogramColor EdgeThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In the context of content-based image retrieval (CBIR) problem, an image is represented by feature vectors called descriptors, whose efficiency is essential to obtain a good performance in the image indexing and retrieval tasks. In this paper, we propose an algorithm to obtain an efficient color-based descriptor, which is a combination of Color Correlogram (CC) and Dominant Color (DC). The color-based descriptors, such as Histogram Intersection (HI) and DC take into account the global distribution of color in an image, while CC takes into account the local color distribution. So the combination of global and local color distribution provides a good image description. By its design, the proposed descriptor is more compact compared with the CC descriptor, which allows reducing computational complexity. Using the Average Retrieval precision (ARP) with different factors the effectiveness of the proposed descriptor is evaluated and compared with the conventional color-based descriptors, such as HI, CC and DC. The image database used in this work contains 500 images with 25 categories randomly selected from the Corel Dataset.

Content-based image retrieval (CBIR) has turned into an important and active potential research field with the advance of multimedia and imaging technology. It makes use of image features, such as color, texture and shape, to index images with minimal human intervention. A CBIR system can be used to locate medical images in large databases. In this paper we propose a CBIR system which describes the methodology for retrieving digital human brain magnetic resonance images (MRI) based on textural features and the Adaptive neuro-fuzzy inference system (ANFIS) learning to retrieve similar images from database in two categories: normal and tumoral. A fuzzy classifier has been used, because of the uncertainty in the results of classifier and capacity of learning. ANFIS is a good candidate for our categorization problem. Our proposed CBIR system can locate a query image in the category of normal or tumoral images in the online retrieval part. Finally, using a relevance feedback, we improve the effectiveness of our retrieval system. This research uses the knowledge of the CBIR approach to the application of medical decision support and discrimination between the normal and abnormal medical images based on features. We present and compare the results of the proposed method with the CBIR systems used in recent works. The experimental results indicate that the proposed method is reliable and has high image retrieval efficiency compared with the previous works.

This paper presents an image representation and matching framework for image categorization in medical image archives. Categorization enables one to determine automatically, based on the image content, the examined body region and imaging modality. It is a basic step in content-based image retrieval (CBIR) systems, the goal of which is to augment text-based search with visual information analysis. CBIR systems are currently being integrated with picture archiving and communication systems for increasing the overall search capabilities and tools available to radiologists. The proposed methodology is comprised of a continuous and probabilistic image representation scheme using Gaussian mixture modeling (GMM) along with information-theoretic image matching via the Kullback-Leibler (KL) measure. The GMM-KL framework is used for matching and categorizing X-ray images by body regions. A multidimensional feature space is used to represent the image input, including intensity, texture, and spatial information. Unsupervised clustering via the GMM is used to extract coherent regions in feature space that are then used in the matching process. A dominant characteristic of the radiological images is their poor contrast and large intensity variations. This presents a challenge to matching among the images, and is handled via an illumination-invariant representation. The GMM-KL framework is evaluated for image categorization and image retrieval on a dataset of 1500 radiological images. A classification rate of 97.5% was achieved. The classification results compare favorably with reported global and local representation schemes. Precision versus recall curves indicate a strong retrieval result as compared with other state-of-the-art retrieval techniques. Finally, category models are learned and results are presented for comparing images to learned category models.