Text-based image retrieval system using semantic visual content for re-ranking

‘A visual idea is more powerful than verbal idea’, ‘A picture is worth more than ten thousand words’, ‘No words can convey what a picture speaks’, ‘A picture has to be seen and searched as a picture only’ are few of the well-known sayings that imply the certainty for the widespread availability of images. Common sense evidence suggests that images are required for a variety of reasons, like, illustration of text articles, conveying information or emotions that are difficult to describe in words, display of detailed data for analysis (medical images), formal recording of design data for later use (architectural plans) etc. The advent of digital photography combined with decreasing storage and processing cost, allows more and more people to have their personal collection of photographs and other visual content available on the internet. Organising these digital images into a small number of categories and providing effective indexing is imperative for accessing, browsing and retrieving useful data in “real time”. The process of digitization does not in itself make image collections easier to manage. Some form of indexing (cataloguing) is still necessary. People’s interest to have their own digital libraries has burgeoned and hence requires a data structure to preserve the images for a long time and also provide easy access to the desired images. These requirements have indeed forced the design of specialized imaging systems/ image databases, such that an access to any image is effective and efficient. An efficient image archival and retrieval system is characterized by its ability to retrieve relevant images based on their visual and semantic contents rather than using simple attributes or keywords assigned to them. Thus, it is necessary to support queries based on image semantics rather than mere-pixel-to-pixel matching. An image archival and retrieval system should therefore allow adequate abstraction mechanisms for capturing higher level semantics of images in order to support content addressability as far as possible. That is, for two images to be similar, not only the shape, color and texture properties of individual image regions must be similar, but also they must have the same arrangement (i.e., spatial relationships) in both the images. In fact, this is the strategy, which is generally being employed by our vision system most of the times. An effective method of representing images depends on the perception of knowledge embedded in images in terms of objects/components (generally known as elements) present in them along with their topological relationships. The perception of topological relationships, especially spatial relationships existing among the significant elements of an image, helps in making the image database system more intelligent, fast and flexible. An obvious method to search an image database is sequential scanning. The query is matched with all stored images (i.e., the representation of the query is matched with all representations stored in the image database) one by one. Retrievals may become extremely slow, especially when database search involves time consuming image matching operations. To deal with slow retrieval response times, and high complexity matching, an image database must utilize indexing methods that are faster than sequential scanning methods. In traditional image database systems, the use of indexing to allow database accessing has been well established. Analogously, image indexing techniques have been studied during the last decade to support representation of pictorial information in an image database and also to retrieve information from an image database. The use of significant elements present in images along with their topological relationships as indexes is the basic issue of the indexing methodologies developed to this aim.

Nowadays, image retrieval systems, emphasize the combining of high level semantics and low level visual features in the image indexing process and of course the performance of such systems for image retrieval is increased with perfect semantic network. In this research, a dynamic high level semantic network in synonym sets of keywords is used and a dynamic approach is presented for updating the semantic network and semantic contents of the images in an interactive way. The proposed approach, certainly, executes a novel kind of long term learning-based on user's opinion in the several interactions with system, and relates the behavior of system to the behavior of users in the recognition process of semantic similarity between the images. The evaluating results of system's performance show improving the accuracy of semantic network and semantic contents related to the images after applying the long term learning process.

Recently, digital content has become a significant and inevitable asset of or any enterprise and the need for visual content management is on the rise as well. Content-based image retrieval has attracted voluminous research in the last decade paving way for development of numerous techniques and systems besides creating interest on fields that support these systems. CBIR indexes the images based on the features obtained from visual content so as to facilitate speedy retrieval. In this thesis work, we present a steerable pyramid based image retrieval system that uses color, contours and texture as visual features to describe the content of an image region. We have initially used steerable pyramid to extract texture features from query image and database images and store them in feature vectors. Second, to speed up retrieval and similarity computation, the database images are classified and the extracted regions are clustered according to their feature vectors using median vector algorithm. This process is performed before query matching takes place. Therefore to answer a query our system does not need to search the entire database images; instead just a number of candidate images are required to be searched for image similarity. Our proposed system has the advantage of increasing the retrieval accuracy and decreasing the retrieval time. The experimental evaluation of the system is based on a satellite and medical image database. From the experimental results, it is evident that our system performs significantly better and faster compared with other existing systems. In our analysis, we provide a comparison between retrieval results based on features extracted from the whole image using steerable pyramid with median vector and features extracted from same image without median vector. The results demonstrate that each type of feature is effective for a particular type of images according to its semantic contents, and using a combination of them giving better retrieval results for almost all different classes of images in the dataset. Â

Relevance feedback is a powerful technique for image retrieval and has been an active research direction for the past few years. Various ad hoc parameter estimation techniques have been proposed for relevance feedback. In addition, methods that perform optimization on multilevel image content model have been formulated. However, these methods only perform relevance feedback on low-level image features and fail to address the images' semantic content. In this paper, we propose a relevance feedback framework to take advantage of the semantic contents of images in addition to low-level features. By forming a semantic network on top of the keyword association on the images, we are able to accurately deduce and utilize the images' semantic contents for retrieval purposes. We also propose a ranking measure that is suitable for our framework. The accuracy and effectiveness of our method is demonstrated with experimental results on real-world image collections.

One of the main challenges in medical image retrieval is the increasing volume of image data, which render it difficult for domain experts to find relevant information from large data sets. Effective and efficient medical image retrieval systems are required to better manage medical image information. Text‐based image retrieval (TBIR) was very successful in retrieving images with textual descriptions. Several TBIR approaches rely on models based on bag‐of‐words approaches, in which the image retrieval problem turns into one of standard text‐based information retrieval; where the meanings and values of specific medical entities in the text and metadata are ignored in the image representation and retrieval process. However, we believe that TBIR should extract specific medical entities and terms and then exploit these elements to achieve better image retrieval results. Therefore, we propose a novel reranking method based on medical‐image‐dependent features. These features are manually selected by a medical expert from imaging modalities and medical terminology. First, we represent queries and images using only medical‐image‐dependent features such as image modality and image scale. Second, we exploit the defined features in a new reranking method for medical image retrieval. Our motivation is the large influence of image modality in medical image retrieval and its impact on image‐relevance scores. To evaluate our approach, we performed a series of experiments on the medical ImageCLEF data sets from 2009 to 2013. The BM25 model, a language model, and an image‐relevance feedback model are used as baselines to evaluate our approach. The experimental results show that compared to the BM25 model, the proposed model significantly enhances image retrieval performance. We also compared our approach with other state‐of‐the‐art approaches and show that our approach performs comparably to those of the top three runs in the official ImageCLEF competition.

Multimodal image retrieval, which involves retrieving images using various modalities such as text, audio, or other images, has significant research importance due to its wide-ranging applications and potential to enhance user experiences across multiple domains. Traditional image retrieval systems, content-based image retrieval (CBIR) systems rely solely on visual features, which can be limiting. By integrating multiple modalities, multimodal retrieval systems can offer more robust and accurate results. A research problem is considered multimodal when it integrates information from more than one type of data source. In multimodal image retrieval (MMIR) systems, one form of data is used to search for outcomes in the same or different modalities. In contrast, cross-modal systems strictly retrieve information from a different modality. A significant challenge in these systems is the effective comparison of input-output queries from different data types, due to their basic forms and the subjective nature of content similarity. Researchers have proposed various techniques to address this challenge and to bridge the semantic gap in information retrieval across different modalities. This article presents comprehensive analysis of various research works pertained in this multimodal image retrieval. The results and comparative analysis of current research works on benchmark datasets have also been discussed. At the end of the paper, some of open issues are presented for future research directions.

Currently, image retrieval by content is a research problem of great interest in academia and the industry, due to the large collections of images available in different contexts. One of the main challenges to develop effective image retrieval systems is the automatic identification of semantic image contents. This research proposal aims to design a model for image retrieval able to take advantage of different data sources, i.e. using multimodal information, to improve the response of an image retrieval system. In particular two data modalities associated to contents and context of images are considered in this proposal: visual features and unstructured text annotations. The proposed framework is based on kernel methods that provide two main important advantages over the traditional multimodal approaches: first, the structure of each modality is preserved in a high dimensional feature space, and second, they provide natural ways to fuse feature spaces in a unique information space. This document presents the research agenda to build a Multimodal Information Space for searching images by content.

In this paper, we design and construct a multi-label image annotation and retrieval system. Various MPEG-7 low level visual features are employed for representing images. For image annotation, a bi-coded genetic algorithm is employed to select optimal feature subsets and corresponding optimal weights for every one vs. one SVM classifiers. After an unlabelled image is segmented into several regions, pre-trained SVMs are used to annotate each region, final label is obtained by merging all the region labels. Based on multi-label of image, image retrieval system provides keyword-based image retrieval service. Multi-labels can provide abundant descriptions for image content in semantic level, high precision annotation algorithm further improve annotation performance.

There is a growing trend for using content‐based image retrieval (CBIR) systems these days because of the constantly growing interest in digital content. Therefore, the ability of the CBIR to perform the CBIR process will depend on the feature extraction process and its basis, for the retrieval will be done on. Numerous researchers put forward various techniques for feature extraction to enhance the nature of the system. Since features play a very key role in enhancing performance, various features can be used collectively to attain the requisite goal. To retain this in mind, we present in this paper a multifeature fusion system, where three features are integrated and form one feature to improve the situation of retrieval. For this purpose, scale‐invariant feature transform (SIFT), speeded‐up robust features (SURF), and histogram of oriented gradients (HOG) features are adopted. These features are common features that deliver information about the shape of the object and for matching purposes, two techniques of distance matching such as Euclidean and Hausdrauff distance are adopted. To assess the performance of the proposed multifeature‐based CBIR approach, experiments were conducted with the usage of a MATLAB simulator. The Corel‐1000 dataset, consisting of 10,000 images in 100 semantic classes, turned into applied, with each magnificence containing 100 images. A subset of 2500 images across 50 semantic classes was used to train the system. This research aligns with industry, innovation, and infrastructure by contributing to advancements in image processing and retrieval systems. Key characteristic descriptors, along with SIFT, SURF, HOG, texture, and multicharacteristic combinations, were extracted for retrieval functions. The results display that the usage of the Hausdrauff distance as a similarity degree outperforms Euclidean distance, accomplishing retrieval accuracies of 80.02% for HOG, 77.9% for SIFT, 79. 8% for SURF, 77.2% for texture, and 84.2% for multicharacteristic combinations, surpassing Euclidean distance results via 1.7%–3.6% across capabilities. These findings underscore the effectiveness of Hausdrauff distance in enhancing retrieval precision within the CBIR framework.

A text-based image retrieval system based on a brain function model, called the combination confabulation model, was developed to recognize how contextual word meanings change depending on the situation to overcome semantic gaps. In addition, a recurrent version of this system was developed to improve retrieval accuracy. The system uses a casebase that contains matched pairs of descriptions and images and searches for images that match the input queries. Experimental results showed that the proposed system exhibited more accurate image retrieval performances than conventional systems. © 2013 Wiley Periodicals, Inc.

Problem statement: The Semantic Region Based Image Retrieval (SRBIR) system that automatically segments the dominant foreground region, consisting of the semantic concept of the image, such as elephants, roses and does the semantic learning, is proposed. Approach: The system segments an image into different regions and finds the dominant foreground region in it, which is the semantic concept of that image. Then it extracts the low-level features of that dominant foreground region. The Support Vector Machine-Binary Decision Tree (SVM-BDT) is used for semantic learning and it finds the semantic category of an image. The low level features of the dominant region of each category image are used to find the semantic template of that category. The SVM-BDT is constructed with the help of these semantic templates. The high level concept of the query image is obtained using this SVM-BDT. Similarity matching is done between the query image and the set of images belonging to the semantic category of the query image and the top images with least distances are retrieved. Results: Experiments were conducted using the COREL dataset consisting of 10,000 images and its subset with 1000 images of 10 different semantic categories. The obtained results demonstrate the effectiveness of the proposed framework, compared to those of the commonly used region based image retrieval approaches. Conclusion: Efficient image searching, browsing and retrieval are required by users from various domains, such as medicine, fashion, architecture, training and teaching. The proposed SRBIR system aims at retrieving images based on their semantic content by extracting the dominant foreground region in the image and learning its semantic concept with the help of the SVM-BDT. The proposed SRBIR system provides an efficient image search based on semantics, with high accuracy and less access time.

With the rapid growing volume of images in medical databases, development of efficient image retrieval systems to retrieve relevant or similar images to a query image has become an active research area. Despite many efforts to improve the performance of techniques for accurate image retrieval, its success in biomedicine thus far has been quite limited. This article presents an adaptive content-based image retrieval (CBIR) system for improving the performance of image retrieval in mammographic databases. In this work, the authors propose a new relevance feedback approach based on incremental learning with support vector machine (SVM) regression. Also, the authors present a new local perturbation method to further improve the performance of the proposed relevance feedback system. The approaches enable efficient online learning by adapting the current trained model to changes prompted by the user's relevance feedback, avoiding the burden of retraining the CBIR system. To demonstrate the proposed image retrieval system, the authors used two mammogram data sets: A set of 76 mammograms scored based on geometrical similarity and a larger set of 200 mammograms scored by expert radiologists based on pathological findings. The experimental results show that the proposed relevance feedback strategy improves the retrieval precision for both data sets while achieving high efficiency compared to offline SVM. For the data set of 200 mammograms, the authors obtained an average precision of 0.48 and an area under the precision-recall curve of 0.79. In addition, using the same database, the authors achieved a high pathology matching rate greater than 80% between the query and the top retrieved images after relevance feedback. Using mammographic databases, the results demonstrate that the proposed approach is more accurate than the model without using relevance feedback not only in image retrieval but also in pathology matching while maintaining its effectiveness for online relevance feedback applications.

Content-based Image retrieval systems extract and retrieve images using their low-level features, such as color, texture, and shape. Nevertheless, these visual contents do not allow a user to formulate semantically meaningful image query. Image annotation systems are a solution to solve the inadequacy of CBIR systems and allow text based image retrieval. There have been several studies on automatic image annotation utilizing machine learning techniques and images' representation with low level features extracted using either global or local methods. However, typically, these approaches suffer from low correlation between the globally assigned annotations and the visual features used to obtain annotations automatically. In this paper, we present an approach to enhance the effectiveness of CBIR using learning based automatic images annotation based on bag of visual word images representation that is created automatically using a set of manually annotated training images. The experimentation is performed with 4,000 annotated images for training, 1000 images for testing from ImageNet. The result has shown 77.5% of performance accuracy. The result of this work is believed to be one step towards enhancing the performance and effectiveness of existing CBIR and minimizing the semantic gap.

Sketch-based image retrieval (SBIR) is a cross-modal matching problem which is typically solved by learning a joint embedding space where the semantic content shared between photo and sketch modalities are preserved. However, a fundamental challenge in SBIR has been largely ignored so far, that is, sketches are drawn by humans and considerable style variations exist amongst different users. An effective SBIR model needs to explicitly account for this style diversity, crucially, to generalise to unseen user styles. To this end, a novel style-agnostic SBIR model is proposed. Different from existing models, a cross-modal variational autoencoder (VAE) is employed to explicitly disentangle each sketch into a semantic content part shared with the corresponding photo, and a style part unique to the sketcher. Importantly, to make our model dynamically adaptable to any unseen user styles, we propose to meta-train our cross-modal VAE by adding two style-adaptive components: a set of feature transformation layers to its encoder and a regulariser to the disentangled semantic content latent code. With this meta-learning framework, our model can not only disentangle the cross-modal shared semantic content for SBIR, but can adapt the disentanglement to any unseen user style as well, making the SBIR model truly style-agnostic. Extensive experiments show that our style-agnostic model yields state-of-the-art performance for both category-level and instance-level SBIR.

A new medical image retrieval approach based on low-level features and high-level semantic features is proposed. The image is segmented into several sub-images by fuzzy C-mean clustering algorithm. After extracting the three gray features, in order to emphasize shape and texture features, the sub-image is changed to binary image, and seven shape features and four texture features are extracted. To decrease feature vector dimension, genetic algorithm is used. The optimizing features constitute the visual contents. Only using these low-level features can't get satisfying results, because these features can't entirely describe human recognition to images, the semantic features are needed, but it is difficult to extract the image semantic contents directly at the present time, so the text information in the image report by radiologists is chosen for semantic content. Experimental results show that the retrieval integrating low-level visual features and text information is better than that only by visual contents.