Comprehensive review and analysis on multi modal image retrieval

There is a growing trend for using content‐based image retrieval (CBIR) systems these days because of the constantly growing interest in digital content. Therefore, the ability of the CBIR to perform the CBIR process will depend on the feature extraction process and its basis, for the retrieval will be done on. Numerous researchers put forward various techniques for feature extraction to enhance the nature of the system. Since features play a very key role in enhancing performance, various features can be used collectively to attain the requisite goal. To retain this in mind, we present in this paper a multifeature fusion system, where three features are integrated and form one feature to improve the situation of retrieval. For this purpose, scale‐invariant feature transform (SIFT), speeded‐up robust features (SURF), and histogram of oriented gradients (HOG) features are adopted. These features are common features that deliver information about the shape of the object and for matching purposes, two techniques of distance matching such as Euclidean and Hausdrauff distance are adopted. To assess the performance of the proposed multifeature‐based CBIR approach, experiments were conducted with the usage of a MATLAB simulator. The Corel‐1000 dataset, consisting of 10,000 images in 100 semantic classes, turned into applied, with each magnificence containing 100 images. A subset of 2500 images across 50 semantic classes was used to train the system. This research aligns with industry, innovation, and infrastructure by contributing to advancements in image processing and retrieval systems. Key characteristic descriptors, along with SIFT, SURF, HOG, texture, and multicharacteristic combinations, were extracted for retrieval functions. The results display that the usage of the Hausdrauff distance as a similarity degree outperforms Euclidean distance, accomplishing retrieval accuracies of 80.02% for HOG, 77.9% for SIFT, 79. 8% for SURF, 77.2% for texture, and 84.2% for multicharacteristic combinations, surpassing Euclidean distance results via 1.7%–3.6% across capabilities. These findings underscore the effectiveness of Hausdrauff distance in enhancing retrieval precision within the CBIR framework.

Introduction: Content-based image retrieval (CBIR) retrieves the images from the vast image repositories. Research in the CBIR domain gets attention due to the massive number of images generated by the mobiles and various image-capturing machines. Objectives: Any image retrieval system’s primary goal is to reduce the semantic gap between low-level features and high-level perception. Methods: The CBIR techniques are classified into multiple categories based on the feature extraction and retrieval mechanism. These categories are feature-based, machine-learning-based, and deep-learning-based methods. The pioneer techniques for each category are explained in detail in this chapter. Results: The comparative analysis has been done to highlight the advantages of techniques over others. Conclusion: The various application area of the CBIR has been described. The most evolving application is the content-based medical image retrieval (CBMIR) system. The applicability of the CBMIR system in different medical science domains is explained in detail.

Medical imaging is becoming a vital component of war on cancer. Tremendous amounts of medical image data are captured and recorded in a digital format during cancer care and cancer research. Facing such an unprecedented volume of image data with heterogeneous image modalities, it is necessary to develop effective and efficient content-based medical image retrieval systems for cancer clinical practice and research. While substantial progress has been made in different areas of content-based image retrieval (CBIR) research, direct applications of existing CBIR techniques to the medical images produced unsatisfactory results, because of the unique characteristics of medical images. In this paper, we develop a new multimodal medical image retrieval approach based on the recent advances in the statistical graphic model and deep learning. Specifically, we first investigate a new extended probabilistic Latent Semantic Analysis model to integrate the visual and textual information from medical images to bridge the semantic gap. We then develop a new deep Boltzmann machine-based multimodal learning model to learn the joint density model from multimodal information in order to derive the missing modality. Experimental results with large volume of real-world medical images have shown that our new approach is a promising solution for the next-generation medical imaging indexing and retrieval system.

Textual data such as tags, sentence descriptions are combined with visual cues to reduce the semantic gap for image retrieval applications in today's Multimodal Image Retrieval (MIR) systems. However, all tags are treated as equally important in these systems, which may result in misalignment between visual and textual modalities during MIR training. This will further lead to degenerated retrieval performance at query time. To address this issue, we investigate the problem of tag importance prediction, where the goal is to automatically predict the tag importance and use it in image retrieval. To achieve this, we first propose a method to measure the relative importance of object and scene tags from image sentence descriptions. Using this as the ground truth, we present a tag importance prediction model to jointly exploit visual, semantic and context cues. The Structural Support Vector Machine (SSVM) formulation is adopted to ensure efficient training of the prediction model. Then, the Canonical Correlation Analysis (CCA) is employed to learn the relation between the image visual feature and tag importance to obtain robust retrieval performance. Experimental results on three real-world datasets show a significant performance improvement of the proposed MIR with Tag Importance Prediction (MIR/TIP) system over other MIR systems.

Content-based image retrieval (CBIR) techniques will be part of the next generation of search engines. However, even with a suitable retrieval method, real-time image retrieval from a large image database still requires extremely high computational power. In this paper, a Web-based image database indexing and retrieval system is described. It uses fast CBIR with Beowulf-Class parallel computing engine (Super Abacus) to efficiently and effectively shorten the retrieval time. Content-based image retrieval with several parallel algorithms for images which query different color spaces and the compressed-domain of the images are developed to achieve higher efficiency. Text-based query is also supported for images using a conventional annotation approach.

With the rapid growing volume of images in medical databases, development of efficient image retrieval systems to retrieve relevant or similar images to a query image has become an active research area. Despite many efforts to improve the performance of techniques for accurate image retrieval, its success in biomedicine thus far has been quite limited. This article presents an adaptive content-based image retrieval (CBIR) system for improving the performance of image retrieval in mammographic databases. In this work, the authors propose a new relevance feedback approach based on incremental learning with support vector machine (SVM) regression. Also, the authors present a new local perturbation method to further improve the performance of the proposed relevance feedback system. The approaches enable efficient online learning by adapting the current trained model to changes prompted by the user's relevance feedback, avoiding the burden of retraining the CBIR system. To demonstrate the proposed image retrieval system, the authors used two mammogram data sets: A set of 76 mammograms scored based on geometrical similarity and a larger set of 200 mammograms scored by expert radiologists based on pathological findings. The experimental results show that the proposed relevance feedback strategy improves the retrieval precision for both data sets while achieving high efficiency compared to offline SVM. For the data set of 200 mammograms, the authors obtained an average precision of 0.48 and an area under the precision-recall curve of 0.79. In addition, using the same database, the authors achieved a high pathology matching rate greater than 80% between the query and the top retrieved images after relevance feedback. Using mammographic databases, the results demonstrate that the proposed approach is more accurate than the model without using relevance feedback not only in image retrieval but also in pathology matching while maintaining its effectiveness for online relevance feedback applications.

As the use of digital media increases, effective retrieval and management techniques become more important. In order to extract useful information from this huge amount of data, many content based image retrieval (CBIR) systems have been developed. A typical CBIR system captures image features such as color, texture, or shape of objects in the query image and tries to retrieve images from the database with similar features. The main advantage of CBIR systems with relevance feedback is that these systems take into account the gap between the high-level concepts and low-level features and subjectivity of human perception of visual content. In this paper, a CBIR system based on color histogram has been presented and a new approach for image storage and retrieval called association-based image retrieval (ABIR) has been studied. The results of both systems have been shown.

Radiologists are now confronted with the difficulty of interpreting cross-sectional studies composed of thousands of images. In hospitals, all acquired imaging data are stored in a picture archiving and communication system (PACS). To take advantage of these masses of previously interpreted images, with the ultimate goal to facilitate the diagnosis of new cases, a promising approach would be to integrate a Content-Based Image Retrieval (CBIR) system into PACS. CBIR system performances are inherently limited by the features considered to represent the images. The current state of the art for the extraction of visual features relies on deep learning which requires a sufficient amount of annotated data to learn a generalizable model, such annotated data being rare and difficult to use in medical imaging. At the same time, PACS contain additional information such as radiological reports which supplement visual information carried by the images. We study here how such semantic information, hidden in these reports, can be used to supervise the learning of neuronal models to build a better visual representation of images. In this context our contribution is threefold. We first adapted a contrastive learning approach, which is usually used to learn representation from pairs of positive images in an unsupervised manner, to deal with in-domain medical data. Second, to train such a model to be robust and generalizable with a sufficient amount of data, we propose to re-employ the "dormant" medical imaging literature. Finally, the visual features and the deep models learned in this way, can be considered in CBIR systems as coarse-grained information which can then be fine-tuned in PACS, with more specific images depending on the applications. The obtained experimental result with state of the art contrastive learning methods highlight the interest of this approach.

According to the recent literature, it is pragmatic that image storage and retrieval through the World Wide Web (WWW) has made impressive progress. Practical searching for image still confronts us with present retrieval systems. A Content Based Image Retrieval (CBIR) system provides an efficient way of retrieving related images from image collections. In this paper we present a new technique to extract the images from the database to achieve better performance and efficiency. The novel method uses an approach, which combines texture information and modified Block Truncation Coding (BTC) method to extract the color features from the image. The whole image is divided into a fixed block of size 4 × 4, 8 × 8, 16 × 16, 32 × 32, 64 × 64, 128 × 128 and 256 × 256 non overlapped blocks. For each block, we calculated the threshold as a mean of the pixel values and the number of pixels which are lower than threshold value and upper than the threshold value are counted and mean of these lower and upper values have been taken and these two are considered as features. Now this process is repeated for the all blocks to extract the features of the image. The second step will compute the four texture features using the Gray-Level Co-occurrence Matrix (GLCM) as a statistical approach to extract energy, contrast, entropy and uniformity. We have used this modified BTC and texture features of the image to retrieve images. Finally, we have conducted experiments on a ground truth image database that has 1000 images of different categories. A different block sizes have been used to demonstrate the experimental results and compared with tabulation. From this we have observed that the performance is better and efficiency is low with smaller block sizes and vice versa with higher block sizes. So we have to choose the middle ranges of block sizes for better performance with better efficiency. As we have observed from the results that the performance of the retrieval system is improved because of combination of texture and color features. The proposed system's performance has been improved by an average precision of 2.41% and recall of 8.22% when compared to existing systems.

An image retrieval system was developed purposely to provide an efficient tool for a set of images from a collection of images in the large database that matches the user’s requirements in similarity evaluations such as image content similarity, edge, and colour similarity. Retrieving images based on the contents which are colour, texture, and shape is called content-based image retrieval (CBIR). This paper discusses and describes about the colour features technique for image retrieval systems. Several colour features technique and algorithms produced by the previous researcher are used to calculate the similarity between extracted features. This paper also describes about the specific technique about the colour basis features and combined features (hybrid techniques) between colour and shape features.

The images found within biomedical articles are sources of essential information useful for a variety of tasks. Due to the rapid growth of biomedical knowledge, image retrieval systems are increasingly becoming necessary tools for quickly accessing the most relevant images from the literature for a given information need. Unfortunately, article text can be a poor substitute for image content, limiting the effectiveness of existing text-based retrieval methods. Additionally, the use of visual similarity by content-based retrieval methods as the sole indicator of image relevance is problematic since the importance of an image can depend on its context rather than its appearance. For biomedical image retrieval, multimodal approaches are often desirable. We describe in this work a practical multimodal solution for indexing and retrieving the images contained in biomedical articles. Recognizing the importance of text in determining image relevance, our method combines a predominately text-based image representation with a limited amount of visual information, in the form of quantized content-based visual features, through a process called global feature mapping. The resulting multimodal image surrogates are easily indexed and searched using existing text-based retrieval systems. Our experimental results demonstrate that our multimodal strategy significantly improves upon the retrieval accuracy of existing approaches. In addition, unlike many retrieval methods that utilize content-based visual features, the response time of our approach is negligible, making it suitable for use with large collections.

Image retrieval is a critical task in many fields such as indexing and Content-Based Image Retrieval is a tried and tested method for image retrieval. With advances in graphics and visualization techniques, image retrieval should be easy, quick and flexible. The aim of this work is to develop an efficient image indexing and retrieval system to collate similar images or images based on conditioned queries in the form of an image itself. The indexing method has been performed using wavelet transform and related mechanisms. Its detailing can easily entail description from a broad range to a narrow range. The indexing algorithm applies a Haar Wavelet-Based Perceptual Similarity Index (HaarPSI) for Image Quality Assessment. The HaarPSI utilizes the coefficients obtained from a Haar wavelet decomposition to assess local similarities between two images, as well as the relative importance of image areas. Results show that the algorithm used is able to retrieve and display images with high retrieval accuracy and medium relevance.The proposed methodology has been tested against known image retrieval techniques and was found to be comparable and in some cases better than most prior known techniques. The testing was performed on images of varied types, from multiple datasets and sizes. The accuracy was found to be 99% in most cases. All in all, this study introduces a simple and convenient way of offline image searches on desktop computers and provides a stepping stone to future content-based image retrieval systems built for similar purposes.

Recently, Content-based medical image retrieval (CBMIR) systems enable fast diagnosis via the assessment of the visual information in medical application. Most of the state-of-the-art CBMIR systems facing few issues: computationally expensive due to the usage of high dimensional feature vectors and complex classifier/clustering approaches. The reasons behind this are, inability to properly handle the “semantic gap” and the high intra-class versus inter-class variability problem of the medical image database (like radiographic image database). This yields a crucial demand for developing computationally efficient and highly effective retrieval system. For this purpose, the present study proposed an efficient retrieval system which has a four-fold: First, pre-processing and Feature extraction of input image using canonical correlation analysis (CCA). By this approach extracted the feature in both pixel and feature domains and examined more rigorously. Second, applied Fuzzy C means clustering of pixel intensity values as features based on the singular value decomposition. Through this can grouping, the image based on the pixel intensity value. Third, deep convolutional neural network with SVM classifier which makes implementable and requires only a compact feature vector representation of the stored database image with their class levels during retrieval. Finally evaluated the performance based on the measure of Mean Average Precision, Correct rate (CR), Error rate (ER), Accuracy. The classification results and learned features are used for the purpose of retrieving the medical images in a database. The proposed retrieval system performs better than the traditional approach in terms of measuring average value of precision, recall, f-measure and accuracy 95.9%, 94.96%, 95.37% and 95.798% respectively. The suggested approach is best suited towards retrieving the medical images for various part of the body.

This article proposes a content‐based image retrieval (CBIR) system that employs an informative pattern‐based descriptor. Recent literature has reported the development of efficient local‐pattern‐based descriptors, including the local vector pattern (LVP). This article extends the LVP formulation by making it more computationally efficient and informative. In the extended LVP‐based extraction process, the global index angles are determined using the mutual information between the patterns, which are obtained from a pair of indexed angles. Thus, the Proposed LVP (PLVP) no longer requires a step to identify patterns in every indexed angle found in the querying phase of the CBIR system. A CBIR system with the PLVP is developed in this article, and the system and its associated methods are tested using data from a benchmark texture database and a natural image database. A performance comparison of the PLVP and traditional patterns, such as the local binary pattern (LBP), completely modeled local binary pattern (CLBP) and local tetra pattern (LTrP), is conducted using the CBIR system. The experimental results reveal the superiority of the PLVP in terms of precision, recall, F‐score and computational efficiency.

Bag of Features (BoF) has been successfully exploited in content-based image retrieval for several years. Due to its performance and popularity, several extensions have been proposed that involve feature description, dictionary building, feature encoding and post-query process, etc. This paper proposes a dynamically weighting scheme for BoF-based image retrieval based on feedback. It involves two contributions: (i) analyzing the statistical distribution characteristic of similar BoF representations and (ii) computing weights dynamically based on the feedback obtained from different initial query results. We quantitatively evaluate the proposed method on two different databases. Experiments confirm that the proposed weighting scheme has better performance than the baseline of BoF-based image retrieval systems. Meanwhile, the results demonstrate the effectiveness of the weighting scheme in terms of the precision of top-N returned images.