Improving image retrieval effectiveness via query expansion using MeSH hierarchical structure

Editorial Board

In this paper, we present an empirical study for monolingual medical image retrieval. In particular, we present a series of experiments in ImageCLEFmed 2009 task. There are three main goals. First, we evaluate traditional well-known weighting models in the text retrieval domain, such as BM25, TFIDF and Language Model (LM), for context-based image retrieval. Second, we evaluate statistical-based feedback models and ontology-based feedback models. Third, we investigate how content-based image retrieval can be integrated with these two basic technologies in traditional text retrieval domain. The experimental results have shown that: 1) traditional weighting models work well in context-based medical image retrieval task especially when the parameters are tuned properly; 2) statistical-based feedback models can further improve the retrieval performance when a small number of documents are used for feedback; however, the medical image retrieval can not benefit from ontology-based query expansion method used in this paper; 3) the retrieval performance can be slightly boosted via an integrated retrieval approach.

Medical imaging is becoming a vital component of war on cancer. Tremendous amounts of medical image data are captured and recorded in a digital format during cancer care and cancer research. Facing such an unprecedented volume of image data with heterogeneous image modalities, it is necessary to develop effective and efficient content-based medical image retrieval systems for cancer clinical practice and research. While substantial progress has been made in different areas of content-based image retrieval (CBIR) research, direct applications of existing CBIR techniques to the medical images produced unsatisfactory results, because of the unique characteristics of medical images. In this paper, we develop a new multimodal medical image retrieval approach based on the recent advances in the statistical graphic model and deep learning. Specifically, we first investigate a new extended probabilistic Latent Semantic Analysis model to integrate the visual and textual information from medical images to bridge the semantic gap. We then develop a new deep Boltzmann machine-based multimodal learning model to learn the joint density model from multimodal information in order to derive the missing modality. Experimental results with large volume of real-world medical images have shown that our new approach is a promising solution for the next-generation medical imaging indexing and retrieval system.

As artificial intelligence and digital medicine increasingly permeate healthcare systems, robust governance frameworks are essential to ensure ethical, secure, and effective implementation. In this context, medical image retrieval becomes a critical component of clinical data management, playing a vital role in decision-making and safeguarding patient information. Existing methods usually learn hash functions using bottleneck features, which fail to produce representative hash codes from blended embeddings. Although contrastive hashing has shown superior performance, current approaches often treat image retrieval as a classification task, using category labels to create positive/negative pairs. Moreover, many methods fail to address the out-of-distribution (OOD) issue when models encounter external OOD queries or adversarial attacks. In this work, we propose a novel method to consolidate knowledge of hierarchical features and optimization functions. We formulate the knowledge consolidation by introducing Depth-aware Representation Fusion (DaRF) and Structure-aware Contrastive Hashing (SCH). DaRF adaptively integrates shallow and deep representations into blended features, and SCH incorporates image fingerprints to enhance the adaptability of positive/negative pairings. These blended features further facilitate OOD detection and content-based recommendation, contributing to a secure AI-driven healthcare environment. Moreover, we present a content-guided ranking to improve the robustness and reproducibility of retrieval results. Our comprehensive assessments demonstrate that the proposed method could effectively recognize OOD samples and significantly outperform existing approaches in medical image retrieval (p<0.05). In particular, our method achieves a 5.6-38.9% improvement in mean Average Precision on the anatomical radiology dataset.

In this paper we explain experiments in the medical information retrieval task (ImageCLEFmed). We experimented with query expansion and the amount of textual information obtained from the collection. For expansion, we carried out experiments using MeSH ontology and UMLS separately. With respect to textual collection, we produced three different collections, the first one with caption and title, the second one with caption, title and the text of the section where the image appears, and the third one with the full text article. Moreover, we experimented with textual and visual search, along with the combination of these two results. For image retrieval we used the results generated by the FIRE software. The best results were obtained using MeSH query expansion on shortest textual collection (only caption and title) merging with the FIRE results.

One of the main challenges in medical image retrieval is the increasing volume of image data, which render it difficult for domain experts to find relevant information from large data sets. Effective and efficient medical image retrieval systems are required to better manage medical image information. Text‐based image retrieval (TBIR) was very successful in retrieving images with textual descriptions. Several TBIR approaches rely on models based on bag‐of‐words approaches, in which the image retrieval problem turns into one of standard text‐based information retrieval; where the meanings and values of specific medical entities in the text and metadata are ignored in the image representation and retrieval process. However, we believe that TBIR should extract specific medical entities and terms and then exploit these elements to achieve better image retrieval results. Therefore, we propose a novel reranking method based on medical‐image‐dependent features. These features are manually selected by a medical expert from imaging modalities and medical terminology. First, we represent queries and images using only medical‐image‐dependent features such as image modality and image scale. Second, we exploit the defined features in a new reranking method for medical image retrieval. Our motivation is the large influence of image modality in medical image retrieval and its impact on image‐relevance scores. To evaluate our approach, we performed a series of experiments on the medical ImageCLEF data sets from 2009 to 2013. The BM25 model, a language model, and an image‐relevance feedback model are used as baselines to evaluate our approach. The experimental results show that compared to the BM25 model, the proposed model significantly enhances image retrieval performance. We also compared our approach with other state‐of‐the‐art approaches and show that our approach performs comparably to those of the top three runs in the official ImageCLEF competition.

Research and experimentation using Arabic WordNet in the field of information retrieval are relatively new. It is limited compared to the research that has been done using Princeton WordNet. This work attempts to study the impact of Arabic WordNet on the performance of Arabic information retrieval. We extend Lucene with Arabic WordNet to expand user's queries. The major contribution of this study is to propose an interactive query expansion (IQE) methodology using the word's part-of-speech, according to the part it plays in a query. First, the user selects the appropriate part of speech for each term in the original query, and then he reselects the appropriate synonyms. Experimental results show that our IQE strategy produces a good Mean Average Precision (MAP), it is able to improve MAP by 12.6%, but no variant of automatic query expansion (AQE) strategies did. Nevertheless, the experiments allow us to conclude that with an appropriate use of Arabic WordNet as a source of linguistic information for AQE can improve effectiveness for Arabic information retrieval.

This paper describes and evaluates the use of Geographical Knowledge Re-Ranking, Linguistic Processing, and Query Expansion techniques to improve Geographical Information Retrieval effectiveness. Geographical Knowledge Re-Ranking is performed with Geographical Gazetteers and conservative Toponym Disambiguation techniques that boost the ranking of the geographically relevant documents retrieved by standard state-of-the-art Information Retrieval algorithms. Linguistic Processing is performed in two ways: 1) Part-of-Speech tagging and Named Entity Recognition and Classification are applied to analyze the text collections and topics to detect toponyms, 2) Stemming (Porter’s algorithm) and Lemmatization are also applied in combination with default stopwords filtering. The Query Expansion methods tested are the Bose-Einstein (Bo1) and Kullback-Leibler term weighting models. The experiments have been performed with the English Monolingual test collections of the GeoCLEF evaluations (from years 2005, 2006, 2007, and 2008) using the TF-IDF, BM25, and InL2 Information Retrieval algorithms over unprocessed texts as baselines. The experiments have been performed with each GeoCLEF test collection (25 topics per evaluation) separately and with the fusion of all these collections (100 topics). The results of evaluating separately Geographical Knowledge Re-Ranking, Linguistic Processing (lemmatization, stemming, and the combination of both), and Query Expansion with the fusion of all the topics show that all these processes improve the Mean Average Precision (MAP) and RPrecision effectiveness measures in all the experiments and show statistical significance over the baselines in most of them. The best results in MAP and RPrecision are obtained with the InL2 algorithm using the following techniques: Geographical Knowledge Re-Ranking, Lemmatization with Stemming, and Kullback-Leibler Query Expansion. Some configurations with Geographical Knowledge Re-Ranking, Linguistic Processing and Query Expansion have improved the MAP of the best official results at GeoCLEF evaluations of 2005, 2006, and 2007.KeywordsInformation retrievalGeographical gazetteersNatural language processingToponym disambiguationQuery expansionEfectiveness measures

In this paper, we present a knowledge-assisted approach to index and retrieve large volume of medical images. Both images and associated texts are indexed using medical concepts from the Unified Medical Language System (UMLS) meta-thesaurus. We propose a structured learning framework for modular acquisition of medical semantics from images with complementary global and local image indexing schemes. Two fusion approaches are also developed to improve text retrieval using the UMLS-based image indexing: a simple post-query fusion and a visual modality filtering to remove visually aberrant images according to the query modality concepts. On the ImageCLEFmed 2005 database, our framework outperformed our previous result which ranked top in the ImageCLEFmed 2005 Medical Image Retrieval task benchmark.

Relevance feedback (RF) approaches are use to improve the performance of content-based image retrieval (CBIR) systems. In this paper, a RF approach based on modification of similarity measure using particle swarm optimization (PSO) in a medical X-ray image retrieval system is proposed. In this algorithm, using PSO, the significance of each feature in the similarity measure is modified to image retrieval. This modification causes that good features have major effect in relevant image retrieval. The defined fitness function in PSO uses relevant and irrelevant retrieved images with different strategies, simultaneously. The relevant and irrelevant images are used to exhort and penalize similarity measure, respectively. To evaluate, the proposed RF is integrated to a CBIR system based on semantic classification. In this system, using merging scheme in a hierarchical structure, the overlapped classes are merged together and determined search space for each query image. The proposed RF evaluated on a database consisting of 10000 medical X-ray images of 57 classes. The proposed algorithm provides the improvement, effectiveness more than the literature.

Interpreting medical images is certainly a complex task which requires extensive knowledge [1]. According to Computer Aided Diagnosis (CAD) serves as a second opinion that will help radiologists in diagnosis and on the other hand Content-based Image Retrieval uses visual content to help users browse, search and retrieve similar medical images from a database based on the user’s interest [2-4]. The competency of the CBMIR system depends on feature extraction methods [5]. The textural features are very important to determine the content of a medical image. Textural features provide scenic depth, the spatial distribution of tonal variation, and surface orientation [6]. Therefore, this study seeks to compare and evaluate some of the hand-crafted texture feature extraction techniques in CBMIR. This is to help those concerned in enhancing CBIR systems to make informed decisions concerning the selection of the best textural feature extraction techniques. Since there is no clear indication of which of the various texture feature extraction techniques is best suited for a given performance metric when considering which of the techniques to choose for a particular study in CBMIR systems. The objective of this work, therefore, is to comparatively evaluate the performance of the following texture feature extraction techniques; Local Binary Pattern (LBP), Gabor Filter, Gray-Level Co-occurrence Matrix (GLCM), Haralick Descriptor, Features from Accelerated Segment Test (FAST) and Features from Accelerated Segment Test and Binary Robust Independent Elementary Features (FAST &amp;BRIEF) using the metrics; precision, recall, f1-score, mean squared error (MSE), accuracy and time. These techniques are coupled with specific similarity measure to obtain results. The results showed that LBP, Haralick Descriptor, FAST, and GLCM had the best results in terms of (Precision and Accuracy), Time, F1-Score, and Recall respectively. LBP had 82.05% and 88.23% scores for precision and accuracy respectively. The following scores represent the performance of the Haralick descriptor, FAST, and GLCM models respectively; 0.88s, 38.7%, and 44.82%. These test scores are obtained from datasets ranging from 1k-10.5k. Aside from LBP outperforming the other 5 models mentioned, it still outperformed the following proposed models ' [7]’, ‘Tamura texture feature and wavelet transform combined with Hausdorff distance - [8]’, ‘ [9]’ in terms of (precision, accuracy, and recall) and (precision and recall) respectively and probably f1-score (since f1-score is the weighted average of precision and recall). It is believed that an ensemble of LBP, Haralick descriptors, and Support Vector Machine (SVM) can represent a robust system for both medical image retrieval and classification.

Medical image retrieval can assist physicians in finding information supporting their diagnosis and fulfilling information needs. Systems that allow searching for medical images need to provide tools for quick and easy navigation and query refinement as the time available for information search is often short. Relevance feedback is a powerful tool in information retrieval. This study evaluates relevance feedback techniques with regard to the content they use. A novel relevance feedback technique that uses both text and visual information of the results is proposed. The two information modalities from the image examples are fused either at the feature level using the Rocchio algorithm or at the query list fusion step using a common late fusion rule. Results using the ImageCLEF 2012 benchmark database for medical image retrieval show the potential of relevance feedback techniques in medical image retrieval. The mean average precision (mAP) is used as the evaluation metric and the proposed method outperforms commonly-used methods. The baseline without feedback reached 16 % whereas the relevance feedback with 20 images reached up to 26.35 % with three steps and when using 100 images up to 34.87 % in four steps. Most improvements occur in the first two steps of relevance feedback and then results start to become relatively flat. This might also be due to only using positive feedback as negative feeback often also improves results after more steps. The effect of relevance feedback in automatically spelling corrected and translated queries is investigated as well. Results without mistakes were better than spell-corrected results but the spelling correction more than double results over non-corrected retrieval. Multimodal relevance feedback has shown to be able to help visual medical information retrieval. Next steps include integrating semantics into relevance feedback techniques to benefit from the structured knowledge of ontologies and experimenting on the fusion of text and visual information.

The large scale of image databases opens up new challenges for storing, organizing, and searching for data. The common challenge of image retrieval systems is to emphasize object detection and recognition from images to bridge the semantic gap between the Open-Vocabulary and Visual Content. This paper proposes a framework for a personalized image retrieval system which selects the most relevant images according to the specified query, the user interest, and the semantic interpretation. Our framework is composed of three major modules: (1) Users processing using Movielens as a social database. (2) Query processing and personalization. (3) Images processing by semantic annotation and clustering based on a new semantic similarity measure named SRSiM (Statistical Retina Similarity Measure) using statistical and knowledge-based approaches. The experimental study is performed on the NUS-WIDE dataset, a list of queries covering several social subjects and various user profiles extracted from the MovieLens database. The evaluation of the overall system shows promising results going up to 0.675 in terms of MAP (Mean Average Precision) measurement, which overcomes the score of other works established in the same conditions and using the same databases. The SRSiM measurement also reaches an accuracy of 77.77\% compared to Wordnet, Wikipedia, and Retina-based measurements

Searching in M edline: Query expansion and manual indexing evaluation

This paper presents a review of online systems for content-based medical image retrieval (CBIR). The objective of this review is to evaluate the capabilities and gaps in these systems and to determine ways of improving relevance of multi-modal (text and image) information retrieval in the iMedline system, being developed at the National Library of Medicine (NLM). Seven medical information retrieval systems: Figuresearch, BioText, GoldMiner, Yale Image Finder, Yottalook, Image Retrieval for Medical Applications (IRMA), and iMedline have been evaluated here using the system of gaps defined in [1]. Not all of these systems take advantage of the visual information contained in biomedical literature as figures and illustrations. However, all attempt to extract metadata about the image from the full-text of the articles and retrieve figures/images in response to a query. iMedline aims to advance the state-of-the-art in multimodal information retrieval by unifying image and text features in computing relevance. We discuss the shortcomings of these current systems and discuss future directions and next steps in iMedline toward context-based medical image retrieval.