Measuring and Predicting Tag Importance for Image Retrieval.

Multimodal image retrieval, which involves retrieving images using various modalities such as text, audio, or other images, has significant research importance due to its wide-ranging applications and potential to enhance user experiences across multiple domains. Traditional image retrieval systems, content-based image retrieval (CBIR) systems rely solely on visual features, which can be limiting. By integrating multiple modalities, multimodal retrieval systems can offer more robust and accurate results. A research problem is considered multimodal when it integrates information from more than one type of data source. In multimodal image retrieval (MMIR) systems, one form of data is used to search for outcomes in the same or different modalities. In contrast, cross-modal systems strictly retrieve information from a different modality. A significant challenge in these systems is the effective comparison of input-output queries from different data types, due to their basic forms and the subjective nature of content similarity. Researchers have proposed various techniques to address this challenge and to bridge the semantic gap in information retrieval across different modalities. This article presents comprehensive analysis of various research works pertained in this multimodal image retrieval. The results and comparative analysis of current research works on benchmark datasets have also been discussed. At the end of the paper, some of open issues are presented for future research directions.

This paper presents a highly scalable and adaptable co-learning framework on multimodal image retrieval and image annotation. The co-learning framework is based on the multiple instance learning theory. While this framework is a general framework that may be used in any specific domains, to evaluate this framework, we apply it to the Berkeley Drosophila ISH embryo image database for the evaluations of the retrieval and annotation performance. In addition, we also apply this framework to across-stage inferencing for the embryo images for knowledge discovery. We have compared the performance of the framework for retrieval, annotation, and inferencing on the Berkeley Drosophila ISH database with a state-of-the-art multimodal image retrieval and annotation method to demonstrate the effectiveness and the promise of the framework.

The analysis of large collections of image data is still a challenging\nproblem due to the difficulty of capturing the true concepts in visual data.\nThe similarity between images could be computed using different and possibly\nmultimodal features such as color or edge information or even text labels. This\nmotivates the design of image analysis solutions that are able to effectively\nintegrate the multi-view information provided by different feature sets. We\ntherefore propose a new image retrieval solution that is able to sort images\nthrough a random walk on a multi-layer graph, where each layer corresponds to a\ndifferent type of information about the image data. We study in depth the\ndesign of the image graph and propose in particular an effective method to\nselect the edge weights for the multi-layer graph, such that the image ranking\nscores are optimised. We then provide extensive experiments in different\nreal-world photo collections, which confirm the high performance of our new\nimage retrieval algorithm that generally surpasses state-of-the-art solutions\ndue to a more meaningful image similarity computation.\n

The images found within biomedical articles are sources of essential information useful for a variety of tasks. Due to the rapid growth of biomedical knowledge, image retrieval systems are increasingly becoming necessary tools for quickly accessing the most relevant images from the literature for a given information need. Unfortunately, article text can be a poor substitute for image content, limiting the effectiveness of existing text-based retrieval methods. Additionally, the use of visual similarity by content-based retrieval methods as the sole indicator of image relevance is problematic since the importance of an image can depend on its context rather than its appearance. For biomedical image retrieval, multimodal approaches are often desirable. We describe in this work a practical multimodal solution for indexing and retrieving the images contained in biomedical articles. Recognizing the importance of text in determining image relevance, our method combines a predominately text-based image representation with a limited amount of visual information, in the form of quantized content-based visual features, through a process called global feature mapping. The resulting multimodal image surrogates are easily indexed and searched using existing text-based retrieval systems. Our experimental results demonstrate that our multimodal strategy significantly improves upon the retrieval accuracy of existing approaches. In addition, unlike many retrieval methods that utilize content-based visual features, the response time of our approach is negligible, making it suitable for use with large collections.

Medical imaging is becoming a vital component of war on cancer. Tremendous amounts of medical image data are captured and recorded in a digital format during cancer care and cancer research. Facing such an unprecedented volume of image data with heterogeneous image modalities, it is necessary to develop effective and efficient content-based medical image retrieval systems for cancer clinical practice and research. While substantial progress has been made in different areas of content-based image retrieval (CBIR) research, direct applications of existing CBIR techniques to the medical images produced unsatisfactory results, because of the unique characteristics of medical images. In this paper, we develop a new multimodal medical image retrieval approach based on the recent advances in the statistical graphic model and deep learning. Specifically, we first investigate a new extended probabilistic Latent Semantic Analysis model to integrate the visual and textual information from medical images to bridge the semantic gap. We then develop a new deep Boltzmann machine-based multimodal learning model to learn the joint density model from multimodal information in order to derive the missing modality. Experimental results with large volume of real-world medical images have shown that our new approach is a promising solution for the next-generation medical imaging indexing and retrieval system.

This paper introduces the technique of graph cut into the extraction of object region and applies the corresponding result of object region extraction into the image retrieval based on object region. The main idea of image retrieval based on object region is to use the feature of object region instead of the feature of global image to participate in the image retrieval. In the field of graphics there is a technique called graph cut, which can be used to figure out the contour of object under the interaction of users. The graph cut algorithm can be used to verify the correctness of object region extraction, and the users' input about seeds can be simulated according to the initial object region extracted. The usage of graph cut can make the object region extracted more precisely and thus the performance of image retrieval based on object region can be improved. Experiments show that the object region extraction algorithm based on graph cut is valid and the subsequent image retrieval results accord with the human visual perception much more than the ones without the usage of graph cut.

Different types of search engines are used to search image and text contents. Two types of image search methods are available in the Internet. They are query keyword based model and content based image retrieval models. Text query strings are used in the textual image retrieval model. Content based image retrieval (CBIR) model uses the visual information of the images. Image search methods use the text annotation and image visual features. Google image search and Bing image search engines are used to fetch images from the web. Image query string is used to search image on Internet. One click query image selection method is used to submit user intention for image retrieval. Content based image re-ranking is performed with visual and textual similarity metrics. Adaptive Weight Schema is used for the similarity analysis. Feature weight learning algorithm is applied to estimate feature weights for the images and its category. Query is expanded with keyword and visual information. Rank boost framework algorithm is enhanced to rank images with photographic quality. Content similarity and visual quality factors are used for the re-ranking process. In this paper, we propose an image indexing and retrieval using speech annotations based on a predefined structured syntax. To improve the retrieval effectively, N-best lists for index generation is used .so, a query expansion technique is explored to enhance the query terms. All this process is automatic, without extra effort from the user. This is critically important in web-based image search engine for any commercial, where the user interface has to be extremely simple.

Achieving a balance between security and retrieval accuracy presents a significant challenge in secure content-based image retrieval (SCBIR), particularly in untrusted cloud environments. Conventional SCBIR schemes often struggle with trade-offs, either compromising data privacy, reducing retrieval performance, or introducing significant computational overhead, making them unsuitable for large-scale real-world applications. To overcome these challenges, we propose SCBIR-PE, a novel SCBIR scheme that integrates block-based perceptual encryption with an efficient histogram-based feature extraction method. The encryption incorporates geometric and color transformations at block and sub-block levels across three distinct extensions, confirming security robustness while preserving image features necessary for an image retrieval application. Thus, feature extraction is performed using the Color and Edge Directivity Descriptor (CEDD), enabling direct feature computation in the encrypted domain. Additionally, Euclidean distance is utilized as a similarity measurement technique, ensuring efficient image retrieval while avoiding computationally intensive tasks like clustering, index generation, or feature mapping, which are at the core of existing SCBIR schemes. Experimental evaluations demonstrate that SCBIR-PE scheme outperforms conventional privacy preserving image retrieval schemes in terms of retrieval accuracy (up to 4% better mean average precision score), security robustness, and computational efficiency. By ensuring strong privacy protection with minimal computational overhead, our proposed scheme offers a scalable and efficient solution for cloud-assisted secure image retrieval applications.

Feature extraction and similarity measure are two basic key issues in image retrieval. Combining the advantages of SNN in image recognition and selective attention for image retrieval, a novel visual keywords-driven image retrieval approach based on these properties has been proposed. By using a predefined set of visual keywords as prototype patterns stored with the SNN and then measuring the degree of similiarity of the stored images to the visual keywords, we show that such a visual keyword driven SNN can provide the framework for image indexing or retrieval which is scalable, robust and efficient for web-based search.

This paper presents a face detection technique and its applications in image retrieval. Even though this face detection method has relatively high false positives and a low detection rate (as compared with the dedicated face detection systems in the literature of image understanding), because of its simple and fast nature, it has been shown that this system may be well applied in image retrieval in certain focused application domains. Two application examples are given: one combining face detection with indexed collateral text for image retrieval regarding human beings, and the other combining face detection with conventional similarity matching techniques for image retrieval with similar background. Experimental results show that our proposed approaches have significantly improved image retrieval precision over existing search engines in these focused application domains.

Visual information retrieval (VIR) is an active and vibrant research area, which attempts at providing means for organizing, indexing, annotating, and retrieving visual information (images and videos) form large, unstructured repositories. The goal of VIR is to retrieve the highest number of relevant matches to a given query (often expressed as an example image and/or a series of keywords). In its early years (1995-2000) the research efforts were dominated by content-based approaches contributed primarily by the image and video processing community. During the past decade, it was widely recognized that the challenges imposed by the semantic gap (the lack of coincidence between an image's visual contents and its semantic interpretation) required a clever use of textual metadata (in addition to information extracted from the image's pixel contents) to make image and video retrieval solutions efficient and effective. The need to bridge (or at least narrow) the semantic gap has been one of the driving forces behind current VIR research. Additionally, other related research problems and market opportunities have started to emerge, offering a broad range of exciting problems for computer scientists and engineers to work on. In this tutorial, we present an overview of visual information retrieval (VIR) concepts, techniques, algorithms, and applications. Several topics are supported by examples written in Java, using Lucene (an open-source Java-based indexing and search implementation) and LIRE (Lucene Image REtrieval), an open-source Java-based library for content-based image retrieval (CBIR) written by Mathias Lux.After motivating the topic, we briefly review the fundamentals of information retrieval, present the most relevant and effective visual descriptors currently used in VIR, the most common indexing approaches for visual descriptors, the most prominent machine learning techniques used in connection with contemporary VIR solutions, as well as the challenges associated with building real-world, large scale VIR solutions, including a brief overview of publicly available datasets used in worldwide challenges, contests, and benchmarks. Throughout the tutorial, we integrate examples using LIRE, whose main features and design principles are also discussed. Finally, we conclude the tutorial with suggestions for deepening the knowledge in the topic, including a brief discussion of the most relevant advances, open challenges, and promising opportunities in VIR and related areas.The tutorial is primarily targeted at experienced Information Retrieval researchers and practitioners interested in extending their knowledge of document-based IR to equivalent concepts, techniques, and challenges in VIR. The acquired knowledge should allow participants to derive insightful conclusions and promising avenues for further investigation.

Relevance feedback (RF) schemes have been widely designed to improve the performance of content-based image retrieval. Despite the success, it is not appropriate to require the user to label a large number of samples in RF. Collaborative image retrieval (CIR) aims to reduce the labeling efforts of the user by resorting to the auxiliary information. Support vector machine (SVM) active learning can select ambiguous samples as the most informative ones for the user to label with the help of the optimal hyperplane of SVM, and thus alleviate the labeling efforts of conventional RF. However, the optimal hyperplane of SVM is usually unstable and inaccurate with small-sized training data, and this is always the case in image retrieval since the user would not like to label a large number of feedback samples and cannot label each sample accurately all the time. In this paper, we propose a novel active learning method, i.e., geometric optimum experimental design (GOED), to select multiple representative samples in the database as the most informative ones for the user to label. Especially, GOED can alleviate the small-sized training data problem by leveraging the geometric structure of unlabeled samples in the reproducing kernel Hilbert space and thus further enhance the performance of image retrieval. Different from the conventional manifold regularization framework, the new method can effectively select the most informative samples for the user to label in image retrieval. By minimizing the expected average prediction variance on the test data, GOED has a clear geometric interpretation to select a set of the most representative samples in the database iteratively with the global optimum. Compared with the popular SVM active learning, our method is label-independent and can effectively avoid various potential problems caused by insufficient and inexactly labeled samples in RF, and is more appropriate and useful for image retrieval. Extensive experiments on both synthetic datasets and a real-world image database have been conducted to show the advantages of the proposed GOED for CIR.

Motor activity (exploration) and formation of home bases in mice (C57BL/6) influenced by visual and tactile cues: Modification of movement distribution, distance, location, and speed

Objectives: The objective of the examined active salient component classifier is to extract the image features from the visual substance through a novel non-parametric example based positioning methodology to utilizing comparability and learning strategies. Methods/Statistical Analysis: To establish the authenticity to avoid improper retrieval on online based image classification system called, lite Dynamic Pattern Extraction Algorithm (DPEA) for Image Classification and salient image extraction algorithm is designed. DPEA Algorithm on each benchmark dataset works on user query basis and reducing the overhead incurred during user query processing by applying labeling process. Next, by retrieving salient features on images with zero mean for each client query and each image retrieval reduces the execution time and complexity as the database does not maintain the related features. Finally, the Potential image Classification and retrieval prevents the unauthorized user modification on image data, therefore improving the reality. Here a Web Image Dataset (NUS-WIDE) created by lab of media search in National University of Singapore. The computer vision CIFAR-10 dataset, Modified National Institute of Standards and Technology database of USA (MNIST) dataset is used for experiment. A series of simulation results are performed to test the image retrieval efficiency, execution time, performance fitness for obtaining efficiency of image data handling and measure the effectiveness of DPEA Algorithm. Findings: The tests were directed on the dataset for image classification and retrieval on each region of the images. The Proposed DPEA offers better performance with an improvement of the data confidentiality by 18.72%, reduces execution complexity by 7.98%, reduces average retrieval overhead by 6.65% and also minimize retrieval complexity by 15.57% compared to existing algorithms of LFDA, PCA and PCCA respectively. Application/Improvements: It can be further extended with implementation of new retrieval techniques with different parameters which improves more retrieval efficiency and performance integrity.

The image tagging task aims to assign relevant known tags to an image. It is an active research topic in computer vision and machine learning because of the diversity of its applications in semantic search and image retrieval. Earlier efforts on image tagging address this problem as a multi-level classification problem using visual features from images and semantic word vectors of tags. In most cases, a pre-trained language model like word2vec or Globe is used to obtain those word vectors. Because of using a pre-trained language model, an image tagging approach cannot scale itself to the context of the targeted application. This paper fine-tunes a language (BERT) model using text descriptions obtained from web (Wikipedia) scraping to learn a rich distributed representation of tags. Then, we employ word vectors of tags extracted from finetuned language (BERT) model to solve the image tagging task. Our method is more specialized to the particular application by incorporating context information between targeted tags and images. As a result, word vectors obtained from the fine-tuned model perform better than those from pre-trained language models. We evaluate our method on the widely used NUS-WIDE dataset and achieve competitive results compared with state-of-the-art methods.