Visual information retrieval using Java and LIRE

Information retrieval algorithms have changed the way we manage and use various data sources, such as images, music or multimedia collections. First, free text information of documents from varying sources became accessible in addition to structured data in databases, initially for exact search and then for more probabilistic models. Novel approaches enable content-based visual search of images using computerized image analysis making visual image content searchable without requiring high quality manual annotations. Other multimedia data followed such as video and music retrieval, sometimes based on techniques such as extracting objects and classifying genre. 3D (surface) objects and solid textures have also been produced in quickly increasing quantities, for example in medical tomographic imaging. For these two types of 3D information sources, systems have become available to characterize the objects or textures and search for similar visual content in large databases. With 3D moving sequences (i.e., 4D), in particular medical imaging, even higher-dimensional data have become available for analysis and retrieval and currently present many multimedia retrieval challenges. This article systematically reviews current techniques in various fields of 3D and 4D visual information retrieval and analyses the currently dominating application areas. The employed techniques are analysed and regrouped to highlight similarities and complementarities among them in order to guide the choice of optimal approaches for new 3D and 4D retrieval problems. Opportunities for future applications conclude the article. 3D or higher-dimensional visual information retrieval is expected to grow quickly in the coming years and in this respect this article can serve as a basis for designing new applications.

As the use of digital media increases, effective retrieval and management techniques become more important. In order to extract useful information from this huge amount of data, many content based image retrieval (CBIR) systems have been developed. A typical CBIR system captures image features such as color, texture, or shape of objects in the query image and tries to retrieve images from the database with similar features. The main advantage of CBIR systems with relevance feedback is that these systems take into account the gap between the high-level concepts and low-level features and subjectivity of human perception of visual content. In this paper, a CBIR system based on color histogram has been presented and a new approach for image storage and retrieval called association-based image retrieval (ABIR) has been studied. The results of both systems have been shown.

In content-based image retrieval (CBIR), a set of low-level features are extracted from an image to represent its visual content. Retrieval is performed by image example where a query image is given as input by the user and an appropriate similarity measure is used to find the best matches in the corresponding feature space. This approach suffers from the fact that there is a large discrepancy between the low-level visual features that one can extract from an image and the semantic interpretation of the image’s content that a particular user may have in a given situation. That is, users seek semantic similarity, but we can only provide similarity based on low-level visual features extracted from the raw pixel data, a situation known as the semantic gap. The selection of an appropriate similarity measure is thus an important problem. Since visual content can be represented by different attributes, the combination and importance of each set of features varies according to the user’s semantic intent. Thus, the retrieval strategy should be adaptive so that it can accommodate the preferences of different users. Relevance feedback (RF) learning has been proposed as a technique aimed at reducing the semantic gap. It works by gathering semantic information from user interaction. Based on the user’s feedback on the retrieval results, the retrieval scheme is adjusted. By providing an image similarity measure under human perception, RF learning can be seen as a form of supervised learning that finds relations between high-level semantic interpretations and low-level visual properties. That is, the feedback obtained within a single query session is used to personalize the retrieval strategy and thus enhance retrieval performance. In this chapter we present an overview of CBIR and related work on RF learning. We also present our own previous work on a RF learning-based probabilistic region relevance learning algorithm for automatically estimating the importance of each region in an image based on the user’s semantic intent.

Content Based Image Retrieval (CBIR) depends on the visual content of an image for obtaining similar images. Research on CBIR faces a challenge of semantic gap between low level features and high level visual content. The optimal solution must reduce the computational complexity and computational time in retrieval process. Relevance Feedback is a solution to reduce the semantic gap. A new framework for content based image retrieval is proposed which is Bipartite Reiterative algorithm. This gives an approach of iteration to relevance feedback mechanism. It consists of two levels which are basic retrieval scheme and reiterative feedback scheme. Positive images are obtained from the user as part of the basic retrieval scheme. New feedback algorithm is developed based on weights assigned to visual attributes. The weights are inversely proportional to the visual attribute distance between query image and the positive images given by the user. Iterative feedback query can be performed after the user feedback. The query will be progressed according to the user feedback. The algorithm reduces the computational complexity and computational time in image retrieval.

Different types of search engines are used to search image and text contents. Two types of image search methods are available in the Internet. They are query keyword based model and content based image retrieval models. Text query strings are used in the textual image retrieval model. Content based image retrieval (CBIR) model uses the visual information of the images. Image search methods use the text annotation and image visual features. Google image search and Bing image search engines are used to fetch images from the web. Image query string is used to search image on Internet. One click query image selection method is used to submit user intention for image retrieval. Content based image re-ranking is performed with visual and textual similarity metrics. Adaptive Weight Schema is used for the similarity analysis. Feature weight learning algorithm is applied to estimate feature weights for the images and its category. Query is expanded with keyword and visual information. Rank boost framework algorithm is enhanced to rank images with photographic quality. Content similarity and visual quality factors are used for the re-ranking process. In this paper, we propose an image indexing and retrieval using speech annotations based on a predefined structured syntax. To improve the retrieval effectively, N-best lists for index generation is used .so, a query expansion technique is explored to enhance the query terms. All this process is automatic, without extra effort from the user. This is critically important in web-based image search engine for any commercial, where the user interface has to be extremely simple.

Recent rapid growth in the demand for technology and image investigation in many applications, such as image retrieval systems and Visual Object Categorization (VOC), effective management of these applications has become crucial. Computer vision and its various applications are a primary focus of research. Content-based image retrieval is considered an extremely challenging issue and has remained an open research area. Obviously, the main challenge associated with this kind of research is the gap between the low-level features and the richness of the semantic concept of the human mind. This problem is called the semantic gap. Several methods have been proposed to increase the performance of the system and reduce the semantic gap. These proposed techniques make use of either global or local features or a combination of both global and local features on one side and the visual content and keyword-based retrieval on the other side. However, the aim of this study is to provide a constructive critique of the algorithms used in extracting the low-level features, either globally or locally or as a combination of both. In addition, it identifies the factors that can affect the low-level features that lead to the semantic gap. As well as, proposed a new framework to improve the Gabor filter and the edge histogram limitations. Finally, recommendations are made for the choice of the descriptors used to describe the low-level features, both locally and globally, depending on the area of limitations or drawbacks of the previous state-of-the-art research.

Given the growing amount of very large image databases, content based image retrieval (CBIR) is becoming more and more important. One of the major challenges in CBIR is the semantic gap - commonly used feature-based algorithms are not able to identify what really draws human attention in an image. This problem is more crucial for localized CBIR, where certain regions / parts of the image are what the user is really interested in. This paper explores how human gaze can be utilized to extract regions of interest (ROIs) of an image to perform attention based image retrieval. Using eye tracking data and knowledge about foveal and peripheral vision of humans, we present a foveal fixation clustering algorithm that automatically generates ROIs in an image while a person is viewing it. To objectively set different parameters of the algorithm, a small user study was conducted. The method was evaluated for use in a localized CBIR system. Image retrieval results using the publicly available SIVAL dataset were scored using mean average precision (MAP). Comparison to a saliency-based visual attention algorithm as well as to manually labeled regions showed that the retrieval results of the developed algorithm are nearly two times better than the saliency-based visual attention algorithm and very close to the results using hand-labeled regions.

Developments in multimedia technologies have paved way for the storage of huge collections of video documents on computer systems. It is essential to design tools for content-based access to the documents, so as to allow an efficient exploitation of these collections. Content based analysis provides a flexible and powerfulway to access video data when compared with the other traditional video analysis techniques. The area of content based video indexing and retrieval (CBVIR), focusing on automating the indexing, retrieval and management of video, has attracted extensive research in the last decade. CBVIR is a lively area of research with enduring acknowledgments from several domains. Herein a vital assessment of contemporary researches associated with the content-based indexing and retrieval of visual information. In this paper, we present an extensive review of significant researches on CBVIR. Concise description of content based video analysis along with the techniques associated with the content based video indexing and retrieval is presented.

This paper presents novel approaches for combining re-ranking and rank aggregation methods aiming at improving the effectiveness of Content-Based Image Retrieval (CBIR) systems. Given a query image as input, CBIR systems retrieve the most similar images in a collection by taking into account image visual properties. In this scenario, accurately ranking collection images is of great relevance. Aiming at improving the effectiveness of CBIR systems, re-ranking and rank aggregation algorithms have been proposed. However, different re-ranking and rank aggregation approaches, applied to different image descriptors, may produce different and complementary image rankings. In this paper, we present four novel approaches for combining these rankings aiming at obtaining more effective results. Several experiments were conducted involving shape, color, and texture descriptors. The proposed approaches are also evaluated on multimodal retrieval tasks, considering visual and textual descriptors. Experimental results demonstrate that our approaches can improve significantly the effectiveness of image retrieval systems.

3D CBIR with sparse coding for image-guided neurosurgery

Recently, advances in storage techniques have conducted to an increase in a lot amount of digital images all around the world. These developments have heightened the need for effective image retrieval techniques. Roughly speaking, image retrieval techniques can be divided into two areas: the traditional keyword-based approach and content-based image retrieval. But they still fail on low precision and lack semantic that will cause semantic gap between image features and the user. Previous studies have demonstrated that non-professional users prefer using event-based conceptual descriptions, such as “a woman wearing a hat”, to describe and search images. In many art image archives, these conceptual descriptions are manually annotated in free-text fields. We propose a novel approach for extracting event-based semantic knowledge automatically from free-text image descriptions. The semantic role labeling techniques appear to be a promising technology for transforming the unrestricted natural language texts into structured records that is easy to index/retrieve using relational database technologies. The precision of such SRL-assisted event-based image retrieval is much higher than that of the conventional keyword-based approach. This thesis proposes an ontology-based approach for integrating metadata-based image annotations (administrative knowledge) with event-based knowledge (administrative and conceptual knowledge), including subject, verb, object, location and temporal information from free-text image descriptions. The goal is to automatically derive certain interesting knowledge, such as, all the paintings with a same painting style and a same conceptual event in the image content that implicitly dispersed around different image archives. In practice, an answer to these questions requires a series of field-based queries, across different digital archives. This thesis investigated the possibility to utilize standardized ontology, CIDOC-CRM, and semantic web tools, such as Protege, SWRL and Jess inference engine in order to model an inference platform. We discuss the need for a robust inference platform for real-life knowledge discovery and integration among heterogeneous image archives. Experimental results indicate the approach can be implemented in real-life applications.

Content-Based Image Retrieval (CBIR) systems play a crucial role in managing the exponential growth of digital image collections. Their task is to retrieve relevant images by analysing the content of a query image rather than relying on metadata. The idea here is to extract relevant features from the query image and find a similar set of features in other images. The current generation of CBIR systems faces difficulties in bridging the “semantic gap” between image features at low levels and human perception at high levels which reduces retrieval precision. The research targets CBIR performance enhancement by developing state-of-the-art deep learning techniques for feature extraction. We implemented two approaches which included a custom Convolutional Neural Network (CNN) with six convolutional layers and a finetuned ResNet50 model trained on the CIFAR-10 dataset, and the finetuned model demonstrated better performance than numerous existing CBIR systems. The networks extracted 512-dimensional feature vectors from their penultimate layer for subsequent similarity-matching operations. The fine-tuned ResNet50 model delivered better retrieval results than our custom CNN model since it reached a Mean Average Precision (MAP) score of 98.7%, while the custom CNN achieved 87.3%. We built a web-based interface that allows users to perform real-time image retrieval through multiple evaluation metrics.

With the rapid advancement of digital imaging technologies, and the use of large volume image databases in various applications, it becomes imperative to build an automatic and an efficient image retrieval system. Content Based Image Retrieval (CBIR) is most emerging and vivid research area in computer vision, in which unknown query image assigns to the closest possible similar images available in the database. Current systems mainly use colour, texture, and shape information for image retrieval using similarity measures between query and database images features. Here this work, proposed a classification system that allows recognizing and recovering the class of a query image based on its visual content. This successful categorization of images greatly enhances the performance of retrieval by filtering out irrelevant classes. In this way we have done the comparative analysis of various features as an individual or in combinations, with direct similarity measure and proposed framework. Experimentson benchmark Wang database show that the proposed classification & retrieval framework performs significantly better than the common framework of distances.

Research in content-based image retrieval is an expanding discipline with an accelerated growth in the last ten years. Advances in telecommunications and the huge demand of visual information on Internet and mobile devices is occupying the attention of the researchers in developing efficient systems to ease the task of useful visual information retrieval by the users. We present a semiautomatic image annotation process using the low-level image descriptor fuzzy color signature to extract the most similar images from an annotated database and frequent pattern mining to select the candidates keywords for annotating the new image. The idea is aimed at establishing a bridge between visual data and their interpretation using a weak semantic approach.

Image annotation and retrieval has been a popular research topic for decades. Based on published journals from 2012 until 2015, a lot of research and studies has been focused on Content Based Image Retrieval (CBIR). In most cases, CBIR systems that use an image as the input query always face a problem called semantic gap due to the use of low-level features for similarity matching. The semantic gap will consequently reduce the performance of the CBIR systems. To overcome this problem, a new class of CBIR known as Annotation Based Image Retrieval (ABIR) has been developed. The ABIR systems, based on the annotation process of the images, employ bag of words for query. However it is found that employing pure bag of word only is not adequate to solve semantic problem. In this study, an attempt to use a Gaussian Mixture Model (GMM) based approach and spatial related information of the annotated objects has been performed in order to improve the performance of the ABIR systems. From the retrieval experiments, it is found that ABIR could achieve average performance up to 88% which is two times better than the CBIR systems in reducing the semantic problem.