Learning image semantics from users relevance feedback

Recent works of music representation learning mainly focus on learning acoustic music representations with unlabeled audios or further attempt to acquire multi-modal music representations with scarce annotated audio-text pairs. They either ignore the language semantics or rely on labeled audio datasets that are difficult and expensive to create. Moreover, merely modeling semantic space usually fails to achieve satisfactory performance on music recommendation tasks since the user preference space is ignored. In this paper, we propose a novel Hierarchical Two-stage Contrastive Learning (HTCL) method that models similarity from the semantic perspective to the user perspective hierarchically to learn a comprehensive music representation bridging the gap between semantic and user preference spaces. We devise a scalable audio encoder and leverage a pre-trained BERT model as the text encoder to learn audio-text semantics via large-scale contrastive pre-training. Further, we explore a simple yet effective way to exploit interaction data from our online music platform to adapt the semantic space to user preference space via contrastive fine-tuning, which differs from previous works that follow the idea of collaborative filtering. As a result, we obtain a powerful audio encoder that not only distills language semantics from the text encoder but also models similarity in user preference space with the integrity of semantic space preserved. Experimental results on both music semantic and recommendation tasks confirm the effectiveness of our method.

Correlating image semantics with its low level features is a challenging task. Although, humans are adept in distinguishing object categories, both in visual as well as in semantic space, but to accomplish this computationally is yet to be fully explored. The learning based techniques do minimize the semantic gap, but unlimited possible categorization of objects in real world is a major challenge to these techniques. This work analyzes and utilizes the strength of a semantically categorized image database to assign semantics to query images. Semantics based categorization of images would result in image hierarchy. The algorithms proposed in this work exploit visual image descriptors and similarity measures in the context of a semantically categorized image database. A novel 'Branch Selection Algorithm' is developed for a highly categorized and dense image database, which drastically reduces the search space. The search space so obtained is further reduced by applying any one of the four proposed 'Pruning Algorithms'. Pruning algorithms maintain accuracy while reducing the search space. These algorithms use an adaptive combination of multiple visual features of an image database to find semantics of query images. Branch Selection Algorithm tested on a subset of 'ImageNet' database reduces search space by 75%. The best pruning algorithm further reduces this search space by 26% while maintaining 95% accuracy.

One of the major challenges in the content-based information retrieval and machine learning techniques is to-build-the-so-called "semantic classifier" which is able to effectively and efficiently classify semantic concepts in a large database. This paper dealt with semantic image classification based on hierarchical Fuzzy Association Rules (FARs) mining in the image database. Intuitively, an association rule is a unique and significant combination of image features and a semantic concept, which determines the degree of correlation between features and concept. The main idea behind this approach is that any image visual concept has some associated features, so that, there are strong correlations between the concepts and their corresponding features. Regardless of the semantic gap, an image concept appears when the corresponding features emerge in an image and vice versa. Specially, this paper's contribution was to propose a novel Fuzzy Association Rule for improving traditional association rules. Moreover, it was concerned with establishing a hierarchical fuzzy rule base in the training phase and setup corresponding fuzzy inference engine in order to classify images in the testing phase. The presented approach was independent from image segmentation and can be applied on multi-label images. Experimental results on a database of 6000 general-purpose images demonstrated the superiority of the proposed algorithm.

Query by image content is a method to retrieve the most important images from the image database. It is an answer for the problem of searching for digital images in large database. A large number of relevance feedback schemes have been developed to improve the performance of content based image retrieval. In this paper we propose biased discriminant Euclidean embedding that form intraclass geometry and interclass discrimination. In this method images can be grouped by their similarity. In order to achieve this, firstly the visual features of the images are found out. In addition to this we use two clustering algorithms to assemble the images into clusters using their low level visual features. Here we have to filter the images in the hierarchical clustering and then apply the clustered images to K-Means, so that we can get the most relevant images. Query image is compared directly with the images in the clusters. Therefore number of comparisons is reduced because query image is compared only with the images in the clusters instead of comparing with all images in the database.

Clustering is an important unsupervised machine learning techniqueused in diverse fields to explore the inherent structure of the data. In most of the real life datasets, one object resides in many clusters with different membership values. Many clustering algorithms have been proposed for finding such overlapping clusters for knowledge extraction and future trend prediction. In the paper, multi-objective genetic algorithm based cluster analysis technique is proposed for finding the optimal set of overlapping clusters. As most of the real world search and optimization problems involve multiple objectives, multi-objective Genetic Algorithm is an obvious choice for capturing multiple optimal solutions. Thus the usefulness of applying the multi-objective Genetic Algorithm is to grouping the objects based on different objective functions for finding optimal set of overlapping clusters. The advantage of this algorithm is that it assigns a membership value only to the objects which are the members of several clusters, instead of assigning membership values for all clusters like fuzzy clustering algorithm. If any object positively belongs only to a single cluster, its membership value for this cluster is ' 1' and '0' for all other clusters. The overall performance of the method is investigated on some popular UCI and microarray datasets and the optimality of the clusters is measured by some important cluster validation indices. The experimental results show the effectiveness of the proposed method.

This book explores the cross-linguistic variation in the structural encoding of the Domain of Intransitive Predication. This domain is defined as the set of non-embedded propositions which consist essentially of a one-place predicate and its argument; the typical linguistic reflection of such propositions is formed by intransitive main clauses. I assume that the domain can be seen as some ‘cognitive/ semantic space’, the topography of which is the same for all languages. Furthermore, I assume that the domain consists of four (non-discrete, and prototypically defined) subdomains, which correspond to a four-way distinction between the semantic classes of event predicates, property predicates, class predicates, and locational predicates; these four semantic predicate classes have their typical linguistic reflection in the lexical classes of verbs, predicate adjectives, predicate nominals, and predicatively used adverbials. In short, the data base of this study consists of the equivalents of English sentences such as given in (1) for each language of the sample:

The proliferation of computer technology and digital image‐acquisition hardware has led to the widespread use of image data across a variety of applications including astronomy, art, natural resources, engineering design, military, business operation, medicine, education, etc. Major research activities in the digital image databases surged after the U.S. government's Digital Library Initiative (DLI) from 1994 to 1998. The DLI research team at the University of California at Berkeley developed a work‐centered digital information system that contains 450 digital text documents, 200 air photos, and over 11,000 ground photographs. The system allowed a user or a working group to access its own collections of varying data types, and to generate new materials to be added to the collection. The DLI project at the University of California (UC) at Santa Barbara focused on the development of digital databases that contain geographically referenced materials such as maps and aerial photos. The research team developed a number of tools for browsing and retrieving map images at multiple resolution. Two other major efforts in developing digital image databases were at The National Library of Medicine (NLM), a component of the National Institutes of Health (NIH), and at Time Warner. More recently, research scientists at the University of California at Berkeley and the Fine Arts Museums of San Francisco successfully launched on the Internet the largest art image database in the world, the Thinker ImageBase.Automatic image indexing and retrieving technology is fundamental in digital image databases. There are two general strategies in image retrieval, browsing and searching. Browsing is an information retrieval strategy in which a user navigates through an ordered arrangement of images by making selections from the progressive levels of a hierarchy into which the available images have been logically grouped. Image browsing relies on the user's cognitive abilities to recognize images of interest without having to formulate a specific query. Searching is an information retrieval strategy in which the user communicates with an information retrieval system by an interface that requires the user to input a search query.Images can be indexed and retrieved by text‐tag information and/or by image content. Conventionally, images are indexed using text‐tag information, such as title, key words, date of the work, artist, author, photographer, legend, captions, etc. Images are often subject to a wide range of interpretations, and textual descriptions can only begin to capture the richness and complexity of the semantic content of a visual image. In many applications, the complexity of the information embedded in image content such as types of objects, object attributes, and spatial relationship of objects, etc, cannot be synthesized in a few key words. It has been reported that users querying an image collection tend to be much more specific in their requests and information needs than when querying a text database. Furthermore, text indexing requires large human effort in creating the meta‐data that enables visual queries and is language dependent. Large databases containing thousands and millions of digital images that can occupy gigabytes of space are almost impossible for manual indexing and searching. The demand for systems using pictorial information combined with textual description in image retrieval is growing and automatic indexing and retrieval based on image content has become the most promising techniques for large image databases and digital image libraries. This article describes the techniques developed for content‐based retrieval.

Several criminal profiling systems have been developed to assist the Law Enforcement Agencies in solving crimes but the techniques employed in most of the systems lack the ability to cluster criminal based on their behavioral characteristics. This paper reviewed different clustering techniques used in criminal profiling and then selects one fuzzy clustering algorithm (Expectation Maximization) and two hard clustering algorithm (K-means and Hierarchical). The selected algorithms were then developed and tested on real life data to produce "profiles" of criminal activity and behavior of criminals. The algorithms were implemented using WEKA software package. The performance of the algorithms was evaluated using cluster accuracy and time complexity. The results show that Expectation Maximization algorithm gave a 90.5% clusters accuracy in 8.5s, while K-Means had 62.6% in 0.09s and Hierarchical with 51.9% in 0.11s. In conclusion, soft clustering algorithm performs better than hard clustering algorithm in analyzing criminal data.
 Keywords: Clustering Algorithm, Profiling, Crime, Membership value

Automatic medical image annotation and retrieval

Class label autoencoder with structure refinement for zero-shot learning

Endoscopic image analysis in semantic space

Content-based image retrieval (CBIR) systems utilize low level query image feature as identifying similarity between a query image and the image database.Image contents are plays significant role for image retrieval.There are three fundamental bases for content-based image retrieval, i.e. visual feature extraction, multidimensional indexing, and retrieval system design.Each image has three contents such as: color, texture and shape features.Color and texture both plays important image visual features used in Content-Based Image Retrieval to improve results.Color histogram and texture features have potential to retrieve similar images on the basis of their properties.As the feature extracted from a query is low level, it is extremely difficult for user to provide an appropriate example in based query.To overcome these problems and reach higher accuracy in CBIR system, providing user with relevance feedback is famous for provide promising solution.

We describe a hierarchical linear subspace method to query large on-line image databases using image similarity as the basis of the queries. The method is based on the generic multimedia indexing (GEMINI) approach which is used in the IBM query through the image content search system. Our approach is demonstrated on image indexing, in which the subspaces correspond to different resolutions of the images. During content-based image retrieval, the search starts in the subspace with the lowest resolution of the images. In this subspace, the set of all possible similar images is determined. In the next subspace, additional metric information corresponding to a higher resolution is used to reduce this set. This procedure is repeated until the similar images can be determined. For evaluation we used three image databases and two different subspace sequences.

With the rapid growing volume of images in medical databases, development of efficient image retrieval systems to retrieve relevant or similar images to a query image has become an active research area. Despite many efforts to improve the performance of techniques for accurate image retrieval, its success in biomedicine thus far has been quite limited. This article presents an adaptive content-based image retrieval (CBIR) system for improving the performance of image retrieval in mammographic databases. In this work, the authors propose a new relevance feedback approach based on incremental learning with support vector machine (SVM) regression. Also, the authors present a new local perturbation method to further improve the performance of the proposed relevance feedback system. The approaches enable efficient online learning by adapting the current trained model to changes prompted by the user's relevance feedback, avoiding the burden of retraining the CBIR system. To demonstrate the proposed image retrieval system, the authors used two mammogram data sets: A set of 76 mammograms scored based on geometrical similarity and a larger set of 200 mammograms scored by expert radiologists based on pathological findings. The experimental results show that the proposed relevance feedback strategy improves the retrieval precision for both data sets while achieving high efficiency compared to offline SVM. For the data set of 200 mammograms, the authors obtained an average precision of 0.48 and an area under the precision-recall curve of 0.79. In addition, using the same database, the authors achieved a high pathology matching rate greater than 80% between the query and the top retrieved images after relevance feedback. Using mammographic databases, the results demonstrate that the proposed approach is more accurate than the model without using relevance feedback not only in image retrieval but also in pathology matching while maintaining its effectiveness for online relevance feedback applications.

Currently, most image retrieval systems rely on text annotation of pictures as the basis to index and retrieve image data. The main limitations of using free-text to describe images are the problems of completeness and consistency. Recent works have addressed the completeness problem by using the document space modification technique to update the image descriptions using the users' relevant feedback. However, the updating process can be inconsistent and that the terms used to update the image descriptions can be ambiguous. The authors describe the use of a concept model of the image collection as the basis to guide the retrieval and updating of image contents. Relevance feedbacks from the users during retrieval activities are used to update the concept representations and image descriptions. The concept structure helps to ensure that only meaningful terms in the domain are used to update the image contents, and to overcome the ambiguity of the meanings of terms in a free-text environment. The system has been tested on an image database in the domain of Singapore history and has been found to be feasible. >