HANOLISTIC: A Hierarchical automatic Image Annotation System Using Holistic Approach

Automatic image annotation is the process of assigning keywords to digital images depending on the content information. In one sense, it is a mapping from the visual content information to the semantic context information. In this study, we propose a novel approach for automatic image annotation problem, where the annotation is formulated as a multivariate mapping from a set of independent descriptor spaces, representing a whole image, to a set of words, representing class labels. For this purpose, a hierarchical annotation architecture, named as HANOLISTIC (Hierarchical Image Annotation System Using Holistic Approach), is defined with two layers. The first layer, called level-0 consists of annotators each of which is fed by a set of distinct descriptors, extracted from the whole image. This enables us to represent the image at each annotator by a different visual property of a descriptor. Since, we use the whole image, the problematic segmentation process is avoided. Training of each annotator is accomplished by a supervised learning paradigm, where each word is considered as a class label. Note that, this approach is slightly different then the classical training approaches, where each data has a unique label. In the proposed system, since each image has one or more annotating words, we assume that an image belongs to more than one class. The output of the level-0 annotators indicate the membership values of the words in the vocabulary, to belong an image. These membership values from each annotator is, then, aggregated at the second layer to obtain meta-level annotator. Finally, a set of words from the vocabulary is selected based on the ranking of the output of meta-level. The hierarchical annotation system proposed in this study outperforms state of the art annotation systems based on segmental and holistic approaches.

A Survey on Automatic Image Annotation and Trends of the New Age

This article presents an automatic image level annotation approach that takes advantage of both context and semantics presented in segmented images. The proposed approach is based on the optimization of classes’ scores using particle swarm optimization. In addition, random forest classifier and normalized cuts algorithm have been applied for automatic image classification, annotation, and clustering. For the proposed approach, each input image is segmented using the normalized cuts segmentation algorithm in order to create a descriptor for each segment. Two parameter selection models have been selected for particle swarm optimization algorithm and many voting techniques have been implemented to find the most suitable set of annotation words per image. Experimental results, using Corel5k benchmark annotated images dataset, demonstrate that applying optimization algorithms along with random forest classifier achieved noticeable increase in image annotation performance measures compared to related researches on the same dataset.

A survey and analysis on automatic image annotation

Automatic image annotation is a critical and challenging problem in pattern recognition and image understanding areas. There are some problems in existing automatic image annotation areas. For example, the size of unlabeled data is much larger than the labeled data. Besides, most image annotation models can only use one kind of image segmentation strategy and certain image description method. According to the above problems, an automatic image annotation model based on Co-training is proposed. In this model, four independent feature properties are constructed and then four corresponding sub-classifiers are built. In this way, different image segmentation strategies and feature representation methods can be integrated into a unified framework. An adaptive algorithm based on vote and consistency is proposed to extend the training dataset. The proposed method use Co-training algorithm and mass unlabeled data to improve the performance of automatic image annotation. Experiments conducted on Corel 5 K dataset verify the effectiveness of proposed method.

Automatic image annotation has been extensively studied, mostly from a content-based approach, whose effectiveness is restricted by the “semantic gap” between low-level image features and semantic annotations, and by the irrelevance of annotations to image content. We propose a social diffusion analysis approach to image annotation, which exploits abundant social diffusion records about how images are disseminated within online social networks. Specifically, we propose a common-interest model to analyze social diffusion records, with the assumption that the diffusion pattern of an image in social networks is highly related to the relevance between image annotations and user preferences. In our proposed model, user preferences are represented as common interests of pairwise users instead of individual user interests. We find the notion of common interests not only facilitates the analysis of social diffusion patterns, but also leads to more accurate profiling of user preferences compared to individual interests. Based on the common-interest model, we design an image annotation framework via social diffusion analysis, which consists of the mining of common interests from social diffusion records, the feature extraction from diffusion graphs and common interests, and the automatic annotation by the learning-to-rank method. Experimental results on real-world data sets show that our proposed common-interest based approach outperforms individual-interest based methods, and also achieves superior performance than state-of-the-art content-based image annotation methods.

The explosive growth of digital images requires effective methods to manage these images. Among various existing methods, automatic image annotation has proved to be an important technique for image management tasks, e.g., image retrieval over large-scale image databases. Automatic image annotation has been widely studied during recent years and a considerable number of approaches have been proposed. However, the performance of these methods is yet to be satisfactory, thus demanding more effort on research of image annotation. In this paper, we propose a novel semi supervised framework built upon feature selection for automatic image annotation. Our method aims to jointly select the most relevant features from all the data points by using a sparsity-based model and exploiting both labeled and unlabeled data to learn the manifold structure. Our framework is able to simultaneously learn a robust classifier for image annotation by selecting the discriminating features related to the semantic concepts. To solve the objective function of our framework, we propose an efficient iterative algorithm. Extensive experiments are performed on different real-world image datasets with the results demonstrating the promising performance of our framework for automatic image annotation.

Nowadays, the number of digital images has increased so that the management of this volume of data needs an efficient system for browsing, categorising and searching. Automatic image annotation is designed for assigning tags to images for more accurate retrieval. Non‐negative matrix factorisation (NMF) is a traditional machine learning technique for decomposing a matrix into a set of basis and coefficients under the non‐negative constraints. In this study, the authors propose a two‐step algorithm for designing an automatic image annotation system that employs the NMF framework for its first step and a variant of K‐nearest neighbourhood as its second step. In the first step, a new multimodal NMF algorithm is proposed to extract the latent factors which reflect the content of images. This is done by jointly factorising the visual and textual data feature matrices so that they have close representation, although not necessarily the same. In the second step, after mapping images to the latent factors space a few tags are predicted for the new images based on a weighted average of similar data. They evaluated the performance of the proposed method and compared it to the state‐of‐the‐art literature. Comparison results demonstrate the effectiveness and potential of the proposed method in image annotation applications.

The development of technology generates huge amounts of non-textual information, such as images. An efficient image annotation and retrieval system is highly desired. Clustering algorithms make it possible to represent visual features of images with finite symbols. Based on this, many statistical models, which analyze correspondence between visual features and words and discover hidden semantics, have been published. These models improve the annotation and retrieval of large image databases. However, image data usually have a large number of dimensions. Traditional clustering algorithms assign equal weights to these dimensions, and become confounded in the process of dealing with these dimensions. In this paper, we propose weighted feature selection algorithm as a solution to this problem. For a given cluster, we determine relevant features based on histogram analysis and assign greater weight to relevant features as compared to less relevant features. We have implemented various different models to link visual tokens with keywords based on the clustering results of K-means algorithm with weighted feature selection and without feature selection, and evaluated performance using precision, recall and correspondence accuracy using benchmark dataset. The results show that weighted feature selection is better than traditional ones for automatic image annotation and retrieval.

Since most computer vision approaches are now driven by machine learning, the current bottleneck is the annotation of images. This time-consuming task is usually performed manually after the acquisition of images. In this article, we assess the value of various egocentric vision approaches in regard to performing joint acquisition and automatic image annotation rather than the conventional two-step process of acquisition followed by manual annotation. This approach is illustrated with apple detection in challenging field conditions. We demonstrate the possibility of high performance in automatic apple segmentation (Dice 0.85), apple counting (88 percent of probability of good detection, and 0.09 true-negative rate), and apple localization (a shift error of fewer than 3 pixels) with eye-tracking systems. This is obtained by simply applying the areas of interest captured by the egocentric devices to standard, non-supervised image segmentation. We especially stress the importance in terms of time of using such eye-tracking devices on head-mounted systems to jointly perform image acquisition and automatic annotation. A gain of time of over 10-fold by comparison with classical image acquisition followed by manual image annotation is demonstrated.

A Hierarchical Image Annotation Method Based on SVM and Semi-supervised EM

Image distance metric learning based on neighborhood sets for automatic image annotation

摘要: 自动图像标识就是自动识别图像中的有意义目标并赋予其相应的语义关键词, 该过程虽然对于人类来说并不难, 但是对于计算机而言却是一项艰巨而有挑战性的任务. 鉴于人类识别物体通常是一个由粗到细的过程, 本文提出一种层次标识方案. 首先, 输入图像被自动分割成多个区域, 每个区域由支持向量机进行粗分类. 由于粗分类结果会直接影响后续细分类, 本文建立统计的上下文语义关系以修订不正确的粗标识. 接着为了对每个获得粗标识的区域进行细分类, 本文提出一种半监督期望最大化算法, 该算法不仅能为每一粗类别下的细类找到代表模式, 而且能对粗分类区域进行二次分类, 使其获得细标识. 最后我们再次应用上下文语义关系修订不合适的细标识. 为了证明上述识别方案的有效性, 我们开发了一个原型图像标识系统, 实验结果证明该层次标识方案是有效的. 关键词: 层次图像标识 / 支持向量机(SVM) / 半监督期望最大化 / 共存性 / 相对位置关系

Content-based Image retrieval systems extract and retrieve images using their low-level features, such as color, texture, and shape. Nevertheless, these visual contents do not allow a user to formulate semantically meaningful image query. Image annotation systems are a solution to solve the inadequacy of CBIR systems and allow text based image retrieval. There have been several studies on automatic image annotation utilizing machine learning techniques and images' representation with low level features extracted using either global or local methods. However, typically, these approaches suffer from low correlation between the globally assigned annotations and the visual features used to obtain annotations automatically. In this paper, we present an approach to enhance the effectiveness of CBIR using learning based automatic images annotation based on bag of visual word images representation that is created automatically using a set of manually annotated training images. The experimentation is performed with 4,000 annotated images for training, 1000 images for testing from ImageNet. The result has shown 77.5% of performance accuracy. The result of this work is believed to be one step towards enhancing the performance and effectiveness of existing CBIR and minimizing the semantic gap.

Automatic image annotation is one of the research fields helping to extract the meaning of images, which aims at the production of a set of semantic annotations for an image to help better present the concept. Over the past few decades, researchers have developed a number of approaches for automatic image annotation. Nevertheless, previous studies have not fully accounted for visual features and annotated features. Therefore, it is still possible to achieve a better annotation performance by combining visual and annotated information. In this study, our aim is to associate multiple semantic tags with a given image. In particular, we detect how to obtain the image annotation by utilizing visual and annotated information. To take advantage of visual information, we first designed a modified neural network method to acquire the features of the image content. In addition, to obtain the annotated features, we exploit an aggregated network embedding approach that consists of annotation embedding, social embedding, profile embedding, and semantic embedding. Finally, to produce an accurate image annotation, we integrate the two aforementioned methods, that is, combining the visual and annotated information, to build a unified cooperative training framework. The experimental results on three real-world datasets clarify that our presented method is superior to the current popular image annotation approaches.