English

Joint multi-view representation and image annotation via optimal predictive subspace learning

In order to investigate the performance of visual feature extraction method for automatic image annotation, three visual feature extraction methods, namely discrete cosine transform, Gabor transform and discrete wavelet transform, are studied in this paper. These three methods are used to extract low-level visual feature vectors from images in a given database separately, then these feature vectors are mapped to high-level semantic words to annotate images with labels in a given semantic label set. As it is more efficient to depict the visual features of an image by the feature distribution than to resort to image segmentation technology for semantic image blocks, this paper is going to find out which of the three feature extraction methods performs better in image annotation based on the distribution of feature vectors from the image. The performance of three different kinds of feature extraction method is fully analyzed, and it is found that discrete cosine transform method is more suitable for Gaussian mixture model in automatic image annotation.

Automatic image annotation is crucial for keyword-based image retrieval because it can be used to improve the textual description of images efficiently. For this purpose, many methods have been developed. Due to the restrictions of computational complexity and small training set, the image annotation methods are usually based on the probability of individual word, instead of the joint probability of a set of words. Therefore the correlation between words is omitted. In this paper, we propose a method to approximate the joint probability of words in a progressive way. Given an image, the word with highest probability is first annotated. Then, the successive words are annotated by incorporating the information of previously annotated words. It can be seen as a greedy algorithm to calculate the joint probability of multiple words. The experiments show that the proposed progressive annotation method can effectively improve the annotation performance.

This paper presents a highly scalable and adaptable co-learning framework on multimodal image retrieval and image annotation. The co-learning framework is based on the multiple instance learning theory. While this framework is a general framework that may be used in any specific domains, to evaluate this framework, we apply it to the Berkeley Drosophila ISH embryo image database for the evaluations of the retrieval and annotation performance. In addition, we also apply this framework to across-stage inferencing for the embryo images for knowledge discovery. We have compared the performance of the framework for retrieval, annotation, and inferencing on the Berkeley Drosophila ISH database with a state-of-the-art multimodal image retrieval and annotation method to demonstrate the effectiveness and the promise of the framework.

Automatic Image Annotation is important research topic in machine vision as it enables one to retrieve images from large databases by using textual queries. In recent years many machine learning techniques have been proposed to build detectors of concepts present on the images. In this paper we present a novel approach for image auto-annotation based on transfer of annotations from most similar images to the query image. We model image features by Multivariate Gaussian Distribution and measure distance between images by using Jensen-Shannon divergence. In spite of its simplicity, the proposed solution outperforms the state-of-the-art methods for image annotation and thus can be used as a baseline for developing other more elaborate methods.KeywordsQuery ImageSimilar ImageImage AnnotationSemantic LabelAutomatic Image AnnotationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

With the advance of internet, multi-modal data can be easily collected from many social websites such as Wikipedia, Flickr, YouTube, etc. Images shared on the web are usually associated with social tags or other textual information. Although existing multi-modal methods can make use of associated text to improve image annotation, the disadvantages of them are that associated text is also required for a new image to be predicted. In this paper, we propose the cross-modal multi-task learning (CMMTL) framework for image annotation. Labeled and unlabeled multi-modal data are both levaraged for training in CMMTL, and it finally obtains visual classifiers which can predict concepts for a single image without any associated information. CMMTL integrates graph learning, multi-task learning and cross-modal learning into a joint framework, where a shared subspace is learned to preserve both cross-modal correlation and concept correlation. The optimal solution of the proposed framework can be obtained by solving a generalized eigenvalue problem. We conduct comprehensive experiments on two real world image datasets: MIR Flickr and NUS-WIDE, to evaluate the performance of the proposed framework. Experimental results demonstrate that CMMTL obtains a significant improvement over several representative methods for cross-modal image annotation.

Current coral reef health monitoring programs rely on biodiversity data obtained through the acquisition and annotation of underwater photographs. Manual annotation of these photographs is a necessary step, but has become problematic due to the high volume of images and the high cost of human resources. While automated and reliable multi-spectral annotation methods exist, coral reef images are often limited to visible light, which makes automation difficult. Much of the previous work has focused on popular texture recognition methods, but the results remain unsatisfactory when compared to human performance for the same task. In this work, we present an improved automatic method for coral image annotation that yields consistent accuracy improvements over existing methods. Our method builds on previous work by combining multiple feature representations. We demonstrate that the aggregation of multiple methods outperforms any single method. Furthermore, our proposed system requires virtually no parameter tuning, and supports rejection for improved results. Firstly, the complex texture diversity of corals is handled by combining multiple feature representations: local binary patterns, hue and opponent angle histograms, textons, and deep convolutional activation feature. Secondly, these multiple representations are aggregated using a score-level fusion of multiple support vector machines. Thirdly, rejection can optionally be applied to enhance classification results, and allows efficient semi-supervised image annotation in collaboration with human experts.

Image annotation is an open and challenging task, especially with large label vocabulary. In this paper, we propose a novel graph learning based method for image annotation, which takes both advantages of the nearest neighbour based and the graph-based methods, by exploiting the graph learning method to propagate the labels on the graph corresponding to the K nearest neighbours of a test image. To acquire more effective graph weights for computing score for each label, besides the similarity of visual features, our method also considers the similarity of two label sets, which is computed based on the label correlation that captures the semantic information between two labels. In addition, we combine the image-to-label distance with the graph learning based score to compute the final decision value for labelling. The proposed method is evaluated on three benchmark datasets for image annotation. The result shows our method substantially outperforms the previous graph learning based methods, and our result matches the current state-of-the-art results in annotation quality.

Automatic image annotation based on an improved nearest neighbor technique with tag semantic extension model

Region-Based Shape Matching for Automatic Image Annotation and Query-by-Example

Even within constrained domains like medicine there are no truly generic methods for automatic image parsing and annotation. Despite the fact that the precision and sophistication of image understanding methods have improved to cope with the increasing amount and complexity of the data, the improvements have not resulted in more flexible or generic image understanding techniques. Instead, the analysis methods are object specific and modality dependent. Consequently, current image search techniques are still dependent on the manual and subjective association of keywords to images for retrieval. Manually annotating the vast numbers of images which are generated and archived in the medical practice is not an option.

Automatic image annotation has been extensively studied, mostly from a content-based approach, whose effectiveness is restricted by the “semantic gap” between low-level image features and semantic annotations, and by the irrelevance of annotations to image content. We propose a social diffusion analysis approach to image annotation, which exploits abundant social diffusion records about how images are disseminated within online social networks. Specifically, we propose a common-interest model to analyze social diffusion records, with the assumption that the diffusion pattern of an image in social networks is highly related to the relevance between image annotations and user preferences. In our proposed model, user preferences are represented as common interests of pairwise users instead of individual user interests. We find the notion of common interests not only facilitates the analysis of social diffusion patterns, but also leads to more accurate profiling of user preferences compared to individual interests. Based on the common-interest model, we design an image annotation framework via social diffusion analysis, which consists of the mining of common interests from social diffusion records, the feature extraction from diffusion graphs and common interests, and the automatic annotation by the learning-to-rank method. Experimental results on real-world data sets show that our proposed common-interest based approach outperforms individual-interest based methods, and also achieves superior performance than state-of-the-art content-based image annotation methods.

Automatic image annotation (AIA) refers to the association of words to whole images which is considered as a promising and effective approach to bridge the semantic gap between low-level visual features and high-level semantic concepts. In this paper, we formulate the task of image annotation as a multi-label multi class semantic image classification problem and propose a simple yet effective algorithm: hybrid self-learning with alternating space between uni-modality and bi-modality, which integrate multi-label boosting with asymmetric binary SVM-based active learning into a joint hierarchical classification framework to perform cross-modal image annotation by incorporating unlabeled images. We conducted experiments on a medium-sized image collection including about 15000 images from Corel Stock Photo CDs. The experimental results demonstrated that our proposed method can enhance a given annotation model by using unlabeled images to some extent, showing the effectiveness of the proposed algorithm and the feasibility of unlabeled data to help the annotation accuracy.

Digital imaging has become an essential element in every medical institution. Therefore, medical image retrieval such as chest X-ray (CXR) must be improved via novel feature extraction and annotation activities before they are stored into image databases. To date, many methods have been introduced to annotate medical images using spatial relationships after these features are extracted. However, the annotation performance for each method is inconsistent and does not show promising achievement to retrieve images. It is noticed that each method is still struggling with at least two big problems. Firstly, the recommended annotation model is weak because the method does not consider the object shape and rely on gross object shape estimation. Secondly, the suggested annotation model can only be functional for simple object placement. As a result, it is difficult to determine the spatial relationship feature after they are extracted to annotate images accurately. Hence, this study aims to propose a new model to annotate nodule location within lung zone for CXR image with extracted spatial relationship feature to improve image retrieval. In order to achieve the aim, a methodology that consists of six phases of CXR image annotation using the extracted spatial relationship features is introduced. This comprehensive methodology covers all cycles for image annotation tasks starting from image pre-processing until determination of spatial relationship features for the lung zone in the CXR. The outcome from applying the methodology also enables us to produce a new semi-automatic annotation system named CHEXRIARS which acts as a tool to annotate the extracted spatial relationship features in CXR images. The CHEXRIARS performance is tested using a retrieval test with two common tests namely the precision and recall (PNR). Apart from CHEXRIARS, three other annotation methods that are object slope, object projection and comparison of region boundaries are also included in the retrieval performance test. Overall, the CHEXRIARS interpolated PNR curve shows the best shape because it is the closest curve approaching the value of 1 on the X-axis and Y-axis. Meanwhile the value of area under curve for CHEXRIARS also revealed that this system attained the highest score at 0.856 as compared to the other three annotation methods. The outcome from the retrieval performance test indicated that the proposed annotation model has produced outstanding outcome and improved the image retrieval.

Multilabel remote sensing (RS) image annotation is a challenging and time-consuming task that requires a considerable amount of expert knowledge. Most existing RS image annotation methods are based on handcrafted features and require multistage processes that are not sufficiently efficient and effective. An RS image can be assigned with a single label at the scene level to depict the overall understanding of the scene and with multiple labels at the object level to represent the major components. The multiple labels can be used as supervised information for annotation, whereas the single label can be used as additional information to exploit the scene-level similarity relationships. By exploiting the dual-level semantic concepts, we propose an end-to-end deep learning framework for object-level multilabel annotation of RS images. The proposed framework consists of a shared convolutional neural network for discriminative feature learning, a classification branch for multilabel annotation and an embedding branch for preserving the scene-level similarity relationships. In the classification branch, an attention mechanism is introduced to generate attention-aware features, and skip-layer connections are incorporated to combine information from multiple layers. The philosophy of the embedding branch is that images with the same scene-level semantic concepts should have similar visual representations. The proposed method adopts the binary cross-entropy loss for classification and the triplet loss for image embedding learning. The evaluations on three multilabel RS image data sets demonstrate the effectiveness and superiority of the proposed method in comparison with the state-of-the-art methods.