A Hierarchical Image Annotation Method Based on SVM and Semi-supervised EM

A Hierarchical Image Annotation Method Based on SVM and Semi-supervised EM

Automatic image annotation (AIA) refers to the association of words to whole images which is considered as a promising and effective approach to bridge the semantic gap between low-level visual features and high-level semantic concepts. In this paper, we formulate the task of image annotation as a multi-label multi class semantic image classification problem and propose a simple yet effective algorithm: hybrid self-learning with alternating space between uni-modality and bi-modality, which integrate multi-label boosting with asymmetric binary SVM-based active learning into a joint hierarchical classification framework to perform cross-modal image annotation by incorporating unlabeled images. We conducted experiments on a medium-sized image collection including about 15000 images from Corel Stock Photo CDs. The experimental results demonstrated that our proposed method can enhance a given annotation model by using unlabeled images to some extent, showing the effectiveness of the proposed algorithm and the feasibility of unlabeled data to help the annotation accuracy.

Automatic image annotation is a technique that can be used to quickly generate tags for a massive dataset based on the content of the images. Nearest-neighbor-based methods such as TagProp are successful methods which have been used for image annotation. However, these methods focus more on weights based on the distances between the images and their neighbors, and ignore the weights of the different labels which can co-occur in the same image. In this paper, an improved method is proposed for automatic semantic annotation of images, which tags rare labels more effectively by processing the label matrix of the training set. In addition, image segmentation and data-driven methods are adopted to provide differential weights to the tags in one image, to improve the accuracy of the predicted tags. Experimental results show that the proposed method outperforms many classical baseline methods and can generate better annotation results than state-of-the-art nearest-neighbor based methods.

Automatic Image annotation is an important open problem in computer vision. In real world dataset environment, image labels are often noisy. For the task of image annotation with weakly labels, we propose SNLWL, a semantic neighborhood learning model from weakly labeled dataset. Missing labels are replenished using reweighting the error loss function. Then semantic balanced neighborhood is construct for samples in the training set. The methods allows the integration of multiple label metric learning and local nonnegative sparse coding. In this manner, we can optimally construct semantic consistent neighborhood where neighbors have higher global similarity, partial correlation, conceptual similarity along with semantic balance for samples in the training set. We also introduce an iterative denoising method of the label predictions to handle the noise. We investigate the performance of different variants of our method and compare to existing work. We present experimental results for various data sets. On all datasets, SNLWL makes a marked improvement as compared to the current state-of-the-art.

Automatic image annotation is an attractive service for users and administrators of online photo sharing websites. In this paper, we propose an image annotation approach exploiting visual and textual saliency. For textual saliency, a concept graph is firstly established based on the association between the labels. Then semantic communities and latent textual saliency are detected; For visual saliency, we adopt a dual-layer BoW (DL-BoW) model integrated with the local features and salient regions of the image. Experiments on NUS-WIDE dataset demonstrate that the proposed method outperforms other state-of-the-art approaches.

A survey and analysis on automatic image annotation

Image annotation systems aim at automatically annotating images with some predefined keywords. In this paper, we propose an automatic image annotation approach by incorporating word correlations into multi-class support vector machine (SVM). At first, each image is segmented into five fixed-size blocks or tiles and MPEG-7 visual descriptors are applied to represent color and texture features of blocks. Keywords are manually assigned to every block of training images. Then, multi-class SVM classifier is trained for semantic concepts. Word or concept correlations are computed by a co-occurrence matrix. The probability outputs from SVM and word correlations are combined to obtain the final results. The minimal-redundancy-maximum-relevance (mRMR) method is used to reduce feature dimensions. The experiments on Corel 5000 dataset demonstrate our approach is effective and efficient.

Joint multi-view representation and image annotation via optimal predictive subspace learning

This article presents an automatic image level annotation approach that takes advantage of both context and semantics presented in segmented images. The proposed approach is based on the optimization of classes’ scores using particle swarm optimization. In addition, random forest classifier and normalized cuts algorithm have been applied for automatic image classification, annotation, and clustering. For the proposed approach, each input image is segmented using the normalized cuts segmentation algorithm in order to create a descriptor for each segment. Two parameter selection models have been selected for particle swarm optimization algorithm and many voting techniques have been implemented to find the most suitable set of annotation words per image. Experimental results, using Corel5k benchmark annotated images dataset, demonstrate that applying optimization algorithms along with random forest classifier achieved noticeable increase in image annotation performance measures compared to related researches on the same dataset.

Image classification with the use of radial basis function neural networks and the minimization of the localized generalization error

Automatic image annotation is the process of assigning keywords to digital images depending on the content information. In one sense, it is a mapping from the visual content information to the semantic context information. In this study, we propose a novel approach for automatic image annotation problem, where the annotation is formulated as a multivariate mapping from a set of independent descriptor spaces, representing a whole image, to a set of words, representing class labels. For this purpose, a hierarchical annotation architecture, named as HANOLISTIC (Hierarchical Image Annotation System Using Holistic Approach), is defined with two layers. The first layer, called level-0 consists of annotators each of which is fed by a set of distinct descriptors, extracted from the whole image. This enables us to represent the image at each annotator by a different visual property of a descriptor. Since, we use the whole image, the problematic segmentation process is avoided. Training of each annotator is accomplished by a supervised learning paradigm, where each word is considered as a class label. Note that, this approach is slightly different then the classical training approaches, where each data has a unique label. In the proposed system, since each image has one or more annotating words, we assume that an image belongs to more than one class. The output of the level-0 annotators indicate the membership values of the words in the vocabulary, to belong an image. These membership values from each annotator is, then, aggregated at the second layer to obtain meta-level annotator. Finally, a set of words from the vocabulary is selected based on the ranking of the output of meta-level. The hierarchical annotation system proposed in this study outperforms state of the art annotation systems based on segmental and holistic approaches.

Automatic image annotation is the process of assigning keywords to digital images depending on the content information. In one sense, it is a mapping from the visual content information to the semantic context information. In this study, we propose a novel approach for automatic image annotation problem, where the annotation is formulated as a multivariate mapping from a set of independent descriptor spaces, representing a whole image, to a set of words, representing class labels. For this purpose, a hierarchical annotation architecture, named as HANOLISTIC (Hierarchical Image Annotation System Using Holistic Approach), is defined with two layers. The first layer, called level-0 consists of annotators each of which is fed by a set of distinct descriptor, extracted from the whole image. This enables us to represent the image at each annotator by a different visual property of a descriptor. Since, we use the whole image, the problematic segmentation process is avoided. Training of each annotator is accomplished by a supervised learning paradigm, where each word is represented by a class label. Note that, this approach is slightly different then the classical training approaches, where each data has a unique label. In the proposed system, since each image has one or more annotating words, we assume that an image belongs to more than one class. The output of the level-0 annotators indicate the membership values of the words in the vocabulary, to belong an image. These membership values from each annotator is, then, aggregated at the second layer to obtain meta-level annotator. Finally, a set of words from the vocabulary is selected based on the ranking of the output of meta-level. The hierarchical annotation system proposed in this study outperforms state of the art annotation systems based on segmental and holistic approaches.

A probabilistic topic model using deep visual word representation for simultaneous image classification and annotation

As digital images are increasing exponentially; it is very attractive to develop more effective machine learning frameworks for automatic image annotation. In order to address the most prominent issues (huge inter‐concept visual similarity and huge intra‐concept visual diversity) more effectively, an inter‐related non‐parametric Bayesian classifier training framework to support multi‐label image annotation is developed. For this purpose, an image is viewed as a bag, and its instances are the over‐segmented regions within it found automatically with an adopted Otsu's method segmentation algorithm. Here firefly algorithm (FA) is utilised to enhance Otsu's method in the direction of finding optimal multilevel thresholds using the maximum variance intra‐clusters. FA has high convergence speed and less computation rate as compared with some evolutionary algorithms. By generating blobs, the extracted features for segmented regions, the concepts which are learned by the classifier tend to relate textually to the words which occur most often in the data and visually to the easiest to recognise segments. This allowing the opportunity to assign a word to each object (localised labelling). Extensive experiments on Corel benchmark image datasets will validate the effectiveness of the proposed solution to multi‐label image annotation and label ranking problem.

Semantic image retrieval using text such keywords or captions at different semantic levels has attracted considerable research attention in recent years. Automatic image annotation (AIA) has been proved to be an effective and promising solution to automatically deduce the high-level semantics from low-level visual features. Due to the complexity of image-label mapping, it has become a challenge to systematically develop precise models with better performance. In this paper, we try to address the problem using Latent Semantic Indexing (LSI) together with mixed Bagging (Bootstrap Aggregating). Given a training set of annotated images, we first perform vector quantizing of visual space to construct a visual vocabulary consisting of visual terms based on K-means constrained by the semantic relationships derived from the associated annotations using Latent Semantic Indexing; then the statistical hidden correlation between the visual terms and keywords is modeled by a two-level mixed Bagging model. At the base level, each multi-class SVM is trained independently using the replicate training set via Bootstrap technique, and the decision of each multi-class SVM is integrated based on Average aggregating to generate a combined confidence label vector (CLV) associated with each visual term. All the CLVs are then propagated to the subsequent language model fusion level. At his level, WordNet is used for re-weighting the CLV for each visual term by incorporating the word correlations to obtain an appropriate and consistent image annotation. We carried out experiments on a medium-sized image collection with about 1000 images from Corel Stock Photo CDs. The experimental results demonstrated that the annotation performance of this method outperforms some traditional approaches, showing the feasibility and effectiveness of the proposed unified framework into which semantic image classification and language models are seamlessly integrated.