General Multi-label Image Classification with Transformers

While deep convolutional neural networks (CNNs) have shown a great success in single-label image classification, it is important to note that real world images generally contain multiple labels, which could correspond to different objects, scenes, actions and attributes in an image. Traditional approaches to multi-label image classification learn independent classifiers for each category and employ ranking or thresholding on the classification results. These techniques, although working well, fail to explicitly exploit the label dependencies in an image. In this paper, we utilize recurrent neural networks (RNNs) to address this problem. Combined with CNNs, the proposed CNN-RNN framework learns a joint image-label embedding to characterize the semantic label dependency as well as the image-label relevance, and it can be trained end-to-end from scratch to integrate both information in a unified framework. Experimental results on public benchmark datasets demonstrate that the proposed architecture achieves better performance than the state-of-the-art multi-label classification model

The task of multi-label image classification involves recognizing multiple objects within a single image. Considering both valuable semantic information contained in the labels and essential visual features presented in the image, tight visual-linguistic interactions play a vital role in improving classification performance. Moreover, given the potential variance in object size and appearance within a single image, attention to features of different scales can help to discover possible objects in the image. Recently, Transformer-based methods have achieved great success in multi-label image classification by leveraging the advantage of modeling long-range dependencies, but they have several limitations. Firstly, existing methods treat visual feature extraction and cross-modal fusion as separate steps, resulting in insufficient visual-linguistic alignment in the joint semantic space. Additionally, they only extract visual features and perform cross-modal fusion at a single scale, neglecting objects with different characteristics. To address these issues, we propose a Hierarchical Scale-Aware Vision-Language Transformer (HSVLT) with two appealing designs: (1)A hierarchical multi-scale architecture that involves a Cross-Scale Aggregation module, which leverages joint multi-modal features extracted from multiple scales to recognize objects of varying sizes and appearances in images. (2)Interactive Visual-Linguistic Attention, a novel attention mechanism module that tightly integrates cross-modal interaction, enabling the joint updating of visual, linguistic and multi-modal features. We have evaluated our method on three benchmark datasets. The experimental results demonstrate that HSVLT surpasses state-of-the-art methods with lower computational cost.

Large-scale multi-label classification using unknown streaming images

Multi-label image classification focuses on predicting a set of object labels presented in an image, which is a practical yet challenging task since labels are normally co-occurred in an image while mutual interactions among labels and the correspondence between a given image and the corresponding labels are rarely considered in the existing methods. To address above challenges, we propose a Multiple Semantic Embedding model with Graph Convolutional Networks (MSEGCN) for multi-label image classification by capturing important dependencies related labels. Specifically, the proposed MSEGCN leverages graph structure to guide label co-occurrence propagation among different categories to obtain appropriate label representations. Then, by formulating multi-label classification problem as a label ranking problem with the aid of end-to-end convolutional neural network framework, we focus on learning a transform matrix to seek the image-label relevance relations in embedding space. Furthermore, an adaptive weighting strategy is introduced to effectively improve the classification performance. Experimental studies across a wide range of benchmark datasets show that our method achieves highly competitive performances against other state-of-the-art approaches.

Machine learning involves the task of training systems to be able to make decisions without being explicitly programmed. Important among machine learning tasks is classification involving the process of training machines to make predictions from predefined labels. Classification is broadly categorized into three distinct groups: single-label (SL), multi-class, and multi-label (ML) classification. This research work presents an application of a multi-label classification (MLC) technique in automating Quranic verses labeling. MLC has been gaining attention in recent years. This is due to the increasing amount of works based on real-world classification problems of multi-label data. In traditional classification problems, patterns are associated with a single-label from a set of disjoint labels. However, in MLC, an instance of data is associated with a set of labels. In this paper, three standard <em>MLC</em> methods: <span>binary relevance (BR), classifier chain (CC), and label powerset (LP) algorithms are implemented with four baseline classifiers: support vector machine (SVM), naïve Bayes (NB), k-nearest neighbors (k-NN), and J48. The research methodology adopts the multi-label problem transformation (PT) approach. The results are validated using six conventional performance metrics. These include: hamming loss, accuracy, one error, micro-F1, macro-F1, and avg. precision. From the results, the classifiers effectively achieved above 70% accuracy mark. Overall, SVM achieved the best results with CC and LP algorithms.</span>

Multi-label fundus disease classification using dual-branch deep learning: an intelligent diagnosis framework inspired by traditional Chinese medicine Five Wheels theory

Hierarchical GAN-Tree and Bi-Directional Capsules for multi-label image classification

Learning to rank has recently emerged as an attractive technique to train deep convolutional neural networks for various computer vision tasks. Pairwise ranking, in particular, has been successful in multi-label image classification, achieving state-of-the-art results on various benchmarks. However, most existing approaches use the hinge loss to train their models, which is non-smooth and thus is difficult to optimize especially with deep networks. Furthermore, they employ simple heuristics, such as top-k or thresholding, to determine which labels to include in the output from a ranked list of labels, which limits their use in the real-world setting. In this work, we propose two techniques to improve pairwise ranking based multi-label image classification: (1) we propose a novel loss function for pairwise ranking, which is smooth everywhere and thus is easier to optimize; and (2) we incorporate a label decision module into the model, estimating the optimal confidence thresholds for each visual concept. We provide theoretical analyses of our loss function in the Bayes consistency and risk minimization framework, and show its benefit over existing pairwise ranking formulations. We demonstrate the effectiveness of our approach on three large-scale datasets, VOC2007, NUS-WIDE and MS-COCO, achieving the best reported results in the literature.

Most existing work on multi-label learning focused on supervised learning which requires manual annotation samples that is labor-intensive, time-consuming and costly. To address such a problem, we present a novel method that incorporates active learning into the semi-supervised learning for multi-label image classification. What’s more, aiming at the curse of dimensionality existing in high-dimensional data, we explore a dimensionality reduction technique with non-negative sparseness constraint to extract a group of features that can completely describe the data and hence make the learning model more efficiently. Experimental results on common data sets validate that the proposed algorithm is relatively effective to improve the performance of the learner in multi-label classification, and the obtained learner is with reliability and robustness after data dimensionality using NNS-DR (Non-Negative Sparseness for Dimensionality Reduction).

Traditional supervised learning heavily relies on human-annotated datasets, especially in data-hungry neural approaches.However, various tasks, especially multi-label tasks like document-level relation extraction, pose challenges in fully manual annotation due to the specific domain knowledge and large class sets.Therefore, we address the multi-label positiveunlabelled learning (MLPUL) problem, where only a subset of positive classes is annotated.We propose Mixture Learner for Partially Annotated Classification (MLPAC), an RL-based framework combining the exploration ability of reinforcement learning and the exploitation ability of supervised learning.Experimental results across various tasks, including documentlevel relation extraction, multi-label image classification, and binary PU learning, demonstrate the generalization and effectiveness of our framework.

Multi-Label Image Classification (MLIC) appro-aches usually exploit label correlations to achieve good performance. However, emphasizing correlation like co-occurrence may overlook discriminative features and lead to model overfitting. In this study, we propose a generic framework named Parallel Self-Distillation (PSD) for boosting MLIC models. PSD decomposes the original MLIC task into several simpler MLIC sub-tasks via two elaborated complementary task decomposition strategies named Co-occurrence Graph Partition (CGP) and Dis-occurrence Graph Partition (DGP). Then, the MLIC models of fewer categories are trained with these sub-tasks in parallel for respectively learning the joint patterns and the category-specific patterns of labels. Finally, knowledge distillation is leveraged to learn a compact global ensemble of full categories with these learned patterns for reconciling the label correlation exploitation and model overfitting. Extensive results on MS-COCO and NUS-WIDE datasets demonstrate that our framework can be easily plugged into many MLIC approaches and improve performances of recent state-of-the-art approaches. The source code is released at https://github.com/Robbie-Xu/CPSD.

Multi-label image classification is a very challenging task, where data are often associated with multiple labels and represented with multiple views. In this paper, we propose a novel multi-view distance metric learning approach to dealing with the multi-label image classification problem. In particular, we attempt to concatenate multiple types of features after learning one optimal distance metric for each view, so as to obtain a better joint representation across multi-view spaces. To preserve the intrinsic geometric structure of the data in the low-dimensional feature space, we introduce a manifold regularization with the adjacency graph being constructed based on all labels. Moreover, we learn another distance metric to capture the dependency of labels, which can further improve the classification performance. Experimental results on publicly available image datasets demonstrate that our method achieves superior performance, compared with the state-of-the-arts.

Human visual perception shows good consistency for many multi-label image classification tasks under certain spatial transforms, such as scaling, rotation, flipping and translation. This has motivated the data augmentation strategy widely used in CNN classifier training -- transformed images are included for training by assuming the same class labels as their original images. In this paper, we further propose the assumption of perceptual consistency of visual attention regions for classification under such transforms, i.e., the attention region for a classification follows the same transform if the input image is spatially transformed. While the attention regions of CNN classifiers can be derived as an attention heatmap in middle layers of the network, we find that their consistency under many transforms are not preserved. To address this problem, we propose a two-branch network with an original image and its transformed image as inputs and introduce a new attention consistency loss that measures the attention heatmap consistency between two branches. This new loss is then combined with multi-label image classification loss for network training. Experiments on three datasets verify the superiority of the proposed network by achieving new state-of-the-art classification performance.

In our daily life, the scenes around us are always with multiple labels especially in a smart city, i.e., recognizing the information of city operation to response and control. Great efforts have been made by using Deep Neural Networks to recognize multi-label images. Since multi-label image classification is very complicated, people seek to use the attention mechanism to guide the classification process. However, conventional attention-based methods always analyzed images directly and aggressively. It is difficult for them to well understand complicated scenes. In this paper, we propose a global/local attention method that can recognize an image from coarse to fine by mimicking how human-beings observe images. Specifically, our global/local attention method first concentrates on the whole image, and then focuses on local specific objects in the image. We also propose a joint max-margin objective function, which enforces that the minimum score of positive labels should be larger than the maximum score of negative labels horizontally and vertically. This function can further improve our multi-label image classification method. We evaluate the effectiveness of our method on two popular multilabel image datasets (i.e., Pascal VOC and MS-COCO). Our experimental results show that our method outperforms state-of-the-art methods.

Multi-label image classification (MLIC) deals with assigning multiple labels to each image, a easy task for human being while still a open problem in machine learning. The greatest challenge in MLIC lies in that different target objects in one image keep distinct viewpoints and scales. One effective way is to borrow the label-related information to guide the selection of interesting region, which will act an important role in classification. By leveraging the attention mechanism in transformer, we propose a region-awared transformer to focus on top related regions and neglect background interference. Furthermore, our approach can cope with the positive-negative imbalance by assigning them different exponential decay factors of positive and negative samples separately. Experiments on MS-COCO show a competitive performance against other state-of-the-art methods.