A combinatorial optimization approach for multi-label associative classification

Lately, multi-label classification (MLC) problems have drawn a lot of attention in a wide range of fields including medical, web, and entertainment. The scale and the diversity of MLC problems is much larger than single-label classification problems. Especially we have to face all possible combinations of labels. To solve MLC problems more efficiently, we focus on three kinds of locality hidden in a given MLC problem. In this paper, first we show how large degree of locality exists in nine datasets, then examine how closely they are related to labels, and last propose a method of reducing the problem size using one kind of locality.

PurposeImage classification is becoming a supporting technology in several image-processing tasks. Due to rich semantic information contained in the images, it is very popular for an image to have several labels or tags. This paper aims to develop a novel multi-label classification approach with superior performance.Design/methodology/approachMany multi-label classification problems share two main characteristics: label correlations and label imbalance. However, most of current methods are devoted to either model label relationship or to only deal with unbalanced problem with traditional single-label methods. In this paper, multi-label classification problem is regarded as an unbalanced multi-task learning problem. Multi-task least-squares support vector machine (MTLS-SVM) is generalized for this problem, renamed as multi-label LS-SVM (ML2S-SVM).FindingsExperimental results on the emotions, scene, yeast and bibtex data sets indicate that the ML2S-SVM is competitive with respect to the state-of-the-art methods in terms of Hamming loss and instance-based F1 score. The values of resulting parameters largely influence the performance of ML2S-SVM, so it is necessary for users to identify proper parameters in advance.Originality/valueOn the basis of MTLS-SVM, a novel multi-label classification approach, ML2S-SVM, is put forward. This method can overcome the unbalanced problem but also explicitly models arbitrary order correlations among labels by allowing multiple labels to share a subspace. In addition, the multi-label classification approach has a wider range of applications. That is to say, it is not limited to the field of image classification.

Multi label classification is concerned with learning from a set of instances that are associated with a set of labels, that is, an instance could be associated with multiple labels at the same time. This task occurs frequently in application areas like text categorization, multimedia classification, bioinformatics, protein function classification and semantic scene classification. Current multi-label classification methods could be divided into two categories. The first is called problem transformation methods, which transform multi-label classification problem into single label classification problem, and then apply any single label classifier to solve the problem. The second category is called algorithm adaptation methods, which adapt an existing single label classification algorithm to handle multi-label data. In this paper, we propose a multi-label classification approach based on correlations among labels that use both problem transformation methods and algorithm adaptation methods. The approach begins with transforming multi-label dataset into a single label dataset using least frequent label criteria, and then applies the PART algorithm on the transformed dataset. The output of the approach is multi-labels rules. The approach also tries to get benefit from positive correlations among labels using predictive Apriori algorithm. The proposed approach has been evaluated using two multi-label datasets named (Emotions and Yeast) and three evaluation measures (Accuracy, Hamming Loss, and Harmonic Mean). The experiments showed that the proposed approach has a fair accuracy in comparison to other related methods.

Genetic programming (GP) is an artificial intelligence technique that benefits from evolutionary computations allowing computers to solve problems automatically. In this paper, we present an optimized genetic-programming-based classifier that directly solves the multi-class classification problems in data mining and image analysis. A new fitness function is proposed for multiclass classification and brain tumor detection, which is validated by 10-fold cross validation. Instead of defining static thresholds as boundaries to differentiate between multiple labels, our work presents a method of classification where a GP system learns the relationships among experiential data and models them mathematically during the evolutionary process. We propose an optimized GP classifier based on a combination of pruning subtrees and a new fitness function. An orthogonal least squares algorithm is also applied in the training phase to create a robust GP classifier. Our approach has been assessed on six multiclass datasets and on a magnetic resonance imaging (MRI) brain image for tumor detection. The results of data classification for Iris, Wine, Glass, Pima, BUPA Liver and Balance Scale datasets are compared with existing algorithms. The high accuracy of brain tumor classification provided by our GP classifier confirms the strong ability of the developed technique for complicated classification problems. We compared our approach in terms of speed with previous GP algorithms as well. The analyzed results illustrate that the developed classifier produces a productive and rapid method for classification tasks that outperforms the previous methods for more challenging multiclass classification problems.

Many computer vision applications, such as image classification and video indexing, are usually multi-label classification problems in which an instance can be assigned to more than one category. In this paper, we present a novel multi-label classification approach with hypergraph regularization that addresses the correlations among different categories. First, a hypergraph is constructed to capture the correlations among different categories, in which each vertex represents one training instance and each hyperedge for one category contains all the instances belonging to the same category. Then, an improved SVM like learning system incorporating the hypergraph regularization, called Rank-HLapSVM, is proposed to handle the multi-label classification problems. We find that the corresponding optimization problem can be efficiently solved by the dual coordinate descent method. Many promising experimental results on the real datasets including ImageCLEF and MediaMill demonstrate the effectiveness and efficiency of the proposed algorithm.

Convolutional Neural Networks (CNNs) have proven to be very effective in image classification due to their status as a powerful feature learning algorithm. Traditional approaches have considered the problem of multiclass classification, where the goal is to classify a set of objects at once. However, co-occurrence can make the discriminative features of the target less salient and may lead to overfitting of the model, resulting in lower performance. To address this, we propose a multi-label classification ensemble model including a Vision Transformer (ViT) and CNN for directly detecting one or multiple objects in an image. First, we improve the MobileNetV2 and DenseNet201 models using extra convolutional layers to strengthen image classification. In detail, three convolution layers are applied in parallel at the end of both models. ViT can learn dependencies among distant positions and local detail, making it an effective tool for multi-label classification. Finally, an ensemble learning algorithm is used to combine the classification predictions of the ViT, the modified MobileNetV2, and DenseNet201 bands for increased image classification accuracy using a voting system. The performance of the proposed model is examined on four benchmark datasets, achieving accuracies of 98.24%, 98.89%, 99.91%, and 96.69% on ASCAL VOC 2007, PASCAL VOC 2012, MS-COCO, and NUS-WIDE 318, respectively, showing that our framework can enhance current state-of-the-art methods.

Social media is a widely used platform that provides a huge amount of user-generated content that can be processed to extract information about users’ emotions. This has numerous benefits, such as understanding how individuals feel about certain news or events. It can be challenging to categorize emotions from text created on social media, especially when trying to identify several different emotions from a short text length, as in a multi-label classification problem. Most previous work on emotion classification has focused on deep neural networks such as Convolutional Neural Networks and Recurrent Neural Networks. However, none of these networks have used semantic and syntactic knowledge to classify multiple emotions from a text. In this study, semantic and syntactic aware graph attention networks were proposed to classify emotions from a text with multiple labels. We integrated semantic information in the graph attention network in the form of Universal Conceptual Cognitive Annotation and syntactic information in the form of dependency trees. Our extensive experimental results showed that our two models, UCCA-GAT (accuracy = 71.2) and Dep-GAT (accuracy = 68.7), were able to outperform the state-of-the-art performance on both the challenging SemEval-2018 E-c: Detecting Emotions (multi-label classification) English dataset (accuracy = 58.8) and GoEmotions dataset (accuracy = 65.9).

Multi-label classification aims to associate multiple labels to a given data/object instance to better describe them. Multi-label data sets are common in a lot of emerging application areas like: Text/Multimedia classification, Bio-Informatics, Medical image annotations and Computer Vision to name a few. There is a growing interest in efficient and accurate multi-label classification. There are two major approaches to perform multi-label classification (i) problem transformation methods and (ii) algorithm adaptation methods. In algorithm adaptation, the traditional classification algorithms are modified to handle multi-label data sets. One classification algorithm which is often modified to do multi-label classification is k- nearest neighbor (kNN). k-nearest neighbor is popular due to its simplicity, easy to implement and seamlessly adaptability. Despite its merits it has several drawbacks like: sensitivity of noisy data, missing values and outliers; feature scaling and often becoming inaccurate for large overlapping solution space. In this paper, a modification to kNN method is suggested for multi-label classification with three improvement strategies (i) selection of local example w.r.t. unknown example – the motivation for this comes from the fact that local and relevant space is vital for the improvement in multi-label classification; (ii) Splitting the input space into multiple sub-spaces for optimal label estimation – the motivation is to estimate label accurately in the presence of noisy labels; And (iii) selection of labels using Mean Average Precision (MAP) estimates – here our motivation is to utilize the training data effectively to maximize the hidden distribution and optimal parameters for the method. The proposed method is implemented and compared with state-of-the-art approaches based on kNN or similar approaches that effectively select and optimize relevant spaces for multi-label classification. Evaluation based on multiple metrics like Hamming loss, Precision/Recall and F-measure are used for evaluation. The suggested approach performed much better than the state-of-the-art on the datasets with strong label cardinalities.

Multi-label image classification is a very challenging task, where data are often associated with multiple labels and represented with multiple views. In this paper, we propose a novel multi-view distance metric learning approach to dealing with the multi-label image classification problem. In particular, we attempt to concatenate multiple types of features after learning one optimal distance metric for each view, so as to obtain a better joint representation across multi-view spaces. To preserve the intrinsic geometric structure of the data in the low-dimensional feature space, we introduce a manifold regularization with the adjacency graph being constructed based on all labels. Moreover, we learn another distance metric to capture the dependency of labels, which can further improve the classification performance. Experimental results on publicly available image datasets demonstrate that our method achieves superior performance, compared with the state-of-the-arts.

Neuroevolution is a powerful and general technique for evolving the structure and weights of artificial neural networks. Though neuroevolutionary approaches such as NeuroEvolution of Augmenting Topologies (NEAT) have been successfully applied to various problems including classification, regression, and reinforcement learning problems, little work has explored application of these techniques to larger-scale multiclass classification problems. In this paper, NEAT is evaluated in several multiclass classification problems, and then extended via two ensemble approaches: One-vs-All and One-vs-One. These approaches decompose multiclass classification problems into a set of binary classification problems, in which each binary problem is solved by an instance of NEAT. These ensemble models exhibit reduced variance and increasingly superior accuracy as the number of classes increases. Additionally, higher accuracy is achieved early in training, even when artificially constrained for the sake of fair comparison with standard NEAT. However, because the approach can be trivially distributed, it can be applied quickly at large scale to solve real problems. In fact, these approaches are incorporated into DarwinTM, an enterprise automatic machine learning solution that also incorporates various other algorithmic enhancements to NEAT. The resulting complete system has proven robust to a wide variety of client datasets.

Medication safety in residential care facilities is a critical concern, particularly when nonmedical staff provide medication assistance. The complex nature of medication-related incidents in these settings, coupled with the psychological impact on health care providers, underscores the need for effective incident analysis and preventive strategies. A thorough understanding of the root causes, typically through incident-report analysis, is essential for mitigating medication-related incidents. We aimed to develop and evaluate a multilabel classifier using natural language processing to identify factors contributing to medication-related incidents using incident report descriptions from residential care facilities, with a focus on incidents involving nonmedical staff. We analyzed 2143 incident reports, comprising 7121 sentences, from residential care facilities in Japan between April 1, 2015, and March 31, 2016. The incident factors were annotated using sentences based on an established organizational factor model and previous research findings. The following 9 factors were defined: procedure adherence, medicine, resident, resident family, nonmedical staff, medical staff, team, environment, and organizational management. To assess the label criteria, 2 researchers with relevant medical knowledge annotated a subset of 50 reports; the interannotator agreement was measured using Cohen κ. The entire data set was subsequently annotated by 1 researcher. Multiple labels were assigned to each sentence. A multilabel classifier was developed using deep learning models, including 2 Bidirectional Encoder Representations From Transformers (BERT)-type models (Tohoku-BERT and a University of Tokyo Hospital BERT pretrained with Japanese clinical text: UTH-BERT) and an Efficiently Learning Encoder That Classifies Token Replacements Accurately (ELECTRA), pretrained on Japanese text. Both sentence- and report-level training were performed; the performance was evaluated by the F1-score and exact match accuracy through 5-fold cross-validation. Among all 7121 sentences, 1167, 694, 2455, 23, 1905, 46, 195, 1104, and 195 included "procedure adherence," "medicine," "resident," "resident family," "nonmedical staff," "medical staff," "team," "environment," and "organizational management," respectively. Owing to limited labels, "resident family" and "medical staff" were omitted from the model development process. The interannotator agreement values were higher than 0.6 for each label. A total of 10, 278, and 1855 reports contained no, 1, and multiple labels, respectively. The models trained using the report data outperformed those trained using sentences, with macro F1-scores of 0.744, 0.675, and 0.735 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. The report-trained models also demonstrated better exact match accuracy, with 0.411, 0.389, and 0.399 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. Notably, the accuracy was consistent even when the analysis was confined to reports containing multiple labels. The multilabel classifier developed in our study demonstrated potential for identifying various factors associated with medication-related incidents using incident reports from residential care facilities. Thus, this classifier can facilitate prompt analysis of incident factors, thereby contributing to risk management and the development of preventive strategies.

This paper explores an interesting way to solve a multi-label classification problem using an ensemble learning with natural language processing approach. The multi-label classification problem selected is that of movie genre prediction where each movie can possibly belong to multiple genres which essentially differentiates it from a much more classical multi-class classification problem. In the current machine learning environment, there are a host of different classifiers, some which are more suited to handle more than binary classes at once such as random forests and decision trees while some which are modelled around solving an inherently binary classification problem. As we are using textual data in the form of synopsis of various movies, we use Word2Vec to vectorize the textual data in order to fit the classifiers. In this project, we use classifiers belonging to both these types, namely Naive Bayes, random forest classifier and XGBoost classifier to individually predict the multiple labels for each movie, and then we construct an ensemble model using a voting classier in the purview of ensemble learning by adjusting the weights of the combining classifiers and show that the ensemble model performs much better than each of the individual classifiers. The final micro-f-score achieved is 0.6738 which is commendable with using non-neural traditional machine learning techniques and is an improvement from the referred work for this paper.

Boosting ICD multi-label classification of health records with contextual embeddings and label-granularity

Support vector machines (SVMs) can solve structured multioutput learning problems such as multilabel classification, multiclass classification, and vector regression. SVM training is expensive, especially for large and high-dimensional data sets. The bottleneck of the SVM training often lies in the kernel value computation. In many real-world problems, the same kernel values are used in many iterations during the training, which makes the caching of kernel values potentially useful. The majority of the existing studies simply adopt the least recently used (LRU) replacement strategy for caching kernel values. However, as we analyze in this article, the LRU strategy generally achieves high hit ratio near the final stage of the training but does not work well in the whole training process. Therefore, we propose a new caching strategy called EFU (less frequently used), which replaces the EFU kernel values that enhance least frequently used (LFU). Our experimental results show that EFU often has 20% higher hit ratio than LRU in the training with the Gaussian kernel. To further optimize the strategy, we propose a caching strategy called hybrid caching for the SVM training (HCST), which has a novel mechanism to automatically adapt the better caching strategy in different stages of the training. We have integrated the caching strategy into ThunderSVM, a recent SVM library on many-core processors. Our experiments show that HCST adaptively achieves high hit ratios with little runtime overhead among different problems including multilabel classification, multiclass classification, and regression problems. Compared with other existing caching strategies, HCST achieves 20% more reduction in training time on average.

In the last decades, the use of ensembles of classifiers in different domains of application has received significant attention of the Machine Learning community. In a typical architecture of ensemble, a new input pattern is presented to all K components. The individual classifiers provide their output and send them to a combination method, which is responsible for providing the final output. Ensemble of classifiers can be applied to any classification problem, single-label or multi- label problems. Multi-label (ML) classification has been received much attention from various research domains, such as text categorization, bioinformatics, computer vision, among others. In this paper, we investigate the use of ensemble of classifiers when applied to multi-label problems, focusing on the use of diversity measures in these systems when applied to multi-label problems. In order to do this, the present work proposes an adapted approach for two well-known diversity measures, good and bad, for ensembles of multi-label classifiers. In order to assess the feasibility of the proposed diversity measures in ensemble of multi-label classifiers, an empirical analysis is conducted, with eight multi-label classification problems. Generally, our finds indicate that there is a strong correlation between performance of the ML ensembles with the proposed diversity measures, showing that these diversity measures can be used as an important tool in the design of efficient ensembles of multi-label classifiers.