A flexible class of dependence-aware multi-label loss functions

Machine learning involves the task of training systems to be able to make decisions without being explicitly programmed. Important among machine learning tasks is classification involving the process of training machines to make predictions from predefined labels. Classification is broadly categorized into three distinct groups: single-label (SL), multi-class, and multi-label (ML) classification. This research work presents an application of a multi-label classification (MLC) technique in automating Quranic verses labeling. MLC has been gaining attention in recent years. This is due to the increasing amount of works based on real-world classification problems of multi-label data. In traditional classification problems, patterns are associated with a single-label from a set of disjoint labels. However, in MLC, an instance of data is associated with a set of labels. In this paper, three standard <em>MLC</em> methods: <span>binary relevance (BR), classifier chain (CC), and label powerset (LP) algorithms are implemented with four baseline classifiers: support vector machine (SVM), naïve Bayes (NB), k-nearest neighbors (k-NN), and J48. The research methodology adopts the multi-label problem transformation (PT) approach. The results are validated using six conventional performance metrics. These include: hamming loss, accuracy, one error, micro-F1, macro-F1, and avg. precision. From the results, the classifiers effectively achieved above 70% accuracy mark. Overall, SVM achieved the best results with CC and LP algorithms.</span>

Bayesian decision problems require subjective elicitation of the inputs: beliefs and preferences. Sometimes, elicitation methods may not represent perfectly the judgements of the decision maker. Several foundations propose to overlay this problem using robust approaches. In these models, beliefs are modelled by a class of probability distributions and preferences by a class of loss functions. Then, we are in the conditions of a Pareto order. Hence the solution concept is the set of non dominated alternatives. In this article we focus on the computation of the efficient set when the preferences are modeled by a class of convex loss functions.

Using tensor products to detect unconditional label dependence in multilabel classifications

Multi label classification is concerned with learning from a set of instances that are associated with a set of labels, that is, an instance could be associated with multiple labels at the same time. This task occurs frequently in application areas like text categorization, multimedia classification, bioinformatics, protein function classification and semantic scene classification. Current multi-label classification methods could be divided into two categories. The first is called problem transformation methods, which transform multi-label classification problem into single label classification problem, and then apply any single label classifier to solve the problem. The second category is called algorithm adaptation methods, which adapt an existing single label classification algorithm to handle multi-label data. In this paper, we propose a multi-label classification approach based on correlations among labels that use both problem transformation methods and algorithm adaptation methods. The approach begins with transforming multi-label dataset into a single label dataset using least frequent label criteria, and then applies the PART algorithm on the transformed dataset. The output of the approach is multi-labels rules. The approach also tries to get benefit from positive correlations among labels using predictive Apriori algorithm. The proposed approach has been evaluated using two multi-label datasets named (Emotions and Yeast) and three evaluation measures (Accuracy, Hamming Loss, and Harmonic Mean). The experiments showed that the proposed approach has a fair accuracy in comparison to other related methods.

In multi-label classification, each example can be associated with multiple labels simultaneously. The task of learning from multilabel data can be addressed by methods that transform the multi-label classification problem into several single-label classification problems. The binary relevance approach is one of these methods, where the multilabel learning task is decomposed into several independent binary classification problems, one for each label in the set of labels, and the final labels for each example are determined by aggregating the predictions from all binary classifiers. However, this approach fails to consider any dependency among the labels. In this paper, we consider a simple approach which can be used to explore labels dependency aiming to accurately predict label combinations. An experimental study using decision trees, a kernel method as well as Naive Bayes as base-learning techniques shows the potential of the proposed approach to improve the multi-label classification performance.

Multi-label classification (MLC) deals with the task of assigning an instance to all its relevant classes. This task becomes challenging in the presence of the label dependencies. The MLC methods that assume label independence do not use the dependencies among labels. We present a two-stage framework which improves the performance of MLC by using label dependencies. In the first stage, a standard MLC method is used to get the confidence scores for different labels. A conditional random field (CRF) is used in the second stage that improves the performance of the first-stage MLC by using the label dependencies among labels. An optimization-based framework is used to learn the structure and parameters of the CRF. Experiments show that the proposed model performs better than the state-of-the-art methods for MLC.

Multi-label classification has attracted significant interests in various domains. However, only few positive labels are labeled. Treating unknown labels as negative labels brings the one-side noise, which causes the severe performance degradation. Motivated by this, we present the positive and unlabeled (PU) matrix completion based Multi-Label Classification with Missing Labels called PUMCML, that focus on finding all positive labels and construct a classifier based on completion matrix under instance and label dependencies. More specifically, an asymmetrical PU learning loss function is firstly used to solve the one-side noise from missing labels. Secondly, an instance dependencies assumption is proposed that similar instances in multi-label data will share similar labels. Furthermore, considering the label dependencies, the completion matrix consisting of the predicted labels and origin labels is constrained by a nuclear norm. Finally, the proposed algorithm is optimization problem which is convex but not smooth, so Alternating Direction Multiplier Method is design for handling the optimization problem by dividing the problem into multiple subproblems. Experimental results on four benchmarks demonstrate our method achieves a huge improvement in performance and robustness to missing labels compared to other advanced algorithms.

This paper presents a pruned sets method (PS) for multi-label classification. It is centred on the concept of treating sets of labels as single labels. This allows the classification process to inherently take into account correlations between labels. By pruning these sets, PS focuses only on the most important correlations, which reduces complexity and improves accuracy. By combining pruned sets in an ensemble scheme (EPS), new label sets can be formed to adapt to irregular or complex data. The results from experimental evaluation on a variety of multi-label datasets show that [E]PS can achieve better performance and train much faster than other multi-label methods.

The rapid growth of unstructured textual data in various domains has necessitated the development of sophisticated techniques for multi-label text classification. Traditional methods often struggle with handling the complexity and interdependence of multiple labels, leading to suboptimal performance. This paper presents an advanced approach that integrates the Beta Ant Colony Optimization (BACO) algorithm with deep learning techniques for multi-label text classification. The BACO algorithm effectively explores the feature space and selects relevant features by leveraging pheromone trails, while the deep learning model captures intricate patterns and relationships within the textual data. The integration of these two methodologies aims to enhance the efficiency and accuracy of multi-label classification tasks, particularly in domains where label dependency is prominent. Empirical evaluations on benchmark datasets demonstrate that the proposed hybrid approach outperforms existing state-of-the-art techniques in terms of precision, recall, F1-score, and computational efficiency. The findings suggest that combining heuristic optimization algorithms with deep learning can significantly improve multi-label text classification performance, providing a robust solution for real-world applications.

The aim of this paper is to give criteria of the sufficiency of certain complete σ-algebras occurring in the theory of unbiased estimation by means of minimal unbiased estimators. The class ofYoung-functions is used to construct a large class of convex loss functions and a theorem ofBahadur based on quadratic loss is generalized to this general class of loss functions.

Top-k error is currently a popular performance measure on large scale image classification benchmarks such as ImageNet and Places. Despite its wide acceptance, our understanding of this metric is limited as most of the previous research is focused on its special case, the top-1 error. In this work, we explore two directions that shed more light on the top-k error. First, we provide an in-depth analysis of established and recently proposed single-label multiclass methods along with a detailed account of efficient optimization algorithms for them. Our results indicate that the softmax loss and the smooth multiclass SVM are surprisingly competitive in top-k error uniformly across all k, which can be explained by our analysis of multiclass top-k calibration. Further improvements for a specific k are possible with a number of proposed top-k loss functions. Second, we use the top-k methods to explore the transition from multiclass to multilabel learning. In particular, we find that it is possible to obtain effective multilabel classifiers on Pascal VOC using a single label per image for training, while the gap between multiclass and multilabel methods on MS COCO is more significant. Finally, our contribution of efficient algorithms for training with the considered top-k and multilabel loss functions is of independent interest.

While deep convolutional neural networks (CNNs) have shown a great success in single-label image classification, it is important to note that real world images generally contain multiple labels, which could correspond to different objects, scenes, actions and attributes in an image. Traditional approaches to multi-label image classification learn independent classifiers for each category and employ ranking or thresholding on the classification results. These techniques, although working well, fail to explicitly exploit the label dependencies in an image. In this paper, we utilize recurrent neural networks (RNNs) to address this problem. Combined with CNNs, the proposed CNN-RNN framework learns a joint image-label embedding to characterize the semantic label dependency as well as the image-label relevance, and it can be trained end-to-end from scratch to integrate both information in a unified framework. Experimental results on public benchmark datasets demonstrate that the proposed architecture achieves better performance than the state-of-the-art multi-label classification model

Recently, multilabel classification algorithms play an increasingly significant role in data mining and machine learning. However, some existing mutual information-based algorithms ignore the influence of the proportions of labels on the correlation degree between features and label sets. Besides, the correlation degree of label sets cannot be accurately measured in most traditional ReliefF algorithms, and the repeated calculation arises from the division of heterogeneous neighbors. To overcome these shortcomings, this paper proposes a multilabel feature selection method using mutual information and improved multilabel ReliefF (ML-ReliefF). First, the proportion of each label is calculated in label space and combined with the mutual information of features and labels to construct a novel correlation degree between features and label sets to preprocess multilabel datasets, which is used to reduce runtime of ML-ReliefF. Second, the mutual information of label sets is introduced into improving accuracy of the correlation degree among label sets. Furthermore, two types of correlation degree for label sets based on ML-ReliefF are developed to divide similar and heterogeneous samples more clearly. Third, a divided method of heterogeneous neighbors is presented to effectively avoid the repeated calculation in ML-ReliefF, and a novel method of feature weighting based on ML-ReliefF is constructed to evaluate the importance of features. Finally, a multilabel feature selection algorithm based on mutual information and ML-ReliefF for multilabel classification is designed to improve the performance of multilabel classification. Experiments under fourteen multilabel datasets show the effectiveness of our algorithm and improve the classification performance for multilabel datasets.

For a multivariate normal set-up, it is well known that themaximumlikelihood estimator (MLE) of covariance matrix is neither admissible nor minimax under the Stein loss function. In this paper, we reveal that the MLE based on the Iwasawa parameterization leads to minimaxity with respect to the Stein loss function. Furthermore, a novel class of loss functions is proposed so that the minimum risks of the MLEs are identical in different coordinate systems, Cholesky parameterization and full Iwasawa parameterization. In other words, the MLEs based on these two different parameterizations are characterized by the property of minimaxity, without a Stein paradox. The application of our novel method to the high-dimensional covariance matrix problem is also discussed.

Over the last few years, loss functions have become increasingly important in quality assurance settings. In this paper, we develop a class of loss functions based on inversions of probability density functions and present industrial examples to illustrate process monitoring and control using economic loss instead of measurements.