Cost-sensitive learning and decision making revisited

Cost-sensitive matrixized classification learning with information entropy

Skewed class distribution and non-uniform misclassification cost are pervasive in many real-world domains such as bankruptcy prediction, medical diagnosis, and intrusion detection. Although class imbalance learning and cost-sensitive learning can be

Class imbalance poses a major challenge for machine learning as most supervised learning models might exhibit bias towards the majority class and under-perform in the minority class. Cost-sensitive learning tackles this problem by treating the classes differently, formulated typically via a user-defined fixed misclassification cost matrix provided as input to the learner. Such parameter tuning is a challenging task that requires domain knowledge and moreover, wrong adjustments might lead to overall predictive performance deterioration. In this work, we propose a novel cost-sensitive boosting approach for imbalanced data that dynamically adjusts the misclassification costs over the boosting rounds in response to model’s performance instead of using a fixed misclassification cost matrix. Our method, called AdaCC, is parameter-free as it relies on the cumulative behavior of the boosting model in order to adjust the misclassification costs for the next boosting round and comes with theoretical guarantees regarding the training error. Experiments on 27 real-world datasets from different domains with high class imbalance demonstrate the superiority of our method over 12 state-of-the-art cost-sensitive boosting approaches exhibiting consistent improvements in different measures, for instance, in the range of [0.3–28.56%] for AUC, [3.4–21.4%] for balanced accuracy, [4.8–45%] for gmean and [7.4–85.5%] for recall.

Cost-sensitive active learning with a label uniform distribution model

Cost-sensitive active learning through statistical methods

In many real-world applications, the costs of different errors are often unequal. Therefore, the inclusion of costs into learning, also named cost-sensitive learning, has been regarded as one of the most relevant topics of future machine learning research. Rough set theory is a powerful mathematic tool dealing with inconsistent information for attribute dependence analysis, knowledge reduction and decision rule extraction. However, it is insensitive to the costs of misclassification due to the absence of a mechanism of considering the subjective knowledge. This paper discusses problems connected with introducing the subjective knowledge into rough set learning and proposes a weighted rough set approach for cost-sensitive learning. In this method, weights are employed to represent the subjective knowledge of costs and a weighted information system is defined firstly. With the introduction of weights, weighted attribute dependence analysis is carried out and an index of weighted approximate quality is given. Furthermore, weighted attribute reduction algorithm and weighted rule extraction algorithm are designed to find the reducts and rules with the consideration of weights. Based on the proposed weighted rough set, a series of comparing experimentations with several familiar general techniques on cost-sensitive learning are constructed. The results show that the approach of weighted rough set produces averagely the minimum misclassification costs and the lowest high cost errors.KeywordsWeighted rough setknowledge reductionrule extractioncost-sensitive learning

Real-time transient stability status prediction (RTSSP) is very important to maintain the safety and stability of electrical power systems, where any unstable contingency will be likely to cause large-scale blackout. Most of machine learning methods used for RTSSP attempt to attain a low classification error, which implies that the misclassification costs of different categories are the same. However, misclassifying an unstable case as stable one usually leads to much higher costs than misclassifying a stable case as unstable one. In this paper, a new RTSSP method based on cost-sensitive extreme learning machine (CELM) is proposed, which recognizes the RTSSP as a cost-sensitive classification problem. The CELM is constructed pursuing the minimum misclassification costs, and its detailed implementation procedures for RSSTP are also researched in this work. The proposed method is implemented on the New England 39-bus electrical power system. Compared with three cost-blind methods (ELM, SVM and DT) and two cost-sensitive methods (cost-sensitive DT, cost-sensitive SVM), the simulation results have proved that the lower total misclassification costs and false dismissal rate with low computational complexity can be achieved by the proposed method, which meets the demands for the computation speed and the reliability of RTSSP.

Cost-sensitive continuous ensemble kernel learning for imbalanced data streams with concept drift

Diabetes is the fourth cause of death in the world and has some complications such as amputation, visual impairment, kidney disorder and early death. 80% of diabetes symptoms is avoidable by early diagnosis. There are standard methods to diagnose diabetes by measuring the plasma glucose concentration. However, screening all people is impossible due to financial shortages especially in developing countries. Therefore, it is proposed that the people more than 20 years old who are prone to diabetes be tested. Identifying diabetic people is possible by using different methods including machine learning algorithms. Standard machines assume balance in data and use all the available and related features to achieve lower error rates. However, in medical applications misclassification cost should be minimized as misclassification costs for healthy and patient instances are different. In addition, we are facing imbalanced data in the most medical issues including diabetes diagnosis. The features will also have different knowledge levels and costs. As a result, cost-sensitive methods should be applied in such situations. In this paper, we present and compare different cost-sensitive learning methods for diagnosis of type 2 diabetes. For evaluation of the methods two different data sets are used, which one of them is data set of Tabriz, Iran and the other one is for Pima Indian data set. By defining different scenarios, we will show that none of the studied methods has absolute superiority over the other methods and the performance of algorithms will be different due to used data set and defined scenario.

Cost-sensitive learning is popular during the process of classification. A fundamental issue in decision tree inductive learning is the attribute selection measure at each non-terminal node of the tree. However, existing literatures have not taken the trade-off between cost and benefit into account well. In this paper, we present a new strategy for attributes selection, which is a trade-off method between classification ability and cost-sensitive learning including misclassification costs and test costs with different units, for selecting splitting attributes in cost-sensitive decision trees induction. The experimental results show our method outperform the existed methods in terms of the decrease of misclassification cost.

Cost-sensitive learning methods guaranteeing privacy are becoming crucial nowadays in many applications where increasing use of sensitive personal information is observed. However, there has no optimal learning scheme developed in the literature to learn cost-sensitive classifiers under constraint of enforcing differential privacy. Our approach is to first develop a unified framework for existing cost-sensitive learning methods by incorporating the weight constant and weight functions into the classical regularized empirical risk minimization framework. Then, we propose two privacy-preserving algorithms with output perturbation and objective perturbation methods, respectively, to be integrated with the cost-sensitive learning framework. We showcase how this general framework can be used analytically by deriving the privacy-preserving cost-sensitive extensions of logistic regression and support vector machine. Experimental evidence on both synthetic and real data sets verifies that the proposed algorithms can reduce the misclassification cost effectively while satisfying the privacy requirement. A theoretical investigation is also conducted, revealing a very interesting analytic relation, i.e., that the choice of the weight constant and weight functions does not only influence the Fisher-consistent property (population minimizer of expected risk with a specific loss function leads to the Bayes optimal decision rule) but also interacts with privacy-preserving levels to affect the performance of classifiers significantly.

Research in machine learning, data mining and related areas has produced a wide variety of algorithms for cost-sensitive (CS) classification, where instead of maximizing the classification accuracy, minimizing the misclassification cost becomes the objective. However, these methods assume that training sets do not contain significant noise, which is rarely the case in real-world environments. In this paper, we systematically study the impacts of class noise on CS learning, and propose a cost-guided class noise handling algorithm to identify noise for effective CS learning. We call it cost-guided iterative classification filter (CICF), because it seamlessly integrates costs and an existing classification filter (C. Brodley and M. Friedl, 1999) for noise identification. Instead of putting equal weights to handle noise in all classes in existing efforts, CICF puts more emphasis on expensive classes, which makes it especially successful in dealing with datasets with a large cost-ratio. Experimental results and comparative studies from real-world datasets indicate that the existence of noise may seriously corrupt the performance of CS classifiers, and by adopting the proposed CICF algorithm, we can significantly reduce the misclassification cost of a CS classifier in noisy environments.

Unsolicited commercial or bulk emails or emails containing virus currently pose a great threat to the utility of email communications. A recent solution for filtering is reputation systems that can assign a value of trust to each IP address sending email messages. By analyzing the query patterns of each participating node, reputation systems can calculate a reputation score for each queried IP address and serve as a platform for global collaborative spam filtering for all participating nodes. In this research, we explore a behavioral classification approach based on spectral sender characteristics retrieved from such global messaging patterns. Due to the large amount of bad senders, this classification task has to cope with highly imbalanced data. In order to solve this challenging problem, a novel granular support vector machine - boundary alignment algorithm (GSVM-BA) is designed. GSVM-BA looks for the optima] decision boundary by repetitively removing positive support vectors from the training dataset and rebuilding another SVM. Compared to the original SVM algorithm with cost-sensitive learning, GSVM-BA demonstrates superior performance on spam IP detection, in terms of both effectiveness and efficiency

Abstract: Decision tree induction has been widely studied and applied. In safety applications, such as determining whether a chemical process is safe or whether a person has a medical condition, the cost of misclassification in one of the classes is significantly higher than in the other class. Several authors have tackled this problem by developing cost-sensitive decision tree learning algorithms or have suggested ways of changing the distribution of training examples to bias the decision tree learning process so as to take account of costs. A prerequisite for applying such algorithms is the availability of costs of misclassification. Although this may be possible for some applications, obtaining reasonable estimates of costs of misclassification is not easy in the area of safety. This paper presents a new algorithm for applications where the cost of misclassifications cannot be quantified, although the cost of misclassification in one class is known to be significantly higher than in another class. The algorithm utilizes linear discriminant analysis to identify oblique relationships between continuous attributes and then carries out an appropriate modification to ensure that the resulting tree errs on the side of safety. The algorithm is evaluated with respect to one of the best known cost-sensitive algorithms (ICET), a well-known oblique decision tree algorithm (OC1) and an algorithm that utilizes robust linear programming.

High dimensionality and class imbalance have been largely recognized as important issues in machine learning. A vast amount of literature has indeed investigated suitable approaches to address the multiple challenges that arise when dealing with high-dimensional feature spaces (where each problem instance is described by a large number of features). As well, several learning strategies have been devised to cope with the adverse effects of imbalanced class distributions, which may severely impact on the generalization ability of the induced models. Nevertheless, although both the issues have been largely studied for several years, they have mostly been addressed separately, and their combined effects are yet to be fully understood. Indeed, little research has been so far conducted to investigate which approaches might be best suited to deal with datasets that are, at the same time, high-dimensional and class-imbalanced. To make a contribution in this direction, our work presents a comparative study among different learning strategies that leverage both feature selection, to cope with high dimensionality, as well as cost-sensitive learning methods, to cope with class imbalance. Specifically, different ways of incorporating misclassification costs into the learning process have been explored. Also different feature selection heuristics have been considered, both univariate and multivariate, to comparatively evaluate their effectiveness on imbalanced data. The experiments have been conducted on three challenging benchmarks from the genomic domain, gaining interesting insight into the beneficial impact of combining feature selection and cost-sensitive learning, especially in the presence of highly skewed data distributions.