- Research Article
47
- 10.1016/j.ins.2018.03.033
Semi-supervised trees for multi-target regression
- Mar 12, 2018
- Information Sciences
- Jurica Levatić + 3 more +3
Semi-supervised trees for multi-target regression
Decision trees are one of the most widely used predictive modelling methods primarily because they are readily interpretable and fast to learn. These nice properties come at the price of predictive performance. Moreover, the standard induction of decision trees suffers from myopia: a single split is chosen in each internal node which is selected in a greedy manner; hence, the resulting tree may be sub-optimal. To address these issues, option trees have been proposed which can include several alternative splits in a new type of internal nodes called option nodes. Considering all of this, an option tree can be also regarded as a condensed representation of an ensemble. In this work, we propose to learn option trees for multi-target regression (MTR) based on the predictive clustering framework. The resulting models are thus called option predictive clustering trees (OPCTs). Multi-target regression is concerned with learning predictive models for tasks with multiple numeric target variables.We evaluate the proposed OPCTs on 11 benchmark MTR data sets. The results reveal that OPCTs achieve statistically significantly better predictive performance than a single predictive clustering tree (PCT) and are competitive with bagging and random forests of PCTs. By limiting the number of option nodes, we can achieve a good trade-off between predictive power and efficiency (model size and learning time).We also perform parameter sensitivity analysis and bias-variance decomposition of the mean squared error. Our analysis shows that OPCTs can reduce the variance of PCTs nearly as much as ensemble methods do. In terms of bias, OPCTs occasionally outperform other methods. Finally, we demonstrate the potential of OPCTs for multifaceted interpretability and illustrate the potential for inclusion of domain knowledge in the tree learning process.
Semi-supervised trees for multi-target regression
Semi-supervised trees for multi-target regression
Detection of Visual Concepts and Annotation of Images Using Ensembles of Trees for Hierarchical Multi-Label Classification
In this paper, we present a hierarchical multi-label classification system for visual concepts detection and image annotation. Hierarchical multi-label classification (HMLC) is a variant of classification where an instance may belong to multiple classes at the same time and these classes/labels are organized in a hierarchy. The system is composed of two parts: feature extraction and classification/annotation. The feature extraction part provides global and local descriptions of the images. These descriptions are then used to learn a classifier and to annotate an image with the corresponding concepts. To this end, we use predictive clustering trees (PCTs), which are able to classify target concepts that are organized in a hierarchy. Our approach to HMLC exploits the annotation hierarchy by building a single predictive clustering tree that can simultaneously predict all of the labels used to annotate an image. Moreover, we constructed ensembles (random forests) of PCTs, to improve the predictive performance. We tested our system on the image database from the ImageCLEF@ICPR 2010 photo annotation task. The extensive experiments conducted on the benchmark database show that our system has very high predictive performance and can be easily scaled to large number of visual concepts and large amounts of data.
Read moreTree-based methods for online multi-target regression
Methods that address the task of multi-target regression on data streams are relatively weakly represented in the current literature. We present several different approaches to learning trees and ensembles of trees for multi-target regression based on the Hoeffding bound. First, we introduce a local method, which learns multiple single-target trees to produce multiple predictions, which are then aggregated into a multi-target prediction. We follow with a tree-based method (iSOUP-Tree) which learns trees that predict all of the targets at once. We then introduce iSOUP-OptionTree, which extends iSOUP-Tree through the use of option nodes. We continue with ensemble methods, and describe the use of iSOUP-Tree as a base learner in the online bagging and online random forest ensemble approaches. We describe an evaluation scenario, and present and discuss the results of the described methods, most notably in terms of predictive performance and the use of computational resources. Finally, we present two case studies where we evaluate the introduced methods in terms of their efficiency and viability of application to real world domains.
Read moreThe Use of Hybrid Predictive C&RT-Logit Models in Analytical CRM
Predictive models in analytical CRM (customer relationship management) are closely related to the customer’s life cycle. Prediction of binary dependent variable refers to the most common areas such as customer acquisition, customer development (cross-selling and up-selling), and customer retention (churn analysis). While building static predictive models one usually applies decision trees, logistic regression, support vector machines or ensemble methods such as different algorithms of boosted decision trees or random forest. Recently one can observe increasing use of hybrid models in the analytical CRM, i.e. those that combine several different analytical tools, e.g. cluster analysis with decision trees, genetic algorithms with neural networks, or decision trees with logistic regression. The purpose of this paper is to compare the results obtained by using hybrid predictive CART-logit models with single decision tree models and logistic regression models. All analyses have been conducted on the basis of data sets relating to analytical CRM.
Read moreEnsemble of naive Bayes, decision tree, and random forest to predict air quality
<p>Air quality prediction is an important research issue because air quality can affect many areas of life. This study aims to predict air quality using the ensemble method and compare the results with the prediction results using a single method. The proposed ensemble method is built from three singlesupervised methods: naïve Bayes, decision trees, and random forests. The results show that the ensemble method performs better than the single methods. The ensemble method achieves the highest performance with scores of 99.89% accuracy, 79.6% precision, 79.81% recall, and 79.7% F1-score. The performance comparison between single and ensemble models is expected to provide information on the percentage increase in predictive model performance metrics from the single to ensemble methods.</p>
Read morePredictive modeling of the hot metal silicon content in blast furnace based on ensemble method
The silicon content of the hot metal is not only an important indicator of the quality of the hot metal and blast furnace (BF) operation but also reflects the level of energy utilization and the thermal state within a BF. It is important to develop an accurate prediction model for hot metal silicon content. In present study, two models for predicting hot metal silicon content are developed based on two ensemble learning methods, random forest regression (RFR) and extreme gradient boosting (XGBoost). First, Box plot was used to visualize the collected data and determine extreme outliers in the raw data. Extreme outliers are replaced with null value, and all null value are filled by linear interpolation. Secondly, feature selection is performed using recursive feature elimination. Cross-validation is performed to optimize machine learning hyperparameters and having a robust accuracy measure. Based on this, two hot metal silicon content prediction models are developed. Finally, the prediction results of the two models are compared and evaluated. The results show that both ensemble learning models show good prediction performance in predicting hot metal silicon content, but the prediction performance of the RFR model is better than that of the XGBoost model and reaching 98.77%.
Read moreRotation Forest for multi-target regression
The prediction of multiple numeric outputs at the same time is called multi-target regression (MTR), and it has gained attention during the last decades. This task is a challenging research topic in supervised learning because it poses additional difficulties to traditional single-target regression (STR), and many real-world problems involve the prediction of multiple targets at once. One of the most successful approaches to deal with MTR, although not the only one, consists in transforming the problem in several STR problems, whose outputs will be combined building up the MTR output. In this paper, the Rotation Forest ensemble method, previously proposed for single-label classification and single-target regression, is adapted to MTR tasks and tested with several regressors and data sets. Our proposal rotates the input space in an efficient and novel fashion, avoiding extra rotations forced by MTR problem decomposition. Four approaches for MTR are used: single-target (ST), stacked-single target (SST), Ensembles of Regressor Chains (ERC), and Multi-target Regression via Quantization (MRQ). For assessing the benefits of the proposal, a thorough experimentation with 28 MTR data sets and statistical tests are used, concluding that Rotation Forest, adapted by means of these approaches, outperforms other popular ensembles, such as Bagging and Random Forest.
Read moreStructure-driven induction of decision tree classifiers through neural learning
Structure-driven induction of decision tree classifiers through neural learning
Habitat modeling with single- and multi-target trees and ensembles
Habitat modeling with single- and multi-target trees and ensembles
Flood susceptibility analysis and its verification using a novel ensemble support vector machine and frequency ratio method
Flood is one of the most commonly occurred natural hazards worldwide. Severe flood occurrences in Kelantan, Malaysia cause damage to both life and property every year. Due to the huge losses in this area, development of appropriate flood modeling is required for the government. Remote sensing and geographic information system techniques can support overall flood management as they can produce rapid data collection and analysis for hydrological studies. The existing models for flood mapping have some weak points that may improve through more sophisticated and ensemble methods. The current research aimed to propose a novel ensemble method by integrating support vector machine (SVM) and frequency ratio (FR) to produce spatial modeling in flood susceptibility assessment. In the literature, mostly statistical and machine learning methods are used individually; however, their integration can enhance the final output. The FR model can perform bivariate statistical analysis and evaluate the correlation between the flooding and classes of each conditioning factors. The weights achieved by FR can be assigned to each conditioning factor and the resulted factors can be used in SVM analysis. In order to examine the efficiency of the proposed ensemble method and to show the proficiency of SVM, another machine learning algorithm such as decision tree (DT) was applied and the results were compared. To perform the methods, the upper catchment of the Kelantan basin in Malaysia was chosen. First, a flood inventory map with a total of 155 flood locations were extracted from various sources over the study area. The flood inventory map was randomly divided into two dataset; 70 % (115 flood locations) for the purpose of training and the remaining 30 % (40 flood locations) was used for validation. The spatial database included digital elevation model, curvature, geology, river, stream power index, rainfall, land use/cover, soil type, topographic wetness index and slope. For model validation, area under curve method was used and both success and prediction rate curves were calculated. The validation results for ensemble method showed 88.71 and 85.21 % for success rate and prediction rate respectively. The DT model showed 87.00 and 82.00 % for the success rate and prediction rate respectively. It is evident that the accuracies were increased using the ensemble method. The acquired results proved the efficiency of the proposed ensemble method as rapid, accurate and reasonable in flood susceptibility assessment.
Read moreReviewing ensemble classification methods in breast cancer
Reviewing ensemble classification methods in breast cancer
Development of Predictive Models for Rights Issues Using Financial Analysis Indices and Decision Tree Technique
This study focuses on predicting which firms will increase capital by issuing new stocks in the near future. Many stakeholders, including banks, credit rating agencies and investors, performs a variety of analyses for firms' growth, profitability, stability, activity, productivity, etc., and regularly report the firms' financial analysis indices. In the paper, we develop predictive models for rights issues using these financial analysis indices and data mining techniques. This study approaches to building the predictive models from the perspective of two different analyses. The first is the analysis period. We divide the analysis period into before and after the IMF financial crisis, and examine whether there is the difference between the two periods. The second is the prediction time. In order to predict when firms increase capital by issuing new stocks, the prediction time is categorized as one year, two years and three years later. Therefore Total six prediction models are developed and analyzed. In this paper, we employ the decision tree technique to build the prediction models for rights issues. The decision tree is the most widely used prediction method which builds decision trees to label or categorize cases into a set of known classes. In contrast to neural networks, logistic regression and SVM, decision tree techniques are well suited for high-dimensional applications and have strong explanation capabilities. There are well-known decision tree induction algorithms such as CHAID, CART, QUEST, C5.0, etc. Among them, we use C5.0 algorithm which is the most recently developed algorithm and yields performance better than other algorithms. We obtained data for the rights issue and financial analysis from TS2000 of Korea Listed Companies Association. A record of financial analysis data is consisted of 89 variables which include 9 growth indices, 30 profitability indices, 23 stability indices, 6 activity indices and 8 productivity indices. For the model building and test, we used 10,925 financial analysis data of total 658 listed firms. PASW Modeler 13 was used to build C5.0 decision trees for the six prediction models. Total 84 variables among financial analysis data are selected as the input variables of each model, and the rights issue status (issued or not issued) is defined as the output variable. To develop prediction models using C5.0 node (Node Options: Output type = Rule set, Use boosting = false, Cross-validate = false, Mode = Simple, Favor = Generality), we used 60% of data for model building and 40% of data for model test. The results of experimental analysis show that the prediction accuracies of data after the IMF financial crisis (59.04% to 60.43%) are about 10 percent higher than ones before IMF financial crisis (68.78% to 71.41%). These results indicate that since the IMF financial crisis, the reliability of financial analysis indices has increased and the firm intention of rights issue has been more obvious. The experiment results also show that the stability-related indices have a major impact on conducting rights issue in the case of short-term prediction. On the other hand, the long-term prediction of conducting rights issue is affected by financial analysis indices on profitability, stability, activity and productivity. All the prediction models include the industry code as one of significant variables. This means that companies in different types of industries show their different types of patterns for rights issue. We conclude that it is desirable for stakeholders to take into account stability-related indices and more various financial analysis indices for short-term prediction and long-term prediction, respectively. The current study has several limitations. First, we need to compare the differences in accuracy by using different data mining techniques such as neural networks, logistic regression and SVM. Second, we are required to develop and to evaluate new prediction models including variables which research in the theory of capital structure has mentioned about the relevance to rights issue.
Read morePredicting Survival of Patients With Rectal Neuroendocrine Tumors Using Machine Learning: A SEER-Based Population Study
Background: The number of patients diagnosed with rectal neuroendocrine tumors (R-NETs) is increasing year by year. An integrated survival predictive model is required to predict the prognosis of R-NETs. The present study is aimed at exploring epidemiological characteristics of R-NETs based on a retrospective study from the Surveillance, Epidemiology, and End Results (SEER) database and predicting survival of R-NETs with machine learning.Methods: Data of patients with R-NETs were extracted from the SEER database (2000–2017), and data were also retrospectively collected from a single medical center in China. The main outcome measure was the 5-year survival status. Risk factors affecting survival were analyzed by Cox regression analysis, and six common machine learning algorithms were chosen to build the predictive models. Data from the SEER database were divided into a training set and an internal validation set according to the year 2010 as a time point. Data from China were chosen as an external validation set. The best machine learning predictive model was compared with the American Joint Committee on Cancer (AJCC) seventh staging system to evaluate its predictive performance in the internal validation dataset and external validation dataset.Results: A total of 10,580 patients from the SEER database and 68 patients from a single medical center were included in the analysis. Age, gender, race, histologic type, tumor size, tumor number, summary stage, and surgical treatment were risk factors affecting survival status. After the adjustment of parameters and algorithms comparison, the predictive model using the eXtreme Gradient Boosting (XGBoost) algorithm had the best predictive performance in the training set [area under the curve (AUC) = 0.87, 95%CI: 0.86–0.88]. In the internal validation, the predictive ability of XGBoost was better than that of the AJCC seventh staging system (AUC: 0.90 vs. 0.78). In the external validation, the XGBoost predictive model (AUC = 0.89) performed better than the AJCC seventh staging system (AUC = 0.83).Conclusions: The XGBoost algorithm had better predictive power than the AJCC seventh staging system, which had a potential value of the clinical application.
Read moreMulti-objective Optimization for Incremental Decision Tree Learning
Decision tree learning can be roughly classified into two categories: static and incremental inductions. Static tree induction applies greedy search in splitting test for obtaining a global optimal model. Incremental tree induction constructs a decision model by analyzing data in short segments; during each segment a local optimal tree structure is formed. Very Fast Decision Tree [4] is a typical incremental tree induction based on the principle of Hoeffding bound for node-splitting test. But it does not work well under noisy data. In this paper, we propose a new incremental tree induction model called incrementally Optimized Very Fast Decision Tree (iOVFDT), which uses a multi-objective incremental optimization method. iOVFDT also integrates four classifiers at the leaf levels. The proposed incremental tree induction model is tested with a large volume of data streams contaminated with noise. Under such noisy data, we investigate how iOVFDT that represents incremental induction method working with local optimums compares to C4.5 which loads the whole dataset for building a globally optimal decision tree. Our experiment results show that iOVFDT is able to achieve similar though slightly lower accuracy, but the decision tree size and induction time are much smaller than that of C4.5.KeywordsDecision TreeClassificationIncremental OptimizationStream Mining
Read moreA new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees
A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees