On Decision Trees with Minimal Average Depth

State-of-the-art decision tree methods apply heuristics recursively to create each split in isolation, which may not capture well the underlying characteristics of the dataset. The optimal decision tree problem attempts to resolve this by creating the entire decision tree at once to achieve global optimality. In the last 25?years, algorithmic advances in integer optimization coupled with hardware improvements have resulted in an astonishing 800 billion factor speedup in mixed-integer optimization (MIO). Motivated by this speedup, we present optimal classification trees, a novel formulation of the decision tree problem using modern MIO techniques that yields the optimal decision tree for axes-aligned splits. We also show the richness of this MIO formulation by adapting it to give optimal classification trees with hyperplanes that generates optimal decision trees with multivariate splits. Synthetic tests demonstrate that these methods recover the true decision tree more closely than heuristics, refuting the notion that optimal methods overfit the training data. We comprehensively benchmark these methods on a sample of 53 datasets from the UCI machine learning repository. We establish that these MIO methods are practically solvable on real-world datasets with sizes in the 1000s, and give average absolute improvements in out-of-sample accuracy over CART of 1---2 and 3---5% for the univariate and multivariate cases, respectively. Furthermore, we identify that optimal classification trees are likely to outperform CART by 1.2---1.3% in situations where the CART accuracy is high and we have sufficient training data, while the multivariate version outperforms CART by 4---7% when the CART accuracy or dimension of the dataset is low.

Cybernetics studies information process in the context of interaction with physical systems. Because such information is sometimes vague and exhibits complex interactions; it can only be discerned using approximate representations. Machine learning provides solutions that create approximate models of information and decision trees are one of its main components. However, decision trees are susceptible to information overload and can get overly complex when a large amount of data is inputted in them. Granulation of decision tree remedies this problem by providing the essential structure of the decision tree, which can decrease its utility. To evaluate the relationship that exists between granulation and decision tree complexity, data uncertainty and prediction accuracy, the deficiencies obtained by nursing homes during annual inspections were taken as a case study. Using rough sets, three forms of granulation were performed: (1) attribute grouping, (2) removing insignificant attributes and (3) removing uncertain records. Attribute grouping significantly reduces tree complexity without having any strong effect upon data consistency and accuracy. On the other hand, removing insignificant features decrease data consistency and tree complexity, while increasing the error in prediction. Finally, decrease in the uncertainty of the dataset results in an increase in accuracy and has no impact on tree complexity.

Although conceptually quite simple, decision trees are still among the most popular classifiers applied to real-world problems. Their popularity is due to a number of factors – core among these is their ease of comprehension, robust performance and fast data processing capabilities. Additionally feature selection is implicit within the decision tree structure. This chapter introduces the basic ideas behind decision trees, focusing on decision trees which only consider a rule relating to a single feature at a node (therefore making recursive axis-parallel slices in feature space to form their classification boundaries). The use of particle swarm optimization (PSO) to train near optimal decision trees is discussed, and PSO is applied both in a single objective formulation (minimizing misclassification cost), and multi-objective formulation (trading off misclassification rates across classes). Empirical results are presented on popular classification data sets from the well-known UCI machine learning repository, and PSO is demonstrated as being fully capable of acting as an optimizer for trees on these problems. Results additionally support the argument that multi-objectification of a problem can improve uni-objective search in classification problems.

Decision trees are a popular classification model in machine learning due to their interpretability and performance. Traditionally, decision-tree classifiers are constructed using greedy heuristic algorithms, however these algorithms do not provide guarantees on the quality of the resultant trees. Instead, a recent line of work has studied the use of exact optimization approaches for constructing optimal decision trees. Most of the recent approaches that employ exact optimization are designed for datasets with binary features. While numeric and categorical features can be transformed to binary features, this transformation can introduce a large number of binary features and may not be efficient in practice. In this work, we present a novel SAT-based encoding for decision trees that supports non-binary features and demonstrate how it can be used to solve two well-studied variants of the optimal decision tree problem. We perform an extensive empirical analysis that shows our approach obtains superior performance and is often an order of magnitude faster than the current state-of-the-art exact techniques on non-binary datasets.

In this paper, we expand the process of classification to an ensemble of fuzzy decision tree. For indoor space recognition, many research use Boosted Tree, consists of Adaboost and decision tree. The Boosted Tree extracts an optimal decision tree in stages. On each stage, Boosted Tree extracts the good decision tree by minimizing the weighted error of classification. This decision tree performs a hard decision. In most case, hard decision offer some error when they classify nearby a dividing point. Therefore, We suggest an ensemble of fuzzy decision tree, which offer some flexibility to the Boosted Tree algorithm as well as a high performance. In experimental results, we evaluate that the accuracy of suggested methods improved about 13% than the traditional one.

Optimal Decision Trees for Feature Based Parameter Tuning: Integer Programming Model and VNS Heuristic

Teicoplanin 24-h loading dose regimen using a decision tree model to target serum trough concentration of 15–30 μg/mL: A retrospective study

In this research, we consider decision trees that incorporate standard queries with one feature per query as well as hypotheses consisting of all features' values. These decision trees are used to represent knowledge and are comparable to those investigated in exact learning, in which membership queries and equivalence queries are used. As an application, we look into the issue of creating decision trees for two cases: the sorting of a sequence that contains equal elements and multiple-value decision tables which are modified from UCI Machine Learning Repository. We contrast the efficiency of several forms of optimal (considering the parameter depth) decision trees with hypotheses for the aforementioned applications. We also investigate the efficiency of decision trees built by dynamic programming and by an entropy-based greedy method. We discovered that the greedy algorithm produces very similar results compared to the results of dynamic programming algorithms. Therefore, since the dynamic programming algorithms take a long time, we may readily apply the greedy algorithms.

To develop and test decision tree (DT) models to classify physical activity (PA) intensity from accelerometer output and Gross Motor Function Classification System (GMFCS) classification level in ambulatory youth with cerebral palsy (CP) and compare the classification accuracy of the new DT models to that achieved by previously published cut points for youth with CP. Youth with CP (GMFCS levels I-III) (N = 51) completed seven activity trials with increasing PA intensity while wearing a portable metabolic system and ActiGraph GT3X accelerometers. DT models were used to identify vertical axis (VA) and vector magnitude (VM) count thresholds corresponding to sedentary (SED) (<1.5 METs), light-intensity PA (LPA) (≥1.5 and <3 METs) and moderate-to-vigorous PA (MVPA) (≥3 METs). Models were trained and cross-validated using the "rpart" and "caret" packages within R. For the VA (VA_DT) and VM DT (VM_DT), a single threshold differentiated LPA from SED, whereas the threshold for differentiating MVPA from LPA decreased as the level of impairment increased. The average cross-validation accuracies for the VC_DT were 81.1%, 76.7%, and 82.9% for GMFCS levels I, II, and III. The corresponding cross-validation accuracies for the VM_DT were 80.5%, 75.6%, and 84.2%. Within each GMFCS level, the DT models achieved better PA intensity recognition than previously published cut points. The accuracy differential was greatest among GMFCS level III participants, in whom the previously published cut points misclassified 40% of the MVPA activity trials. The GMFCS-specific cut points provide more accurate assessments of MVPA levels in youth with CP across the full spectrum of ambulatory ability.

Biological databases related to medical science, containing pathological, radiological and genetic information of patients is undergoing tremendous growth, beyond our analyzing capability. However such analysis can reveal new findings about the cause and subsequent treatment of any disease. Here the genetic information of Lung Adenocarcinoma, in the form of microarray dataset has been investigated which have five different stages. Rough Set Theory (RST) has been used in analysis with an aim to effectively extract biologically relevant information, as RST is a tool that works well in an environment, heavy with inconsistent and ambiguous data, or with missing data and provides efficient algorithms for finding hidden patterns in data. The investigation has been carried out on the publicly available microarray dataset obtained from the GEO profiles at National Centre for Biotechnology Information (NCBI) website. Cross validation of the generated rule sets shows 100% accuracy. Now to extract the hidden biological dependencies between responsible genes, Decision Tree is used at consecutive two stages of cancer development to identify the main culprit genes for cancer development from one stage to another and that may lead to the drug design. The analysis revealed that hybrid Rough- Decision Tree is able to extract hidden relationships among the various genes which play an important role in causing the disease and also able to provide a unique rule set for automated medical diagnosis. Moreover at the end, the functions of the identified genes are studied and validated from Gene Ontology website DAVID which clearly shows the direct or indirect relation of genes with the cancer. This study highlights the usefulness and efficiency of RST and Decision Tree in the disease diagnosis process and its potential use in inductive learning and as a valuable aid for building more biologically significant expert systems in medical sciences

Rough set theory (RST), support vector machine (SVM), and decision tree (DT) are brightly data mining methodologies for classification prediction tasks. While the accuracy for class prediction is highly emphasized, the ability to generate rules for decision support is also important in some practical applications. Studies have shown the ability of RST for feature selection while SVM and DT are significantly on their predictive power. Moreover, the ability of DT for rule generation is an attractive function. This study intents to integrate the advantages of RST, SVM and DT approaches to develop a hybrid data mining approach to improve the performance of class prediction as well as rule generation.

Implementation of Improved ID3 Algorithm to Obtain more Optimal Decision Tree.

In this study, unsupervised and supervised machine learning techniques, principal component analysis and classification tree modelling which could be improved with additional input variables were applied on iodine oxidation voltammetric data in order to determine routes and extract information about the electrochemical conditions leading to different damaged starch ratios in flour. For this purpose a database of 3542 observations which was normalized and filtered from outliers was used. It was seen that although it was almost impossible to generalize information or determine correlations from voltammetric data at different conditions, principal component analysis indicate that on platinum electrode UCD values of 16.5 mostly seen at high potentials, optimized decision tree indicate that the impact of variables on UCD values can be ordered as current density &amp;gt; potential &amp;gt; electrode type &amp;gt; KI concentration and give routes to UCD values with high class membership leaf nodes. Therefore machine learning with decision tree modelling could open perspectives for practical and fast prediction of damaged starch ratio which would help food industry to speed up and economize costs for analysis in flour.

The purpose of the work is to investigate the use of decision tree (DT) enhanced by bootstrap aggregates (Bag) and least-squares boosting (Lsboost) in modeling the organic matter of water according to its physicochemical parameters. An entire database of 500 samples of 21 physicochemical parameters, including organic matter, was used to build the DT, DT_Bag, and DT_Lsboost models. Training data (364 data points) is resampled using a bootstrap technique to form different training datasets to train different models. The models built were validated by a dataset of 91 samples. The predicted outputs obtained from the developed DT models are then combined by simple averaging. On the other hand, the data was also boosted with the Lsboost technical aid to increase the strength of a weak learning algorithm. The model trains the first weak learner with equal weight across all data points in the training set, then trains all other weak learners based on the updated weight aimed at the validation result to minimize the squared error medium. Good agreement between the predicted and experimental organic matter concentrations for the DT_Lsboost model was obtained (the correlation coefficient for the validation dataset was 0.9992), followed by the DT-Bag model with a correlation coefficient of 0.9949. The comparison between DT, DT_Bag, and DT_Lsboost revealed the superiority of the DT_Lsboost model (the mean root of the squared errors for the dataset were 0.1295 for the DT_Lsboost, 0.1664 for the DT_Bag, and 0.5444 for the DT). These results show that Lsboost technology dramatically improved the DT model. This result is also confirmed by the results of tests on models (interpolation data of 45 points). It should also be noted that the Bag technique was also very effective in optimizing the DT model, as the results obtained with this technique were very close to the DT_Lsboost model.

In this paper, we consider decision trees that use two types of queries: queries based on one attribute each and queries based on hypotheses about values of all attributes. Such decision trees are similar to the ones studied in exact learning, where membership and equivalence queries are allowed. We present dynamic programming algorithms for minimization of the depth and number of nodes of above decision trees and discuss results of computer experiments on various data sets and randomly generated Boolean functions. Decision trees with hypotheses generally have less complexity, i.e., they are more understandable and more suitable as a means for knowledge representation.