An efficient algorithm for optimal pruning of decision trees

This chapter is devoted to the study of 16 types of greedy algorithms for decision tree construction. The dynamic programming approach is used for construction of optimal decision trees. Optimization is performed relative to minimal values of average depth, depth, number of nodes, number of terminal nodes, and number of nonterminal nodes of decision trees. We compare average depth, depth, number of nodes, number of terminal nodes and number of nonterminal nodes of constructed trees with minimum values of the considered parameters obtained based on a dynamic programming approach. We report experiments performed on data sets from UCI ML Repository and randomly generated binary decision tables. As a result, for depth, average depth, and number of nodes we propose a number of good heuristics.

Decision bees have been widely and successfully used in machine learning. However, they have suffered from overfitting in noisy domains. This problem has been remedied, in C4.5 for example, by tree pruning, resulting in improved performance. More recently, fuzzy representations have been combined with decision trees. How does the performance of fuzzy decision trees compare to that of pruned decision trees? The authors propose a comparative study of pruned decision trees and fuzzy decision trees. Further, for continuous inputs, they explore different ways: (1) for selecting the granularity of the fuzzy input variables; and (2) for defining the membership functions of the fuzzy input values. We carry out an empirical study using 12 data sets. The results show that a fuzzy decision tree constructed using FID3, combined with fuzzy clustering (to build membership functions) and cluster validity (to decide on granularity), is superior to pruned decision trees.

C4.5 algorithm is used to simplify decision tree from decision table by generating decision tree from an existing decision tree. With this algorithm, knowledge base in the decision table can be simplified. This research will build a consultation program using C4.5 algorithm which is called decision tree generator. Decision tree generator provides inference facility and a user interface for consultation. The user is required to build a knowledge base first, and the application will generate user interface automatically. There are two steps in decision tree generator: firstly, the application will build the decision tree, and after that the application will build the user interface for consultation session. The results of this research show that the decision tree generator can get goal and advice from tree exploration in consultation session.

Decision tree induction is one of the widely used classification approaches. It constructs a tree in which an internal node is split based on values of a selected attribute. Depending on the attribute selected at each level of the tree, a training dataset could lead to many different trees. Consequently, it is possible that an unseen case is classified into different and conflict classes using different trees. On the other band, recently developed association role based classifications are able to generate more interesting and useful rules than decision trees. However, the large number of rules without appropriate data structure brings the issue of efficiency. In this paper, we propose to unify decision tree classification with association-based classification using generalized decision trees (GDT). GDT generalizes the concept of decision tree to encode all interesting classification rules discovered based on association in a tree form. it inherits the merits of both approaches and removes their drawbacks. The data structure and related algorithms are discussed. Results of an experimental study are presented to indicate the advantages of GDT.

Induction of decision trees as classification models through metaheuristics

In data analysis, induction of decision trees serves two main goals: first, induced decision trees can be used for classification /prediction of new instances, and second, they represent an easy-to-interpret model of the problem domain that can be used for explanation. The accuracy of the induced classifier is usually estimated using N-fold cross validation, whereas for explanation purposes a decision tree induced from all the available data is used. Decision tree learning is relatively non-robust: a small change in the training set may significantly change the structure of the induced decision tree. This paper presents a decision tree construction method in which the domain model is constructed by consensus clustering of N decision trees induced in N-fold cross-validation. Experimental results show that consensus decision trees are simpler than C4.5 decision trees, indicating that they may be a more stable approximation of the intended domain model than decision tree, constructed from the entire set of training instances.

Using decision trees as a concept description language, we examine the time complexity for learning Boolean functions with polynomial‐sized disjunctive normal form expressions when feature construction is performed on an initial decision tree containing only primitive attributes. A shortcoming of several feature‐construction algorithms found in the literature is that it is difficult to develop time complexity results for them. We illustrate a way to determine a limit on the number of features to use for building more concise trees within a standard amount of time. We introduce a practical algorithm that forms a finite number of features using a decision tree in a polynomial amount of time. We show empirically that our procedure forms many features that subsequently appear in a tree and the new features aid in producing simpler trees when concepts are being learned from certain problem domains. Expert systems developers can use a method such as this to create a knowledge base of information that contains specific knowledge in the form of If‐Then rules.

Post-pruning in decision tree induction using multiple performance measures

Unlike a univariate decision tree, a multivariate decision tree is not restricted to splits of the instance space that are orthogonal to the features' axes. This article addresses several issues for constructing multivariate decision trees: representing a multivariate test, including symbolic and numeric features, learning the coefficients of a multivariate test, selecting the features to include in a test, and pruning of multivariate decision trees. We present several new methods for forming multivariate decision trees and compare them with several well-known methods. We compare the different methods across a variety of learning tasks, in order to assess each method's ability to find concise, accurate decision trees. The results demonstrate that some multivariate methods are in general more effective than others (in the context of our experimental assumptions). In addition, the experiments confirm that allowing multivariate tests generally improves the accuracy of the resulting decision tree over a univariate tree.

Inducing decision trees with an ant colony optimization algorithm

Co-location decision tree for enhancing decision-making of pavement maintenance and rehabilitation

Structure-driven induction of decision tree classifiers through neural learning

Decision tree learning can be roughly classified into two categories: static and incremental inductions. Static tree induction applies greedy search in splitting test for obtaining a global optimal model. Incremental tree induction constructs a decision model by analyzing data in short segments; during each segment a local optimal tree structure is formed. Very Fast Decision Tree [4] is a typical incremental tree induction based on the principle of Hoeffding bound for node-splitting test. But it does not work well under noisy data. In this paper, we propose a new incremental tree induction model called incrementally Optimized Very Fast Decision Tree (iOVFDT), which uses a multi-objective incremental optimization method. iOVFDT also integrates four classifiers at the leaf levels. The proposed incremental tree induction model is tested with a large volume of data streams contaminated with noise. Under such noisy data, we investigate how iOVFDT that represents incremental induction method working with local optimums compares to C4.5 which loads the whole dataset for building a globally optimal decision tree. Our experiment results show that iOVFDT is able to achieve similar though slightly lower accuracy, but the decision tree size and induction time are much smaller than that of C4.5.KeywordsDecision TreeClassificationIncremental OptimizationStream Mining

Data mining is nontrivial extraction of implicit, previously unknown and potential useful information from the data. For a database with number of records and for a set of classes such that each record belongs to one of the given classes, the problem of classification is to decide the class to which the given record belongs. The classification problem is also to generate a model for each class from given data set. We are going to make use of supervised classification in which we have training dataset of record, and for each record the class to which it belongs is known. There are many approaches to supervised classification. Decision tree is attractive in data mining environment as they represent rules. Rules can readily expressed in natural languages and they can be even mapped o database access languages. Now a days classification based on decision trees is one of the important problems in data mining which has applications in many areas. Now a days database system have become highly distributed, and we are using many paradigms. we consider the problem of inducing decision trees in a large distributed network of highly distributed databases. The classification based on decision tree can be done on the existence of distributed databases in healthcare and in bioinformatics, human computer interaction and by the view that these databases are soon to contain large amounts of data, characterized by its high dimensionality. Current decision tree algorithms would require high communication bandwidth, memory, and they are less efficient and scalability reduces when executed on such large volume of data. So there are some approaches being developed to improve the scalability and even approaches to analyse the data distributed over a network.[keywords: Data mining, Decision tree, decision tree induction, distributed data, classification]

State-of-the-art decision tree methods apply heuristics recursively to create each split in isolation, which may not capture well the underlying characteristics of the dataset. The optimal decision tree problem attempts to resolve this by creating the entire decision tree at once to achieve global optimality. In the last 25?years, algorithmic advances in integer optimization coupled with hardware improvements have resulted in an astonishing 800 billion factor speedup in mixed-integer optimization (MIO). Motivated by this speedup, we present optimal classification trees, a novel formulation of the decision tree problem using modern MIO techniques that yields the optimal decision tree for axes-aligned splits. We also show the richness of this MIO formulation by adapting it to give optimal classification trees with hyperplanes that generates optimal decision trees with multivariate splits. Synthetic tests demonstrate that these methods recover the true decision tree more closely than heuristics, refuting the notion that optimal methods overfit the training data. We comprehensively benchmark these methods on a sample of 53 datasets from the UCI machine learning repository. We establish that these MIO methods are practically solvable on real-world datasets with sizes in the 1000s, and give average absolute improvements in out-of-sample accuracy over CART of 1---2 and 3---5% for the univariate and multivariate cases, respectively. Furthermore, we identify that optimal classification trees are likely to outperform CART by 1.2---1.3% in situations where the CART accuracy is high and we have sufficient training data, while the multivariate version outperforms CART by 4---7% when the CART accuracy or dimension of the dataset is low.