Optimal classification trees

Optimal classification trees with leaf-branch and binary constraints

Margin optimal classification trees

Decision trees are a popular classification model in machine learning due to their interpretability and performance. Traditionally, decision-tree classifiers are constructed using greedy heuristic algorithms, however these algorithms do not provide guarantees on the quality of the resultant trees. Instead, a recent line of work has studied the use of exact optimization approaches for constructing optimal decision trees. Most of the recent approaches that employ exact optimization are designed for datasets with binary features. While numeric and categorical features can be transformed to binary features, this transformation can introduce a large number of binary features and may not be efficient in practice. In this work, we present a novel SAT-based encoding for decision trees that supports non-binary features and demonstrate how it can be used to solve two well-studied variants of the optimal decision tree problem. We perform an extensive empirical analysis that shows our approach obtains superior performance and is often an order of magnitude faster than the current state-of-the-art exact techniques on non-binary datasets.

Although conceptually quite simple, decision trees are still among the most popular classifiers applied to real-world problems. Their popularity is due to a number of factors – core among these is their ease of comprehension, robust performance and fast data processing capabilities. Additionally feature selection is implicit within the decision tree structure. This chapter introduces the basic ideas behind decision trees, focusing on decision trees which only consider a rule relating to a single feature at a node (therefore making recursive axis-parallel slices in feature space to form their classification boundaries). The use of particle swarm optimization (PSO) to train near optimal decision trees is discussed, and PSO is applied both in a single objective formulation (minimizing misclassification cost), and multi-objective formulation (trading off misclassification rates across classes). Empirical results are presented on popular classification data sets from the well-known UCI machine learning repository, and PSO is demonstrated as being fully capable of acting as an optimizer for trees on these problems. Results additionally support the argument that multi-objectification of a problem can improve uni-objective search in classification problems.

Decision trees are studied in rough set theory [6],[7] and test theory [1], [2], [3] and are used in different areas of applications. The complexity of optimal decision tree (a decision tree with minimal average depth) construction is very high. In the paper some conditions reducing the search are formulated. If these conditions are satisfied, an optimal decision tree for the problem is a result of simple transformation of optimal decision trees for some problems, obtained by decomposition of the initial problem. The decompostion properties are used to show that bounds given in [4] are unimprovable bounds on minimal average depth of decision tree.KeywordsDecision TreeAverage DepthTest TheoryDiagnostic ProblemTerminal VertexThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we expand the process of classification to an ensemble of fuzzy decision tree. For indoor space recognition, many research use Boosted Tree, consists of Adaboost and decision tree. The Boosted Tree extracts an optimal decision tree in stages. On each stage, Boosted Tree extracts the good decision tree by minimizing the weighted error of classification. This decision tree performs a hard decision. In most case, hard decision offer some error when they classify nearby a dividing point. Therefore, We suggest an ensemble of fuzzy decision tree, which offer some flexibility to the Boosted Tree algorithm as well as a high performance. In experimental results, we evaluate that the accuracy of suggested methods improved about 13% than the traditional one.

Optimal Decision Trees for Feature Based Parameter Tuning: Integer Programming Model and VNS Heuristic

We consider the problem of constructing optimal decision trees: given a collection of tests which can disambiguate between a set of m possible diseases, each test having a cost, and the a-priori likelihood of the patient having any particular disease, what is a good adaptive strategy to perform these tests to minimize the expected cost to identify the disease? We settle the approximability of this problem by giving a tight O(logm)-approximation algorithm. We also consider a more substantial generalization, the Adaptive TSP problem, which can be used to model switching costs between tests in the optimal decision tree problem. Given an underlying metric space, a random subset S of cities is drawn from a known distribution, but S is initially unknown to us--we get information about whether any city is in S only when we visit the city in question. What is a good adaptive way of visiting all the cities in the random subset S while minimizing the expected distance traveled? For this adaptive TSP problem, we give the first poly-logarithmic approximation, and show that this algorithm is best possible unless we can improve the approximation guarantees for the well-known group Steiner tree problem.

In this research, we consider decision trees that incorporate standard queries with one feature per query as well as hypotheses consisting of all features' values. These decision trees are used to represent knowledge and are comparable to those investigated in exact learning, in which membership queries and equivalence queries are used. As an application, we look into the issue of creating decision trees for two cases: the sorting of a sequence that contains equal elements and multiple-value decision tables which are modified from UCI Machine Learning Repository. We contrast the efficiency of several forms of optimal (considering the parameter depth) decision trees with hypotheses for the aforementioned applications. We also investigate the efficiency of decision trees built by dynamic programming and by an entropy-based greedy method. We discovered that the greedy algorithm produces very similar results compared to the results of dynamic programming algorithms. Therefore, since the dynamic programming algorithms take a long time, we may readily apply the greedy algorithms.

Following work of Stroud and Saeger (Proceedings of ISI, Springer Verlag, New York, 2006) and Anand et al. (Proceedings of Computer, Communication and Control Technologies, 2003), we formulate a port of entry inspection sequencing task as a problem of finding an optimal binary decision tree for an appropriate Boolean decision function. We report on new algorithms for finding such optimal trees that are more efficient computationally than those presented by Stroud and Saeger and Anand et al. We achieve these efficiencies through a combination of specific numerical methods for finding optimal thresholds for sensor functions and two novel binary decision tree search algorithms that operate on a space of potentially acceptable binary decision trees. The improvements enable us to analyze substantially larger applications than was previously possible. © 2011 Wiley Periodicals, Inc. Naval Research Logistics, 2011

Optimal randomized classification trees

Implementation of Improved ID3 Algorithm to Obtain more Optimal Decision Tree.

In this paper, we consider decision trees that use two types of queries: queries based on one attribute each and queries based on hypotheses about values of all attributes. Such decision trees are similar to the ones studied in exact learning, where membership and equivalence queries are allowed. We present dynamic programming algorithms for minimization of the depth and number of nodes of above decision trees and discuss results of computer experiments on various data sets and randomly generated Boolean functions. Decision trees with hypotheses generally have less complexity, i.e., they are more understandable and more suitable as a means for knowledge representation.

The rapid growth of the Internet of Things (IoT) has increased the risk of malware attacks, posing serious threats especially to micro, small, and medium enterprises (MSMEs) that often lack sufficient cybersecurity resources. This study aims to optimize Decision Tree (DT) and Random Forest (RF) classifiers using Grid Search, while addressing the class imbalance problem through the Synthetic Minority Oversampling Technique (SMOTE). The Security Attacks Malware IoT Networks dataset with five classes (Benign, Malware, DDoS, Brute Force, Scanning) was used and divided into training and testing sets with stratified 80:20 split. Experimental results show that DT achieved 67.3% accuracy with a macro F1-score of 42.9%, while RF achieved 70.7% accuracy but a very low macro F1-score of 21.4%, indicating bias toward the majority class despite balancing. Boosting methods provided stronger baselines, with XGBoost reaching 87.0% accuracy and 66.7% F1-score, while LightGBM achieved 85.6% accuracy and 64.4% F1-score. ROC curves and confusion matrices confirmed that boosting methods were more balanced in recognizing minority classes. In terms of efficiency, DT required the shortest training time (8 seconds), while LightGBM provided the best trade-off between accuracy and computational cost (26 seconds). Paired t-tests further confirmed that performance differences between DT and RF were not significant, while boosting methods significantly outperformed RF. Overall, optimizing DT and RF with Grid Search and SMOTE enhances their performance, but boosting methods remain more robust for malware detection in IoT traffic. These findings provide practical insights for MSMEs in balancing accuracy and efficiency when deploying intrusion detection systems.

An efficient algorithm for optimal pruning of decision trees