Optimal Decision Trees for Feature Based Parameter Tuning: Integer Programming Model and VNS Heuristic

State-of-the-art decision tree methods apply heuristics recursively to create each split in isolation, which may not capture well the underlying characteristics of the dataset. The optimal decision tree problem attempts to resolve this by creating the entire decision tree at once to achieve global optimality. In the last 25?years, algorithmic advances in integer optimization coupled with hardware improvements have resulted in an astonishing 800 billion factor speedup in mixed-integer optimization (MIO). Motivated by this speedup, we present optimal classification trees, a novel formulation of the decision tree problem using modern MIO techniques that yields the optimal decision tree for axes-aligned splits. We also show the richness of this MIO formulation by adapting it to give optimal classification trees with hyperplanes that generates optimal decision trees with multivariate splits. Synthetic tests demonstrate that these methods recover the true decision tree more closely than heuristics, refuting the notion that optimal methods overfit the training data. We comprehensively benchmark these methods on a sample of 53 datasets from the UCI machine learning repository. We establish that these MIO methods are practically solvable on real-world datasets with sizes in the 1000s, and give average absolute improvements in out-of-sample accuracy over CART of 1---2 and 3---5% for the univariate and multivariate cases, respectively. Furthermore, we identify that optimal classification trees are likely to outperform CART by 1.2---1.3% in situations where the CART accuracy is high and we have sufficient training data, while the multivariate version outperforms CART by 4---7% when the CART accuracy or dimension of the dataset is low.

The purpose of this work is to investigate some of the algorithms used for representing switching functions by space efficient decision diagrams. This representation is an optimization problem. Adopting the terminology from combinatorial optimization problems, we have:Problem Instance: A multivalued function f(x) where x is a vector, and a storing cost, s(xi), associated with each variable xi in x.Feasible Solution: A decision diagram D (acyclic, directed graph with indegrees of the internal nodes possibly exeeding one) representing f(x).Objective Function: The total storage cost,S(D) = Σ s(xi) for each feasible solution D.xie DOptimal Solution: A decision diagram D representing the function with minimum total storage cost S(D), i.e. a feasible solution which minimizes the objective function.To date, only two algorithms for the construction of optimal decision diagrams with respect to the total storage cost have been developed; Reinwald & Soland's branch & bound algorithm and Thayse's, based on the extension of Shannon's expansion formula. The second algorithm is analyzed and proven to be highly exponential in the number of variables. This comes as no surprise since, in most cases, the construction of optimal decision trees is an NP-hard problem. The construction of decision diagrams is generally more complicated than that of trees. One cannot seriously doubt that the constructions of optimal diagrams is also NP-hard for most costs including the total storage cost. Therefore, a heuristic approach to this problem is needed. Two heuristics, based on the greedy algorithmic technique, are presented. These heuristics with the analysis of Thayse's algorithm constitute the major achievements of this work.

Although conceptually quite simple, decision trees are still among the most popular classifiers applied to real-world problems. Their popularity is due to a number of factors – core among these is their ease of comprehension, robust performance and fast data processing capabilities. Additionally feature selection is implicit within the decision tree structure. This chapter introduces the basic ideas behind decision trees, focusing on decision trees which only consider a rule relating to a single feature at a node (therefore making recursive axis-parallel slices in feature space to form their classification boundaries). The use of particle swarm optimization (PSO) to train near optimal decision trees is discussed, and PSO is applied both in a single objective formulation (minimizing misclassification cost), and multi-objective formulation (trading off misclassification rates across classes). Empirical results are presented on popular classification data sets from the well-known UCI machine learning repository, and PSO is demonstrated as being fully capable of acting as an optimizer for trees on these problems. Results additionally support the argument that multi-objectification of a problem can improve uni-objective search in classification problems.

Decision trees are a popular classification model in machine learning due to their interpretability and performance. Traditionally, decision-tree classifiers are constructed using greedy heuristic algorithms, however these algorithms do not provide guarantees on the quality of the resultant trees. Instead, a recent line of work has studied the use of exact optimization approaches for constructing optimal decision trees. Most of the recent approaches that employ exact optimization are designed for datasets with binary features. While numeric and categorical features can be transformed to binary features, this transformation can introduce a large number of binary features and may not be efficient in practice. In this work, we present a novel SAT-based encoding for decision trees that supports non-binary features and demonstrate how it can be used to solve two well-studied variants of the optimal decision tree problem. We perform an extensive empirical analysis that shows our approach obtains superior performance and is often an order of magnitude faster than the current state-of-the-art exact techniques on non-binary datasets.

Real-time power optimization for application server clusters based on Mixed-Integer Programming

Decision trees are studied in rough set theory [6],[7] and test theory [1], [2], [3] and are used in different areas of applications. The complexity of optimal decision tree (a decision tree with minimal average depth) construction is very high. In the paper some conditions reducing the search are formulated. If these conditions are satisfied, an optimal decision tree for the problem is a result of simple transformation of optimal decision trees for some problems, obtained by decomposition of the initial problem. The decompostion properties are used to show that bounds given in [4] are unimprovable bounds on minimal average depth of decision tree.KeywordsDecision TreeAverage DepthTest TheoryDiagnostic ProblemTerminal VertexThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we expand the process of classification to an ensemble of fuzzy decision tree. For indoor space recognition, many research use Boosted Tree, consists of Adaboost and decision tree. The Boosted Tree extracts an optimal decision tree in stages. On each stage, Boosted Tree extracts the good decision tree by minimizing the weighted error of classification. This decision tree performs a hard decision. In most case, hard decision offer some error when they classify nearby a dividing point. Therefore, We suggest an ensemble of fuzzy decision tree, which offer some flexibility to the Boosted Tree algorithm as well as a high performance. In experimental results, we evaluate that the accuracy of suggested methods improved about 13% than the traditional one.

This chapter is devoted to the study of 16 types of greedy algorithms for decision tree construction. The dynamic programming approach is used for construction of optimal decision trees. Optimization is performed relative to minimal values of average depth, depth, number of nodes, number of terminal nodes, and number of nonterminal nodes of decision trees. We compare average depth, depth, number of nodes, number of terminal nodes and number of nonterminal nodes of constructed trees with minimum values of the considered parameters obtained based on a dynamic programming approach. We report experiments performed on data sets from UCI ML Repository and randomly generated binary decision tables. As a result, for depth, average depth, and number of nodes we propose a number of good heuristics.

The rapid growth of the Internet of Things (IoT) has increased the risk of malware attacks, posing serious threats especially to micro, small, and medium enterprises (MSMEs) that often lack sufficient cybersecurity resources. This study aims to optimize Decision Tree (DT) and Random Forest (RF) classifiers using Grid Search, while addressing the class imbalance problem through the Synthetic Minority Oversampling Technique (SMOTE). The Security Attacks Malware IoT Networks dataset with five classes (Benign, Malware, DDoS, Brute Force, Scanning) was used and divided into training and testing sets with stratified 80:20 split. Experimental results show that DT achieved 67.3% accuracy with a macro F1-score of 42.9%, while RF achieved 70.7% accuracy but a very low macro F1-score of 21.4%, indicating bias toward the majority class despite balancing. Boosting methods provided stronger baselines, with XGBoost reaching 87.0% accuracy and 66.7% F1-score, while LightGBM achieved 85.6% accuracy and 64.4% F1-score. ROC curves and confusion matrices confirmed that boosting methods were more balanced in recognizing minority classes. In terms of efficiency, DT required the shortest training time (8 seconds), while LightGBM provided the best trade-off between accuracy and computational cost (26 seconds). Paired t-tests further confirmed that performance differences between DT and RF were not significant, while boosting methods significantly outperformed RF. Overall, optimizing DT and RF with Grid Search and SMOTE enhances their performance, but boosting methods remain more robust for malware detection in IoT traffic. These findings provide practical insights for MSMEs in balancing accuracy and efficiency when deploying intrusion detection systems.

The construction of decision trees is an efficient tool for inductive learning, and fuzzy decision trees are particularly interesting because they enable the user to take into account imprecise descriptions of the cases, or heterogeneous values (symbolic, numerical, or fuzzy). However, since the method to construct a fuzzy decision tree is not unique, in this paper, a comparative study is presented to point out differences between three methods. This study focus on differences between methods when ranking attributes during the construction of a fuzzy decision tree. The aim is to enable the reader to understand what kind of fuzzy decision tree is obtained by each method.

The growing availability of large amounts of multimedia contents in science and industry have made data mining applications such as data classification highly demanding. The contribution of this paper is two-fold. First, we propose an approach for constructing a decision tree based classification model for multimedia contents. Second, in order to speed up the performance of the proposed model, we propose a hybrid CPU-GPU approach for construction of decision tree on Graphic Processing Unit (GPU). Our approach not only accelerates the construction of decision tree via GPU computing, but also does so by considering the power and energy consumption of the GPU. Through the experiments, we demonstrate that the proposed hybrid CPU-GPU approach outperforms CPU-based sequential implementation by several times.

In this paper, we consider simultaneous wireless information and energy transfer in an orthogonal-frequency-division-multiple-access decode-and-forward relay network, in which an energy-constrained relay node harvests energy from a source node and uses the harvested energy to forward information to multiple destination nodes. Our objective is to maximize the end-to-end sum rate by resource allocation, subject to transmit power constraint at the source and energy-harvesting (EH) constraint at the relay. A non-convex and mixed-integer programming (MIP) problem is formulated to optimize time-switching (TS) ratios of EH and information decoding at the relay, TS ratio of information transmission from relay to destinations, subcarrier allocation as well as power allocation (PA) over all subcarriers at source and relay. We propose to decouple this problem into a convex problem and an MIP problem in fractional form. To solve the MIP problem, we transform it into an equivalent optimization problem in subtractive form which has a tractable solution. As a result, we propose a novel scheme to achieve jointly optimal TS ratios, subcarrier allocation and PA. Simulation results verify the optimality of our proposed resource allocation scheme.

An Application of Interior Point Method to 0-1 Mixed Integer Programing Problems

In this research, we consider decision trees that incorporate standard queries with one feature per query as well as hypotheses consisting of all features' values. These decision trees are used to represent knowledge and are comparable to those investigated in exact learning, in which membership queries and equivalence queries are used. As an application, we look into the issue of creating decision trees for two cases: the sorting of a sequence that contains equal elements and multiple-value decision tables which are modified from UCI Machine Learning Repository. We contrast the efficiency of several forms of optimal (considering the parameter depth) decision trees with hypotheses for the aforementioned applications. We also investigate the efficiency of decision trees built by dynamic programming and by an entropy-based greedy method. We discovered that the greedy algorithm produces very similar results compared to the results of dynamic programming algorithms. Therefore, since the dynamic programming algorithms take a long time, we may readily apply the greedy algorithms.

Centralized algorithms are widely used for optimization and control in power system applications. These algorithms require all the measurements and data to be accumulated at central location and hence suffer from single-point-of-failure. Additionally, these algorithms lack scalability with an increasing number of sensors and actuators, specially with the increasing integration of distributed energy resources (DERs). As the power system becomes a confluence of a diverse set of decision-making entities with a multitude of objectives, the preservation of privacy and operation of the system with limited information has been a growing concern. Distributed optimization techniques solve these challenges while also ensuring resilient computational solutions for the power system operation in the presence of both natural and man-made adversaries. A detailed discussion of possible applications of distributed optimization in power systems is provided in this work. However, there exist multiple challenges for accurate and computationally efficient distributed solutions. Commonly-used distributed optimization approaches include Lagrange relaxation, augmented Lagrangian, approximate network directions in conjunction with standard Lagrangian, auxiliary problem principle, alternating directions method of multipliers, optimality condition decomposition, proximal atomic coordination, and optimal feedback-based voltage control. A comprehensive classification of the distributed optimization problems has been discussed and detailed in this work. All of these algorithms have displayed efficient identification of global optimum solutions for convex continuous distributed optimization problems. The algorithms discussed so far are predominantly used to manage continuous state variables. Inclusion of integer variables in the decision support are needed for specific power system problems. Mixed integer programming (MIP) problem arises in a power system operation and control due to tap changing transformers, capacitors and switches. The global optimization techniques for MIPs are Branch and Bound, Branch and Cut, Cutting planes, Adaptive coordinate search, Nelder-Mead, Genetic algorithm etc. Although the above optimization techniques are able to solve NP-hard convexified MIP problems centrally, but are time consuming and do not scale well for large scale distributed problems. Decomposition and solution approach of distributed coordination can resolve the scalability issue. Despite the fact that a large body of work is present on the centralized solution methods for convexified MIP problems, the literature on distributed MIPs is relatively limited. The distributed optimization algorithms applied in power network to solve MIPs are reported here. ML based solutions can help to get faster convergence for distributed optimization or can replace optimization techniques depending on the problem as discussed in this work. Finally, a summary and path forward are provided, and the advancement needed in distributed optimization for the power grid is also presented.