Improving GP classification performance by injection of decision trees

Large vocabulary continuous speech recognition (LVCSR) systems often use a multi-pass decoding strategy with a combination of multiple systems in the final stage. To reduce the error rate, these models must be complementary, i.e. make different errors. Previously, complementary systems have been generated by independently training a number of models, explicitly performing all combinations and picking the best performance. This method becomes infeasible as the potential number of systems increases, and does not guarantee that any of the models will be complementary. This paper presents an algorithm for generating complementary systems by altering the decision tree generation. Confusions made by a baseline system are resolved by separating confusable states, which might previously have been clustered together using the standard decision tree algorithm. Experimental results presented on a broadcast news Mandarin task show gains when combining the baseline with a complementary directed decision tree system.

We apply multiparty computation (MPC) techniques to show, given a database that is secret-shared among multiple mutually distrustful parties, how the parties may obliviously construct a decision tree based on the secret data. We consider data with continuous attributes (i.e., coming from a large domain), and develop a secure version of a learning algorithm similar to the C4.5 or CART algorithms. Previous MPC-based work only focused on decision tree learning with discrete attributes (De Hoogh et al. 2014). Our starting point is to apply an existing generic MPC protocol to a standard decision tree learning algorithm, which we then optimize in several ways. We exploit the fact that even if we allow the data to have continuous values, which a priori might require fixed or floating point representations, the output of the tree learning algorithm only depends on the relative ordering of the data. By obliviously sorting the data we reduce the number of comparisons needed per node to O(N log2 N) from the naive O(N 2), where N is the number of training records in the dataset, thus making the algorithm feasible for larger datasets. This does however introduce a problem when duplicate values occur in the dataset, but we manage to overcome this problem with a relatively cheap subprotocol. We show a procedure to convert a sorting network into a permutation network of smaller complexity, resulting in a round complexity of O(log N) per layer in the tree. We implement our algorithm in the MP-SPDZ framework and benchmark our implementation for both passive and active three-party computation using arithmetic modulo 264. We apply our implementation to a large scale medical dataset of ≈ 290 000 rows using random forests, and thus demonstrate practical feasibility of using MPC for privacy-preserving machine learning based on decision trees for large datasets.

Decision Trees have been applied widely for classification in many fields such as finance, marketing, engineering, and medicine. The increased field of application, made the requirement for understanding various aspects of decision trees in deep. In addition, it is crucial to understand the different type of costs associated with the classification task in a decision tree classifier and their relationship with the classifier’s accuracy, as balancing the two is a major concern these days in many fields such as medical diagnosis. This paper introduces the concept of decision trees, presents their various areas of application in data mining, summarizes the standard decision tree algorithms, and identifies their main advantages and disadvantages. It mainly aims to clarify relationship between the classification accuracy and classification cost in decision trees.

About two-thirds of patients with minor strokes are discharged home. However, these patients may have difficulties returning to their usual living activities. To investigate the factors associated with successful home discharge, our aim was to provide a decision tree (based on clinical data) that could identify if a patient discharged home could return to pre-stroke activities and to perform an external validation of this decision tree on an independent cohort. Two cohorts of patients with minor strokes gathered from stroke registries at the Hôpital Pitié-Salpêtrière and University Hospital Bern were included in this study (n = 105 for the construction cohort coming from France; n = 100 for the second cohort coming from Switzerland). The decision tree was built using the classification and regression tree (CART) analysis on the construction cohort. It was then applied to the validation cohort. Accuracy, sensitivity, specificity, false positive, and false-negative rates were reported for both cohorts. In the construction cohort, 60 patients (57%) returned to their usual, pre-stroke level of independence. The CART analysis produced a decision tree with the Montreal Cognitive Assessment (MoCA) as the first decision point, followed by discharge NIHSS score or age, and then by the occupational status. The overall prediction accuracy to the favorable outcome was 80% in the construction cohort and reached 72% accuracy in the validation cohort. This decision tree highlighted the role of cognitive function as a crucial factor for patients to return to their usual activities after a minor stroke. The algorithm may help clinicians to tailor planning of patients' discharge.

Decision trees are a fundamental statistical learning tool for addressing classification and regression problems through a recursive partitioning approach that effectively accommodates numerical and categorical data [1, 2]. The Classification and regression tree (CART) algorithm underlies modern Boosting methodologies such as Gradient boosting machine (GBM), Extreme gradient boosting (XGBoost), and Light gradient boosting machine (LightGBM). However, the standard CART algorithm may require improvement due to its inability to learn from unlabeled data. This study proposes several modifications to incorporate test data into the training phase. Specifically, we introduce a method based on Graph-based semi-supervised learning called “Distance-based Weighting,” which calculates and removes irrelevant records from the training set to accelerate the training process and improve performance. We present Semi-supervised classification and regression tree (Semi-Cart), a new implementation of CART that constructs a decision tree using weighted training data. We evaluated its performance on thirteen datasets from various domains. Our results demonstrate that Semi-Cart outperforms standard CART methods and contributes to statistical learning.

In data analysis, induction of decision trees serves two main goals: first, induced decision trees can be used for classification /prediction of new instances, and second, they represent an easy-to-interpret model of the problem domain that can be used for explanation. The accuracy of the induced classifier is usually estimated using N-fold cross validation, whereas for explanation purposes a decision tree induced from all the available data is used. Decision tree learning is relatively non-robust: a small change in the training set may significantly change the structure of the induced decision tree. This paper presents a decision tree construction method in which the domain model is constructed by consensus clustering of N decision trees induced in N-fold cross-validation. Experimental results show that consensus decision trees are simpler than C4.5 decision trees, indicating that they may be a more stable approximation of the intended domain model than decision tree, constructed from the entire set of training instances.

In uses of pre-trained machine learning models, it is a known issue that the target population in which the model is being deployed may not have been reflected in the source population with which the model was trained. This can result in a biased model when deployed, leading to a reduction in model performance. One risk is that, as the population changes, certain demographic groups will be under-served or otherwise disadvantaged by the model, even as they become more represented in the target population. The field of domain adaptation proposes techniques for a situation where label data for the target population does not exist, but some information about the target distribution does exist. In this paper we contribute to the domain adaptation literature by introducing domain-adaptive decision trees (DADT). We focus on decision trees given their growing popularity due to their interpretability and performance relative to other more complex models. With DADT we aim to improve the accuracy of models trained in a source domain (or training data) that differs from the target domain (or test data). We propose an in-processing step that adjusts the information gain split criterion with outside information corresponding to the distribution of the target population. We demonstrate DADT on real data and find that it improves accuracy over a standard decision tree when testing in a shifted target population. We also study the change in fairness under demographic parity and equal opportunity. Results show an improvement in fairness with the use of DADT.

When performing predictive modeling, the key criterion is always accuracy. With this in mind, complex techniques like neural networks or ensembles are normally used, resulting in opaque models impossible to interpret. When models need to be comprehensible, accuracy is often sacrificed by using simpler techniques directly producing transparent models; a tradeoff termed the accuracy vs. comprehensibility tradeoff. In order to reduce this tradeoff, the opaque model can be transformed into another, interpretable, model; an activity termed rule extraction. In this paper, it is argued that rule extraction algorithms should gain from using oracle data; i.e. test set instances, together with corresponding predictions from the opaque model. The experiments, using 17 publicly available data sets, clearly show that rules extracted using only oracle data were significantly more accurate than both rules extracted by the same algorithm, using training data, and standard decision tree algorithms. In addition, the same rules were also significantly more compact; thus providing better comprehensibility. The overall implication is that rules extracted in this fashion explain the predictions made on novel data better than rules extracted in the standard way; i.e. using training data only

The objective of this work is to exploit the potential of latest pattern recognition techniques in power quality applications. This paper presents a novel hybrid pattern recognizer for classification of power quality disturbances. The hybrid learning methodology integrates a multiobjective genetic algorithm (GA) and decision trees (CART) in order to evolve optimal subsets of discriminatory features for robust pattern classification. In the training phase the multiobjective GA based on the wrapper approach is used to find a subset of relevant attributes that minimizes both classification error rate and size of the tree discovered by the classification algorithm, namely CART, using the Pareto dominance approach. For a given feature subset, CART is invoked to produce a decision tree; the classification error and the complexity of the decision tree are used as the fitness functions by the GA to evolve better subsets. Experimental results reveal that the proposed multiobjective GA-CART combined approach yields improved classification performance and reduced classification time as compared to standard CART decision trees.

We investigate the problem of learning Boolean functions with a short DNF representation using decision trees as a concept description language. Unfortunately, Boolean concepts with a short description may not have a small decision tree representation when the tests at the nodes are limited to the primitive attributes. This representational shortcoming may be overcome by using Boolean features at the decision nodes. We present two new methods that adaptively introduce relevant features while learning a decision tree from examples. We show empirically that these methods outperform a standard decision tree algorithm for learning small random DNF functions when the examples are drawn at random from the uniform distribution.

Learning from structured data is becoming increasingly important. Besides the well-known approaches which deal directly with complex data representations (inductive logic programming and multi-relational data mining), new techniques have been recently proposed by upgrading propositional learning algorithms. Focusing on distance-based methods, these techniques are extended by incorporating similarity functions defined over structured domains, for instance a k-NN algorithm solving a graph classification problem. Since a measure between objects is the essential component for this kind of methods, this paper starts with a description of some of the recent similarity functions defined over common structured data (lists, sets, terms, etc.). However, many of the most common classification techniques, such as decision tree learning, are not distance-based methods or cannot be directly adapted to be so (as kernel methods and neural networks have been adapted). In this work, we extend decision trees to use any kind of similarity function. The method is inspired by centre splitting, which constructs decision trees by defining splits based on the distance to two or more centroids. We include an experimental analysis with both propositional data and complex data. Apart from the advantages of the new proposed method, it can be used as an example of how other partition-based methods can be adapted to deal with distances and, hence, with structured data.

ABSTRACTIn order to achieve the accurate estimation of state of charge (SOC) of the battery in a hybrid electric vehicle (HEV), this paper proposed a new estimation model based on the classification and regression tree (CART) which belongs to a kind of decision tree. The basic principle and modelling process of the CART decision tree were introduced in detail in this paper, and we used the voltage, current, and temperature of the battery in an HEV to estimate the value of SOC under the driving cycle. Meanwhile, we took the energy feedback of the HEV under the regenerative braking into consideration. The simulation data and experimental data were used to test the effectiveness of the estimation model of CART, and the results indicate that the proposed estimation model has high accuracy, the relative error of simulation is within 0.035, while the relative error of experiment is less than 0.05.

Motor bearing state directly affects the reliability of motor operation. The decision tree algorithm has strong anti-noise and adaptability, which is widely used in motor bearing fault diagnosis. This paper proposes constructing many decision trees with similar structures by using the classification and regression tree (CART) algorithm to diagnose the fault of motor bearing. To verify its generality, effectiveness, and anti-interference under the actual operation of the motor, we add noises on the bearing dataset obtained from the Case Western Reserve University and use it to verify the effect of our method. Firstly, we construct feature vectors by the time domain and frequency domain feature extraction method. Then we use the new feature vector dataset to train and test the CART-based diagnosis model. The experimental results show that the proposed algorithm is suitable for motor bearing diagnosis under a noisy environment with high accuracy.

A segmentation method for processing greenhouse vegetable foliar disease symptom images

Decision tree learning through a Predictive Model for Student Academic Performance in Intelligent M-Learning environments