Vanilla Gradient Descent for Oblique Decision Trees

We present methods for learning and pruning oblique decision trees. We propose a new function for evaluating different split rules at each node while growing the decision tree. Unlike the other evaluation functions currently used in the literature (which are all based on some notion of purity of a node), this new evaluation function is based on the concept of degree of linear separability. We adopt a correlation based optimization technique called the Alopex algorithm (K.P. Unnikrishnaan and K.P. Venugopal, 1994) for finding the split rule that optimizes our evaluation function at each node. The algorithm we present is applicable only for 2-class problems. Through empirical studies, we demonstrate that our algorithm learns good compact decision trees. We suggest a representation scheme for oblique decision trees that makes explicit the fact that an oblique decision tree represents each class as a union of convex sets bounded by hyperplanes in the feature space. Using this representation, we present a new pruning technique. Unlike other pruning techniques, which generally replace heuristically selected subtrees of the original tree by leaves, our method can radically restructure the decision tree. Through empirical investigation, we demonstrate the effectiveness of our method.

We are investigating the combination of wavelets and decision trees to detect ships and other maritime surveillance targets from medium resolution SAR images. Wavelets have inherent advantages to extract image descriptors while decision trees are able to handle different data sources. In addition, our work aims to consider oceanic features such as ship wakes and ocean spills. In this incipient work, Haar and Cohen-Daubechies-Feauveau 9/7 wavelets obtain detailed descriptors from targets and ocean features and are inserted with other statistical parameters and wavelets into an oblique decision tree.

Bio-informatics data sets may be large in the number of examples and/or the number of features. Predicting the secondary structure of proteins from amino acid sequences is one example of high dimensional data for which large training sets exist. The data from the KDD Cup 2001 on the binding of compounds to thrombin is another example of a very high dimensional data set. This type of data set can require significant computing resources to train a neural network. In general, decision trees will require much less training time than neural networks. There have been a number of studies on the advantages of decision trees relative to neural networks for specific data sets. There are often statistically significant, though typically not very large, differences. Here, we examine one case in which a neural network greatly outperforms a decision tree; predicting the secondary structure of proteins. The hypothesis that the neural network learns important features of the data through its hidden units is explored by a using a neural network to transform data for decision tree training. Experiments show that this explains some of the performance difference, but not all. Ensembles of decision trees are compared with a single neural network. It is our conclusion that the problem of protein secondary structure prediction exhibits some characteristics that are fundamentally better exploited by a neural network model.

This paper proposes several IP cores for the hardware implementation of the complete decision tree inference algorithm. Evolving decision trees in hardware is motivated by the significant improvement in the evolution time compared to the time needed for software evolution. Several architectures for the hardware evolution of single oblique or nonlinear decision trees are presented. The proposed architectures are suitable for the implementation using both Field Programmable Gate Arrays (FPGA) and Application Specific Integrated Circuits (ASIC).

Introduction: Machine learning has emerged as a powerful tool for data analysis, enabling systems to identify patterns and make predictions without explicit programming. Among its various approaches, unsupervised learning plays a crucial role in discovering hidden structures within data, especially in scenarios where labeled examples are scarce or costly to obtain. This study provides a comprehensive analysis of unsupervised learning techniques, with a particular focus on clustering and reinforcement learning. Research significance: This study provides an in-depth exploration of unsupervised learning techniques, emphasizing their ability to identify patterns in data without the need for labeled training examples. This is particularly significant in domains where acquiring labeled data is costly or impractical. By highlighting the role of reinforcement learning in unsupervised systems, the research advances the understanding of how agents improve behavior through rewards and penalties, which has implications for robotics and strategic game-playing applications. Methodology: Other options include K-Nearest Neighbors (KNN), Neural Networks, Support Vector Machines (SVM), and Decision Trees. Assessment Criteria: Memory Usage, Accuracy, Training Speed, and Error Rate. Result: According to the results, K-Nearest Neighbors (KNN) had the lowest quality, while neural networks had the highest quality. Conclusion: According to the GRA approach, neural networks are the most valuable datasets for machine learning algorithms. Key words: Unsupervised Learning, Reinforcement Learning, Clustering, Generalization & Over fitting, Decision Trees & Random Forests, Neural Networks & Deep Learning, Ensemble Methods, and Medical Imaging & Cyber security.

This article describes a new system for induction ofoblique decision trees. This system, OC1, combines deterministic hill-climbing with two forms of randomization to find a goodoblique split (in the form of a hyperplane) at each node of a decisiontree. Oblique decision tree methods are tuned especially for domains in which the attributes are numeric, although they can be adapted to symbolic or mixed symbolic/numeric attributes. We presentextensive empirical studies, using both real and artificial data, thatanalyze OC1's ability to construct oblique trees that are smaller and more accurate than their axis-parallel counterparts. We also examinethe benefits of randomization for the construction of oblique decisiontrees.

The decision tree-based neural network is introduced to combine neural networks with decision trees. Its key idea is using decision trees to form the structure of the neural network: to construct a decision tree and then convert the tree into a neural network. The decision tree and neural network for classification are similar and have equivalent property. Thereby a decision tree can be used to provide a systematic design method of neural network. We propose a new mapping between decision tree and neural network that can accurately specify the number of units, layers, connection and initial setting of parameters of neural network. Furthermore, we quote two theorems to show that the mapping is reasonable. In this paper, we use the decision tree-based neural network to solve the XOR problem.

Background: Recently, readmission rate, one of the indicators that reflect quality of hospital care, has been recieved much attention. Readmission refers to being hospitalized again after being discharged and is significant for two reasons: low quality and high cost of health care. This study aims to explore risk factors for readmission using three statistical models, and select the most successful model by comparing among them. Methods: Models were constructed with three methods, i.e. logistic regression, decision tree and neural network, in order to predict the risk of readmission within 28 days after discharge. Then the models were compared and evaluated in light of their misclassification rate, root asymptotic standard error, lift chart, and receiver operating characteristic curve. Inpatients database including 11951 patients of one university hospital was used. Results: From a comparison of the root ASE to evaluate the predictive power of the three models, the decision tree showed the highest predictive power (where regression=0.385, decision tree=0.369 and neural network=0.383). In the comparison of the misclassification rate, the decision tree also showed the highest predictive power (where regression=0.217, decision tree=0.177 and neural network= 0.211). The lift chart and ROC curve, which are widely used to evaluate a given model’s predictive power, were also used, and from the results, both the lift chart and ROC curve found the de decision tree to have stronger predictive power. Thus, from the model comparison, the decision tree was selected as the most successful model. According to this model, the risk of readmission was higher when the length of stay (LOS) was less than 2 days, route of admission was through the out-patient department (OPD), medical department was in internal medicine, 10th revision of the International Classification of Diseases code was neoplasm, LOS was relatively shorter, and the frequency of OPD visit was greater. Conclusions: This study introduces how to make and select best prediction model for readmission in order to explore the risk factors associated with readmission. According to present results, when a patient is to be discharged within 2 days, the appropriateness of discharge should be considered, with special concern of undiscovered complications and co-morbidities. In particular, if the patient is admitted through the OPD, any suspected disease should be appropriately examined and prompt outcomes of tests should be secured. Moreover, for patients of internal medicine practitioners, co-morbidity and complications caused by chronic illness should be given greater attention. International Digital Health and Care Congress, The King’s Fund, London, September 10-12 2014. International Journal of Integrated Care – Volume 14, 01 November – URN:NBN:NL:UI:10-1-116472– http://www.ijic.org/

Convolutional neural networks (CNNs) have shown vulnerable to adversarial examples which contain imperceptible perturbations. In this paper, we propose an approach to defend against adversarial examples with soft decision trees embedding. Firstly, we extract the semantic features of adversarial examples with a feature extraction network. Then, a specific soft decision tree is trained and embedded to select the key semantic features for each feature map from convolutional layers and the selected features are fed to a light-weight classification network. To this end, we use the probability distributions of each tree node to quantify the semantic features. In this way, some small perturbations can be effectively removed and the selected features are more discriminative in identifying adversarial examples. Moreover, the influence of adversarial perturbations on classification can be reduced by migrating the interpretability of soft decision trees into the black-box neural networks. We conduct experiments to defend the state-of-the-art adversarial attacks. The experimental results demonstrate that our proposed approach can effectively defend against these attacks and improve the robustness of deep neural networks.

This chapter aims to introduce several machine learning models based on a tree structure called the decision tree, which are widely believed to be among the most popular methods for both classification and regression tasks. The basic structure and concepts as well as the tree-growing algorithms of a single decision tree are first introduced. As a single decision tree is prone to over-fitting, ensemble models consisting of a certain number of decision trees are developed. Random forest-based on bagging and gradient boosting decision trees based on boosting will be introduced as the representatives of ensemble models on decision trees.

Policy sharing between multiple mobile robots using decision trees

DTOF-ANN: An Artificial Neural Network phishing detection model based on Decision Tree and Optimal Features

This study focuses on predicting which firms will increase capital by issuing new stocks in the near future. Many stakeholders, including banks, credit rating agencies and investors, performs a variety of analyses for firms' growth, profitability, stability, activity, productivity, etc., and regularly report the firms' financial analysis indices. In the paper, we develop predictive models for rights issues using these financial analysis indices and data mining techniques. This study approaches to building the predictive models from the perspective of two different analyses. The first is the analysis period. We divide the analysis period into before and after the IMF financial crisis, and examine whether there is the difference between the two periods. The second is the prediction time. In order to predict when firms increase capital by issuing new stocks, the prediction time is categorized as one year, two years and three years later. Therefore Total six prediction models are developed and analyzed. In this paper, we employ the decision tree technique to build the prediction models for rights issues. The decision tree is the most widely used prediction method which builds decision trees to label or categorize cases into a set of known classes. In contrast to neural networks, logistic regression and SVM, decision tree techniques are well suited for high-dimensional applications and have strong explanation capabilities. There are well-known decision tree induction algorithms such as CHAID, CART, QUEST, C5.0, etc. Among them, we use C5.0 algorithm which is the most recently developed algorithm and yields performance better than other algorithms. We obtained data for the rights issue and financial analysis from TS2000 of Korea Listed Companies Association. A record of financial analysis data is consisted of 89 variables which include 9 growth indices, 30 profitability indices, 23 stability indices, 6 activity indices and 8 productivity indices. For the model building and test, we used 10,925 financial analysis data of total 658 listed firms. PASW Modeler 13 was used to build C5.0 decision trees for the six prediction models. Total 84 variables among financial analysis data are selected as the input variables of each model, and the rights issue status (issued or not issued) is defined as the output variable. To develop prediction models using C5.0 node (Node Options: Output type = Rule set, Use boosting = false, Cross-validate = false, Mode = Simple, Favor = Generality), we used 60% of data for model building and 40% of data for model test. The results of experimental analysis show that the prediction accuracies of data after the IMF financial crisis (59.04% to 60.43%) are about 10 percent higher than ones before IMF financial crisis (68.78% to 71.41%). These results indicate that since the IMF financial crisis, the reliability of financial analysis indices has increased and the firm intention of rights issue has been more obvious. The experiment results also show that the stability-related indices have a major impact on conducting rights issue in the case of short-term prediction. On the other hand, the long-term prediction of conducting rights issue is affected by financial analysis indices on profitability, stability, activity and productivity. All the prediction models include the industry code as one of significant variables. This means that companies in different types of industries show their different types of patterns for rights issue. We conclude that it is desirable for stakeholders to take into account stability-related indices and more various financial analysis indices for short-term prediction and long-term prediction, respectively. The current study has several limitations. First, we need to compare the differences in accuracy by using different data mining techniques such as neural networks, logistic regression and SVM. Second, we are required to develop and to evaluate new prediction models including variables which research in the theory of capital structure has mentioned about the relevance to rights issue.

In this paper we enhance the notion of anomaly detection and use both neural network (NN) and decision tree (DT) for intrusion detection. While DTs are highly successful in detecting known attacks, NNs are more interesting to detect new attacks. In our method we proposed a new approach to design the system using both DT and combination of unsupervised and supervised NN for Intrusion Detection System (IDS). By applying DT known attacks would be recognized with a quick execution time. Unknown attacks would be detected by applying the unsupervised NN based on hybrid of Self Organizing Map (SOM) for clustering attacks into smaller categories and supervised NN based on Backpropagation for detailed clustering.

Decision trees are versatile machine learning algorithms that are frequently used in classification and regression tasks. In this work, decision trees are employed as tools for sediment classification using sound waves propagating in the ocean and their behavior and features. We first analyze the structure of received time series at deployed hydrophones, extracting characteristic features (kurtosis and skewness, for example). Feature values then form vectors that are used as input patterns to the trees. A training step is the first stage of the machine learning approach with the trees trained to recognize sediment types based on feature values. The method is subsequently tested on feature vectors obtained from noise-corrupted time series. The performance depends on Signal-to-Noise Ratio values as expected and the method is found to be superior to conventional machine learning approaches. The addition of tools such as principal component analysis as well as spectrogram processing and time-frequency curve fitting further enhances the method. The decision tree technique provides an effective and efficient solution to the problem of sediment classification using acoustic data. [Work supported by ONR.]