Accuracy vs. Cost in Decision Trees: A Survey

Abstract: The research study in this paper aims to propose advanced research methodologies for classifying sequence data, addressing a fundamental challenging task in contemporary real time applications: the categorization of sequential information. Traditional classification approaches, such as Classification and Regression Trees (CART) and decision tree algorithms like C4.5, have been widely utilized in many real-time applications due to their attractive features such as classification accuracy, scalability, simplicity, interpretability, and general applicability. These attributes have solidified decision trees as benchmark models in both data mining and machine learning applications. However, standard C4.5 and CART are inherently limited to processing fixed-length vector data and are incapable of handling more complex data structures such as sequences, trees, graphs and special structures. To facilitate efficient and effective classification of sequence data, there is an increasing demand for cutting-edge, autonomous, highly intelligent models capable of real-time analysis. This particular study introduces four novel methodologies for decision tree-based sequence data classification: Global Energy Technique-1 (GET-1), Maximum Minus Minimum Energy Technique-2 (MMET-2), Maximum Energy Technique-3 (MET-3), and Square Maximum Energy Technique-4 (SMET-4). These special techniques leverage the concept of data energy or information value of data, which is analogous to its informational value, to enhance classification accuracy and efficiency.

Machine learning based algorithms, such as a decision tree (DT) classifier, have been applied to automated process monitoring and fault identification in manufacturing processes, however the current DT-based process control models employ a single DT classifier for both mean shift detection and fault identification. As many manufacturing processes use automated data collection for multiple process parameters, a DT classifier would have to handle a large number of classes. Previous research shows that a large number of classes can degrade the accuracy of a DT multiclass classifier. In this study we propose a new process monitoring model using multiple DT classifiers with each handling a small number of classes. Moreover, we not only detect mean shifts but also identify process variability levels that may cause out-of-control signals. Experimental results show that our proposed model achieves satisfactory performance in process monitoring and fault identification with various parameter settings. It achieves better ARL performance compared with the baseline method based on a single DT classifier.

Aim: The aim of the work is to evaluate the accuracy and precision in predicting heart disease using Support Vector Machine (SVM), Random forest (RF), Logistic Regression (LR), Decision Tree (DT) Classification algorithms. Materials and Methods: Classification algorithm is appealed on a heart dataset which consists of 180 records. A framework for heart disease prediction in the medical sector comparing Random forest, Logistic Regression, Decision Tree and SVM classifiers has been proposed and developed. The sample size was calculated as 55 in each group using G power 80%. Sample size was calculated using clincalc analysis, with alpha and beta values 0.05 and 0.5, 95% confidence, pretest power 80% and enrolment ratio 1. Results: The Novel Random Forest Algorithm (92.13%), Support Vector Machine (62.51%), Logistic Regression (84.89%), Decision Tree (86.25%) classifiers produce respectively. SVM, RF exists a statistically significant difference between the two groups (p=0.001,p=.004;p<0.05).LR, RF exists a statistically insignificant difference between the two groups (p=.103, P=.080;p>0.05) both with confidence interval 95%. Hence Random forest is better than SVM, RF, DT classifiers. Conclusion: The results show that the performance of RF is better when compared with SVM, LR and DT in terms of both precision and accuracy.

In this paper, we have compared the classifier algorithms including C4.5 decision tree, le artificial neural network (ANN), artificial immune recognition system (AIRS), and adaptive neuro-fuzzy inference system (ANFIS) in the diagnosis of obstructive sleep apnea syndrome (OSAS), which is an important disease that affects both the right and the left cardiac ventricle. The goal of this study was to find the best classifier model on the diagnosis of OSAS. The clinical features were obtained from Polysomnography device as a diagnostic tool for obstructive sleep apnea in patients clinically suspected of suffering this disease in this study. The clinical features are arousals index, apneahypopnea index (AHI), SaO2 minimum value in stage of rapid eye movement, and percent sleep time in stage of SaO2 intervals bigger than 89%. In our experiments, a total of 83 patients (58 with a positive OSAS (AHI>5) and 25 with a negative OSAS such that normal subjects) were examined. The decision support systems can help to physicians in the diagnosing of any disorder or disease using clues obtained from signal or images taken from subject having any disorder. In order to compare the used classifier algorithms, the mean square error, classification accuracy, area under the receiver operating characteristics curve (AUC), and sensitivity and specificity analysis have been used. The obtained AUC values of C4.5 decision tree, ANN, AIRS, and ANFIS classifiers are 0.971, 0.96, 0.96, and 0.922, respectively. These results have shown that the best classifier system is C4.5 decision tree classifier on the diagnosis of obstructive sleep apnea syndrome.

Motivation and Objectives Neuroblastoma (NB) is the most common pediatric solid tumor characterized by clinical and molecular risk factors. The mortality is about fifty percent and this makes exploration of new and more effective risk factors for improving stratification mandatory. Hypoxia is a condition of low oxygen tension occurring in poorly vascularized areas of the tumor associated with poor prognosis. We had previously defined a robust gene expression signature measuring the hypoxic component of NB tumors (NB-hypo) that is a novel, independent risk factor (Fardin et al., 2010). Integrating classical risk factors with NB-hypo could improve the stratification of NB patients. We wanted to develop a prognostic classifier of NB patients’ outcome blending existing knowledge on clinical and molecular risk factors with the prognostic NB-hypo signature. Furthermore, we were interested in the decision tree classifier that outputs explicit rules easily translated into the clinical setting. Methods A total of 182 NB patients were enrolled on the bases of availability of gene expression profile by Affymetrix GeneChip HG-U133plus2.0, clinical and molecular information. NB tumor stage was defined according to the International NB Staging System (INSS). Age at diagnosis was dichotomized as greater or equal than 1 year and less than one year. MYCN status was amplified or normal. Good and poor outcome were defined as the patient’s status alive or dead 5 years after diagnosis respectively. The risk group was assigned according to the International Neuroblastoma Risk Group (INRG) Consensus Pretreatment Classification Schema. The 182 NB patients cohort was clustered in High and low hypoxia by k-means analysis of the 62 probsets constituting the NB-hypo signature previously described to measure tumor hypoxia (Fardin et al., 2009). We utilized the k-means algorithm implemented in the WEKA software (Hall et al., 2009) setting up number of clusters to 2, 500 iterations, preserving instances order and using Manhattan distance. The classification was performed by induction of decision trees. We utilized the Weka J48 implementation of the popular C4.5 algorithm (Kotsiantis, 2007; Murthy, 1998) and we set up the following options: pruning parameter was 0.25, pruning method was sub-tree raising and minimum number of instances per leaf was 2. Each leaf of the decision tree classifier identifies a non overlapping group of patients and each decision node identifies a branch which splits the dataset. We utilized Fisher’s exact test to measure the statistical significance of groups and branches. Fisher’s test was utilized in the context of decision trees to design a top-down approach to prune out non statistically significant branches (Liu et al., 2010). For each leaf we counted the number of correctly classified (named n) and the number of incorrectly classified instances (named m). We considered the marginal totals y and ¬y which represent poor and good outcome patients respectively. We designed a 2x2 contingency table of the two possible outcomes (Good or Poor) against the number of instances included in a give leaf and the remaining instances. Application of the Fisher exact test to this table generates a p value giving the probability of observing 2x2 table, or more extreme tables, knowing the marginal totals (y, ¬y) and assuming independency among the patients in a specific leaf and those in other leaves. Results and Discussion Patients were stratified in good and poor outcome on the bases of the following risk factors: Age at diagnosis, INSS stage, MYCN status and NB-hypo. The algorithm generated a decision tree classifier composed by 3 decision nodes and 7 leaves covering 87% of non overlapping good outcome patients and 100% of non overlapping poor outcome patients. Each path from the root to a leaf utilizes some, but not all, considered risk factors. Interestingly NB-hypo was included in the decision node that stratified stage 3 tumors demonstrating its usefulness in NB patients’ stratification. The leaves classifying good outcome patients had the very low error of 2% indicating a good performance of the algorithm in predicting this class. In contrast, the classification of poor outcome patients produced the leaf with the highest error of 13%. This leaf includes stage 4 tumors that are traditionally difficult to stratify by any known risk factor. To test statistical significance of the splits performed by the algorithm and the groups of patients, we utilized the Fisher’s exact test with a confidence set at 95%. The results showed that the groups obtained at each split were statistically significant. Furthermore, analysis of single groups of patients identified by leaves demonstrated that some, but no all reached the significance threshold. The significant leaves included the NB-hypo among the utilized risk factors. These results further strengthen the value of NB-hypo in predicting patients’ outcome. Lack of significance was often associated with a rather low number of patients in the leaf. This is a limitation of the divide-and-conquer approach of the algorithm, applied to relatively small patients’ cohorts, that recurrently splits the dataset. We then assessed the concordance of the predictions with INRG risk assessment. High Risk patients were correctly included in the leaves classifying poor outcome patients and Low Risk patients mapped correctly in the good outcome leaves. Interestingly, NB-hypo generated a leaf identifying a new group of poor outcome patients, sharing the high NB-hypo, whose characteristic fell into both High and Intermediate Risk. We collected and analyzed the results of 1000 10-fold cross validations and we observed that most classifiers had 7 leaves and only 5 out of 10^4 deviated from this pattern. The recurrence of 7 leaves demonstrated the high stability of the decision tree classifier that we generated. Analysis of the pruning parameters revealed optimal performance in the range of 0.1-0.3 in line with what used in this study. The path to reach each leaf can be easily transposed into a “if…than…” rule that, in turn provides an easy readout of the classifier, precious for translating the classification into the clinical setting. In conclusion, we demonstrated that the decision tree algorithm C4.5 can derive explicit rules for NB patients stratification if classical risk factors are blended with the NB hypo signature. These rules are statistically significant and quite stable and suitable to be conveyed to the clinic to design new therapies perhaps taking hypoxia into consideration as a potential target. Acknowledgements The work was supported by the Fondazione Italiana per la Lotta al Neuroblastoma, the Associazione Italiana per la Ricerca sul Cancro, the Società Italiana Glicogenosi, the Fondazione Umberto Veronesi and the Ministero della Salute Italiano. Davide Cangelosi and Fabiola Blengio are recipients of a fellowship from the Fondazione Italiana per la Lotta al Neuroblastoma. References Fardin P, Barla A, Mosci S, Rosasco L, Verri A, Varesio L. (2009) The l1-l2 regularization framework unmasks the hypoxia signature hidden in the transcriptome of a set of heterogeneous neuroblastoma cell lines. BMC Genomics, 10:474. doi:10.1186/1471-2164-10-474 Fardin P, Barla A, Mosci S, Rosasco L, Verri A, et al. (2010) A biology-driven approach identifies the hypoxia gene signature as a predictor of the outcome of neuroblastoma patients, Journal of Molecular Cancer 9:1. 185. doi:10.1186/1476-4598-9-185 Hall M, Eibe F, Holmes G, Pfahringer B, Reutemann P, Witten IH. (2009) The WEKA Data Mining Software: An Update: SIGKDD Explorations. doi: 10.1145/1656274.1656278 Kotsiantis S.B. (2007) Supervised Machine Learning: A Review of Classification Techniques. Informatica 31:249-268. Liu W, Chawla S, Cieslak D, Chawla N. (2010) A Robust Decision Tree Algorithm for Imbalanced Data Sets. In: SDM. 766-777. Murthy S.K. (1998) Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey, Data Mining Knowledge Discovery 2:4. 345-389. doi: 10.1023/A:1009744630224

Metabolic syndrome (MS) is a condition associated with metabolic abnormalities that are characterized by central obesity (e.g. waist circumference or body mass index), hypertension (e.g. systolic or diastolic blood pressure), hyperglycemia (e.g. fasting plasma glucose) and dyslipidemia (e.g. triglyceride and high-density lipoprotein cholesterol). It is also associated with the development of diabetes mellitus (DM) type 2 and cardiovascular disease (CVD). Therefore, the rapid identification of MS is required to prevent the occurrence of such diseases. Herein, we review the utilization of data mining approaches for MS identification. Furthermore, the concept of quantitative population-health relationship (QPHR) is also presented, which can be defined as the elucidation/understanding of the relationship that exists between health parameters and health status. The QPHR modeling uses data mining techniques such as artificial neural network (ANN), support vector machine (SVM), principal component analysis (PCA), decision tree (DT), random forest (RF) and association analysis (AA) for modeling and construction of predictive models for MS characterization. The DT method has been found to outperform other data mining techniques in the identification of MS status. Moreover, the AA technique has proved useful in the discovery of in-depth as well as frequently occurring health parameters that can be used for revealing the rules of MS development. This review presents the potential benefits on the applications of data mining as a rapid identification tool for classifying MS.

Email is one of the most widely used forms of written communication over the Internet, and its use has increased tremendously for both personal and professional purposes. The increase in email traffic comes also with an increase in the use of emails for illegitimate purposes to commit all sort of crimes. Phishing, spamming, email bombing, threatening, cyber bullying, racial vilification, child pornography, viruses and malware propagation, and sexual harassments are common examples of email abuses. Terrorist groups and criminal gangs are also using email systems as a safe channel for their communication. The alarming increase in the number of cybercrime incidents using email is mostly due to the fact that email can be easily anonymized. The problem of email authorship attribution is to identify the most plausible author of an anonymous email from a group of potential suspects. Most previous contributions employed a traditional classification approach, such as decision tree and Support Vector Machine (SVM), to identify the author and studied the effects of different writing style features on the classification accuracy. However, little attention has been given on ensuring the quality of the evidence. In this work, we introduce an innovative data mining method to capture the write-print of every suspect and model it as combinations of features that occur frequently in the suspect's emails. This notion is called frequent pattern, which has proven to be effective in many data mining applications, but has not been applied to the problem of authorship attribution. Unlike traditional approaches, the extracted write-print by our method is unique among the suspects and, therefore, provides convincing and credible evidence for presenting it in a court of law. Experiments on real-life emails suggest that the proposed method can effectively identify the author and the results are supported by a strong evidence.

Data mining (DM) is a powerful information technology (IT) tool in today’s competitive business world, especially as our human society entered the Big Data era. From academic point of view, it is an area of the intersection of human intervention, machine learning, mathematical modeling and databases. In recent years, data mining applications have become an important business strategy for most companies that want to attract new customers and retain existing ones. Using mathematical techniques, such as, neural networks, decision trees, mathematical programming, fuzzy logic and statistics, data mining software can help the company discover previously unknown, valid, and actionable information from various and large sources (either databases or open data sources like internet) for crucial business decisions. The algorithms of the mathematical models are implemented through some sort of computer languages, such as C++, JAVA, structured query language (SQL), on-line analysis processing (OLAP) and R. The process of data mining can be categorized as selecting, transforming, mining, and interpreting data. The ultimate goal of doing data mining is to find knowledge from data to support user’s decision. Therefore, data mining is strongly related with knowledge and knowledge management.

This paper presents a novel hybrid method combining genetic programming and decision tree learning. The method starts by estimating a benchmark level of reasonable accuracy, based on decision tree performance on bootstrap samples of the training set. Next, a normal GP evolution is started with the aim of producing an accurate GP. At even intervals, the best GP in the population is evaluated against the accuracy benchmark. If the GP has higher accuracy than the benchmark, the evolution continues normally until the maximum number of generations is reached. If the accuracy is lower than the benchmark, two things happen. First, the fitness function is modified to allow larger GPs, able to represent more complex models. Secondly, a decision tree with increased size and trained on a bootstrap of the training data is injected into the population. The experiments show that the hybrid solution of injecting decision trees into a GP population gives synergetic effects producing results that are better than using either technique separately. The results, from 18 UCI data sets, show that the proposed method clearly outperforms normal GP, and is significantly better than the standard decision tree algorithm.

Partial discharge (PD) is a common phenomenon that occurs in insulation of high voltage equipment such as transformers and has a damaging effect to the insulation. In this paper, the application of descriptive data mining on PD occurring in insulating systems is shown. Experiments were set up to create three basic types of PD: corona, surface discharges and internal discharges. PD data were analysed using phase resolved analysis and pulse height analysis. Descriptive data mining was applied on the collected data using decision tree with genetic algorithm (GA) to mine the rules/relationships that can be used to differentiate the PD. These extracted rules are useful as input to predictive data mining such as fuzzy logics.

Decision trees on parallel processors

Decision trees are considered as one of the efficient classification techniques in data mining fields. But the standard decision trees are unfit to cope with data pervaded with uncertainty both at the construction and classification phase. Dempster-Shafer theory offers an alternative to traditional probabilistic theory for the mathematical representation of uncertainty. This paper added new aggregation combination operator and uncertainty measure operator into general framework for data mining based on evidence theory. Combining these two operators with decision tree and multidimensional cube, decision tree technique can be extended to uncertain environment. In the phase of node splitting, this algorithm can pre-prune the decision tree and generate a decision tree with fewer branches. Simulations have shown the effectiveness of this method.

Today’s buildings are adequately informative because of the wide implementation of the building automation system (BAS). In this regard, data mining (DM) tools have an excellent ability to interpret and discover interesting, unknown information from raw data collected from BAS. The present work is a case study, in which DM tools (such as clustering and decision tree) are applied to understand the operational pattern of a hot water storage tank (sanicube) used in a residential building, located in Scotland. The main objective of this study is to explore the correlation among the important features (storage tank temperature, solar collector, the operation of the sanicube heating element, etc.) in the chosen building. Interesting correlations and patterns between the solar collector, storage tank, sanicube-heating element are explored. The clustering results show the existence of different charging patterns of the storage tank and it highly influences the space heating (SH) and domestic hot water (DHW) temperature. The main inference from the result is that the charging of the storage tank is not automatic, and it is influenced by the occupancy behaviour. To maintain the temperature of the storage tank and subsequently to meet the SH and DHW demand, the operation of both solar collector and sanicube heating element (during night time) is recommended rather than operating only the solar collector.

Decision tree classification is one of the most efficient methods for obtaining land use/land cover (LULC) information from remotely sensed imageries. However, traditional decision tree classification methods cannot effectively eliminate the influence of mixed pixels. This study aimed to integrate pixel unmixing and decision tree to improve LULC classification by removing mixed pixel influence. The abundance and minimum noise fraction (MNF) results that were obtained from mixed pixel decomposition were added to decision tree multi-features using a three-dimensional (3D) Terrain model, which was created using an image fusion digital elevation model (DEM), to select training samples (ROIs), and improve ROI separability. A Landsat-8 OLI image of the Yunlong Reservoir Basin in Kunming was used to test this proposed method. Study results showed that the Kappa coefficient and the overall accuracy of integrated pixel unmixing and decision tree method increased by 0.093% and 10%, respectively, as compared with the original decision tree method. This proposed method could effectively eliminate the influence of mixed pixels and improve the accuracy in complex LULC classifications.

The online version contains supplementary material available at 10.1007/s40200-021-00884-2.