TWO ALGORITHMS THAT LEARN DNF BY DISCOVERING RELEVANT FEATURES

Influence diagrams are a powerful graphic representation for decision models, complementary to decision trees. Influence diagrams and decision trees are different graphic representations for the same underlying mathematical model and operations. This article describes the elements of an influence diagram, and shows several familiar decision problems represented as decision trees and as influence diagrams. The authors also contrast the information highlighted in each graphic representation, demonstrate how to calculate the expected utilities of decision alternatives modeled with an influence diagram, provide an overview of the conceptual basis of the solution algorithms that have been developed for influence diagrams, discuss the strengths and limitations of influence diagrams relative to decision trees, and describe the mathematical operations that are used to evaluate both decision trees and influence diagrams. They use clinical examples to illustrate the mathematical operations of the influence-diagram-evaluation algorithm; these operations are arc reversal, chance node removal by averaging, and decision node removal by policy determination. Influence diagrams may be helpful when problems have a high degree of conditional independence, when large models are needed, when communication of the probabilistic relationships is important, or when the analysis requires extensive Bayesian updating. The choice of graphic representation should be governed by convenience, and will depend on the problem being analyzed, on the experience of the analyst, and on the background of the consumers of the analysis.

This article, written by Senior Technology Editor Dennis Denney, contains highlights of paper SPE 127761, ’Probabilistic Modeling for Decision Support in Integrated Operations,’ by Martin Giese, University of Oslo, and Reidar B. Bratvold, SPE, University of Stavanger, prepared for the 2010 SPE Intelligent Energy Conference and Exhibition, Utrecht, Netherlands, 23-25 March. The paper has not been peer reviewed. A system was designed to assist operational decisions on the basis of real-time sensor readings in a typical scenario: While drilling close to the transition to a high-pressure formation, gas influx is observed. The technology used Bayesian networks and influence diagrams (IDs) to apply the method of decision analysis. The resulting ID was tested with a simulation. Introduction An assumption of integrated operations is that all information provided by the technology will help to make better decisions faster. But for this to happen, information must be presented in an accessible, useful, and decision-relevant format. Decision-analysis tools provide decision makers with logical and consistent recommendations based on the available data. This decision-analysis method was applied to a real case, representative of a typical scenario in drilling operations. The tool enabled good decisions consistently. However, effort is required and challenges exist in the modeling process along with problems regarding the way decision making is treated in the industry. Modeling and Solving A decision tree provides a graphical illustration of all the uncertainties, decisions, and payoffs associated with a decision situation. Although application of decision trees is intuitive for simple problems, the method loses its appeal for complex decisions. A symmetric decision tree grows exponentially with the number of variables in the decision domain, which often makes its use impractical. For complex decisions, the ID can provide a level of insight and transparency that cannot be achieved with the equivalent decision tree. The ID is a compact graphical-model representation for reasoning under uncertainty and was developed to assist structuring and analysis of complex decisions. The ID has two advantages—structuring and probabilistic evaluation. The structuring provides a graphical representation of the inter-relationships among the decision-basis elements (i.e., alternatives, information, and preferences), while the evaluation appraises the resulting decision structure and provides an assessment of its expected utility given the quantified decision basis. The representational framework of an ID captures the probabilistic relationships among decisions (alternatives), key uncertainties (information), and utilities (values or preferences). The ID model comprises four types of nodes. Decision nodes specify a set of alternatives. Chance nodes represent the key uncertainties in the decision domain. Deterministic nodes represent either constants or intermediate calculations. Value nodes represent the criteria used to make the decisions.

We investigate the problem of learning Boolean functions with a short DNF representation using decision trees as a concept description language. Unfortunately, Boolean concepts with a short description may not have a small decision tree representation when the tests at the nodes are limited to the primitive attributes. This representational shortcoming may be overcome by using Boolean features at the decision nodes. We present two new methods that adaptively introduce relevant features while learning a decision tree from examples. We show empirically that these methods outperform a standard decision tree algorithm for learning small random DNF functions when the examples are drawn at random from the uniform distribution.

Chapter 5 - Decision Analysis Fundamentals

Conversion of a colorectal cancer guideline into clinical decision trees with assessment of validity

Motivation and Objectives Neuroblastoma (NB) is the most common pediatric solid tumor characterized by clinical and molecular risk factors. The mortality is about fifty percent and this makes exploration of new and more effective risk factors for improving stratification mandatory. Hypoxia is a condition of low oxygen tension occurring in poorly vascularized areas of the tumor associated with poor prognosis. We had previously defined a robust gene expression signature measuring the hypoxic component of NB tumors (NB-hypo) that is a novel, independent risk factor (Fardin et al., 2010). Integrating classical risk factors with NB-hypo could improve the stratification of NB patients. We wanted to develop a prognostic classifier of NB patients’ outcome blending existing knowledge on clinical and molecular risk factors with the prognostic NB-hypo signature. Furthermore, we were interested in the decision tree classifier that outputs explicit rules easily translated into the clinical setting. Methods A total of 182 NB patients were enrolled on the bases of availability of gene expression profile by Affymetrix GeneChip HG-U133plus2.0, clinical and molecular information. NB tumor stage was defined according to the International NB Staging System (INSS). Age at diagnosis was dichotomized as greater or equal than 1 year and less than one year. MYCN status was amplified or normal. Good and poor outcome were defined as the patient’s status alive or dead 5 years after diagnosis respectively. The risk group was assigned according to the International Neuroblastoma Risk Group (INRG) Consensus Pretreatment Classification Schema. The 182 NB patients cohort was clustered in High and low hypoxia by k-means analysis of the 62 probsets constituting the NB-hypo signature previously described to measure tumor hypoxia (Fardin et al., 2009). We utilized the k-means algorithm implemented in the WEKA software (Hall et al., 2009) setting up number of clusters to 2, 500 iterations, preserving instances order and using Manhattan distance. The classification was performed by induction of decision trees. We utilized the Weka J48 implementation of the popular C4.5 algorithm (Kotsiantis, 2007; Murthy, 1998) and we set up the following options: pruning parameter was 0.25, pruning method was sub-tree raising and minimum number of instances per leaf was 2. Each leaf of the decision tree classifier identifies a non overlapping group of patients and each decision node identifies a branch which splits the dataset. We utilized Fisher’s exact test to measure the statistical significance of groups and branches. Fisher’s test was utilized in the context of decision trees to design a top-down approach to prune out non statistically significant branches (Liu et al., 2010). For each leaf we counted the number of correctly classified (named n) and the number of incorrectly classified instances (named m). We considered the marginal totals y and ¬y which represent poor and good outcome patients respectively. We designed a 2x2 contingency table of the two possible outcomes (Good or Poor) against the number of instances included in a give leaf and the remaining instances. Application of the Fisher exact test to this table generates a p value giving the probability of observing 2x2 table, or more extreme tables, knowing the marginal totals (y, ¬y) and assuming independency among the patients in a specific leaf and those in other leaves. Results and Discussion Patients were stratified in good and poor outcome on the bases of the following risk factors: Age at diagnosis, INSS stage, MYCN status and NB-hypo. The algorithm generated a decision tree classifier composed by 3 decision nodes and 7 leaves covering 87% of non overlapping good outcome patients and 100% of non overlapping poor outcome patients. Each path from the root to a leaf utilizes some, but not all, considered risk factors. Interestingly NB-hypo was included in the decision node that stratified stage 3 tumors demonstrating its usefulness in NB patients’ stratification. The leaves classifying good outcome patients had the very low error of 2% indicating a good performance of the algorithm in predicting this class. In contrast, the classification of poor outcome patients produced the leaf with the highest error of 13%. This leaf includes stage 4 tumors that are traditionally difficult to stratify by any known risk factor. To test statistical significance of the splits performed by the algorithm and the groups of patients, we utilized the Fisher’s exact test with a confidence set at 95%. The results showed that the groups obtained at each split were statistically significant. Furthermore, analysis of single groups of patients identified by leaves demonstrated that some, but no all reached the significance threshold. The significant leaves included the NB-hypo among the utilized risk factors. These results further strengthen the value of NB-hypo in predicting patients’ outcome. Lack of significance was often associated with a rather low number of patients in the leaf. This is a limitation of the divide-and-conquer approach of the algorithm, applied to relatively small patients’ cohorts, that recurrently splits the dataset. We then assessed the concordance of the predictions with INRG risk assessment. High Risk patients were correctly included in the leaves classifying poor outcome patients and Low Risk patients mapped correctly in the good outcome leaves. Interestingly, NB-hypo generated a leaf identifying a new group of poor outcome patients, sharing the high NB-hypo, whose characteristic fell into both High and Intermediate Risk. We collected and analyzed the results of 1000 10-fold cross validations and we observed that most classifiers had 7 leaves and only 5 out of 10^4 deviated from this pattern. The recurrence of 7 leaves demonstrated the high stability of the decision tree classifier that we generated. Analysis of the pruning parameters revealed optimal performance in the range of 0.1-0.3 in line with what used in this study. The path to reach each leaf can be easily transposed into a “if…than…” rule that, in turn provides an easy readout of the classifier, precious for translating the classification into the clinical setting. In conclusion, we demonstrated that the decision tree algorithm C4.5 can derive explicit rules for NB patients stratification if classical risk factors are blended with the NB hypo signature. These rules are statistically significant and quite stable and suitable to be conveyed to the clinic to design new therapies perhaps taking hypoxia into consideration as a potential target. Acknowledgements The work was supported by the Fondazione Italiana per la Lotta al Neuroblastoma, the Associazione Italiana per la Ricerca sul Cancro, the Società Italiana Glicogenosi, the Fondazione Umberto Veronesi and the Ministero della Salute Italiano. Davide Cangelosi and Fabiola Blengio are recipients of a fellowship from the Fondazione Italiana per la Lotta al Neuroblastoma. References Fardin P, Barla A, Mosci S, Rosasco L, Verri A, Varesio L. (2009) The l1-l2 regularization framework unmasks the hypoxia signature hidden in the transcriptome of a set of heterogeneous neuroblastoma cell lines. BMC Genomics, 10:474. doi:10.1186/1471-2164-10-474 Fardin P, Barla A, Mosci S, Rosasco L, Verri A, et al. (2010) A biology-driven approach identifies the hypoxia gene signature as a predictor of the outcome of neuroblastoma patients, Journal of Molecular Cancer 9:1. 185. doi:10.1186/1476-4598-9-185 Hall M, Eibe F, Holmes G, Pfahringer B, Reutemann P, Witten IH. (2009) The WEKA Data Mining Software: An Update: SIGKDD Explorations. doi: 10.1145/1656274.1656278 Kotsiantis S.B. (2007) Supervised Machine Learning: A Review of Classification Techniques. Informatica 31:249-268. Liu W, Chawla S, Cieslak D, Chawla N. (2010) A Robust Decision Tree Algorithm for Imbalanced Data Sets. In: SDM. 766-777. Murthy S.K. (1998) Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey, Data Mining Knowledge Discovery 2:4. 345-389. doi: 10.1023/A:1009744630224

Text mining is a special case of data mining which explore unstructured or semi-structured text documents, to establish valuable patterns and rules that indicate trends and significant features about specific topics. Text mining has been in pattern recognition, predictive studies, sentiment analysis and statistical theories in many areas of research, medicine, financial analysis, social life analysis, and business intelligence. Text mining uses concept of natural language processing and machine learning. Machine learning algorithms have been used and reported to give great results, but their performance of machine learning algorithms is affected by factors such as dataset domain, number of classes, length of the corpus, and feature selection techniques used. Redundant attribute affects the performance of the classification algorithm, but this can be reduced by using different feature selection techniques and dimensionality reduction techniques. Feature selection is a data preprocessing step that chooses a subset of input variable while eliminating features with little or no predictive information. Feature selection techniques are Information gain, Term Frequency, Term Frequency-Inverse document frequency, Mutual Information, and Chi-Square, which can use a filters, wrappers, or embedded approaches. To get the most value from machine learning, pairing the best algorithms with the right tools and processes is necessary. Little research has been done on the effect of feature selection techniques on classification accuracy for pairing of these algorithms with the best feature selection techniques for optimal results. In this research, a text classification experiment was conducted using incident management dataset, where incidents were classified into their resolver groups. Support vector machine (SVM), K-Nearest Neighbors (KNN), Naïve Bayes (NB) and Decision tree (DT) machine learning algorithms were examined. Filtering approach was used on the feature selection techniques, with different ranking indices applied for optimal feature set and classification accuracy results analyzed. The classification accuracy results obtained using TF were, 88% for SVM, 70% for NB, 79% for Decision tree, and KNN had 55%, while Boolean registered 90%, 83%, 82% and 75%, for SVM, NB, DT, and KNN respectively. TF-IDF, had 91%, 83%, 76%, and 56% for SVM, NB, DT, and KNN respectively. The results showed that algorithm performance is affected by feature selection technique applied. SVM performed best, followed by DT, KNN and finally NB. In conclusion, presence of noisy data leads to poor learning performance and increases the computational time. The classifiers performed differently depending on the feature selection technique applied. For optimal results, the classifier that performed best together with the feature selection technique with the best feature subset should be applied for all types of data for accurate classification performance. Keywords: Text Classification, Supervised Machine Learning, Feature Selection DOI : 10.7176/JIEA/9-3-06 Publication date :May 31 st 2019

A fundamental problem in model-based computer vision is that of identifying to which of a given set of concept classes of geometric models an observed model belongs. Considering a “probe” to be an oracle that tells whether or not the observed model is present at a given point in an image, we study the problem of computing efficient strategies (“decision trees”) for probing an image, with the goal to minimize the number of probes necessary (in the worst case) to determine in which class the observed model belongs. We prove a hardness result and give strategies that obtain decision trees whose height is within a log factor of optimal.

Decision Model for Acute Appendicitis Treatment With Decision Tree Technology—A Modification of the Alvarado Scoring System

AimsThis study aimed to clarify the different topographical distribution of tau pathology between progressive supranuclear palsy (PSP) and corticobasal degeneration (CBD) and establish a machine learning‐based decision tree classifier.MethodsParaffin‐embedded sections of the temporal cortex, motor cortex, caudate nucleus, globus pallidus, subthalamic nucleus, substantia nigra, red nucleus, and midbrain tectum from 1020 PSP and 199 CBD cases were assessed by phospho‐tau immunohistochemistry. The severity of tau lesions (i.e., neurofibrillary tangle, coiled body, tufted astrocyte or astrocytic plaque, and tau threads) was semi‐quantitatively scored in each region. Hierarchical cluster analysis was performed using tau pathology scores. A decision tree classifier was made with tau pathology scores using 914 cases. Cross‐validation was done using 305 cases. An additional ten cases were used for a validation study.ResultsCluster analysis displayed two distinct clusters; the first cluster included only CBD, and the other cluster included all PSP and six CBD cases. We built a decision tree, which used only seven decision nodes. The scores of tau threads in the caudate nucleus were the most decisive factor for predicting CBD. In a cross‐validation, 302 out of 305 cases were correctly diagnosed. In the pilot validation study, three investigators made a correct diagnosis in all cases using the decision tree.ConclusionRegardless of the morphology of astrocytic tau lesions, semi‐quantitative tau pathology scores in select brain regions are sufficient to distinguish PSP and CBD. The decision tree simplifies neuropathologic differential diagnosis of PSP and CBD.

Warning and evacuation are among the most effective ways for saving human lives and properties from landslide dam hazards. A new warning decision model for landslide dam break is developed using Influence Diagrams to minimize the total losses. An Influence Diagram is a simple visual representation of a decision problem. It analyzes the qualitative (causal) relationships between the variables via a logic diagram and determines the quantitative relationships via conditional probability and Bayes’ theorem. The model is applied for the warning decision-making of the 2008 Tangjiashan landslide dam. The new model unifies the dam failure probability, evacuation, life loss, and flood damage in an Influence Diagram. Besides, a warning criterion is proposed for efficient decision-making. The model is more advanced than the decision tree since the inter-relationships of influence factors are qualitatively analyzed with causality connections and quantitatively analyzed with conditional probabilities. It is more efficient than a dynamic decision-making model (DYDEM) as it can directly calculate the three types of flood loss (i.e., evacuation cost, flood damage, and monetized life loss) and the expected total loss. Moreover, the probabilities of the influence factors leading to known results can be obtained through inversion analysis based on Bayesian theory. The new warning decision model offers an efficient way to save lives from landslide dam breaking and avoid unnecessary expenses from premature warning and evacuation.

Dynamic decision making without expected utility: An operational approach

Association rule mining (ARM) has been studied in the areas of content-based multimedia retrieval and semantic concept detection due to its high efficiency and accuracy. Two important processes in mining the association rules for classification are rule generation and rule selection. In this paper, a novel high-level feature detection framework using the ARM technique together with the correlations among the feature-value pairs is proposed. A new association rule mining (ARM) algorithm has been developed, where the N-feature-value pairs are generated using a combined measure based on (1) the existence of the (N-1)-feature-value pairs (where N is larger than 1), (2) the correlation between different N-feature-value pairs and the concept classes through multiple correspondence analysis (MCA), and (3) the similarity representing the harmonic mean of the inter-similarity and intra-similarity. The final association classification rules are selected by using the calculated harmonic mean of the similarity values. The proposed framework enables the automatic discovery and generation of the N-feature-value pair association rules from the N-feature-value pairs for classification. Experimenting with 15 high-level features (concepts) and benchmark data sets from TRECVID, our proposed framework achieves promising performance and outperforms three other well-known classifiers (decision trees, support vector machine, and neural networks) which are commonly used for performance comparison in the TRECVID community.

Using decision trees as a concept description language, we examine the time complexity for learning Boolean functions with polynomial‐sized disjunctive normal form expressions when feature construction is performed on an initial decision tree containing only primitive attributes. A shortcoming of several feature‐construction algorithms found in the literature is that it is difficult to develop time complexity results for them. We illustrate a way to determine a limit on the number of features to use for building more concise trees within a standard amount of time. We introduce a practical algorithm that forms a finite number of features using a decision tree in a polynomial amount of time. We show empirically that our procedure forms many features that subsequently appear in a tree and the new features aid in producing simpler trees when concepts are being learned from certain problem domains. Expert systems developers can use a method such as this to create a knowledge base of information that contains specific knowledge in the form of If‐Then rules.

In this paper, we study arbitrary subword-closed languages over the alphabet (binary subword-closed languages). For the set of words of the length n belonging to a binary subword-closed language L, we investigate the depth of the decision trees solving the recognition and the membership problems deterministically and nondeterministically. In the case of the recognition problem, for a given word from , we should recognize it using queries, each of which, for some , returns the ith letter of the word. In the case of the membership problem, for a given word over the alphabet of the length n, we should recognize if it belongs to the set using the same queries. With the growth of n, the minimum depth of the decision trees solving the problem of recognition deterministically is either bounded from above by a constant or grows as a logarithm, or linearly. For other types of trees and problems (decision trees solving the problem of recognition nondeterministically and decision trees solving the membership problem deterministically and nondeterministically), with the growth of n, the minimum depth of the decision trees is either bounded from above by a constant or grows linearly. We study the joint behavior of the minimum depths of the considered four types of decision trees and describe five complexity classes of binary subword-closed languages.