A weighted random forests approach to improve predictive performance

BackgroundIdentifying variants associated with complex human traits in high-dimensional data is a central goal of genome-wide association studies. However, complicated etiologies such as gene-gene interactions are ignored by the univariate analysis usually applied in these studies. Random Forests (RF) are a popular data-mining technique that can accommodate a large number of predictor variables and allow for complex models with interactions. RF analysis produces measures of variable importance that can be used to rank the predictor variables. Thus, single nucleotide polymorphism (SNP) analysis using RFs is gaining popularity as a potential filter approach that considers interactions in high-dimensional data. However, the impact of data dimensionality on the power of RF to identify interactions has not been thoroughly explored. We investigate the ability of rankings from variable importance measures to detect gene-gene interaction effects and their potential effectiveness as filters compared to p-values from univariate logistic regression, particularly as the data becomes increasingly high-dimensional.ResultsRF effectively identifies interactions in low dimensional data. As the total number of predictor variables increases, probability of detection declines more rapidly for interacting SNPs than for non-interacting SNPs, indicating that in high-dimensional data the RF variable importance measures are capturing marginal effects rather than capturing the effects of interactions.ConclusionsWhile RF remains a promising data-mining technique that extends univariate methods to condition on multiple variables simultaneously, RF variable importance measures fail to detect interaction effects in high-dimensional data in the absence of a strong marginal component, and therefore may not be useful as a filter technique that allows for interaction effects in genome-wide data.

Because of the poor performance of the Random Forest algorithm in processing the classification of high-dimensional unbalanced data, a Hybrid Samping&Feature Selection Random Forest optimization strategy (Hybrid Samping&Feature Selection Random Forest (HF_RF) is proposed in this paper. First, from the data level, the high-dimensional unbalanced data set is preprocessed by SMOTE algorithm combined with random undersampling to achieve balanced unbalanced data. At the same time, the clustering algorithm is combined with SMOTE algorithm to improve the processing ability of the algorithm for negative samples; On the algorithm level, through the Relief F algorithm, different weight values are given to the preprocessed high-dimensional data, irrelevant and redundant features are eliminated, and high-dimensional data is reduced for dimensionality; Finally, the weighted voting principle is used to further elevate the predictive performance of HF_RF. The experimental results show that compared with the traditional algorithm, the proposed algorithm has higher indicators when dealing with high-dimensional unbalanced data, which proves that the HF_RF proposed in this paper is The correctness of the algorithm and its effectiveness in improving the classification performance of high-dimensional unbalanced data.

Random forests (RFs) have been widely used as a powerful classification method. However, with the randomization in both bagging samples and feature selection, the trees in the forest tend to select uninformative features for node splitting. This makes RFs have poor accuracy when working with high-dimensional data. Besides that, RFs have bias in the feature selection process where multivalued features are favored. Aiming at debiasing feature selection in RFs, we propose a new RF algorithm, called xRF, to select good features in learning RFs for high-dimensional data. We first remove the uninformative features using p-value assessment, and the subset of unbiased features is then selected based on some statistical measures. This feature subset is then partitioned into two subsets. A feature weighting sampling technique is used to sample features from these two subsets for building trees. This approach enables one to generate more accurate trees, while allowing one to reduce dimensionality and the amount of data needed for learning RFs. An extensive set of experiments has been conducted on 47 high-dimensional real-world datasets including image datasets. The experimental results have shown that RFs with the proposed approach outperformed the existing random forests in increasing the accuracy and the AUC measures.

Interest in predictive analytics of big data has grown exponentially in the four years since the publication of Statistical and Machine-Learning Data Mining: Techniques for Better Predictive Modeling and Analysis of Big Data, Second Edition. In the third edition of this bestseller, the author has completely revised, reorganized, and repositioned the original chapters and produced 13 new chapters of creative and useful machine-learning data mining techniques. In sum, the 43 chapters of simple yet insightful quantitative techniques make this book unique in the field of data mining literature. What is new in the Third Edition: The current chapters have been completely rewritten. The core content has been extended with strategies and methods for problems drawn from the top predictive analytics conference and statistical modeling workshops. Adds thirteen new chapters including coverage of data science and its rise, market share estimation, share of wallet modeling without survey data, latent market segmentation, statistical regression modeling that deals with incomplete data, decile analysis assessment in terms of the predictive power of the data, and a user-friendly version of text mining, not requiring an advanced background in natural language processing (NLP). Includes SAS subroutines which can be easily converted to other languages. As in the previous edition, this book offers detailed background, discussion, and illustration of specific methods for solving the most commonly experienced problems in predictive modeling and analysis of big data. The author addresses each methodology and assigns its application to a specific type of problem. To better ground readers, the book provides an in-depth discussion of the basic methodologies of predictive modeling and analysis. While this type of overview has been attempted before, this approach offers a truly nitty-gritty, step-by-step method that both tyros and experts in the field can enjoy playing with.

Ensemble classifiers are being widely used for the classification of spectroscopic data. In this regard, the random forest (RF) ensemble has been successfully applied in an array of applications, and has proven to be robust in handling high dimensional data. More recently, several variants of the traditional RF algorithm including rotation forest (rotF) and oblique random forest (oRF) have been applied to classifying high dimensional data. In this study we compare the traditional RF, rotF, and oRF (using three different splitting rules, i.e., ridge regression, partial least squares, and support vector machine) for the classification of healthy and infected Pinus radiata seedlings using high dimensional spectroscopic data. We further test the robustness of these five ensemble classifiers to reduced spectral resolution by spectral resampling (binning) of the original spectral bands. The results showed that the three oblique random forest ensembles outperformed both the traditional RF and rotF ensembles. Additionally, the rotF ensemble proved to be the least robust of the five ensembles tested. Spectral resampling of the original bands provided mixed results. Nevertheless, the results demonstrate that using spectral resampled bands is a promising approach to classifying asymptomatic stress in Pinus radiata seedlings.

Outlier detection is an important data mining task for consistency checks, fraud detection, etc. Binary decision making on whether or not an object is an outlier is not appropriate in many applications and moreover hard to parametrize. Thus, recently, methods for outlier ranking have been proposed. Determining the degree of deviation, they do not require setting a decision boundary between outliers and the remaining data. High dimensional and heterogeneous (continuous and categorical attributes) data, however, pose a problem for most outlier ranking algorithms. In this work, we propose our OutRank approach for ranking outliers in heterogeneous high dimensional data. We introduce a consistent model for different attribute types. Our novel scoring functions transform the analyzed structure of the data to a meaningful ranking. Promising results in preliminary experiments show the potential for successful outlier ranking in high dimensional data.

Random Forest is a supervised classification method based on bagging (Bootstrap aggregating) Breiman and random selection of features. The choice of features randomly assigned to the Random Forest makes it possible that the selected feature is not necessarily informative. So it is necessary to select features in the Random Forest. The purpose of choosing this feature is to select an optimal subset of features that contain valuable information in the hope of accelerating the performance of the Random Forest method. Mainly for the execution of high-dimensional datasets such as the Parkinson, CNAE-9, and Urban Land Cover dataset. The feature selection is done using the Correlation-Based Feature Selection method, using the BestFirst method. Tests were carried out 30 times using the K-Cross Fold Validation value of 10 and dividing the dataset into 70% training and 30% testing. The experiments using the Parkinson dataset obtained a time difference of 0.27 and 0.28 seconds faster than using the Random Forest method without feature selection. Likewise, the trials in the Urban Land Cover dataset had 0.04 and 0.03 seconds, while for the CNAE-9 dataset, the difference time was 2.23 and 2.81 faster than using the Random Forest method without feature selection. These experiments showed that the Random Forest processes are faster when using the first feature selection. Likewise, the accuracy value increased in the two previous experiments, while only the CNAE-9 dataset experiment gets a lower accuracy. This research’s benefits is by first performing feature selection steps using the Correlation-Base Feature Selection method can increase the speed of performance and accuracy of the Random Forest method on high-dimensional data.

The primary goal of genome-wide association studies (GWAS) is to discover genes or variants associated with complex diseases. Most GWA studies use single SNP (single nucleotide polymorphism) approaches that mainly focused on assessing the association between each individual SNP and disease; therefore they cannot take into account the combinations of SNPs. However, complex diseases are thought to involve complex etiologies including complicated interactions between many SNPs. Thus, different approaches are necessary to identify SNPs that influence disease risk jointly or in complex interactions. To discover SNP-SNP interactions, in this paper we propose first to use an improvement of Random Forest algorithm tailored for structured GWAS data, all rules are then extracted from the trees to analyse SNPs interactions. Our method allows one to select subgroups of informative SNPs which are most relevant to disease for building accurate decision trees and then we enable educe SNPs interactions from these trees. By this way, it reduces the dimensionality and can perform well with high-dimensional SNPs data sets. We conducted experiments on two genome-wide SNP data sets to demonstrate the effectiveness of the method for the SNP-SNP interactions.

Random Forest (RF) has been widely used in the classification of high dimensional data. However, all the features of high dimensional data are used for classification, which will increase the computation time and reduce the classification accuracy. Therefore, feature selection is critical to high dimensional data classification. In order to solve this problem, this paper presents a method of Conditional Mutual Information (CMI) and Random Forest (CMI-RF). CMI is used to remove irrelevant and redundant information. The optimal subset of features with higher classification accuracy is obtained by RF. In this paper, the high dimensional near infrared spectral data is taken as experimental data. The experimental results demonstrate that CMI-RF method can select the feature subset with stronger correlation, no redundancy and high classification accuracy.

Self-organizing subspace clustering for high-dimensional and multi-view data

<p>Classifying high-dimensional data are a challenging task in data mining. Gene expression data is a type of high-dimensional data that has thousands of features. The study was proposing a method to extract knowledge from high-dimensional gene expression data by selecting features and classifying. Lasso was used for selecting features and the classification and regression tree (CART) algorithm was used to construct the decision tree model. To examine the stability of the lasso decision tree, we performed bootstrap aggregating (Bagging) with 50 replications. The gene expression data used was an ovarian tumor dataset that has 1,545 observations, 10,935 gene features, and binary class. The findings of this research showed that the lasso decision tree could produce an interpretable model that theoretically correct and had an accuracy of 89.32%. Meanwhile, the model obtained from the majority vote gave an accuracy of 90.29% which showed an increase in accuracy of 1% from the single lasso decision tree model. The slightly increasing accuracy shows that the lasso decision tree classifier is stable.</p>

This paper establishes a negative result for clustering: above a certain ratio of random noise to nonrandom information, it is impossible for a large class of cost functions to distinguish between two partitions of a data set. In particular, it is shown that as the dimension increases, the ability to distinguish an accurate partitioning from an inaccurate one is lost unless the informative components are both sufficiently numerous and sufficiently informative. We examine squared error cost functions in detail. More generally, it is seen that the VC‐dimension is an essential hypothesis for the class of cost functions to satisfy for an impossibility proof to be feasible. Separately, we provide bounds on the probabilistic behavior of cost functions that show how rapidly the ability to distinguish two clusterings decays. In two examples, one simulated and one with genomic data, bounds on the ability of squared‐error and other cost functions to distinguish between two partitions are computed. Thus, one should not rely on clustering results alone for high dimensional low sample size data and one should do feature selection. Copyright © 2010 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 4: 30–53 2011

The last three decades have witnessed significant advances in psychotherapy. Numerous scholarly articles and books have been devoted to pertinent topics in the field, making it difficult for the practicing clinician to keep up with this rapidly growing area. The purpose of this article is to provide some guidelines on how to evaluate the empirical literature in psychotherapy and then to explore three key areas: evidence-based psychotherapies for patients with psychiatric disorders, individual variables that predict differential outcome to treatment, and the therapeutic alliance. Finally, two case examples will be presented to illustrate how knowledge of the empirical literature can facilitate an evidence-based approach to the daily practice of psychotherapy in general psychiatry.

Incremental linear model trees with approximate lookahead are fast, but produce overly large trees. This is due to non-optimal splitting decisions boosted by a possibly unlimited number of examples obtained from a data source. To keep the processing speed high and the tree complexity low, appropriate incremental pruning techniques are needed. In this paper, we introduce a pruning technique for the class of incremental linear model trees with approximate lookahead on stationary data sources. Experimental results show that the advantage of approximate lookahead in terms of processing speed can be further improved by producing much smaller and consequently more explanatory, less memory consuming trees on high-dimensional data. This is done at the expense of only a small increase in prediction error. Additionally, the pruning algorithm can be tuned to either produce less accurate model trees at a much higher processing speed or, alternatively, more accurate trees at the expense of higher processing times.

High dimension data is growing rapidly, which challenges the existing data mining and learning systems. A stable performance can be achieved precisely in high dimension data using machine learning algorithm, ensemble learning and feature selection methods. The ensemble classifiers like Decision tree, Random forest, support vector machine, Navies Bayes, Linear Discriminant Analysis are applied as a self-contained module and the output label is reliably produced by the fusion of majority voting for data analysis. At present a lot of investigations for reduction of high dimensional data set are implemented, but the study result shows only a minor improvement in accuracy. In this paper, a wrapper method is properly used to improve certain key features such as random state, tune max depth, tune number samples split, and tuned loss function. The proposed work is carried out using real-time garbage data set collected from the municipal corporation of Tiruchirappalli, India and the selected sample is tested with FGDT (Forward Gradient Boost Decision Tree) shows improvement in accuracy.