Feature selection to increase the random forest method performance on high dimensional data

BackgroundSuccessfully modeling high-dimensional data involving thousands of variables is challenging. This is especially true for gene expression profiling experiments, given the large number of genes involved and the small number of samples available. Random Forests (RF) is a popular and widely used approach to feature selection for such "small n, large p problems." However, Random Forests suffers from instability, especially in the presence of noisy and/or unbalanced inputs.ResultsWe present RKNN-FS, an innovative feature selection procedure for "small n, large p problems." RKNN-FS is based on Random KNN (RKNN), a novel generalization of traditional nearest-neighbor modeling. RKNN consists of an ensemble of base k-nearest neighbor models, each constructed from a random subset of the input variables. To rank the importance of the variables, we define a criterion on the RKNN framework, using the notion of support. A two-stage backward model selection method is then developed based on this criterion. Empirical results on microarray data sets with thousands of variables and relatively few samples show that RKNN-FS is an effective feature selection approach for high-dimensional data. RKNN is similar to Random Forests in terms of classification accuracy without feature selection. However, RKNN provides much better classification accuracy than RF when each method incorporates a feature-selection step. Our results show that RKNN is significantly more stable and more robust than Random Forests for feature selection when the input data are noisy and/or unbalanced. Further, RKNN-FS is much faster than the Random Forests feature selection method (RF-FS), especially for large scale problems, involving thousands of variables and multiple classes.ConclusionsGiven the superiority of Random KNN in classification performance when compared with Random Forests, RKNN-FS's simplicity and ease of implementation, and its superiority in speed and stability, we propose RKNN-FS as a faster and more stable alternative to Random Forests in classification problems involving feature selection for high-dimensional datasets.

Random Forest (RF) has been widely used in the classification of high dimensional data. However, all the features of high dimensional data are used for classification, which will increase the computation time and reduce the classification accuracy. Therefore, feature selection is critical to high dimensional data classification. In order to solve this problem, this paper presents a method of Conditional Mutual Information (CMI) and Random Forest (CMI-RF). CMI is used to remove irrelevant and redundant information. The optimal subset of features with higher classification accuracy is obtained by RF. In this paper, the high dimensional near infrared spectral data is taken as experimental data. The experimental results demonstrate that CMI-RF method can select the feature subset with stronger correlation, no redundancy and high classification accuracy.

Cashmere and wool play an important role in the wool industry and textile industry, and suitable features are the key to identifying them. To obtain effective features and improve the accuracy of cashmere and wool classification, the multi-feature selection and random forest method is used to express in this article. Firstly, the gray-gradient co-occurrence matrix model is used for texture feature extraction to construct the original high-dimensional feature data set; secondly, considering that the original feature data set contains a large number of invalid and redundant features, the feature selection algorithm combining correlation analysis and principal component analysis–weight coefficient evaluation is used to obtain important features, independent features, and principal component sensitive features to complement each other; last but not least, the optimized random forest model analyzes the results. The results show that the combination of multi-feature selection subsets and random forest makes the classification accuracy of cashmere and wool more reliable, and the accuracy fluctuates around 90%.

Because of the poor performance of the Random Forest algorithm in processing the classification of high-dimensional unbalanced data, a Hybrid Samping&Feature Selection Random Forest optimization strategy (Hybrid Samping&Feature Selection Random Forest (HF_RF) is proposed in this paper. First, from the data level, the high-dimensional unbalanced data set is preprocessed by SMOTE algorithm combined with random undersampling to achieve balanced unbalanced data. At the same time, the clustering algorithm is combined with SMOTE algorithm to improve the processing ability of the algorithm for negative samples; On the algorithm level, through the Relief F algorithm, different weight values are given to the preprocessed high-dimensional data, irrelevant and redundant features are eliminated, and high-dimensional data is reduced for dimensionality; Finally, the weighted voting principle is used to further elevate the predictive performance of HF_RF. The experimental results show that compared with the traditional algorithm, the proposed algorithm has higher indicators when dealing with high-dimensional unbalanced data, which proves that the HF_RF proposed in this paper is The correctness of the algorithm and its effectiveness in improving the classification performance of high-dimensional unbalanced data.

Feature selection is one of the most significant procedures in machine learning algorithms. It is particularly to improve the performance and prediction accuracy for complex data classification. This paper discusses a hybrid feature selection technique with the decision tree-based classification algorithm. The feature selected using information gain (IG) is combined with the features selected from ReliefF which generates the resultant feature subset. Then the resultant feature subset is in turn combined with a correlation-based feature selection (CFS) method to generate the aggregated feature subset. To perform classification accuracy on the aggregated feature subset, different decision trees-based classification algorithm such as C4.5, decision stumps, naive Bayes tree, and random forest with ten-fold cross-validation have been deployed. To check the prediction accuracy of the proposed work eight different multilevel University of California, Irvine (UCI) machine learning datasets have been used with minimum to maximum numbers of features. The main objective of the hybrid feature selection is to improve the classification accuracy, prediction and to reduce the execution time using standard datasets.

Cybersecurity in modern systems, particularly in the context of space systems and network infrastructure, faces increasing challenges due to the sophistication and variety of attacks. To effectively secure such systems, it is crucial to implement robust detection and monitoring methods that can identify malicious activity in real-time. This study explores the application of machine learning techniques, particularly random forest and feature selection methods like particle swarm optimization (PSO), to enhance cybersecurity measures. The AWID2 dataset, sourced from IEEE 802.11 wireless networks, was utilized to simulate real-world network conditions and different types of attacks, including impersonation, injection, and flooding. The research focused on pre-processing the dataset through normalization and scaling, followed by feature extraction using principal component analysis (PCA) and feature selection using PSO. The performance of the system was evaluated by training a random forest classifier, demonstrating its ability to effectively differentiate between normal and attack traffic. The findings suggest that the combination of these techniques can significantly improve the accuracy and efficiency of attack detection systems, offering insights into how machine learning can be integrated into cybersecurity strategies for space and network infrastructures.

Random forests (RFs) have been widely used as a powerful classification method. However, with the randomization in both bagging samples and feature selection, the trees in the forest tend to select uninformative features for node splitting. This makes RFs have poor accuracy when working with high-dimensional data. Besides that, RFs have bias in the feature selection process where multivalued features are favored. Aiming at debiasing feature selection in RFs, we propose a new RF algorithm, called xRF, to select good features in learning RFs for high-dimensional data. We first remove the uninformative features using p-value assessment, and the subset of unbiased features is then selected based on some statistical measures. This feature subset is then partitioned into two subsets. A feature weighting sampling technique is used to sample features from these two subsets for building trees. This approach enables one to generate more accurate trees, while allowing one to reduce dimensionality and the amount of data needed for learning RFs. An extensive set of experiments has been conducted on 47 high-dimensional real-world datasets including image datasets. The experimental results have shown that RFs with the proposed approach outperformed the existing random forests in increasing the accuracy and the AUC measures.

Based on benchmarking various methods and analyzing multiple real scRNA-seq datasets, a computational platform/workflow and a set of tips for best practices are developed for analyzing causal interactions or relationships in single cells.

<p>Soil properties could be assessed with reflectance spectroscopy (soil spectroscopy, SS) in vis-NIR region (400-2500 nm) through absorption features found in soil spectra. A high spectral resolution (up to 1 nm) drives to high dimensional and multicollinear data. This issue is usually addressed prior to modelling with feature extraction methods such as Principal Component Analysis, or embedded methods such as Partial Least Squares Regression (PLSR). Feature Selection (FS) wrapper methods are promising dimensionality reduction approaches barely used in SS. The objective of this study was two-fold: i) evaluate the performance of FS wrapper methods built from Random Forest (RF) algorithm to predict soil organic matter (SOM), clay and carbonates using laboratory spectroscopy, ii) test the performance of FS methods for dimensionality reduction in SS. The reflectance of 100 soil samples from Sierra de las Nieves National Park (Spain), was measured under laboratory conditions using an ASD FieldSpec Pro JR. A spectral preprocessing method, Continumm Removal (CR), was applied to raw spectra. The RF wrapper considered two different feature searching approaches: Sequential Forward Selection (SFS) and Sequential Flotant Forward Selection (SFFS). The performance of RF with FS (RF-FS) was compared to that of Partial Least Squares Regression (PLSR) and RF (without FS). Models were evaluated with R-squared, root mean squared error (RMSE) and ratio of prediction to deviation (RPD).</p><p>RF-FS models outperformed PLSR and RF models for the three SAP. RF-FS best models had a RPD of 2.19 for SOM, 1.64 for carbonates and 1.52 for clay, whereas PLSR models had RPD values of 1.59, 1.22 and 1.3, and RF 1.38, 1.23 and 1.23 for SOM, carbonates, and clay, respectively. Therefore, FS was useful in obtaining models with improved accuracy by reducing redundant features and avoiding multicollinearity (Hughes effect). The application of FS wrapper methods reduced the number of features in the RF-FS models to less than 1% of the starting features. Features were selected across all spectra from SOM and clay, and around 900, 1900 and 2350 nm for carbonates. This research, thus, shows an alternative to different feature extraction approaches for modelling soil properties based on FS methods and machine learning.</p>

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

Introduction: Coronary stenosis due to atherosclerosis restricts blood flow. Stenosis progression would lead to increased clinical risk such as heart attack. Although many risk factors were found to contribute to atherosclerosis progression, factors associated with fatigue is underemphasized. Our goal is to investigate the relationship between fatigue and stenosis progression based on in vivo intravascular ultrasound (IVUS) images and finite element models.Methods: Baseline and follow-up in vivo IVUS and angiography data were acquired from seven patients using Institutional Review Board approved protocols with informed consent obtained. Three hundred and five paired slices at baseline and follow-up were matched and used for plaque modeling and analysis. IVUS-based thin-slice models were constructed to obtain the coronary biomechanics and stress/strain amplitudes (stress/strain variations in one cardiac cycle) were used as the measurement of fatigue. The change of lumen area (DLA) from baseline to follow-up were calculated to measure stenosis progression. Nineteen morphological and biomechanical factors were extracted from 305 slices at baseline. Correlation analyses of these factors with DLA were performed. Random forest (RF) method was used to fit morphological and biomechanical factors at baseline to predict stenosis progression during follow-up.Results: Significant correlations were found between stenosis progression and maximum stress amplitude, average stress amplitude and average strain amplitude (p < 0.05). After factors selection implemented by random forest (RF) method, eight morphological and biomechanical factors were selected for classification prediction of stenosis progression. Using eight factors including fatigue, the overall classification accuracy, sensitivity and specificity of stenosis progression prediction with RF method were 83.61%, 86.25% and 80.69%, respectively.Conclusion: Fatigue correlated positively with stenosis progression. Factors associated with fatigue could contribute to better prediction for atherosclerosis progression.

In the present study, gene expression profiles of acute myeloid leukemia (AML) samples were analyzed to identify feature genes with the capacity to predict the mutation status of FLT3/ITD. Two machine learning models, namely the support vector machine (SVM) and random forest (RF) methods, were used for classification. Four datasets were downloaded from the European Bioinformatics Institute, two of which (containing 371 samples, including 281 FLT3/ITD mutation-negative and 90 mutation-positive samples) were randomly defined as the training group, while the other two datasets (containing 488 samples, including 350 FLT3/ITD mutation-negative and 138 mutation-positive samples) were defined as the test group. Differentially expressed genes (DEGs) were identified by significance analysis of the micro-array data by using the training samples. The classification efficiency of the SCM and RF methods was evaluated using the following parameters: Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and the area under the receiver operating characteristic curve. Functional enrichment analysis was performed for the feature genes with DAVID. A total of 585 DEGs were identified in the training group, of which 580 were upregulated and five were downregulated. The classification accuracy rates of the two methods for the training group, the test group and the combined group using the 585 feature genes were >90%. For the SVM and RF methods, the rates of correct determination, specificity and PPV were >90%, while the sensitivity and NPV were >80%. The SVM method produced a slightly better classification effect than the RF method. A total of 13 biological pathways were overrepresented by the feature genes, mainly involving energy metabolism, chromatin organization and translation. The feature genes identified in the present study may be used to predict the mutation status of FLT3/ITD in patients with AML.

RNA editing is a post-transcriptional RNA process that provides RNA and protein complexity for regulating gene expression in eukaryotes. It is challenging to predict RNA editing by computational methods. In this study, we developed a novel method to predict RNA editing based on a random forest method. A careful feature selection procedure was performed based on the Maximum Relevance Minimum Redundancy (mRMR) and Incremental Feature Selection (IFS) algorithms. Eighteen optimal features were selected from the 77 features in our dataset and used to construct a final predictor. The accuracy and MCC (Matthews correlation coefficient) values for the training dataset were 0.866 and 0.742, respectively; for the testing dataset, the accuracy and MCC were 0.876 and 0.576, respectively. The performance was higher using 18 features than all 77, suggesting that a small feature set was sufficient to achieve accurate prediction. Analysis of the 18 features was performed and may shed light on the mechanism and dominant factors of RNA editing, providing a basis for future experimental validation.

High dimension and complexity of network high-dimensional data lead to poor feature selection effect network high-dimensional data. To effectively solve this problem, feature selection algorithms for high-dimensional network data based on supervised discriminant projection (SDP) have been designed. The sparse representation problem of high-dimensional network data is transformed into an Lp norm optimization problem, and the sparse subspace clustering method is used to cluster high-dimensional network data. Dimensionless processing is carried out for the clustering processing results. Based on the linear projection matrix and the best transformation matrix, the dimensionless processing results are reduced by combining the SDP. The sparse constraint method is used to achieve feature selection of high-dimensional data in the network, and the relevant feature selection results are obtained. The experimental findings demonstrate that the suggested algorithm can effectively cluster seven different types of data and converges when the number of iterations approaches 24. The F1 value, recall, and precision are all kept at high levels. High-dimensional network data feature selection accuracy on average is 96.9%, and feature selection time on average is 65.1 milliseconds. The selection effect for network high-dimensional data features is good.

Fire is a natural disaster that can be caused by many different reasons. Recently, more environmentally friendly and innovative extinguishing methods have started to be tested, some of which are also used. For this purpose, a sound wave fire-extinguishing system was created and firefighting tests were performed. With the data obtained, as a result of 17,442 tests, a data set was created. In this study, five different machine learning methods were used by using the data set created. These are artificial neural network, k-nearest neighbor, random forest, stacking and deep neural network methods. Stacking method is an ensemble method created by using artificial neural network, k-nearest neighbor, random forest models together. Classification of extinction and non-extinction states of the flame was made with the models created with these methods. The accuracy of models in classification should be analyzed in detail in order to be used as a decision support system in the sound wave fire-extinguishing system. Hence, the classification processes were carried out through the 10-fold cross-validation method. As a result of these tests, the performance analysis of the models was carried out, and the results showed that the highest classification accuracy belongs to the stacking model with 97.06&#x0025;. The classification accuracy was determined 96.58&#x0025; in random forest method, 96.03&#x0025; in artificial neural network model, 94.88&#x0025; in deep neural network model and 92.62&#x0025; in k-NN model. The performance of the methods was compared by analyzing the performance metrics of machine learning methods. Thanks to the decision support system to be obtained based on the results of the analyzes, the sound wave fire-extinguishing system can be used efficiently.