EFS: an ensemble feature selection tool implemented as R-package and web-application

Software defect prediction models are used for predicting high risk software components. Feature selection has significant impact on the prediction performance of the software defect prediction models since redundant and unimportant features make the prediction model more difficult to learn. Ensemble feature selection has recently emerged as a new methodology for enhancing feature selection performance. This paper proposes a new multi-criteria-decision-making (MCDM) based ensemble feature selection (EFS) method. This new method is termed as MCDM-EFS. The proposed method, MCDM-EFS, first generates the decision matrix signifying the feature’s importance score with respect to various existing feature selection methods. Next, the decision matrix is used as the input to well-known MCDM method TOPSIS for assigning a final rank to each feature. The proposed approach is validated by an experimental study for predicting software defects using two classifiers K-nearest neighbor (KNN) and naïve bayes (NB) over five open-source datasets. The predictive performance of the proposed approach is compared with existing feature selection algorithms. Two evaluation metrics – nMCC and G-measure are used to compare predictive performance. The experimental results show that the MCDM-EFS significantly improves the predictive performance of software defect prediction models against other feature selection methods in terms of nMCC as well as G-measure.

A hybrid ensemble feature selection design for candidate biomarkers discovery from transcriptome profiles

On the benefit of feature selection and ensemble feature selection for fuzzy k-nearest neighbor classification

Human activity recognition (HAR) based on sensors has been widely used in many fields. Instead of using multi-sensor system which is not convenient in practical applications and requires high computational cost, this paper utilizes a single wearable accelerometer to collect human activity information. In order to improve the recognition performance of the whole system and select the features that are most relevant to the wearing position of sensor, the wavelet decomposition-based features and a novel feature selection method are introduced. Considering the limitation of single filter feature selection method, this paper proposes an ensemble-based filter feature selection (EFFS) approach to optimize the feature set. Experiment results show that the wavelet decomposition-based features can increase the discrimination of activities and significantly and improve the activity recognition accuracy. Compared with other four popular feature selection methods, the proposed EFFS approach provides higher accuracy with fewer features.

This study discusses the problem of feature selection as one of the most fundamental problems in the field of the machine learning. Two novel approaches for feature selection in order to select a subset with relevant features are proposed. These approaches can be considered as a direct extension of the ensemble feature selection approach. The first one deals with identifying relevant features by using a single feature selection method. While, the second one uses different feature selection methods in order to identify more correctly the relevant features. An illustration shows the effectiveness of the proposed methods on artificial databases where we have a priori the informations about the relevant features.KeywordsFeature SelectionRelevant FeatureFeature SubsetFeature Selection MethodNeural Information Processing SystemThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

The use of ensemble paradigm with classifiers is a proven approach that involves combining the outcomes of several classifiers. It has recently been extrapolated to feature selection methods to find the most relevant features. Earlier, ensemble feature selection has been used in high dimensional, low sample size datasets like bioinformatics. To one's knowledge there is no such endeavor in the text classification domain. In this work, the ensemble feature selection using data perturbation in the text classification domain has been used with an aim to enhance predictability and stability. This approach involves application of the same feature selector to different perturbed versions of training data, obtaining different ranks for a feature. Previous works focus only on one of the metrics, that is, stability or accuracy. In this work, a combined framework is adopted that assesses both the predictability and stability of the feature selection method by using feature selection ensemble. This approach has been explored on univariate and multivariate feature selectors, using two rank aggregators.

MIC-SHAP: An ensemble feature selection method for materials machine learning

Ensemble feature selection algorithms aggregate the results of multiple feature selection methods in order to select an effective subset of features. However, typically, ensemble algorithms treat each feature selection method equally and do not consider performance differences. Consequently, features selected by a relatively smaller number of methods may not be included. To address this problem, we propose an interactive feature selection method that can more effectively aggregate the results of multiple feature selection methods and iteratively improve the selected features by integrating expert knowledge. The proposed method includes a learning-from-crowds-based ensemble feature selection algorithm and a visual analysis system. The algorithm models the performance of multiple feature selection methods, calculates their reliabilities, and aggregates results. To integrate expert knowledge, the visual analysis system provides a set of ranking schemes to assist experts in understanding the results of an individual feature selection method and the roles played by the features in classification tasks. A numerical experiment conducted on four real-world datasets shows that the proposed algorithm can improve classification accuracy by 0.63%–2.85% compared to state-of-the-art ensemble feature selection algorithms. In addition, we conducted case studies on text and image data to demonstrate that the proposed visual analysis system can further improve classification accuracy by 0.28%–5.24%.

The right security solutions in the information and communication world are critical to network security by providing real-time network protection against network vulnerabilities and data usage. An effective intrusion detection strategy is able to use a holistic approach to protect critical systems from unauthorized access or attacks. The paper examines the latest scientific achievements and research related to the analysis of detection of network intrusions using machine learning (ML) methods. The article describes a complex security solution based on machine learning (ML) for network intrusion detection using a complex controlled ML structure and ensemble feature selection methods. In addition, a comparative analysis of several MH models and function selection methods is provided. The article develops a general mechanism for detecting and achieving higher accuracy with a minimum frequency of false positive results (FPR). The paper uses datasets and the results show that the detection model can successfully identify 99.3% of intrusions with the lowest error rate of 0.5%, which shows better performance compared to existing solutions. The article combines the selection of ensemble functions and ensemble machine learning approaches as a detection mechanism in SBB to detect network anomalies. An experimental study was conducted with feature sets obtained from nine feature selection methods, and then these feature sets were combined to obtain the minimum number of features using majority voting. A comparative analysis of sets of functions was carried out. Controlled methods are used, which are more efficient with a balanced data set. To make the training dataset balanced, the data type (benign or attacking) with the minimum number of data instances in that training dataset was first selected. An ensemble feature selection and ensemble classification algorithm is implemented to improve the overall performance of the proposed machine learning model. Prospects for the development of further research are proposed.

Ensemble gene (feature) selection is a promising new strategy with many benefits including more stable gene lists and improved classification results. The ensemble portion is achieved through multiple runs of feature selection which are then aggregated into a single result. The critical question is how many iterations of feature selection are appropriate. Too few iterations can make classification performance suffer. However, too many iterations will cause issues regarding computational costs. The goal is to choose the correct number of iterations to maximize classification performance without expending too much computational power. Our paper is an in-depth study on the effect of the number of iterations of feature selection on classification performance. Our work employs eleven DNA microarray datasets, on which we apply various ensemble methods, feature selection techniques, classifiers, and feature subset sizes. The results show that using 10 iterations of feature ranking during ensemble feature selection is not sufficient to optimize classification results and that a larger number of iterations is required (20 or 50). However, there is very little distinction between 20 iterations and 50 iterations, as both produce very similar classification results. Our recommendation is to use 20 iterations because while 20 iterations and 50 iterations perform similarly, 20 iterations has a much smaller computation time. To our knowledge there has not been a previous study as expansive as this one on the effects of the number of iterations of feature selection on ensemble feature selection.

Federated learning (FL) has emerged as a significant method for developing machine learning models across multiple devices without centralized data collection. Candidemia, a critical but rare disease in ICUs, poses challenges in early detection and treatment. The goal of this study is to develop a privacy-preserving federated learning framework for predicting candidemia in ICU patients. This approach aims to enhance the accuracy of antifungal drug prescriptions and patient outcomes. This study involved the creation of four predictive FL models for candidemia using data from ICU patients across three hospitals in China. The models were designed to prioritize patient privacy while aggregating learnings across different sites. A unique ensemble feature selection strategy was implemented, combining the strengths of XGBoost’s feature importance and statistical test p values. This strategy aimed to optimize the selection of relevant features for accurate predictions. The federated learning models demonstrated significant improvements over locally trained models, with a 9% increase in the area under the curve (AUC) and a 24% rise in true positive ratio (TPR). Notably, the FL models excelled in the combined TPR + TNR metric, which is critical for feature selection in candidemia prediction. The ensemble feature selection method proved more efficient than previous approaches, achieving comparable performance. The study successfully developed a set of federated learning models that significantly enhance the prediction of candidemia in ICU patients. By leveraging a novel feature selection method and maintaining patient privacy, the models provide a robust framework for improved clinical decision-making in the treatment of candidemia.

Biomarker discovery from high-dimensional data is a crucial problem with enormous applications in biology and medicine. It is also extremely challenging from a statistical viewpoint, but surprisingly few studies have investigated the relative strengths and weaknesses of the plethora of existing feature selection methods. In this study we compare feature selection methods on public gene expression datasets for breast cancer prognosis, in terms of predictive performance, stability and functional interpretability of the signatures they produce. We observe that the feature selection method has a significant influence on the accuracy, stability and interpretability of signatures. Surprisingly, complex wrapper and embedded methods generally do not outperform simple univariate feature selection methods, and ensemble feature selection has generally no positive effect. Overall a simple Student's t-test seems to provide the best results.

Last century, the challenge was to develop new technologies that store large amount of data. Recently, the challenges are to effectively utilize the incredible amount of data and to obtain knowledge that benefits business, scientific, and government transactions by using subset of features rather than the whole features in the dataset. In the present paper, we have focused on feature selection techniques as a method to gain high quality attributes to enhance the mining process. Feature selection techniques touch all disciplines that require knowledge discovery from large data. In our study, we made a comparison between benchmark feature selection methods based on three well-known datasets and three well-recognized machine learning algorithms. The study found that feature selection methods are capable to improve the performance of learning algorithms. However, no single feature selection methods that best satisfy all datasets and learning algorithms. Therefore, machine learning researchers should understand the nature of datasets and learning algorithms characteristics in order to obtain better outcome. Overall, correlation based feature selection (CFS) and consistency based subset evaluation (CB) performed better than information gain, symmetrical uncertainty, Relief (RF), and principle components analysis (PC).

© 2020 by the authors. The main challenge of classification systems is the processing of undesirable data. Filter-based feature selection is an effective solution to improve the performance of classification systems by selecting the significant features and discarding the undesirable ones. The success of this solution depends on the extracted information from data characteristics. For this reason, many research theories have been introduced to extract different feature relations. Unfortunately, traditional feature selection methods estimate the feature significance based on either individually or dependency discriminative ability. This paper introduces a new ensemble feature selection, called fuzzy feature selection based on relevancy, redundancy, and dependency (FFS-RRD). The proposed method considers both individually and dependency discriminative ability to extract all possible feature relations. To evaluate the proposed method, experimental comparisons are conducted with eight state-of-the-art and conventional feature selection methods. Based on 13 benchmark datasets, the experimental results over four well-known classifiers show the outperformance of our proposed method in terms of classification performance and stability.