Robust Feature Selection Using Ensemble Feature Selection Techniques

Identifying important biomarkers to improve disease diagnosis and treatment is a significant topic of research in bioinformatics. However, bioinformatics datasets frequently have a large number of features per sample or instance. This problem, known as “high dimensionality,” can be alleviated through the use of dimension reducing techniques such as feature (gene) selection which remove unnecessary features. There are many versions of feature selection, with varying biases and predictive abilities. However, predictive power is but one factor to consider when choosing a feature selection technique: one must also consider the technique's stability, that is, its ability to create feature subsets which remain valid in the face of changes to the data. While there has been work in determining the relative stability of different feature selection techniques, this does not always help determine whether a chosen feature selection technique will give stable feature subsets for a specific dataset. Factors such as difficulty of learning (e.g., dataset difficulty) may also influence feature selection stability, making generally-true facts about different techniques not applicable to a given dataset. In this work, we study how dataset difficulty can affect the stability of feature selection techniques, leading to good performance from bad techniques and vice versa. We use a set of twenty-six DNA microarray datasets with varying levels of difficulty of learning, along with four levels of dataset perturbation, six feature selection techniques with various levels of stability, and twelve feature subset sizes. The results show that as the dataset difficulty increases, the stability decreases. However, the relative stability between the techniques remains the same. Additionally, the more difficult the dataset, the more the stability is affected by changes to the data. We also found that unstable rankers are more affected by the transition between Easy and Moderate datasets, whereas the stable techniques are more affected by the change between Moderate and Hard datasets. Lastly, as the feature subset size increases, the stability increases and the difference between the levels of dataset difficulty decreases. Overall, we conclude that difficulty of learning must be taken into account before interpreting stability results.

Innovations in machine learning algorithms have enhanced the effectiveness of malware detection systems during the previous decades. However, the advancement of high throughput technologies results in high dimensional malware data, making feature selection useful and mandatory in such datasets. The feature selection technique is an information retrieval tool that aims to improve classifiers by listing important features, which also aids in reducing computational overload. However, different feature selection algorithms select representative features using various criteria, making it difficult to determine the optimal technique for distinct domain datasets. Ensemble feature selection approaches, which integrate the results of several feature selections, can be used to overcome the inadequacies of single-feature selection methods. Therefore, this paper attempts to determine whether the heterogeneous ensemble of filter and embedded feature selection approaches, namely the heterogenous ensemble of ANOVA F-test, ReliefF, L1-penalized logistic regression, LASSO regression, Extra-Tree Classifier and XGBoost feature selection techniques, namely HEFS-ARLLEX, can provide a better classification performance than the single feature selection technique and other ensemble feature selection approaches for malware classification data. The experimental results show that HEFS-ARLLEX, which combines both filters and embedded, is a better choice, providing consistently high classification accuracy, recall, precision, specificity and F-measure and a reasonable feature reduction rate for malware classification dataset.

Ensemble gene (feature) selection is a promising new strategy with many benefits including more stable gene lists and improved classification results. The ensemble portion is achieved through multiple runs of feature selection which are then aggregated into a single result. The critical question is how many iterations of feature selection are appropriate. Too few iterations can make classification performance suffer. However, too many iterations will cause issues regarding computational costs. The goal is to choose the correct number of iterations to maximize classification performance without expending too much computational power. Our paper is an in-depth study on the effect of the number of iterations of feature selection on classification performance. Our work employs eleven DNA microarray datasets, on which we apply various ensemble methods, feature selection techniques, classifiers, and feature subset sizes. The results show that using 10 iterations of feature ranking during ensemble feature selection is not sufficient to optimize classification results and that a larger number of iterations is required (20 or 50). However, there is very little distinction between 20 iterations and 50 iterations, as both produce very similar classification results. Our recommendation is to use 20 iterations because while 20 iterations and 50 iterations perform similarly, 20 iterations has a much smaller computation time. To our knowledge there has not been a previous study as expansive as this one on the effects of the number of iterations of feature selection on ensemble feature selection.

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

Text mining is a special case of data mining which explore unstructured or semi-structured text documents, to establish valuable patterns and rules that indicate trends and significant features about specific topics. Text mining has been in pattern recognition, predictive studies, sentiment analysis and statistical theories in many areas of research, medicine, financial analysis, social life analysis, and business intelligence. Text mining uses concept of natural language processing and machine learning. Machine learning algorithms have been used and reported to give great results, but their performance of machine learning algorithms is affected by factors such as dataset domain, number of classes, length of the corpus, and feature selection techniques used. Redundant attribute affects the performance of the classification algorithm, but this can be reduced by using different feature selection techniques and dimensionality reduction techniques. Feature selection is a data preprocessing step that chooses a subset of input variable while eliminating features with little or no predictive information. Feature selection techniques are Information gain, Term Frequency, Term Frequency-Inverse document frequency, Mutual Information, and Chi-Square, which can use a filters, wrappers, or embedded approaches. To get the most value from machine learning, pairing the best algorithms with the right tools and processes is necessary. Little research has been done on the effect of feature selection techniques on classification accuracy for pairing of these algorithms with the best feature selection techniques for optimal results. In this research, a text classification experiment was conducted using incident management dataset, where incidents were classified into their resolver groups. Support vector machine (SVM), K-Nearest Neighbors (KNN), Naïve Bayes (NB) and Decision tree (DT) machine learning algorithms were examined. Filtering approach was used on the feature selection techniques, with different ranking indices applied for optimal feature set and classification accuracy results analyzed. The classification accuracy results obtained using TF were, 88% for SVM, 70% for NB, 79% for Decision tree, and KNN had 55%, while Boolean registered 90%, 83%, 82% and 75%, for SVM, NB, DT, and KNN respectively. TF-IDF, had 91%, 83%, 76%, and 56% for SVM, NB, DT, and KNN respectively. The results showed that algorithm performance is affected by feature selection technique applied. SVM performed best, followed by DT, KNN and finally NB. In conclusion, presence of noisy data leads to poor learning performance and increases the computational time. The classifiers performed differently depending on the feature selection technique applied. For optimal results, the classifier that performed best together with the feature selection technique with the best feature subset should be applied for all types of data for accurate classification performance. Keywords: Text Classification, Supervised Machine Learning, Feature Selection DOI : 10.7176/JIEA/9-3-06 Publication date :May 31 st 2019

Text clustering involves data that are of very high dimension. Feature selection techniques find subsets of relevant features from the original feature space that help in efficient and effective clustering. Selection of relevant features merely on ranking scores without considering correlation interferes with the clustering performance. An efficient feature selection technique should be capable of preserving the multi-cluster structure of the data. The purpose of the present work is to demonstrate that feature selection techniques which take into consideration the correlation among features in multi-cluster scenario show better clustering results than those techniques that simply rank features independent of each other. This paper compares two feature selection techniques in this regard viz. the traditional Tf-Idf and the Multi-Cluster Feature Selection (MCFS) technique. The experimental results over the TDT2 and Reuters-21,578 datasets show the superior clustering results of MCFS over traditional Tf-Idf.

The investigation of potential microarray markers, which in turn will speed up the molecular analysis and provide reliable results on the benefit of patient care is of significant importance. Feature selection techniques, which aim at minimizing the dimensionality of the microarray data by keeping the most significant genes according to their expression values is a necessary component towards this goal. In the current article, we present an investigation regarding the minimum required subsets of genes, which best classify myopathy data. For this purpose, we developed a tool that facilitates the users to easily access/use multiple feature selection methods and subsequently perform classification of data. For the current study, five feature selection methods on datasets from two different myopathies have been utilized. Our findings reveal subsets of very small number of genes, which can successfully classify gene expression datasets from different patients with skeletal myopathies. In addition, we observe that similar classification results may be obtained from completely different subsets of genes. The developed tool can expedite the identification of small gene subsets with high classification accuracy that could ultimately be used in the genetics clinics for diagnostic, prognostic and pharmacogenomic purposes.

As climate change intensifies, the frequency and severity of waterlogging are expected to increase, necessitating a deeper understanding of the cucumber response to this stress. In this study, three public RNA-seq datasets (PRJNA799460, PRJNA844418, and PRJNA678740) comprising 36 samples were analyzed. Various feature selection algorithms including Uncertainty, Relief, SVM (Support Vector Machine), Correlation, and logistic least absolute shrinkage, and selection operator (LASSO) were performed to identify the most significant genes related to the waterlogging stress response. These feature selection techniques, which have different characteristics, were used to reduce the complexity of the data and thereby identify the most significant genes related to the waterlogging stress response. Uncertainty, Relief, SVM, Correlation, and LASSO identified 4, 4, 10, 21, and 13 genes, respectively. Differential gene correlation analysis (DGCA) focusing on the 36 selected genes identified changes in correlation patterns between the selected genes under waterlogged versus control conditions, providing deeper insights into the regulatory networks and interactions among the selected genes. DGCA revealed significant changes in the correlation of 13 genes between control and waterlogging conditions. Finally, we validated 13 genes using the Random Forest (RF) classifier, which achieved 100% accuracy and a 1.0 Area Under the Curve (AUC) score. The SHapley Additive exPlanations (SHAP) values clearly showed the significant impact of LOC101209599, LOC101217277, and LOC101216320 on the model’s predictive power. In addition, we employed the Boruta as a wrapper feature selection method to further validate our gene selection strategy. Eight of the 13 genes were common across the four feature weighting algorithms, LASSO, DGCA, and Boruta, underscoring the robustness and reliability of our gene selection strategy. Notably, the genes LOC101209599, LOC101217277, and LOC101216320 were among genes identified by multiple feature selection methods from different categories (filtering, wrapper, and embedded). Pathways associated with these specific genes play a pivotal role in regulating stress tolerance, root development, nutrient absorption, sugar metabolism, gene expression, protein degradation, and calcium signaling. These intricate regulatory mechanisms are crucial for cucumbers to adapt effectively to waterlogging conditions. These findings provide valuable insights for uncovering targets in breeding new cucumber varieties with enhanced stress tolerance.

Non-Intrusive Load Monitoring (NILM) aids in estimating the individual energy consumed by different appliances using aggregate signals measured at point of common connection. This paper explores various unexplored aspects concerning load classification in NILM. Rich feature sets are extracted from load signatures using auto-correlogram, Variational Mode Decomposition (VMD), and Wavelet Packet Tree (WPT) decomposition. Typically, previous works employ a single Feature Selection (FS) technique to earmark the relevant features for improved performance of NILM. However, each FS technique behaves uniquely, and the effects of using different FS techniques or their combinations have yet to be explored for NILM. Hence, Combinatorial Fusion Analysis (CFA), which surpasses the performance of a single FS technique by considering an optimal combination of multiple FS techniques, is used in this paper. The final selected features are fused and provided to a classifier. Rather than manually tuning or performing an exhaustive search for setting the classifier’s hyper-parameters, as prevalent in previous works, the present work utilizes Bayesian Optimization (BO) comprising surrogate and acquisition functions. The proposed work’s efficacy is validated on publicly available data-sets in a host computer and an embedded device, namely, R-Pi, via comparison with previous works.

Continuous blood pressure (BP) provides essential information for monitoring one's health condition. However, BP is currently monitored using uncomfortable cuff-based devices, which does not support continuous BP monitoring. This paper aims to introduce a blood pressure monitoring algorithm based on only photoplethysmography (PPG) signals using the deep neural network (DNN). The PPG signals are obtained from 125 unique subjects with 218 records and filtered using signal processing algorithms to reduce the effects of noise, such as baseline wandering, and motion artifacts. The proposed algorithm is based on pulse wave analysis of PPG signals, extracted various domain features from PPG signals, and mapped them to BP values. Four feature selection methods are applied and yielded four feature subsets. Therefore, an ensemble feature selection technique is proposed to obtain the optimal feature set based on major voting scores from four feature subsets. DNN models, along with the ensemble feature selection technique, outperformed in estimating the systolic blood pressure (SBP) and diastolic blood pressure (DBP) compared to previously reported approaches that rely only on the PPG signal. The coefficient of determination ( ) and mean absolute error (MAE) of the proposed algorithm are 0.962 and 2.480 mmHg, respectively, for SBP and 0.955 and 1.499 mmHg, respectively, for DBP. The proposed approach meets the Advancement of Medical Instrumentation standard for SBP and DBP estimations. Additionally, according to the British Hypertension Society standard, the results attained Grade A for both SBP and DBP estimations. It concludes that BP can be estimated more accurately using the optimal feature set and DNN models. The proposed algorithm has the potential ability to facilitate mobile healthcare devices to monitor continuous BP.

In the past decade, big data has become increasingly prevalent in a large number of applications. As a result, datasets suffering from noise and redundancy issues have necessitated the use of feature selection across multiple domains. However, a common concern in feature selection is that different approaches can give very different results when applied to similar datasets. Aggregating the results of different selection methods helps to resolve this concern and control the diversity of selected feature subsets. In this work, we implemented a general framework for the ensemble of multiple feature selection methods. Based on diversified datasets generated from the original set of observations, we aggregated the importance scores generated by multiple feature selection techniques using two methods: the Within Aggregation Method (WAM), which refers to aggregating importance scores within a single feature selection; and the Between Aggregation Method (BAM), which refers to aggregating importance scores between multiple feature selection methods. We applied the proposed framework on 13 real datasets with diverse performances and characteristics. The experimental evaluation showed that WAM provides an effective tool for determining the best feature selection method for a given dataset. WAM has also shown greater stability than BAM in terms of identifying important features. The computational demands of the two methods appeared to be comparable. The results of this work suggest that by applying both WAM and BAM, practitioners can gain a deeper understanding of the feature selection process.

Tuberculosis is one of the leading causes of millions of deaths across the world, mainly due to growth of drug-resistant strains. Anti-tubercular peptides may facilitate an alternate way to combat antibiotic tolerance. This study describes a novel approach for enhancing the prediction of anti-tubercular peptides by feature extraction from sequence of the peptides, selection of optimal features from the extracted features, and selection of suitable learning algorithm. Firstly, we extract different sequence features by using iFeature web server. Then, the optimal features are obtained by using a novel divergence measure-based intuitionistic fuzzy rough sets-assisted feature selection technique. Furthermore, an attempt has been made to develop models using different machine learning techniques for enhancing the prediction of anti-tubercular (or anti-mycobacterial peptides) with other antibacterial peptides (ABP) as well non-antibacterial peptides (non-ABP). Moreover, the best prediction result is obtained by vote-based classifier. Using 80:20 percentage split, the proposed method performs well, with sensitivity of 92.0%, 96.4%, specificity of 83.3%, 88.4%, overall accuracy of 87.80%, 92.90%, Mathews correlation coefficient of 0.757, 0.857, AUC of 0.922, 0.914, and g-means of 87.5%, 92.3% for anti-tubercular and ABP (primary dataset), anti-tubercular and non-ABP (secondary dataset), respectively. Finally, we have evaluated the performances of different machine learning algorithms by using the reduced training sets as produced by our proposed feature selection technique as well as already existing intuitionistic fuzzy rough set based and ensemble feature selection technique. Moreover, the performance of our proposed approach is evaluated on few benchmark and AMP datasets. From the experimental results, it can be observed that our proposed method is outperforming the previous methods.

Introduction: The early and accurate detection of COVID-19 remains a life-threatening challenge in medical analysis. Machine learning is used for predicting disease outcomes based on clinical parameters. This analysis proposes a comparative analysis of feature selection method and classification techniques to enhance COVID-19 detection accuracy using blood biomarkers. We used a pensourse dataset of 1,724 cases, including 35 features. To improve the model performance data preprocessing process included outlier handling, normalization, and transformation techniques to improve model performance. To identify the relevant features, we employed the three-feature selection methods Chi-Square test, Pearson correlation coefficient, and Random Forest. The model prediction accuracy was enhanced using a stacking ensemble classication techniques. The machine learning based classification models effectively predicted COVID-19 infectious disease using blood biomarkers with optimized feature selection techniques. Objectives: To enhance the accuracy of COVID-19 prediction using machine learning techniques by applying feature selection and classification techniques on blood biomarkers. Methods: The comparative analysis utilized a publicly available dataset containing 1,724 cases with 35 attributes. Data preprocessing involved outlier handling, normalization, and transformation techniques. Employed Chi-Square test, Pearson correlation coefficient, and Random Forest feature selection techniques. Stacking ensemble classification algorithm was utilized for the better performance of a model. Results: The classification models demonstrated efficiency in predicting COVID-19 using blood biomarkers. Optimized feature selection significantly improved predictive accuracy, highlighting the importance of selecting relevant features for model performance enhancement. Conclusions: This study showcases the potential of ML-driven approaches for COVID-19 detection, emphasizing the role of feature selection in improving classification accuracy. The findings contribute to the advancement of diagnostic tools, offering a data-driven solution for rapid and reliable COVID-19 screening.

Credit risk assessment is an essential task for financial institutions, as it evaluates the likelihood of loan defaults. Misclassifying loan applications can result in substantial economic losses. However, accurately predicting customers' creditworthiness presents challenges for machine learning models due to high dimensionality and imbalanced class distributions, which can severely impact overall model performance. This implements three feature selection (FS) techniques, such as Random Forest with Highly Correlated features (RFHC), Information Gain, and Chi-Square to address the high dimensional problem. Additionally, SMOTE is implemented to address the imbalanced dataset. The model performance is analyzed using three different approaches. First, employing only feature selection techniques, in the second approach, we employed feature selection followed by oversampling, and in the final approach, oversampling prior to feature selection is employed. The experimental results indicate that implementing feature selection followed by oversampling gives better performance compared to the other two approaches. Among the feature selection techniques, RFHC outperforms the others, achieving a highest F1-score of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{95.12 \%}$</tex> and an AUC value of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{85.76 \%}$</tex>.

Last century, the challenge was to develop new technologies that store large amount of data. Recently, the challenges are to effectively utilize the incredible amount of data and to obtain knowledge that benefits business, scientific, and government transactions by using subset of features rather than the whole features in the dataset. In the present paper, we have focused on feature selection techniques as a method to gain high quality attributes to enhance the mining process. Feature selection techniques touch all disciplines that require knowledge discovery from large data. In our study, we made a comparison between benchmark feature selection methods based on three well-known datasets and three well-recognized machine learning algorithms. The study found that feature selection methods are capable to improve the performance of learning algorithms. However, no single feature selection methods that best satisfy all datasets and learning algorithms. Therefore, machine learning researchers should understand the nature of datasets and learning algorithms characteristics in order to obtain better outcome. Overall, correlation based feature selection (CFS) and consistency based subset evaluation (CB) performed better than information gain, symmetrical uncertainty, Relief (RF), and principle components analysis (PC).