Feature Selection for Data Mining

Classification problem in biological Omics data has gained in popularity in recent years. In consideration of the high dimension of the Omics dataset, the importance of feature selection technology becomes apparent. Feature selection can not only improve the classification accuracy but also avoid overfitting which is a common issue in machine learning. There are three main categories of feature selection methods: filter methods, wrapper methods, and hybrid methods. Yet it is difficult for researchers to choose among them. In this study, we conducted a comprehensive comparison for filter methods, wrapper methods, and hybrid methods. Specifically, we selected information gain as the filter method, genetic algorithm, and binary particle swarms optimization as the wrapper methods, IG-GA and IG-BPSO as the hybrid methods for comparison. The experimental results show that the IG-BPSO, a hybrid method, has the highest classification accuracy among these feature selection methods on three Omics datasets.

In many applications, like function approximation, pattern recognition, time series prediction, and data mining, one has to build a model relating some features describing the data to some response value. Often, the features that are relevant for building the model are not known in advance. Feature selection methods allow removing irrelevant and/or redundant features to only keep the feature subset that are most useful to build a prediction model. The model is simpler and easier to interpret, reducing the risks of overfitting, non-convergence, etc. By contrast with other dimensionality reduction techniques such as principal component analysis or more recent nonlinear projection techniques (Lee & Verleysen 2007), which build a new, smaller set of features, the features that are selected by feature selection methods preserve their initial meaning, potentially bringing extra information about the process being modeled (Guyon 2006). Recently, the advent of high-dimensional data has raised new challenges for feature selection methods, both from the algorithmic point of view and the conceptual point of view (Liu & Motoda 2007). The problem of feature selection is exponential in nature, and many approximate algorithms are cubic with respect to the initial number of features, which may be intractable when the dimensionality of the data is large. Furthermore, high-dimensional data are often highly redundant, and two distinct subsets of features may have very similar predictive power, which can make it difficult to identify the best subset.

A new hybrid filter/wrapper algorithm for feature selection in classification

Computing significant factors quantitatively is an imperative task in understanding the underlying reasons that contribute to a final outcome. In this paper, a case of e-Government ranking is studied by attempting to find the significance of each KPIs which leads to resultant rank of a country. Significant factors in this context are inferred as some degrees of relations between the input variables (which are the KPIs in this case) and the final outcome (the rank). In the past, significant factors were either acquired as first-hand information via direct questioning from users’ satisfaction survey or qualitative inference; typical question is ‘You are satisfied with a particular e-Government service’ by applying a multi-level Likert scale. Respondents answered by choosing one of the following: Strongly agree, Agree, Neutral, Disagree, and Strongly Disagree. The replies are then counted and studied using traditional statistical methods. In this paper, an alternative method by feature selection in data mining is proposed which computes quantitatively the relative importance of each KPI with respective to the predicted class, the rank. The main advantage of feature selection by data mining (FSDM) method is that it considers the cross-dependencies of the variables and how they contribute as a whole predictive model to a particular predicted outcome. In contrast, classical significant factor analysis such as correlogram tells only the strength of correlation between an individual pair of factor and outcome. Another advantage of using data mining method over simple statistic is that the inferred predictive model could be used as a predictor and/or what-if decision simulator; given some values of KPIs a corresponding rank could be guesstimated. A case study of computing significant factors in terms of KPIs that lead to the world rank in from the data of UN e-Government Survey 2010, is presented.

Feature selection is an important topic in data mining, especially for high dimensional dataset. Feature selection is a process commonly used in machine learning, wherein subsets of the features available from the data are selected for application of learning algorithm. The best subset contains the least number of dimensions that most contribute to accuracy. Feature selection methods can be decomposed into three main classes, one is filter method, another one is wrapper method and third one is embedded method. This chapter presents an empirical comparison of feature selection methods and its algorithm. In view of the substantial number of existing feature selection algorithms, the need arises to count on criteria that enable to adequately decide which algorithm to use in certain situation. This chapter reviews several fundamental algorithms found in the literature and assess their performance in a controlled scenario.

With the advancement of industrial digitization, utilizing large datasets for model training to boost performance is a pivotal technical approach for industry progress. However, raw training datasets often contain abundant redundant features, which increase model training's computational cost and impair generalization ability. To tackle this, this study proposes the bionic ABCCOA algorithm, an enhanced version of the bionic Coati Optimization Algorithm (COA), to improve redundant feature elimination in datasets. To address the bionic COA's inadequate global search performance in feature selection (FS) problems, leading to lower classification accuracy, an adaptive search strategy is introduced. This strategy combines individual learning capability and the learnability of disparities, enhancing global exploration. For the imbalance between the exploration and exploitation phases in the bionic COA algorithm when solving FS problems, which often traps it in suboptimal feature subsets, a balancing factor is proposed. By integrating phase control and dynamic adjustability, a good balance between the two phases is achieved, reducing the likelihood of getting stuck in suboptimal subsets. Additionally, to counter the bionic COA's insufficient local exploitation performance in FS problems, increasing classification error rates, a centroid guidance strategy is presented. By combining population centroid guidance and fractional-order historical memory, the algorithm lowers the classification error rate of feature subsets and speeds up convergence. The bionic ABCCOA algorithm was tested on the CEC2020 test functions and engineering problem, achieving an over 90% optimization success rate and faster convergence, confirming its efficiency. Applied to 27 FS problems, it outperformed comparative algorithms in best, average, and worst fitness function values, classification accuracy, feature subset size, and running time, proving it an efficient and robust FS algorithm.

Feature selection is an essential step in the preprocessing of data in pattern recognition and data mining. Nowadays, the feature selection problem as an optimization problem can be solved with nature-inspired algorithm. In this paper, we propose an efficient feature selection method based on the cuckoo search algorithm called CBCSEM. The proposed method avoids the premature convergence of traditional methods and the tendency to fall into local optima, and this efficient method is attributed to three aspects. Firstly, the chaotic map increases the diversity of the initialization of the algorithm and lays the foundation for its convergence. Then, the proposed two-population elite preservation strategy can find the attractive one of each generation and preserve it. Finally, Lévy flight is developed to update the position of a cuckoo, and the proposed uniform mutation strategy avoids the trouble that the search space is too large for the convergence of the algorithm due to Lévy flight and improves the algorithm exploitation ability. The experimental results on several real UCI datasets show that the proposed method is competitive in comparison with other feature selection algorithms.

There is a consistent rise in chronic diseases worldwide. These diseases decrease immunity and the quality of daily life. The treatment of these disorders is a challenging task for medical professionals. Dimensionality reduction techniques make it possible to handle big data samples, providing decision support in relation to chronic diseases. These datasets contain a series of symptoms that are used in disease prediction. The presence of redundant and irrelevant symptoms in the datasets should be identified and removed using feature selection techniques to improve classification accuracy. Therefore, the main contribution of this paper is a comparative analysis of the impact of wrapper and filter selection methods on classification performance. The filter methods that have been considered include the Correlation Feature Selection (CFS) method, the Information Gain (IG) method and the Chi-Square (CS) method. The wrapper methods that have been considered include the Best First Search (BFS) method, the Linear Forward Selection (LFS) method and the Greedy Step Wise Search (GSS) method. A Decision Tree algorithm has been used as a classifier for this analysis and is implemented through the WEKA tool. An attribute significance analysis has been performed on the diabetes, breast cancer and heart disease datasets used in the study. It was observed that the CFS method outperformed other filter methods concerning the accuracy rate and execution time. The accuracy rate using the CFS method on the datasets for heart disease, diabetes, breast cancer was 93.8%, 89.5% and 96.8% respectively. Moreover, latency delays of 1.08 s, 1.02 s and 1.01 s were noted using the same method for the respective datasets. Among wrapper methods, BFS’ performance was impressive in comparison to other methods. Maximum accuracy of 94.7%, 95.8% and 96.8% were achieved on the datasets for heart disease, diabetes and breast cancer respectively. Latency delays of 1.42 s, 1.44 s and 132 s were recorded using the same method for the respective datasets. On the basis of the obtained result, a new hybrid Attribute Evaluator method has been proposed which effectively integrates enhanced K-Means clustering with the CFS filter method and the BFS wrapper method. Furthermore, the hybrid method was evaluated with an improved decision tree classifier. The improved decision tree classifier combined clustering with classification. It was validated on 14 different chronic disease datasets and its performance was recorded. A very optimal and consistent classification performance was observed. The mean values for accuracy, specificity, sensitivity and f-score metrics were 96.7%, 96.5%, 95.6% and 96.2% respectively.

Feature selection plays an important role in data preprocessing. The aim of feature selection is to recognize and remove redundant or irrelevant features. The key issue is to use as few features as possible to achieve the lowest classification error rate. This paper formulates feature selection as a multi-objective problem. In order to address feature selection problem, this paper uses the multi-objective bacterial foraging optimization algorithm to select the feature subsets and k-nearest neighbor algorithm as the evaluation algorithm. The wheel roulette mechanism is further introduced to remove duplicated features. Four information exchange mechanisms are integrated into the bacteria-inspired algorithm to avoid the individuals getting trapped into the local optima so as to achieve better results in solving high-dimensional feature selection problem. On six small datasets and ten high-dimensional datasets, comparative experiments with different conventional wrapper methods and several evolutionary algorithms demonstrate the superiority of the proposed bacteria-inspired based feature selection method.

Feature selection is an important data preprocessing technique in classification problems. This paper focuses on a new feature selection problem, in which sampling data of different features have different reliability degree. First, the problem is modeled as a multi-objective optimization. There two objectives should be optimized simultaneously: reliability and classifying accuracy of feature subset. Then, a multi-objective feature selection method based on particle swarm optimization, called JMOPSO, is proposed by incorporating several effective operators. Finally, experimental results suggest that the proposed JMOPSO is a highly competitive feature selection method for solving the feature selection problem with unreliable data.

Abstract. Dissolved oxygen (DO) holds an important role in water quality monitoring due to its impact on the aquatic organisms inhabiting the water body. Sampaloc lake, located in San Pablo City in Laguna Province, Philippines, one of the major fishing lakes, has been affected by the decrease of DO concentration leading to continuous fish kill incidents. This study used supervised machine learning-based model utilizing the filter and wrapper method in feature selection to predict the DO levels in Sampaloc Lake. The goal of the study is to provide a baseline for building an early warning system for fish kills events in the lake. Water quality, physical and hydroclimatic parameters are the range of features selected as predictors to create a reliable DO predictive model. Based on the feature selection process, water temperature, atmospheric temperature, biochemical oxygen demand, surface pressure and pH are the consistent important feature in predicting DO for both filter and wrapper method. Overall, the result of the eXtreme Gradient Boosting Regression (XGBR) Model of the filter method with R² = 0.95, RMSE = 0.14, and MAE = 0.10 showed a reliable performance among the predicted models (Multiple Linear Regression, Random Forest, Kernel Ridge Regression, and Bootstrap Aggregating) tested in the study.

The explosive usage of social media produces massive amount of unlabeled and high-dimensional data. Feature selection has been proven to be effective in dealing with high-dimensional data for efficient learning and data mining. Unsupervised feature selection remains a challenging task due to the absence of label information based on which feature relevance is often assessed. The unique characteristics of social media data further complicate the already challenging problem of unsupervised feature selection, e.g., social media data is inherently linked, which makes invalid the independent and identically distributed assumption, bringing about new challenges to unsupervised feature selection algorithms. In this paper, we investigate a novel problem of feature selection for social media data in an unsupervised scenario. In particular, we analyze the differences between social media data and traditional attribute-value data, investigate how the relations extracted from linked data can be exploited to help select relevant features, and propose a novel unsupervised feature selection framework, LUFS, for linked social media data. We systematically design and conduct systemic experiments to evaluate the proposed framework on data sets from real-world social media websites. The empirical study demonstrates the effectiveness and potential of our proposed framework.

A new hybrid filter–wrapper feature selection method for clustering based on ranking

Many evolutionary computation (EC) methods have been used to solve feature selection problems and they perform well on most small-scale feature selection problems. However, as the dimensionality of feature selection problems increases, the solution space increases exponentially. Meanwhile, there are more irrelevant features than relevant features in datasets, which leads to many local optima in the huge solution space. Therefore, the existing EC methods still suffer from the problem of stagnation in local optima on large-scale feature selection problems. Furthermore, large-scale feature selection problems with different datasets may have different properties. Thus, it may be of low performance to solve different large-scale feature selection problems with an existing EC method that has only one candidate solution generation strategy (CSGS). In addition, it is time-consuming to find a suitable EC method and corresponding suitable parameter values for a given large-scale feature selection problem if we want to solve it effectively and efficiently. In this article, we propose a self-adaptive particle swarm optimization (SaPSO) algorithm for feature selection, particularly for large-scale feature selection. First, an encoding scheme for the feature selection problem is employed in the SaPSO. Second, three important issues related to self-adaptive algorithms are investigated. After that, the SaPSO algorithm with a typical self-adaptive mechanism is proposed. The experimental results on 12 datasets show that the solution size obtained by the SaPSO algorithm is smaller than its EC counterparts on all datasets. The SaPSO algorithm performs better than its non-EC and EC counterparts in terms of classification accuracy not only on most training sets but also on most test sets. Furthermore, as the dimensionality of the feature selection problem increases, the advantages of SaPSO become more prominent. This highlights that the SaPSO algorithm is suitable for solving feature selection problems, particularly large-scale feature selection problems.

Feature selection plays a critical role in improving the efficiency, accuracy, and interpretability of machine learning models, particularly when dealing with high-dimensional datasets. Among various approaches, wrapper-based feature selection methods are known for their ability to capture feature interactions by directly optimizing model performance. This study presents a comprehensive comparative analysis of six wrapper feature selection techniques—Recursive Feature Elimination (RFE), Sequential Forward Selection (SFS), Sequential Backward Selection (SBS), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and Differential Evolution (DE)—in conjunction with five widely used classification algorithms: Decision Tree, K-Nearest Neighbour, Random Forest, Logistic Regression, and Support Vector Machine. Experiments are conducted on an educational dataset comprising 395 student records with 30 attributes obtained from the UCI repository, using different feature subset sizes (all features, top 20, top 15, and top 10). Model performance is evaluated using accuracy, precision, recall, F1-score, and AUC. The results demonstrate that wrapper methods significantly enhance classification performance while reducing dimensionality, with GA and RFE consistently emerging as the most effective techniques across multiple classifiers. DE also shows strong performance, particularly with Logistic Regression and Random Forest, whereas PSO generally underperforms in terms of AUC. Furthermore, reducing the feature set does not adversely affect predictive accuracy and, in several cases, leads to improved generalization. The findings confirm the effectiveness of wrapper methods for educational data mining and provide practical insights for selecting optimal feature–classifier combinations.