A genetic algorithm based wrapper feature selection method for classification of hyperspectral images using support vector machine

A new feature selection method based on a validity index of feature subset

Feature selection plays a critical role in improving the efficiency, accuracy, and interpretability of machine learning models, particularly when dealing with high-dimensional datasets. Among various approaches, wrapper-based feature selection methods are known for their ability to capture feature interactions by directly optimizing model performance. This study presents a comprehensive comparative analysis of six wrapper feature selection techniques—Recursive Feature Elimination (RFE), Sequential Forward Selection (SFS), Sequential Backward Selection (SBS), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and Differential Evolution (DE)—in conjunction with five widely used classification algorithms: Decision Tree, K-Nearest Neighbour, Random Forest, Logistic Regression, and Support Vector Machine. Experiments are conducted on an educational dataset comprising 395 student records with 30 attributes obtained from the UCI repository, using different feature subset sizes (all features, top 20, top 15, and top 10). Model performance is evaluated using accuracy, precision, recall, F1-score, and AUC. The results demonstrate that wrapper methods significantly enhance classification performance while reducing dimensionality, with GA and RFE consistently emerging as the most effective techniques across multiple classifiers. DE also shows strong performance, particularly with Logistic Regression and Random Forest, whereas PSO generally underperforms in terms of AUC. Furthermore, reducing the feature set does not adversely affect predictive accuracy and, in several cases, leads to improved generalization. The findings confirm the effectiveness of wrapper methods for educational data mining and provide practical insights for selecting optimal feature–classifier combinations.

A new hybrid filter/wrapper algorithm for feature selection in classification

Classification problem in biological Omics data has gained in popularity in recent years. In consideration of the high dimension of the Omics dataset, the importance of feature selection technology becomes apparent. Feature selection can not only improve the classification accuracy but also avoid overfitting which is a common issue in machine learning. There are three main categories of feature selection methods: filter methods, wrapper methods, and hybrid methods. Yet it is difficult for researchers to choose among them. In this study, we conducted a comprehensive comparison for filter methods, wrapper methods, and hybrid methods. Specifically, we selected information gain as the filter method, genetic algorithm, and binary particle swarms optimization as the wrapper methods, IG-GA and IG-BPSO as the hybrid methods for comparison. The experimental results show that the IG-BPSO, a hybrid method, has the highest classification accuracy among these feature selection methods on three Omics datasets.

Chapter 14 - Review on hybrid feature selection and classification of microarray gene expression data

In the field of hand gesture recognition based on electromyography, the selection of features is still a matter of research because the accuracy of feature evaluation techniques depends on the distribution underling a feature space. Currently, there are a lot of kinds of features, so a selection method is required in order to find an optimal feature subset. The selection of features can be done based on either classifier methods or statistical measurement indexes. On one hand, the use of classifier methods takes considerable time, consequently, they are computationally expensive. On the other hand, the use of a statistical index allows to select features based on properties, and they are less expensive in terms of time and computational costs. In this context, this study analyzed different statistical indexes to discover optimal feature subsets and predicts which one may have the best performance in terms of classification and recognition accuracy. For this purpose, four indexes were analyzed: the sum of RES (separation and compactness index), the sum of Fuzzy Entropies, and both of these indexes divided by the mutual information of feature subset. These index values were calculated for 10 different feature subsets obtained from nineteen time-domain features which have been investigated by many researchers for extracting useful information from an electromyography signal. The feature subsets were implemented in four classifiers (Artificial Neural Network, k-nearest Neighbors Algorithm, Support Vector Machine, and decision trees) to find the best statistical index which tries to quantify the suitableness of the feature space. Each tested index in this work was analyzed in terms of classification and recognition accuracy for each machine learning algorithm.KeywordsHand gesture recognitionElectromyographyFeature evaluationFuzzy entropyMutual informationMyo armband sensor

Automatic recognition of skin symptom plays an importance role in the skin diagnosis and treatment. Feature selection is to increase the classification performance of skin symptom. In this paper, the effects of feature selection on the classification of 4-class skin symptoms (chloasma, blackhead, freckled and comedone) are analyzed. Support vector machine (SVM) is employed to construct classifier, the parameters setting is very crucial to the learning results and generalization ability of SVM, and many irrelevant and redundant features degrade the performance of classification, furthermore, the choice of SVM parameters is influenced by the feature subset taken into account and vice versa. So a joint optimization method for selecting features and SVM parameters is proposed. Genetic algorithms, which have a lot of improvements about fitness, crossover and mutation in comparison with simple genetic algorithms, is to achieve a balance between the classification accuracy and the size of the feature subsets selected and have a fast convergence rate. Furthermore, ‘one-against-one’ approach is used to solve multi-class classification problems and a comparison of the model performances is made among the results of the joint optimization and the single optimization of features or SVM parameters in the experiments. The simulation result shows that our approach can find subsets of features for 4-class skin symptoms classification with higher average cross validation accuracy and a higher rate of convergence compared with single optimization of features or SVM parameters.

In text classification, feature selection is essential to improve the classification effectiveness. This paper provides an empirical study of a feature selection method based on genetic algorithms for different text representation methods. This feature selection algorithm can accomplish two goals: in one hand is the search of a feature subset such that the performance of classifier is best; in other hands is find a feature subset with the smallest dimensionality which achieves higher accuracy in classification. To evaluate the performance of this approach, three from the best classifiers have been selected: Naive Bayes (NB), Nearest Neighbors (KNN) and Support Vector Machines (SVMs). Our objective is to determine whether the genetic algorithms based feature selection will improve the performances in text classification with smaller size using F-measure. Experimentations were carried out on two benchmark document collections 20Newsgroups, and Reuters-21578. And the results were very interesting.

We present a detailed study on the classification of urban hyperspectral data with morphological profiles (MP). Although such a spectral-spatial classification approach may significantly increase achieved accuracy, the computational complexity as well as the increased dimensionality and redundancy of such data sets are potential drawbacks. This can be overcome by feature selection. Moreover it is useful to derive detailed information on the contribution of different components from MP to the classification accuracy by evaluating these subsets. We apply a wrapper approach for feature selection based on support vector machines (SVM) with sequential feature forward selection (FFS) search strategy to two hyperspectral data sets that contain the first principal components (PC) and various corresponding MP from an urban area. In doing so, we identify feature subsets of increasing size that perform best in terms of kappa for the given setup. Results clearly demonstrate that maximum classification accuracies are achieved already on small feature subsets with few morphological profiles.

Objectives: To develop a machine learning-based model to select the most important features from a high-dimensional dataset to classify the patterns at high accuracy and reduce their dimensionality. Methods: The proposed feature selection method (FSIFC) forms and combines feature clusters incrementally and produces feature subsets each time. The method uses K-means clustering and Mutual Information (MI) to refine the feature selection process iteratively. Initially, two clusters of features are formed using K-means clustering (K=2) by taking features as the basis of clustering instead of taking the patterns (a traditional way). From these two clusters, the features with the highest MI value in each cluster are kept in a feature subset. Classification accuracies (CA) of the feature subset are calculated using three classifiers namely Support Vector Machines (SVM), Random Forest (RF), and k-nearest Neighbor (knn). The process is repeated by incrementing the value of K i.e. number of clusters; until a maximum user-defined value of K is reached. The best value of CA obtained from these trials is recorded and the corresponding feature set is finally accepted. Findings: The proposed method is demonstrated using ten datasets and the results are compared with the existing published results using three classifiers to determine the method's performance. The ten datasets are classified with average CAs of 92.72%, 93.13%, and 91.5%, using the SVM, RF, and K-NN classifiers respectively. The proposed method selects a maximum of thirty features from the datasets. In terms of selecting the most effective and the smallest feature sets, the proposed method outperforms eight other feature selection methods considering CAs. Novelty: The proposed model applies feature reduction using combined feature clustering and filter methods in an incremental way. This provides an improved selection of relevant features while removing those which are irrelevant at different trials. Keywords: Feature selection, High-dimensional datasets, K-means algorithm, Mutual information, Machine learning

This study is aimed at assessing the usefulness of different feature selection and classification methodologies in the context of sleep apnea hypopnea syndrome (SAHS) detection. Feature extraction, selection and classification stages were applied to analyze blood oxygen saturation (SaO2) recordings in order to simplify polysomnography (PSG), the gold standard diagnostic methodology for SAHS. Statistical, spectral and nonlinear measures were computed to compose the initial feature set. Principal component analysis (PCA), forward stepwise feature selection (FSFS) and genetic algorithms (GAs) were applied to select feature subsets. Fisher's linear discriminant (FLD), logistic regression (LR) and support vector machines (SVMs) were applied in the classification stage. Optimum classification algorithms from each combination of these feature selection and classification approaches were prospectively validated on datasets from two independent sleep units. FSFS + LR achieved the highest diagnostic performance using a small feature subset (4 features), reaching 83.2% accuracy in the validation set and 88.7% accuracy in the test set. Similarly, GAs + SVM also achieved high generalization capability using a small number of input features (7 features), with 84.2% accuracy on the validation set and 84.5% accuracy in the test set. Our results suggest that reduced subsets of complementary features (25% to 50% of total features) and classifiers with high generalization ability could provide high-performance screening tools in the context of SAHS.

Support vector machines (SVM) are attractive for the classification of remotely sensed data with some claims that the method is insensitive to the dimensionality of the data and, therefore, does not require a dimensionality-reduction analysis in preprocessing. Here, a series of classification analyses with two hyperspectral sensor data sets reveals that the accuracy of a classification by an SVM does vary as a function of the number of features used. Critically, it is shown that the accuracy of a classification may decline significantly (at 0.05 level of statistical significance) with the addition of features, particularly if a small training sample is used. This highlights a dependence of the accuracy of classification by an SVM on the dimensionality of the data and, therefore, the potential value of undertaking a feature-selection analysis prior to classification. Additionally, it is demonstrated that, even when a large training sample is available, feature selection may still be useful. For example, the accuracy derived from the use of a small number of features may be noninferior (at 0.05 level of significance) to that derived from the use of a larger feature set providing potential advantages in relation to issues such as data storage and computational processing costs. Feature selection may, therefore, be a valuable analysis to include in preprocessing operations for classification by an SVM.

In network security, intrusion detection plays an important role. Feature subsets obtained by different feature selection methods will lead to different accuracy of intrusion detection. Using individual feature selection method can be unstable in different intrusion detection scenarios. In this paper, the idea of ensemble is applied to feature selection to adjust feature subsets. Feature selection is converted into a two-category problem, and odd number of feature selection methods is used for voting method to decide whether a feature is required or discarded. In actual operation, mean decrease impurity, random forest classifier, stability selection, recursive feature elimination and chi-square test are used. Feature subsets obtained from them will be adjusted by our proposed method to get ensemble feature subsets. To test the performance, support vector machine, decision tree, knn and multi-layer perception are used to observe and compare the classification accuracy with ensemble feature subsets. Three intrusion detection data sets, including kddcup99, cidds-001 and unsw_nb15 are used in our experiments. The best result is reflected on cidds-001 with a 99.40% classification accuracy. The investigation shows that our method has a certain improvement on classification accuracy of intrusion detection.

In many pattern recognition applications feature selection and instance selection can be used as two data preprocessing methods that aim at reducing the computational cost of the learning process. Moreover, in some cases, feature subset selection can improve the classification performance. Feature selection and instance selection can be interesting since the choice of features and instances greatly influence the performance of the learnt models as well as their training costs. In the past, unifying both problems was carried out by solving a global optimization problem using meta-heuristics. This paradigm not only does not exploit the manifold structure of data but can be computationally expensive. To the best of our knowledge, the joint use of sparse modeling representative and feature subset relevance have not been exploited by the joint feature and selection methods. In this paper, we target the joint feature and instance selection by adopting feature subset relevance and sparse modeling representative selection. More precisely, we propose three schemes for the joint feature and instance selection. The first is a wrapper technique while the two remaining ones are filter approaches. In the filter approaches, the search process adopts a genetic algorithm in which the evaluation is mainly given by a score that quantify the goodness of the features and instances. An efficient instance selection technique is used and integrated in the search process in order to adapt the instances to the candidate feature subset. We evaluate the performance of the proposed schemes using image classification where classifiers are the nearest neighbor classifier and support vector machine classifier. The study is conducted on five public image datasets. These experiments show the superiority of the proposed schemes over various baselines. The results confirm that the filter approaches leads to promising improvement on classification accuracy when both feature selection and instance selection are adopted.

In order to obtain the higher classification accuracy in specific categories for the different feature subset, a hierarchical classification algorithm based on Feature Selection is proposed, and is used for synthetic aperture radar (SAR) image classification, and feature selection is achieved by Genetic algorithm. The algorithm has two main characteristics: one is hierarchical classification which consists of many two-class classifier, and the two-class classifier is trained by the optimal feature subset which is selected according to different categories; the second is the classifier of support vector machine SVM (Support Vector Machine); the two is Genetic algorithm which can search out the optimal feature subset and parameters of support vector machine that is most suitable for the category, by unified coding the feature set and the parameters of SVM to constitute the Chromosome. The experiment on the first polarimetric SAR data show that the algorithm can obtain higher classification accuracy rate.