A new hybrid filter/wrapper algorithm for feature selection in classification

The high-dimensional feature vectors of hyper spectral data often impose a high computational cost as well as the risk of “over fitting ” when classification is performed. Therefore it is necessary to reduce the dimensionality through ways like feature selection. Currently, there are two kinds of feature selection methods: filter methods and wrapper methods. The form kind requires no feedback from classifiers and estimates the classification performance indirectly. The latter kind evaluates the “goodness ” of selected feature subset directly based on the classification accuracy. Many experimental results have proved that the wrapper methods can yield better performance, although they have the disadvantage of high computational cost. In this paper, we present a Genetic Algorithm (GA) based wrapper method for classification of hyper spectral data using Support Vector Machine (SVM), a state-of-art classifier that has found success in a variety of areas. The genetic algorithm (GA), which seek to solve optimization problems using the methods of evolution, specifically survival of the fittest, was used to optimize both the feature subset, i.e. band subset, of hyper spectral data and SVM kernel parameters simultaneously. A special strategy was adopted to reduce computation cost caused by the high-dimensional feature vectors of hyper spectral data when the feature subset part of chromosome was designed. The GA-SVM method was realized using the ENVI/IDL language, and was then tested by applying to a HYPERION hyper spectral image. Comparison of the optimized results and the un-optimized results showed that the GA-SVM method could significantly reduce the computation cost while improving the classification accuracy. The number of bands used for classification was reduced from 198 to

Classification problem in biological Omics data has gained in popularity in recent years. In consideration of the high dimension of the Omics dataset, the importance of feature selection technology becomes apparent. Feature selection can not only improve the classification accuracy but also avoid overfitting which is a common issue in machine learning. There are three main categories of feature selection methods: filter methods, wrapper methods, and hybrid methods. Yet it is difficult for researchers to choose among them. In this study, we conducted a comprehensive comparison for filter methods, wrapper methods, and hybrid methods. Specifically, we selected information gain as the filter method, genetic algorithm, and binary particle swarms optimization as the wrapper methods, IG-GA and IG-BPSO as the hybrid methods for comparison. The experimental results show that the IG-BPSO, a hybrid method, has the highest classification accuracy among these feature selection methods on three Omics datasets.

Objectives: To develop a machine learning-based model to select the most important features from a high-dimensional dataset to classify the patterns at high accuracy and reduce their dimensionality. Methods: The proposed feature selection method (FSIFC) forms and combines feature clusters incrementally and produces feature subsets each time. The method uses K-means clustering and Mutual Information (MI) to refine the feature selection process iteratively. Initially, two clusters of features are formed using K-means clustering (K=2) by taking features as the basis of clustering instead of taking the patterns (a traditional way). From these two clusters, the features with the highest MI value in each cluster are kept in a feature subset. Classification accuracies (CA) of the feature subset are calculated using three classifiers namely Support Vector Machines (SVM), Random Forest (RF), and k-nearest Neighbor (knn). The process is repeated by incrementing the value of K i.e. number of clusters; until a maximum user-defined value of K is reached. The best value of CA obtained from these trials is recorded and the corresponding feature set is finally accepted. Findings: The proposed method is demonstrated using ten datasets and the results are compared with the existing published results using three classifiers to determine the method's performance. The ten datasets are classified with average CAs of 92.72%, 93.13%, and 91.5%, using the SVM, RF, and K-NN classifiers respectively. The proposed method selects a maximum of thirty features from the datasets. In terms of selecting the most effective and the smallest feature sets, the proposed method outperforms eight other feature selection methods considering CAs. Novelty: The proposed model applies feature reduction using combined feature clustering and filter methods in an incremental way. This provides an improved selection of relevant features while removing those which are irrelevant at different trials. Keywords: Feature selection, High-dimensional datasets, K-means algorithm, Mutual information, Machine learning

Feature selection is an important topic in data mining, especially for high dimensional dataset. Feature selection is a process commonly used in machine learning, wherein subsets of the features available from the data are selected for application of learning algorithm. The best subset contains the least number of dimensions that most contribute to accuracy. Feature selection methods can be decomposed into three main classes, one is filter method, another one is wrapper method and third one is embedded method. This chapter presents an empirical comparison of feature selection methods and its algorithm. In view of the substantial number of existing feature selection algorithms, the need arises to count on criteria that enable to adequately decide which algorithm to use in certain situation. This chapter reviews several fundamental algorithms found in the literature and assess their performance in a controlled scenario.

Feature selection is the process of choosing, or removing, features to obtain the most informative feature subset of minimal size. Such subsets are used to improve performance of machine learning algorithms and enable human understanding of the results. Approaches to feature selection in literature exploit several optimization algorithms. Multi-objective methods also have been proposed, minimizing at the same time the number of features and the error. While most approaches assess error resorting to the average of a stochastic K-fold cross-validation, comparing averages might be misleading. In this paper, we show how feature subsets with different average error might in fact be non-separable when compared using a statistical test. Following this idea, clusters of non-separable optimal feature subsets are identified. The performance in feature selection can thus be evaluated by verifying how many of these optimal feature subsets an algorithm is able to identify. We thus propose a multi-objective optimization approach to feature selection, EvoFS, with the objectives to i. minimize feature subset size, ii. minimize test error on a 10-fold cross-validation using a specific classifier, iii. maximize the analysis of variance value of the lowest-performing feature in the set. Experiments on classification datasets whose feature subsets can be exhaustively evaluated show that our approach is able to always find the best feature subsets. Further experiments on a high-dimensional classification dataset, that cannot be exhaustively analyzed, show that our approach is able to find more optimal feature subsets than state-of-the-art feature selection algorithms.

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

There is a consistent rise in chronic diseases worldwide. These diseases decrease immunity and the quality of daily life. The treatment of these disorders is a challenging task for medical professionals. Dimensionality reduction techniques make it possible to handle big data samples, providing decision support in relation to chronic diseases. These datasets contain a series of symptoms that are used in disease prediction. The presence of redundant and irrelevant symptoms in the datasets should be identified and removed using feature selection techniques to improve classification accuracy. Therefore, the main contribution of this paper is a comparative analysis of the impact of wrapper and filter selection methods on classification performance. The filter methods that have been considered include the Correlation Feature Selection (CFS) method, the Information Gain (IG) method and the Chi-Square (CS) method. The wrapper methods that have been considered include the Best First Search (BFS) method, the Linear Forward Selection (LFS) method and the Greedy Step Wise Search (GSS) method. A Decision Tree algorithm has been used as a classifier for this analysis and is implemented through the WEKA tool. An attribute significance analysis has been performed on the diabetes, breast cancer and heart disease datasets used in the study. It was observed that the CFS method outperformed other filter methods concerning the accuracy rate and execution time. The accuracy rate using the CFS method on the datasets for heart disease, diabetes, breast cancer was 93.8%, 89.5% and 96.8% respectively. Moreover, latency delays of 1.08 s, 1.02 s and 1.01 s were noted using the same method for the respective datasets. Among wrapper methods, BFS’ performance was impressive in comparison to other methods. Maximum accuracy of 94.7%, 95.8% and 96.8% were achieved on the datasets for heart disease, diabetes and breast cancer respectively. Latency delays of 1.42 s, 1.44 s and 132 s were recorded using the same method for the respective datasets. On the basis of the obtained result, a new hybrid Attribute Evaluator method has been proposed which effectively integrates enhanced K-Means clustering with the CFS filter method and the BFS wrapper method. Furthermore, the hybrid method was evaluated with an improved decision tree classifier. The improved decision tree classifier combined clustering with classification. It was validated on 14 different chronic disease datasets and its performance was recorded. A very optimal and consistent classification performance was observed. The mean values for accuracy, specificity, sensitivity and f-score metrics were 96.7%, 96.5%, 95.6% and 96.2% respectively.

In the last few decades, Feature selection is one of the most challenging and open problem to researchers. The rapid progress in computational techniques causes the generation and recording of data in huge size. Though there exists various feature ranking methods, the processing of data is still a challenging task due to its computational complexity. The filter method has many advantages over the wrapper method. The filter methods are classifier independent and have better computational efficiency. Here, a subset of features is selected based on a certain goal function. Most of these goal functions employs the principle of information theory. Most of the algorithms in earlier studies addressed two factors, that is, maximization of relevancy and minimization of redundancy without considering the interaction among the features. This paper developed a new forward filter feature selection algorithm based on mutual information known as Maximum Dual Interaction and Maximum Feature Relevance(MDIMFR). This method considers all the three factors: relevance, redundancy, and feature interaction. This method is experimented on three datasets and compares the performance with existing methods. The results show that MDIMFR outperforms the existing competitive feature selection methods of recent studies: mRMR, JMIM and CMIM. MDIMFR also achieves good stability in average classification accuracy for a certain number of features, say k and above. Hence, these k features can be considered as an optimal feature set.

In the field of data processing and analysis, the dataset may be a large set of features that restrict data usability and applicability, and thus the dimensions of data sets need to be reduced. Feature selection is the process of removing as much of the redundant and irrelevant features as possible from the original dataset to improve the mining process efficiency. This paper presented a study to evaluate and compare the effect of filter and wrapper methods as feature selection approaches in terms of classification accuracy and time complexity. The Naive Bayes Classifier and three classification datasets from the UCI repository are utilizing in the classification procedure. To investigate the effect of feature selection methods, they are applied to the different characteristics datasets to obtain the selected feature vectors which are then classified according to each dataset category. The datasets used in this paper are the Iris, Ionosphere, and Ovarian Cancer dataset. Experimental results indicate that the filter and wrapper methods provide approximately equal classification accuracy where the average accuracy value of the Ionosphere and Ovarian Cancer dataset is 0.78 and 0.91 for the same selected feature vectors respectively. For Iris dataset, the filter method outperforms the wrapper method by achieving the same accuracy value using only half number of selected features. The results also show that the filter method surpasses when considering the execution time.

In order to obtain the higher classification accuracy in specific categories for the different feature subset, a hierarchical classification algorithm based on Feature Selection is proposed, and is used for synthetic aperture radar (SAR) image classification, and feature selection is achieved by Genetic algorithm. The algorithm has two main characteristics: one is hierarchical classification which consists of many two-class classifier, and the two-class classifier is trained by the optimal feature subset which is selected according to different categories; the second is the classifier of support vector machine SVM (Support Vector Machine); the two is Genetic algorithm which can search out the optimal feature subset and parameters of support vector machine that is most suitable for the category, by unified coding the feature set and the parameters of SVM to constitute the Chromosome. The experiment on the first polarimetric SAR data show that the algorithm can obtain higher classification accuracy rate.

Computer-aided diagnosis has the potential of increasing diagnostic accuracy by providing a second reading to radiologists. In many computerized schemes, numerous features can be extracted to describe suspect image regions. A subset of these features is then employed in a data classifier to determine whether the suspect region is abnormal or normal. Different subsets of features will, in general, result in different classification performances. A feature selection method is often used to determine an "optimal" subset of features to use with a particular classifier. A classifier performance measure (such as the area under the receiver operating characteristic curve) must be incorporated into this feature selection process. With limited datasets, however, there is a distribution in the classifier performance measure for a given classifier and subset of features. In this paper, we investigate the variation in the selected subset of "optimal" features as compared with the true optimal subset of features caused by this distribution of classifier performance. We consider examples in which the probability that the optimal subset of features is selected can be analytically computed. We show the dependence of this probability on the dataset sample size, the total number of features from which to select, the number of features selected, and the performance of the true optimal subset. Once a subset of features has been selected, the parameters of the data classifier must be determined. We show that, with limited datasets and/or a large number of features from which to choose, bias is introduced if the classifier parameters are determined using the same data that were employed to select the "optimal" subset of features.

Abstract. Dissolved oxygen (DO) holds an important role in water quality monitoring due to its impact on the aquatic organisms inhabiting the water body. Sampaloc lake, located in San Pablo City in Laguna Province, Philippines, one of the major fishing lakes, has been affected by the decrease of DO concentration leading to continuous fish kill incidents. This study used supervised machine learning-based model utilizing the filter and wrapper method in feature selection to predict the DO levels in Sampaloc Lake. The goal of the study is to provide a baseline for building an early warning system for fish kills events in the lake. Water quality, physical and hydroclimatic parameters are the range of features selected as predictors to create a reliable DO predictive model. Based on the feature selection process, water temperature, atmospheric temperature, biochemical oxygen demand, surface pressure and pH are the consistent important feature in predicting DO for both filter and wrapper method. Overall, the result of the eXtreme Gradient Boosting Regression (XGBR) Model of the filter method with R² = 0.95, RMSE = 0.14, and MAE = 0.10 showed a reliable performance among the predicted models (Multiple Linear Regression, Random Forest, Kernel Ridge Regression, and Bootstrap Aggregating) tested in the study.

<p>Classification problems often have a large number of features, but not all of them are useful for classification. Irrelevant and redundant features may even reduce the classification accuracy. Feature selection is a process of selecting a subset of relevant features, which can decrease the dimensionality, shorten the running time, and/or improve the classification accuracy. There are two types of feature selection approaches, i.e. wrapper and filter approaches. Their main difference is that wrappers use a classification algorithm to evaluate the goodness of the features during the feature selection process while filters are independent of any classification algorithm. Feature selection is a difficult task because of feature interactions and the large search space. Existing feature selection methods suffer from different problems, such as stagnation in local optima and high computational cost. Evolutionary computation (EC) techniques are well-known global search algorithms. Particle swarm optimisation (PSO) is an EC technique that is computationally less expensive and can converge faster than other methods. PSO has been successfully applied to many areas, but its potential for feature selection has not been fully investigated. The overall goal of this thesis is to investigate and improve the capability of PSO for feature selection to select a smaller number of features and achieve similar or better classification performance than using all features. This thesis investigates the use of PSO for both wrapper and filter, and for both single objective and multi-objective feature selection, and also investigates the differences between wrappers and filters. This thesis proposes a new PSO based wrapper, single objective feature selection approach by developing new initialisation and updating mechanisms. The results show that by considering the number of features in the initialisation and updating procedures, the new algorithm can improve the classification performance, reduce the number of features and decrease computational time. This thesis develops the first PSO based wrapper multi-objective feature selection approach, which aims to maximise the classification accuracy and simultaneously minimise the number of features. The results show that the proposed multi-objective algorithm can obtain more and better feature subsets than single objective algorithms, and outperform other well-known EC based multi-objective feature selection algorithms. This thesis develops a filter, single objective feature selection approach based on PSO and information theory. Two measures are proposed to evaluate the relevance of the selected features based on each pair of features and a group of features, respectively. The results show that PSO and information based algorithms can successfully address feature selection tasks. The group based method achieves higher classification accuracies, but the pair based method is faster and selects smaller feature subsets. This thesis proposes the first PSO based multi-objective filter feature selection approach using information based measures. This work is also the first work using other two well-known multi-objective EC algorithms in filter feature selection, which are also used to compare the performance of the PSO based approach. The results show that the PSO based multiobjective filter approach can successfully address feature selection problems, outperform single objective filter algorithms and achieve better classification performance than other multi-objective algorithms. This thesis investigates the difference between wrapper and filter approaches in terms of the classification performance and computational time, and also examines the generality of wrappers. The results show that wrappers generally achieve better or similar classification performance than filters, but do not always need longer computational time than filters. The results also show that wrappers built with simple classification algorithms can be general to other classification algorithms.</p>

Selecting highly predictive features from the high dimensional dataset is a formidable task. The existing feature selection algorithms available today are not dealing with multiple equally predictive subsets of features. We strongly believe that there is other subsets of features as well, which can give equivalent predictive accuracy as that of state-of-the-art algorithms. Statistically equivalent signature (SES) is one such feature selection algorithm, which is centred on constraint-based learning of Bayesian networks. The proposed model selects equivalent subsets of features from oral squamous cell carcinoma (OSCC) dataset with the help of SES. To inspect the validity of SES algorithms output, we are using K-nearest neighbour (KNN), support vector machine (SVM) and neural networks (NN) on each subset of predictive features. Finally, the results of proposed technique is compared with support vector machine - recursive feature elimination (SVM-RFE). SES produces more stable accuracy as compared to SVM-RFE.

Hybrid filter–wrapper feature selection for short-term load forecasting