FeatureSelect: a software for feature selection based on machine learning approaches

A new hybrid filter/wrapper algorithm for feature selection in classification

Classification problem in biological Omics data has gained in popularity in recent years. In consideration of the high dimension of the Omics dataset, the importance of feature selection technology becomes apparent. Feature selection can not only improve the classification accuracy but also avoid overfitting which is a common issue in machine learning. There are three main categories of feature selection methods: filter methods, wrapper methods, and hybrid methods. Yet it is difficult for researchers to choose among them. In this study, we conducted a comprehensive comparison for filter methods, wrapper methods, and hybrid methods. Specifically, we selected information gain as the filter method, genetic algorithm, and binary particle swarms optimization as the wrapper methods, IG-GA and IG-BPSO as the hybrid methods for comparison. The experimental results show that the IG-BPSO, a hybrid method, has the highest classification accuracy among these feature selection methods on three Omics datasets.

Hybrid filter–wrapper feature selection for short-term load forecasting

In the field of data processing and analysis, the dataset may be a large set of features that restrict data usability and applicability, and thus the dimensions of data sets need to be reduced. Feature selection is the process of removing as much of the redundant and irrelevant features as possible from the original dataset to improve the mining process efficiency. This paper presented a study to evaluate and compare the effect of filter and wrapper methods as feature selection approaches in terms of classification accuracy and time complexity. The Naive Bayes Classifier and three classification datasets from the UCI repository are utilizing in the classification procedure. To investigate the effect of feature selection methods, they are applied to the different characteristics datasets to obtain the selected feature vectors which are then classified according to each dataset category. The datasets used in this paper are the Iris, Ionosphere, and Ovarian Cancer dataset. Experimental results indicate that the filter and wrapper methods provide approximately equal classification accuracy where the average accuracy value of the Ionosphere and Ovarian Cancer dataset is 0.78 and 0.91 for the same selected feature vectors respectively. For Iris dataset, the filter method outperforms the wrapper method by achieving the same accuracy value using only half number of selected features. The results also show that the filter method surpasses when considering the execution time.

There is a consistent rise in chronic diseases worldwide. These diseases decrease immunity and the quality of daily life. The treatment of these disorders is a challenging task for medical professionals. Dimensionality reduction techniques make it possible to handle big data samples, providing decision support in relation to chronic diseases. These datasets contain a series of symptoms that are used in disease prediction. The presence of redundant and irrelevant symptoms in the datasets should be identified and removed using feature selection techniques to improve classification accuracy. Therefore, the main contribution of this paper is a comparative analysis of the impact of wrapper and filter selection methods on classification performance. The filter methods that have been considered include the Correlation Feature Selection (CFS) method, the Information Gain (IG) method and the Chi-Square (CS) method. The wrapper methods that have been considered include the Best First Search (BFS) method, the Linear Forward Selection (LFS) method and the Greedy Step Wise Search (GSS) method. A Decision Tree algorithm has been used as a classifier for this analysis and is implemented through the WEKA tool. An attribute significance analysis has been performed on the diabetes, breast cancer and heart disease datasets used in the study. It was observed that the CFS method outperformed other filter methods concerning the accuracy rate and execution time. The accuracy rate using the CFS method on the datasets for heart disease, diabetes, breast cancer was 93.8%, 89.5% and 96.8% respectively. Moreover, latency delays of 1.08 s, 1.02 s and 1.01 s were noted using the same method for the respective datasets. Among wrapper methods, BFS’ performance was impressive in comparison to other methods. Maximum accuracy of 94.7%, 95.8% and 96.8% were achieved on the datasets for heart disease, diabetes and breast cancer respectively. Latency delays of 1.42 s, 1.44 s and 132 s were recorded using the same method for the respective datasets. On the basis of the obtained result, a new hybrid Attribute Evaluator method has been proposed which effectively integrates enhanced K-Means clustering with the CFS filter method and the BFS wrapper method. Furthermore, the hybrid method was evaluated with an improved decision tree classifier. The improved decision tree classifier combined clustering with classification. It was validated on 14 different chronic disease datasets and its performance was recorded. A very optimal and consistent classification performance was observed. The mean values for accuracy, specificity, sensitivity and f-score metrics were 96.7%, 96.5%, 95.6% and 96.2% respectively.

Abstract. Dissolved oxygen (DO) holds an important role in water quality monitoring due to its impact on the aquatic organisms inhabiting the water body. Sampaloc lake, located in San Pablo City in Laguna Province, Philippines, one of the major fishing lakes, has been affected by the decrease of DO concentration leading to continuous fish kill incidents. This study used supervised machine learning-based model utilizing the filter and wrapper method in feature selection to predict the DO levels in Sampaloc Lake. The goal of the study is to provide a baseline for building an early warning system for fish kills events in the lake. Water quality, physical and hydroclimatic parameters are the range of features selected as predictors to create a reliable DO predictive model. Based on the feature selection process, water temperature, atmospheric temperature, biochemical oxygen demand, surface pressure and pH are the consistent important feature in predicting DO for both filter and wrapper method. Overall, the result of the eXtreme Gradient Boosting Regression (XGBR) Model of the filter method with R² = 0.95, RMSE = 0.14, and MAE = 0.10 showed a reliable performance among the predicted models (Multiple Linear Regression, Random Forest, Kernel Ridge Regression, and Bootstrap Aggregating) tested in the study.

A new hybrid filter–wrapper feature selection method for clustering based on ranking

The high-dimensional feature vectors of hyper spectral data often impose a high computational cost as well as the risk of “over fitting ” when classification is performed. Therefore it is necessary to reduce the dimensionality through ways like feature selection. Currently, there are two kinds of feature selection methods: filter methods and wrapper methods. The form kind requires no feedback from classifiers and estimates the classification performance indirectly. The latter kind evaluates the “goodness ” of selected feature subset directly based on the classification accuracy. Many experimental results have proved that the wrapper methods can yield better performance, although they have the disadvantage of high computational cost. In this paper, we present a Genetic Algorithm (GA) based wrapper method for classification of hyper spectral data using Support Vector Machine (SVM), a state-of-art classifier that has found success in a variety of areas. The genetic algorithm (GA), which seek to solve optimization problems using the methods of evolution, specifically survival of the fittest, was used to optimize both the feature subset, i.e. band subset, of hyper spectral data and SVM kernel parameters simultaneously. A special strategy was adopted to reduce computation cost caused by the high-dimensional feature vectors of hyper spectral data when the feature subset part of chromosome was designed. The GA-SVM method was realized using the ENVI/IDL language, and was then tested by applying to a HYPERION hyper spectral image. Comparison of the optimized results and the un-optimized results showed that the GA-SVM method could significantly reduce the computation cost while improving the classification accuracy. The number of bands used for classification was reduced from 198 to

Feature selection is an important topic in data mining, especially for high dimensional dataset. Feature selection is a process commonly used in machine learning, wherein subsets of the features available from the data are selected for application of learning algorithm. The best subset contains the least number of dimensions that most contribute to accuracy. Feature selection methods can be decomposed into three main classes, one is filter method, another one is wrapper method and third one is embedded method. This chapter presents an empirical comparison of feature selection methods and its algorithm. In view of the substantial number of existing feature selection algorithms, the need arises to count on criteria that enable to adequately decide which algorithm to use in certain situation. This chapter reviews several fundamental algorithms found in the literature and assess their performance in a controlled scenario.

In the last few decades, Feature selection is one of the most challenging and open problem to researchers. The rapid progress in computational techniques causes the generation and recording of data in huge size. Though there exists various feature ranking methods, the processing of data is still a challenging task due to its computational complexity. The filter method has many advantages over the wrapper method. The filter methods are classifier independent and have better computational efficiency. Here, a subset of features is selected based on a certain goal function. Most of these goal functions employs the principle of information theory. Most of the algorithms in earlier studies addressed two factors, that is, maximization of relevancy and minimization of redundancy without considering the interaction among the features. This paper developed a new forward filter feature selection algorithm based on mutual information known as Maximum Dual Interaction and Maximum Feature Relevance(MDIMFR). This method considers all the three factors: relevance, redundancy, and feature interaction. This method is experimented on three datasets and compares the performance with existing methods. The results show that MDIMFR outperforms the existing competitive feature selection methods of recent studies: mRMR, JMIM and CMIM. MDIMFR also achieves good stability in average classification accuracy for a certain number of features, say k and above. Hence, these k features can be considered as an optimal feature set.

Feature Selection methods in Data Mining and Data Analysis problems aim at selecting a subset of the variables, or features, that describe the data in order to obtain a more essential and compact representation of the available information. The selected subset has to be small in size and must retain the information that is most useful for the specific application. The role of Feature Selection is particularly important when computationally expensive Data Mining tools are used, or when the data collection process is difficult or costly. Feature Selection problems are typically solved in the literature using search techniques, where the evaluation of a specific subset is accomplished by a proper function (filter methods) or directly by the performance of a Data Mining tool (wrapper methods). In this work we show how the Feature Selection problem can be formulated as a subgraph selection problem derived from the lightest k-subgraph problem, and solved as an Integer Program. The proposed formulation is very flexible, as additional conditions on the solution can be added in the formulation. Although optimal solutions for such problems are difficult to find in the worst case, a large number of test instances have been solved efficiently by commercial tools. Finally, an application to a database on urban mobility is presented, where the proposed method is integrated in the Data Mining tool named Lsquare and is compared with other approaches.

Feature selection, as a pre-processing step to machine learning, is effective in reducing dimensionality, removing irrelevant data, increasing learning accuracy, and improving result comprehensibility.There are two main approaches for feature selection: wrapper methods, in which the features are selected using the supervised learning algorithm, and filter methods, in which the selection of features is independent of any learning algorithm.However, most of these techniques use feature scoring algorithms that make some basic assumptions about the distribution of the data like normality, balanced distribution of classes, non-sparsity or dense data-set, etc.The data generated in the real world rarely follow such strict criteria.In some cases such as digital advertising, the generated data matrix is actually very sparse and follows no distinct distribution.For this reason, we have come up with a new approach towards feature selection for cases where the data-sets do not follow the above-mentioned assumptions.Our methodology also presents an approach to solve the problem of skewness of data.The efficiency and effectiveness of our methods is then demonstrated by comparison with other well-known techniques of statistics like ANOVA, mutual information, KL divergence, Fisher score, Bayes' error, Chi-square, etc.The data-set used for validation is a real-world user-browsing history data-set used for ad-campaign targeting.It has very high dimensions and is highly sparse as well.Our approach reduces the number of features to a significant degree without compromising on the accuracy of the final predictions.

Classification is data mining techniques which used for the purposes of diagnosis in the medical field as measured by the high accuracy produced. The accuracy of classification algorithm is influenced by the use of features and dimensions in dataset. In this study, Chronic Kidney Disease (CKD) dataset was used where the data is one of the high dimension datasets. Support Vector Machine (SVM) algorithm is used because its ability to handle high-dimensional data. In the dataset, it consists of 24 attributes and 1 class which if all are used results accuracy of classification will be diminished. Method for selecting features with Particle Swarm Optimization (PSO) is applied to reduce redundant features and produce optimal features. In addition, ensemble AdaBoost also applied in this research to increase performance of entirety classification algorithm. The results showed that the optimization of SVM algorithm by using PSO as a selection and ensemble feature of AdaBoost with an average of selected features of 18 features could increase the accuracy of 36.20% to 99.50% in the diagnosis of CKD compared to the SVM algorithm without optimization only resulting in accuracy 63.30%. This research can be used as a reference for further research in focusing on the preprocessing stage.

Besides chemical composition, microstructure plays a key role to control the properties of engineering materials. A strong correlation exists between microstructure and many mechanical and physical properties of a metal. It has the utmost importance to understand the microstructure and distinguish the microstructure accurately for the appropriate selection of engineering materials in product fabrication. Computer vision and machine learning play a major role to extract the feature and predict the most probable class of a 7-class microstructural image with a high degree of accuracy. Features contain information about the image, and the classification function is defined in terms of features. Feature selection plays an important role in the classification problem to improve the classification accuracy and also to reduce the computational time by eliminating redundant or non-influential features. The current research aims at classifying microstructure image datasets by an improved wrapper-filter based feature selection method using texture-based feature descriptor. Before applying the feature selection method, a feature descriptor, called rotational local tetra pattern (RLTrP), is applied to extract the features from the input images. Then, an ensemble of three filter methods is developed by considering the union of the top-n features selected by Chi-square, Fisher score, and Gini impurity-based filter methods. The objective of this ensemble is to combine all possible important features selected by three filter methods which will be used to create an initial population of the wrapper-based meta-heuristic feature selection algorithm called, harmony search (HS). The novelty of this HS method lies in the objective function, which is defined as a function of Pearson correlation coefficient and mutual information to calculate the fitness value. The proposed method not only optimizes features with reduced dimension but also improves the performance of classification accuracy of the 7-class microstructural images. Moreover, the proposed HS model has also been compared with some standard optimization algorithms like whale optimization algorithm, particle swarm optimization, and Grey wolf optimization on the present dataset, and in every case, the HS method ensures better agreement between feature selection and classification accuracy than the other methods.

Feature selection, the process of removing irrelevant features can be extremely useful in reducing the dimensionality of the data to be processed, in reducing execution time and improving predictive accuracy of the classifier. Selecting the right set of features for classification is one of the most important problems in designing a good classifier. Feature selection methods can be decomposed into two broad classes. One is Filter methods and another is Wrapper method. This paper proposes hybrid method which is based on the Mutual Information and Genetic Algorithm.