Information gain-based multi-objective evolutionary algorithm for feature selection

Multi-objective feature selection based on artificial bee colony: An acceleration approach with variable sample size

<p>Classification problems often have a large number of features, but not all of them are useful for classification. Irrelevant and redundant features may even reduce the classification accuracy. Feature selection is a process of selecting a subset of relevant features, which can decrease the dimensionality, shorten the running time, and/or improve the classification accuracy. There are two types of feature selection approaches, i.e. wrapper and filter approaches. Their main difference is that wrappers use a classification algorithm to evaluate the goodness of the features during the feature selection process while filters are independent of any classification algorithm. Feature selection is a difficult task because of feature interactions and the large search space. Existing feature selection methods suffer from different problems, such as stagnation in local optima and high computational cost. Evolutionary computation (EC) techniques are well-known global search algorithms. Particle swarm optimisation (PSO) is an EC technique that is computationally less expensive and can converge faster than other methods. PSO has been successfully applied to many areas, but its potential for feature selection has not been fully investigated. The overall goal of this thesis is to investigate and improve the capability of PSO for feature selection to select a smaller number of features and achieve similar or better classification performance than using all features. This thesis investigates the use of PSO for both wrapper and filter, and for both single objective and multi-objective feature selection, and also investigates the differences between wrappers and filters. This thesis proposes a new PSO based wrapper, single objective feature selection approach by developing new initialisation and updating mechanisms. The results show that by considering the number of features in the initialisation and updating procedures, the new algorithm can improve the classification performance, reduce the number of features and decrease computational time. This thesis develops the first PSO based wrapper multi-objective feature selection approach, which aims to maximise the classification accuracy and simultaneously minimise the number of features. The results show that the proposed multi-objective algorithm can obtain more and better feature subsets than single objective algorithms, and outperform other well-known EC based multi-objective feature selection algorithms. This thesis develops a filter, single objective feature selection approach based on PSO and information theory. Two measures are proposed to evaluate the relevance of the selected features based on each pair of features and a group of features, respectively. The results show that PSO and information based algorithms can successfully address feature selection tasks. The group based method achieves higher classification accuracies, but the pair based method is faster and selects smaller feature subsets. This thesis proposes the first PSO based multi-objective filter feature selection approach using information based measures. This work is also the first work using other two well-known multi-objective EC algorithms in filter feature selection, which are also used to compare the performance of the PSO based approach. The results show that the PSO based multiobjective filter approach can successfully address feature selection problems, outperform single objective filter algorithms and achieve better classification performance than other multi-objective algorithms. This thesis investigates the difference between wrapper and filter approaches in terms of the classification performance and computational time, and also examines the generality of wrappers. The results show that wrappers generally achieve better or similar classification performance than filters, but do not always need longer computational time than filters. The results also show that wrappers built with simple classification algorithms can be general to other classification algorithms.</p>

Multi-objective feature selection aims to find a set of feature subsets that achieves a trade-off between two objectives, i.e., reducing the number of selected features and improving the classification performance. However, these two objectives might not be always conflicting during the optimization process and have varying difficulties in optimization. Such characteristics pose a great challenge to existing multi-objective evolutionary approaches, which often treat two objectives equally. Specifically, a large number of feature subsets with few features may appear in the population and compete for survival opportunities with promising feature subsets located in not fully explored regions, leading to poor performance. To this end, we propose a two-archive evolutionary feature selection algorithm for multi-objective feature selection. In the proposed method, all individuals are equally allocated into two independent archives. A two-archive based solution generation strategy is proposed, where a dynamic dimensionality reduction operator is used to exploit small features subsets while a diversity-based mutation operator is utilized to find feature subsets with better classification performance. Moreover, a novel environmental selection scheme is proposed, which aims to improve the survival probability of promising feature subsets by providing different selection environments. Experimental results on 23 datasets demonstrate that the proposed algorithm is superior to the other five state-of-the-art algorithms.

Multi-surrogate assisted multi-objective evolutionary algorithms for feature selection in regression and classification problems with time series data

Feature selection (FS) using meta-heuristic algorithms on high-dimensional datasets (HDD) is becoming more prevalent due to the continuous advancement in data mining. However, the difficulty in identifying the threshold of features in a dataset to be categorised as HDD remains an issue due to the different schools of thought on this matter. Therefore, this survey intended to determine the threshold for a number of features to be HDD, and subsequently identify the trend or potential FS method for HDD and the most preferred meta-heuristic algorithms and classifiers for both wrapper-based and filter-based FS methods to analyse HDD. This study performed an extensive systematic literature review by implementing the PRISMA guidelines on 62 research articles that were published between 2016 to 2021. This survey proposed a novel grouping technique called literal grouping and data grouping (LGDG) to accurately group the chosen articles based on HDD. The LGDG method serves as a guide for other researchers who intend to perform FS research related to HDD. Literal grouping refers to searching for selected papers using specific keywords, like HDD in this case. While data grouping compares the number of features in datasets towards the threshold, which is set at 2,000 features by the majority. Based on the analyses of all the LGDG groupings, the filter-based FS method gained more attention in recent years with competent results no less than wrapper-based, especially on HDD. Besides that, Moth Flame Optimisation works well in filter-based methods, whereas Cuckoo Optimisation Algorithm works well in wrapper-based, while Whale Optimisation Algorithm works well in both FS methods. As for the classifier’s preferences, SVM, DT, and NB are preferred by the filter-based, while KNN is preferred by the wrapper-based method. It can be recommended that reviewing other aspects such as multi-objective FS on HDD and including more FS methods could be included in future studies as an extension to this survey.

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

Feature selection is an NP-hard problem from the viewpoint of algorithm design and it is one of the main open problems in pattern recognition. In this paper, we propose a new evolutionary-incremental framework for feature selection. The proposed framework can be applied on an ordinary evolutionary algorithm (EA) such as genetic algorithm (GA) or invasive weed optimization (IWO). This framework proposes some generic modifications on ordinary EAs to be compatible with the variable length of solutions. In this framework, the solutions related to the primary generations have short length. Then, the length of solutions may be increased through generations gradually. In addition, our evolutionary-incremental framework deploys two new operators called addition and deletion operators which change the length of solutions randomly. For evaluation of the proposed framework, we use that for feature selection in the application of face recognition. In this regard, we applied our feature selection method on a robust face recognition algorithm which is based on the extraction of Gabor coefficients. Experimental results show that our proposed evolutionary-incremental framework can select a few number of features from existing thousands features efficiently. Comparison result of the proposed methods with the previous methods shows that our framework is comprehensive, robust, and well-defined to apply on many EAs for feature selection.

The conceptual fusion of smart city and sustainability indicators has inspired the emergence of the smart sustainable city (SSC). Given the early stage of development in this field, most SSC studies have been primarily theoretical. Notably, existing empirical studies have overlooked the crucial aspect of feature engineering in the context of SSC, despite its significance in advancing SSC initiatives. This paper introduces an approach advocating for feature subset selection to maximize prediction accuracy and minimize computational time across diverse SSC indicators encompassing socio-cultural, economic, environmental, and governance categories. The study systematically collected multiple datasets on SSC indicators, covering various themes within the SSC framework. Employing six carefully chosen multiple-objective evolutionary feature selection algorithms, the research selected feature subsets. These subsets were then utilized in modeling algorithms to predict SSC indicators. The proposal enhanced prediction accuracy for life expectancy, online shopping intentions, energy consumption, air quality, water quality, and traffic flow for a smart and sustainable city by minimizing the subset features. The findings underscore the efficacy of feature subset selection in generating minimal features, thereby enhancing both prediction accuracy and computational efficiency in the realm of SSC indicators. For researchers aiming to develop sustainable systems for real-time data monitoring within SSC, the identified subset features offer a valuable resource, negating the necessity for extensive dataset collection. The provided SSC datasets are anticipated to serve as a catalyst, inspiring researchers to embark on empirical studies that explore SSC development from diverse perspectives, ultimately contributing to a more profound understanding of the SSC dynamics.

Evolutionary computation for feature selection in classification: A comprehensive survey of solutions, applications and challenges

The aim of this paper is to present an intelligent method for fault diagnosis of induction motors which doesn't need any expert to analyze the signals. A new intelligent fault diagnosis scheme based on multi-objective feature selection using non-dominated sorting genetic algorithm II (NSGA-II) is proposed. Firstly, to improve the signal-to-noise ratio, wavelet packet decomposition is performed. Multiple statistical features are then extracted from the decomposed signals. Some of these features contain unhelpful information, so the most superior features are selected using NSGA-II. Finally, the classification of type and severity of faults is performed using a multilayer perceptron (MLP) neural network. The proposed scheme is tested on a bearing fault dataset, and the results show that it, unlike signal processing techniques, is able to detect the faults of induction motor without any expert. It also achieves a better classification rate comparing with the methods based on conventional feature selection algorithms.

New hybrid data mining model for credit scoring based on feature selection algorithm and ensemble classifiers

PurposeThis paper develops a robust hospital readmission prediction framework by combining the feature selection algorithm and machine learning (ML) classifiers. The improved feature selection is proposed by considering the uncertainty in patient's attributes that leads to the output variable.Design/methodology/approachFirst, data preprocessing is conducted which includes how raw data is managed. Second, the impactful features are selected through feature selection process. It started with calculating the relational grade of each patient towards readmission using grey relational analysis (GRA) and the grade is used as the target values for feature selection. Then, the influenced features are selected using the Least Absolute Shrinkage and Selection Operator (LASSO) method. This proposed method is termed as Grey-LASSO feature selection. The final task is the readmission prediction using ML classifiers.FindingsThe proposed method offered good performances with a minimum feature subset up to 54–65% discarded features. Multi-Layer Perceptron with Grey-LASSO gave the best performance.Research limitations/implicationsThe performance of Grey-LASSO is justified in two readmission datasets. Further research is required to examine the generalisability to other datasets.Originality/valueIn designing the feature selection algorithm, the selection on influenced input variables was based on the integration of GRA and LASSO. Specifically, GRA is a part of the grey system theory, which was employed to analyse the relation between systems under uncertain conditions. The LASSO approach was adopted due to its ability for sparse data representation.

The prediction of liquefaction susceptibility for highly unbalanced database with limited and important input parameters is a crucial issue. The proposed multi-objective feature selection algorithms (MOFS) were applied to highly unbalanced databases of in situ tests: standard penetration test (SPT), cone penetration test (CPT) and shear wave velocity (Vs) test. Two multi-objective algorithms, non-dominated sorting genetic algorithm (NSGA-II) and multi-objective symbiotic organisms search algorithm (MOSOS), were coupled with learning algorithms, artificial neural network (ANN) and multivariate adaptive regression spline (MARS) separately to effectively select the optimal parameters and simultaneously minimize the error. The obtained optimal point has approximately equal accuracy in both liquefiable and non-liquefiable conditions for training and testing. The important inputs found for models based on SPT are: (N1)60, amax and Mw; CPT: qc1, amax and CSR and Vs: Vs1, CSR, amax and Mw. The CPT-based models were found to be the most efficient.

In classification, feature selection (FS) removes irrelevant and redundant features to improve the classification performance. Most of the existing FS approaches are designed specifically for single-label classification, wherein each instance is associated to a single class label. However, many real-world problems have more than one class label for each instance, also known as multi-label classification (MLC). Label-to-label interactions in MLC information presents a new challenge for FS, whereas single-label FS methods typically only consider feature-to-feature (redundancy) and feature-to-label (relevancy) information. Sparsity-based FS is an efficient and effective method for MLC which has demonstrated its ability in considering label-to-label interactions. However, existing sparsity-based approaches can be trapped at local optima due to their gradient-based search. In addition, they cannot consider different trade-off between the two objectives of FS: to minimise the number of features and to maximise the classification performance. Evolutionary multi-objective optimisation (EMO) can address the above issues, but EMO has not been applied to achieve sparsity-based FS for MLC. This paper proposes a novel sparsity-based representation, objective function, and local search method to achieve sparsity-based EMO FS for MLC. The experimental results indicate improved efficiency and effectiveness in the MLC performance in comparison to existing state-of-the-art multi-label feature selection methods.

A federated feature selection algorithm based on particle swarm optimization under privacy protection