Evolutionary Feature Selection for Big Data Classification: A MapReduce Approach

Designing efficient learning models capable of dealing with tons of data has become a reality in the era of big data. However, the amount of available data is too much for traditional data mining techniques to be applicable. This issue is even more serious when evolutionary algorithms are a key part of the learning algorithm. In this scenario, one typical approach is to follow a divide-and-conquer strategy, where data is divided into different chunks that are individually and independently addressed. Afterwards, the partial knowledge obtained from each chunk of data is combined in order to give a solution to the problem. Nevertheless, these kinds of local approaches do not look at data as a whole, missing a global view of the problem, which may result in less accurate models that also depend on how data is split. In this work, we focus on evolutionary feature selection algorithms. A divide-and-conquer approach to handle evolutionary feature selection in big data was already developed. We aim at designing its global counterpart, which looks at the feature selection problem from a global perspective, making use of the data as a whole to select the most appropriate features. In order to do so, we consider Apache Spark as a big data technology where our algorithm is implemented. We design a genetic algorithm capable of dealing with big datasets by selecting the proper parameters for our base algorithm (the well-known CHC) and adapting the evaluation procedure to take all the distributed data into account. Several preliminary results are discussed to study the feasibility of global evolutionary feature selection methods for big datasets.

<p>Classification problems often have a large number of features, but not all of them are useful for classification. Irrelevant and redundant features may even reduce the classification accuracy. Feature selection is a process of selecting a subset of relevant features, which can decrease the dimensionality, shorten the running time, and/or improve the classification accuracy. There are two types of feature selection approaches, i.e. wrapper and filter approaches. Their main difference is that wrappers use a classification algorithm to evaluate the goodness of the features during the feature selection process while filters are independent of any classification algorithm. Feature selection is a difficult task because of feature interactions and the large search space. Existing feature selection methods suffer from different problems, such as stagnation in local optima and high computational cost. Evolutionary computation (EC) techniques are well-known global search algorithms. Particle swarm optimisation (PSO) is an EC technique that is computationally less expensive and can converge faster than other methods. PSO has been successfully applied to many areas, but its potential for feature selection has not been fully investigated. The overall goal of this thesis is to investigate and improve the capability of PSO for feature selection to select a smaller number of features and achieve similar or better classification performance than using all features. This thesis investigates the use of PSO for both wrapper and filter, and for both single objective and multi-objective feature selection, and also investigates the differences between wrappers and filters. This thesis proposes a new PSO based wrapper, single objective feature selection approach by developing new initialisation and updating mechanisms. The results show that by considering the number of features in the initialisation and updating procedures, the new algorithm can improve the classification performance, reduce the number of features and decrease computational time. This thesis develops the first PSO based wrapper multi-objective feature selection approach, which aims to maximise the classification accuracy and simultaneously minimise the number of features. The results show that the proposed multi-objective algorithm can obtain more and better feature subsets than single objective algorithms, and outperform other well-known EC based multi-objective feature selection algorithms. This thesis develops a filter, single objective feature selection approach based on PSO and information theory. Two measures are proposed to evaluate the relevance of the selected features based on each pair of features and a group of features, respectively. The results show that PSO and information based algorithms can successfully address feature selection tasks. The group based method achieves higher classification accuracies, but the pair based method is faster and selects smaller feature subsets. This thesis proposes the first PSO based multi-objective filter feature selection approach using information based measures. This work is also the first work using other two well-known multi-objective EC algorithms in filter feature selection, which are also used to compare the performance of the PSO based approach. The results show that the PSO based multiobjective filter approach can successfully address feature selection problems, outperform single objective filter algorithms and achieve better classification performance than other multi-objective algorithms. This thesis investigates the difference between wrapper and filter approaches in terms of the classification performance and computational time, and also examines the generality of wrappers. The results show that wrappers generally achieve better or similar classification performance than filters, but do not always need longer computational time than filters. The results also show that wrappers built with simple classification algorithms can be general to other classification algorithms.</p>

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

In network security, intrusion detection plays an important role. Feature subsets obtained by different feature selection methods will lead to different accuracy of intrusion detection. Using individual feature selection method can be unstable in different intrusion detection scenarios. In this paper, the idea of ensemble is applied to feature selection to adjust feature subsets. Feature selection is converted into a two-category problem, and odd number of feature selection methods is used for voting method to decide whether a feature is required or discarded. In actual operation, mean decrease impurity, random forest classifier, stability selection, recursive feature elimination and chi-square test are used. Feature subsets obtained from them will be adjusted by our proposed method to get ensemble feature subsets. To test the performance, support vector machine, decision tree, knn and multi-layer perception are used to observe and compare the classification accuracy with ensemble feature subsets. Three intrusion detection data sets, including kddcup99, cidds-001 and unsw_nb15 are used in our experiments. The best result is reflected on cidds-001 with a 99.40% classification accuracy. The investigation shows that our method has a certain improvement on classification accuracy of intrusion detection.

Smart meters play a significant role in power systems, but their condition assessment faces challenges such as inconsistent evaluation criteria and inaccurate assessment results. This paper proposes feature engineering including feature construction and feature selection for smart meter group failure rate prediction. First, the basic structure and common fault types of smart meters are introduced. Smart meters are grouped by batch and distribution area. Next, 25 condition features are constructed based on failure mechanisms and technical specifications. Then, an evolutionary multi-objective feature selection algorithm combining NSGA-II, Jaccard similarity, and XGBoost is developed, where feature subsets are encoded as binary individuals optimized for three objectives: MSE, 1 − R2, and the number of features. The experimental results demonstrate that the proposed method not only reduces the number of features (25→7) but also improves the prediction accuracy (MSE: 0.0049 → 0.0042, R2: 0.6638 → 0.7228) of smart meter group failure rates. Comparative studies with other feature selection methods further confirm the superiority of our approach. The optimized features enhance interpretability and computational efficiency, providing a practical solution for large-scale smart meter condition assessment in power systems.

Filters and wrappers represent two mainstream approaches to feature selection (FS). Although evolutionary wrapper-based FS outperforms filters in addressing real-world classification problems, extending these methods to high-dimensional, many-objective optimization problems with imbalanced data poses substantial challenges. Overcoming computational costs and identifying suitable performance metrics are vital for navigating search operation complexities. Here, we propose using the Jaccard similarity (JS) in a set-based evolutionary many-objective (JSEMO) FS search, addressing both evolutionary FS and imbalanced classifier choice concurrently. This study highlights the mutual influence between these aspects, impacting overall algorithm performance. JSEMO integrates JS into population initialization, reproduction, and elitism steps, enhancing diversity and avoiding duplicate solutions. The set-based variation operator utilizes intersection and union operators for compatibility with binary coding. We also introduce a double-weighted KNN (KNN2W) classifier with four supportive objectives as a many-objective FS problem to handle imbalanced distributions. Compared with 20 methods on 15 benchmark problems, JSEMO produces distinct optimal features, significantly improving overall accuracy, balance accuracy, and g-mean metrics with comparable feature set size and computational cost. The ablation study underscores the positive impact of all JSEMO components, highlighting the set-based variation operation with JS and KNN2W with relevant evaluation metrics as the most influential aspects.

• Neural Network feature selection methodology for both numerical and categorical data. • Embedded feature selection provides model and feature subset in a single training. • Novel feature embedding for dealing with high-cardinality categorical features. • Validation of the methodology on several open and real industry dataset. • Comparison with several state-of-the-art feature selection methodologie. In an era of effortless data collection, the impact of machine learning — especially neural networks (NNs) — is undeniable. As datasets grow in size and complexity, efficiently handling mixed data types, including categorical and numerical features, becomes critical. Feature encoding and selection play a key role in improving NN performance, efficiency, interpretability, and generalisation. This paper presents GLEm-Net (Grouped Lasso with Embeddings Network), a novel NN-based approach that seamlessly integrates feature encoding and selection directly into the training process. GLEm-Net uses embedding layers to process categorical features with high cardinality, simplifying the model and improving generalisation. By extending the grouped Lasso regularisation to explicitly consider categorical features, GLEm-Net automatically identifies the most relevant features during training and returns them to the analyst. We evaluate GLEm-Net on open and proprietary industry datasets and compare it to state-of-the-art feature selection methodologies. Results show that GLEm-Net adapts to each dataset by allowing the NN to directly select subsets of most important features, offering on par performance with the best state-of-the-art feature selection methods, while eliminating the need for the external feature encoding and selection steps that are now incorporated in the NN training stage.

Classification has been widely studied due to its practical applications. Feature selection aims to improve the classification accuracy by selecting a small feature subset from the original full feature set. However, identification of relevant features is not trivial due to the large search space. Particle swarm optimisation (PSO) is an efficient meta-heuristic algorithm which has shown to be promising in feature selection. However, traditional PSO uses its personal best experience and its historical best experience to determine its search direction, but this learning strategy may limit its performance for feature selection due to the premature convergence. Therefore, the potential of PSO needs to be further explored. In this paper, a new evolutionary learning algorithm termed hybridising PSO with differential evolution (HPSO-DE) is proposed to develop new feature selection methods. In HPSO-DE, differential evolution is applied to breed promising and efficient exemplars for PSO to guide its search, which is expected to not only preserve the diversity of the population but also guide particles to fly to promising areas. HPSO-DE is compared with three classic PSO variants and five traditional feature selection methods on 15 classification problems. The results show that the proposed algorithm can effectively achieve a higher classification accuracy with a smaller feature subset than the compared methods.

Objectives: To develop a machine learning-based model to select the most important features from a high-dimensional dataset to classify the patterns at high accuracy and reduce their dimensionality. Methods: The proposed feature selection method (FSIFC) forms and combines feature clusters incrementally and produces feature subsets each time. The method uses K-means clustering and Mutual Information (MI) to refine the feature selection process iteratively. Initially, two clusters of features are formed using K-means clustering (K=2) by taking features as the basis of clustering instead of taking the patterns (a traditional way). From these two clusters, the features with the highest MI value in each cluster are kept in a feature subset. Classification accuracies (CA) of the feature subset are calculated using three classifiers namely Support Vector Machines (SVM), Random Forest (RF), and k-nearest Neighbor (knn). The process is repeated by incrementing the value of K i.e. number of clusters; until a maximum user-defined value of K is reached. The best value of CA obtained from these trials is recorded and the corresponding feature set is finally accepted. Findings: The proposed method is demonstrated using ten datasets and the results are compared with the existing published results using three classifiers to determine the method's performance. The ten datasets are classified with average CAs of 92.72%, 93.13%, and 91.5%, using the SVM, RF, and K-NN classifiers respectively. The proposed method selects a maximum of thirty features from the datasets. In terms of selecting the most effective and the smallest feature sets, the proposed method outperforms eight other feature selection methods considering CAs. Novelty: The proposed model applies feature reduction using combined feature clustering and filter methods in an incremental way. This provides an improved selection of relevant features while removing those which are irrelevant at different trials. Keywords: Feature selection, High-dimensional datasets, K-means algorithm, Mutual information, Machine learning

Fault diagnosis methods based on machine learning have made great progress for rotating machinery. The main steps of the machine learning process involve feature extraction, selection, and classification. Feature selection improves classification accuracy and reduces diagnosis time by selecting the better features. Due to the difficulty of traditional feature selection methods to rank the feature importance of each class, the best subset of features could hardly be obtained. Therefore, this article proposes a new feature selection method to address the shortcomings of the above traditional methods, called Feature Ranking based on Optimal Class Distance Ratio (FROCDR), which can choose the optimal features between every two classes of samples to obtain feature ranking that is conducive to classification. In order to comprehensively extract the fault information in the signal, the multiscale analysis and the variational mode decomposition (VMD) method are applied to process the vibration signals under different scales and frequency bands, and the processed signals are visualized by symmetrized dot pattern (SDP). In addition, features are extracted from the obtained SDP images, and the proposed FROCDR method is used to select the best subset of features. The final diagnosis task is accomplished by a random forest (RF) classifier. Experimental cases of bearing and gear data show that the proposed method has higher diagnostic accuracy and stability.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

Hyperspectral images (HSIs) generally contain a large amount of spectral bands (features), and the redundant information in them will cause the Hughes phenomenon in the classification process. And feature extraction and feature selection are the two main existing methods to effectively reduce the redundancy of spectral information in the field of HSIs classification. Compared with feature extraction methods, feature selection methods can preserve most of the features of the original HSIs data without losing their valuable details. However, most existing feature selection methods based on single scene (domain) perform poorly in some scenes (domains) with insufficient labeled samples. Therefore, how to adopt an efficient feature selection method to select the optimal feature subsets of source scene and target scene and use the sample information of source scene to assist in the classification of target scene so as to improve the classification accuracy of images in the target scene as much as possible is still very challenging. In order to solve the above problem, this paper proposes a new cross-scene algorithm: Improved Ant Colony Optimization Algorithm-Based Cross-Scene Feature Selection Algorithm (IMACO-CSFS). In order to obtain more accurate feature subsets of the two scenes, IMACO-CSFS proposes a priority sorting-based ant colony strategy to make the subsequent search process focus on the global optimal solution (optimal feature subset) found in the previous iteration. In addition, in order to further accelerate the convergence speed of the global optimal solution, an ant colony strategy based on elite ants is proposed in IMACO-CSFS to more efficiently obtain the optimal feature subsets of the two scenes for training the classifier. Furthermore, this paper simultaneously considers overall classification accuracies of the optimal feature subsets for both scenes and dynamically adjusts their scale to ensure the consistency of the selected features between the two scenes, attenuating the effect of spectral shift and achieving the higher image classification accuracy in the target scene. Experimental results on three cross-scene HSI data pairs demonstrate that IMACO-CSFS is superior in cross-scene feature selection and cross-scene HSIs classification.

Feature subset selection (FSS) has received a great deal of attention in statistics, machine learning, and data mining. Real world data analyzed by data mining algorithms can involve a large number of redundant or irrelevant features or simply too many features for a learning algorithm to handle them efficiently. Feature selection is becoming essential as databases grow in size and complexity. The selection process is expected to bring benefits in terms of better performing models, computational efficiency, and simpler more understandable models. Evolutionary computation (EC) encompasses a number of naturally inspired techniques such as genetic algorithms, genetic programming, ant colony optimization, or particle swarm optimization algorithms. Such techniques are well suited to feature selection because the representation of a feature subset is straightforward and the evaluation can also be easily accomplished through the use of wrapper or filter algorithms. Furthermore, the capability of such heuristic algorithms to efficiently search large search spaces is of great advantage to the feature selection problem. Here, we review the use of different EC paradigms for feature selection in classification problems. We discuss details of each implementation including representation, evaluation, and validation. The review enables us to uncover the best EC algorithms for FSS and to point at future research directions.

The “Curse of Dimensionality” induced by the rapid development of information science might have a negative impact when dealing with big datasets, and it also makes the problems of symmetry and asymmetry increasingly prominent. Feature selection (FS) can eliminate irrelevant information in big data and improve accuracy. As a recently proposed algorithm, the Sparrow Search Algorithm (SSA) shows its advantages in the FS tasks because of its superior performance. However, SSA is more subject to the population’s poor diversity and falls into a local optimum. Regarding this issue, we propose a variant of the SSA called the Tent Lévy Flying Sparrow Search Algorithm (TFSSA) to select the best subset of features in the wrapper-based method for classification purposes. After the performance results are evaluated on the CEC2020 test suite, TFSSA is used to select the best feature combination to maximize classification accuracy and simultaneously minimize the number of selected features. To evaluate the proposed TFSSA, we have conducted experiments on twenty-one datasets from the UCI repository to compare with nine algorithms in the literature. Nine metrics are used to evaluate and compare these algorithms’ performance properly. Furthermore, the method is also used on the coronavirus disease (COVID-19) dataset, and its classification accuracy and the average number of feature selections are 93.47% and 2.1, respectively, reaching the best. The experimental results and comparison in all datasets demonstrate the effectiveness of our new algorithm, TFSSA, compared with other wrapper-based algorithms.

In many pattern recognition applications feature selection and instance selection can be used as two data preprocessing methods that aim at reducing the computational cost of the learning process. Moreover, in some cases, feature subset selection can improve the classification performance. Feature selection and instance selection can be interesting since the choice of features and instances greatly influence the performance of the learnt models as well as their training costs. In the past, unifying both problems was carried out by solving a global optimization problem using meta-heuristics. This paradigm not only does not exploit the manifold structure of data but can be computationally expensive. To the best of our knowledge, the joint use of sparse modeling representative and feature subset relevance have not been exploited by the joint feature and selection methods. In this paper, we target the joint feature and instance selection by adopting feature subset relevance and sparse modeling representative selection. More precisely, we propose three schemes for the joint feature and instance selection. The first is a wrapper technique while the two remaining ones are filter approaches. In the filter approaches, the search process adopts a genetic algorithm in which the evaluation is mainly given by a score that quantify the goodness of the features and instances. An efficient instance selection technique is used and integrated in the search process in order to adapt the instances to the candidate feature subset. We evaluate the performance of the proposed schemes using image classification where classifiers are the nearest neighbor classifier and support vector machine classifier. The study is conducted on five public image datasets. These experiments show the superiority of the proposed schemes over various baselines. The results confirm that the filter approaches leads to promising improvement on classification accuracy when both feature selection and instance selection are adopted.