A Multiobjective Genetic Programming-Based Ensemble for Simultaneous Feature Selection and Classification.

PSOLDA: A particle swarm optimization approach for enhancing classification accuracy rate of linear discriminant analysis

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

<p>Classification problems often have a large number of features, but not all of them are useful for classification. Irrelevant and redundant features may even reduce the classification accuracy. Feature selection is a process of selecting a subset of relevant features, which can decrease the dimensionality, shorten the running time, and/or improve the classification accuracy. There are two types of feature selection approaches, i.e. wrapper and filter approaches. Their main difference is that wrappers use a classification algorithm to evaluate the goodness of the features during the feature selection process while filters are independent of any classification algorithm. Feature selection is a difficult task because of feature interactions and the large search space. Existing feature selection methods suffer from different problems, such as stagnation in local optima and high computational cost. Evolutionary computation (EC) techniques are well-known global search algorithms. Particle swarm optimisation (PSO) is an EC technique that is computationally less expensive and can converge faster than other methods. PSO has been successfully applied to many areas, but its potential for feature selection has not been fully investigated. The overall goal of this thesis is to investigate and improve the capability of PSO for feature selection to select a smaller number of features and achieve similar or better classification performance than using all features. This thesis investigates the use of PSO for both wrapper and filter, and for both single objective and multi-objective feature selection, and also investigates the differences between wrappers and filters. This thesis proposes a new PSO based wrapper, single objective feature selection approach by developing new initialisation and updating mechanisms. The results show that by considering the number of features in the initialisation and updating procedures, the new algorithm can improve the classification performance, reduce the number of features and decrease computational time. This thesis develops the first PSO based wrapper multi-objective feature selection approach, which aims to maximise the classification accuracy and simultaneously minimise the number of features. The results show that the proposed multi-objective algorithm can obtain more and better feature subsets than single objective algorithms, and outperform other well-known EC based multi-objective feature selection algorithms. This thesis develops a filter, single objective feature selection approach based on PSO and information theory. Two measures are proposed to evaluate the relevance of the selected features based on each pair of features and a group of features, respectively. The results show that PSO and information based algorithms can successfully address feature selection tasks. The group based method achieves higher classification accuracies, but the pair based method is faster and selects smaller feature subsets. This thesis proposes the first PSO based multi-objective filter feature selection approach using information based measures. This work is also the first work using other two well-known multi-objective EC algorithms in filter feature selection, which are also used to compare the performance of the PSO based approach. The results show that the PSO based multiobjective filter approach can successfully address feature selection problems, outperform single objective filter algorithms and achieve better classification performance than other multi-objective algorithms. This thesis investigates the difference between wrapper and filter approaches in terms of the classification performance and computational time, and also examines the generality of wrappers. The results show that wrappers generally achieve better or similar classification performance than filters, but do not always need longer computational time than filters. The results also show that wrappers built with simple classification algorithms can be general to other classification algorithms.</p>

Automated design of job shop scheduling rules using genetic programming as a hyper-heuristic is an emerging topic that has become more and more popular in recent years. For evolving dispatching rules, feature selection is an important issue for deciding the terminal set of genetic programming. There can be a large number of features, whose importance/relevance varies from one to another. It has been shown that using a promising feature subset can lead to a significant improvement over using all the features. However, the existing feature selection algorithm for job shop scheduling is too slow and inapplicable in practice. In this paper, we propose the first “practical” feature selection algorithm for job shop scheduling. Our contributions are twofold. First, we develop a Niching-based search framework for extracting a diverse set of good rules. Second, we reduce the complexity of fitness evaluation by using a surrogate model. As a result, the proposed feature selection algorithm is very efficient. The experimental studies show that it takes less than 10% of the training time of the standard genetic programming training process, and can obtain much better feature subsets than the entire feature set. Furthermore, it can find better feature subsets than the best-so-far feature subset. © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.<div><br></div><div><div><table><tr><td>© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.</td></tr></table></div></div>

Most feature selection methods only perform well on datasets with relatively small set of features. In the case of large feature sets and small number of data points, almost none of the existing feature selection methods help in achieving high accuracy. This paper proposes a novel approach to optimize the feature selection process through Frequent Pattern Growth algorithm to find sets of features that appear frequently among the top features selected by the main feature selection methods. Our experimental evaluation on two datasets containing a small and very large number of features shows that our approach significantly improves the accuracy results of the dataset with a very large number of features.

<p>Symbolic regression (SR) is a function identification process, the task of which is to identify and express the relationship between the input and output variables in mathematical models. SR is named to emphasise its ability to find the structure and coefficients of the model simultaneously. Genetic Programming (GP) is an attractive and powerful technique for SR, since it does not require any predefined model and has a flexible representation. However, GP based SR generally has a poor generalisation ability which degrades its reliability and hampers its applications to science and real-world modeling. Therefore, this thesis aims to develop new GP approaches to SR that evolve/learn models exhibiting good generalisation ability. This thesis develops a novel feature selection method in GP for high-dimensional SR. Feature selection can potentially contribute not only to improving the efficiency of learning algorithms but also to enhancing the generalisation ability. However, feature selection is seldom considered in GP for high-dimensional SR. The proposed new feature selection method utilises GP’s built-in feature selection ability and relies on permutation to detect the truly relevant features and discard irrelevant/noisy features. The results confirm the superiority of the proposed method over the other examined feature selection methods including random forests and decision trees on identifying the truly relevant features. Further analysis indicates that the models evolved by GP with the proposed feature selection method are more likely to contain only the truly relevant features and have better interpretability. To address the overfitting issue of GP when learning from a relatively small number of instances, this thesis proposes a new GP approach by incorporating structural risk minimisation (SRM), which is a framework to estimate the generalisation performance of models, into GP. The effectiveness of SRM highly depends on the accuracy of the Vapnik-Chervonenkis (VC) dimension measuring model complexity. This thesis significantly extends an experimental method (instead of theoretical estimation) to measure the VC-dimension of a mixture of linear and nonlinear regression models in GP for the first time. The experimental method has been conducted using uniform and non-uniform settings and provides reliable VC-dimension values. The results show that our methods have an impressively better generalisation gain and evolve more compact model, which have a much smaller behavioural difference from the target models than standard GP and GP with bootstrap, The proposed method using the optimised non-uniform setting further improves the one using the uniform setting. This thesis employs geometric semantic GP (GSGP) to tackle the unsatisfied generalisation performance of GP for SR when no overfitting occurs. It proposes three new angle-awareness driven geometric semantic operators (GSO) including selection, crossover and mutation to further explore the geometry of the semantic space to gain a greater generalisation improvement in GP for SR. The angle-awareness brings new geometric properties to these geometric operators, which are expected to provide a greater leverage for approximating the target semantics in each operation, and more importantly, to be resistant to overfitting. The results show that compared with two kinds of state-of-the-art GSOs, the proposed new GSOs not only drive the evolutionary process fitting the target semantics more efficiently but also significantly improve the generalisation performance. A further comparison on the evolved models shows that the new method generally produces simpler models with a much smaller size and containing important building blocks of the target models.</p>

We consider feature selection for text classification both theoretically and empirically. Our main result is an unsupervised feature selection strategy for which we give worst-case theoretical guarantees on the generalization power of the resultant classification function f with respect to the classification function f obtained when keeping all the features. To the best of our knowledge, this is the first feature selection method with such guarantees. In addition, the analysis leads to insights as to when and why this feature selection strategy will perform well in practice. We then use the TechTC-100, 20-Newsgroups, and Reuters-RCV2 data sets to evaluate empirically the performance of this and two simpler but related feature selection strategies against two commonly-used strategies. Our empirical evaluation shows that the strategy with provable performance guarantees performs well in comparison with other commonly-used feature selection strategies. In addition, it performs better on certain datasets under very aggressive feature selection.

In the present state of affairs, the motive behind every educational organization is to uplift the academic achievement of students. Educational data mining (EDM) is an upward field of research, and it is very helpful for academic institutions to predict the academic performance of the students. Educational datasets are the basis of various predictive models. Quality of these models can be improved by using feature selection (FS). To get the required benefits from the available data, there must be some tools for analysis and prediction. In lieu of the above, machine learning/data mining are most suitable. In educational data mining, for better accuracy of prediction models’ and quality of various educational datasets, feature selection (FS) plays a vital role. Feature selection (FS) algorithms abolish inappropriate information from the repositories of educational background so that performance of classifier in terms of accuracy could be increased and the same could be used for better decision. In lieu of the above, a best feature selection algorithm must be selected. In this paper, two filter selection approaches namely correlation feature selection (CFS) and wrapper-based feature selection have been used to demonstrate the importance of selection of a feature subset for a classification problem. The present paper aims to find the detailed investigation of filter feature selection algorithms along with the classification algorithms on a given dataset. We found result with numerous numbers of features from various Feature selection algorithms and classifiers which will help the researcher to discover the most excellent mixture of filter feature selection algorithms and its associated classifiers. The result indicates that SMO and J48 have the highest accuracy measures with the correlation feature selection algorithms, while Naïve Bayes has the highest accuracy measures with the wrapper subset feature selection algorithms for predicting high, medium and low grade for the students.

Cancer is a world-famous deadly disease. According to the World Health Organization (WHO), cancer is the second leading cause of death globally and is responsible for an estimated 9.6 million deaths in 2018. One well-known technique for cancer detection is the DNA microarray technique. DNA microarray technology provides an opportunity for researchers to analyze thousands of gene expression profiles at the same time to determine whether a person has cancer or not. However, one of the problems in DNA microarray data is the large number of features that require feature selection. In overcoming these problems, this study will use the feature selection Support Vector Machine-Recursive Feature Elimination (SVM-RFE) and Chi-Square and use the Naive Bayes classification method. The accuracy results from using feature selection with those that are not will be compared. The accuracy between using the two feature selection methods will also be compared to find which feature selection method is better when combined with the Naive Bayes classification method. To get an overall picture of the performance comparison, this study also considers precision, recall, and F1-score. The best accuracy results obtained were 100% lung cancer data with SVM-RFE and Chi-Square, 99.6% ovarian cancer with SVM-RFE, 93.7% breast cancer with SVM-RFE, and 90% colon cancer with SVM- RFE.

Feature selection methods and their combinations in high-dimensional classification of speaker likability, intelligibility and personality traits

Automatic epileptic seizure detection technologies for clinical diagnosis mainly rely on electroencephalogram (EEG) recordings, which are immensely useful tools for epileptic location and identification. Currently, traditional seizure detection methods based only on single-view features have great limitations for the typical dynamic and nonlinear EEG signals. An objective of this paper is to investigate the effect of multiview feature selection and multilevel spectral analysis methods on the identification of the EEG signals for seizure detection. Here, multiview features are extracted from time domain, frequency domain, and information theory to collect adequate information of EEG signals. And a feature selection algorithm based on particle swarm optimization (PSO) is proposed for automatic seizure detection. Moreover, due to the different frequency components of the EEG signals, they are divided into four kinds of brain waves for multilevel spectral analysis. The effect of these four rhythm waves on seizure detection is compared. Three well-known classifiers are employed to classify EEG signals concerning seizure or nonseizure events. The result shows that the average accuracy, specificity, and sensitivity of classification with the CHB-MIT database are 98.14%, 98.64%, and 96.79%, respectively. The application of the PSO-based feature selection method for automatic seizure detection improves accuracy by 5.99% with the SVM classifier. Compared with the state-of-the-art methods, the proposed method has superior competence with high performance for automatic seizure detection. It is further shown that the feature selection method is an indispensable step in seizure detection. With PSO-based feature selection and multilevel spectral analysis, the θ wave in the frequency range of 4-7 Hz shows better performance in the identification of EEG signals and is more suitable for the proposed method. The PSO-based feature selection algorithm for automatic seizure detection can be a useful assistant tool for clinical diagnosis.

Background: Cholangiocarcinoma is a molecularly heterogenous cancer arising from the biliary epithelium. Individual omic techniques have identified clinically relevant, targetable subtypes but most tumors have no targetable alterations based on exome or transcriptomic evaluation. Integrated multiomics approaches incorporating transcriptomic, proteomic, and phosphoproteomic characterization provide deeper understanding and identify non-mutated, activated pathways that can be targeted. To optimize the potential clinical utility of these approaches, minimal classifier features must be defined. Methods: Approximately 60,000 RNA, protein, and phosphoprotein features were extracted from 35 cholangiocarcinomas treated at the primary US study site. 206 patients treated and profiled at an institution in China were also included. Using the Multiomics Factor Analysis clustering method, patients were sorted into 3 distinct molecular profiles. Important molecular features were selected using competing feature selection (FS) methods: Boruta, RreliefF, variable selection using random forests (RF), recursive factor elimination (RFE), median decrease accuracy in RF, and Concrete Autoencoders. Machine learning classification prediction models including Extreme Gradient Boosting, gradient-boosted trees, multinomial logit, and support vector machine (SVM) were tested on each FS. Each combination of FS and predictive model was tested with 10 replicates of 10-fold cross-validation of a randomly-selected balanced test set and an oversampled, balanced training subset. Accuracy, F1 score, AUC, precision, and recall were assessed for each FS and predictive model combination. A custom function of physical assay cost and classification accuracy was optimized to find the best assay. KEGG pathway overrepresentation analysis was used to analyze feature subsets. Results: A consistent set of 50 important RNA and protein features was identified, including previously reported prognostic factors, like SpryD4 protein expression, as well as previously unreported pathway surrogates, such as Slc2A2 or Pipox protein. Pathway overrepresentation analysis found insulin signaling and resistance and amino acid biosynthesis/metabolism pathways were overrepresented in the discriminative feature set. The gradient-boosted trees algorithm combined with the RFE FS method had the best performance, with a multiclass macro average (±sd) cross-validation accuracy 89% (±6%), F1 score 89% (±8%), AUC 0.98 (±0.02), precision 90% (±10%), recall 89% (±6%). Conclusions: Feature selection methods and predictive machine learning models work synchronously to develop accurate molecular subtype assays based on just 50 protein and RNA levels. This strategy has the potential to make molecular subtyping for cholangiocarcinoma and other molecularly complex cancers fast and widely available. Citation Format: Ellen Larson, Erik Jessen, Dong-Gi Mun, Jennifer Tomlinson, Amro Abdelrahman, Danielle Carlson, Hojjat Salehinejad, Rory Smoot. Feature selection and machine learning strategies optimize an affordable molecular assay for cholangiocarcinoma subtype [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 1107.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

Maximum relevancy maximum complementary feature selection for multi-sensor activity recognition

Stock market prediction is a very noisy problem and the use of any additional information to increase accuracy is necessary. In this paper, for the stock daily return prediction problem, the set of features is expanded to include indicators not only for the stock to be predicted itself but also a set of other stocks and currencies. Afterwards, different feature selection and classification methods are utilized for prediction. The daily close returns of the 3 most traded stocks (GARAN, THYAO, and ISCTR) in Borsa Istanbul (BIST) are predicted using indicators computed on those stocks, indicators for all the other stocks listed in the BIST100 index, and indicators on the dollar-gold prices. Twenty-five different indicators on daily stock prices are computed to form feature vectors for each trading day. These feature vectors are assigned class labels according to the daily close returns. Expanding the feature space with BIST100 stocks features results in a high dimensional feature space, with possibly noisy or irrelevant features. Therefore, feature selection methods are utilized to select the most informative features. In order to determine relevance scores of features, fast filter-based methods, gain ratio and relief, are used. Experiments are performed based on individual stock features, dollar-gold features (DG), BIST100 stock features (BIST100), and a combination of BIST100 and DG with and without feature selection. Using the gain ratio feature selection with a gradient boosting machine (GBM), the movements of GARAN stock were predicted with an accuracy of 0.599 and an F-measure of 0.614. For THYAO, the relief feature selection with the GBM gave an accuracy of 0.558, and for ISCTR, the gain ratio feature selection with logistic regression achieved an accuracy of 0.581. It was found that using BIST100 stock features boosts classification results for all stocks in terms of accuracy.