Joint feature and instance selection using manifold data criteria: application to image classification

Co-operative co-evolution approach solves problems by breaking them into subproblems. The proposed framework for Clinical Decision Support System (CDSS) uses a co-operative coevolution approach which treats Feature Selection (FS) and Instance Selection (IS) as independent subproblems. FS and IS remove less relevant features and instances, respectively, thereby improving the overall performance of the system. In this work, both Feature and Instance selection are done using the wrapper approach, which uses co-operative co-evolution and random forest classifier. The reduced dataset is used to train a random forest classifier and this trained model helped in making clinical decisions. These decisions assist physicians as the second opinion for diagnosis and treatment. Wisconsin Diagnostic Breast Cancer (WDBC), Hepatitis, Pima Indian Diabetes (PID), Cleveland Heart Disease (CHD), Statlog Heart Disease (SHD), Vertebral Column, and Hepatocellular Carcinoma (HCC) datasets from the University of California Irvine (UCI) Machine Learning repository are used for experimentation. The proposed framework achieved an accuracy of 97.1%, 82.3%, 81.01%, 93.4%, 96.8%, 91.4%, and 72.2% for datasets WDBC, Hepatitis, PID, CHD, SHD, vertebral column, and HCC, respectively. The results prove that the CDSS-developed using co-operative coevolution can efficiently assist the physicians in decision-making.

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

<p>Classification problems often have a large number of features, but not all of them are useful for classification. Irrelevant and redundant features may even reduce the classification accuracy. Feature selection is a process of selecting a subset of relevant features, which can decrease the dimensionality, shorten the running time, and/or improve the classification accuracy. There are two types of feature selection approaches, i.e. wrapper and filter approaches. Their main difference is that wrappers use a classification algorithm to evaluate the goodness of the features during the feature selection process while filters are independent of any classification algorithm. Feature selection is a difficult task because of feature interactions and the large search space. Existing feature selection methods suffer from different problems, such as stagnation in local optima and high computational cost. Evolutionary computation (EC) techniques are well-known global search algorithms. Particle swarm optimisation (PSO) is an EC technique that is computationally less expensive and can converge faster than other methods. PSO has been successfully applied to many areas, but its potential for feature selection has not been fully investigated. The overall goal of this thesis is to investigate and improve the capability of PSO for feature selection to select a smaller number of features and achieve similar or better classification performance than using all features. This thesis investigates the use of PSO for both wrapper and filter, and for both single objective and multi-objective feature selection, and also investigates the differences between wrappers and filters. This thesis proposes a new PSO based wrapper, single objective feature selection approach by developing new initialisation and updating mechanisms. The results show that by considering the number of features in the initialisation and updating procedures, the new algorithm can improve the classification performance, reduce the number of features and decrease computational time. This thesis develops the first PSO based wrapper multi-objective feature selection approach, which aims to maximise the classification accuracy and simultaneously minimise the number of features. The results show that the proposed multi-objective algorithm can obtain more and better feature subsets than single objective algorithms, and outperform other well-known EC based multi-objective feature selection algorithms. This thesis develops a filter, single objective feature selection approach based on PSO and information theory. Two measures are proposed to evaluate the relevance of the selected features based on each pair of features and a group of features, respectively. The results show that PSO and information based algorithms can successfully address feature selection tasks. The group based method achieves higher classification accuracies, but the pair based method is faster and selects smaller feature subsets. This thesis proposes the first PSO based multi-objective filter feature selection approach using information based measures. This work is also the first work using other two well-known multi-objective EC algorithms in filter feature selection, which are also used to compare the performance of the PSO based approach. The results show that the PSO based multiobjective filter approach can successfully address feature selection problems, outperform single objective filter algorithms and achieve better classification performance than other multi-objective algorithms. This thesis investigates the difference between wrapper and filter approaches in terms of the classification performance and computational time, and also examines the generality of wrappers. The results show that wrappers generally achieve better or similar classification performance than filters, but do not always need longer computational time than filters. The results also show that wrappers built with simple classification algorithms can be general to other classification algorithms.</p>

Joint imbalanced classification and feature selection for hospital readmissions

• Neural Network feature selection methodology for both numerical and categorical data. • Embedded feature selection provides model and feature subset in a single training. • Novel feature embedding for dealing with high-cardinality categorical features. • Validation of the methodology on several open and real industry dataset. • Comparison with several state-of-the-art feature selection methodologie. In an era of effortless data collection, the impact of machine learning — especially neural networks (NNs) — is undeniable. As datasets grow in size and complexity, efficiently handling mixed data types, including categorical and numerical features, becomes critical. Feature encoding and selection play a key role in improving NN performance, efficiency, interpretability, and generalisation. This paper presents GLEm-Net (Grouped Lasso with Embeddings Network), a novel NN-based approach that seamlessly integrates feature encoding and selection directly into the training process. GLEm-Net uses embedding layers to process categorical features with high cardinality, simplifying the model and improving generalisation. By extending the grouped Lasso regularisation to explicitly consider categorical features, GLEm-Net automatically identifies the most relevant features during training and returns them to the analyst. We evaluate GLEm-Net on open and proprietary industry datasets and compare it to state-of-the-art feature selection methodologies. Results show that GLEm-Net adapts to each dataset by allowing the NN to directly select subsets of most important features, offering on par performance with the best state-of-the-art feature selection methods, while eliminating the need for the external feature encoding and selection steps that are now incorporated in the NN training stage.

Feature selection has emerged as a craft, using which we boost the performance of our learning model. Feature or Attribute Selection is a data preprocessing technique, where only the most informative features are considered and given to the predictor. This reduces the computational overhead and improves the correctness of the classifier. Attribute Selection is commonly carried out by applying some filter or by using the performance of the learning model to gauge the quality of the attribute subset. Metric Ranked Feature Inclusion and Accuracy Ranked Feature Inclusion are the two novel hybrid feature selection methods we introduce in this paper. These algorithms follow a two-stage procedure, the first of which is feature ranking, followed by feature subset selection. They differ in the way they rank the features but follow the same subset selection technique. Multiple experiments have been conducted to assess our models. We compare our results with numerous works of the past and validate our models using 12 datasets. From the results, we infer that our algorithms perform better than many existent models.

Both feature selection and hyperparameter tuning are key tasks in machine learning. Hyperparameter tuning is often useful to increase model performance, while feature selection is undertaken to attain sparse models. Sparsity may yield better model interpretability and lower cost of data acquisition, data handling and model inference. While sparsity may have a beneficial or detrimental effect on predictive performance, a small drop in performance may be acceptable in return for a substantial gain in sparseness. We therefore treat feature selection as a multi-objective optimization task. We perform hyperparameter tuning and feature selection simultaneously because the choice of features of a model may influence what hyperparameters perform well. We present, benchmark, and compare two different approaches for multi-objective joint hyperparameter optimization and feature selection: The first uses multi-objective model-based optimization. The second is an evolutionary NSGA-II-based wrapper approach to feature selection which incorporates specialized sampling, mutation and recombination operators. Both methods make use of parameterized filter ensembles. While model-based optimization needs fewer objective evaluations to achieve good performance, it incurs computational overhead compared to the NSGA-II, so the preferred choice depends on the cost of evaluating a model on given data.

This paper describes an original technique for the joint feature and model selection in the context of support vector machine (SVM) classification applied as a diagnosis strategy in model-based fault detection and isolation (FDI). We demonstrate that the proposed technique contributes to the solution of an open research problem: to design a robust FDI procedure, correctly functioning with different operating conditions and fault sizes, specifically settled for an electric generation system based on solid oxide fuel cells (SOFCs). By using a quantitative model of the generation system coupled to an optimized SVM classifier, a satisfactory FDI procedure is achieved, which is robust against modeling and measurement errors and is compliant with practical deployment.

Summary To classify high‐dimensional data, feature selection plays a key role to eliminate irrelevant attributes and enhance the classification accuracy and efficiency. Since feature selection is an NP‐Hard problem, many heuristics and metaheuristics have been used to tackle in practice this problem. In this article, we propose a novel approach that consists in a probabilistic selection hyper‐heuristic called the synergy Thompson sampling hyper‐heuristic. The Thompson sampling selection strategy is a probabilistic reinforcement learning mechanism to assess the behavior of the low‐level heuristics, and to predict which one will be more efficient at each point during the search process. The proposed hyper‐heuristic is combined with a 1 nearest neighbor classifier from the Weka framework. It aims to find the best subset of features that maximizes the classification accuracy rate. Experimental results show a good performance in favor of the proposed method when comparing with other existing approaches.

Based on benchmarking various methods and analyzing multiple real scRNA-seq datasets, a computational platform/workflow and a set of tips for best practices are developed for analyzing causal interactions or relationships in single cells.

Feature subset selection (FSS) has received a great deal of attention in statistics, machine learning, and data mining. Real world data analyzed by data mining algorithms can involve a large number of redundant or irrelevant features or simply too many features for a learning algorithm to handle them efficiently. Feature selection is becoming essential as databases grow in size and complexity. The selection process is expected to bring benefits in terms of better performing models, computational efficiency, and simpler more understandable models. Evolutionary computation (EC) encompasses a number of naturally inspired techniques such as genetic algorithms, genetic programming, ant colony optimization, or particle swarm optimization algorithms. Such techniques are well suited to feature selection because the representation of a feature subset is straightforward and the evaluation can also be easily accomplished through the use of wrapper or filter algorithms. Furthermore, the capability of such heuristic algorithms to efficiently search large search spaces is of great advantage to the feature selection problem. Here, we review the use of different EC paradigms for feature selection in classification problems. We discuss details of each implementation including representation, evaluation, and validation. The review enables us to uncover the best EC algorithms for FSS and to point at future research directions.

The goal of the feature selection process is, given a dataset described by n attributes (features), to find the minimum number m of relevant attributes which describe the data as well as the original set of attributes do. Genetic algorithms have been used to implement feature selection algorithms. Previous algorithms presented in the literature used the predictive accuracy of a specific learning algorithm as the fitness function to maximize over the space of possible feature subsets. Such an approach to feature selection requires a large amount of CPU time to reach a good solution on large datasets. This paper presents a genetic algorithm for feature selection which improves previous results presented in the literature for genetic-based feature selection. It is independent of a specific learning algorithm and requires less CPU time to reach a relevant subset of features. Reported experiments show that the proposed algorithm is at least ten times faster than a standard genetic algorithm for feature selection without a loss of predictive accuracy when a learning algorithm is applied to reduced data.

Fault diagnosis methods based on machine learning have made great progress for rotating machinery. The main steps of the machine learning process involve feature extraction, selection, and classification. Feature selection improves classification accuracy and reduces diagnosis time by selecting the better features. Due to the difficulty of traditional feature selection methods to rank the feature importance of each class, the best subset of features could hardly be obtained. Therefore, this article proposes a new feature selection method to address the shortcomings of the above traditional methods, called Feature Ranking based on Optimal Class Distance Ratio (FROCDR), which can choose the optimal features between every two classes of samples to obtain feature ranking that is conducive to classification. In order to comprehensively extract the fault information in the signal, the multiscale analysis and the variational mode decomposition (VMD) method are applied to process the vibration signals under different scales and frequency bands, and the processed signals are visualized by symmetrized dot pattern (SDP). In addition, features are extracted from the obtained SDP images, and the proposed FROCDR method is used to select the best subset of features. The final diagnosis task is accomplished by a random forest (RF) classifier. Experimental cases of bearing and gear data show that the proposed method has higher diagnostic accuracy and stability.

Finding defects early in a software system is a crucial task, as it creates adequate time for fixing such defects using available resources. Strategies such as symmetric testing have proven useful; however, its inability in differentiating incorrect implementations from correct ones is a drawback. Software defect prediction (SDP) is another feasible method that can be used for detecting defects early. Additionally, high dimensionality, a data quality problem, has a detrimental effect on the predictive capability of SDP models. Feature selection (FS) has been used as a feasible solution for solving the high dimensionality issue in SDP. According to current literature, the two basic forms of FS approaches are filter-based feature selection (FFS) and wrapper-based feature selection (WFS). Between the two, WFS approaches have been deemed to be superior. However, WFS methods have a high computational cost due to the unknown number of executions available for feature subset search, evaluation, and selection. This characteristic of WFS often leads to overfitting of classifier models due to its easy trapping in local maxima. The trapping of the WFS subset evaluator in local maxima can be overcome by using an effective search method in the evaluator process. Hence, this study proposes an enhanced WFS method that dynamically and iteratively selects features. The proposed enhanced WFS (EWFS) method is based on incrementally selecting features while considering previously selected features in its search space. The novelty of EWFS is based on the enhancement of the subset evaluation process of WFS methods by deploying a dynamic re-ranking strategy that iteratively selects germane features with a low subset evaluation cycle while not compromising the prediction performance of the ensuing model. For evaluation, EWFS was deployed with Decision Tree (DT) and Naïve Bayes classifiers on software defect datasets with varying granularities. The experimental findings revealed that EWFS outperformed existing metaheuristics and sequential search-based WFS approaches established in this work. Additionally, EWFS selected fewer features with less computational time as compared with existing metaheuristics and sequential search-based WFS methods.

This paper explores the use of rough set theory for feature selection in high dimensional haptic-based handwritten signatures (exploited for user identification). Two rough set-based methods for feature selection are analyzed, the first is a greedy approach while the second relies on genetic algorithms to find minimal subsets of attributes. Also, to further reduce the haptic feature space while maximizing user identification accuracy, a method is proposed where feature vectors are subsampled prior to the feature selection procedure. Rough set-generated minimal subsets are initially exploited to determine the importance of different haptic data types (e.g. force, position, torque and orientation) in discriminating between different users. In addition, a comparison between rough set-based methods and classical machine learning techniques in the selection of minimal information-preserving subsets of features in high dimensional haptic datasets, is provided. The criteria for comparison are the length of the selected subsets of features and their corresponding discrimination power. Support Vector Machine classifiers are used to evaluate the accuracy of the selected minimal feature vectors. The results demonstrated that the combination of rough set and genetic algorithm techniques can outperform well-established machine learning methods in the selection of minimal subsets of features present in haptic-based handwritten signatures.