Multi-Task Feature Selection for Advancing Performance of Image Segmentation

<p>Classification problems often have a large number of features, but not all of them are useful for classification. Irrelevant and redundant features may even reduce the classification accuracy. Feature selection is a process of selecting a subset of relevant features, which can decrease the dimensionality, shorten the running time, and/or improve the classification accuracy. There are two types of feature selection approaches, i.e. wrapper and filter approaches. Their main difference is that wrappers use a classification algorithm to evaluate the goodness of the features during the feature selection process while filters are independent of any classification algorithm. Feature selection is a difficult task because of feature interactions and the large search space. Existing feature selection methods suffer from different problems, such as stagnation in local optima and high computational cost. Evolutionary computation (EC) techniques are well-known global search algorithms. Particle swarm optimisation (PSO) is an EC technique that is computationally less expensive and can converge faster than other methods. PSO has been successfully applied to many areas, but its potential for feature selection has not been fully investigated. The overall goal of this thesis is to investigate and improve the capability of PSO for feature selection to select a smaller number of features and achieve similar or better classification performance than using all features. This thesis investigates the use of PSO for both wrapper and filter, and for both single objective and multi-objective feature selection, and also investigates the differences between wrappers and filters. This thesis proposes a new PSO based wrapper, single objective feature selection approach by developing new initialisation and updating mechanisms. The results show that by considering the number of features in the initialisation and updating procedures, the new algorithm can improve the classification performance, reduce the number of features and decrease computational time. This thesis develops the first PSO based wrapper multi-objective feature selection approach, which aims to maximise the classification accuracy and simultaneously minimise the number of features. The results show that the proposed multi-objective algorithm can obtain more and better feature subsets than single objective algorithms, and outperform other well-known EC based multi-objective feature selection algorithms. This thesis develops a filter, single objective feature selection approach based on PSO and information theory. Two measures are proposed to evaluate the relevance of the selected features based on each pair of features and a group of features, respectively. The results show that PSO and information based algorithms can successfully address feature selection tasks. The group based method achieves higher classification accuracies, but the pair based method is faster and selects smaller feature subsets. This thesis proposes the first PSO based multi-objective filter feature selection approach using information based measures. This work is also the first work using other two well-known multi-objective EC algorithms in filter feature selection, which are also used to compare the performance of the PSO based approach. The results show that the PSO based multiobjective filter approach can successfully address feature selection problems, outperform single objective filter algorithms and achieve better classification performance than other multi-objective algorithms. This thesis investigates the difference between wrapper and filter approaches in terms of the classification performance and computational time, and also examines the generality of wrappers. The results show that wrappers generally achieve better or similar classification performance than filters, but do not always need longer computational time than filters. The results also show that wrappers built with simple classification algorithms can be general to other classification algorithms.</p>

High-feature dimensions is one of the obvious challenges facing large-scale hierarchical classification problem. One promising way to overcome such problem is reducing the features size by removing irrelevant, noise and redundant features. Filter-based feature selection methods are commonly used to construct smaller subset of most relevant features in either adaptive or global approach at each non leaf node within the hierarchy. Both the Feature selection approaches do not only reduce time and space complexity but also improve classification accuracy of the learning models. In this study, the effectiveness of the main approaches for constructing relevant features subset, that is adaptive and global approaches are analyzed using the two filter-based methods suitable for text classification. Experiment evaluation conducted on 20NG, IPC, and DMOZ-small datasets shows that implementing feature selection adaptively has more capability of selecting most relevant features, thus, achieving the best classification accuracy. Moreover, further evaluation shown integrating feature selection with LSHC records improve performance when compared to without feature selection.

In recent years, detecting credit card fraud transactions has been a difficult task due to the high dimensions and imbalanced datasets. Selecting a subset of important features from a high-dimensional dataset has proven to be the most prominent approach for solving high-dimensional dataset issues, and the selection of features is critical for improving classification performance, such as the fraud transaction identification process. To contribute to the field, this paper proposes a novel feature selection (FS) approach based on a metaheuristic algorithm called Rock Hyrax Swarm Optimization Feature Selection (RHSOFS), inspired by the actions of rock hyrax swarms in nature, and implements supervised machine learning techniques to improve credit card fraud transaction identification approaches. This approach is used to select a subset of optimal relevant features from a high-dimensional dataset. In a comparative efficiency analysis, RHSOFS is compared with Differential Evolutionary Feature Selection (DEFS), Genetic Algorithm Feature Selection (GAFS), Particle Swarm Optimization Feature Selection (PSOFS), and Ant Colony Optimization Feature Selection (ACOFS) in a comparative efficiency analysis. The proposed RHSOFS outperforms existing approaches, such as DEFS, GAFS, PSOFS, and ACOFS, according to the experimental results. Various statistical tests have been used to validate the statistical significance of the proposed model.

Bi-stage feature selection for crop mapping using grey wolf metaheuristic optimization

There are two fundamentally different approaches for feature selection: wrapper and filter. It is also possible to combine them, obtaining hybrid approaches. This paper describes a hybrid method for selecting relevant features in clustering problems. The proposed approach is based on the combination of the widely known k-means algorithm and a Bayesian filter, which is based on the Markov Blanket concept. Since the number of clusters and the subset of relevant features are usually inter-related, we propose a method that iterates between clustering (assuming that the number of clusters is not known a priori) and filtering. Experiments in a number of datasets show that the proposed approach allows selecting features that provide good partitions.

Manifold regularized multi-view feature selection for social image annotation

In this work, we propose a novel Feature Selection framework, called Sparse-Modeling Based Approach for Class Specific Feature Selection (SMBA-CSFS), that simultaneously exploits the idea of Sparse Modeling and Class-Specific Feature Selection. Feature selection plays a key role in several fields (e.g., computational biology), making it possible to treat models with fewer variables which, in turn, are easier to explain, by providing valuable insights on the importance of their role, and might speed the experimental validation up. Unfortunately, also corroborated by the no free lunch theorems, none of the approaches in literature is the most apt to detect the optimal feature subset for building a final model, thus it still represents a challenge. The proposed feature selection procedure conceives a two steps approach: (a) a sparse modeling-based learning technique is first used to find the best subset of features, for each class of a training set; (b) the discovered feature subsets are then fed to a class-specific feature selection scheme, in order to assess the effectiveness of the selected features in classification tasks. To this end, an ensemble of classifiers is built, where each classifier is trained on its own feature subset discovered in the previous phase, and a proper decision rule is adopted to compute the ensemble responses. In order to evaluate the performance of the proposed method, extensive experiments have been performed on publicly available datasets, in particular belonging to the computational biology field where feature selection is indispensable: the acute lymphoblastic leukemia and acute myeloid leukemia, the human carcinomas, the human lung carcinomas, the diffuse large B-cell lymphoma, and the malignant glioma. SMBA-CSFS is able to identify/retrieve the most representative features that maximize the classification accuracy. With top 20 and 80 features, SMBA-CSFS exhibits a promising performance when compared to its competitors from literature, on all considered datasets, especially those with a higher number of features. Experiments show that the proposed approach might outperform the state-of-the-art methods when the number of features is high. For this reason, the introduced approach proposes itself for selection and classification of data with a large number of features and classes.

In machine learning, Case Based Reasoning is a prominent technique for harvesting knowledge from past experiences. The past experiences are represented in the form of a repository of cases having a set of features. But each feature may not have the equal relevancy in describing a case. Measuring the relevancy of each feature is always a prime issue. A subset of relevant features describes a case with adequate accuracy. An appropriate subset of relevant features should be selected for improving the performance of the system and to reduce dimensionality. In case based domain, feature selection is a process of selecting an appropriate subset of relevant features. There are various real domains which are inherently case based and features are expressed in terms of linguistic variables. To assign a numerical weight to each linguistic feature, a lot of feature subset selection algorithms have been proposed. But the weighting values are usually determined using subjective judgement or a trial and error basis.

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

In many pattern recognition applications feature selection and instance selection can be used as two data preprocessing methods that aim at reducing the computational cost of the learning process. Moreover, in some cases, feature subset selection can improve the classification performance. Feature selection and instance selection can be interesting since the choice of features and instances greatly influence the performance of the learnt models as well as their training costs. In the past, unifying both problems was carried out by solving a global optimization problem using meta-heuristics. This paradigm not only does not exploit the manifold structure of data but can be computationally expensive. To the best of our knowledge, the joint use of sparse modeling representative and feature subset relevance have not been exploited by the joint feature and selection methods. In this paper, we target the joint feature and instance selection by adopting feature subset relevance and sparse modeling representative selection. More precisely, we propose three schemes for the joint feature and instance selection. The first is a wrapper technique while the two remaining ones are filter approaches. In the filter approaches, the search process adopts a genetic algorithm in which the evaluation is mainly given by a score that quantify the goodness of the features and instances. An efficient instance selection technique is used and integrated in the search process in order to adapt the instances to the candidate feature subset. We evaluate the performance of the proposed schemes using image classification where classifiers are the nearest neighbor classifier and support vector machine classifier. The study is conducted on five public image datasets. These experiments show the superiority of the proposed schemes over various baselines. The results confirm that the filter approaches leads to promising improvement on classification accuracy when both feature selection and instance selection are adopted.

This study is aimed at assessing the usefulness of different feature selection and classification methodologies in the context of sleep apnea hypopnea syndrome (SAHS) detection. Feature extraction, selection and classification stages were applied to analyze blood oxygen saturation (SaO2) recordings in order to simplify polysomnography (PSG), the gold standard diagnostic methodology for SAHS. Statistical, spectral and nonlinear measures were computed to compose the initial feature set. Principal component analysis (PCA), forward stepwise feature selection (FSFS) and genetic algorithms (GAs) were applied to select feature subsets. Fisher's linear discriminant (FLD), logistic regression (LR) and support vector machines (SVMs) were applied in the classification stage. Optimum classification algorithms from each combination of these feature selection and classification approaches were prospectively validated on datasets from two independent sleep units. FSFS + LR achieved the highest diagnostic performance using a small feature subset (4 features), reaching 83.2% accuracy in the validation set and 88.7% accuracy in the test set. Similarly, GAs + SVM also achieved high generalization capability using a small number of input features (7 features), with 84.2% accuracy on the validation set and 84.5% accuracy in the test set. Our results suggest that reduced subsets of complementary features (25% to 50% of total features) and classifiers with high generalization ability could provide high-performance screening tools in the context of SAHS.

Introduction Large language models (LLMs) have the potential to realize accurate risk stratification and disease prediction by integrating multimodal data, such as electronic health records, medical images, and genomic profiles. However, in complex tasks like atherosclerosis risk estimation, the design of the prompt is critical to bring out the performance of LLMs. Several researches have demonstrated that in-context learning, where the prompt contains some cases, improves the performance of LLMs when the contained cases are carefully selected. Because manual selection usually requires enormous costs, automatic case selection has been desired. Methods This study aims to improve the effectiveness of in-context learning in LLM for atherosclerosis risk prediction by introducing a strategy for case selection. Our method involves two key components: feature selection and case selection. In the feature selection phase, we compute the mutual information between the clinical features and atherosclerosis risk using the database to extract the essential clinical features. This feature selection removes the unimportant features that disturb the performance of the case selection mechanism. Then, in the case selection phase, we select several cases from the database that are the most similar to the patient being diagnosed. This study compares three metrics to compute the similarity score: - Mahalanobis distance (trained via large margin nearest neighbor classification; LMNN [1]) - Cosine similarity (between the vector representations in KNN-augmented in-context example selection; KATE [2]) - Euclidean distance (between the prompt features) This combined strategy supplies the LLM with informative cases and thereby enhances the diagnostic performance. Results We used a fine-tuned Llama3-8B model for our target language and analyzed a dataset of 117,709 cases. Each case contained 96 features that were collected from annual health check-up data. From this dataset, 1,000 cases each were randomly selected for validation and test datasets. The remaining data were used as the database in our system. We considered a binary classification task and assigned the positive label when the Cardio-Ankle Vascular Index (CAVI) was greater than 8.0. The proportion of selected features was adjusted in 10% increments to maximize the F1 score on the validation dataset. The result shows that our proposed method, the strategic feature and case selections, significantly improved the F1 score for the test dataset by around 2% compared with zero-shot prompt and random selection (see the attached table). Among three similarity metrics, Euclidean distance yielded the highest F1 scores. We also observed that the feature selection phase with the adjusted number of selected features improved the F1 score in almost all settings. Conclusion Our strategy for feature and case selections improved the performance of in-context learning for LLM-based atherosclerosis risk prediction.

• Neural Network feature selection methodology for both numerical and categorical data. • Embedded feature selection provides model and feature subset in a single training. • Novel feature embedding for dealing with high-cardinality categorical features. • Validation of the methodology on several open and real industry dataset. • Comparison with several state-of-the-art feature selection methodologie. In an era of effortless data collection, the impact of machine learning — especially neural networks (NNs) — is undeniable. As datasets grow in size and complexity, efficiently handling mixed data types, including categorical and numerical features, becomes critical. Feature encoding and selection play a key role in improving NN performance, efficiency, interpretability, and generalisation. This paper presents GLEm-Net (Grouped Lasso with Embeddings Network), a novel NN-based approach that seamlessly integrates feature encoding and selection directly into the training process. GLEm-Net uses embedding layers to process categorical features with high cardinality, simplifying the model and improving generalisation. By extending the grouped Lasso regularisation to explicitly consider categorical features, GLEm-Net automatically identifies the most relevant features during training and returns them to the analyst. We evaluate GLEm-Net on open and proprietary industry datasets and compare it to state-of-the-art feature selection methodologies. Results show that GLEm-Net adapts to each dataset by allowing the NN to directly select subsets of most important features, offering on par performance with the best state-of-the-art feature selection methods, while eliminating the need for the external feature encoding and selection steps that are now incorporated in the NN training stage.

According to whether the systems vary over time, the decision systems can be divided into two categories: static decision systems and dynamic decision systems. Most existing feature selection work is done for the former, few work has been developed recently for the latter. To the best of our knowledge, when an object set varies dynamically in incomplete decision systems, no feature selection approach has been specially designed to select feature subset until now. In this regard, a feature selection algorithm based on compact discernibility matrix is developed. The compact discernibility matrix is firstly introduced, which not only avoids computing the time-consuming lower approximation, but also saves more storage space than classical discernibility matrix. Afterwards, we take the change of lower approximation as a springboard to incrementally update the compact discernibility matrix. On the basis of updated compact discernibility matrix, an efficient feature selection algorithm is provided to compute a new feature subset, instead of retaining the discernibility matrix from scratch to find a new feature subset. The efficiency and effectiveness of the proposed algorithm are demonstrated by the experimental results on different data sets.

© 2015 IEEE. Feature selection often involves two conflicting objectives of minimizing the feature subset size and the maximizing the classification accuracy. In this paper, a multi-objective artificial bee colony (MOABC) framework is developed for feature selection in classification, and a new fuzzy mutual information based criterion is proposed to evaluate the relevance of feature subsets. Three new multi-objective feature selection approaches are proposed by integrating MOABC with three filter fitness evaluation criteria, which are mutual information, fuzzy mutual information and the proposed fuzzy mutual information. The proposed multi-objective feature selection approaches are examined by comparing them with three single-objective ABC-based feature selection approaches on six commonly used datasets. The results show that the proposed approaches are able to achieve better performance than the original feature set in terms of the classification accuracy and the number of features. By using the same evaluation criterion, the proposed multi-objective algorithms generally perform better than the single-objective methods, especially in terms of reducing the number of features. Furthermore, the proposed fuzzy mutual information criterion outperforms mutual information and the original fuzzy mutual information in both single-objective and multi-objective manners. This work is the first study on multi-objective ABC for filter feature selection in classification, which shows that multi-objective ABC can be effectively used to address feature selection problems.<div><br></div><div><div><table><tr><td>© 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.</td></tr></table></div></div>