Multilabel feature selection: A comprehensive review and guiding experiments

Insurance purchasing has been receiving increasing attention nowadays. Facing various insurance products, customers prefer the convincing recommendation by machine learning algorithms. The amount of personal information explosively grows due to the development of network and storage. Redundant features within the personal information may lead to reduced prediction accuracy and low efficiency. As a result, it is important to select effective features for multi-product prediction. Many current multi-label feature selection methods ignore the correlation between either features or labels. Thus, the selected features cannot comprehensively represent the leading factors of each label. In this paper, we propose a Multi-label Entropy-based Feature selection (MEF) method, aiming to provide a clear and reasonable explanation for insurance purchase prediction. Our method considers the relationship between features and removes redundant features. We compare our MEF model with 3 previously proposed multi-label feature selection methods on the Insurance Company Case datasets from CoIL Challenge 2000. The experiment results show that the MEF algorithm outperforms other methods with fewer features by 6 evaluation indexes and can effectively improve the classification performance of the multi-label classifier.

Enhanced multi-label feature selection considering label-specific relevant information

Ant-TD: Ant colony optimization plus temporal difference reinforcement learning for multi-label feature selection

A label-specific multi-label feature selection algorithm based on the Pareto dominance concept

Text datasets have many terms, which decrease the classification accuracy. According to the high-dimensional text data, there are more challenges for these methods. Each classification method has strengths and weaknesses in its feature selection function. Therefore, ensembling should be used for better classifications and exploitation of strengths. In this paper, for the first time, we have presented an ensemble multi-label (ML) feature selection method for the text datasets using the Geometric-Mean (GM) aggregation approach. For this purpose, we have used four multi-label feature selection (MLFS) algorithms with different structures to achieve a good result. Then, the performance and results obtained by the GM method are compared with the four algorithms and based on the six classification criteria on three ML datasets with text domains. According to the obtained outputs, it is possible to realize the ability of the GMA (proposed method) method in using strengths and ignoring weaknesses in the path of feature selection, and be more accurate. Finally, according to experiments and obtained results, superiority of the GMA over other methods can be well seen.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

QSAR analysis of diaryl COX-2 inhibitors: Comparison of feature selection and train-test data selection methods

Prediction of the stage of cancer plays an important role in planning the course of treatment and has been largely reliant on imaging tools which do not capture molecular events that cause cancer progression. Gene-expression data-based analyses are able to identify these events, allowing RNA-sequence and microarray cancer data to be used for cancer analyses. Breast cancer is the most common cancer worldwide, and is classified into four stages - stages 1, 2, 3, and 4 [2]. While machine learning models have previously been explored to perform stage classification with limited success, multi-class stage classification has not had significant progress. There is a need for improved multi-class classification models, such as by investigating deep learning models. Gene-expression-based cancer data is characterised by the small size of available datasets, class imbalance, and high dimensionality. Class balancing methods must be applied to the dataset. Since all the genes are not necessary for stage prediction, retaining only the necessary genes can improve classification accuracy. The breast cancer samples are to be classified into 4 classes of stages 1 to 4. Invasive ductal carcinoma breast cancer samples are obtained from The Cancer Genome Atlas (TCGA) and Molecular Taxonomy of Breast Cancer International Consortium (METABRIC) datasets and combined. Two class balancing techniques are explored, synthetic minority oversampling technique (SMOTE) and SMOTE followed by random undersampling. A hybrid feature selection pipeline is proposed, with three pipelines explored involving combinations of filter and embedded feature selection methods: Pipeline 1 - minimum-redundancy maximum-relevancy (mRMR) and correlation feature selection (CFS), Pipeline 2 - mRMR, mutual information (MI) and CFS, and Pipeline 3 - mRMR and support vector machine-recursive feature elimination (SVM-RFE). The classification is done using deep learning models, namely deep neural network, convolutional neural network, recurrent neural network, a modified deep neural network, and an AutoKeras generated model. Classification performance post class-balancing and various feature selection techniques show marked improvement over classification prior to feature selection. The best multiclass classification was found to be by a deep neural network post SMOTE and random undersampling, and feature selection using mRMR and recursive feature elimination, with a Cohen-Kappa score of 0.303 and a classification accuracy of 53.1%. For binary classification into early and late-stage cancer, the best performance is obtained by a modified deep neural network (DNN) post SMOTE and random undersampling, and feature selection using mRMR and recursive feature elimination, with an accuracy of 81.0% and a Cohen-Kappa score (CKS) of 0.280. This pipeline also showed improved multiclass classification performance on neuroblastoma cancer data, with a best area under the receiver operating characteristic (auROC) curve score of 0.872, as compared to 0.71 obtained in previous work, an improvement of 22.81%. The results and analysis reveal that feature selection techniques play a vital role in gene-expression data-based classification, and the proposed hybrid feature selection pipeline improves classification performance. Multi-class classification is possible using deep learning models, though further improvement particularly in late-stage classification is necessary and should be explored further.

Recently, multilabel classification algorithms play an increasingly significant role in data mining and machine learning. However, some existing mutual information-based algorithms ignore the influence of the proportions of labels on the correlation degree between features and label sets. Besides, the correlation degree of label sets cannot be accurately measured in most traditional ReliefF algorithms, and the repeated calculation arises from the division of heterogeneous neighbors. To overcome these shortcomings, this paper proposes a multilabel feature selection method using mutual information and improved multilabel ReliefF (ML-ReliefF). First, the proportion of each label is calculated in label space and combined with the mutual information of features and labels to construct a novel correlation degree between features and label sets to preprocess multilabel datasets, which is used to reduce runtime of ML-ReliefF. Second, the mutual information of label sets is introduced into improving accuracy of the correlation degree among label sets. Furthermore, two types of correlation degree for label sets based on ML-ReliefF are developed to divide similar and heterogeneous samples more clearly. Third, a divided method of heterogeneous neighbors is presented to effectively avoid the repeated calculation in ML-ReliefF, and a novel method of feature weighting based on ML-ReliefF is constructed to evaluate the importance of features. Finally, a multilabel feature selection algorithm based on mutual information and ML-ReliefF for multilabel classification is designed to improve the performance of multilabel classification. Experiments under fourteen multilabel datasets show the effectiveness of our algorithm and improve the classification performance for multilabel datasets.

A novel approach to improving C-Tree for feature selection

Low-rank learning for feature selection in multi-label classification

In classification, feature selection (FS) removes irrelevant and redundant features to improve the classification performance. Most of the existing FS approaches are designed specifically for single-label classification, wherein each instance is associated to a single class label. However, many real-world problems have more than one class label for each instance, also known as multi-label classification (MLC). Label-to-label interactions in MLC information presents a new challenge for FS, whereas single-label FS methods typically only consider feature-to-feature (redundancy) and feature-to-label (relevancy) information. Sparsity-based FS is an efficient and effective method for MLC which has demonstrated its ability in considering label-to-label interactions. However, existing sparsity-based approaches can be trapped at local optima due to their gradient-based search. In addition, they cannot consider different trade-off between the two objectives of FS: to minimise the number of features and to maximise the classification performance. Evolutionary multi-objective optimisation (EMO) can address the above issues, but EMO has not been applied to achieve sparsity-based FS for MLC. This paper proposes a novel sparsity-based representation, objective function, and local search method to achieve sparsity-based EMO FS for MLC. The experimental results indicate improved efficiency and effectiveness in the MLC performance in comparison to existing state-of-the-art multi-label feature selection methods.

Software defect prediction models are used for predicting high risk software components. Feature selection has significant impact on the prediction performance of the software defect prediction models since redundant and unimportant features make the prediction model more difficult to learn. Ensemble feature selection has recently emerged as a new methodology for enhancing feature selection performance. This paper proposes a new multi-criteria-decision-making (MCDM) based ensemble feature selection (EFS) method. This new method is termed as MCDM-EFS. The proposed method, MCDM-EFS, first generates the decision matrix signifying the feature’s importance score with respect to various existing feature selection methods. Next, the decision matrix is used as the input to well-known MCDM method TOPSIS for assigning a final rank to each feature. The proposed approach is validated by an experimental study for predicting software defects using two classifiers K-nearest neighbor (KNN) and naïve bayes (NB) over five open-source datasets. The predictive performance of the proposed approach is compared with existing feature selection algorithms. Two evaluation metrics – nMCC and G-measure are used to compare predictive performance. The experimental results show that the MCDM-EFS significantly improves the predictive performance of software defect prediction models against other feature selection methods in terms of nMCC as well as G-measure.

In recent years, multi-label study has received extensive attention and research in many fields. The feature dimensions of a multi-label data set are high but contain a large amount of noise as well as irrelevant and redundant features. This not only leads to huge storage and time overhead, but also brings serious dimensional disaster problems, making multi-label learning tasks very difficult. Therefore, how to effectively select multi-label features is an important research content in multi-label learning. However, most of the current methods are converted from the methods of single-label feature selection, and feature selection is easy to fall into the local optimal heuristic search strategy. Time complexity has always been the biggest problem of such methods. Based on those considerations, our paper proposes a fast and effective multi-label feature selection method, which uses the optimization strategy to replace the previous search strategy for multi-label feature selection, and transforms the search problem into convex optimization problem. Therefore, the time performance of the traditional method is improved by two to three orders of magnitude. Finally, the experimental results of five evaluation indicators on the four data sets show that our method is superior to many popular methods in feature selection field.

Accurately classifying medical data is critical for improving diagnostic prediction system and identifying threptic targets for treatments. Analysing gene expression data has a major challenge in extracting disease-related genes from the large number of genes output from next generation sequencing technology. Therefore, eliminating irrelevant and redundant genes is a major step to process data for prediction. Our objective is to predict more accurately the presence of cancer disease in a sample cell from the gene expression.In this paper, we create a function called Classification Technique as Feature Selection (CTFS) as a new feature selection (FS) method to extract a subset (small number) of genes from classified big number of genes expression to improve cancer prediction result. The enrolled classification techniques in CTFS function for selection are K-Nearest Neighbors (K-NN) and Extreme Gradient Boosting (XGBoosting) optimized by Bayesian Parameter Tuning (BPT). The feature selection methods used to investigate the performance of CTFS function are Univariate Feature Selection (UFS) and Feature Importance (FI). The classification stage is carried out after the feature selection stage using three machine learning (ML) algorithms, Naïve Bayes (NB), Linear Support Vector Machine (LSVM), and Random Forest (RF). Results shows that, using XGBoosting optimized by BPT for FS outperforms FI method in terms of increasing the prediction accuracies along with minimum number of features but with higher running time. The performance of K-NN in FS outperforms all other FS methods in terms of accuracies providing an accuracy that is up to 100% when applied with LSVM on simulation dataset.