Regularized sparse feature selection with constraints embedded in graph Laplacian matrix

Sparse feature selection: Relevance, redundancy and locality structure preserving guided by pairwise constraints

Feature selection aims to select the most relevant subset of features to make classifiers more accurate, fast, and easy to understand. Regularized sparse feature selection methods using L 1 -norm regularization term in their optimization problem, have received much attention in recent years. In this paper, besides the L 1 -norm regularization term a locality structure preserving regularization term based on positive data points is proposed. Positive data is defined as the unique data points involved in the must-link constraints, which are the most informative data. The experimental results on a number of datasets, show the efficacy of the proposed method for classification tasks compared with similar feature selection methods.

With the advent of the era of big data, data has become bigger than ever. Recently, as a fundamental task of pattern recognition, predict and data mining, feature selection has aroused wide public concern. However, extant methods on feature selection have an $O(\left|C\right|^x\left|U\right|^y)$ time complexity, which is the bottleneck preventing people from exploring knowledge in large-scale or high-dimensional datasets. Based on bijective soft sets, we propose a new rationale for feature selection, which can help break that bottleneck. Subsequently, this paper proposes an $O(\left|U\right|)$ feature-selection method whose computational time increases linearly only with the number of instances. To validate the proposed method, we conduct extensive experiments on the University of California Irvine (UCI) datasets in which large-scale and high-dimensional datasets containing four million instances and over three million features are included. The results reveal that the proposed method is an efficient, effective, and outperforms traditional methods in runtime, which can save massive computing resources. Moreover, the proposed method can be applied to feature selection for large-scale and gigantic-dimensional datasets, which are difficult to process with traditional methods.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

Feature selection (FS) using meta-heuristic algorithms on high-dimensional datasets (HDD) is becoming more prevalent due to the continuous advancement in data mining. However, the difficulty in identifying the threshold of features in a dataset to be categorised as HDD remains an issue due to the different schools of thought on this matter. Therefore, this survey intended to determine the threshold for a number of features to be HDD, and subsequently identify the trend or potential FS method for HDD and the most preferred meta-heuristic algorithms and classifiers for both wrapper-based and filter-based FS methods to analyse HDD. This study performed an extensive systematic literature review by implementing the PRISMA guidelines on 62 research articles that were published between 2016 to 2021. This survey proposed a novel grouping technique called literal grouping and data grouping (LGDG) to accurately group the chosen articles based on HDD. The LGDG method serves as a guide for other researchers who intend to perform FS research related to HDD. Literal grouping refers to searching for selected papers using specific keywords, like HDD in this case. While data grouping compares the number of features in datasets towards the threshold, which is set at 2,000 features by the majority. Based on the analyses of all the LGDG groupings, the filter-based FS method gained more attention in recent years with competent results no less than wrapper-based, especially on HDD. Besides that, Moth Flame Optimisation works well in filter-based methods, whereas Cuckoo Optimisation Algorithm works well in wrapper-based, while Whale Optimisation Algorithm works well in both FS methods. As for the classifier’s preferences, SVM, DT, and NB are preferred by the filter-based, while KNN is preferred by the wrapper-based method. It can be recommended that reviewing other aspects such as multi-objective FS on HDD and including more FS methods could be included in future studies as an extension to this survey.

Enhanced multi-label feature selection considering label-specific relevant information

High-dimensional datasets with low sample sizes (HDLSS) are pivotal in the fields of biology and bioinformatics. One of core objective of HDLSS is to select most informative features and discarding redundant or irrelevant features. This is particularly crucial in bioinformatics, where accurate feature (gene) selection can lead to breakthroughs in drug development and provide insights into disease diagnostics. Despite its importance, identifying optimal features is still a significant challenge in HDLSS. To address this challenge, we propose an effective feature selection method that combines gradual permutation filtering with a heuristic tribrid search strategy, specifically tailored for HDLSS contexts. The proposed method considers inter-feature interactions and leverages feature rankings during the search process. In addition, a new performance metric for the HDLSS that evaluates both the number and quality of selected features is suggested. Through the comparison of the benchmark dataset with existing methods, the proposed method reduced the average number of selected features from 37.8 to 5.5 and improved the performance of the prediction model, based on the selected features, from 0.855 to 0.927. The proposed method effectively selects a small number of important features and achieves high prediction performance.

Occam's razor in dimension reduction: Using reduced row Echelon form for finding linear independent features in high dimensional microarray datasets

High-dimensional datasets can lead to overfitting and computationally expensive model building on machine learning. This study uses a dimensionality reduction technique, namely feature selection techniques, to overcome these problems. Five feature selection methods were used, i.e., Chi-Square (CS), Information Gain (IG), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and Least Absolute Shrinkage and Selection Operator (LASSO), and three classifier methods viz. Naïve Bayes, Extreme Gradient Boosting (XGB), and RF Classifier. The dataset used is the Heart Attack Analysis & Prediction Dataset. In this study, three scenarios of the best feature selection were carried out, namely: 1. selection of the best feature using a specific feature selection, 2. the intersection of selection of the best feature from the same category, 3. the intersection of selection of the best feature from the five proposed feature selection methods. The performance model is measured using accuracy, precision, recall, f1-score, AUC, and training time. This study reveals that feature selection is very effective in improving the performance of prediction models. Based on the experiment results, the best feature selection is CS and IG in the Filter Category with the XGB model. The best feature selected improved the performance of accuracy, precision, recall, f1-score, and AUC, i.e., 1.7%, 1%, 2.3%, 1.6%, and 0.2%, respectively. Meanwhile, training time requirements decreased by 23.5%. Feature selection with specific techniques performs better than feature selection by selecting the best features from the same category feature selection technique or various other feature selection methods.

Principal Component Analysis (PCA) is the most widely used unsupervised subspace learning method, and lots of its variants have been developed. With so many proposed PCA-like methods, it is still not clear that which features are better or worse for principal components, especially when the data suffers from outliers. To this end, we propose Robust Principal Component Analysis via joint l 2,1 -norms minimization, which provides new insights into two crucial issues of PCA: feature selection and robustness to outliers. Unlike other PCA-like methods, the proposed method is able to select effective features for reconstruction by using the l 2,1 -norm regularization term. More specific, we first use a l 2,1 -norm based transformation matrix to select effective features that can effectively characterize key components (e.g., the eyes and the nose in a face image), and then use an orthogonal transformation matrix to recover the original data from the selected data representation. In this way, the key components can be well recovered by using the effective features selected by a learned transformation matrix. On the other hand, we also impose l 2,1 -norm on a loss term to select clean samples to recover its same class samples but with outliers. A simple yet effective optimization algorithm is proposed to solve the resulting optimization problem. Experiments on six datasets demonstrate the effectiveness of the proposed method.

Software defect prediction models are used for predicting high risk software components. Feature selection has significant impact on the prediction performance of the software defect prediction models since redundant and unimportant features make the prediction model more difficult to learn. Ensemble feature selection has recently emerged as a new methodology for enhancing feature selection performance. This paper proposes a new multi-criteria-decision-making (MCDM) based ensemble feature selection (EFS) method. This new method is termed as MCDM-EFS. The proposed method, MCDM-EFS, first generates the decision matrix signifying the feature’s importance score with respect to various existing feature selection methods. Next, the decision matrix is used as the input to well-known MCDM method TOPSIS for assigning a final rank to each feature. The proposed approach is validated by an experimental study for predicting software defects using two classifiers K-nearest neighbor (KNN) and naïve bayes (NB) over five open-source datasets. The predictive performance of the proposed approach is compared with existing feature selection algorithms. Two evaluation metrics – nMCC and G-measure are used to compare predictive performance. The experimental results show that the MCDM-EFS significantly improves the predictive performance of software defect prediction models against other feature selection methods in terms of nMCC as well as G-measure.

Diabetes is a disease that occurs due to an increase in blood sugar levels. Increased sugar levels can trigger death because they can cause damage to blood vessels, nerves, and other internal structures. To avoid bad consequences, it is important to predict whether a person has diabetes or not based on the information that each patient has. The method that can be used to predict whether someone has diabetes or not is a classification method. The C4.5 algorithm is one of the classification algorithms that can be used to predict diabetics. Although the C4.5 algorithm can be used to predict, getting a high accuracy value is also an important measurement. One thing that can be done to increase the accuracy value is to select the attribute that most influences a person to have diabetes or not. Feature selection greatly affects the accuracy of the C4.5 algorithm, particle swarm optimization is used as a feature selection method in this study. The purpose of this study is to increase the value of accuracy to predict whether someone has diabetes or not through feature selection using particle swarm optimization. The dataset used is the Pima Indian Diabetes Databases (PIDD) from the University of California Irvine (UCI) Machine Learning Repository. The results of the study show that feature selection using particle swarm optimization before using the C4.5 algorithm can increase accuracy by 6% from the previous accuracy value of 75%. The selected features are 4 attributes out of 9 attributes. The four attributes are Glucose, SkinThickness, BMI, and DiabetesPedigreeFunction. The results of feature selection using particle swarm optimization can increase the accuracy of predictions of whether someone suffering from diabetes or not.

Manifold learning and feature selection have been widely studied in face recognition in the past two decades. This paper focuses on making use of the manifold structure of datasets for feature extraction and selection. We propose a novel method called Joint Sparse Locality Preserving Projections (JSLPP). In order to preserve the manifold structure of datasets, we first propose a manifold-based regression model by using a nearest-neighbor graph, then the \( L_{2,1} \)-norm regularization term is imposed on the model to perform feature selection. At last, an efficient iterative algorithm is designed to solve the sparse regression model. The convergence analysis and computational complexity analysis of the algorithm are presented. Experimental results on two face datasets indicate that JSLPP outperforms six classical and state-of-the-art dimensionality reduction algorithms.

Feature Selection for financial data – comparison

An effective bearing fault diagnosis model based on machine learning is proposed in this study. The model can separate into three stages: feature extraction, feature selection, and classification. In the stage of feature extraction, multiresolution analysis (MRA) and fast Fourier transform (FFT) are applied to extract the features from the raw signal measured from the rotating machine. In the second stage, a powerful feature selection method is proposed and utilized in the stage of feature selection. The new feature selection method is based on grey wolf optimization (GWO) and heap-based optimizer (HBO) with new strategies combined. Finally, support vector machine (SVM) and linear discriminant analysis (LDA) are used as the classifier independently. To verify the capability of the proposed model, four different datasets are applied to test the model in this study, respectively University of California Irvine (UCI) benchmark dataset, bearing dataset, Case Western Reserve University (CWRU) benchmark dataset, and Machinery Failure Prevention Technology (MFPT) benchmark dataset. The proposed method is compared with the existing methods and can certify the robustness with the experiment results.