Two-Step Feature Selection Methods for Selection of Very Few Features

Target spectrum based feature selection (TSFS): A new method based on chain coding for target detection problems

A simple and efficient filter feature selection method via document-term matrix unitization

Software defect prediction models are used for predicting high risk software components. Feature selection has significant impact on the prediction performance of the software defect prediction models since redundant and unimportant features make the prediction model more difficult to learn. Ensemble feature selection has recently emerged as a new methodology for enhancing feature selection performance. This paper proposes a new multi-criteria-decision-making (MCDM) based ensemble feature selection (EFS) method. This new method is termed as MCDM-EFS. The proposed method, MCDM-EFS, first generates the decision matrix signifying the feature’s importance score with respect to various existing feature selection methods. Next, the decision matrix is used as the input to well-known MCDM method TOPSIS for assigning a final rank to each feature. The proposed approach is validated by an experimental study for predicting software defects using two classifiers K-nearest neighbor (KNN) and naïve bayes (NB) over five open-source datasets. The predictive performance of the proposed approach is compared with existing feature selection algorithms. Two evaluation metrics – nMCC and G-measure are used to compare predictive performance. The experimental results show that the MCDM-EFS significantly improves the predictive performance of software defect prediction models against other feature selection methods in terms of nMCC as well as G-measure.

With the rapid development of artificial intelligence in recent years, the research on image processing, text mining, and genome informatics has gradually deepened, and the mining of large-scale databases has begun to receive more and more attention. The objects of data mining have also become more complex, and the data dimensions of mining objects have become higher and higher. Compared with the ultra-high data dimensions, the number of samples available for analysis is too small, resulting in the production of high-dimensional small sample data. High-dimensional small sample data will bring serious dimensional disasters to the mining process. Through feature selection, redundancy and noise features in high-dimensional small sample data can be effectively eliminated, avoiding dimensional disasters and improving the actual efficiency of mining algorithms. However, the existing feature selection methods emphasize the classification or clustering performance of the feature selection results and ignore the stability of the feature selection results, which will lead to unstable feature selection results, and it is difficult to obtain real and understandable features. Based on the traditional feature selection method, this paper proposes an ensemble feature selection method, Random Bits Forest Recursive Clustering Eliminate (RBF-RCE) feature selection method, combined with multiple sets of basic classifiers to carry out parallel learning and screen out the best feature classification results, optimizes the classification performance of traditional feature selection methods, and can also improve the stability of feature selection. Then, this paper analyzes the reasons for the instability of feature selection and introduces a feature selection stability measurement method, the Intersection Measurement (IM), to evaluate whether the feature selection process is stable. The effectiveness of the proposed method is verified by experiments on several groups of high-dimensional small sample data sets.

Advances in functional magnetic resonance imaging (fMRI) have led to the ability to study the brain across many contexts. However, the large number of features generated by functional connectivity approaches may overfit the data. These problems can be overcome with either feature selection (FS) or dimensionality reduction (DR), which can be applied to less complex models. We utilize two open source datasets to compare the performance of DR/FS methods on cognitive task decoding using a suite of ML classifiers. While FS and DR methods have been used previously in decoding research, no systematic comparison of their performance has been undertaken. Here, we compare available methods using commonly utilized machine learning libraries to establish which methods provide the best predictive performance. We then conduct statistical tests to examine the relative contributions of DR and FS methods and classifiers on decoding accuracy. Neither DR or FS was found to be superior. However, differences were identified across datasets and tasks. In the majority of methods and datasets, a peak in predictive performance was found using a small percentage (005-.10%) of the total number of original features. Some methods perform better than the baseline method of prediction with all available features or selecting features randomly. Decoding performance with some datasets with some method exceeds that of deep learning approaches. These results suggest a "sweet spot" for the tradeoff between the retention of features and predictive accuracy.

<p>The feature selection method enhances machine learning performance by enhancing learning precision. Determining the optimal feature selection method for a given machine learning task involving big-dimension data is crucial. Therefore, the purpose of this study is to make a comparison of feature selection methods highlighting several filters (information gain, chi-square, ReliefF) and embedded (Lasso, Ridge) hybrid with logistic regression (LR). A sample size of n=100, 75 is chosen randomly, and the reduction features d=50, 22, and 10 are applied. The procedure for feature reduction makes use of the entire sample sizes. Each sample size's results are compared, including tests with no feature selection process. The results indicate that LR+ReliefF is the best method for mammary cancer data, whereas LR+IG is the best for prostatic cancer data, making the filter more suitable than embedded for big-dimension data. This study revealed that the sample's features and size influence the most effective method for selecting features from big-dimension data. Therefore, it provides insight into the most effective methods for particular features and sample sizes in high-dimensional data.</p>

Objective. The identification of functional regions, in particular the subthalamic nucleus, through microelectrode recording (MER) is the key step in deep brain stimulation (DBS). To eliminate variability in a neurosurgeon’s judgment, this study presents an online identification method for identifying functional regions along the electrode trajectory. Approach. Functional regions can be identified through offline clustering and online identification based on the unsupervised random forest (RF) algorithm. We took 106 features from MER and the estimated anatomical distance to target as the dataset to train the RF model. To improve the prediction performance and reduce the computation time, a wrapper feature selection (FS) method was added into the algorithm. The method contains feature ranking based on out-of-bag error or silhouette index and feature subset search based on the roulette selection algorithm. Main results. To evaluate the optimization effect of the FS method on the unsupervised RF algorithm, we compared the results of the algorithm with or without FS on the DBS dataset. In addition, the optimization effect of FS on the computation time is evaluated. The results show that for offline clustering, the accuracy obtained with the selected features is higher than that obtained with all features, and the running time decreased from 259.7 s to 60.8 s in the iteration of the FS. The accuracy in online identification improved from 76.19% to 92.08% through FS. In addition, the functional region online identification time is 41.5 ms, which can meet the requirements of DBS surgery. Significance. In conclusion, using the FS method can improve the accuracy and reduce the computation time of the online identification of functional regions. In addition, the online identification method can provide valuable assistance for both neurosurgeons and stereotactic surgery robots in guiding implantation of the electrode in real time.

A critical problem in radiomic studies is the high dimensionality of the datasets, which stems from small sample sizes and many generic features extracted from the volume of interest. Therefore, feature selection methods are used, which aim to remove redundant as well as irrelevant features. Because there are many feature selection algorithms, it is key to understand their performance in the context of radiomics. A total of 29 feature selection algorithms and 10 classifiers were evaluated on 10 publicly available radiomic datasets. Feature selection methods were compared for training times, for the stability of the selected features, and for ranking, which measures the pairwise similarity of the methods. In addition, the predictive performance of the algorithms was measured by utilizing the area under the receiver operating characteristic curve of the best-performing classifier. Feature selections differed largely in training times as well as stability and similarity. No single method was able to outperform another one consistently in predictive performance. Our results indicated that simpler methods are more stable than complex ones and do not perform worse in terms of area under the receiver operating characteristic curve. Analysis of variance, least absolute shrinkage and selection operator, and minimum redundancy, maximum relevance ensemble appear to be good choices for radiomic studies in terms of predictive performance, as they outperformed most other feature selection methods.

Background: Feature selection techniques are important factors for improving machine learning models because they increase prediction accuracy and decrease the time to create a model. Recently, feature selection techniques have been employed on software quality prediction problems with different results and no clear indication of which techniques are frequently used.Objective: This study aims to conduct a systematic review of the application of feature selection techniques in software quality prediction and answers eight research questions.Method: The review evaluates 15 papers in 9 journals and 6 conference proceedings from 2007 to 2017 using the standard systematic literature review method.Results: The results obtained from this study reveal that the filter feature selection method was the most commonly used in the studies (60%) and RELIEF was the most employed among this method, and a limited number of studies employed an ensemble method. Several studies used public datasets available in the PROMISE software project repository (60%). Most studies focused on software defect prediction (classification problem) using area under curve (AUC) as a primary evaluation measure, whereas only two studies focused on software maintainability prediction (regression problem) using mean magnitude of relative error (MMRE) as a primary evaluation measure. All selected studies performed k-fold cross-validation to evaluate model accuracy. Individual prediction models were mostly employed and ensemble models appeared only in three studies. Naive Bayes was the most investigated among individual models, whereas Random forest was the most investigated among ensemble models.Conclusion: Feature selection techniques used by selected primary studies have a positive impact on the performance of the prediction models. Further, both ensemble feature selection method and ensemble models have the ability for increasing prediction accuracy over single methods or individual models and have reported improvement in the prediction accuracy; however, the application of these techniques in software quality prediction is still limited.

On the benefit of feature selection and ensemble feature selection for fuzzy k-nearest neighbor classification

QSAR analysis of diaryl COX-2 inhibitors: Comparison of feature selection and train-test data selection methods

• Neural Network feature selection methodology for both numerical and categorical data. • Embedded feature selection provides model and feature subset in a single training. • Novel feature embedding for dealing with high-cardinality categorical features. • Validation of the methodology on several open and real industry dataset. • Comparison with several state-of-the-art feature selection methodologie. In an era of effortless data collection, the impact of machine learning — especially neural networks (NNs) — is undeniable. As datasets grow in size and complexity, efficiently handling mixed data types, including categorical and numerical features, becomes critical. Feature encoding and selection play a key role in improving NN performance, efficiency, interpretability, and generalisation. This paper presents GLEm-Net (Grouped Lasso with Embeddings Network), a novel NN-based approach that seamlessly integrates feature encoding and selection directly into the training process. GLEm-Net uses embedding layers to process categorical features with high cardinality, simplifying the model and improving generalisation. By extending the grouped Lasso regularisation to explicitly consider categorical features, GLEm-Net automatically identifies the most relevant features during training and returns them to the analyst. We evaluate GLEm-Net on open and proprietary industry datasets and compare it to state-of-the-art feature selection methodologies. Results show that GLEm-Net adapts to each dataset by allowing the NN to directly select subsets of most important features, offering on par performance with the best state-of-the-art feature selection methods, while eliminating the need for the external feature encoding and selection steps that are now incorporated in the NN training stage.

Accurately classifying medical data is critical for improving diagnostic prediction system and identifying threptic targets for treatments. Analysing gene expression data has a major challenge in extracting disease-related genes from the large number of genes output from next generation sequencing technology. Therefore, eliminating irrelevant and redundant genes is a major step to process data for prediction. Our objective is to predict more accurately the presence of cancer disease in a sample cell from the gene expression.In this paper, we create a function called Classification Technique as Feature Selection (CTFS) as a new feature selection (FS) method to extract a subset (small number) of genes from classified big number of genes expression to improve cancer prediction result. The enrolled classification techniques in CTFS function for selection are K-Nearest Neighbors (K-NN) and Extreme Gradient Boosting (XGBoosting) optimized by Bayesian Parameter Tuning (BPT). The feature selection methods used to investigate the performance of CTFS function are Univariate Feature Selection (UFS) and Feature Importance (FI). The classification stage is carried out after the feature selection stage using three machine learning (ML) algorithms, Naïve Bayes (NB), Linear Support Vector Machine (LSVM), and Random Forest (RF). Results shows that, using XGBoosting optimized by BPT for FS outperforms FI method in terms of increasing the prediction accuracies along with minimum number of features but with higher running time. The performance of K-NN in FS outperforms all other FS methods in terms of accuracies providing an accuracy that is up to 100% when applied with LSVM on simulation dataset.

The paper proposes a hybrid approach to reduce dimension in text classification problems, to overcome the issue of Curse of Dimensionality. This hybrid approach is a combination of Feature Selection (FS) and Feature Extraction (FE) methods, considering different aspects of feature relevance, to effectively reduce the dimension in large text datasets. It prevents feature selection biased in favor of a particular FS method. Many FS methods like Term Variance, Document Frequency, Information Gain, Shannons Entropy measure, Mean-Median and Mean Absolute Difference, were implemented and a comparative study was made on their performance when implemented in a hybrid system. The features selected by the individual FS methods are merged using three approaches, namely, Union, Intersection and Modified Union. The sub lists of features further undergo Feature Extraction by PCA, and the reduced feature sub list is clustered with k-means. Finally, the sentiment-score of the individual clusters are calculated using SentiWordNet database which gives the polarity of the data. The experiments were conducted on the benchmark datasets namely Reuters-21,578 and Classic4. The performance evaluation of the system made using the measures like precision, recall, f-score and accuracy shows that the proposed method has improved performance compared to its competitive methods.

In many real-world unsupervised learning applications, given data with balanced distribution, that is, there are an approximately equal number of instances in each class, we often need to construct a model to reveal such balance. However, in many data, especially the high-dimensional ones, the data in the original feature space often do not present such balance due to the redundant and noisy features. To tackle this problem, we apply an unsupervised spectral feature selection method to select some informative features, which can better reveal the balanced structure of data. Although spectral feature selection is one of the most popular unsupervised feature selection methods and has been widely studied, none of the existing spectral feature selection methods consider the balance property of data. To address this issue, in this article, we propose a novel balanced spectral feature selection (BSFS) method, which not only selects the discriminative features but also picks those to reveal the balanced structure of data. To the best of our knowledge, this is the first spectral feature selection method considering balance structure of data. By introducing a balanced regularization term, we integrate the balanced spectral clustering and feature selection into a unified framework seamlessly. At last, the experiments on benchmark datasets show that the proposed one outperforms the conventional feature selection methods in both clustering performance and balance, which demonstrates the effectiveness and efficiency of the proposed method.