Probabilistic Feature Selection in Machine Learning

To develop a hypothesis-free model that best predicts response to MTX drug in RA patients utilizing biologically meaningful genetic feature selection of potentially functional single nucleotide polymorphisms (pfSNPs) through robust machine learning (ML) feature selection methods. MTX-treated RA patients with known response were divided in a 4:1 ratio into training and test sets. From the patients' exomes, potential features for classifier prediction were identified from pfSNPs and non-genetic factors through ML using recursive feature elimination with cross-validation incorporating the random forest classifier. Feature selection was repeated on random subsets of the training cohort, and consensus features were assembled into the final feature set. This feature set was evaluated for predictive potential using six ML classifiers, first by cross-validation within the training set, and finally by analysing its performance with the unseen test set. The final feature set contains 56 pfSNPs and five non-genetic factors. The majority of these pfSNPs are located in pathways related to RA pathogenesis or MTX action and are predicted to modulate gene expression. When used for training in six ML classifiers, performance was good in both the training set (area under the curve: 0.855-0.916; sensitivity: 0.715-0.892; and specificity: 0.733-0.862) and the unseen test set (area under the curve: 0.751-0.826; sensitivity: 0.581-0.839; and specificity: 0.641-0.923). Sensitive and specific predictors of MTX response in RA patients were identified in this study through a novel strategy combining biologically meaningful and machine learning feature selection and training. These predictors may facilitate better treatment decision-making in RA management.

Image segmentation is a popular application area of machine learning. In this context, each target region drawn from an image is defined as a class towards recognition of instances that belong to this region (class). In order to train classifiers that recognize the target region to which an instance belongs, it is important to extract and select features relevant to the region. In traditional machine learning, all features extracted from different regions are simply used together to form a single feature set for training classifiers, and feature selection is usually designed to evaluate the capability of each feature or feature subset in discriminating one class from other classes. However, it is possible that some features are only relevant to one class but irrelevant to all the other classes. From this point of view, it is necessary to undertake feature selection for each specific class, i.e, a relevant feature subset is selected for each specific class. In this paper, we propose the so-called multi-task feature selection approach for identifying features relevant to each target region towards effective image segmentation. This way of feature selection requires to transform a multi-class classification task into $n$ binary classification tasks, where $n$ is the number of classes. In particular, the Prism algorithm is used to produce a set of rules for class specific feature selection and the K nearest neighbour algorithm is used for training a classifier on a feature subset selected for each class. The experimental results show that the multi-task feature selection approach leads to an significant improvement of classification performance comparing with traditional feature selection approaches.

Bi-stage feature selection for crop mapping using grey wolf metaheuristic optimization

Real-world objects and their features tend to exhibit multiple relationships rather than simple pairwise ones, and as a result basic graph representation can lead to substantial loss of information. Hypergraph representations, on the other hand, allow vertices to be multiply connected by hyperedges and can hence capture multiple or higher order relationships among features. Due to their effectiveness in representing multiple relationships, in this paper, we draw on recent work on hypergraph clustering to select the relevant feature subset (RFS) from a set of features using high-order (rather than pairwise) similarities. Specifically, we first devise a coupled feature representation to represent the data by utilizing self-coupled and inter-feature coupling relationships, which can be more effective to capture the intrinsic linear and nonlinear information on data structure. Regarding the new data representation, we use a new information theoretic criterion referred to as multivariate mutual information to measure the high-order feature combinations with respect to the class labels. Therefore, we construct a coupled feature hypergraph to model the high-order relations among features. Finally, we locate the relevant feature subset (RFS) from feature hypergraph by maximizing features’ average relevance, which has both low redundancy and strong discriminating power. The size of the relevant feature subset (RFS) is determined automatically. Experimental results demonstrate the effectiveness of our feature selection method on a number of standard data-sets.

Feature selection (FS) is a very important pre-processing technique in machine learning and data mining. It aims to select a small subset of relevant and informative features from the original feature space that may contain many irrelevant, redundant and noisy features. Feature selection usually leads to better performance, interpretability, and lower computational cost. In the literature, FS methods are categorized into three main approaches: Filters, Wrappers, and Embedded. In this paper we introduce a new feature selection method called graph feature selection (GFS). The main steps of GFS are the following: first, we create a weighted graph where each node corresponds to each feature and the weight between two nodes is computed using a matrix of individual and pairwise score of a Decision tree classifier. Second, at each iteration, we split the graph into two random partitions having the same number of nodes, then we keep moving the worst node from one partition to another until the global modularity is converged. Third, from the final best partition, we select the best ranked features according to a new proposed variable importance criterion. The results of GFS are compared to three well-known feature selection algorithms using nine benchmarking datasets. The proposed method shows its ability and effectiveness at identifying the most informative feature subset.

Privacy preserving in machine learning is a crucial issue in industry informatics since data used for training in industries usually contain sensitive information. Existing differentially private machine learning algorithms have not considered the impact of data correlation, which may lead to more privacy leakage than expected in industrial applications. For example, data collected for traffic monitoring may contain some correlated records due to temporal correlation or user correlation. To fill this gap, in this article, we propose a correlation reduction scheme with differentially private feature selection considering the issue of privacy loss when data have correlation in machine learning tasks. The proposed scheme involves five steps with the goal of managing the extent of data correlation, preserving the privacy, and supporting accuracy in the prediction results. In this way, the impact of data correlation is relieved with the proposed feature selection scheme, and moreover the privacy issue of data correlation in learning is guaranteed. The proposed method can be widely used in machine learning algorithms, which provide services in industrial areas. Experiments show that the proposed scheme can produce better prediction results with machine learning tasks and fewer mean square errors for data queries compared to existing schemes.

Feature selection in machine learning: A new perspective

In recent years, detecting credit card fraud transactions has been a difficult task due to the high dimensions and imbalanced datasets. Selecting a subset of important features from a high-dimensional dataset has proven to be the most prominent approach for solving high-dimensional dataset issues, and the selection of features is critical for improving classification performance, such as the fraud transaction identification process. To contribute to the field, this paper proposes a novel feature selection (FS) approach based on a metaheuristic algorithm called Rock Hyrax Swarm Optimization Feature Selection (RHSOFS), inspired by the actions of rock hyrax swarms in nature, and implements supervised machine learning techniques to improve credit card fraud transaction identification approaches. This approach is used to select a subset of optimal relevant features from a high-dimensional dataset. In a comparative efficiency analysis, RHSOFS is compared with Differential Evolutionary Feature Selection (DEFS), Genetic Algorithm Feature Selection (GAFS), Particle Swarm Optimization Feature Selection (PSOFS), and Ant Colony Optimization Feature Selection (ACOFS) in a comparative efficiency analysis. The proposed RHSOFS outperforms existing approaches, such as DEFS, GAFS, PSOFS, and ACOFS, according to the experimental results. Various statistical tests have been used to validate the statistical significance of the proposed model.

Network information security risks multiply and become more dangerous. Hackers today generally target end-to-end technology and take advantage of human weaknesses. Furthermore, hackers take advantage of technology weaknesses by applying various methods to attack. Nowadays, one of the greatest dangers to the modern digital world is malicious URLs, and stopping them is one of the biggest challenges in the field of cyber security. Detecting harmful URLs using machine learning and deep learning algorithms have been the subject of various academic papers. However, time and accuracy are the two biggest challenges of these tools. This paper proposes a multilayer perceptron (MLP) model that utilizes two significant aspects to make it more practical, lightweight, and fast: Using only lexical features and a decision tree (DT) algorithm to select the best relevant subset of features. The effectiveness of the experimental outcomes is evaluated in terms of time, accuracy, and error reduction. The results show that a MLP model using 35 features could achieve an accuracy of 94.51% utilizing only URL lexical features. Furthermore, the model is improved in time after applying the DT as feature selection with a slight improvement in accuracy and loss.

We propose a unifying FCA-based framework for some questions in data analysis and data mining, combining ideas from Rough Set Theory, JSM-reasoning, and feature selection in machine learning. Unlike the standard rough set model the indiscernibility relation in our paper is based on a quasi-order, not necessarily an equivalence relation. Feature selection, though algorithmically difficult in general, appears to be easier in many cases of scaled many-valued contexts, because the difficulties can at least partially be projected to the scale contexts. We propose a heuristic algorithm for this.

Human Action Recognition (HAR) is a vital part of the healthcare sector. Medical practitioners are experiencing difficulty in recognizing human physiological conditions due to the enormous quantity of sensory stimulation. Machine learning techniques help to predict human physiological conditions, providing medical practitioners to work more efficiently. Feature selection is an essential part of discovering new knowledge in the majority of real-world problems when there are a lot of features. Feature selection is extremely useful because it speeds up decisions and enhances classification performance. The importance of feature selection in machine learning is dimension reduction in a massive multivariate data collection. This paper presents an effective feature selection method known as the Enhanced Recursive Feature Elimination (EFRE) for selecting key features from the data set for HAR prediction. The experimental results reveal that the ERFE technique selects the most suitable features for HAR prediction. The performance of the proposed ERFE approach is tested using different performance evaluation metrics. The performance analysis shows that the ERFE method outperforms existing feature selection methods with 88% accuracy.

Feature selection entails identifying a subset of the most useful features that produces compatible results as the original entire set of features. A feature selection algorithm can be assessed in terms of both efficiency and effectiveness. While efficiency is concerned with the time required to find a subset of features, effectiveness is concerned with the quality of the subset of features. This paper proposes and experimentally evaluates a fast clustering-based feature selection algorithm, FAST, based on these criteria.  The FAST algorithm operates in two steps. Graph-theoretic clustering methods are used to partition characteristics into clusters in the initial stage. The most representative feature from each cluster that is strongly related to target classes is chosen in the second stage to construct a subset of features. Because the properties in various clusters are relatively independent, FAST's clustering-based technique is likely to produce a subset of valuable and independent features. We use the efficient Minimum-spanning tree clustering method to ensure FAST's efficiency. An empirical study is conducted to assess the efficiency and effectiveness of the FAST algorithm. FAST and several representative feature selection algorithms, such as FCBF, ReliefF, CFS, Consist, and FOCUS-SF, are compared to four types of well-known classifiers, including the probability-based Naive Bayes, the tree-based C4.5, the instance-based IB1, and the rule-based RIPPER, before and after feature selection. FAST not only provides smaller subsets of features but also improves the performances of the four types of classifiers, according to the findings, which were based on 35 publicly accessible real-world high-dimensional image, microarray, and text data.

In many data analysis tasks, one is often confronted with very high dimensional data. Feature selection techniques are designed to find the relevant feature subset of the original features which can facilitate clustering, classification and retrieval. In this paper, we consider the feature selection problem in unsupervised learning scenario, which is particularly difficult due to the absence of class labels that would guide the search for relevant information. The feature selection problem is essentially a combinatorial optimization problem which is computationally expensive. Traditional unsupervised feature selection methods address this issue by selecting the top ranked features based on certain scores computed independently for each feature. These approaches neglect the possible correlation between different features and thus can not produce an optimal feature subset. Inspired from the recent developments on manifold learning and L1-regularized models for subset selection, we propose in this paper a new approach, called Multi-Cluster Feature Selection (MCFS), for unsupervised feature selection. Specifically, we select those features such that the multi-cluster structure of the data can be best preserved. The corresponding optimization problem can be efficiently solved since it only involves a sparse eigen-problem and a L1-regularized least squares problem. Extensive experimental results over various real-life data sets have demonstrated the superiority of the proposed algorithm.

There are two fundamentally different approaches for feature selection: wrapper and filter. It is also possible to combine them, obtaining hybrid approaches. This paper describes a hybrid method for selecting relevant features in clustering problems. The proposed approach is based on the combination of the widely known k-means algorithm and a Bayesian filter, which is based on the Markov Blanket concept. Since the number of clusters and the subset of relevant features are usually inter-related, we propose a method that iterates between clustering (assuming that the number of clusters is not known a priori) and filtering. Experiments in a number of datasets show that the proposed approach allows selecting features that provide good partitions.

Fault diagnosis methods based on machine learning have made great progress for rotating machinery. The main steps of the machine learning process involve feature extraction, selection, and classification. Feature selection improves classification accuracy and reduces diagnosis time by selecting the better features. Due to the difficulty of traditional feature selection methods to rank the feature importance of each class, the best subset of features could hardly be obtained. Therefore, this article proposes a new feature selection method to address the shortcomings of the above traditional methods, called Feature Ranking based on Optimal Class Distance Ratio (FROCDR), which can choose the optimal features between every two classes of samples to obtain feature ranking that is conducive to classification. In order to comprehensively extract the fault information in the signal, the multiscale analysis and the variational mode decomposition (VMD) method are applied to process the vibration signals under different scales and frequency bands, and the processed signals are visualized by symmetrized dot pattern (SDP). In addition, features are extracted from the obtained SDP images, and the proposed FROCDR method is used to select the best subset of features. The final diagnosis task is accomplished by a random forest (RF) classifier. Experimental cases of bearing and gear data show that the proposed method has higher diagnostic accuracy and stability.