LIA: A Label-Independent Algorithm for Feature Selection for Supervised Learning

Previous chapter Next chapter Full AccessProceedings Proceedings of the 2016 SIAM International Conference on Data Mining (SDM)Robust Unsupervised Feature Selection on Networked DataJundong Li, Xia Hu, Liang Wu, and Huan LiuJundong Li, Xia Hu, Liang Wu, and Huan Liupp.387 - 395Chapter DOI:https://doi.org/10.1137/1.9781611974348.44PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAboutAbstract Feature selection has shown its effectiveness to prepare high-dimensional data for many data mining and machine learning tasks. Traditional feature selection algorithms are mainly based on the assumption that data instances are independent and identically distributed. However, this assumption is invalid in networked data since instances are not only associated with high dimensional features but also inherently interconnected with each other. In addition, obtaining label information for networked data is time consuming and labor intensive. Without label information to direct feature selection, it is difficult to assess the feature relevance. In contrast to the scarce label information, link information in networks are abundant and could help select relevant features. However, most networked data has a lot of noisy links, resulting in the feature selection algorithms to be less effective. To address the above mentioned issues, we propose a robust unsupervised feature selection framework NetFS for networked data, which embeds the latent representation learning into feature selection. Therefore, content information is able to help mitigate the negative effects from noisy links in learning latent representations, while good latent representations in turn can contribute to extract more meaningful features. In other words, both phases could cooperate and boost each other. Experimental results on real-world datasets demonstrate the effectiveness of the proposed framework. Previous chapter Next chapter RelatedDetails Published:2016eISBN:978-1-61197-434-8 https://doi.org/10.1137/1.9781611974348Book Series Name:ProceedingsBook Code:PRDT16Book Pages:1-867

In a conventional supervised learning paradigm, each data instance is associated with one single class label. Multilabel learning differs in the way that data instances may belong to multiple concepts simultaneously, which naturally appear in a variety of high impact domains, ranging from bioinformatics and information retrieval to multimedia analysis. It targets leveraging the multiple label information of data instances to build a predictive learning model that can classify unlabeled instances into one or multiple predefined target classes. In multilabel learning, even though each instance is associated with a rich set of class labels, the label information could be noisy and incomplete as the labeling process is both time consuming and labor expensive, leading to potential missing annotations or even erroneous annotations. The existence of noisy and missing labels could negatively affect the performance of underlying learning algorithms. More often than not, multilabeled data often has noisy, irrelevant, and redundant features of high dimensionality. The existence of these uninformative features may also deteriorate the predictive power of the learning model due to the curse of dimensionality. Feature selection, as an effective dimensionality reduction technique, has shown to be powerful in preparing high-dimensional data for numerous data mining and machine-learning tasks. However, a vast majority of existing multilabel feature selection algorithms either boil down to solving multiple single-labeled feature selection problems or directly make use of the imperfect labels to guide the selection of representative features. As a result, they may not be able to obtain discriminative features shared across multiple labels. In this article, to bridge the gap between a rich source of multilabel information and its blemish in practical usage, we propose a novel noise-resilient multilabel informed feature selection framework (MIFS) by exploiting the correlations among different labels. In particular, to reduce the negative effects of imperfect label information in obtaining label correlations, we decompose the multilabel information of data instances into a low-dimensional space and then employ the reduced label representation to guide the feature selection phase via a joint sparse regression framework. Empirical studies on both synthetic and real-world datasets demonstrate the effectiveness and efficiency of the proposed MIFS framework.

Unsupervised feature selection via discrete spectral clustering and feature weights

Credit scoring is extensively used by credit industries and financial institutions for financial decision-making. It is a way to assess the risk associated with an applicant based on historical data. However, the historical data may have large number of redundant and noisy features which could affect performance of credit scoring models. Main focus of this paper is to develop a hybrid credit scoring model by combining the feature selection and multi-layer ensemble classifier framework to improve the prediction performance of credit scoring model. The proposed hybrid credit scoring model uses hybrid binary particle swarm optimization and gravitational search algorithm (BPSOGSA) for feature selection and multi-layer ensemble classifier framework with five heterogeneous classifiers. A novel V-shaped transfer function for BPSOGSA is also designed for effective feature selection, which is used to transform the continuous search space to binary search space. Also, a novel fitness function for BPSOGSA is proposed to calculate the fitness value for each search agent. Further, multi-layer ensemble classifier framework along with a novel aggregation function is designed based on generalized convex function. The proposed hybrid credit scoring model is validated using Australian, German-categorical, German-numerical and Japanese credit scoring datasets. The experimental results on all the datasets demonstrate that the proposed credit scoring model outperforms other methods such as random forest and ensemble frameworks, namely majority voting, layered majority voting, weighted voting and layered weighted voting in terms of accuracy, sensitivity, G-measure and ROC characteristics.

In the present state of affairs, the motive behind every educational organization is to uplift the academic achievement of students. Educational data mining (EDM) is an upward field of research, and it is very helpful for academic institutions to predict the academic performance of the students. Educational datasets are the basis of various predictive models. Quality of these models can be improved by using feature selection (FS). To get the required benefits from the available data, there must be some tools for analysis and prediction. In lieu of the above, machine learning/data mining are most suitable. In educational data mining, for better accuracy of prediction models’ and quality of various educational datasets, feature selection (FS) plays a vital role. Feature selection (FS) algorithms abolish inappropriate information from the repositories of educational background so that performance of classifier in terms of accuracy could be increased and the same could be used for better decision. In lieu of the above, a best feature selection algorithm must be selected. In this paper, two filter selection approaches namely correlation feature selection (CFS) and wrapper-based feature selection have been used to demonstrate the importance of selection of a feature subset for a classification problem. The present paper aims to find the detailed investigation of filter feature selection algorithms along with the classification algorithms on a given dataset. We found result with numerous numbers of features from various Feature selection algorithms and classifiers which will help the researcher to discover the most excellent mixture of filter feature selection algorithms and its associated classifiers. The result indicates that SMO and J48 have the highest accuracy measures with the correlation feature selection algorithms, while Naïve Bayes has the highest accuracy measures with the wrapper subset feature selection algorithms for predicting high, medium and low grade for the students.

Feature selection plays an important role in preprocessing in pattern recognition and data mining, especially in large scale image, digital text, and biological data. Specifically, class label information is unavailable for conducting the selection of a minimal feature subset in unsupervised learning, which is full of challenging and interesting problems. In this paper, we present an unsupervised REPresentative Feature Selection (REPFS) algorithm based on information entropy and relevance analysis. The proposed method seeks to find a high-quality feature subset through feature clustering without using any learning algorithms. More importantly, the features’ relevance will be computed based on an information metric of the relevance gain, which provides an information theoretical foundation for finding a minimum of the redundancy between features. Our results on nine benchmark data sets demonstrate that REPFS can significantly improve upon state-of-the-art unsupervised algorithms.

In this work we propose a novel, sound framework for evolutionary feature selection in unsupervised machine learning problems. We show that unsupervised feature selection is inherently multi-objective and behaves differently from supervised feature selection in that the number of features must be maximized instead of being minimized. Although this might sound surprising from a supervised learning point of view, we exemplify this relationship on the problem of data clustering and show that existing approaches do not pose the optimization problem in an appropriate way. Another important consequence of this paradigm change is a method which segments the Pareto sets produced by our approach. Inspecting only prototypical points from these segments drastically reduces the amount of work for selecting a final solution. We compare our methods against existing approaches on eight data sets.

Automatic clustering and feature selection using multi-objective crow search algorithm

Today, diagnosis of attention deficit hyperactivity disorder (ADHD) still primarily relies on a series of subjective evaluations that highly rely on a doctor’s experiences and intuitions from diagnostic interviews and observed behavior measures. An accurate and objective diagnosis of ADHD is still a challenge and leaves much to be desired. Many children and adults are inappropriately labeled with ADHD conditions, whereas many are left undiagnosed and untreated. Recent advances in neuroimaging studies have enabled us to search for both structural (e.g., cortical thickness, brain volume) and functional (functional connectivity) abnormalities that can potentially be used as new biomarkers of ADHD. However, structural and functional characteristics of neuroimaging data, especially magnetic resonance imaging (MRI), usually generate a large number of features. With a limited sample size, traditional machine learning techniques can be problematic to discover the true characteristic features of ADHD due to the significant issues of overfitting, computational burden, and interpretability of the model. There is an urgent need of efficient approaches to identify meaningful discriminative variables from a higher dimensional feature space when sample size is small compared with the number of features. To tackle this problem, this paper proposes a novel integrated feature ranking and selection framework that utilizes normalized brain cortical thickness features extracted from MRI data to discriminate ADHD subjects against healthy controls. The proposed framework combines information theoretic criteria and the least absolute shrinkage and selection operator (Lasso) method into a two-step feature selection process which is capable of selecting a sparse model while preserving the most informative features. The experimental results showed that the proposed framework generated the highest/comparable ADHD prediction accuracy compared with the state-of-the-art feature selection approaches with minimum number of features in the final model. The selected regions of interest in our model were consistent with recent brain–behavior studies of ADHD development, and thus confirmed the validity of the selected features by the proposed approach.

Technological advancements have generated huge amount of data in recent times. The objective is to analyze this data to get meaningful insights from them. One of the techniques is to perform clustering, an unsupervised learning technique. The performance of clustering is dependent on the size of data (number of features, number of records), a well-known fact. Feature selection/extraction algorithms not only eliminate irrelevant features from the data but also improve the performance of the clustering algorithm applied. Knowing your data sets (type, nature, and size), the appropriate algorithm to use and the appropriate evaluation metrics that will be insightful to improve the quality of clusters is important. No such correlation has been established as per author’s knowledge. In this paper, we report the results of various internal and external evaluation metrics on application of distinct unsupervised feature selection/extraction algorithms on a variety of data sets. The output of various algorithms indicates that retaining 100 features give best results. The experiments prove that Principal Component Analysis (PCA) algorithm is suitable for linear and low-dimensional data sets. Laplacian Score, Non-negative Discriminative Feature Selection (NDFS), and Spectral Feature Selection (SPEC) work well with high dimensional, binary data sets. Low variance algorithm and Unsupervised Discriminative Feature Selection (UDFS) do not perform well on any type of data sets.

The explosive usage of social media produces massive amount of unlabeled and high-dimensional data. Feature selection has been proven to be effective in dealing with high-dimensional data for efficient learning and data mining. Unsupervised feature selection remains a challenging task due to the absence of label information based on which feature relevance is often assessed. The unique characteristics of social media data further complicate the already challenging problem of unsupervised feature selection, e.g., social media data is inherently linked, which makes invalid the independent and identically distributed assumption, bringing about new challenges to unsupervised feature selection algorithms. In this paper, we investigate a novel problem of feature selection for social media data in an unsupervised scenario. In particular, we analyze the differences between social media data and traditional attribute-value data, investigate how the relations extracted from linked data can be exploited to help select relevant features, and propose a novel unsupervised feature selection framework, LUFS, for linked social media data. We systematically design and conduct systemic experiments to evaluate the proposed framework on data sets from real-world social media websites. The empirical study demonstrates the effectiveness and potential of our proposed framework.

Many feature selection algorithms have been proposed in the past focusing on improving classification accuracy. In this work, we point out the importance of stable feature selection for knowledge discovery from high-dimensional data, and identify two causes of instability of feature selection algorithms: selection of a minimum subset without redundant features and small sample size. We propose a general framework for stable feature selection which emphasizes both good generalization and stability of feature selection results. The framework identifies dense feature groups based on kernel density estimation and treats features in each dense group as a coherent entity for feature selection. An efficient algorithm DRAGS (Dense Relevant Attribute Group Selector) is developed under this framework. We also introduce a general measure for assessing the stability of feature selection algorithms. Our empirical study based on microarray data verifies that dense feature groups remain stable under random sample hold out, and the DRAGS algorithm is effective in identifying a set of feature groups which exhibit both high classification accuracy and stability.

Increasingly many applications of machine learning are encountering large data that were almost unimaginable just a few years ago, and hence, many of the current algorithms cannot handle, i.e., do not scale to, today's extremely large volumes of data. The data are made up of a large set of features describing each observation, and the complexity of the models for making predictions tend to increase not only with the number of observations, but also the number of features. Fortunately, not all of the features that make up the data carry meaningful information about making the predictions. Thus irrelevant features should be filtered from the data prior to building a model. Such a process of removing features to produce a subset is commonly referred to as feature subset selection. In this work, we present two new filter-based feature subset selection algorithms that are scalable to large data sets that address: (i) potentially large & distributed data sets, and (ii) they are capable of scaling to very large feature sets. Our first proposed algorithm, Neyman-Pearson Feature Selection (NPFS), uses a statistical hypothesis test derived from the Neyman-Pearson lemma for determining if a feature is statistically relevant. The proposed approach can be applied as a wrapper to any feature selection algorithm, regardless of the feature selection criteria used, to determine whether a feature belongs in the relevant set. Perhaps more importantly, this procedure efficiently determines the number of relevant features given an initial starting point, and it fits into a computationally attractive MapReduce model. We also describe a sequential learning framework for feature subset selection (SLSS) that scales with both the number of features as well as the number of observations. SLSS uses bandit algorithms to process features and form a level of importance for each feature. Feature selection is performed independently from the optimization of any classifier to reduce unnecessary complexity. We demonstrate the capabilities of NPFS and SLSS on synthetic and real-world data sets. We also present a new approach for classifier-dependent feature selection that is an online learning algorithm that easily handles large amounts of missing feature values in a data stream. There are many real-world applications that can benefit from scalable feature subset selection algorithms; one such area is the study of the microbiome (i.e., the study of micro-organisms and their influence on the environments that they inhabit). Feature subset selection algorithms can be used to sift through massive amounts of data collected from the genomic sciences to help microbial ecologists understand the microbes - particularly the micro-organisms that are the best indicators by some phenotype, such as healthy or unhealthy. In this work, we provide insights into data collected from the American Gut Project, and deliver open-source software implementations for feature selection with biological data formats.

Feature selection plays an important role in data mining and pattern recognition. However, most existing feature selection methods suffer from stagnation in local optimal and/or high computational cost. In general, the feature selection process can be considered as a global optimization problem. The swarm intelligence algorithm can effectively find the optimal feature subset due to its global search ability. But it usually consumes very long running time when dealing with large data sets. In this paper, feature selection is transformed into a global optimization problem, which provides a fast and efficient method based on swarm intelligence algorithm. First, we propose a global optimization framework for filter-based feature selection and its mathematical model. Furthermore, to solve the feature selection problem for acoustic defect detection, we combine the shuffled frog leaping algorithm (SFLA) with an improved minimum-redundancy maximum-relevancy (ImRMR), named SFLA-ImRMR. In the experiments, a back propagation neural network is employed to evaluate the classification performance of the selected feature subset on the test sets of acoustic defect detection. The results show that SFLA-ImRMR achieved similar performance to the other algorithms within the shortest time.

ABSTRACTPolycystic ovary syndrome (PCOS) is a prevalent endocrine condition in women of reproductive age that is linked to a number of issues with metabolism and reproductive health. For PCOS to be effectively treated and managed, a precise and early diagnosis is essential. In recent years, machine learning (ML) algorithms have demonstrated promising results in the diagnosis of PCOS by extracting pertinent features from clinical and laboratory data. Motivated by the diagnostic complexity and subjectivity in current PCOS identification techniques, this study addresses the gap in systematic evaluations of feature selection and classifier performance. The key objective is to develop an effective ML‐based diagnostic framework by identifying the most informative features, selecting optimal classifiers, and analyzing the impact of data balancing and validation strategies. In this study, we present a comparative analysis of three feature selection methods, namely chi‐square test, backward elimination, and mutual information gain, for identifying the most significant features from a dataset of 541 patients with and without PCOS. Additionally, we investigate the impact of dataset resampling on classification performance, evaluate the classification performance of individual features, compare the performance of various classifiers, and assess the classification performance for different values of k in k‐fold cross‐validation. As evidenced by our findings, the backward feature elimination method performs better than the other two methods in terms of identifying pertinent features for PCOS diagnosis, obtaining a Random Forest (RF) classifier accuracy of 96.55%. The most important features identified were anti‐Mullerian hormone (AMH) level, regularity or irregularity in cycle, follicle no. in left and right ovaries, and endometrium dimension. Our results illustrate the significance of taking into account various feature selection methods to uncover the most relevant features and demonstrate the potential of ML and feature selection techniques in effectively detecting PCOS. Additionally, we briefly explore the potential integration of real‐time data from Internet of Medical Things (IoMT) devices into the PCOS diagnosis framework to enhance its real‐world applicability.