A Distributed Integrated Feature Selection Scheme for Column Subset Selection

The common criteria for evaluating spectral clustering are NCut and RatioCut. The seemingly unrelated column subset selection (CSS) problem aims to compute a column subset that linearly approximates the entire matrix. A common criterion is the approximation error in the Frobenius norm (ApproxErr). We show that any algorithm for CSS can be viewed as a clustering algorithm that minimizes NCut by applying it to a matrix formed from graph edges. Conversely, any clustering algorithm can be seen as identifying a column subset from that matrix. In both cases, ApproxErr and NCut have the same value. Analogous results hold for RatioCut with a slightly different matrix. Therefore, established results for CSS can be mapped to spectral clustering. We use this to obtain new clustering algorithms, including an optimal one that is similar to A*. This is the first nontrivial clustering algorithm with such an optimality guarantee. A variant of the weighted A* runs much faster and provides bounds on the accuracy. Finally, we use the results from spectral clustering to prove the NP-hardness of CSS from sparse matrices.

Observations from hyperspectral imaging sensors lead to high dimensional data sets from hundreds of images taken at closely spaced narrow spectral bands. High storage and transmission requirements, computational complexity, and statistical modeling problems combined with physical insight motivate the idea of hyperspectral dimensionality reduction using band subset selection. Many algorithms are described in the literature to solve supervised and unsupervised band subset selection problems. This paper explores the use of unsupervised band subset selection methods using column subset selection (CSS). Column subset selection is the problem (CSSP) of selecting the most independent columns of a matrix. A recent variant of this problem is the positive column subset selection problem (pCSSP) which restricts column subset selection to only consider positive linear combinations. Many algorithms have been proposed in the literature for the solution of the CSSP. However, the pCSSP is less studied. This paper will present a comparison of different algorithms to solve the CSSP and the pCSSP for band subset selection. The performance of classifiers using the algorithms as a dimensionality reduction stage will be used to evaluate the usefulness of these algorithms in hyperspectral image exploitation.

Low rank approximation of matrices is an important tool in machine learning. Given a data matrix, low rank approximation helps to find factors, patterns, and provides concise representations for the data. Research on low rank approximation usually focuses on real matrices. However, in many applications data are binary (categorical) rather than continuous. This leads to the problem of low rank approximation of binary matrices. Here we are given a d x n binary matrix A and a small integer k < d. The goal is to find two binary matrices U and V of sizes d x k and k x n respectively, so that the Frobenius norm of A - U V is minimized. There are two models of this problem, depending on the definition of the dot product of binary vectors: The GF(2) model and the Boolean semiring model. Unlike low rank approximation of a real matrix which can be efficiently solved by Singular Value Decomposition, we show that approximation of a binary matrix is NP-hard, even for k=1.\nIn this paper, our main concern is the problem of Column Subset Selection (CSS), in which the low rank matrix U must be formed by k columns of the data matrix, and we are interested in the approximation ratio achievable by CSS for binary matrices. For the GF(2) model, we show that CSS has approximation ratio bounded by k/2+1+k/(2(2^k-1)) and this is asymptotically tight. For the Boolean model, it turns out that CSS is no longer sufficient to obtain a bound. We then develop a Generalized CSS (GCSS) procedure in which the columns of U are generated from Boolean formulas operating bitwise on selected columns of the data matrix. We show that the approximation ratio achieved by GCSS is bounded by 2^(k-1)+1, and argue that an exponential dependency on k is seems inherent.

We consider feature selection for text classification both theoretically and empirically. Our main result is an unsupervised feature selection strategy for which we give worst-case theoretical guarantees on the generalization power of the resultant classification function f with respect to the classification function f obtained when keeping all the features. To the best of our knowledge, this is the first feature selection method with such guarantees. In addition, the analysis leads to insights as to when and why this feature selection strategy will perform well in practice. We then use the TechTC-100, 20-Newsgroups, and Reuters-RCV2 data sets to evaluate empirically the performance of this and two simpler but related feature selection strategies against two commonly-used strategies. Our empirical evaluation shows that the strategy with provable performance guarantees performs well in comparison with other commonly-used feature selection strategies. In addition, it performs better on certain datasets under very aggressive feature selection.

The trend of current brain computer interfaces (BCI) seek to establish bi-directional communication with the brain, for instance, recovering motor functions by externally controlling devices and directly stimulating the brain. This will greatly assist paralyzed individuals through bypassing the damaged brain region. The key process of this communication interface is to decode movements from neural signals and encode information into neural activity. The majority of decoding or pattern classification studies have focused on cortical areas for BCIs, but deep brain structures have also been involved in motor control. The subthalamic nucleus (STN) in the basal ganglia is involved in the preparation, execution and imagining of movements, and may be an alternative source for driving BCIs. This study therefore aimed to classify patterns of deep brain local field potentials (LFPs) related to execution of visually cued movements. LFPs were recorded bilaterally from the STN through deep brain stimulation electrodes implanted in patients with Parkinson's disease. The frequency dependent components of the LFPs were extracted using the wavelet packet transform. In each frequency component, signal features were extracted using an alternative approach called neural synchronization by analyzing Granger causality between the STN. Based on these extracted features, a new feature selection strategy, namely weighted sequential feature selection (WSFS) was developed to efficiently select the optimal feature subset. A support vector machine (SVM) classifier was implemented alongside this novel feature extraction and selection strategy, and evaluated using a cross-validation procedure. Using this optimised feature subset, average correct pattern classification accuracy of movement (left or right) reached 76.0±3.1%. The results obtained in this study are encouraging and suggest that the neural activity in the deep neural circuit (basal ganglia) can be used for controlling BCIs.

Distributed feature selection (DFS) strategy for microarray gene expression data to improve the classification performance

Many attempts have been made to analyze gene expression data. Typical goals of such analysis include discovery of subclasses, designing predictors/classifiers for diseases, identifying marker genes, and trying to get a deeper understanding of underlying biological process. Success of each of these tasks strongly depends on the features used to solve the problem. The high dimensional nature of expression profiles makes the task very difficult. Consequently, many researchers have used some feature selection criteria to reduce the dimensionality of the problem. These approaches are off-line in nature, as feature selection is done in a separate phase from the system design phase. These approaches ignore the fact that utility of features depends on both the problem that is solved and the tool that is used to solve the problem. We here propose to use a novel neural scheme that picks up the necessary features on-line when the system learns the classification task. Because it considers all the features at one go, it does not miss any subtle combination of these features. We demonstrate the effectiveness of our on-line feature selection (OFS) scheme to distinguish between acute myeloid leukemia (AML) and acute lymphoblastic leukemia (ALL) cancer expression data set. Our scheme could identify only five genes that can produce results as good as or even better than what is reported in the literature on this data set. It identifies an important marker gene that alone has a very good discriminating power. This analysis method is quite general in nature and can be effectively used in other areas of bioinformatics. © 2006 Wiley Periodicals, Inc. Int J Int Syst 21: 453–467, 2006.

In this paper, we demonstrate the need for dimensionality reduction to mitigate model overfitting on the nontrivial problem of gender classification from digital images. In this study we explore four feature selection schemes using Genetic Algorithm, Memetic Algorithms, and Random Forest, which are fed to a nonlinear support vector machine (SVM) for final classification. The performance of the model (feature) selection approaches are evaluated against two distinct datasets of facial images: FG-NET which contains toddlers to seniors and the UIUC-PAL which contains faces of adults up to seniors. This work demonstrates that feature selection can, and does, improve performance of an SVM based gender classification system significantly.

A hybrid feature selection scheme and self-organizing map model for machine health assessment

In multi-label text classification, the central aim is to associate an array of descriptive labels for a better understanding of the text. There are three main challenges in doing multi-label text classification (i) a large number of text (input) features, (ii) the underlying implicit relationship between input features and output labels, and (iii) an implicit inter-label dependency. In traditional approaches to multi-label classification, these problems are not being addressed collectively. A feature selection strategy that inherently uses local features to discriminate a class and similarly global features that can distinctly separate classes can be very effective for multi-label classification. In this research, we perform a feature selection and ranking strategy based on local and global features. A Naïve Bayes classifier is being used using a combination of these two -feature sets, it is compared with the baseline implemented with the term frequency-inverse document frequency (TF-IDF). A series of experiments have been carried out on standard multi-label text datasets, using evaluation metrics like Hamming loss, Subset Accuracy and Micro/Macro F1 scores, and encouraging results are obtained.

We present an integrated algorithm for simultaneous feature selection (FS) and designing of diverse classifiers using a steady state multiobjective genetic programming (GP), which minimizes three objectives: 1) false positives (FPs); 2) false negatives (FNs); and 3) the number of leaf nodes in the tree. Our method divides a c -class problem into c binary classification problems. It evolves c sets of genetic programs to create c ensembles. During mutation operation, our method exploits the fitness as well as unfitness of features, which dynamically change with generations with a view to using a set of highly relevant features with low redundancy. The classifiers of i th class determine the net belongingness of an unknown data point to the i th class using a weighted voting scheme, which makes use of the FP and FN mistakes made on the training data. We test our method on eight microarray and 11 text data sets with diverse number of classes (from 2 to 44), large number of features (from 2000 to 49 151), and high feature-to-sample ratio (from 1.03 to 273.1). We compare our method with a bi-objective GP scheme that does not use any FS and rule size reduction strategy. It depicts the effectiveness of the proposed FS and rule size reduction schemes. Furthermore, we compare our method with four classification methods in conjunction with six features selection algorithms and full feature set. Our scheme performs the best for 380 out of 474 combinations of data sets, algorithm and FS method.

A modified genetic algorithm and weighted principal component analysis based feature selection and extraction strategy in agriculture

The efficiency of a binary support vector machine- (SVM-) based classifier depends on the combination and the number of input features extracted from raw signals. Sometimes, a combination of individual good features does not perform well in discriminating a class due to a high level of relevance to a second class also. Moreover, an increase in the dimensions of an input vector also degrades the performance of a classifier in most cases. To get efficient results, it is needed to input a combination of the lowest possible number of discriminating features to a classifier. In this paper, we propose a framework to improve the performance of an SVM-based classifier for sensor fault classification in two ways: firstly, by selecting the best combination of features for a target class from a feature pool and, secondly, by minimizing the dimensionality of input vectors. To obtain the best combination of features, we propose a novel feature selection algorithm that selects m out of M features having the maximum mutual information (or relevance) with a target class and the minimum mutual information with nontarget classes. This technique ensures to select the features sensitive to the target class exclusively. Furthermore, we propose a diversified-input SVM (DI-SVM) model for multiclass classification problems to achieve our second objective which is to reduce the dimensions of the input vector. In this model, the number of SVM-based classifiers is the same as the number of classes in the dataset. However, each classifier is fed with a unique combination of features selected by a feature selection scheme for a target class. The efficiency of the proposed feature selection algorithm is shown by comparing the results obtained from experiments performed with and without feature selection. Furthermore, the experimental results in terms of accuracy, receiver operating characteristics (ROC), and the area under the ROC curve (AUC-ROC) show that the proposed DI-SVM model outperforms the conventional model of SVM, the neural network, and the k-nearest neighbor algorithm for sensor fault detection and classification.

In recent years, deep convolutional features have been deployed in discriminative correlation filters (DCF) to boost object tracking performance. However, features captured from pre-trained classification networks are usually trained for image classification tasks, not object tracking. In this paper, we find that different convolutional feature channels play different roles in tracking different targets. Some feature channels are favorable for tracking a given target and can be acquired based on this target, some are irrelevant to track this target, and some can be the primary cause of trackers' performance degradation when tracking this target. Thus, we perform feature selection before learning correlation filters for object tracking, and the feature selection module is realized by reinforcement learning. We penalize the features non-positive to obtain a DCF tracker based on positive convolutional feature channels. Compared with DCF based trackers without a feature selection technique, our scheme improves the robustness of target representation, lessens the dimension of activations, and achieves better tracking performance. Extensive experiments on the OTB dataset demonstrate our feature selection scheme is simple, robust, and effective for DCF based trackers.

In modern era, Internet plays a key role in accessing and fetching web information and web resources from World Wide Web (WWW). The websites act as a medium for retrieving information from the web. Although it increases the data retrieval and users interactions, it also opens the gate for various types of attacks. For example, spams in the websites attract various Internet users. It has been observed from the literature that many authors attempted to detect the web spam using various machine learning techniques. However, none of these techniques used deep learning architecture for detection of hidden patterns. Hence, in this paper, a deep learning algorithm, i.e., Recurrent Neural Networks (RNN), has been used for the classification of spam nodes. We devise here a framework called FS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> RNN: Feature Selection Scheme using Recurrent Neural Networks. In this framework, the dataset is preprocessed before applying RNN in which principal component analysis (PCA) is used for dimension reduction on the data set and recursive feature elimination (RFE) is used for feature selection. The accuracy of the proposed framework, when compared before and after preprocessing, is improved by 24.2 %, which is excellent result.