Unsupervised feature selection via distributed coding for multi-view object recognition

The problem of feature selection has been an area of considerable research in machine learning. Feature selection is known to be particularly difficult in unsupervised learning because different subgroups of features can yield useful insights into the same dataset. In other words, many theoretically-right answers may exist for the same problem. Furthermore, designing algorithms for unsupervised feature selection is technically harder than designing algorithms for supervised feature selection because unsupervised feature selection algorithms cannot be guided by class labels. As a result, previous work attempts to discover intrinsic structures of data with heavy computation such as matrix decomposition, and require significant time to find even a single solution. This paper proposes a novel algorithm, named Explainability-based Unsupervised Feature Value Selection (EUFVS), which enables a paradigm shift in feature selection, and solves all of these problems. EUFVS requires only a few tens of milliseconds for datasets with thousands of features and instances, allowing the generation of a large number of possible solutions and select the solution with the best fit. Another important advantage of EUFVS is that it selects feature values instead of features, which can better explain phenomena in data than features. EUFVS enables a paradigm shift in feature selection. This paper explains its theoretical advantage, and also shows its applications in real experiments. In our experiments with labeled datasets, EUFVS found feature value sets that explain labels, and also detected useful relationships between feature value sets not detectable from given class labels.KeywordsFeature selectionUnsupervised learningClustering

Feature selection plays a crucial role in scientific research and practical applications. In the real world applications, labeling data is time and labor consuming. Thus, unsupervised feature selection methods are desired for many practical applications. Linear discriminant analysis (LDA) with trace ratio criterion is a supervised dimensionality reduction method that has shown good performance to improve classifications. In this paper, we first propose a unified objective to seamlessly accommodate trace ratio formulation and K-means clustering procedure, such that the trace ratio criterion is extended to unsupervised model. After that, we propose a novel unsupervised feature selection method by integrating unsupervised trace ratio formulation and structured sparsity-inducing norms regularization. The proposed method can harness the discriminant power of trace ratio criterion, thus it tends to select discriminative features. Meanwhile, we also provide two important theorems to guarantee the unsupervised feature selection process. Empirical results on four benchmark data sets show that the proposed method outperforms other sate-of-the-art unsupervised feature selection algorithms in all three clustering evaluation metrics.

Recently, the latest advances in compact feature representation and feature learning have provided an efficient framework for several visual analysis tasks, such as object recognition. However, when multiple cameras with overlapping fields-of-view are employed, other visual analysis tasks such as depth estimation can be supported and object recognition accuracy can be improved. In this paper the problem of distributed visual analysis from multiple views of a scene is addressed, considering that computational power and bandwidth, at each camera sensor, are rather limited. More specifically, an efficient coding technique for local binary features is proposed which exploits the correlation at the decoder side between each descriptor and its quantized representation. Moreover, considering that descriptors representing the same visual feature across different views are well correlated, a technique to avoid the transmission of redundant descriptors from multiple views is proposed. At the decoder, the joint statistics of all descriptors from all views is used to drive the selection of the best descriptors to be transmitted by each sensing node. The proposed multi-view feature coding and selection techniques allow obtaining bitrate reductions up to 80%, with respect to the uncompressed descriptor rate, for a certain task accuracy.

Existing unsupervised feature selection algorithms are designed to extract the most relevant subset of features which can facilitate clustering and interpretation of the obtained results. However, these techniques are not directly applicable in many real-world situations in which one has access to datasets consisting of multiple views or dissimilar representations. By leveraging information from these different views one can obtain more robust and accurate results compared with single-view methods. In this paper Unsupervised Multi-View Feature Selection algorithm is used to simultaneously extract a relevant subset of features and to perform clustering that is consistent across different views. In the empirical evaluation of synthetic multi-view data we demonstrate the advantage of using the proposed algorithm over the use of its single-view analogue. We used the National Cancer Institute's NCI-60 panel dataset, exploiting information from various -omics representations to identify tumor subtypes based on very few requisite, biologically relevant features.

An unsupervised feature selection algorithm based on ant colony optimization

Feature selection plays an important role in preprocessing in pattern recognition and data mining, especially in large scale image, digital text, and biological data. Specifically, class label information is unavailable for conducting the selection of a minimal feature subset in unsupervised learning, which is full of challenging and interesting problems. In this paper, we present an unsupervised REPresentative Feature Selection (REPFS) algorithm based on information entropy and relevance analysis. The proposed method seeks to find a high-quality feature subset through feature clustering without using any learning algorithms. More importantly, the features’ relevance will be computed based on an information metric of the relevance gain, which provides an information theoretical foundation for finding a minimum of the redundancy between features. Our results on nine benchmark data sets demonstrate that REPFS can significantly improve upon state-of-the-art unsupervised algorithms.

Unsupervised feature selection via discrete spectral clustering and feature weights

The application of machine learning in health care often necessitates the use of hierarchical codes such as the International Classification of Diseases (ICD) and Anatomical Therapeutic Chemical (ATC) systems. These codes classify diseases and medications, respectively, thereby forming extensive data dimensions. Unsupervised feature selection tackles the "curse of dimensionality" and helps to improve the accuracy and performance of supervised learning models by reducing the number of irrelevant or redundant features and avoiding overfitting. Techniques for unsupervised feature selection, such as filter, wrapper, and embedded methods, are implemented to select the most important features with the most intrinsic information. However, they face challenges due to the sheer volume of ICD and ATC codes and the hierarchical structures of these systems. The objective of this study was to compare several unsupervised feature selection methods for ICD and ATC code databases of patients with coronary artery disease in different aspects of performance and complexity and select the best set of features representing these patients. We compared several unsupervised feature selection methods for 2 ICD and 1 ATC code databases of 51,506 patients with coronary artery disease in Alberta, Canada. Specifically, we used the Laplacian score, unsupervised feature selection for multicluster data, autoencoder-inspired unsupervised feature selection, principal feature analysis, and concrete autoencoders with and without ICD or ATC tree weight adjustment to select the 100 best features from over 9000 ICD and 2000 ATC codes. We assessed the selected features based on their ability to reconstruct the initial feature space and predict 90-day mortality following discharge. We also compared the complexity of the selected features by mean code level in the ICD or ATC tree and the interpretability of the features in the mortality prediction task using Shapley analysis. In feature space reconstruction and mortality prediction, the concrete autoencoder-based methods outperformed other techniques. Particularly, a weight-adjusted concrete autoencoder variant demonstrated improved reconstruction accuracy and significant predictive performance enhancement, confirmed by DeLong and McNemar tests (P<.05). Concrete autoencoders preferred more general codes, and they consistently reconstructed all features accurately. Additionally, features selected by weight-adjusted concrete autoencoders yielded higher Shapley values in mortality prediction than most alternatives. This study scrutinized 5 feature selection methods in ICD and ATC code data sets in an unsupervised context. Our findings underscore the superiority of the concrete autoencoder method in selecting salient features that represent the entire data set, offering a potential asset for subsequent machine learning research. We also present a novel weight adjustment approach for the concrete autoencoders specifically tailored for ICD and ATC code data sets to enhance the generalizability and interpretability of the selected features.

Compared to supervised feature selection, unsupervised feature selection tends to be more challenging due to the lack of guidance from class labels. Along with the increasing variety of data sources, many datasets are also equipped with certain side information of heterogeneous structure. Such side information can be critical for feature selection when class labels are unavailable. In this paper, we propose a new feature selection method, SideFS, to exploit such rich side information. We model the complex side information as a heterogeneous network and derive instance correlations to guide subsequent feature selection. Representations are learned from the side information network and the feature selection is performed in a unified framework. Experimental results show that the proposed method can effectively enhance the quality of selected features by incorporating heterogeneous side information.

The explosive usage of social media produces massive amount of unlabeled and high-dimensional data. Feature selection has been proven to be effective in dealing with high-dimensional data for efficient learning and data mining. Unsupervised feature selection remains a challenging task due to the absence of label information based on which feature relevance is often assessed. The unique characteristics of social media data further complicate the already challenging problem of unsupervised feature selection, e.g., social media data is inherently linked, which makes invalid the independent and identically distributed assumption, bringing about new challenges to unsupervised feature selection algorithms. In this paper, we investigate a novel problem of feature selection for social media data in an unsupervised scenario. In particular, we analyze the differences between social media data and traditional attribute-value data, investigate how the relations extracted from linked data can be exploited to help select relevant features, and propose a novel unsupervised feature selection framework, LUFS, for linked social media data. We systematically design and conduct systemic experiments to evaluate the proposed framework on data sets from real-world social media websites. The empirical study demonstrates the effectiveness and potential of our proposed framework.

Technological advancements have generated huge amount of data in recent times. The objective is to analyze this data to get meaningful insights from them. One of the techniques is to perform clustering, an unsupervised learning technique. The performance of clustering is dependent on the size of data (number of features, number of records), a well-known fact. Feature selection/extraction algorithms not only eliminate irrelevant features from the data but also improve the performance of the clustering algorithm applied. Knowing your data sets (type, nature, and size), the appropriate algorithm to use and the appropriate evaluation metrics that will be insightful to improve the quality of clusters is important. No such correlation has been established as per author’s knowledge. In this paper, we report the results of various internal and external evaluation metrics on application of distinct unsupervised feature selection/extraction algorithms on a variety of data sets. The output of various algorithms indicates that retaining 100 features give best results. The experiments prove that Principal Component Analysis (PCA) algorithm is suitable for linear and low-dimensional data sets. Laplacian Score, Non-negative Discriminative Feature Selection (NDFS), and Spectral Feature Selection (SPEC) work well with high dimensional, binary data sets. Low variance algorithm and Unsupervised Discriminative Feature Selection (UDFS) do not perform well on any type of data sets.

Simultaneous positive sequential vectors modeling and unsupervised feature selection via continuous hidden Markov models

Detecting cancer at an early stage is an important application, which reduces the risk of death for a cancerous patient. Data from tissues is generated using microarray DNA technology and these data contains genes/attributes of very large number and samples of relatively small number, making it tough to predict the classifier. Hence, several machine learning algorithms with more emphasis on feature selection were proposed to solve the problem of handling large number of genes/attributes. Most of the feature selection (FS) algorithms mentioned in the literature are Supervised learning algorithms. Authors proposed an Unsupervised feature selection algorithm, making it more independent of the class label. Simple Feature ranking algorithm using Single Value Decomposition (SVD)-Entropy is the first step in the proposed feature selection algorithm. SVD-Entropy based method selects attributes independent of the each other, hence reduces complexity involved in multiple association of attributes in large datasets. At second stage, Correlation among attributes is used to remove attributes/features that are highly correlated to each other. Once the features are selected, a logistic regression model for predicting the class label. The model predicts whether its a cancer or non-cancer causing tissue. The proposed algorithm proved to be an efficient approach in terms of accurately predicting the cancer causing tissues. Experiments were carried on three different datasets like Ovarian, Lung and Breast cancer. The accuracy scores achieved for these datasets are 100, 75 and 96.4 percent, respectively, proving the efficiency of the approach.

Feature Selection (FS) has become one of the most active research topics in the area of data mining. It performs to remove redundant and noisy features from high-dimensional data sets. A good feature selection has several advantages for a learning algorithm such as reducing computational cost, increasing its classification accuracy and improving result comprehensibility. In the supervised FS methods various feature subsets are evaluated using an evaluation function or metric to select only those features which are related to the decision classes of the data under consideration. However, for many data mining applications, decision class labels are often unknown or incomplete, thus indicating the significance of unsupervised feature selection. However, in unsupervised learning, decision class labels are not provided. The problem is that not all features are important. Some of the features may be redundant, and others may be irrelevant and noisy. In this paper, a novel unsupervised feature selection in mammogram image, using rough set based measures, is proposed. A typical mammogram image processing system generally consists of mammogram image acquisition, preprocessing of image, segmentation, features extracted from the segmented mammogram image. The proposed method is used to select features from data set, the method is compared with existing rough set based supervised feature selection methods and classification performance of both methods are recorded and demonstrates the efficiency of the method.

The main goal of most information quality (IQ)-based measures is to combine data provided by multiple information sources to enhance the quality of information essential for decision makers to perform their tasks. However, there is few work to fuse multisource information from the perspective of possibility distribution (PD) and use IQ as the evaluation criteria for feature selection. The PD is one of important concepts in the possibility theory, which is a generally acknowledged method for describing a kind of uncertain knowledge. In this article, we propose a novel representation model of PDs based on FMs, namely, a possibility distribution information system (PDIS). Then, several IQ measures are defined in the PDIS, including Gini entropy, compatibility, conflict, credibility, and separability degrees. In view of this, a minimal-separability-minimal-uncertainty-based unsupervised feature selection algorithm (UmSMU) is designed. The proposed UmSMU can sufficiently fuse multiple possibilistic information. Meanwhile, the selected features maintain as much information as possible while minimizing the uncertainty of information. The experimental results show that the proposed algorithm performs well, especially when it comes to selecting fewer features and improving performance.