Exploiting Multilabel Information for Noise-Resilient Feature Selection

Feature selection for multilabel classification with missing labels via multi-scale fusion fuzzy uncertainty measures

Previous chapter Next chapter Full AccessProceedings Proceedings of the 2016 SIAM International Conference on Data Mining (SDM)Robust Unsupervised Feature Selection on Networked DataJundong Li, Xia Hu, Liang Wu, and Huan LiuJundong Li, Xia Hu, Liang Wu, and Huan Liupp.387 - 395Chapter DOI:https://doi.org/10.1137/1.9781611974348.44PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAboutAbstract Feature selection has shown its effectiveness to prepare high-dimensional data for many data mining and machine learning tasks. Traditional feature selection algorithms are mainly based on the assumption that data instances are independent and identically distributed. However, this assumption is invalid in networked data since instances are not only associated with high dimensional features but also inherently interconnected with each other. In addition, obtaining label information for networked data is time consuming and labor intensive. Without label information to direct feature selection, it is difficult to assess the feature relevance. In contrast to the scarce label information, link information in networks are abundant and could help select relevant features. However, most networked data has a lot of noisy links, resulting in the feature selection algorithms to be less effective. To address the above mentioned issues, we propose a robust unsupervised feature selection framework NetFS for networked data, which embeds the latent representation learning into feature selection. Therefore, content information is able to help mitigate the negative effects from noisy links in learning latent representations, while good latent representations in turn can contribute to extract more meaningful features. In other words, both phases could cooperate and boost each other. Experimental results on real-world datasets demonstrate the effectiveness of the proposed framework. Previous chapter Next chapter RelatedDetails Published:2016eISBN:978-1-61197-434-8 https://doi.org/10.1137/1.9781611974348Book Series Name:ProceedingsBook Code:PRDT16Book Pages:1-867

In multi-label learning, each object belongs to multiple class labels simultaneously. In the data explosion age, the size of data is often huge, i.e., large number of instances, features and class labels. The high dimension of both the feature and label spaces has posed great challenges to multi-label learning problems, e.g., high time and memory costs. In this paper, we propose a new framework for multi-label learning with a large number of class labels and features, i.e., M ulti- L abel L earning via F eature and L abel S pace D imension R eduction, namely MLL-FLSDR. Specifically, both the feature space and label space are reduced to low dimensional spaces respectively, in which the local structure of data points is utilized to constrain the geometrical structure on both the learned low dimensional spaces and guarantee the qualities of them. Then, an effective multi-label classifier is constructed from the low dimensional feature space to the latent label space. Last, the final prediction for new test data examples can be obtained by recovering from their prediction results in the latent label space with an encoding matrix learned in the previous stage. Extensive comparison experiments with the state-of-the-art approaches manifest the effectiveness of the proposed method MLL-FLSDR.

Existing multi-label learning classification algorithms ignore that class labels may be determined by some features in the original feature space. And only a partial label of each instance can be obtained for some real applications. Therefore, we propose a novel algorithm named joint Label-Specific features and Label Correlation for multi-label learning with Missing Label (LSLC-ML) and its optimized version to solve the above-mentioned problems. First, a missing label can be recovered by the learned positive and negative label correlations from the incomplete training data sets, then the label-specific features can be selected, finally the multi-label classification task can be modeled by combining the labelspecific feature selections, missing labels and positive and negative label correlations. The experimental results show that our algorithm LSLC-ML has strong competitiveness compared with some state-of-the-art algorithms in evaluation matrices when tested on benchmark multi-label data sets.

Feature selection is an important task in machine learning. It can improve classification accuracy and effectively reduce the dataset dimensionality by removing no discriminative features. Though a large body of researches were focused on feature selection for text classification, few works addressed the problem for multi-label data in big data context. Therefore, this paper proposes a distributed feature selection approach for multi-label textual big data based on the weighted chi-square method. First, a standard multi-label approach to transform the multi-label data into single-label data is applied. Then, the algorithm assigns different weights to the features based on the category term frequency and then calculates the chi-square based on the weight of each feature. The proposed method is implemented on Hadoop framework using MapReduce programming model. At last, a set of experiments were conducted on three benchmarking text datasets to evaluate the effectiveness of the proposed approach. A comparative analysis of the results with the state-of-the-art techniques proves that our method is efficient, robust and scalable.

Multi-label learning draws great interests in many real world applications. It is a highly costly task to assign many labels by the oracle for one instance. Meanwhile, it is also hard to build a good model without diagnosing discriminative labels. Can we reduce the label costs and improve the ability to train a good model for multi-label learning simultaneously? Active learning addresses the less training samples problem by querying the most valuable samples to achieve a better performance with little costs. In multi-label active learning, some researches have been done for querying the relevant labels with less training samples or querying all labels without diagnosing the discriminative information. They all cannot effectively handle the outlier labels for the measurement of uncertainty. Since maximum correntropy criterion (MCC) provides a robust analysis for outliers in many machine learning and data mining algorithms, in this paper, we derive a robust multi-label active learning algorithm based on an MCC by merging uncertainty and representativeness, and propose an efficient alternating optimization method to solve it. With MCC, our method can eliminate the influence of outlier labels that are not discriminative to measure the uncertainty. To make further improvement on the ability of information measurement, we merge uncertainty and representativeness with the prediction labels of unknown data. It cannot only enhance the uncertainty but also improve the similarity measurement of multi-label data with labels information. Experiments on benchmark multi-label data sets have shown a superior performance than the state-of-the-art methods.

Many real-world data have multiple class labels known as multi-label data, where the labels are correlated with each other, and as such, they are not independent. Since these data are usually high-dimensional, and the current multi-label feature selection methods have not been precise enough, then a new feature selection method is necessary. In this paper, for the first time, we have modeled the problem of multi-label feature selection to a bipartite graph matching process. The proposed method constructs a bipartite graph of features (as the left vertices) and labels (as the right vertices), called Feature-Label Graph (FLG), where each feature is connected to the set of labels, where the weight of the edge between each feature and label is equal to their correlation. Then, the Hungarian algorithm estimates the best matching in FLG. The selected features in each matching are sorted by weighted correlation distance and added to the ranking vector. To select the discriminative features, the proposed method considers both the redundancy of features and the relevancy of each feature to the class labels. The results indicate the superiority of the proposed method against the other methods in classification measures.

Semi-supervised multi-label feature selection with adaptive structure learning and manifold learning

Tremendous efforts have been dedicated to improving the effectiveness of multi-label learning with incomplete label assignments. Most of the current techniques assume that the input features of data instances are complete. Nevertheless, the co-occurrence of highly incomplete features and weak label assignments is a challenging and widely perceived issue in real-world multi-label learning applications due to a number of practical reasons including incomplete data collection, moderate labels from annotators, etc. Existing multi-label learning algorithms are not directly applicable when the observed features are highly incomplete. In this work, we attack this problem by proposing a weakly supervised multi-label learning approach, based on the idea of collaborative embedding. This approach provides a flexible framework to conduct efficient multi-label classification at both transductive and inductive mode by coupling the process of reconstructing missing features and weak label assignments in a joint optimisation framework. It is designed to collaboratively recover feature and label information, and extract the predictive association between the feature profile and the multi-label tag of the same data instance. Substantial experiments on public benchmark datasets and real security event data validate that our proposed method can provide distinctively more accurate transductive and inductive classification than other state-of-the-art algorithms.

Information theoretical based methods have attracted a great attention in recent years, and gained promising results to deal with multi-label data with high dimensionality. However, most of the existing methods are either directly transformed from heuristic single-label feature selection methods or inefficient in exploiting labeling information. Thus, they may not be able to get an optimal feature selection result shared by multiple labels. In this paper, we propose a general global optimization framework, in which feature relevance, label relevance (i.e., label correlation), and feature redundancy are taken into account, thus facilitating multi-label feature selection. Moreover, the proposed method has an excellent mechanism for utilizing inherent properties of multi-label learning. Specially, we provide a formulation to extend the proposed method with label-specific features. Empirical studies on twenty multi-label data sets reveal the effectiveness and efficiency of the proposed method. Our implementation of the proposed method is available online at: https://jiazhang-ml.pub/GRRO-master.zip.

Feature selection based on label distribution and fuzzy mutual information

Multi-label feature selection attracts considerable attention from multi-label learning. Information theory-based multi-label feature selection methods intend to select the most informative features and reduce the uncertain amount of information of labels. Previous methods regard the uncertain amount of information of labels as constant. In fact, as the classification information of the label set is captured by features, the remaining uncertainty of each label is changing dynamically. In this paper, we categorize labels into two groups: One contains the labels with few remaining uncertainty, which means that most of classification information with respect to the labels has been obtained by the already-selected features; another group contains the labels with extensive remaining uncertainty, which means that the classification information of these labels is neglected by already-selected features. Feature selection aims to select the new features that are highly relevant to the labels in the second group. Existing methods do not distinguish the difference between two label groups and ignore the dynamic change amount of information of labels. To this end, a Relevancy Ratio is designed to clarify the dynamic change amount of information of each label under the condition of the already-selected features. Afterward, a Weighted Feature Relevancy is defined to evaluate the candidate features. Finally, a new multi-label feature selection method based on Weighted Feature Relevancy (WFRFS) is proposed. The experiments obtain encouraging results of WFRFS in comparison with six multi-label feature selection methods on thirteen real-world data sets.

Feature selection, an effective technique for dimensionality reduction, plays an important role in many machine learning systems. Supervised knowledge can significantly improve the performance. However, faced with the rapid growth of newly emerging concepts, existing supervised methods might easily suffer from the scarcity and validity of labeled data for training. In this paper, the authors study the problem of zero-shot feature selection (i.e., building a feature selection model that generalizes well to “unseen” concepts with limited training data of “seen” concepts). Specifically, they adopt class-semantic descriptions (i.e., attributes) as supervision for feature selection, so as to utilize the supervised knowledge transferred from the seen concepts. For more reliable discriminative features, they further propose the center-characteristic loss which encourages the selected features to capture the central characteristics of seen concepts. Extensive experiments conducted on various real-world datasets demonstrate the effectiveness of the method.

Existing multi-label learning algorithms mostly assume that all class labels in an obtained data set are completely accurate. This assumption is rarely valid. As the scale of the label space increases, owing to limited domain knowledge and differences among methods of data acquisition, some examples will be unlabeled or mislabeled. The introduction of such noise results in the poor performance of trained multi-label classifiers. This paper proposes a new method for multi-label learning with label noise, called low-rank-constraint-based multi-label learning with label noise (ML-LRC), by incorporating low-rank matrix recovery and features selection into the learning process. First, we assume that two strongly correlated labels share more samples than two weakly correlated labels. Based on this assumption, the ML-LRC applies low-rank constraints to the label matrix to mine the local correlations of class labels and remove noise from the observed label matrix. Second, we believe that each class label only correlates to some features. So we control the sparsity of the coefficient matrix to filter out label-specific features, which can reduce the complexity of the model and greatly improve the performance of the classifier. The accelerated proximal gradient method and alternating direction method of multipliers are adopted to solve the problem. Experiments conducted on four benchmark multi-label datasets demonstrate the competitive performance of our proposed method compared with several state-of-the-art methods.

The current study considers an unconventional framework of unsupervised feature selection for supervised learning. We provide a new unsupervised algorithm which we call LIA, for Label-Independent Algorithm, which combines information theory and network science techniques. In addition, we present the results of an empirical study comparing LIA with a standard supervised algorithm (MRMR). The two algorithms are similar in that both minimize redundancy, but MRMR uses the labels of the instances in the input data set to maximize the relevance of the selected features. This is an advantage when such labels are available, but a shortcoming when they are not. We used cross-validation to evaluate the effectiveness of selected features for generating different well-known classifiers for a variety of publicly available data sets. The results show that the performance of classifiers using the features selected by our proposed label-independent algorithm is very close to or in some cases better than their performance when using the features selected by a common label-dependent algorithm (MRMR). Thus, LIA has potential to be useful for a wide variety of applications where dimension reduction is needed and the instances in the data set are unlabeled.