Zero-Shot Feature Selection via Transferring Supervised Knowledge

In a conventional supervised learning paradigm, each data instance is associated with one single class label. Multilabel learning differs in the way that data instances may belong to multiple concepts simultaneously, which naturally appear in a variety of high impact domains, ranging from bioinformatics and information retrieval to multimedia analysis. It targets leveraging the multiple label information of data instances to build a predictive learning model that can classify unlabeled instances into one or multiple predefined target classes. In multilabel learning, even though each instance is associated with a rich set of class labels, the label information could be noisy and incomplete as the labeling process is both time consuming and labor expensive, leading to potential missing annotations or even erroneous annotations. The existence of noisy and missing labels could negatively affect the performance of underlying learning algorithms. More often than not, multilabeled data often has noisy, irrelevant, and redundant features of high dimensionality. The existence of these uninformative features may also deteriorate the predictive power of the learning model due to the curse of dimensionality. Feature selection, as an effective dimensionality reduction technique, has shown to be powerful in preparing high-dimensional data for numerous data mining and machine-learning tasks. However, a vast majority of existing multilabel feature selection algorithms either boil down to solving multiple single-labeled feature selection problems or directly make use of the imperfect labels to guide the selection of representative features. As a result, they may not be able to obtain discriminative features shared across multiple labels. In this article, to bridge the gap between a rich source of multilabel information and its blemish in practical usage, we propose a novel noise-resilient multilabel informed feature selection framework (MIFS) by exploiting the correlations among different labels. In particular, to reduce the negative effects of imperfect label information in obtaining label correlations, we decompose the multilabel information of data instances into a low-dimensional space and then employ the reduced label representation to guide the feature selection phase via a joint sparse regression framework. Empirical studies on both synthetic and real-world datasets demonstrate the effectiveness and efficiency of the proposed MIFS framework.

This paper proposes a feature selection method based on a hybrid optimization algorithm that combines the Golden Jackal Optimization (GJO) and Grey Wolf Optimizer (GWO). The primary objective of this method is to create an effective data dimensionality reduction technique for eliminating redundant, irrelevant, and noisy features within high-dimensional datasets. Drawing inspiration from the Chinese idiom "Chai Lang Hu Bao," hybrid algorithm mechanisms, and cooperative behaviors observed in natural animal populations, we amalgamate the GWO algorithm, the Lagrange interpolation method, and the GJO algorithm to propose the multi-strategy fusion GJO-GWO algorithm. In Case 1, the GJO-GWO algorithm addressed eight complex benchmark functions. In Case 2, GJO-GWO was utilized to tackle ten feature selection problems. Experimental results consistently demonstrate that under identical experimental conditions, whether solving complex benchmark functions or addressing feature selection problems, GJO-GWO exhibits smaller means, lower standard deviations, higher classification accuracy, and reduced execution times. These findings affirm the superior optimization performance, classification accuracy, and stability of the GJO-GWO algorithm.

With the advancement of industrial digitization, utilizing large datasets for model training to boost performance is a pivotal technical approach for industry progress. However, raw training datasets often contain abundant redundant features, which increase model training's computational cost and impair generalization ability. To tackle this, this study proposes the bionic ABCCOA algorithm, an enhanced version of the bionic Coati Optimization Algorithm (COA), to improve redundant feature elimination in datasets. To address the bionic COA's inadequate global search performance in feature selection (FS) problems, leading to lower classification accuracy, an adaptive search strategy is introduced. This strategy combines individual learning capability and the learnability of disparities, enhancing global exploration. For the imbalance between the exploration and exploitation phases in the bionic COA algorithm when solving FS problems, which often traps it in suboptimal feature subsets, a balancing factor is proposed. By integrating phase control and dynamic adjustability, a good balance between the two phases is achieved, reducing the likelihood of getting stuck in suboptimal subsets. Additionally, to counter the bionic COA's insufficient local exploitation performance in FS problems, increasing classification error rates, a centroid guidance strategy is presented. By combining population centroid guidance and fractional-order historical memory, the algorithm lowers the classification error rate of feature subsets and speeds up convergence. The bionic ABCCOA algorithm was tested on the CEC2020 test functions and engineering problem, achieving an over 90% optimization success rate and faster convergence, confirming its efficiency. Applied to 27 FS problems, it outperformed comparative algorithms in best, average, and worst fitness function values, classification accuracy, feature subset size, and running time, proving it an efficient and robust FS algorithm.

Many evolutionary computation (EC) methods have been used to solve feature selection problems and they perform well on most small-scale feature selection problems. However, as the dimensionality of feature selection problems increases, the solution space increases exponentially. Meanwhile, there are more irrelevant features than relevant features in datasets, which leads to many local optima in the huge solution space. Therefore, the existing EC methods still suffer from the problem of stagnation in local optima on large-scale feature selection problems. Furthermore, large-scale feature selection problems with different datasets may have different properties. Thus, it may be of low performance to solve different large-scale feature selection problems with an existing EC method that has only one candidate solution generation strategy (CSGS). In addition, it is time-consuming to find a suitable EC method and corresponding suitable parameter values for a given large-scale feature selection problem if we want to solve it effectively and efficiently. In this article, we propose a self-adaptive particle swarm optimization (SaPSO) algorithm for feature selection, particularly for large-scale feature selection. First, an encoding scheme for the feature selection problem is employed in the SaPSO. Second, three important issues related to self-adaptive algorithms are investigated. After that, the SaPSO algorithm with a typical self-adaptive mechanism is proposed. The experimental results on 12 datasets show that the solution size obtained by the SaPSO algorithm is smaller than its EC counterparts on all datasets. The SaPSO algorithm performs better than its non-EC and EC counterparts in terms of classification accuracy not only on most training sets but also on most test sets. Furthermore, as the dimensionality of the feature selection problem increases, the advantages of SaPSO become more prominent. This highlights that the SaPSO algorithm is suitable for solving feature selection problems, particularly large-scale feature selection problems.

Feature Selection methods in Data Mining and Data Analysis problems aim at selecting a subset of the variables, or features, that describe the data in order to obtain a more essential and compact representation of the available information. The selected subset has to be small in size and must retain the information that is most useful for the specific application. The role of Feature Selection is particularly important when computationally expensive Data Mining tools are used, or when the data collection process is difficult or costly. Feature Selection problems are typically solved in the literature using search techniques, where the evaluation of a specific subset is accomplished by a proper function (filter methods) or directly by the performance of a Data Mining tool (wrapper methods). In this work we show how the Feature Selection problem can be formulated as a subgraph selection problem derived from the lightest k-subgraph problem, and solved as an Integer Program. The proposed formulation is very flexible, as additional conditions on the solution can be added in the formulation. Although optimal solutions for such problems are difficult to find in the worst case, a large number of test instances have been solved efficiently by commercial tools. Finally, an application to a database on urban mobility is presented, where the proposed method is integrated in the Data Mining tool named Lsquare and is compared with other approaches.

Multi-level aircraft feature representation and selection for aviation environmental impact analysis

In many applications, like function approximation, pattern recognition, time series prediction, and data mining, one has to build a model relating some features describing the data to some response value. Often, the features that are relevant for building the model are not known in advance. Feature selection methods allow removing irrelevant and/or redundant features to only keep the feature subset that are most useful to build a prediction model. The model is simpler and easier to interpret, reducing the risks of overfitting, non-convergence, etc. By contrast with other dimensionality reduction techniques such as principal component analysis or more recent nonlinear projection techniques (Lee & Verleysen 2007), which build a new, smaller set of features, the features that are selected by feature selection methods preserve their initial meaning, potentially bringing extra information about the process being modeled (Guyon 2006). Recently, the advent of high-dimensional data has raised new challenges for feature selection methods, both from the algorithmic point of view and the conceptual point of view (Liu & Motoda 2007). The problem of feature selection is exponential in nature, and many approximate algorithms are cubic with respect to the initial number of features, which may be intractable when the dimensionality of the data is large. Furthermore, high-dimensional data are often highly redundant, and two distinct subsets of features may have very similar predictive power, which can make it difficult to identify the best subset.

The problem of feature selection has been an area of considerable research in machine learning. Feature selection is known to be particularly difficult in unsupervised learning because different subgroups of features can yield useful insights into the same dataset. In other words, many theoretically-right answers may exist for the same problem. Furthermore, designing algorithms for unsupervised feature selection is technically harder than designing algorithms for supervised feature selection because unsupervised feature selection algorithms cannot be guided by class labels. As a result, previous work attempts to discover intrinsic structures of data with heavy computation such as matrix decomposition, and require significant time to find even a single solution. This paper proposes a novel algorithm, named Explainability-based Unsupervised Feature Value Selection (EUFVS), which enables a paradigm shift in feature selection, and solves all of these problems. EUFVS requires only a few tens of milliseconds for datasets with thousands of features and instances, allowing the generation of a large number of possible solutions and select the solution with the best fit. Another important advantage of EUFVS is that it selects feature values instead of features, which can better explain phenomena in data than features. EUFVS enables a paradigm shift in feature selection. This paper explains its theoretical advantage, and also shows its applications in real experiments. In our experiments with labeled datasets, EUFVS found feature value sets that explain labels, and also detected useful relationships between feature value sets not detectable from given class labels.KeywordsFeature selectionUnsupervised learningClustering

In many data analysis tasks, one is often confronted with very high dimensional data. Feature selection techniques are designed to find the relevant feature subset of the original features which can facilitate clustering, classification and retrieval. In this paper, we consider the feature selection problem in unsupervised learning scenario, which is particularly difficult due to the absence of class labels that would guide the search for relevant information. The feature selection problem is essentially a combinatorial optimization problem which is computationally expensive. Traditional unsupervised feature selection methods address this issue by selecting the top ranked features based on certain scores computed independently for each feature. These approaches neglect the possible correlation between different features and thus can not produce an optimal feature subset. Inspired from the recent developments on manifold learning and L1-regularized models for subset selection, we propose in this paper a new approach, called Multi-Cluster Feature Selection (MCFS), for unsupervised feature selection. Specifically, we select those features such that the multi-cluster structure of the data can be best preserved. The corresponding optimization problem can be efficiently solved since it only involves a sparse eigen-problem and a L1-regularized least squares problem. Extensive experimental results over various real-life data sets have demonstrated the superiority of the proposed algorithm.

Feature selection is an important research issue in classification. As an effective global optimization technique, Particle Swarm Optimization (PSO) algorithm has been widely employed to solve feature selection problems. Population initialization is an important stage in Evolutionary Computation (EC) techniques. Therefore, in EC research field, many works focus on population initialization and it has been verified that suitable population initialization methods can significantly improve the performance of EC techniques. The efficient of PSO significantly deteriorates when being used to solve the large-scale feature selection problems. At present, only a few works have been done to enhance the performance of PSO for feature selection problems through population initialization. Meanwhile, there are many excellent traditional filter feature selection methods. Their characteristic is quick in speed but low in effect. However, they can provide a lot of useful heuristic information. Therefore, this paper focuses on improving the performance of PSO for feature selection problems by designing filter-based population initialization methods. In the proposed method, the filters are firstly used to evaluate features. Based on the obtained heuristic information, the initialization method combining the mixed initialization and the threshold selection is designed. The experiments are carried out on several datasets and the proposed initialization method is compared to some related initialization methods. Besides, the K-Nearest Neighbour (KNN) and other compatible fitness learners are used for performance assessment for feature selection. The results show that the proposed method using KNN classifier is promising to improve the performance of PSO for feature selection problems.

The explosive usage of social media produces massive amount of unlabeled and high-dimensional data. Feature selection has been proven to be effective in dealing with high-dimensional data for efficient learning and data mining. Unsupervised feature selection remains a challenging task due to the absence of label information based on which feature relevance is often assessed. The unique characteristics of social media data further complicate the already challenging problem of unsupervised feature selection, e.g., social media data is inherently linked, which makes invalid the independent and identically distributed assumption, bringing about new challenges to unsupervised feature selection algorithms. In this paper, we investigate a novel problem of feature selection for social media data in an unsupervised scenario. In particular, we analyze the differences between social media data and traditional attribute-value data, investigate how the relations extracted from linked data can be exploited to help select relevant features, and propose a novel unsupervised feature selection framework, LUFS, for linked social media data. We systematically design and conduct systemic experiments to evaluate the proposed framework on data sets from real-world social media websites. The empirical study demonstrates the effectiveness and potential of our proposed framework.

Subspace analysis is an effective technique for dimensionality reduction, which aims at finding a low-dimensional space of high-dimensional data. Fisher linear discriminant analysis (FLD) and Neural Networks are commonly used techniques of image processing and recognition. In this paper, a new scheme of image feature extraction namely, the FLD and Generalized Regression Neural Networks (GRNN) approach for image recognition is presented. Linear discriminant analysis is usually performed to investigate differences among multivariate classes, to determine which attributes discriminate between the classes, and to determine the most parsimonious way to distinguish among classes. Two-dimensional linear discriminative features and GRNN are used to perform the classification. Experiments on the image database (Face, Object and Character) demonstrate the effectiveness and feasibility of the proposed method.

Feature selection is an important data preprocessing technique in classification problems. This paper focuses on a new feature selection problem, in which sampling data of different features have different reliability degree. First, the problem is modeled as a multi-objective optimization. There two objectives should be optimized simultaneously: reliability and classifying accuracy of feature subset. Then, a multi-objective feature selection method based on particle swarm optimization, called JMOPSO, is proposed by incorporating several effective operators. Finally, experimental results suggest that the proposed JMOPSO is a highly competitive feature selection method for solving the feature selection problem with unreliable data.

In this paper we show that feature selection problem can be formulated as a model selection problem. A Bayesian framework for feature selection in unsupervised learning based on Gaussian mixture models is applied to speech recognition. In the original formulation (Figueiredo (2002)) a minimum message length criterion is used for model selection; we propose a new model selection technique based on variational Bayesian learning that shows a higher robustness to the amount of training data. Results on speech data from the TIMIT database show a high efficiency in determining feature saliency.

Feature selection is an important step of data processing. When feature selection is conducted for multi-label classification problem in online learning fashion, it is the problem of online multi-label feature selection. Online feature selection is very appropriate for some actual situations in which the data is not available in advance, the data size is very large or fast running speed is highly demanding. We propose an online multi-label feature selection algorithm in which the data set is divided into many single-label data sets, feature selection is conducted for each single-label data set and the final features are selected from the selected single-label features. As many data sets are imbalanced, we use the basic idea of cost-sensitive learning to combat it. Experiment results corroborate the performance of our algorithm on various data sets and demonstrate that the proposed algorithm can improve online classification performance on imbalanced data sets effectively.