Multiple Kernel Learning via Low-Rank Nonnegative Matrix Factorization for Classification of Hyperspectral Imagery

Recently, multiple kernel learning (MKL) methods have been developed to improve the flexibility of kernel-based learning machine. The MKL methods generally focus on determining key kernels to be preserved and their significance in optimal kernel combination. Unfortunately, computational demand of finding the optimal combination is prohibitive when the number of training samples and kernels increase rapidly, particularly for hyperspectral remote sensing data. In this paper, we address the MKL for classification in hyperspectral images by extracting the most variation from the space spanned by multiple kernels and propose a representative MKL (RMKL) algorithm. The core idea embedded in the algorithm is to determine the kernels to be preserved and their weights according to statistical significance instead of time-consuming search for optimal kernel combination. The noticeable merits of RMKL consist that it greatly reduces the computational load for searching optimal combination of basis kernels and has no limitation from strict selection of basis kernels like most MKL algorithms do; meanwhile, RMKL keeps excellent properties of MKL in terms of both good classification accuracy and interpretability. Experiments are conducted on different real hyperspectral data, and the corresponding experimental results show that RMKL algorithm provides the best performances to date among several the state-of-the-art algorithms while demonstrating satisfactory computational efficiency.

Multiple Kernel Learning (MKL) algorithms have recently demonstrated their effectiveness for classifying the data with numerous features. These algorithms aim at learning an optimal composite kernel through combining the basis kernels constructed from different features. Despite their satisfactory results, MKL algorithms assume that the basis kernels are a priori computed. Moreover, they adopt complex optimization methods to train the combination of the basis kernels, which are usually hard to solve and can only handle the binary classification problems. In this paper, a novel MKL framework was introduced in order to address all these issues. This framework optimizes a data-dependent kernel evaluation measure in order to learn both the basis kernels and their combination. The kernel evaluation measure should be able to estimate the goodness of the composite kernel for a multiclass classification problem. In this paper, we defined such a measure based on the similarity between the composite kernel and an ideal kernel. To this end, three different kernel-based similarity measures, namely kernel alignment (KA), centered kernel alignment (CKA), and Hilbert-Schmidt independence criterion (HSIC), were presented. For solving the optimization problem of the proposed MKL framework, we used the metaheuristic optimization algorithms, which in addition to being accurate algorithms can be easily implemented. The performance of the proposed framework was evaluated by classifying the features extracted from two multispectral and hyperspectral datasets. The results showed that this framework outperformed the other state-of-the-art MKL algorithms in terms of both classification accuracy and the computational time.

Kernel Methods have been successfully applied in different tasks and used on a variety of data sample sizes. Multiple Kernel Learning (MKL) and Multilayer Multiple Kernel Learning (MLMKL), as new families of kernel methods, consist of learning the optimal kernel from a set of predefined kernels by using an optimization algorithm. However, learning this optimal combination is considered to be an arduous task. Furthermore, existing algorithms often do not converge to the optimal solution (i.e., weight distribution). They achieve worse results than the simplest method, which is based on the average combination of base kernels, for some real-world applications. In this paper, we present a hybrid model that integrates two methods: Support Vector Machine (SVM) and Multiple Classifier (MC) methods. More precisely, we propose a multiple classifier framework of deep SVMs for classification tasks. We adopt the MC approach to train multiple SVMs based on multiple kernel in a multi-layer structure in order to avoid solving the complicated optimization tasks. Since the average combination of kernels gives high performance, we train multiple models with a predefined combination of kernels. Indeed, we apply a specific distribution of weights for each model. To evaluate the performance of the proposed method, we conducted an extensive set of classification experiments on a number of benchmark data sets. Experimental results show the effectiveness and efficiency of the proposed method as compared to various state-of-the-art MKL and MLMKL algorithms.

Multiple kernel learning (MKL) is a promising family of machine learning algorithms using multiple kernel functions for various challenging data mining tasks. Conventional MKL methods often formulate the problem as an optimization task of learning the optimal combinations of both kernels and classifiers, which usually results in some forms of challenging optimization tasks that are often difficult to be solved. Different from the existing MKL methods, in this paper, we investigate a boosting framework of MKL for classification tasks, i.e., we adopt boosting to solve a variant of MKL problem, which avoids solving the complicated optimization tasks. Specifically, we present a novel framework of Multiple kernel boosting (MKBoost), which applies the idea of boosting techniques to learn kernel-based classifiers with multiple kernels for classification problems. Based on the proposed framework, we propose several variants of MKBoost algorithms and extensively examine their empirical performance on a number of benchmark data sets in comparisons to various state-of-the-art MKL algorithms on classification tasks. Experimental results show that the proposed method is more effective and efficient than the existing MKL techniques.

Multiple kernel learning (MKL) is demonstrated to be flexible and effective in depicting heterogeneous data sources since MKL can introduce multiple kernels rather than a single fixed kernel into applications. However, MKL would get a high time and space complexity in contrast to single kernel learning, which is not expected in real-world applications. Meanwhile, it is known that the kernel mapping ways of MKL generally have two forms including implicit kernel mapping and empirical kernel mapping (EKM), where the latter is less attracted. In this paper, we focus on the MKL with the EKM, and propose a reduced multiple empirical kernel learning machine named RMEKLM for short. To the best of our knowledge, it is the first to reduce both time and space complexity of the MKL with EKM. Different from the existing MKL, the proposed RMEKLM adopts the Gauss Elimination technique to extract a set of feature vectors, which is validated that doing so does not lose much information of the original feature space. Then RMEKLM adopts the extracted feature vectors to span a reduced orthonormal subspace of the feature space, which is visualized in terms of the geometry structure. It can be demonstrated that the spanned subspace is isomorphic to the original feature space, which means that the dot product of two vectors in the original feature space is equal to that of the two corresponding vectors in the generated orthonormal subspace. More importantly, the proposed RMEKLM brings a simpler computation and meanwhile needs a less storage space, especially in the processing of testing. Finally, the experimental results show that RMEKLM owns a much efficient and effective performance in terms of both complexity and classification. The contributions of this paper can be given as follows: (1) by mapping the input space into an orthonormal subspace, the geometry of the generated subspace is visualized; (2) this paper first reduces both the time and space complexity of the EKM-based MKL; (3) this paper adopts the Gauss Elimination, one of the on-the-shelf techniques, to generate a basis of the original feature space, which is stable and efficient.

Kernel methods have been successfully used in many practical machine learning problems. Choosing a suitable kernel is left to the practitioner. A common way to an automatic selection of optimal kernels is to learn a linear combination of element kernels. In this paper, a novel framework of multiple kernel learning is proposed based on conditional entropy minimization criterion. For the proposed framework, three multiple kernel learning algorithms are derived. The algorithms are experimentally shown to be comparable to or outperform kernel Fisher discriminant analysis and other multiple kernel learning algorithms on benchmark data sets.

Automatic action recognition in videos is a challenging computer vision task that has become an active research area in recent years. Existing strategies usually use kernel-based learning algorithms that considers a simple combination of different features completely disregarding how such features should be integrated to fit the given problem. Since a given feature is most suitable to describe a given image/video property, the adaptive weighting of such features can improve the performance of the learning algorithm. In this paper, we investigated the use of the Multiple Kernel Learning (MKL) algorithm to adaptive search for the best linear relation among the considered features. MKL is an extension of the support vector machines (SVMs) to work with a weighted linear combination of several single kernels. This approach allows to simultaneously estimate the weights for the multiple kernels combination as well as the underlying SVM parameters. In order to prove the validity of the MKL approach, we considered a descriptor composed of multiple features aligned with dense trajectories. We experimented our approach on a database containing 36 cooking actions. Results confirm that the use of MKL improves the classification performance.

In this paper, we extend the original non-negative matrix factorization (NMF) to kernel NMF (KNMF). The advantages of KNMF over NMF are: 1) it could extract more useful features hidden in the original data through some kernel-induced nonlinear mappings; 2) it can deal with data where only relationships (similarities or dissimilarities) between objects are known; 3) it can process data with negative values by using some specific kernel functions (e.g. Gaussian). Thus, KNMF is more general than NMF. To further improve the performance of KNMF, we also propose the SpKNMF, which performs KNMF on sub-patterns of the original data. The effectiveness of the proposed algorithms is validated by extensive experiments on UCI datasets and the FERET face database.

In this paper, we propose a novel multiple kernel learning (MKL) framework to incorporate both spectral and spatial features for hyperspectral image classification, which is called multiple-structure-element nonlinear MKL (MultiSE-NMKL). In the proposed framework, multiple structure elements (MultiSEs) are employed to generate extended morphological profiles (EMPs) to present spatial-spectral information. In order to better mine interscale and interstructure similarity among EMPs, a nonlinear MKL (NMKL) is introduced to learn an optimal combined kernel from the predefined linear base kernels. We integrate this NMKL with support vector machines (SVMs) and reduce the min-max problem to a simple minimization problem. The optimal weight for each kernel matrix is then solved by a projection-based gradient descent algorithm. The advantages of using nonlinear combination of base kernels and multiSE-based EMP are that similarity information generated from the nonlinear interaction of different kernels is fully exploited, and the discriminability of the classes of interest is deeply enhanced. Experiments are conducted on three real hyperspectral data sets. The experimental results show that the proposed method achieves better performance for hyperspectral image classification, compared with several state-of-the-art algorithms. The MultiSE EMPs can provide much higher classification accuracy than using a single-SE EMP.

Nonnegative matrix factorization with combined kernels for small data representation

Kernel methods have become a standard solution for a large number of data analysis, and extensively utilized in the field of signal processing including analysis of speech, image, time series, and DNA sequences. The main difficulty in applying the kernel method is in designing the appropriate kernel for the specific data, and multiple kernel learning (MKL) is one of the principled approaches for kernel design problem. In this paper, a novel multiple kernel learning method based on the notion of Gaussianity evaluated by entropy power inequality is proposed. The notable characteristics of the proposed method are in utilizing the entropy power inequality for kernel learning, and in realizing an MKL algorithm which only optimizes the kernel combination coefficients, while the conventional methods need optimizing both combination coefficients and classifier parameters. The proposed MKL algorithm is experimentally shown to have good classification accuracy.

In this paper, we proposed an incremental kernel non-negative matrix factorization (IKNMF) to reduce the computing scale in hyperspectral unmixing. Kernel non-negative matrix factorization (KNMF) is an extended non-negative matrix factorization (NMF) able to capture nonlinear dependency features in data matrix through kernel functions. In KNMF algorithm, the size of kernel matrices is closely associated with the input data scale. To reduce calculation and storage of large matrices, we extend KNMF by introducing partition matrix theory. The decomposition results of data matrices are derived from smaller scale matrices incrementally. Experiments are conducted on synthetic hyperspectral images with multiple sizes, and the experimental results show that the proposed algorithm have effect in saving calculation and memory resource without degrading the unmixing performance.

A lot of image processing research works focus on natural images, such as in classification, clustering, and the research on the recognition of artworks (such as oil paintings), from feature extraction to classifier design, is relatively few. This paper focuses on oil painter recognition and tries to find the mobile application to recognize the painter. This paper proposes a cluster multiple kernel learning algorithm, which extracts oil painting features from three aspects: color, texture, and spatial layout, and generates multiple candidate kernels with different kernel functions. With the results of clustering numerous candidate kernels, we selected the sub-kernels with better classification performance, and use the traditional multiple kernel learning algorithm to carry out the multi-feature fusion classification. The algorithm achieves a better result on the Painting91 than using traditional multiple kernel learning directly.

Multi-class kernel margin maximization for kernel learning

Multiple kernel learning (MKL) is a principled way for kernel fusion for various learning tasks such as classification, clustering, and dimensionality reduction. The least absolute shrinkage and selection operator (Lasso) allows computationally efficient feature selection based on the linear dependence between input features and output values. In this paper, we develop a novel MKL model based on a nonlinear Lasso, that is, the Hilbert–Schmidt independence criterion (HSIC) Lasso. In the proposed model, we first propose the HSIC Lasso‐based MKL formulation, which has a clear statistical interpretation that minimum redundant kernels with maximum dependence on output labels are found and combined, and also that the global optimal solution can be computed efficiently by solving a Lasso optimization problem. After the optimal kernel is obtained, the support vector machine (SVM) is used to select the prediction hypothesis. It is evident that the proposed MKL is a two‐stage kernel learning approach. Extensive experiments on real‐world datasets from the UCI benchmark repository validate the superiority of the proposed model in terms of prediction accuracy. © 2018 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.