Multiple Kernel Dimensionality Reduction via Ratio-Trace and Marginal Fisher Analysis

Multiple kernel learning (MKL) is demonstrated to be flexible and effective in depicting heterogeneous data sources since MKL can introduce multiple kernels rather than a single fixed kernel into applications. However, MKL would get a high time and space complexity in contrast to single kernel learning, which is not expected in real-world applications. Meanwhile, it is known that the kernel mapping ways of MKL generally have two forms including implicit kernel mapping and empirical kernel mapping (EKM), where the latter is less attracted. In this paper, we focus on the MKL with the EKM, and propose a reduced multiple empirical kernel learning machine named RMEKLM for short. To the best of our knowledge, it is the first to reduce both time and space complexity of the MKL with EKM. Different from the existing MKL, the proposed RMEKLM adopts the Gauss Elimination technique to extract a set of feature vectors, which is validated that doing so does not lose much information of the original feature space. Then RMEKLM adopts the extracted feature vectors to span a reduced orthonormal subspace of the feature space, which is visualized in terms of the geometry structure. It can be demonstrated that the spanned subspace is isomorphic to the original feature space, which means that the dot product of two vectors in the original feature space is equal to that of the two corresponding vectors in the generated orthonormal subspace. More importantly, the proposed RMEKLM brings a simpler computation and meanwhile needs a less storage space, especially in the processing of testing. Finally, the experimental results show that RMEKLM owns a much efficient and effective performance in terms of both complexity and classification. The contributions of this paper can be given as follows: (1) by mapping the input space into an orthonormal subspace, the geometry of the generated subspace is visualized; (2) this paper first reduces both the time and space complexity of the EKM-based MKL; (3) this paper adopts the Gauss Elimination, one of the on-the-shelf techniques, to generate a basis of the original feature space, which is stable and efficient.

Multiple kernel learning (MKL) is a promising family of machine learning algorithms using multiple kernel functions for various challenging data mining tasks. Conventional MKL methods often formulate the problem as an optimization task of learning the optimal combinations of both kernels and classifiers, which usually results in some forms of challenging optimization tasks that are often difficult to be solved. Different from the existing MKL methods, in this paper, we investigate a boosting framework of MKL for classification tasks, i.e., we adopt boosting to solve a variant of MKL problem, which avoids solving the complicated optimization tasks. Specifically, we present a novel framework of Multiple kernel boosting (MKBoost), which applies the idea of boosting techniques to learn kernel-based classifiers with multiple kernels for classification problems. Based on the proposed framework, we propose several variants of MKBoost algorithms and extensively examine their empirical performance on a number of benchmark data sets in comparisons to various state-of-the-art MKL algorithms on classification tasks. Experimental results show that the proposed method is more effective and efficient than the existing MKL techniques.

Kernel Methods have been successfully applied in different tasks and used on a variety of data sample sizes. Multiple Kernel Learning (MKL) and Multilayer Multiple Kernel Learning (MLMKL), as new families of kernel methods, consist of learning the optimal kernel from a set of predefined kernels by using an optimization algorithm. However, learning this optimal combination is considered to be an arduous task. Furthermore, existing algorithms often do not converge to the optimal solution (i.e., weight distribution). They achieve worse results than the simplest method, which is based on the average combination of base kernels, for some real-world applications. In this paper, we present a hybrid model that integrates two methods: Support Vector Machine (SVM) and Multiple Classifier (MC) methods. More precisely, we propose a multiple classifier framework of deep SVMs for classification tasks. We adopt the MC approach to train multiple SVMs based on multiple kernel in a multi-layer structure in order to avoid solving the complicated optimization tasks. Since the average combination of kernels gives high performance, we train multiple models with a predefined combination of kernels. Indeed, we apply a specific distribution of weights for each model. To evaluate the performance of the proposed method, we conducted an extensive set of classification experiments on a number of benchmark data sets. Experimental results show the effectiveness and efficiency of the proposed method as compared to various state-of-the-art MKL and MLMKL algorithms.

Recently, multiple kernel learning (MKL) methods have been developed to improve the flexibility of kernel-based learning machine. The MKL methods generally focus on determining key kernels to be preserved and their significance in optimal kernel combination. Unfortunately, computational demand of finding the optimal combination is prohibitive when the number of training samples and kernels increase rapidly, particularly for hyperspectral remote sensing data. In this paper, we address the MKL for classification in hyperspectral images by extracting the most variation from the space spanned by multiple kernels and propose a representative MKL (RMKL) algorithm. The core idea embedded in the algorithm is to determine the kernels to be preserved and their weights according to statistical significance instead of time-consuming search for optimal kernel combination. The noticeable merits of RMKL consist that it greatly reduces the computational load for searching optimal combination of basis kernels and has no limitation from strict selection of basis kernels like most MKL algorithms do; meanwhile, RMKL keeps excellent properties of MKL in terms of both good classification accuracy and interpretability. Experiments are conducted on different real hyperspectral data, and the corresponding experimental results show that RMKL algorithm provides the best performances to date among several the state-of-the-art algorithms while demonstrating satisfactory computational efficiency.

Multiple Kernel Learning (MKL) algorithms have recently demonstrated their effectiveness for classifying the data with numerous features. These algorithms aim at learning an optimal composite kernel through combining the basis kernels constructed from different features. Despite their satisfactory results, MKL algorithms assume that the basis kernels are a priori computed. Moreover, they adopt complex optimization methods to train the combination of the basis kernels, which are usually hard to solve and can only handle the binary classification problems. In this paper, a novel MKL framework was introduced in order to address all these issues. This framework optimizes a data-dependent kernel evaluation measure in order to learn both the basis kernels and their combination. The kernel evaluation measure should be able to estimate the goodness of the composite kernel for a multiclass classification problem. In this paper, we defined such a measure based on the similarity between the composite kernel and an ideal kernel. To this end, three different kernel-based similarity measures, namely kernel alignment (KA), centered kernel alignment (CKA), and Hilbert-Schmidt independence criterion (HSIC), were presented. For solving the optimization problem of the proposed MKL framework, we used the metaheuristic optimization algorithms, which in addition to being accurate algorithms can be easily implemented. The performance of the proposed framework was evaluated by classifying the features extracted from two multispectral and hyperspectral datasets. The results showed that this framework outperformed the other state-of-the-art MKL algorithms in terms of both classification accuracy and the computational time.

As a rapid developing research topic in the machine vision field, image-based hand posture recognition has the potential to be an efficient and intuitive tool of human-computer interaction. For improving the accuracy of multi-class hand postures and extending the algorithm generalization,a novel hand posture recognition method is proposed by integrating the multiple image features and multiple kernels learning support vector machine(SVM). Firstly, three types of feature descriptors are extracted to describe the characteristics of a hand posture image. Shape context descriptor represents distribution characteristics of the edge points of the hand posture image. Pyramid histogram of oriented gradient describes characteristics of local and global shape effectively. The Bag of Feature(BOF) algorithm describes the surface texture characteristics of the posture image. Secondly, the Chamfer kernel and histogram intersection kernel are rebuilt to obtain the basis kernels of the features. And the combined kernel is constructed by weighting the basis kernels.So the heterogeneous features fusion realizes. Finally, the classification model and optimal fusion weights are calculated by using multiple kernels learning algorithm. The unknown category posture can be recognized by the trained multiple kernels of SVM. Experiments on Jochen Triesch's hand posture dataset demonstrate that the proposed method obtains higher recognition rate than the traditional single-kernel classifier and other recent methods.

In this paper, a novel multiple kernel learning (MKL) algorithm is proposed for the classification of hyperspectral images. The proposed MKL algorithm adopts a two-step strategy to learn a multiple kernel machine. In the first step, unsupervised learning is carried out to learn a combined kernel from the predefined base kernels. In our algorithms, low-rank nonnegative matrix factorization (NMF) is used to carry out the unsupervised learning and learn an optimal combined kernel. Furthermore, the kernel NMF (KNMF) is introduced to substitute NMF for enhancing the ability of the unsupervised learning with the predefined base kernels. In the second step, the optimal kernel is embedded into the standard optimization routine of support vector machine (SVM). In addition, we address a major challenge in hyperspectral data classification, i.e., using very few labeled samples in a high-dimensional space. Experiments are conducted on three real hyperspectral datasets, and the experimental results show that the proposed algorithms, especially for KNMF-based MKL, achieve the outstanding performance for hyperspectral image classification with few labeled samples when compared with several state-of-the-art algorithms.

Relevance and irrelevance graph based marginal Fisher analysis for image search reranking

Multiple kernel learning is a paradigm which employs a properly constructed chain of kernel functions able to simultaneously analyse different data or different representations of the same data. In this paper, we propose an hybrid classification system based on a linear combination of multiple kernels defined over multiple dissimilarity spaces. The core of the training procedure is the joint optimisation of kernel weights and representatives selection in the dissimilarity spaces. This equips the system with a two-fold knowledge discovery phase: by analysing the weights, it is possible to check which representations are more suitable for solving the classification problem, whereas the pivotal patterns selected as representatives can give further insights on the modelled system, possibly with the help of field-experts. The proposed classification system is tested on real proteomic data in order to predict proteins’ functional role starting from their folded structure: specifically, a set of eight representations are drawn from the graph-based protein folded description. The proposed multiple kernel-based system has also been benchmarked against a clustering-based classification system also able to exploit multiple dissimilarities simultaneously. Computational results show remarkable classification capabilities and the knowledge discovery analysis is in line with current biological knowledge, suggesting the reliability of the proposed system.

Automatic action recognition in videos is a challenging computer vision task that has become an active research area in recent years. Existing strategies usually use kernel-based learning algorithms that considers a simple combination of different features completely disregarding how such features should be integrated to fit the given problem. Since a given feature is most suitable to describe a given image/video property, the adaptive weighting of such features can improve the performance of the learning algorithm. In this paper, we investigated the use of the Multiple Kernel Learning (MKL) algorithm to adaptive search for the best linear relation among the considered features. MKL is an extension of the support vector machines (SVMs) to work with a weighted linear combination of several single kernels. This approach allows to simultaneously estimate the weights for the multiple kernels combination as well as the underlying SVM parameters. In order to prove the validity of the MKL approach, we considered a descriptor composed of multiple features aligned with dense trajectories. We experimented our approach on a database containing 36 cooking actions. Results confirm that the use of MKL improves the classification performance.

Learning linear combinations of multiple kernels is an appealing strategy when the right choice of features is unknown. Previous approaches to multiple kernel learning (MKL) promote sparse kernel c...

Elastic Multiple Kernel Learning

In this paper, we propose a novel multiple kernel learning (MKL) framework to incorporate both spectral and spatial features for hyperspectral image classification, which is called multiple-structure-element nonlinear MKL (MultiSE-NMKL). In the proposed framework, multiple structure elements (MultiSEs) are employed to generate extended morphological profiles (EMPs) to present spatial-spectral information. In order to better mine interscale and interstructure similarity among EMPs, a nonlinear MKL (NMKL) is introduced to learn an optimal combined kernel from the predefined linear base kernels. We integrate this NMKL with support vector machines (SVMs) and reduce the min-max problem to a simple minimization problem. The optimal weight for each kernel matrix is then solved by a projection-based gradient descent algorithm. The advantages of using nonlinear combination of base kernels and multiSE-based EMP are that similarity information generated from the nonlinear interaction of different kernels is fully exploited, and the discriminability of the classes of interest is deeply enhanced. Experiments are conducted on three real hyperspectral data sets. The experimental results show that the proposed method achieves better performance for hyperspectral image classification, compared with several state-of-the-art algorithms. The MultiSE EMPs can provide much higher classification accuracy than using a single-SE EMP.

Kernel methods have been successfully used in many practical machine learning problems. Choosing a suitable kernel is left to the practitioner. A common way to an automatic selection of optimal kernels is to learn a linear combination of element kernels. In this paper, a novel framework of multiple kernel learning is proposed based on conditional entropy minimization criterion. For the proposed framework, three multiple kernel learning algorithms are derived. The algorithms are experimentally shown to be comparable to or outperform kernel Fisher discriminant analysis and other multiple kernel learning algorithms on benchmark data sets.

Multiple kernel learning (MKL) is a principled way for kernel fusion for various learning tasks such as classification, clustering, and dimensionality reduction. The least absolute shrinkage and selection operator (Lasso) allows computationally efficient feature selection based on the linear dependence between input features and output values. In this paper, we develop a novel MKL model based on a nonlinear Lasso, that is, the Hilbert–Schmidt independence criterion (HSIC) Lasso. In the proposed model, we first propose the HSIC Lasso‐based MKL formulation, which has a clear statistical interpretation that minimum redundant kernels with maximum dependence on output labels are found and combined, and also that the global optimal solution can be computed efficiently by solving a Lasso optimization problem. After the optimal kernel is obtained, the support vector machine (SVM) is used to select the prediction hypothesis. It is evident that the proposed MKL is a two‐stage kernel learning approach. Extensive experiments on real‐world datasets from the UCI benchmark repository validate the superiority of the proposed model in terms of prediction accuracy. © 2018 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.