Greedy kernel PCA for training data reduction and nonlinear feature extraction in classification

Kernel PCA is an efficient method for nonlinear feature extraction. We address two issues in kernel PCA based feature extraction and classification. First, it extracts features without utilizing sample label information. Second, it does not provide a practical means to choose the dimensionality for principal subspace. In this paper, one kind of side-information is incorporated into kernel PCA to solve the first problem. And a complete probabilistic density function is estimated in kernel space so that the choice of dimensionality for principal subspace becomes less important. The proposed model is named probabilistic kernel feature subspace (PKFS). Experiments show that it achieves promising performance and outperforms many other algorithms in classification.

Kernel discriminant analysis (KDA) and the kernel principal component analysis (KPCA), which are the extension of the linear discriminant analysis (LDA) and the principal component analysis (PCA), respectively, from linear domain to nonlinear domain via the kernel trick, are two very popular nonlinear feature extraction methods. In this paper, we present a new feature extraction algorithm by combing KDA and KPCA, and then apply it to the face recognition task. The experimental results on Yale face dataset show that the proposed method can significantly improve the performance both KDA and KPCA.

Principal component analysis (PCA) has been proven to be an efficient method in dimensionality reduction, feature extraction and pattern recognition. Kernel principal component analysis (KPCA) can be considered as a natural nonlinear generalization of PCA, which performs linear PCA in a high dimensional space implicitly by using kernel trick. However, both conventional PCA and KPCA suffer from the deficiency of being sensitive to outliers. Existing robust KPCA has to eigen-decompose the gram matrix directly in each step and is much more computationally infeasible due to the large size of the matrix when the number of training samples is large. By extending existing robust PCA algorithm using kernel methods, we present a novel robust adaptive algorithm for calculating the kernel principal components. The proposed method not only preserves the characteristic of capturing underlying nonlinear structure of KPCA but also is robust against outliers by restraining the effect of outlying samples. Compared with existing robust KPCA methods, our method is performed without having to store the kernel matrix, which can reduce significantly the storage burden. In addition, our method shows the potential of expansibility to the incremental learning version. Experimental results on synthetic data indicate that our improved algorithm is effective and promising.

This paper presents a novel algorithm - robust kernel principal component analysis (robust KPCA), on the basis of the research of kernel principal component analysis (KPCA) and robust principal component analysis (RPCA). First, this algorithm sets the radius of the images of the training samples in the feature space using kernel tricks, then determines whether the samples are outliers or not, and finally analyzes the training samples which have eliminated the outliers using KPCA algorithm. The improved KPCA algorithm not only retains the non-linearity property of KPCA algorithm but also gets better robustness. Because the effects of outliers are eliminated, robust KPCA algorithm gets higher accuracy than KPCA algorithm for data analysis. The simulation experiments show that the robust KPCA algorithm developed is better than the KPCA algorithm.

The inherent physical characteristics of many real-life phenomena, including biological and physiological aspects, require adapted nonlinear tools. Moreover, the additive nature in some situations involve solutions expressed as positive combinations of data. In this paper, we propose a nonlinear feature extraction method, with a non-negativity constraint. To this end, the kernel principal component analysis is considered to define the most relevant features in the reproducing kernel Hilbert space. These features are the nonlinear principal components with high-order correlations between input variables. A pre-image technique is required to get back to the input space. With a non-negative constraint, we show that one can solve the pre-image problem efficiently, using a simple iterative scheme. Furthermore, the constrained solution contributes to the stability of the algorithm. Experimental results on event-related potentials (ERP) illustrate the efficiency of the proposed method.

A deep kernel method for lithofacies identification using conventional well logs

The identification of between-group differences and changes in gait mechanics are useful for injury prevention. Previous studies suggest the differences in gait biomechanical variables may interact in a complex non-linear fashion rather than a simple linear fashion. A non-linear multivariate analysis technique is therefore necessary to unravel the inherent structure in the gait data. Kernel principal component analysis (KPCA) is an extension of principal component analysis (PCA), the most widely used method in this field, and can provide inside into non-linear relationships of the variables of interest. Despite the growing use of PCA in running gait research, no prior studies have evaluated the utility of KPCA in determining the between-group differences in running gait patterns. Therefore, the objective of this study was to compare the performance of KPCA and PCA in identifying differences between male and female runners and as well between young and older runners in running gait kinematics. Running kinematic data were analysed on a gender group (female, n = 50; male, n = 50) and an age group (young, n = 50; older, n = 50) for discrete variables and waveform data. The following key results were obtained: (1) the performance of the KPCA for the identification of gender and age differences in running gait kinematics increased as compared to using the linear PCA; (2) the Gaussian function often performed better than the polynomial function for these two experiments; (3) there was no consistent optimal value of either the width σ of the Gaussian kernel or the degree d of the polynomial kernel for different data types and experiments. These results suggest that non-linear features extracted by KPCA could be potentially useful in identifying and discriminating between-group differences and changes in running gait patterns, and could provide useful information about the intrinsic non-linear dynamics of running movement.

Network intrusion detection identifies malicious activity in the network by analyzing the behavior of network traffic. As an important part of network intrusion detection, feature extraction plays a crucial role in improving the performance of intrusion detection. This research proposes a novel secondary feature extraction method called L-KPCA based on the Liner Discriminant Analysis (LDA) and Kernel Principal Component Analysis (KPCA), to provide efficient features for network intrusion detection. While maintaining the effectiveness of processing nonlinear data in network traffic, the use of LDA effectively compensates for the problem that KPCA only focuses on the analysis of features in terms of variance and ignores the performance of features in terms of mean. Extensive experimental results verify that the use of the proposed, L-KPCA can make the intrusion detection classification model perform better in terms of recognition accuracy and recall.

An optimized feature extraction algorithm for abnormal network traffic detection

A kernel based nonlinear subspace projection (KNSP) method is proposed for reduction of hyperspectral image dimensionality. This method involves three steps: subspace partition of full data space, feature extraction based on kernel principal component analysis (KPCA) in subspace and feature selection based on class separability criterion. The main merit of the proposed method is that it is more suitable for feature extraction than linear principal component analysis (PCA) and segmented principal component: transformation (SPCT), in particular, when hyperspectral data have nonlinear characteristics. In order to testify the effectiveness of the KNSP method for reduction of hyperspectral image dimensionality, hyperspectral image classification is performed on AVIRIS data. Experimental results show that when the hyperspectral dimensionality is reduced to a few features, the average classification accuracy of the new method is higher than those of PCA and SPCT methods.

Traditional face recognition methods such as Principal Components Analysis(PCA), Independent Component Analysis(ICA) and Linear Discriminant Analysis(LDA) are linear discriminant methods, but in the real situation, a lot of problems can't be linear discriminated; therefore, researchers proposed face recognition method based on kernel techniques which can transform the nonlinear problem of inputting space into the linear problem of high dimensional space. In this paper, we propose a recognition method based on kernel function which combines kernel Fisher Discriminant Analysis(KFDA) with kernel Principle Components Analysis(KPCA) and use typical ORL(Olivetti Research Laboratory) face database as our experimental database. There are four key steps: constructing feature subspace, image projection, feature extraction and image recognition. We found that the recognition accuracy has been greatly improved by using nonlinear identification method and combined feature extraction methods. We use MATLAB software as the platform, and use the GUI to demonstrate the process of face recognition in order to achieving human-computer interaction and making the process and result more intuitive.

In kernel methods, choosing a suitable kernel is in dispensable for favorable results. No well-founded methods, however, have been established in general for unsupervised learning. We focus on kernel Princ ipal Component Analysis (kernel PCA), which is a nonlinear extension of principal component analysis and ha s been used electively for extracting nonlinear featu res and reducing dimensionality. As a kernel method , kernel PCA also suffers from the problem of kernel choice. Although cross-validation is a popular meth od for choosing hyperparameters, it is not applicable straightforwardly to choose a kernel in kernel PCA because of the incomparable norms given by differen t kernels. It is important, thus, to develop a well founded method for choosing a kernel in kernel PCA. This study proposes a method for choosing hyperparameters in kernel PCA (kernel and the number of components) based on cross-validation for the comparable reconstruction errors of pre-images in t he original space. The experimental results on synthesized and real-world datasets demonstrate tha t the proposed method successfully selects an appropriate kernel and the number of components in kernel PCA in terms of visualization and classification errors on the principal components. The results imply that the proposed method enables automatic design of hyperparameters in kernel PCA.

In this study, a subspace detection technique is developed using a hybrid approach which combines both feature extraction and feature selection for the task of hyperspectral image classification. The proposed approach applies Kernel Principal Component Analysis (KPCA) at the first step, then feature selection from the KPCA images is accomplished by combining the KPCA score with a Jeffries-Matusita (JM) distance based ranking score. Experimental analysis is carried out on a hyperspectral image acquired by the AVIRIS sensor and the results show the advantage of the proposed approach in terms of classification accuracy in the tested cases.

Aimed at the nonlinear properties of analog circuits features , kernel Principal Component Analysis(KPCA) combined with Support Vector Machine(SVM) based diagnostic method for faults of analog circuit was proposed in this paper. Here, uses KPCA to reduce the irrelevant features. The KPCA method is used to extract the initial features. The SVM is then applied to the circuit after features extraction. It can be used as a new idea for Fault Diagnosis of Analog Circuits. To better verify the superiority of the proposed method, KPCA+SVM classification results were compared with the results of PCA+SVM. It is concluded that KPCA+SVM classifier achieved a better performance than PCA+SVM in the terms of the accuracy.

The random forest (RF) classifier, which is a combination of tree predictors, is one of the most powerful classification algorithms that has been recently applied for fault detection and diagnosis (FDD) of industrial processes. However, RF is still suffering from some limitations such as the noncorrelation between variables. These limitations are due to the direct use of variables measured at nodes and therefore the only use of static information from the process data. Thus, this article proposes two enhanced RF classifiers, namely the Euclidean distance based reduced kernel RF (RK-RF <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ED</sub> ) and K-means clustering based reduced kernel RF (RK-RF <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Kmeans</sub> ), for FDD. Based on the kernel principal component analysis, the proposed classifiers consist of two main stages: feature extraction and selection, and fault classification. In the first stage, the number of observations in the training data set is reduced using two methods: the first method consists of using the Euclidean distance as dissimilarity metric so that only one measurement is kept in case of redundancy between samples. The second method aims at reducing the amount of the training data based on the K-means clustering technique. Once the characteristics of the process are extracted, the most sensitive features are selected. During the second phase, the selected features are fed to an RF classifier. An emulated grid-connected PV system is used to validate the performance of the proposed RK-RF <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ED</sub> and RK-RF <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Kmeans</sub> classifiers. The presented results confirm the high classification accuracy of the developed techniques with low computation time.