Nonlinear feature extraction using kernel principal component analysis with non-negative pre-image

As a valid statistical tool, principal component analysis (PCA) has been widely used in industrial process monitoring. But due to its intrinsic linear character, it performs badly in nonlinear process monitoring. Kernel PCA (KPCA) can extract useful information in nonlinear data. However KPCA-based monitoring is not suitable for on-line monitoring because of large calculation and much memory occupation. The paper introduces an on-line monitoring method based on KPCA and neural network (NN), where KPCA is used to extract nonlinear principal components (PCs) and then NN approximates the relationship between process data and nonlinear PCs. We can obtain nonlinear PCs by NN to compute the monitoring indices and then achieve the on-line monitoring. The case study shows the validity of the method.KeywordsNeural NetworkKernel Principal Component AnalysisMonitoring IndexMemory OccupationComplex Industrial ProcessThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Kernel principal component analysis (KPCA) as a powerful nonlinear feature extraction method has proven as a preprocessing step for classification algorithm. A face recognition approach based on KPCA and genetic algorithms (GAs) is proposed. By the use of the polynomial functions as a kernel function in KPCA, the high order relationships can be utilized and the nonlinear principal components can be obtained. After we obtain the nonlinear principal components, we use GAs to select the optimal feature set for classification. At the recognition stage, we employed linear support vector machines (SVM) as classifier for the recognition tasks. Two face databases were used to test our algorithm and higher recognition rates were obtained which show that our algorithm is effective.

The aim of this paper applies greedy kernel principal component analysis (greedy kernel PCA) to deal with training data reduction and nonlinear feature extraction in classification. Kernel PCA is a nonlinear extension of linear PCA. It shows a powerful nonlinear feature extraction technique via kernel trick. A disadvantage of kernel PCA, however, is that the storage of training data in terms of the dot products is too expensive since the size of kernel matrix increases quadratically with the number of training data. So, a more efficient feature extraction method, greedy kernel PCA, is proposed to reduce training data and nonlinear feature extraction for classification. The reduced set method aims to find a new kernel expansion and well approximates the original training data. Simulation results show both kernel PCA and greedy kernel PCA are more superior to linear PCA in feature extraction. Greedy kernel PCA will tend towards kernel PCA feature extraction as more percentage of training data is included in the reduced set, whilst greedy kernel PCA results in lower evaluation cost due to the reduced training set. The experiments show also that greedy kernel PCA can significantly reduce the complexity while retaining their accuracy in classification.

Principal Components Analysis (PCA) is used to classify the running condition of a machine by means of projecting the original data to the Principal Components space. However, if the data are concentrated in a nonlinear subspace, PCA will fail to work well. Kernel Principal Components Analysis (KPCA) transforms the input data from the original input space into a higher dimensional feature space with the nonlinear mapping, and then uses the nonlinear principal components to realize the classification. In this paper a case of gear fault diagnosis was studied with KPCA. The characteristic values of frequent domain from vibration signals of the gearbox under the running condition were extracted, and the KPCA method was used to classify gear crack fault. The result shows that KPCA is more effective to distinguish the state of the gear and more suitable to diagnose the gear faults in early stage.

We present a novel data embedding that significantly reduces the estimation error of locality sensitive hashing (LSH) technique when used in reproducing kernel Hilbert space (RKHS). Efficient and accurate kernel approximation techniques either involve the kernel principal component analysis (KPCA) approach or the Nystrom approximation method. In this work we show that extant LSH methods in this space suffer from a bias problem, that moreover is difficult to estimate apriori. Consequently, the LSH estimate of a kernel is different from that of the KPCA/Nystrom approximation. We provide theoretical rationale for this bias, which is also confirmed empirically. We propose an LSH algorithm that can reduce this bias and consequently our approach can match the KPCA or the Nystrom methods’ estimation accuracy while retaining the traditional benefits of LSH. We evaluate our algorithm on a wide range of realworld image datasets (for which kernels are known to perform well) and show the efficacy of our algorithm using a variety of principled evaluations including mean estimation error, KL divergence and the Kolmogorov-Smirnov test.

This paper presents a novel algorithm - robust kernel principal component analysis (robust KPCA), on the basis of the research of kernel principal component analysis (KPCA) and robust principal component analysis (RPCA). First, this algorithm sets the radius of the images of the training samples in the feature space using kernel tricks, then determines whether the samples are outliers or not, and finally analyzes the training samples which have eliminated the outliers using KPCA algorithm. The improved KPCA algorithm not only retains the non-linearity property of KPCA algorithm but also gets better robustness. Because the effects of outliers are eliminated, robust KPCA algorithm gets higher accuracy than KPCA algorithm for data analysis. The simulation experiments show that the robust KPCA algorithm developed is better than the KPCA algorithm.

Aimed at the non-stationary characteristics of rolling bearing vibration signal,a fault diagnosis method was proposed based on local-wave method and KPCA(kernel principal component analysis)-LSSVM(least squares support vector machine).Firstly,local wave decomposition was used to decompose rolling bearing vibration signal into several intrinsic mode function(IMF),whose feature energy,singular values and AR model parameters were computed as initial feature vectors.Secondly,ini tial feature vectors were mapped into a higher-dimensional space with KPCA,and the kemel principal components were extracted as new feature vectors,which used as the input of LSSVM for fault classification.The experimental results show the KPCA-LSSVM method improves LSSVM's classification performance by KPCA obtaining additional discriminative information,and has better generalization than direct LSSVM method,and can identify rolling bearing fault patterns more accurately.

A vibrating screen efficiency prediction modeling method based on autoregressive (AR) model and least square support vector machine (LS-SVM) was proposed. The vibration signals of a self-synchronized vibrating screen were collected to establish the AR model. Nonlinear principal components of the signals were extracted by the kernel principal component analysis (KPCA), followed by the regression model reconstruction using LS-SVM to accomplish reduced complexity of the prediction model from AR coefficients and improved generalization capacity and learning speed. The results show that the model predictions are consistent with the experimental data, which indicates that the modeling method is applicable and feasible in adjusting the design and process parameters of vibrating screens. Furthermore, the work condition monitoring method in the experiment is feasible for faults diagnosis of mechanical equipment.

Fault discriminant enhanced kernel principal component analysis incorporating prior fault information for monitoring nonlinear processes

Principal component analysis (PCA) is a widely used unsupervised method for dimensionality reduction. The kernelized version is called kernel principal component analysis (KPCA), which can capture the nonlinear data structure. KPCA is derived from the Gram matrix, which is not robust when outliers exist in the data. This may yield the principal axis in the feature space deviated by outliers, leading to misinterpretation of the principal components. In this paper, we propose a robust method for KPCA with a reformulation in Euclidean space to construct a robust KPCA method, where an error measurement is introduced into the loss function, and l 2,1 -regularization is added to the loss function. The idea of l 2,1 -regularization of the proposed method is motivated by sparse PCA via variable projection. However, because orthogonality is not satisfied in the proposed method, orthonormal bases are obtained by using the Gram-Schmidt orthonormalization process. In the experiments, a toy example and real data are used for outlier detection to verify the method's performance and effectiveness. In the toy example, the proposed method reduces the influence of outliers and detects more outliers than KPCA. For the real data, the proposed method improves detection in comparison to other existing methods.

Kernel principal component analysis (KPCA) has been effectively applied as an unsupervised non-linear feature extractor in many machine learning applications and suggested for various data stream classification tasks requiring a nonlinear transformation scheme to reduce dimensions. However, the dimensionality reduction ability is restricted because of KPCA’s high time complexity. So the practicality of KPCA on large datasets is rare. Therefore in this paper, we proposes a novel kind of incremental kernel principal component analysis algorithm: Data characteristic extraction based on IPCA algorithm –DCEIPCA, which allows efficient processing of large datasets and overcome the insufficient of KPCA. On the basis of DCEIPCA, we propose Classification expert system (CES) for intrusion detection system. Extensive experiments on KDDcup99 datasets confirm the superiority of Intrusion detection system based on CES(IDSCES) over other recent Intrusion detection system[4-11].

In kernel methods, choosing a suitable kernel is in dispensable for favorable results. No well-founded methods, however, have been established in general for unsupervised learning. We focus on kernel Princ ipal Component Analysis (kernel PCA), which is a nonlinear extension of principal component analysis and ha s been used electively for extracting nonlinear featu res and reducing dimensionality. As a kernel method , kernel PCA also suffers from the problem of kernel choice. Although cross-validation is a popular meth od for choosing hyperparameters, it is not applicable straightforwardly to choose a kernel in kernel PCA because of the incomparable norms given by differen t kernels. It is important, thus, to develop a well founded method for choosing a kernel in kernel PCA. This study proposes a method for choosing hyperparameters in kernel PCA (kernel and the number of components) based on cross-validation for the comparable reconstruction errors of pre-images in t he original space. The experimental results on synthesized and real-world datasets demonstrate tha t the proposed method successfully selects an appropriate kernel and the number of components in kernel PCA in terms of visualization and classification errors on the principal components. The results imply that the proposed method enables automatic design of hyperparameters in kernel PCA.

Recently, the nonlinear projection trick (NPT) was introduced enabling direct computation of coordinates of samples in a reproducing kernel Hilbert space. With NPT, any machine learning algorithm can be extended to a kernel version without relying on the so called kernel trick. However, NPT is inherently difficult to be implemented incrementally because an ever increasing kernel matrix should be treated as additional training samples are introduced. In this paper, an incremental version of the NPT (INPT) is proposed based on the observation that the centerization step in NPT is unnecessary. Because the proposed INPT does not change the coordinates of the old data, the coordinates obtained by INPT can directly be used in any incremental methods to implement a kernel version of the incremental methods. The effectiveness of the INPT is shown by applying it to implement incremental versions of kernel methods such as, kernel singular value decomposition, kernel principal component analysis, and kernel discriminant analysis which are utilized for problems of kernel matrix reconstruction, letter classification, and face image retrieval, respectively.

This work derives and evaluates a method for Blind Source Extraction (BSE) in a reproducing kernel Hilbert space (RKHS) framework. The a priori information about the autocorrelation function of the signal under study is translated in a linear transformation of the Gram matrix of the transformed data in Hilbert space. Our method proved to be more robust than methods presented in the literature of BSE with respect to ambiguities in the available a priori information of the signal to be extracted. The approach here introduced can also be seen as a generalization of Kernel principal component analysis (KPCA) to analyze autocorrelation matrices at specific time lags.

The identification of between-group differences and changes in gait mechanics are useful for injury prevention. Previous studies suggest the differences in gait biomechanical variables may interact in a complex non-linear fashion rather than a simple linear fashion. A non-linear multivariate analysis technique is therefore necessary to unravel the inherent structure in the gait data. Kernel principal component analysis (KPCA) is an extension of principal component analysis (PCA), the most widely used method in this field, and can provide inside into non-linear relationships of the variables of interest. Despite the growing use of PCA in running gait research, no prior studies have evaluated the utility of KPCA in determining the between-group differences in running gait patterns. Therefore, the objective of this study was to compare the performance of KPCA and PCA in identifying differences between male and female runners and as well between young and older runners in running gait kinematics. Running kinematic data were analysed on a gender group (female, n = 50; male, n = 50) and an age group (young, n = 50; older, n = 50) for discrete variables and waveform data. The following key results were obtained: (1) the performance of the KPCA for the identification of gender and age differences in running gait kinematics increased as compared to using the linear PCA; (2) the Gaussian function often performed better than the polynomial function for these two experiments; (3) there was no consistent optimal value of either the width σ of the Gaussian kernel or the degree d of the polynomial kernel for different data types and experiments. These results suggest that non-linear features extracted by KPCA could be potentially useful in identifying and discriminating between-group differences and changes in running gait patterns, and could provide useful information about the intrinsic non-linear dynamics of running movement.