Bayesian optimization of multiscale kernel principal component analysis and its application to model Gas-to-liquid (GTL) process data

In kernel methods, choosing a suitable kernel is in dispensable for favorable results. No well-founded methods, however, have been established in general for unsupervised learning. We focus on kernel Princ ipal Component Analysis (kernel PCA), which is a nonlinear extension of principal component analysis and ha s been used electively for extracting nonlinear featu res and reducing dimensionality. As a kernel method , kernel PCA also suffers from the problem of kernel choice. Although cross-validation is a popular meth od for choosing hyperparameters, it is not applicable straightforwardly to choose a kernel in kernel PCA because of the incomparable norms given by differen t kernels. It is important, thus, to develop a well founded method for choosing a kernel in kernel PCA. This study proposes a method for choosing hyperparameters in kernel PCA (kernel and the number of components) based on cross-validation for the comparable reconstruction errors of pre-images in t he original space. The experimental results on synthesized and real-world datasets demonstrate tha t the proposed method successfully selects an appropriate kernel and the number of components in kernel PCA in terms of visualization and classification errors on the principal components. The results imply that the proposed method enables automatic design of hyperparameters in kernel PCA.

There is a trend in recent machine learning community to construct a nonlinear version of a linear algorithm using the 'kernel method', e.g. Support Vector Machines (SVMs), kernel principal component analysis, kernel fisher discriminant analysis and the recent kernel clustering algorithms. In unsupervised clustering algorithms using kernel method, typically, a nonlinear mapping is used first to map the data into a potentially much higher feature space, where clustering is then performed. A drawback of these kernel clustering algorithms is thatthe clustering prototypes lie in high dimensional feature space and hence lack clear and intuitive descriptions unless using additional projection approximation from the feature to the data space as done in the existing literatures. In this paper, a novel clustering algorithm using the 'kernel method' based on the classical fuzzy clustering algorithm (FCM) is proposed and called as kernel fuzzy c-means algorithm (KFCM). KFCM adopts a new kernel-induced metric in the data space to replace the original Euclidean norm metric in FCM and the clustered prototypes still lie in the data space so that the clustering results can be reformulated and interpreted in the original space. Our analysis shows that KFCM is robust to noise and outliers and also tolerates unequal sized clusters. And finally this property is utilized to cluster incomplete data. Experiments on two artificial and one real datasets show that KFCM hasbetter clustering performance and more robust than several modifications of FCM for incomplete data clustering.

Kernel principal component analysis (KPCA) is a method for performing a nonlinear form of principal component analysis (PCA). It involves nonlinear mapping of the input data onto a high dimensional feature space and performing PCA there. Through KPCA, nonlinear relations in the input space are detected as linear relations in the feature space. Furthermore, explicit computation of the mapping is not necessary in KPCA. Only dot products of the mapped data items in the feature space are required which are obtained directly from the input data using a kernel function. Moreover, control limits of traditional statistical process monitoring indices are usually determined under assumption of normal (Gaussian) distribution. Although, it is recognized that this is inappropriate in nonlinear processes, the impact of using kernel density estimation (KDE), which is a nonparametric method, to determine control limits in KPCA-based process monitoring has not been reported. This paper seeks to bridge this gap. The KPCA with KDE approach is applied to the Tennessee Eastman process for the detection of faults. The results confirm that associating KPCA with kernel density estimated control limits provides better monitoring performance than using control limits based on the normal probability density function.

Principal Components Analysis (PCA) is used to classify the running condition of a machine by means of projecting the original data to the Principal Components space. However, if the data are concentrated in a nonlinear subspace, PCA will fail to work well. Kernel Principal Components Analysis (KPCA) transforms the input data from the original input space into a higher dimensional feature space with the nonlinear mapping, and then uses the nonlinear principal components to realize the classification. In this paper a case of gear fault diagnosis was studied with KPCA. The characteristic values of frequent domain from vibration signals of the gearbox under the running condition were extracted, and the KPCA method was used to classify gear crack fault. The result shows that KPCA is more effective to distinguish the state of the gear and more suitable to diagnose the gear faults in early stage.

This paper presents a novel algorithm - robust kernel principal component analysis (robust KPCA), on the basis of the research of kernel principal component analysis (KPCA) and robust principal component analysis (RPCA). First, this algorithm sets the radius of the images of the training samples in the feature space using kernel tricks, then determines whether the samples are outliers or not, and finally analyzes the training samples which have eliminated the outliers using KPCA algorithm. The improved KPCA algorithm not only retains the non-linearity property of KPCA algorithm but also gets better robustness. Because the effects of outliers are eliminated, robust KPCA algorithm gets higher accuracy than KPCA algorithm for data analysis. The simulation experiments show that the robust KPCA algorithm developed is better than the KPCA algorithm.

Aimed at the non-stationary characteristics of rolling bearing vibration signal,a fault diagnosis method was proposed based on local-wave method and KPCA(kernel principal component analysis)-LSSVM(least squares support vector machine).Firstly,local wave decomposition was used to decompose rolling bearing vibration signal into several intrinsic mode function(IMF),whose feature energy,singular values and AR model parameters were computed as initial feature vectors.Secondly,ini tial feature vectors were mapped into a higher-dimensional space with KPCA,and the kemel principal components were extracted as new feature vectors,which used as the input of LSSVM for fault classification.The experimental results show the KPCA-LSSVM method improves LSSVM's classification performance by KPCA obtaining additional discriminative information,and has better generalization than direct LSSVM method,and can identify rolling bearing fault patterns more accurately.

Summary The principal component analysis (PCA) is exists the problem of lack of probability model and the absence of higher-order statistics information. Therefore，in order to overcome its shortcomings，this paper studies the method that can overcome two disadvantages of the principal component analysis (PCA) — the probability kernel principal component analysis (PKPCA) which is based on Bayesian theory and kernel principal component analysis (KPCA). First，the sample data are mapped to the high dimensional feature space，then define probability model of the data in high-dimensional space，and finally，expectation maximization (EM) estimated is used to get the best results. This method has both the advantage of probability analysis and kernel principal component analysis (KPCA). It is able to effectively adapt to more complex reservoir conditions and can realize the non-linear probability analysis. The probability of kernel principal component analysis (PKPCA) is applied to reservoir prediction of the oilfield. The predicted results show that the method can improve the precision attribute optimization，while improving the accuracy of the forecasts of reservoir.

As a valid statistical tool, principal component analysis (PCA) has been widely used in industrial process monitoring. But due to its intrinsic linear character, it performs badly in nonlinear process monitoring. Kernel PCA (KPCA) can extract useful information in nonlinear data. However KPCA-based monitoring is not suitable for on-line monitoring because of large calculation and much memory occupation. The paper introduces an on-line monitoring method based on KPCA and neural network (NN), where KPCA is used to extract nonlinear principal components (PCs) and then NN approximates the relationship between process data and nonlinear PCs. We can obtain nonlinear PCs by NN to compute the monitoring indices and then achieve the on-line monitoring. The case study shows the validity of the method.KeywordsNeural NetworkKernel Principal Component AnalysisMonitoring IndexMemory OccupationComplex Industrial ProcessThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Principal component analysis (PCA) has been proven to be an efficient method in dimensionality reduction, feature extraction and pattern recognition. Kernel principal component analysis (KPCA) can be considered as a natural nonlinear generalization of PCA, which performs linear PCA in a high dimensional space implicitly by using kernel trick. However, both conventional PCA and KPCA suffer from the deficiency of being sensitive to outliers. Existing robust KPCA has to eigen-decompose the gram matrix directly in each step and is much more computationally infeasible due to the large size of the matrix when the number of training samples is large. By extending existing robust PCA algorithm using kernel methods, we present a novel robust adaptive algorithm for calculating the kernel principal components. The proposed method not only preserves the characteristic of capturing underlying nonlinear structure of KPCA but also is robust against outliers by restraining the effect of outlying samples. Compared with existing robust KPCA methods, our method is performed without having to store the kernel matrix, which can reduce significantly the storage burden. In addition, our method shows the potential of expansibility to the incremental learning version. Experimental results on synthetic data indicate that our improved algorithm is effective and promising.

A deep kernel method for lithofacies identification using conventional well logs

A novel Gabor-Kernel face recognition method is proposed in this paper. This involves convolving a face image with a series of Gabor wavelets at different scales, locations, and orientations and extracting features from resulting Gabor filtered images. kernel discriminant analysis (KDDA) is then applied to the feature vectors for dimension reduction as well as class separability enhancement. A database of 600 frontal-view face images from the FERET face database is used to test the method. Experimental results demonstrate the advantage of KDDA over other Kernel methods such as kernel principal component analysis (KPCA) and general discriminant analysis (GDA). Significant improvements are also observed when features are extracted from Gabor filtered images instead of the original images. A 94% accuracy has been observed for the novel Gabor + KDDA method on the FERET database using a simple classifier, which could be further improved by employing a more complex classifier and distance measurer.

Support Vector Machines and Kernel methods have become very popular as methods for learning from examples. In this paper, in order to solve the non-linear and the time of recognition problem, recent advanced techniques are presented. For example, Kernel Principal Components and Proximal Support Vector Machines, Kernel Principal Components Analysis can describe multiple correlations between pixels in the image recognition, and can handle the high order statistics of the original data, at the same time the nonlinear features of the image can be preferably extracted. Proximal Support Vector Machines is an improvement for these characters of Support Vector Machines, combines the proximal algorithm with Support Vector Machines, has a better ability of nonlinear mapping and a stronger generalization capability, and the pace of Proximal Support Vector Machines is very enhanced. In this paper, the computer simulation was progressed based on ORL face database, the effectiveness of the Proximal Support Vector Machines algorithm and Kernel Principal Components Analysis algorithm were shown by experimental results.

Manufacturing industries collect massive amounts of multivariate measurement through automated inspection processes. Noisy measurements and high-dimensional, irrelevant features make it difficult to identify useful patterns in the data. Principal component analysis provides linear summaries of datasets with fewer latent variables. Kernel Principal Component Analysis (KPCA), however, identifies nonlinear patterns. One challenge in KPCA is to inverse map the denoised signal from a high-dimensional feature space into its preimage in input space to visualize the nonlinear variation sources. However, such an inverse map is not always defined. This article provides a new meta-method applicable to any KPCA algorithm to approximate the preimage. It improves upon previous work where a strong assumption was the availability of noise-free training data. This is problematic for applications such as manufacturing variation analysis. To attenuate noise in kernel subspace estimation the final preimage is estimated as the average from bagged samples drawn from the original dataset. The improvement is most pronounced when the parameters differ from those that minimize the error rate. Consequently, the proposed approach improves the robustness of any base KPCA algorithm. The usefulness of the proposed method is demonstrated by analyzing a classic handwritten digit dataset and a face dataset. Significant improvement over the existing methods is observed.

Kernel function implicitly maps data from its original space to a higher dimensional feature space. Kernel based machine learning algorithms are typically applied to data that is not linearly separable in its original space. Although kernel methods are among the most elegant part of machine learning, it is challenging for users to define or select a proper kernel function with optimized parameter settings for their data. In this paper, we propose a novel method called Deep Kernel that can automatically learn a kernel function from data using deep learning. The deep kernel is currently utilized in classification, and dimension reduction and visualization. For the classification task, we evaluate the deep kernel method by comparing its performance with the optimized Gaussian kernels, both using support vector machines as the decision model, on different types of datasets. The experimental results show that the proposed deep kernel method outperforms the traditional methods with Gaussian kernels on most of the data sets. For the dimension reduction and visualization task, the deep kernel is used along with kernel PCA. The results are also compared and contrasted with using the RBF kernel with multiple parameters. The deep kernel is shown to be more powerful in dimension reduction and visualization than the RBF kernel.

Nonlinear Process Monitoring Using Dynamic Sparse Kernel Classifier