Clustering Incomplete Data Using Kernel-Based Fuzzy C-means Algorithm

Support Vector Machines and Kernel methods have become very popular as methods for learning from examples. In this paper, in order to solve the non-linear and the time of recognition problem, recent advanced techniques are presented. For example, Kernel Principal Components and Proximal Support Vector Machines, Kernel Principal Components Analysis can describe multiple correlations between pixels in the image recognition, and can handle the high order statistics of the original data, at the same time the nonlinear features of the image can be preferably extracted. Proximal Support Vector Machines is an improvement for these characters of Support Vector Machines, combines the proximal algorithm with Support Vector Machines, has a better ability of nonlinear mapping and a stronger generalization capability, and the pace of Proximal Support Vector Machines is very enhanced. In this paper, the computer simulation was progressed based on ORL face database, the effectiveness of the Proximal Support Vector Machines algorithm and Kernel Principal Components Analysis algorithm were shown by experimental results.

In kernel methods, choosing a suitable kernel is in dispensable for favorable results. No well-founded methods, however, have been established in general for unsupervised learning. We focus on kernel Princ ipal Component Analysis (kernel PCA), which is a nonlinear extension of principal component analysis and ha s been used electively for extracting nonlinear featu res and reducing dimensionality. As a kernel method , kernel PCA also suffers from the problem of kernel choice. Although cross-validation is a popular meth od for choosing hyperparameters, it is not applicable straightforwardly to choose a kernel in kernel PCA because of the incomparable norms given by differen t kernels. It is important, thus, to develop a well founded method for choosing a kernel in kernel PCA. This study proposes a method for choosing hyperparameters in kernel PCA (kernel and the number of components) based on cross-validation for the comparable reconstruction errors of pre-images in t he original space. The experimental results on synthesized and real-world datasets demonstrate tha t the proposed method successfully selects an appropriate kernel and the number of components in kernel PCA in terms of visualization and classification errors on the principal components. The results imply that the proposed method enables automatic design of hyperparameters in kernel PCA.

There is always much difficult in the MR image segmentation. Although fuzzy c-means(FCM) clustering algorithm has been widely used in the field of image segmentation study, some inherent deficiencies of this algorithm especially the high cost of computation made the algorithm to be difficult widely used in practice. A novel algorithm, based on kernel fuzzy c-means (KFCM) clustering algorithm and the k-nearest neighbor (KNN) algorithm, is proposed to improve the performance of MR image segmentation. In this algorithm, the statistical gray level histogram of image is used in KFCM algorithm to speed up the algorithm. Furthermore, the spatial information of image is also considered by k-nearest neighbor algorithm based on kernel methods. With kernel methods each pixel of image is mapped into a high-dimensional feature space where FCM algorithm and KNN algorithm are carried out. Experiments show that the proposed algorithm is effective and efficient in image segmentation.

The aim of this letter is to propose a theory of deep restricted kernel machines offering new foundations for deep learning with kernel machines. From the viewpoint of deep learning, it is partially related to restricted Boltzmann machines, which are characterized by visible and hidden units in a bipartite graph without hidden-to-hidden connections and deep learning extensions as deep belief networks and deep Boltzmann machines. From the viewpoint of kernel machines, it includes least squares support vector machines for classification and regression, kernel principal component analysis (PCA), matrix singular value decomposition, and Parzen-type models. A key element is to first characterize these kernel machines in terms of so-called conjugate feature duality, yielding a representation with visible and hidden units. It is shown how this is related to the energy form in restricted Boltzmann machines, with continuous variables in a nonprobabilistic setting. In this new framework of so-called restricted kernel machine (RKM) representations, the dual variables correspond to hidden features. Deep RKM are obtained by coupling the RKMs. The method is illustrated for deep RKM, consisting of three levels with a least squares support vector machine regression level and two kernel PCA levels. In its primal form also deep feedforward neural networks can be trained within this framework.

In kernel methods such as kernel principal component analysis (PCA) and support vector machines, the so called kernel trick is used to avoid direct calculations in a high (virtually infinite) dimensional kernel space. In this brief, based on the fact that the effective dimensionality of a kernel space is less than the number of training samples, we propose an alternative to the kernel trick that explicitly maps the input data into a reduced dimensional kernel space. This is easily obtained by the eigenvalue decomposition of the kernel matrix. The proposed method is named as the nonlinear projection trick in contrast to the kernel trick. With this technique, the applicability of the kernel methods is widened to arbitrary algorithms that do not use the dot product. The equivalence between the kernel trick and the nonlinear projection trick is shown for several conventional kernel methods. In addition, we extend PCA-L1, which uses L1-norm instead of L2-norm (or dot product), into a kernel version and show the effectiveness of the proposed approach.

This paper presents a novel algorithm - robust kernel principal component analysis (robust KPCA), on the basis of the research of kernel principal component analysis (KPCA) and robust principal component analysis (RPCA). First, this algorithm sets the radius of the images of the training samples in the feature space using kernel tricks, then determines whether the samples are outliers or not, and finally analyzes the training samples which have eliminated the outliers using KPCA algorithm. The improved KPCA algorithm not only retains the non-linearity property of KPCA algorithm but also gets better robustness. Because the effects of outliers are eliminated, robust KPCA algorithm gets higher accuracy than KPCA algorithm for data analysis. The simulation experiments show that the robust KPCA algorithm developed is better than the KPCA algorithm.

Bayesian optimization of multiscale kernel principal component analysis and its application to model Gas-to-liquid (GTL) process data

In this paper, we present dictionary learning methods for sparse signal representations in a high dimensional feature space. Using the kernel method, we describe how the well known dictionary learning approaches, such as the method of optimal directions and KSVD, can be made nonlinear. We analyze their kernel constructions and demonstrate their effectiveness through several experiments on classification problems. It is shown that nonlinear dictionary learning approaches can provide significantly better performance compared with their linear counterparts and kernel principal component analysis, especially when the data is corrupted by different types of degradations.

Evaluation of the performance of clustering algorithms in kernel-induced feature space

Principal Components Analysis (PCA) is used to classify the running condition of a machine by means of projecting the original data to the Principal Components space. However, if the data are concentrated in a nonlinear subspace, PCA will fail to work well. Kernel Principal Components Analysis (KPCA) transforms the input data from the original input space into a higher dimensional feature space with the nonlinear mapping, and then uses the nonlinear principal components to realize the classification. In this paper a case of gear fault diagnosis was studied with KPCA. The characteristic values of frequent domain from vibration signals of the gearbox under the running condition were extracted, and the KPCA method was used to classify gear crack fault. The result shows that KPCA is more effective to distinguish the state of the gear and more suitable to diagnose the gear faults in early stage.

A deep kernel method for lithofacies identification using conventional well logs

Classification of hyperspectral remote sensing data with support vector machines (SVMs) is investigated. SVMs have been introduced recently in the field of remote sensing image processing. Using the kernel method, SVMs map the data into higher dimensional space to increase the separability and then fit an optimal hyperplane to separate the data. In this paper, two kernels have been considered. The generalization capability of SVMs as well as the ability of SVMs to deal with high dimensional feature spaces have been tested in the situation of very limited training set. SVMs have been tested on real hyperspectral data. The experimental results show that SVMs used with the two kernels are appropriate for remote sensing classification problems.

Machine Learning is a modern and actively developing field of computer science, devoted to extracting and estimating dependencies from empirical data. It combines such fields as statistics, optimization theory and artificial intelligence. In practical tasks, the general aim of Machine Learning is to construct algorithms able to generalize and predict in previously unseen situations based on some set of examples. Given some finite information, Machine Learning provides ways to exract knowledge, describe, explain and predict from data. Kernel Methods are one of the most successful branches of Machine Learning. They allow applying linear algorithms with well-founded properties such as generalization ability, to non-linear real-life problems. Support Vector Machine is a well-known example of a kernel method, which has found a wide range of applications in data analysis nowadays. In many practical applications, some additional prior knowledge is often available. This can be the knowledge about the data domain, invariant transformations, inner geometrical structures in data, some properties of the underlying process, etc. If used smartly, this information can provide significant improvement to any data processing algorithm. Thus, it is important to develop methods for incorporating prior knowledge into data-dependent models. The main objective of this thesis is to investigate approaches towards learning with kernel methods using prior knowledge. Invariant learning with kernel methods is considered in more details. In the first part of the thesis, kernels are developed which incorporate prior knowledge on invariant transformations. They apply when the desired transformation produce an object around every example, assuming that all points in the given object share the same class. Different types of objects, including hard geometrical objects and distributions are considered. These kernels were then applied for images classification with Support Vector Machines. Next, algorithms which specifically include prior knowledge are considered. An algorithm which linearly classifies distributions by their domain was developed. It is constructed such that it allows to apply kernels to solve non-linear tasks. Thus, it combines the discriminative power of support vector machines and the well-developed framework of generative models. It can be applied to a number of real-life tasks which include data represented as distributions. In the last part of the thesis, the use of unlabelled data as a source of prior knowledge is considered. The technique of modelling the unlabelled data with a graph is taken as a baseline from semi-supervised manifold learning. For classification problems, we use this apporach for building graph models of invariant manifolds. For regression problems, we use unlabelled data to take into account the inner geometry of the input space. To conclude, in this thesis we developed a number of approaches for incorporating some prior knowledge into kernel methods. We proposed invariant kernels for existing algorithms, developed new algorithms and adapted a technique taken from semi-supervised learning for invariant learning. In all these cases, links with related state-of-the-art approaches were investigated. Several illustrative experiments were carried out on real data on optical character recognition, face image classification, brain-computer interfaces, and a number of benchmark and synthetic datasets.

Principal component analysis (PCA) and kernel Fisher discriminant analysis (KFDA) were applied to grade Fuji apples combined with near infrared reflectance (NIR) spectroscopy. Firstly, PCA was used to reduce the dimensionality of NIR spectra acquired by the Antaris II FT-NIR spectrophotometer on apples. Secondly, nonlinear discriminant information was extracted by kernel Fisher discriminant analysis (KFDA). Finally, the k-nearest neighbors algorithm with leave one out strategy was utilized to classify apple samples into two grades. LDA can only solve linearly separable problems, and it is not suitable in solving some nonlinear problems. But unlike LDA, KFDA can solve nonlinearly separable problems, and it projects data onto a high-dimensional feature space by the nonlinear mapping. Experimental results showed that KFDA achieved higher classification rate compared with LDA.

A novel Gabor-Kernel face recognition method is proposed in this paper. This involves convolving a face image with a series of Gabor wavelets at different scales, locations, and orientations and extracting features from resulting Gabor filtered images. kernel discriminant analysis (KDDA) is then applied to the feature vectors for dimension reduction as well as class separability enhancement. A database of 600 frontal-view face images from the FERET face database is used to test the method. Experimental results demonstrate the advantage of KDDA over other Kernel methods such as kernel principal component analysis (KPCA) and general discriminant analysis (GDA). Significant improvements are also observed when features are extracted from Gabor filtered images instead of the original images. A 94% accuracy has been observed for the novel Gabor + KDDA method on the FERET database using a simple classifier, which could be further improved by employing a more complex classifier and distance measurer.