Nonlinear Sufficient Dimension Reduction for Metric Space Objects

Principal support vector machine was proposed recently by Li, Artemiou and Li (2011) to combine L$1$ support vector machine and sufficient dimension reduction. We introduce the principal L$q$ support vector machine as a unified framework for linear and nonlinear sufficient dimension reduction. By noticing that the solution of L$1$ support vector machine may not be unique, we set $q>1$ to ensure the uniqueness of the solution. The asymptotic distribution of the proposed estimators are derived for $q> 1$. We demonstrate through numerical studies that the proposed L$2$ support vector machine estimators improve existing methods in accuracy, and are less sensitive to the tuning parameter selection.

Dimension reduction has become an important tool for dealing with high dimensional data. Locally linear embedding (LLE) is a nonlinear dimension reduction method which can preserve local configurations of nearest neighbors. However, finding the nearest neighbors requires the definition of a distance measure, which is a critical step in LLE. In this paper, the Rank-order distance measure is used to substitute the traditional Euclidean distance measure in order to find better nearest neighbor candidates for preserving local configurations of the manifolds. The Rank-order distance between the data points is calculated using their neighbors’ ranking orders, and is shown to be able to improve the clustering of high dimensional data. The proposed method is called Rank-order based LLE (RLLE). The RLLE method is evaluated by comparing with the original LLE, ISO-LLE and IED-LLE on two handwritten datasets. It is shown that the effectiveness of a distance measure in the LLE method is closely related to whether it can be used to find good nearest neighbors. The experimental results show that the proposed RLLE method can improve the process of dimension reduction effectively, and C-index is another good candidate for evaluating the dimension reduction results.

Modern manufacturing processes are often characterized by high dimensionality and nonlinearity; dimension reduction is usually required to identify the critical features and help improve the process yield. Dimension reduction and analysis of data with high nonlinearity has been a challenging task. Many methods have been proposed based on manifold learning, which aims to learn a lower-dimensional manifold from the original high-dimensional space. However, since the mappings are usually implicit, it is difficult to build a connection between the classification results and the original features. Apart from the prediction results, industry data analysis also focus on the features themselves. Therefore, methods that are able to combine feature selection with dimension reduction is often needed. This paper proposed a hybrid method for nonlinear dimension reduction and feature selection based on Locally Linear Embedding (LLE). LLE and many filter feature selection shares a common procedure of neighbor search. By integrating nearest neighbor search for both LLE and ReliefF, and adjusting the distance measure in supervised LLE with results from feature selection, the proposed method could connect the feature selection process with the nonlinear model, extract the critical features for further analysis, and improve the performance of process modelling. Two dataset from real industry processes are also illustrated as case studies.

In this article, we propose a general nonlinear sufficient dimension reduc-tion (SDR) framework when both the predictor and the response lie in some general metric spaces. We construct reproducing kernel Hilbert spaces with kernels that are fully determined by the distance functions of the metric spaces, and then leverage the inherent structures of these spaces to define a nonlinear SDR framework. We adapt the classical sliced inverse regression within this framework for the metric space data. Next we build an estimator based on the corresponding linear opera-tors, and show that it recovers the regression information in an unbiased manner. We derive the estimator at both the operator level and under a coordinate system, and establish its convergence rate. Lastly, we illustrate the proposed method using synthetic and real data sets that exhibit non-Euclidean geometry.

In multimedia applications, dimension reduction is essential to the effectiveness and efficiency of an algorithm due to the curse of dimensionality. Recently, its adaptive variants have received considerable attention in unsupervised learning since a single pass without label information often fails to guarantee an optimal representation, especially when the parameters are not set properly. However, most such methods are basically linear, therefore unable to consider the geometrical structure of the data space. In this paper, we propose a novel algorithm called Nonlinear Adaptive Dimension Reduction (NADR), which adaptively learns the optimal low-dimensional coordinates that preserve the intrinsic geometric structure of the original data. Moreover, the incorporation of K-means enables NADR to be a powerful alternative for cluster analysis. Experiments on benchmark image data sets illustrate that NADR outperforms the state-of-the-art adaptive dimension reduction methods.

Detection of maize seed germination rate based on improved locally linear embedding

Locally linear embedding with additive noise

We propose a general theory and the estimation procedures for nonlinear sufficient dimension reduction where both the predictor and the response may be random functions. The relation between the response and predictor can be arbitrary and the sets of observed time points can vary from subject to subject. The functional and nonlinear nature of the problem leads to construction of two functional spaces: the first representing the functional data, assumed to be a Hilbert space, and the second characterizing nonlinearity, assumed to be a reproducing kernel Hilbert space. A particularly attractive feature of our construction is that the two spaces are nested, in the sense that the kernel for the second space is determined by the inner product of the first. We propose two estimators for this general dimension reduction problem, and establish the consistency and convergence rate for one of them. These asymptotic results are flexible enough to accommodate both fully and partially observed functional data. We investigate the performances of our estimators by simulations, and applied them to data sets about speech recognition and handwritten symbols.

Recently, many dimensionality reduction algorithms, including local methods and global methods, have been presented. The representative local linear methods are locally linear embedding (LLE) and linear preserving projections (LPP), which seek to find an embedding space that preserves local information to explore the intrinsic characteristics of high dimensional data. However, both of them still fail to nicely deal with the sparsely sampled or noise contaminated datasets, where the local neighborhood structure is critically distorted. On the contrary, principal component analysis (PCA), the most frequently used global method, preserves the total variance by maximizing the trace of feature variance matrix. But PCA cannot preserve local information due to pursuing maximal variance. In order to integrate the locality and globality together and avoid the drawback in LLE and PCA, in this paper, inspired by the dimensionality reduction methods of LLE and PCA, we propose a new dimensionality reduction method for face recognition, namely, unsupervised linear difference projection (ULDP). This approach can be regarded as the integration of a local approach (LLE) and a global approach (PCA), so that it has better performance and robustness in applications. Experimental results on the ORL, YALE and AR face databases show the effectiveness of the proposed method on face recognition.

Manifold learning and its various algorithms have attracted more and more attention in recent years. Locally linear embedding (LLE) algorithm, as a classical nonlinear dimensionality reduction algorithm, has been widely used in many technical fields, such as image classification and object recognition. This paper proposes an improved algorithm based on LLE algorithm–Sparse LLE, which applies a new method of classification in the process of finding K-nearest neighbours with original LLE algorithm. According to the experiment results, the new algorithm can not only guarantee the effect of dimensionality reduction, but also greatly reduce the processing time of the dimensionality reduction of datasets.

In this paper, a novel technique for the recognition of weed seeds, known as Locally Linear Embedding (LLE), is presented. This experiment contains two steps: dimensionality reduction with LLE and classification. In dimensionality reduction part, LLE, which is an unsupervised non-linear technique, is used to map high dimensional data into low dimensional, neighborhood-preserving embeddings and extract features. In classification part, minimum Euclidean distance classifier is used to recognize different weed seeds. This paper presents a comparison between LLE and PCA. Experimental results demonstrate that the representation of LLE is superior significantly as expected and gains much higher recognition rates, amounting to 96%, which is 8% higher than those of PCA.

Locally Linear Embedding (LLE) is a recently proposed algorithm for non-linear dimensionality reduction and manifold learning. However, it may not be optimal for classification problem. In this paper, an improved version of LLE, namely KFDA-LLE, is proposed using kernel Fisher discriminant analysis (KFDA) method, combined with SVM classifier for face recognition task. Firstly, the input training samples are projected into the low-dimensional space by LLE. Then KFDA is introduced for finding the optimal projection direction. Finally, SVM classifier is used for face recognition. Experimental results on face database demonstrate that the extended LLE method is more efficient and robust.

The locally linear embedding (LLE) is an effective algorithm for dimensional reduction, visualization and classification, which can automatically discover the low-dimensional nonlinear manifold in a high-dimensional data space and then embed the data points into a low-dimensional embedding space, using tractable linear algebraic techniques that are easy to implement. Despite its appealing properties, LLE is not robust against outliers in the data, yet so far very little has been done to address the robustness problem. To improve the performance of LLE, some modified LLE algorithms were proposed by rigidly restraining the influence of the noise and outliers in the data embedding. In this paper, a weighted LLE (WLLE) is proposed. The experiments on synthetic data and real plant leaf image data demonstrate that WLLE is effective and feasible.KeywordsLocally linear embedding (LLE)Weighted LLE (WLLE)Plant leaf visualization

Hyperspectral imagery (HSI) has high spectral and spatial resolutions, which are essential for anomaly detection (AD). Many anomaly detectors assume that the spectrum signature of HSI pixels can be modeled with a Gaussian distribution, which is actually not accurate and often leads to many false alarms. Therefore, a sparsity model without any distribution hypothesis is usually employed. Dimensionality reduction (DR) as a preprocessing step for HSI is important. Principal component analysis as a conventional DR method is a linear projection and cannot exploit the nonlinear properties in hyperspectral data, whereas locally linear embedding (LLE) as a local, nonlinear manifold learning algorithm works well for DR of HSI. A modified algorithm of sparsity divergence index based on locally linear embedding (SDI-LLE) is thus proposed. First, kernel collaborative representation detection is adopted to calculate the sparse dictionary matrix of local reconstruction weights in LLE. Then, SDI is obtained both in the spectral and spatial domains, where spatial SDI is computed after DR by LLE. Finally, joint SDI, combining spectral SDI and spatial SDI, is computed, and the optimal SDI is performed for AD. Experimental results demonstrate that the proposed algorithm significantly improves the performance, when compared with its counterparts.

Visualizing Alzheimer's disease progression in low dimensional manifolds