Adaptive Nonlinear Dimensionality Reduction with a Local Metric

Feature extraction is an important preprocessing step which is encountered in many areas such as data mining, pattern recognition and scientific visualization. In this paper, a new method for sparse feature extraction using local manifold learning is proposed. Similarities in a neighborhood are first computed to explore local geometric structures, producing sparse feature representation. Based on the constructed similarity matrix, linear dimension reduction is applied to enhance similarities among the elements in the same class and extract optimal features for classification performances. Since it only computes similarities in a neighborhood, sparsity in the similarity matrix can give computational efficiency and memory savings. Experimental results demonstrate superior performances of the proposed method.

Locally Linear Embedding (LLE) is a recently proposed algorithm for non-linear dimensionality reduction and manifold learning. However, it may not be optimal for classification problem. In this paper, an improved version of LLE, namely KFDA-LLE, is proposed using kernel Fisher discriminant analysis (KFDA) method, combined with SVM classifier for face recognition task. Firstly, the input training samples are projected into the low-dimensional space by LLE. Then KFDA is introduced for finding the optimal projection direction. Finally, SVM classifier is used for face recognition. Experimental results on face database demonstrate that the extended LLE method is more efficient and robust.

Modern manufacturing processes are often characterized by high dimensionality and nonlinearity; dimension reduction is usually required to identify the critical features and help improve the process yield. Dimension reduction and analysis of data with high nonlinearity has been a challenging task. Many methods have been proposed based on manifold learning, which aims to learn a lower-dimensional manifold from the original high-dimensional space. However, since the mappings are usually implicit, it is difficult to build a connection between the classification results and the original features. Apart from the prediction results, industry data analysis also focus on the features themselves. Therefore, methods that are able to combine feature selection with dimension reduction is often needed. This paper proposed a hybrid method for nonlinear dimension reduction and feature selection based on Locally Linear Embedding (LLE). LLE and many filter feature selection shares a common procedure of neighbor search. By integrating nearest neighbor search for both LLE and ReliefF, and adjusting the distance measure in supervised LLE with results from feature selection, the proposed method could connect the feature selection process with the nonlinear model, extract the critical features for further analysis, and improve the performance of process modelling. Two dataset from real industry processes are also illustrated as case studies.

Manifold learning and its various algorithms have attracted more and more attention in recent years. Locally linear embedding (LLE) algorithm, as a classical nonlinear dimensionality reduction algorithm, has been widely used in many technical fields, such as image classification and object recognition. This paper proposes an improved algorithm based on LLE algorithm–Sparse LLE, which applies a new method of classification in the process of finding K-nearest neighbours with original LLE algorithm. According to the experiment results, the new algorithm can not only guarantee the effect of dimensionality reduction, but also greatly reduce the processing time of the dimensionality reduction of datasets.

The present paper put forward a non-destructive detection method which combines semi-transmission hyperspectral imaging technology with manifold learning dimension reduction algorithm and least squares support vector machine (LSSVM) to recognize internal and external defects in potatoes simultaneously. Three hundred fifteen potatoes were bought in farmers market as research object, and semi-transmission hyperspectral image acquisition system was constructed to acquire the hyperspectral images of normal external defects (bud and green rind) and internal defect (hollow heart) potatoes. In order to conform to the actual production, defect part is randomly put right, side and back to the acquisition probe when the hyperspectral images of external defects potatoes are acquired. The average spectrums (390-1,040 nm) were extracted from the region of interests for spectral preprocessing. Then three kinds of manifold learning algorithm were respectively utilized to reduce the dimension of spectrum data, including supervised locally linear embedding (SLLE), locally linear embedding (LLE) and isometric mapping (ISOMAP), the low-dimensional data gotten by manifold learning algorithms is used as model input, Error Correcting Output Code (ECOC) and LSSVM were combined to develop the multi-target classification model. By comparing and analyzing results of the three models, we concluded that SLLE is the optimal manifold learning dimension reduction algorithm, and the SLLE-LSSVM model is determined to get the best recognition rate for recognizing internal and external defects potatoes. For test set data, the single recognition rate of normal, bud, green rind and hollow heart potato reached 96.83%, 86.96%, 86.96% and 95% respectively, and he hybrid recognition rate was 93.02%. The results indicate that combining the semi-transmission hyperspectral imaging technology with SLLE-LSSVM is a feasible qualitative analytical method which can simultaneously recognize the internal and external defects potatoes and also provide technical reference for rapid on-line non-destructive detecting of the internal and external defects potatoes.

Local manifold learning has been successfully applied to hyperspectral dimensionality reduction in order to embed nonlinear and nonconvex manifolds in the data. Local manifold learning is mainly characterized by affinity matrix construction, which is composed of two steps: neighbor selection and computation of affinity weights. There is a challenge in each step: First, the neighbor selection is sensitive to complex spectral variability due to nonuniform data distribution, illumination variations, and sensor noise; second, the computation of affinity weights is challenging due to highly correlated spectral signatures in the neighborhood. To address the two issues, in this paper, a novel manifold learning methodology based on locally linear embedding is proposed through learning a robust local manifold representation. More specifically, a hierarchical neighbor selection is designed to progressively eliminate the effects of complex spectral variability using joint normalization and to robustly compute affinity (or reconstruction) weights reducing multicollinearity via the refined neighbor selection. Additionally, an idea that combines spatial–spectral information is introduced into the proposed manifold learning methodology to further improve the robustness of affinity calculations. Classification is explored as a potential application for validating the proposed algorithm. The classification accuracy in the use of different dimensionality reduction methods is evaluated and compared, while two kinds of strategies are applied in selecting the training and test samples: random sampling and region-based sampling. Experimental results show the classification accuracy obtained by the proposed method is superior to those state-of-the-art dimensionality reduction methods.

The locally linear embedding (LLE) is an effective algorithm for dimensional reduction, visualization and classification, which can automatically discover the low-dimensional nonlinear manifold in a high-dimensional data space and then embed the data points into a low-dimensional embedding space, using tractable linear algebraic techniques that are easy to implement. Despite its appealing properties, LLE is not robust against outliers in the data, yet so far very little has been done to address the robustness problem. To improve the performance of LLE, some modified LLE algorithms were proposed by rigidly restraining the influence of the noise and outliers in the data embedding. In this paper, a weighted LLE (WLLE) is proposed. The experiments on synthetic data and real plant leaf image data demonstrate that WLLE is effective and feasible.KeywordsLocally linear embedding (LLE)Weighted LLE (WLLE)Plant leaf visualization

Dimensionality reduction is a key step in hyperspectral image processing. Recent investigations are looking into nonlinear manifold learning for dimensionality reduction in hyperspectral imagery. Nonlinear manifold learning methods such as Isomap, and Local Linear Embedding are use to recover the low dimensional representation of an unknown nonlinear manifold for high dimensional data where it is important to retain the neighborhood structure of the manifold. Although these algorithms use different philosophies for recovering the nonlinear manifold, they all incorporate neighborhood information from each data point to construct a weighted graph having the data points as vertices. Thus the performance of these methods highly depends on how well these neighborhoods are selected, since all subsequent steps rely on it. The k -NN algorithm is the most widely used technique for neighbor selection in manifold learning. However, it can result in a disconnected graph and it does not fully exploit spatial neighborhood information from the image in selecting points to form the neighborhoods. In this paper, recently proposed methods for constructing the weighted graph in manifold learning are studied: k -VC, k -EC and k -MST, which have the advantage of creating connected graphs and have performed well in artificial data sets. Spatial information of the hyperspectral images is included in the manifold learning process by using spatial coherence. Experiments are conducted with artificial data and hyperspectral images. For hyperspectral images, classification accuracy was used as an indirect measure to understand how the low dimensional embedding of the data reduces dimensionality while still maintaining good discrimination performance.

At present, manifold learning is mainly applied to dimensionality reduction. However, from viewpoint of dimensionality reduction, manifold learning algorithms are only local feature preserving algorithms. For example, Local Linear Embedding is local linear preserving, Local Tangent Space Alignment is local homeomorphic preserving and Laplacian Eigenmap is local similarity preserving. The community of dimensionality reduction is now pursuing the algorithms which can preserve both local and global features of data during dimensionality reduction. In this paper, a new algorithm of dimensionality reduction, called Hilbert-Schmidt Independence Criterion Regularized Manifold Learning (HSIC-ML for short), is proposed, in which HSIC between the high dimensional data and the dimension-reduced data is added as a regularization term to the objective functions of manifold learning. The addition of HSIC regularization term makes HSIC-ML capable of preserving both local and global features during dimensionality reduction. HSIC is a criterion measuring the statistical dependence between two data sets and has been widely applied to machine learning in recent years. However, since HSIC was first proposed around 2005, there seems to have not been applied directly to dimensionality reduction, not applied as a regularization term either. The proposed HSIC-ML may be the first try in this respect. The experimental results presented in this paper show that the manifold learning with HSIC regularization performs better than that without HSIC regularization.

Locally linear embedding (LLE) is an elegant nonlinear method for feature extraction and manifold learning, which attempt to project the original data into a lower dimensional feature space by preserving the local neighborhood structure. However, LLE algorithm fails when it is directly applied to video with multi-shot. In this paper, video manifold feature is defined firstly, and then using LLE we extract video manifold feature through adding virtual frames on an enriched set. The proposed manifold feature extraction method is applied to shot transition and video trajectories. The video manifold feature description gives a new tool for video analysis.

Background:In our recent work, we developed a novel dynamic programming algorithm to find optimal Bayesian networks (BNs) with parent set constraints. This ‘generational orderings’ based dynamic programming search algorithm - CausNet - efficiently searches the space of possible BNs given the possible parent sets. The algorithm supports both continuous and categorical data, as well as continuous, binary and survival outcomes. In the present work, we develop a variant of CausNet - CausNet-partial - which searches the space of ‘partial generational orderings’, which further reduces the search space and is suited for finding smaller sparse optimal Bayesian networks; and can be applied to 1000s of variables.Results:We test this method both on synthetic and real data. Our algorithm performs better than three state-of-art algorithms that are currently used extensively to find optimal BNs. We apply it to simulated continuous data and also to a benchmark discrete Bayesian network ALARM, a Bayesian network designed to provide an alarm message system for patient monitoring. We first apply the original CausNet and then CausNet-partial varying the partial order from 5 to 2. CausNet-partial discovers small sparse networks with drastically reduced runtime as expected from theory.Conclusions:Our partial generational orderings based search for small optimal networks, is both an efficient and highly scalable approach for finding optimal sparse and small Bayesian Networks and can be applied to 1000s of variables. Using specifiable parameters - correlation, FDR cutoffs, in-degree, and partial order - one can increase or decrease the number of nodes and density of the networks. Availability of two scoring option - BIC and Bge - and implementation for survival outcomes and mixed data types makes our algorithm very suitable for many types of high dimensional data in a variety of fields.

The patterns of communication among different chondrocyte subtypes in human cartilage degeneration and regeneration help us understand the microenvironment of osteoarthritis and optimize cell-targeted therapies. Here, a single-cell transcriptome dataset of chondrocytes is used to explore the synergistic and communicative patterns of different chondrocyte subtypes. We collected 1600 chondrocytes from 10 patients with osteoarthritis and analyzed the active communication patterns for the first time based on network analysis and pattern recognition at the single-cell level. Manifold learning and quantitative contrasts were performed to analyze conserved and specific communication pathways. We found that ProCs (Proliferative chondrocytes), ECs (Effector chondrocytes), preHTCs (Prehypertrophic chondrocytes), HTCs (Hypertrophic chondrocytes), and FCs (Fibrocartilage chondrocytes) are more active in incoming and outgoing signaling patterns, which is consistent with studies on their close functional cooperation. Among them, preHTCs play multiple roles in chondrocyte communication, and ProCs and preHTCs have many overlapping pathways. These two subtypes are the most active among all chondrocyte subtypes. Interestingly, ECs and FCs are a pair of “mutually exclusive” subtypes, of which ECs are predominant in incoming patterns and FCs in outgoing patterns. The active signaling pathways of ECs and FCs largely do not overlap. COLLAGEN and LAMININ are the main pivotal pathways, which means they are very important in the repair and expansion of joint homeostasis. Notably, only preHTCs assume multiple roles (including sender, receiver, mediator, and influencer) and are involved in multiple communication pathways. We have examined their communication patterns from the perspective of cellular interactions, revealed the relationships among different chondrocyte subtypes, and, in particular, identified a number of active subtypes and pathways that are important for targeted therapy in the osteoarthritic microenvironment. Our findings provide a new research paradigm and new insights into understanding chondrocyte activity patterns in the osteoarthritic microenvironment.

<p>Classifying high-dimensional data are a challenging task in data mining. Gene expression data is a type of high-dimensional data that has thousands of features. The study was proposing a method to extract knowledge from high-dimensional gene expression data by selecting features and classifying. Lasso was used for selecting features and the classification and regression tree (CART) algorithm was used to construct the decision tree model. To examine the stability of the lasso decision tree, we performed bootstrap aggregating (Bagging) with 50 replications. The gene expression data used was an ovarian tumor dataset that has 1,545 observations, 10,935 gene features, and binary class. The findings of this research showed that the lasso decision tree could produce an interpretable model that theoretically correct and had an accuracy of 89.32%. Meanwhile, the model obtained from the majority vote gave an accuracy of 90.29% which showed an increase in accuracy of 1% from the single lasso decision tree model. The slightly increasing accuracy shows that the lasso decision tree classifier is stable.</p>

Manifold learning is a field of study in machine learning and statistics that is closely associated with dimensionality reduction algorithmic techniques is gaining popularity these days. There are two types of manifold learning approaches: linear and nonlinear. Principal component analysis (PCA) and multidimensional scaling (MDS) are two examples of linear techniques that have long been staples in the statistician's arsenal for evaluating multivariate data. Nonlinear manifold learning, which encompasses diffusion maps, Laplacian Eigenmaps, Hessian Eigenmaps, Isomap, and local linear embedding, has seen a surge in research effort recently. A few of these methods are nonlinear extensions of linear approaches. A nearest search, the definition of distances or affinities between points (a crucial component of these methods' effectiveness), and an Eigen problem for embedding high-dimensional points into a lower dimensional space make up the algorithmic process of the majority of these techniques. The strengths and weaknesses of the new method are briefly reviewed in this article. In the field of computer graphics, we utilize a particular manifold learning method was first presented in statistics and machine learning to create a global, Spectral-based shape descriptor.

Sparse regularization discriminant analysis for face recognition