Trace Ratio Optimization-Based Semi-Supervised Nonlinear Dimensionality Reduction for Marginal Manifold Visualization

High-dimensional semi-supervised learning via a fusion-refinement procedure

Systematic comparison of unsupervised feature extraction methods for enhancing discrimination accuracy of brown rice flour adulteration in LIBS

A multitask multiview clustering algorithm in heterogeneous situations based on LLE and LE

To investigate whether nonlinear dimensionality reduction improves unsupervised classification of (1) H MRS brain tumor data compared with a linear method. In vivo single-voxel (1) H magnetic resonance spectroscopy (55 patients) and (1) H magnetic resonance spectroscopy imaging (MRSI) (29 patients) data were acquired from histopathologically diagnosed gliomas. Data reduction using Laplacian eigenmaps (LE) or independent component analysis (ICA) was followed by k-means clustering or agglomerative hierarchical clustering (AHC) for unsupervised learning to assess tumor grade and for tissue type segmentation of MRSI data. An accuracy of 93% in classification of glioma grade II and grade IV, with 100% accuracy in distinguishing tumor and normal spectra, was obtained by LE with unsupervised clustering, but not with the combination of k-means and ICA. With (1) H MRSI data, LE provided a more linear distribution of data for cluster analysis and better cluster stability than ICA. LE combined with k-means or AHC provided 91% accuracy for classifying tumor grade and 100% accuracy for identifying normal tissue voxels. Color-coded visualization of normal brain, tumor core, and infiltration regions was achieved with LE combined with AHC. The LE method is promising for unsupervised clustering to separate brain and tumor tissue with automated color-coding for visualization of (1) H MRSI data after cluster analysis.

Many works have posited the benefit of depth in deep networks. However, one of the problems encountered in the training of very deep networks is feature reuse; that is, features are ‘diluted’ as they are forward propagated through the model. Hence, later network layers receive less informative signals about the input data, consequently making training less effective. In this work, we address the problem of feature reuse by taking inspiration from an earlier work which employed residual learning for alleviating the problem of feature reuse. We propose a modification of residual learning for training very deep networks to realize improved generalization performance; for this, we allow stochastic shortcut connections of identity mappings from the input to hidden layers. We perform extensive experiments using the USPS and MNIST datasets. On the USPS dataset, we achieve an error rate of 2.69% without employing any form of data augmentation (or manipulation). On the MNIST dataset, we reach a comparable state-of-the-art error rate of 0.52%. Particularly, these results are achieved without employing any explicit regularization technique.

In recent years, a variety of nonlinear dimensionality reduction techniques (NLDR) have been proposed in the literature. They aim to address the limitations of traditional techniques such as PCA and classical scaling. Most of these techniques assume that the data of interest lie on an embedded non-linear manifold within the higher-dimensional space. They provide a mapping from the high-dimensional space to the low-dimensional embedding and may be viewed, in the context of machine learning, as a preliminary feature extraction step, after which pattern recognition algorithms are applied. Laplacian Eigenmaps (LE) is a nonlinear graph-based dimensionality reduction method. It has been successfully applied in many practical problems such as face recognition. However the construction of LE graph suffers, similarly to other graph-based DR techniques from the following issues: (1) the neighborhood graph is artificially defined in advance, and thus does not necessary benefit the desired DR task; (2) the graph is built using the nearest neighbor criterion which tends to work poorly due to the high-dimensionality of original space; and (3) its computation depends on two parameters whose values are generally uneasy to assign, the neighborhood size and the heat kernel parameter. To address the above-mentioned problems, for the particular case of the LPP method (a linear version of LE), L. Zhang et al.1 have developed a novel DR algorithm whose idea is to integrate graph construction with specific DR process into a unified framework. This algorithm results in an optimized graph rather than a predefined one.

Dozens of manifold learning-based dimensionality reduction algorithms have been proposed in the literature. The most representative ones are locally linear embedding (LLE) [65], ISOMAP [76], Laplacian eigenmaps (LE) [4], Hessian eigenmaps (HLLE) [20], and local tangent space alignment (LTSA) [102]. LLE uses linear coefficients, which reconstruct a given example by its neighbors, to represent the local geometry, and then seeks a low-dimensional embedding, in which these coefficients are still suitable for reconstruction. ISOMAP preserves global geodesic distances of all the pairs of examples.

Visible-NIR spectral characterization and grade inversion modelling study of the Derni copper deposit

The diffusion geometry of fibre bundles: Horizontal diffusion maps

Chapter 19 - Dimensionality Reduction and Latent Variable Modeling

At present, manifold learning is mainly applied to dimensionality reduction. However, from viewpoint of dimensionality reduction, manifold learning algorithms are only local feature preserving algorithms. For example, Local Linear Embedding is local linear preserving, Local Tangent Space Alignment is local homeomorphic preserving and Laplacian Eigenmap is local similarity preserving. The community of dimensionality reduction is now pursuing the algorithms which can preserve both local and global features of data during dimensionality reduction. In this paper, a new algorithm of dimensionality reduction, called Hilbert-Schmidt Independence Criterion Regularized Manifold Learning (HSIC-ML for short), is proposed, in which HSIC between the high dimensional data and the dimension-reduced data is added as a regularization term to the objective functions of manifold learning. The addition of HSIC regularization term makes HSIC-ML capable of preserving both local and global features during dimensionality reduction. HSIC is a criterion measuring the statistical dependence between two data sets and has been widely applied to machine learning in recent years. However, since HSIC was first proposed around 2005, there seems to have not been applied directly to dimensionality reduction, not applied as a regularization term either. The proposed HSIC-ML may be the first try in this respect. The experimental results presented in this paper show that the manifold learning with HSIC regularization performs better than that without HSIC regularization.

This work presents a comparative analysis between the linear combination of em-bedded spaces resulting from two approaches: (1) The application of dimensional reduction methods (DR) in their standard implementations, and (2) Their corresponding kernel-based approximations. Namely, considered DR methods are: CMDS (Classical Multi- Dimensional Scaling), LE (Laplacian Eigenmaps) and LLE (Locally Linear Embedding). This study aims at determining -through objective criteria- what approach obtains the best performance of DR task for data visualization. The experimental validation was performed using four databases from the UC Irvine Machine Learning Repository. The quality of the obtained embedded spaces is evaluated regarding the \({\varvec{R_{NX}(K)}}\) criterion. The \({\varvec{R_{NX}(K)}}\) allows for evaluating the area under the curve, which indicates the performance of the technique in a global or local topology. Additionally, we measure the computational cost for every comparing experiment. A main contribution of this work is the provided discussion on the selection of an interactivity model when mixturing DR methods, which is a crucial aspect for information visualization purposes.

Usually, most of the data generated in real-world such as images, speech signals, or fMRI scans has a high dimensionality. Therefore, dimensionality reduction techniques can be used to reduce the number of variables in that data and then the system performance can be improved. Because the processing of the high dimensional data leads the increase of complexity both in execution time and memory usage. In the previous work, we developed an offline writer identification system using a combination of Oriented Basic Image features (OBI) and the concept of graphemes codebook. In order to measure and optimise the system performance, a variety of nonlinear dimensionality reduction algorithms such as Kernel Principal Component Analysis (KPCA), Isomap, Locally linear embedding (LLE), Hessian LLE and Laplacian Eigenmaps have been used. The performance has been evaluated based on IAM dataset for English handwriting and ICFHR 2012 dataset for Arabic handwriting. The results obtained indicated the system performance based KPCA was better than the other reduction techniques that have been used and investigated in this work.

Manifold learning is a field of study in machine learning and statistics that is closely associated with dimensionality reduction algorithmic techniques is gaining popularity these days. There are two types of manifold learning approaches: linear and nonlinear. Principal component analysis (PCA) and multidimensional scaling (MDS) are two examples of linear techniques that have long been staples in the statistician's arsenal for evaluating multivariate data. Nonlinear manifold learning, which encompasses diffusion maps, Laplacian Eigenmaps, Hessian Eigenmaps, Isomap, and local linear embedding, has seen a surge in research effort recently. A few of these methods are nonlinear extensions of linear approaches. A nearest search, the definition of distances or affinities between points (a crucial component of these methods' effectiveness), and an Eigen problem for embedding high-dimensional points into a lower dimensional space make up the algorithmic process of the majority of these techniques. The strengths and weaknesses of the new method are briefly reviewed in this article. In the field of computer graphics, we utilize a particular manifold learning method was first presented in statistics and machine learning to create a global, Spectral-based shape descriptor.

We review the ideas, algorithms, and numerical performance of manifold-based machine learning and dimension reduction methods. The representative methods include locally linear embedding (LLE), ISOMAP, Laplacian eigenmaps, Hessian eigenmaps, local tangent space alignment (LTSA), and charting. We describe the insights from these developments, as well as new opportunities for both researchers and practitioners. Potential applications in image and sensor data are illustrated. This chapter is based on an invited survey presentation that was delivered by Huo at the 2004 INFORMS Annual Meeting, which was held in Denver, CO, U.S.A.