Semi-supervised Local Discriminant Embedding

Semisupervised Discriminant Analysis (SDA) aims at dimensionality reduction with both limited labeled data and copious unlabeled data, but it may fail to discover the intrinsic geometry structure when the data manifold is highly nonlinear. The kernel trick is widely used to map the original nonlinearly separable problem to an intrinsically larger dimensionality space where the classes are linearly separable. Inspired by low-rank representation (LLR), we proposed a novel kernel SDA method called low-rank kernel-based SDA (LRKSDA) algorithm where the LRR is used as the kernel representation. Since LRR can capture the global data structures and get the lowest rank representation in a parameter-free way, the low-rank kernel method is extremely effective and robust for kinds of data. Extensive experiments on public databases show that the proposed LRKSDA dimensionality reduction algorithm can achieve better performance than other related kernel SDA methods.

Hyperspectral image (HSI) has shown promising results in many fields because of its high spectral resolution. However, redundancy and noise in spectral dimension seriously affect the classification of HSI. For this reason, many popular dimensionality reduction (DR) methods are proposed to solve the problem. The local discriminant embedding (LDE) as an effective non-linear method for DR can be more discriminative by constructing two neighborhood graphs. However, HSI is very easy influenced by noise, and the LDE algorithm based on K nearest neighborhood is highly susceptible to interference from extreme point, which may lead to inaccurate graph construction and poor performance of classification. To overcome the problem and retain the advantages of LDE, a modified local discriminant embedding (MLDE) is firstly applied on HSI by constructing neighborhood graphs on a new spectral feature space instead of the original space. We use variance to characterize the pixels similarity of the same class and use covariance to characterize the separation of different classes of pixels. The combination of variance and covariance makes pixels in the same class to be closer and makes greater separation of pixels from different classes, which enhances classification performance of HSI. The way of representing data by using variance and covariance can attenuate the effects of noise. The Log-Euclidean metric is used to capture the similarity between spectral vectors, which can provide a more accurate similarity evaluation than euclidean distance. The experimental results of two hyperspectral datasets demonstrate the effectiveness of our proposed MLDE method.

Recently, deep learning techniques achieve remarkable classification performance on histopathology images. How-ever, they usually require a large amount of labeled training images to obtain satisfactory accuracy, and manual labeling is labor expensive and time consuming. To address this issue, in this paper, we propose a novel semi-supervised deep learning framework, namely semi-supervised deep linear discriminant analysis, by taking advantage of the deep neural network (DNN) and the graph to simultaneously exploit the semantic information of labeled and unlabeled images for classification. Specifically, we first replace the loss function of the DNN with the objective function of linear discriminant analysis to produce features minimizing the intra-class distance yet maximizing the inter-class distance, in order to construct a robust and effective graph Laplacian. Afterwards, we design a new objective function via employing the graph constructed by features of labeled and un-labeled images, and then adopt this objective as the loss function of the DNN to produce features for classification. Experiments on skeletal muscle and lung cancer images demonstrate the proposed framework outperforms several recent state of the arts.

Local Discriminant Embedding (LDE) was recently proposed to overcome some limitations of the global Linear Discriminant Analysis (LDA) method. In the case of a small training data set, however, LDE cannot directly be applied to high-dimensional data. This case is the so-called small-sample-size (SSS) problem. In this paper, we introduce an Exponential Local Discriminant Embedding (ELDE) technique to overcome the SSS problem. The advantages of ELDE are that, compared with Principal Component Analysis (PCA) + LDE, the ELDE method can extract the most discriminant information that was contained in the null space of the locality preserving between-class and within-class scatter matrices. In addition, the proposed ELDE is equivalent to transforming original data into a new space by distance diffusion mapping, and then, LDE is applied in such a new space. As a result of diffusion mapping, the margin between samples belonging to different classes is enlarged, which is helpful in improving classification accuracy. The experiments are conducted on four public face databases, YALE, PIE, Extended Yale and PF01. Experiments conducted on real data show that the performance of ELDE is better than that of LDE and many state-of-the-art discriminant analysis techniques.

Dimensionality reduction is the process of mapping high-dimension patterns to a lower dimension subspace. When done prior to classification, estimates obtained in the lower dimension subspace are more reliable. For some classifiers, there is also an improvement in performance due to the removal of the diluting effect of redundant information. A majority of the present approaches to dimensionality reduction are based on scatter matrices or other statistics of the data which do not directly correlate to classification accuracy. The optimality criteria of choice for the purposes of classification is the Bayes error. Usually however, Bayes error is difficult to express analytically. We propose an optimality criteria based on an approximation of the Bayes error and use it to formulate a linear and a nonlinear method of dimensionality reduction. The nonlinear method we propose, relies on using a multilayered perceptron which produces as output the lower dimensional representation. It thus differs from autoassociative like multilayered perceptrons which have been proposed and used for dimensionality reduction. Our results show that the nonlinear method is, as anticipated, superior to the linear method in that it can perform unfolding of a nonlinear manifold. In addition, the nonlinear method we propose provides substantially better lower dimension representation (for classification purposes) than Fisher's linear discriminant (FLD) and two other nonlinear methods of dimensionality reduction that are often used.

As one of the most popular dimensionality reduction techniques, locality preserving projections (LPP) has been widely used in computer vision and pattern recognition. However, in practical applications, data is always corrupted by noises. For the corrupted data, samples from the same class may not be distributed in the nearest area, thus LPP may lose its effectiveness. In this paper, it is assumed that data is grossly corrupted and the noise matrix is sparse. Based on these assumptions, we propose a novel dimensionality reduction method, named low-rank preserving projections (LRPP) for image classification. LRPP learns a low-rank weight matrix by projecting the data on a low-dimensional subspace. We use the L21 norm as a sparse constraint on the noise matrix and the nuclear norm as a low-rank constraint on the weight matrix. LRPP keeps the global structure of the data during the dimensionality reduction procedure and the learned low rank weight matrix can reduce the disturbance of noises in the data. LRPP can learn a robust subspace from the corrupted data. To verify the performance of LRPP in image dimensionality reduction and classification, we compare LRPP with the state-of-the-art dimensionality reduction methods. The experimental results show the effectiveness and the feasibility of the proposed method with encouraging results.

The extraction of concise and discriminative features from hyperspectral remote sensing images (HSIs) holds great significance for land cover classification tasks. Nevertheless, the intricate spatial-spectral tensor structure and inaccurate class labels undermine the performance of existing vector-based or Tucker tensor-based feature dimensionality reduction (DR) approaches. To tackle this problem, we propose an Adaptive Tensor-Ring Manifold Maintaining Embedding (ATRMME) approach for HSI feature DR in the presence of noisy labels. Firstly, a tensor-ring subspace representation theory is established to characterize the low-dimensional hyperspectral tensor subspace. To mine concise features from complex spatial-spectral data, differing from existing Tucker-based DR methods that involve exorbitant parameter overheads, a nuclear norm-constrained Tensor-Ring (TR) feature representation term is designed. This term enables the compression of key features within the low-dimensional subspace with a linear parameter cost. For the adaptive perception and capture of the manifold structure under noisy labels, an adaptive adjacency matrix is constructed, and the adjacency relationships are reorganized based on the TR feature distribution. To guarantee the effective execution of model training, an alternating optimization algorithm is developed for parameter estimation. Experimental results on multiple hyperspectral datasets verify that the ATRMME approach exhibits strong robustness against noisy labels, outperforms conventional vector-based and Tucker tensor-based methods, and has lower parameter requirements.

This paper proposes a new manifold learning-based scheme for multi-fault detection and classification, which utilizes local and nonlocal embedding method to build a statistic index for fault detection and subsequently develops a joint global and local discriminant embedding (GLDE) approach to discover the discriminant features of multiple faults for fault classification. The proposed GLDE approach can capture the global and local/nonlocal structure information of complicated data and obtain the concise discriminant information for classification. Compared with the conventional Fisher discriminant analysis method, GLDE has a strong discriminant power and provides better monitoring results for complex multi-fault Tennessee Eastman process.

Supervised feature extraction algorithms usually require lots of labeled samples to achieve good performance. However, labeling the samples is often time-consuming and even impractical. Therefore, in this paper, a semi-supervised manifold local Fisher discriminant analysis (SMLFDA) is proposed to take advantage of unlabeled samples as well as labeled samples. The proposed algorithm utilizes local scatter matrix and manifold structure to extract the information from labeled and unlabeled samples, respectively, which significantly improves the accuracy of successive classification application when labeled samples are insufficient. In addition, an exponential form weighting coefficient is proposed to further improve the classification performance. Experiments of hyperspectral classification demonstrate the effectiveness of the proposed semi-supervised feature extraction algorithm.

Extracting discriminative features and reducing the dimensionality of data are two main objectives of manifold learning. Among different techniques, nonlinear manifold learning methods have been proposed in order to extract features from data which are not linearly distributed. Kernel trick is one of the famous nonlinear techniques which helps to project the data without an explicit mapping which can be used in combination with different linear techniques (e.g., Linear discriminant analysis and local discriminant embedding (LDE)). In this paper, we propose a Two Subspace-based Kernel Local Discriminant Embedding (TSKLDE) method which extract features from both non-null and null space of the within-class locality preserving scatter matrix of LDE in the kernel space. We evaluated the proposed algorithm using three publicly available face databases. The obtained results demonstrate that the use of both features in TSKLDE leads to more noise tolerant features compared to other kernel methods and to higher discriminant ability than many existing manifold learning techniques.

We consider popular dimension reduction techniques that project data on a low dimensional subspace. They include Principal Component Analysis, Column Subset Selection, and Johnson-Lindenstrauss projections. These techniques have been classically used to efficiently compute various approximations. We propose the following three-step procedure for enhancing the accuracy of such approximations: 1. Unknown quantities in the approximation are replaced with random variables. 2. The Maximum Entropy method is applied to infer the most likely probability distribution. 3. Expected values of the random variables are used to compute the enhanced estimates. Our use of the Maximum Entropy method requires knowledge of vector norms that can be easily computed during the dimension reduction. We demonstrate significant enhancements in average accuracy for Euclidean distance and Mahalanobis distance, and improvements in evaluating k-nearest neighbors and k-furthest neighbors by using the enhanced Euclidean distance formula.

Multi-view Sparsity Preserving Projection for dimension reduction

We present a subspace learning method, called local discriminant embedding with tensor representation (LDET), that addresses simultaneously the generalization and data representation problems in subspace learning. LDET learns multiple interrelated subspaces for obtaining a lower-dimensional embedding by incorporating both class label information and neighborhood information. By encoding each object as a second- or higher-order tensor, LDET can capture higher-order structures in the data without requiring a large sample size. Extensive empirical studies have been performed to compare LDET with a second- or third-order tensor representation and the original LDE on their face recognition performance. Not only does LDET have a lower computational complexity than LDE, but LDET is also superior to LDE in terms of its recognition accuracy.

The solution of the unconstrained weighted linear least-squares problem is known to be a convex combination of the basic solutions formed by the nonsingular subsystems if the weight matrix is diagonal and positive definite. In particular, this implies that the norm of this solution is uniformly bounded for any diagonal and positive definite weight matrix. In addition, the solution set is known to be the relative interior of a finite set of polytopes if the weight matrix varies over the set of positive definite diagonal matrices. In this paper, these results are reviewed and generalized to the set of weight matrices that are symmetric, positive semidefinite, and diagonally dominant and that give unique solution to the least-squares problem. This is done by means of a particular symmetric diagonal decomposition of the weight matrix, giving a finite number of diagonally weighted problems but in a space of higher dimension. Extensions to equality-constrained weighted linear least-squares problems are given. A discussion of why the boundedness properties do not hold for general symmetric positive definite weight matrices is given. The motivation for this research is from interior methods for optimization.

Spectral clustering requires the time-consuming decomposition of the Laplacian matrix of the similarity graph, thus limiting its applicability to large datasets. To improve the efficiency of spectral clustering, a top-down approach was recently proposed, which first divides the data into several micro-clusters (granular-balls), then splits these micro-clusters when they are not ``compact'', and finally uses these micro-clusters as nodes to construct a similarity graph for more efficient spectral clustering. However, this top-down approach is challenging to adapt to unevenly distributed or structurally complex data. This is because constructing micro-clusters as a rough ball struggles to capture the shape and structure of data in a local range, and the simplistic splitting rule that solely targets ``compactness'' is susceptible to noise and variations in data density and leads to micro-clusters with varying shapes, making it challenging to accurately measure the similarity between them. To resolve these issues and improve spectral clustering, this paper first proposes to start from local structures to obtain micro-clusters, such that the complex structural information inside local neighborhoods is well captured by them. Moreover, by noting that Euclidean distance is more suitable for convex sets, this paper further proposes a data splitting rule that couples local density and data manifold structures, so that the similarities of the obtained micro-clusters can be easily characterized. A novel similarity measure between micro-clusters is then proposed for the final spectral clustering. A series of experiments based on synthetic and real-world datasets demonstrate that the proposed method has better adaptability to structurally complex data than granular-ball based methods.