Supervised Discriminative Sparse PCA with Adaptive Neighbors for Dimensionality Reduction

Over the years, many dimensionality reduction algorithms have been proposed for learning the structure of high dimensional data by linearly or non-linearly transforming it into a low-dimensional space. Some techniques can keep the local structure of data, while the others try to preserve the global structure. In this paper, we propose a linear dimensionality reduction technique that characterizes the local and global properties of data by firstly applying k-means algorithm on original data, and then finding the projection by simultaneously globally maximizing the between-cluster scatter matrix and locally minimizing the within-cluster scatter matrix, which actually keeps both local and global structure of data. Low complexity and structure preserving are two main advantages of the proposed technique. The experiments on both artificial and real data sets show the effectiveness and novelty of proposed algorithm in visualization and classification tasks.

The linear discriminant analysis (LDA) is a very popular linear feature extraction approach. The algorithms of LDA usually perform well under the following two assumptions. The first assumption is that the global data structure is consistent with the local data structure. The second assumption is that the input data classes are Gaussian distributions. However, in real-world applications, these assumptions are not always satisfied. In this paper, we propose an improved LDA framework, the local LDA (LLDA), which can perform well without needing to satisfy the above two assumptions. Our LLDA framework can effectively capture the local structure of samples. According to different types of local data structure, our LLDA framework incorporates several different forms of linear feature extraction approaches, such as the classical LDA and principal component analysis. The proposed framework includes two LLDA algorithms: a vector-based LLDA algorithm and a matrix-based LLDA (MLLDA) algorithm. MLLDA is directly applicable to image recognition, such as face recognition. Our algorithms need to train only a small portion of the whole training set before testing a sample. They are suitable for learning large-scale databases especially when the input data dimensions are very high and can achieve high classification accuracy. Extensive experiments show that the proposed algorithms can obtain good classification results.

A new nonlinear dimensionality reduction method called data-dependent kernel global–local preserving projections (DDKGLPP) is proposed and used for process monitoring. To achieve performance improvements, DDKGLPP uses a data-dependent kernel rather than a conventional kernel. A unified kernel optimization framework is developed to optimize the data-dependent kernel by minimizing a data structure preserving index. The optimized kernel can unfold both global and local data structures in the feature space. The data-dependent kernel principal component (DDKPCA) and data-dependent kernel locality preserving projections (DDKLPP) also can be developed under the unified kernel optimization framework. However, unlike DDKPCA and DDKLPP, DDKGLPP is able to preserve both global and local structures of the data set when performing dimensionality reduction. Consequently, DDKGLPP is more powerful in capturing useful data characteristics. A DDKGLPP-based monitoring method is then proposed for nonlinear processes. Its per...

The dimension reduction methods have been proved powerful and practical to extract latent features hidden in the signal for process monitoring. A novel nonlinear dimension reduction method called kernel orthogonal global-local preserving projections (KOGLPP) is proposed and applied for condition monitoring and fault detection. To overcome the shortcomings of kernel locality preserving projections (KLPP) and kernel principal component analysis (KPCA), the KOGLPP model aims at preserving the global and local data structures simultaneously by constructing a dual-objective optimization function, and a tuning parameter is introduced to adjust the trade-off between the global and local data structures. For the purpose of condition monitoring and fault detection, monitoring statistics are constructed in low dimensional feature space. As KOGLPP combines the advantages of both KPCA and KLPP, KOGLPP is also more powerful in extracting potential useful data characteristics. Finally, the effectiveness of the proposed nonlinear dimension reduction method is evaluated experimentally on a numerical example and a bearing test-rig. The results indicate its potential applications as an effective and reliable tool for condition monitoring and fault detection.

Identifying the key genes related to tumors from gene expression data with a large number of features is important for the accurate classification of tumors and to make special treatment decisions. In recent years, unsupervised feature selection algorithms have attracted considerable attention in the field of gene selection as they can find the most discriminating subsets of genes, namely the potential information in biological data. Recent research also shows that maintaining the important structure of data is necessary for gene selection. However, most current feature selection methods merely capture the local structure of the original data while ignoring the importance of the global structure of the original data. We believe that the global structure and local structure of the original data are equally important, and so the selected genes should maintain the essential structure of the original data as far as possible. In this paper, we propose a new, adaptive, unsupervised feature selection scheme which not only reconstructs high-dimensional data into a low-dimensional space with the constraint of feature distance invariance but also employs -norm to enable a matrix with the ability to perform gene selection embedding into the local manifold structure-learning framework. Moreover, an effective algorithm is developed to solve the optimization problem based on the proposed scheme. Comparative experiments with some classical schemes on real tumor datasets demonstrate the effectiveness of the proposed method.

Recently, low-rank and sparse model-based dimensionality reduction (DR) methods have aroused lots of interest. In this paper, we propose an effective supervised DR technique named block-diagonal constrained low-rank and sparse-based embedding (BLSE). BLSE has two steps, i.e., block-diagonal constrained low-rank and sparse representation (BLSR) and block-diagonal constrained low-rank and sparse graph embedding (BLSGE). Firstly, the BLSR model is developed to reveal the intrinsic intra-class and inter-class adjacent relationships as well as the local neighborhood relations and global structure of data. Particularly, there are mainly three items considered in BLSR. First, a sparse constraint is required to discover the local data structure. Second, a low-rank criterion is incorporated to capture the global structure in data. Third, a block-diagonal regularization is imposed on the representation to promote discrimination between different classes. Based on BLSR, informative and discriminative intra-class and inter-class graphs are constructed. With the graphs, BLSGE seeks a low-dimensional embedding subspace by simultaneously minimizing the intra-class scatter and maximizing the inter-class scatter. Experiments on public benchmark face and object image datasets demonstrate the effectiveness of the proposed approach.

Magnetic resonance imaging and computed tomography produce three-dimensional volumetric medical images. While a scalar value represents each individual volume element, or voxel, volumetric data are characterized by features derived from groups of neighboring voxels and their inherent relationships, which may vary depending on the specific clinical application. Labeled samples are also required in most applications, which can be problematic for large datasets such as medical images. We propose a direct volume rendering (DVR) framework based on multi-scale dimensionality reduction neighbor embedding that generates two-dimensional transfer function (TF) domains. In this way, we present FSS.t-SNE, a fast semi-supervised version of the t-distributed stochastic neighbor embedding (t-SNE) method that works over hundreds of thousands of voxels without the problem of crowding and with better separation in a 2D histogram compared to traditional TF domains. Our FSS.t-SNE scatters voxels of the same sub-volume in a wider region through multi-scale neighbor embedding, better preserving both local and global data structures and allowing for its internal exploration based on the original features of the multi-dimensional space, taking advantage of the partially provided labels. Furthermore, FSS.t-SNE untangles sample paths among sub-volumes, allowing us to explore edges and transitions. In addition, our approach employs a Barnes–Hut approximation to reduce computational complexity from O(N2) (t-SNE) to O(NlogN). Although we require the additional step of generating the 2D TF domain from multiple features, our experiments show promising performance in volume segmentation and visual inspection.

Subspace learning and reconstruction have been widely explored in recent transfer learning work and generally a specially designed projection and reconstruction transfer matrix are wanted. However, existing subspace reconstruction based algorithms neglect the class prior such that the learned transfer function is biased, especially when data scarcity of some class is encountered. Different from those previous methods, in this paper, we propose a novel reconstruction-based transfer learning method called Class-specific Reconstruction Transfer Learning (CRTL), which optimizes a well-designed transfer loss function without class bias. Using a class-specific reconstruction matrix to align the source domain with the target domain which provides help for classification with class prior modeling. Furthermore, to keep the intrinsic relationship between data and labels after feature augmentation, a projected Hilbert-Schmidt Independence Criterion (pHSIC), that measures the dependency between two sets, is first proposed by mapping the data from original space to RKHS in transfer learning. In addition, combining low-rank and sparse constraints on the class-specific reconstruction coefficient matrix, the global and local data structures can be effectively preserved. Extensive experiments demonstrate that the proposed method outperforms conventional representation-based domain adaptation methods.

Feature extraction by learning Lorentzian metric tensor and its extensions

Despite the widespread adoption of deep learning models, shallow machine learning (SML) algorithms are still used for image classification due to simplicity, interpretability and efficiency. This study aims to bridge this gap by investigating the robustness dynamics of SML techniques under more complex scenarios, such as adversarial perturbations and geometric transformation. Five popular classification algorithms, including k-nearest neighbour and support vector machines, were employed to build classification models. Methodology involves investigating the robustness of proposed methods, first, on original and corrupted data by utilising benchmark datasets across several image domains. To strengthen the investigation, the models were trained using a new low-rank representation (LRR) strategy. This hybrid model simultaneously addresses two key limitations in classical LRR models: overcoming the sequential learning process and effectively capturing both local and global data structures. By introducing a dual regularisation mechanism, it integrates a k-nearest neighbour graph to preserve local consistency, while a global low-rank constraint ensures coherent data representation. Experimental results reveal significant drops in accuracy of most SML methods, especially under adversarial attacks and geometric transformations, but LRR approach mitigates these effects to a notable extent, boosting performance across data variations. The results also show that the proposed method outperforms state-of-the-art LRR techniques in most experiments.

Dimensionality reduction plays a fundamental role in data processing, for which principal component analysis (PCA) is widely used. In this paper, we develop the Laplacian PCA (LPCA) algorithm which is the extension of PCA to a more general form by locally optimizing the weighted scatter. In addition to the simplicity of PCA, the benefits brought by LPCA are twofold: the strong robustness against noise and the weak metric-dependence on sample spaces. The LPCA algorithm is based on the global alignment of locally Gaussian or linear subspaces via an alignment technique borrowed from manifold learning. Based on the coding length of local samples, the weights can be determined to capture the local principal structure of data. We also give the exemplary application of LPCA to manifold learning. Manifold unfolding (non-linear dimensionality reduction) can be performed by the alignment of tangential maps which are linear transformations of tangent coordinates approximated by LPCA. The superiority of LPCA to PCA and kernel PCA is verified by the experiments on face recognition (FRGC version 2 face database) and manifold (Scherk surface) unfolding.

In this paper, we propose a manifold geometry based projective nonnegative matrix factorization linear dimensionality reduction method, called neighborhood preserving orthogonal projective nonnegative matrix factorization (NPOPNMF), for feature extraction of hyperspectral image. By adding constraints on projective nonnegative matrix factorization (PNMF) that each data point can be represented as a linear combination of its neighbors, NPOPNMF preserves local neighborhood geometrical structure of hyperspectral data in the reduced space, and overcomes the Euclidean limitation of PNMF. The metric structure of original high-dimensional hyperspectral data space is preserved due to the orthogonality of projection matrix. NPOPNMF can be performed in either supervised or unsupervised mode according to the construction of adjacency graph and it can improve the discriminant performance of PNMF. Theoretical analysis and experimental results on hyperspectral data sets demonstrate that the proposed method is an effective and promising method for hyperspectral image feature extraction.

In the last few years, great efforts have been made to extend the linear projection technique (LPT) for multidimensional data (i.e., tensor), generally referred to as the multilinear projection technique (MPT). The vectorized nature of LPT requires high-dimensional data to be converted into vector, and hence may lose spatial neighborhood information of raw data. MPT well addresses this problem by encoding multidimensional data as general tensors of a second or even higher order. In this paper, we propose a novel multilinear projection technique, called multilinear spatial discriminant analysis (MSDA), to identify the underlying manifold of high-order tensor data. MSDA considers both the nonlocal structure and the local structure of data in the transform domain, seeking to learn the projection matrices from all directions of tensor data that simultaneously maximize the nonlocal structure and minimize the local structure. Different from multilinear principal component analysis (MPCA) that aims to preserve the global structure and tensor locality preserving projection (TLPP) that is in favor of preserving the local structure, MSDA seeks a tradeoff between the nonlocal (global) and local structures so as to drive its discriminant information from the range of the non-local structure and the range of the local structure. This spatial discriminant characteristic makes MSDA have more powerful manifold preserving ability than TLPP and MPCA. Theoretical analysis shows that traditional MPTs, such as multilinear linear discriminant analysis, TLPP, MPCA, and tensor maximum margin criterion, could be derived from the MSDA model by setting different graphs and constraints. Extensive experiments on face databases (ORL, CMU PIE, and the extended Yale-B) and the Weizmann action database demonstrate the effectiveness of the proposed MSDA method.

Multiple kernel locality-constrained collaborative representation-based discriminant projection for face recognition

Local Linear Embedding with Adaptive Neighbors