Nonconvex low-rank tensor approximation with graph and consistent regularizations for multi-view subspace learning

Advances in single-cell biotechnologies have generated the single-cell RNA sequencing (scRNA-seq) of gene expression profiles at cell levels, providing an opportunity to study cellular distribution. Although significant efforts developed in their analysis, many problems remain in studying cell types distribution because of the heterogeneity, high dimensionality, and noise of scRNA-seq. In this study, a multi-view clustering with graph learning algorithm (MCGL) for scRNA-seq data is proposed, which consists of multi-view learning, graph learning, and cell type clustering. In order to avoid a single feature space of scRNA-seq being inadequate to comprehensively characterize the functions of cells, MCGL constructs the multiple feature spaces and utilizes multi-view learning to comprehensively characterize scRNA-seq data from different perspectives. MCGL adaptively learns the similarity graphs of cells that overcome the dependence on fixed similarity, transforming scRNA-seq analysis into the analysis of multi-view clustering. MCGL decomposes the networks of cells into view-specific and common networks in multi-view learning, which better characterizes the topological relationship of cells. MCGL simultaneously utilizes multiple types of cell-cell networks and fully exploits the connection relationship between cells through the complementarity between networks to improve clustering performance. The graph learning, graph factorization, and cell-type clustering processes are accomplished simultaneously under one optimization framework. The performance of the MCGL algorithm is validated with ten scRNA-seq datasets from different scales, and experimental results imply that the proposed algorithm significantly outperforms fourteen state-of-the-art scRNA-seq algorithms.

Extracting information from large-scale high-dimensional data is a fundamentally important task in high performance computing, where the hierarchical Tucker (HT) tensor learning approach (learning a tensor-tree structure) has been widely used in many applications. However, HT tensor learning algorithms are compute-intensive due to the " <i>curse of dimensionality</i> ," i.e., the time complexity grows exponentially with the order of the data tensor. The computation of HT tensor learning algorithms boils down to tensor primitives, which are amenable to computing on GPU tensor cores. Existing work does not support HT tensor learning using GPU tensor cores. There are three main challenges to address: 1) to accelerate tensor learning primitives using GPU tensor cores; 2) to implement the tensor learning algorithms using GPU tensor cores and multiple GPUs; 3) to support large-scale data tensors exceeding the GPU memory capacity. In this paper, we present efficient HT tensor learning primitives using GPU tensor cores and demonstrate three applications. First, we utilize GPU tensor cores to optimize HT tensor learning primitives, including tensor contractions, tensor matricizations and tensor singular value decomposition (SVD). We employ the optimized primitives to optimize HT tensor decomposition algorithms for Big Data analysis. Second, we propose a novel HT tensor layer for deep neural networks, whose training process only involves a forward pass without back propagation. The forward pass consists of tensor operations, thus further exploiting the computing power of GPU tensor cores. Third, we apply the optimized primitives to develop a tensor-tree structured quantum machine learning algorithm <i>tree-tensor network (TTN)</i> . Compared with TensorLy and TensorNetwork on NVIDIA A100 GPUs, our third-order HT tensor decomposition algorithm achieves up to <inline-formula><tex-math notation="LaTeX">$8.92 \times$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$6.42 \times$</tex-math></inline-formula> speedups, respectively, and our high-order case achieves up to <inline-formula><tex-math notation="LaTeX">$32.67 \times$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$23.97 \times$</tex-math></inline-formula> speedups, respectively. Our HT tensor layer for a fully connected neural network achieves <inline-formula><tex-math notation="LaTeX">$49.2 \times$</tex-math></inline-formula> compression at the cost of 0.5% drops in accuracy and <inline-formula><tex-math notation="LaTeX">$1.42 \times$</tex-math></inline-formula> speedup compared with the implementation on CUDA cores; for the AlexNet, our HT tensor layer achieves <inline-formula><tex-math notation="LaTeX">$9.45 \times$</tex-math></inline-formula> compression at the cost of 0.8% drops in accuracy and <inline-formula><tex-math notation="LaTeX">$1.87 \times$</tex-math></inline-formula> speedup compared with the implementation on CUDA cores. Our TTN algorithm achieves up to <inline-formula><tex-math notation="LaTeX">$11.17\times$</tex-math></inline-formula> speedup compared with TensorNetwork, indicating the potential of optimized tensor learning primitives for the classical simulation of quantum machine learning algorithms.

Recent researches on multiview learning have received widespread attention due to the increasing generalization of multiview data. As an effective probabilistic model, random walk has also shown encouraging performance in various fields. To further exploit the potential of utilizing random walk schemes to address multiview learning problems, this article proposes a simple yet efficient multigraph random walk scheme for both multiview clustering and semisupervised classification tasks. The proposed model integrates random walk with multiview learning, and recursively learns a globally stable probability distribution matrix from multiple views, on the basis of which the label indicator is obtained in the scene of clustering or semisupervised classification. Furthermore, an adaptive weight vector is learned to incorporate the diversity and complementarity of multiview data. Besides, the relationships between the proposed scheme and spectral clustering, neighborhood embedding and manifold embedding are analyzed theoretically. Finally, comprehensive comparative experiments are conducted with several state-of-the-art multiview clustering and semisupervised classification methods on eight real-world datasets. The experimental results demonstrate the superiority of the proposed method in terms of both clustering and classification performance.

Significant efforts have been focused on enhancing the utilization of multiple node features and topological structures in multi-view graph learning through explicit model-driven and implicit deep learning-based methodologies. The former excels in embedding prior knowledge, thereby offering theoretical interpretability but is limited in application flexibility due to manual parameter selection. In contrast, the latter leverages automatic differentiation, providing greater flexibility but lacking theoretical interpretability due to their opaque nature. Motivated by these observations, we propose an interpretable deep unfolding network for mutual-benefit multi-view graph learning, aiming to combine the strengths of both approaches. Specifically, we employ the Alternating Direction Method of Multipliers (ADMM) to solve a multi-view graph learning model with sparse and low-rank constraints. This solution is then integrated into deep unfolding networks to enhance interpretability. Furthermore, we convert optimization conditions into implicit losses and utilize automatic differentiation to update parameters, reducing the need for manual tuning and increasing flexibility. This integration optimizes multi-view learning for a graph representation that balances interpretability and flexibility. Empirical evaluations on six diverse datasets demonstrate the effectiveness and superiority of the proposed method over state-of-the-art approaches.

Multi-view clustering utilizes information from diverse views to improve the performance of clustering. For most existing multi-view spectral clustering methods, information of different views is integrated by pursuing a consensus similarity matrix for clustering. However, view-specific structures, which contain the complementary information of multi-view data, may be lost during the clustering process. Actually, in multi-view spectral clustering, similarity matrices of multiple views would have the same clustering structures or properties rather than be numerically uniform. To overcome the aforementioned problem, a novel essential multi-view graph learning (EMGL) method for clustering is proposed in this paper. Different from most existing multi-view spectral clustering, an orthogonal matrix factorization is imposed on multi-view similarity matrices for making them have the same nuclear norm, which indicates the same clustering structures of different views. Furthermore, we also propose an alternating direction method of multipliers (ADMM) based optimization algorithm to address the objective function of our method. Extensive experiments on several datasets demonstrate the superior performance of our proposed method.

&lt;p style='text-indent:20px;'&gt;In 2011, Kilmer and Martin proposed tensor singular value decomposition (T-SVD) for third order tensors. Since then, T-SVD has applications in low rank tensor approximation, tensor recovery, multi-view clustering, multi-view feature extraction, tensor sketching, etc. By going through the Discrete Fourier Transform (DFT), matrix SVD and inverse DFT, a third order tensor is mapped to an f-diagonal third order tensor. We call this a Kilmer-Martin mapping. We show that the Kilmer-Martin mapping of a third order tensor is invariant if that third order tensor is taking T-product with some orthogonal tensors. We define singular values and T-rank of that third order tensor based upon its Kilmer-Martin mapping. Thus, tensor tubal rank, T-rank, singular values and T-singular values of a third order tensor are invariant when it is taking T-product with some orthogonal tensors. Some properties of singular values, T-rank and best T-rank one approximation are discussed.&lt;/p&gt;

Multi-view semi-supervised learning for classification on dynamic networks

Graph learning for latent-variable Gaussian graphical models under laplacian constraints

Self-taught Multi-view Spectral Clustering

Adaptive multi-view subspace learning based on distributed optimization

Multi-view subspace learning via bidirectional sparsity

Deep multi-view representation learning focuses on training a unified low-dimensional representation for data with multiple sources or modalities. With the rapidly growing attention of graph neural networks, more and more researchers have introduced various graph models into multi-view learning. Although considerable achievements have been made, most existing methods usually propagate information in a single view and fuse multiple information only from the perspective of attributes or relationships. To solve the aforementioned problems, we propose an efficient model termed Dual Fusion-Propagation Graph Neural Network (DFP-GNN) and apply it to deep multi-view clustering tasks. The proposed method is designed with three submodules and has the following merits: a) The proposed view-specific and cross-view propagation modules can capture the consistency and complementarity information among multiple views; b) The designed fusion module performs multi-view information fusion with the attributes of nodes and the relationships among them simultaneously. Experiments on popular databases show that DFP-GNN achieves significant results compared with several state-of-the-art algorithms.

In recent years, block-based compressive sensing (BCS) has been extensively studied because it can reduce computational complexity and data storage by dividing the image into smaller patches, but the performance of the reconstruction algorithm is not satisfactory. In this paper, a new reconstruction model for image and video is proposed. The model makes full use of spatio-temporal correlation and utilizes low-rank tensor approximation to improve the quality of the reconstructed image and video. For image recovery, the proposed model obtains a low-rank approximation of a tensor formed by non-local similar patches, and improves the reconstruction quality from a spatial perspective by combining non-local similarity and low-rank property. For video recovery, the reconstruction process is divided into two phases. In the first phase, each frame of the video sequence is regarded as an independent image to be reconstructed by taking advantage of spatial property. The second phase performs tensor approximation through searching similar patches within frames near the target frame, to achieve reconstruction by putting the spatio-temporal correlation into full play. The resulting model is solved by an efficient Alternating Direction Method of Multipliers (ADMM) algorithm. A series of experiments show that the quality of the proposed model is comparable to the current state-of-the-art recovery methods.

Compressive sensing (CS) has been employed to compress and store hyperspectral images (HSI) to transfer extensive data efficiently, and obtaining high-quality reconstructed images is crucial for subsequent applications. Model-based methods usually capture the priors of HSI, including sparsity, global and nonlocal low-rankness, nonlocal self-similarity and global correlation. However, it is challenging to reasonably integrate different regularizations in a united framework and exploit various priors mentioned to reconstruct high-quality images. In this paper, a tensor optimization model based on the low-rank coefficient tensor and global prior (LRCTGP) is proposed for HSI reconstruction. Complementary regularization terms are integrated into a united framework and effectively promote the reconstruction results. First, we apply the mode- tensor-matrix product to decompose the original HSI rather than employing the regularization to capture the spectral low-rankness and simultaneously constrain the original and feature HSI. Then, tensor ring decomposition is employed to constrain the coefficient tensor with fewer bands, which is more efficient in the low-rank approximation and has smaller calculation costs than applying it to the original HSI. Moreover, integrating BM3D as a regularizer is more efficient than other patch-based models. Finally, considering the smoothness and global correlation, spatial-spectral total variation (SSTV) is applied to compensate for the shortcomings after decomposing the original and feature HSI, which complements the overall structure and details of reconstructed images and improves the reconstruction quality. Alternating direction method of multipliers (ADMM) is used to optimize the proposed model. Experimental results of the LRCTGP model on different HSI datasets are better than existing state-of-the-art approaches, which proves the superiority.

This work presents the first support vector machine (SVM) processor that supports on-chip active learning for seizure detection. Alternating direction method of multipliers (ADMM), minimum-redundancy maximum-relevance (mRMR), and low-rank approximation are utilized to reduce the computational complexity and memory storage by 99.4% and 90.4%, respectively. Hardware complexity is reduced by 87% through folded architecture with reconfigurable processing elements. The chip achieves the best detection performance with a 96.1% accuracy and a 0.34% false alarm rate in 0.71s with 1.9mW. It also achieves a 153,310x higher energy efficiency than a high-end CPU for SVM training.