Smoothed analysis of tensor decompositions

This survey gives a comprehensive overview of tensor techniques and applications in machine learning. Tensor represents higher order statistics. Nowadays, many applications based on machine learning algorithms require a large amount of structured high-dimensional input data. As the set of data increases, the complexity of these algorithms increases exponentially with the increase of vector size. Some scientists found that using tensors instead of the original input vectors can effectively solve these high-dimensional problems. This survey introduces the basic knowledge of tensor, including tensor operations, tensor decomposition, some tensor-based algorithms, and some applications of tensor in machine learning and deep learning for those who are interested in learning tensors. The tensor decomposition is highlighted because it can effectively extract structural features of data and many algorithms and applications are based on tensor decomposition. The organizational framework of this paper is as follows. In part one, we introduce some tensor basic operations, including tensor decomposition. In part two, applications of tensor in machine learning and deep learning, including regression, supervised classification, data preprocessing, and unsupervised classification based on low rank tensor approximation algorithms are introduced detailly. Finally, we briefly discuss urgent challenges, opportunities and prospects for tensor.

Tensor decomposition is one of the model reduction techniques for compressing deep neural networks. Existing methods use either Tucker decomposition (TD) or Canonical Polyadic decomposition (CPD) for model compression, but none of them tried to combine those two methods, owing to the complexity of choosing a proper decomposition method for each layer. In this paper, we adopted the automatic tuning technique to design an algorithm that can mix both tensor decomposition methods, called Mixed Tensor Decomposition (MTD). The goal is to achieve better compression ratio while keeping similar accuracy as the original models. We used VGG type networks for the case study since they are relatively heavy and computationally expensive. We first studied the relation of model accuracy and compression ratio for Tucker and CPD applying to convolution neural networks (CNN). Based on the studied results, we designed a strategy to select the most suitable decomposition method for each layer, and further fine-tunes the models to recover the accuracy. We have conducted experiments using VGG11 and VGG16 with CIFAR10 dataset, and compared MTD with other tensor decomposition algorithms. The results show that MTD can achieve compression ratio 32 × and 37 × for VGG11 and VGG16 respectively with less than 1% accuracy drops, which is much better than the state-of-the-art tensor decomposition algorithms for model compression.

The separation of Electroencephalography (EEG) sources is a typical application of tensor decompositions in biomedical engineering. The objective of most approaches studied in the literature consists in providing separate spatial maps and time signatures for the identified sources. However, for some applications, a precise localization of each source is required. To achieve this, a two-step approach has been proposed. The idea of this approach is to separate the sources using the canonical polyadic decomposition in the first step and to employ the results of the tensor decomposition to estimate distributed sources in the second step, using the so-called disk algorithm. In this paper, we propose to combine the tensor decomposition and the source localization in a single step. To this end, we directly impose structural constraints, which are based on a priori information on the possible source locations, on the factor matrix of spatial characteristics. The resulting optimization problem is solved using the alternating direction method of multipliers, which is incorporated in the alternating least squares tensor decomposition algorithm. Realistic simulations with epileptic EEG data confirm that the proposed single-step source localization approach outperforms the previously developed two-step approach.

Group analysis of ongoing EEG data based on fast double-coupled nonnegative tensor decomposition

Mechanical vibration signal mapped into a high-dimensional space tends to exhibit a special distribution and movement characteristics, which can further reveal the dynamic behavior of the original time series. As the most natural representation of high-dimensional data, tensor can preserve the intrinsic structure of the data to the maximum extent. Thus, the tensor decomposition algorithm has broad application prospects in signal processing. High-dimensional tensor can be obtained from a one-dimensional vibration signal by using phase space reconstruction, which is called the tensorization of data. As a new signal decomposition method, tensor-based singular spectrum algorithm (TSSA) fully combines the advantages of phase space reconstruction and tensor decomposition. However, TSSA has some problems, mainly in estimating the rank of tensor and selecting the optimal reconstruction tensor. In this paper, the improved TSSA algorithm based on convex-optimization and permutation entropy (PE) is proposed. Firstly, aiming to accurately estimate the rank of tensor decomposition, this paper presents a convex optimization algorithm using non-convex penalty functions based on singular value decomposition (SVD). Then, PE is employed to evaluate the desired tensor and improve the denoising performance. In order to verify the effectiveness of proposed algorithm, both numerical simulation and experimental bearing failure data are analyzed.

CANDECOMP/PARAFAC (CP) tensor decomposition is a popular unsupervised machine learning method with numerous applications. This process involves modeling a high-dimensional, multi-modal array (a tensor) as the sum of several low-dimensional components. In order to decompose a tensor, one must solve an optimization problem, whose objective is often given by the sum of the squares of the tensor and decomposition model entry differences. One algorithm occasionally utilized to solve such problems is CP-OPT-DGN, a damped Gauss-Newton all-at-once optimization method for CP tensor decomposition. However, there are currently no published results that consider the decomposition of large-scale (with up to billions of non-zeros), sparse tensors using this algorithm. This work considers the decomposition of large-scale tensors using an efficiently implemented CP-OPT-DGN method. It is observed that CP-OPT-DGN significantly outperforms CP-ALS (CP-Alternating Least Squares) and CP-OPT-QNR (a quasi-Newton-Raphson all-at-once optimization method for CP tensor decomposition), two other widely used tensor decomposition algorithms, in terms of accuracy and latent behavior detection.

Rank-1 L1-norm-based TUCKER2 (L1-TUCKER2) decomposition of 3-way tensors was recently solved exactly, for the first time, by Markopoulos et al.¹ The exact solution to general-rank L1-TUCKER2 remains to date unknown. In this work, we present a novel approximate algorithm for general-rank L1-TUCKER2 decomposition of 3-way tensors. Our algorithm is accompanied by formal convergence and complexity analysis. Our numerical studies illustrate the sturdy corruption resistance of the proposed algorithm compared to state-of-the-art TUCKER2-decomposition counterparts such as GLRAM, HOSVD, and HOOI.

Tensor decomposition and recomputation are two common strategies used to to accelerate training and save memory during the training phase of deep neural network, respectively, allowing researchers to consider larger models, batch sizes, or higher-resolution data. Tensor decomposition reduces parameters and computation by decomposing weight tensors, while recomputation trades computation time for memory. As These two strategies are independent and have their own advantages and disadvantages, it is applealing to consider the simultaneous combination of two strategies to save more memory and complement each other. We analyze existing tensor decomposition algorithms and recomputation techniques, and we propose a memory-optimized algorithm that effectively combines them, and finally evaluate the performance of the proposed algorithm on commonly used deep neural networks and computing platforms. Our experiments show that tensor decomposition can partially or even completely eliminate the extra time overhead caused by recomputation, and that together they can reduce the memory consumption of the training phase by about 41 %, while the loss of accuracy is less than 1.5%.

Acquired images in hyperspectral imagery are disturbed by additive noise which is usually assumes as zero-mean white one. In fact, there is still non-white noise in hyperspectral images (HSIs). The 2-dimensional filtering methods and multidimensional tensor decomposition algorithms cannot be used to remove non-white noise from HSIs directly. Therefore, a prewhitening denoising solution based on tensor decomposition is proposed in this paper. The prewhitening procedure can help to transform the non-white noise into a white one and the tensor decomposition method removes the noise without affecting the related information between image planes. The simulation results show that this non-white noise reducing approach has promising prospects in this field compared to existing denoising methods.

The widespread use of multi-sensor technology and the emergence of big datasets has highlighted the limitations of standard flat-view matrix models and the necessity to move towards more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift towards models that are essentially polynomial and whose uniqueness, unlike the matrix methods, is guaranteed under verymild and natural conditions. Benefiting fromthe power ofmultilinear algebra as theirmathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints that match data properties, and to find more general latent components in the data than matrix-based methods. A comprehensive introduction to tensor decompositions is provided from a signal processing perspective, starting from the algebraic foundations, via basic Canonical Polyadic and Tucker models, through to advanced cause-effect and multi-view data analysis schemes. We show that tensor decompositions enable natural generalizations of some commonly used signal processing paradigms, such as canonical correlation and subspace techniques, signal separation, linear regression, feature extraction and classification. We also cover computational aspects, and point out how ideas from compressed sensing and scientific computing may be used for addressing the otherwise unmanageable storage and manipulation problems associated with big datasets. The concepts are supported by illustrative real world case studies illuminating the benefits of the tensor framework, as efficient and promising tools for modern signal processing, data analysis and machine learning applications; these benefits also extend to vector/matrix data through tensorization. Keywords: ICA, NMF, CPD, Tucker decomposition, HOSVD, tensor networks, Tensor Train.

To better explore the underlying intrinsic structure of tensorial data, in this Letter, the authors propose a low‐rank approximation‐based tensor decomposition (LRATD) algorithm for subspace clustering. LRATD aims to seek a low‐dimensional intrinsic core tensor representation by projecting the original tensor into a subspace spanned by projection matrices. Different from traditional approaches that impose additional constraints on basis matrices to further eliminate the influence of data noise or corruption, they directly add a low‐rank regulariser on the core tensor to encourage more robust feature representation. Noticeably, they develop an accelerated proximal gradient algorithm to solve the problem of LRATD. Experimental results demonstrate the excellent performance compared with state‐of‐the‐art methods.

Fast deployment of mobile Internet makes Web services often consumed under a multi-dimensional spatiotemporal model, wherein a specific service client could keep active while its location is changing. Recommending Web services for such clients must be able to predict unknown QoS values with the target client's service requesting time and location taken into account, e.g., Performing the prediction via a set of measured multi-dimensional QoS data. Most QoS prediction methods focus on the QoS characteristics for one specific dimension, e.g., Time or location, and do not exploit the structural relationships among the multi-dimensional QoS data. This paper proposes an integrated QoS prediction approach which unifies the modeling of multi-dimensional QoS data via multi-linear-algebra based concepts of tensor and enables efficient service recommendation for Web service based mobile clients via tensor decomposition and reconstruction optimization algorithms. Comparative experimental evaluation results show that the proposed QoS prediction approach could result in much better accuracy in recommending Web services than several other representative ones.

Cross-bispectral analysis identifies higher-order neural interactions, but the resulting high-dimensional tensors are difficult to interpret at the source level. Standard dimensionality reduction often lacks a physical link to the underlying neural generators. We introduce a low-rank tensor decomposition framework explicitly grounded in the physics of EEG signal generation. The method represents the cross-bispectrum using a single spatial mixing matrix and a compact source-interaction tensor, yielding a structured and physically interpretable model. This formulation enables a direct source-space interpretation and supports subsequent spatial demixing using MOCA to recover distinct source maps. Through extensive simulations, we show that the method robustly captures the dominant bispectral structure across a range of conditions, including varying signal-to-noise ratio, dipole orientation, amplitude balance, and source complexity. Applied to resting-state EEG, the approach identifies anatomically plausible parietal generators associated with prominent alpha-band bispectral interactions. In contrast to standard tensor decompositions such as Tucker and PARAFAC, which rely on unconstrained cores or rank-one assumptions, the proposed framework enforces a compact and interpretable source-interaction structure. This structure significantly improves interpretability and spatial localization while suppressing spurious high-dimensional variability. The proposed low-rank decomposition offers a principled and computationally efficient approach for reducing and interpreting cross-bispectral EEG data. By bridging tensor decomposition with biophysical source models, it enables insights into complex neural coupling that are inaccessible with standard matrix or tensor techniques.

Lots of data is generated around us in today's big data age. Much of this data is time-varying or dynamic, such as the social network connections that change across time or dynamic functional brain connectivity networks constructed across multiple subjects. In these dynamic (time-varying) networks, it is important to reduce the large amount of data into a few meaningful descriptors. One way to achieve this goal is to detect change points or anomalies in the connectivity patterns across time. Recently, there has been an interest in describing the time-varying network activity as a tensor and detecting the anomalies in terms of the changes in the subspaces of the tensor along each mode [1], [2]. However, the current approaches to tensor decomposition are not robust to non-Gaussian noise, outliers, and corruption in the data. For this reason, a robust low-rank tensor recovery algorithm similar to robust principal components analysis (RPCA) has been recently proposed. In this paper, we employ higher order robust PCA (HoRPCA) for tracking dynamic networks in time and detecting anomalies using a subspace distance measure. The proposed approach assumes that most real life networks are low-rank in nature and considers a low-rank plus sparse tensor decomposition at each time point. The subspaces corresponding to each mode and each time point are described through a projection operator and the subspace distance is quantified through a Hausdorff distance measure. The proposed framework is evaluated on both simulated networks and dynamic functional connectivity brain networks.

How can we accurately and efficiently decompose a tensor stream? Tensor decomposition is a crucial task in a wide range of applications and plays a significant role in latent feature extraction and estimation of unobserved entries of data. The problem of efficiently decomposing tensor streams has been of great interest because many real-world data dynamically change over time. However, existing methods for dynamic tensor decomposition sacrifice the accuracy too much, which limits their usages in practice. Moreover, the accuracy loss becomes even more serious when the tensor stream has an inconsistent temporal pattern since the current methods cannot adapt quickly to a sudden change in data. In this paper, we propose DAO-CP, an accurate and efficient online CP decomposition method which adapts to data changes. DAO-CP tracks local error norms of the tensor streams, detecting a change point of the error norms. It then chooses the best strategy depending on the degree of changes to balance the trade-off between speed and accuracy. Specifically, DAO-CP decides whether to (1) reuse the previous factor matrices for the fast running time or (2) discard them and restart the decomposition to increase the accuracy. Experimental results show that DAO-CP achieves the state-of-the-art accuracy without noticeable loss of speed compared to existing methods.