Tensor Ring decomposition for context-aware recommendation

Tensor decomposition is one of the model reduction techniques for compressing deep neural networks. Existing methods use either Tucker decomposition (TD) or Canonical Polyadic decomposition (CPD) for model compression, but none of them tried to combine those two methods, owing to the complexity of choosing a proper decomposition method for each layer. In this paper, we adopted the automatic tuning technique to design an algorithm that can mix both tensor decomposition methods, called Mixed Tensor Decomposition (MTD). The goal is to achieve better compression ratio while keeping similar accuracy as the original models. We used VGG type networks for the case study since they are relatively heavy and computationally expensive. We first studied the relation of model accuracy and compression ratio for Tucker and CPD applying to convolution neural networks (CNN). Based on the studied results, we designed a strategy to select the most suitable decomposition method for each layer, and further fine-tunes the models to recover the accuracy. We have conducted experiments using VGG11 and VGG16 with CIFAR10 dataset, and compared MTD with other tensor decomposition algorithms. The results show that MTD can achieve compression ratio 32 × and 37 × for VGG11 and VGG16 respectively with less than 1% accuracy drops, which is much better than the state-of-the-art tensor decomposition algorithms for model compression.

Tensor decomposition has been proved to be effective for solving many problems in signal processing and machine learning[1]. Recently, tensor decomposition finds its advantage for compressing deep neural networks. In many applications of deep neural networks, it is critical to reduce the number of parameters and computation workload to accelerate inference speed in deployment of the network. Modern deep neural network consists of multiple layers with multi-array weights where tensor decomposition is a natural way to perform compression. It is achieved by decomposing the weight tensors in convolutional layers or fully-connected layers with specified tensor ranks (e.g. canonical ranks, tensor train ranks). Conventional tensor decomposition in compressing deep neural networks selects the ranks manually that requires tedious human efforts to finetune the performance. To overcome this issue, we propose a novel rank selection scheme, which is inspired by reinforcement learning, to automatically select ranks in recently studied tensor ring decomposition in each convolutional layer. Experimental results validate that our learning based rank selection significantly outperforms hand-crafted rank selection heuristics on a number of benchmark datasets, for the purpose of effectively compressing deep neural networks while maintaining comparable accuracy.

Tensor decomposition of EEG signals: A brief review

In tensor completion tasks, the traditional low-rank tensor decomposition models suffer from the laborious model selection problem due to their high model sensitivity. In particular, for tensor ring (TR) decomposition, the number of model possibilities grows exponentially with the tensor order, which makes it rather challenging to find the optimal TR decomposition. In this paper, by exploiting the low-rank structure of the TR latent space, we propose a novel tensor completion method which is robust to model selection. In contrast to imposing the low-rank constraint on the data space, we introduce nuclear norm regularization on the latent TR factors, resulting in the optimization step using singular value decomposition (SVD) being performed at a much smaller scale. By leveraging the alternating direction method of multipliers (ADMM) scheme, the latent TR factors with optimal rank and the recovered tensor can be obtained simultaneously. Our proposed algorithm is shown to effectively alleviate the burden of TR-rank selection, thereby greatly reducing the computational cost. The extensive experimental results on both synthetic and real-world data demonstrate the superior performance and efficiency of the proposed approach against the state-of-the-art algorithms.

The so-called block-term decomposition (BTD) tensor model has been recently receiving increasing attention due to its enhanced ability of representing systems and signals that are composed of blocks of rank higher than one, a scenario encountered in numerous diverse applications. Its uniqueness and approximation have thus been thoroughly studied. Nevertheless, the problem of estimating the BTD model structure, namely the number of block terms and their individual ranks, has only recently started to attract significant attention, as it is more challenging compared to more classical tensor models such as canonical polyadic decomposition (CPD) and Tucker decomposition (TD). This paper reports our recent results on this topic, which are based on an appropriate extension to the BTD model of our earlier rank-revealing work on low-rank matrix approximation. The idea is to impose column sparsity jointly on the factors and successively estimate the ranks as the numbers of factor columns of non-negligible magnitude, with the aid of alternating iteratively reweighted least squares (IRLS). Simulation results are reported that demonstrate the effectiveness of our method in accurately estimating both the ranks and the factors of the least squares BTD approximation.

Missing data are critical deficiency in the investigation of displacement measurement in urban Internet of Things system. In the insight of recovering missing displacement data, this article presents a data-driven and high-dimensional gap-imputation method, Tucker decomposition with L2 regularization. Results on the global navigation satellite system (GNSS) time series collected from an intelligent structural health monitoring system show that the recovery accuracy is improved compared with some popular benchmark methods. When the missing rate is 50%, compared with singular spectrum analysis, singular value decomposition, CP optimization algorithm, k -nearest neighbors, and Tucker decomposition via alternating least squares, Tucker decomposition with L2 regularization can improve the average mean absolute error by about 4.74, 4.95, 5.82, 2.29, and 5.67 mm for all locations. It can be concluded that the consideration of multiple temporal correlations is necessary for missing data imputation. Compared with matrix decomposition, tensor decomposition can improve the ability for high-dimensional correlations in the GNSS time series.

Many spectral unmixing approaches ranging from geometry, algebra to statistics have been proposed, in which nonnegative matrix factorization (NMF)-based ones form an important family. The original NMF-based unmixing algorithm loses the spectral and spatial information between mixed pixels when stacking the spectral responses of the pixels into an observed matrix. Therefore, various constrained NMF methods are developed to impose spectral structure, spatial structure, and spectral-spatial joint structure into NMF to enforce the estimated endmembers and abundances preserve these structures. Compared with matrix format, the third-order tensor is more natural to represent a hyperspectral data cube as a whole, by which the intrinsic structure of hyperspectral imagery can be losslessly retained. Extended from NMF-based methods, a matrix-vector nonnegative tensor factorization (NTF) model is proposed in this paper for spectral unmixing. Different from widely used tensor factorization models, such as canonical polyadic decomposition CPD) and Tucker decomposition, the proposed method is derived from block term decomposition, which is a combination of CPD and Tucker decomposition. This leads to a more flexible frame to model various application-dependent problems. The matrix-vector NTF decomposes a third-order tensor into the sum of several component tensors, with each component tensor being the outer product of a vector (endmember) and a matrix (corresponding abundances). From a formal perspective, this tensor decomposition is consistent with linear spectral mixture model. From an informative perspective, the structures within spatial domain, within spectral domain, and cross spectral-spatial domain are retreated interdependently. Experiments demonstrate that the proposed method has outperformed several state-of-the-art NMF-based unmixing methods.

The Canonical Polyadic (CP) decomposition of R-way arrays is a powerful tool in multilinear algebra. Algorithms to compute an approximate CP decomposition from noisy observations are often based on Alternating Least Squares (ALS) which may require a large number of iterations to converge. To avoid this drawback we investigate semi-algebraic approaches that algebraically reformulate the CP decomposition into a set of simultaneous matrix diagonalization (SMD) problems. In particular, we propose a SEmi-algebraic framework for approximate CP decompositions via SImultaneous matrix diagonalization (SMD) and generalized unfoldings (SECSI-GU). SECSI-GU combines the benefits of two existing semi-algebraic approaches based on SMDs: the SECSI framework which selects the model estimate from multiple candidates obtained by solving multiple SMDs and the “Semi-Algebraic Tensor Decomposition” (SALT) algorithm which considers a “generalized” unfolding of the tensor in order to enhance the identifiability for tensors with R > 3 dimensions. The resulting SECSI-GU framework offers a large number of degrees of freedom to flexibly adapt the performance-complexity trade-off. As we show in numerical simulations, it outperforms SECSI and SALT for tensors with R > 3 dimensions.

The separation of Electroencephalography (EEG) sources is a typical application of tensor decompositions in biomedical engineering. The objective of most approaches studied in the literature consists in providing separate spatial maps and time signatures for the identified sources. However, for some applications, a precise localization of each source is required. To achieve this, a two-step approach has been proposed. The idea of this approach is to separate the sources using the canonical polyadic decomposition in the first step and to employ the results of the tensor decomposition to estimate distributed sources in the second step, using the so-called disk algorithm. In this paper, we propose to combine the tensor decomposition and the source localization in a single step. To this end, we directly impose structural constraints, which are based on a priori information on the possible source locations, on the factor matrix of spatial characteristics. The resulting optimization problem is solved using the alternating direction method of multipliers, which is incorporated in the alternating least squares tensor decomposition algorithm. Realistic simulations with epileptic EEG data confirm that the proposed single-step source localization approach outperforms the previously developed two-step approach.

Functional near-infrared spectroscopy (fNIRS) provides an effective tool in neuroscience studies of cognition in infants. fNIRS signals are normally processed by applying ANOVA analysis on the grand average of the hemodynamic responses to investigate the cognitive-related differences between experimental groups. However, this averaging approach does not account for any differences in the temporal patterns of the responses. Therefore, we propose a new approach based on a combination of tensor decomposition and ANOVA. First, a four-way tensor of the hemodynamic responses is arranged as time × frequency × channel× subject and decomposed using Canonical Polyadic Decomposition (CPD). Next, ANOVA is applied to identify significant patterns between subjects. Instead of averaging, the CPD can capture the distinct patterns between groups in all the dimensions. We used fNIRS dataset of 70 infants who participated in an experiment to investigate cortical activation to an agent (i.e., mechanical claws vs. human hands) with different events (i.e., function and non-function). In the comparison with the traditional ANOVA, CPD+ANOVA identified the same significance factors. However, CPD+ANOVA discovered new information on the temporal and spatial patterns indicating a longer interval hemodynamic responses, which was missed using the traditional ANOVA. This new analysis of hemodynamic responses as captured using fNIRS will improve neuroscience and cognitive studies.

Recurrent Neural Networks (RNNs) and their variants, such as Long-Short Term Memory (LSTM) networks, and Gated Recurrent Unit (GRU) networks, have achieved promising performance in sequential data modeling. The hidden layers in RNNs can be regarded as the memory units, which are helpful in storing information in sequential contexts. However, when dealing with high dimensional input data, such as video and text, the input-to-hidden linear transformation in RNNs brings high memory usage and huge computational cost. This makes the training of RNNs very difficult. To address this challenge, we propose a novel compact LSTM model, named as TR-LSTM, by utilizing the low-rank tensor ring decomposition (TRD) to reformulate the input-to-hidden transformation. Compared with other tensor decomposition methods, TR-LSTM is more stable. In addition, TR-LSTM can complete an end-to-end training and also provide a fundamental building block for RNNs in handling large input data. Experiments on real-world action recognition datasets have demonstrated the promising performance of the proposed TR-LSTM compared with the tensor-train LSTM and other state-of-the-art competitors.

In this work, we consider the problem of factoring a third-order tensor into multilinear rank -($L_{r}, L_{r}$, 1) terms. This model, referred to as the rank-($L_{r}, L_{r}$, 1) block-term decomposition (BTD), finds many applications in signal processing, especially blind separation of smooth sources and unmixing spectral-spatial data (e.g., hyper-spectral image). On the other hand, finding latent factors of rank-($L_{r}, L_{r}$, 1) BTD poses a very challenging optimization problem. Some computational tools designed for canonical polyadic decomposition (CPD) (e.g., alternating least squares (ALS) and Levenberg-Marquardt (LM) based algorithms) can be modified to handle rank-($L_{r}, L_{r}$, 1) BTD. Nonetheless, these methods essentially treat rank-($L_{r}, L_{r}$, 1) BTD as a special CPD problem. This raises a number of challenges, since rank -($L_{r}, L_{r}$, 1) BTD can be viewed as a CPD problem with rank-deficient latent factors and high CP rank–and these are known as hard cases for CPD algorithms. In this work, we reformulate the rank-($L_{r}, L_{r}$, 1) BTD problem as a matrix rank-constrained matrix factorization problem. We propose a simple algorithm that combines alternating optimization and projected gradient. This way, the per-iteration complexity is much smaller than those of the ALS and LM-based algorithms. We also show that the algorithm converges to a stationary point of the problem of interest at a sublinear rate–although nonconvex constraints are involved. Numerical experiments are conducted to showcase the effectiveness of our algorithm.

The Canonical Polyadic Decomposition (CPD) is a fundamental tensor decomposition which has widespread use in signal processing due to its ability to extract component information. A popular for algorithm CPD is the generalized eigenvalue decomposition (GEVD) which is based on the generalized eigenvectors of a subpencil of a tensor. GEVD plays an important role in applications as it provides strong algebraic initializations for optimization routines for CPD computation. In fact, using GEVD initializations can improve final accuracy and reduce computation time. However, despite GEVD’s success, the algorithm underperforms in some settings and exhibits pencil-based instability.We present a recursive generalized eigenspace decomposition (GESD) for CPD computation. Rather using one sub-pencil, GESD combines generalized eigenspace information from many subpencils to compute a CPD. GESD is more accurate than GEVD and thereby improves the reliability of the components extracted by CPD. We also give a Cramér-Rao based analysis for the accuracy of GESD.

Integrated sensing and communication (ISAC) has emerged as a key enabler for future wireless networks by unifying communication and sensing functionalities within a shared framework. However, achieving the coordination gains of these two functionalities critically depends on accurate estimation of the sensing targets and communication channels, while their joint estimation remains challenging. To address this, this paper proposes a Bayesian tensor decomposition (BTD) approach for joint channel estimation and target sensing in multiple-input multiple-output (MIMO)-ISAC systems, where parts of sensing targets also act as communication scatterers. Specifically, we develop space-frequency domain received signal models for target sensing and channel estimation and formulate them as canonical polyadic decomposition (CPD) problems under the tensor decomposition framework. This formulation reveals the common multilinear structure and the partially shared physical parameters between sensing and communication, which underpins the ensuing joint estimation task. To solve these problems, we propose a dual-module Vandermonde structure-assisted BTD (V-BTD) algorithm that incorporates propagation-induced Vandermonde structure constraints within a Bayesian framework to enable effective sensing-communication collaboration while maintaining problem feasibility. In this algorithm, Module A estimates the factor matrices via unstructured BTD with Gaussian priors, whereas Module B exploits the Vandermonde structure to recover the underlying physical parameters using generalized von Mises priors. The dual-module design alternates between an unstructured tensor decomposition step and a structure-aware parameter recovery step, yielding a favorable trade-off between inference exactness and computational tractability. With the flexible prior models in the BTD framework, the proposed algorithm supports both uninformative and informative settings, thereby allowing sensing-derived information to be incorporated for communication channel estimation to further improve estimation accuracy. Simulation results demonstrate that the proposed method significantly outperforms the benchmarks, highlighting its superiority for advanced ISAC systems.

The widespread use of multi-sensor technology and the emergence of big datasets has highlighted the limitations of standard flat-view matrix models and the necessity to move towards more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift towards models that are essentially polynomial and whose uniqueness, unlike the matrix methods, is guaranteed under verymild and natural conditions. Benefiting fromthe power ofmultilinear algebra as theirmathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints that match data properties, and to find more general latent components in the data than matrix-based methods. A comprehensive introduction to tensor decompositions is provided from a signal processing perspective, starting from the algebraic foundations, via basic Canonical Polyadic and Tucker models, through to advanced cause-effect and multi-view data analysis schemes. We show that tensor decompositions enable natural generalizations of some commonly used signal processing paradigms, such as canonical correlation and subspace techniques, signal separation, linear regression, feature extraction and classification. We also cover computational aspects, and point out how ideas from compressed sensing and scientific computing may be used for addressing the otherwise unmanageable storage and manipulation problems associated with big datasets. The concepts are supported by illustrative real world case studies illuminating the benefits of the tensor framework, as efficient and promising tools for modern signal processing, data analysis and machine learning applications; these benefits also extend to vector/matrix data through tensorization. Keywords: ICA, NMF, CPD, Tucker decomposition, HOSVD, tensor networks, Tensor Train.