Tensor decomposition exploiting structural constraints for brain source imaging

Tensor decomposition is one of the model reduction techniques for compressing deep neural networks. Existing methods use either Tucker decomposition (TD) or Canonical Polyadic decomposition (CPD) for model compression, but none of them tried to combine those two methods, owing to the complexity of choosing a proper decomposition method for each layer. In this paper, we adopted the automatic tuning technique to design an algorithm that can mix both tensor decomposition methods, called Mixed Tensor Decomposition (MTD). The goal is to achieve better compression ratio while keeping similar accuracy as the original models. We used VGG type networks for the case study since they are relatively heavy and computationally expensive. We first studied the relation of model accuracy and compression ratio for Tucker and CPD applying to convolution neural networks (CNN). Based on the studied results, we designed a strategy to select the most suitable decomposition method for each layer, and further fine-tunes the models to recover the accuracy. We have conducted experiments using VGG11 and VGG16 with CIFAR10 dataset, and compared MTD with other tensor decomposition algorithms. The results show that MTD can achieve compression ratio 32 × and 37 × for VGG11 and VGG16 respectively with less than 1% accuracy drops, which is much better than the state-of-the-art tensor decomposition algorithms for model compression.

Integrated sensing and communication (ISAC) has emerged as a key enabler for future wireless networks by unifying communication and sensing functionalities within a shared framework. However, achieving the coordination gains of these two functionalities critically depends on accurate estimation of the sensing targets and communication channels, while their joint estimation remains challenging. To address this, this paper proposes a Bayesian tensor decomposition (BTD) approach for joint channel estimation and target sensing in multiple-input multiple-output (MIMO)-ISAC systems, where parts of sensing targets also act as communication scatterers. Specifically, we develop space-frequency domain received signal models for target sensing and channel estimation and formulate them as canonical polyadic decomposition (CPD) problems under the tensor decomposition framework. This formulation reveals the common multilinear structure and the partially shared physical parameters between sensing and communication, which underpins the ensuing joint estimation task. To solve these problems, we propose a dual-module Vandermonde structure-assisted BTD (V-BTD) algorithm that incorporates propagation-induced Vandermonde structure constraints within a Bayesian framework to enable effective sensing-communication collaboration while maintaining problem feasibility. In this algorithm, Module A estimates the factor matrices via unstructured BTD with Gaussian priors, whereas Module B exploits the Vandermonde structure to recover the underlying physical parameters using generalized von Mises priors. The dual-module design alternates between an unstructured tensor decomposition step and a structure-aware parameter recovery step, yielding a favorable trade-off between inference exactness and computational tractability. With the flexible prior models in the BTD framework, the proposed algorithm supports both uninformative and informative settings, thereby allowing sensing-derived information to be incorporated for communication channel estimation to further improve estimation accuracy. Simulation results demonstrate that the proposed method significantly outperforms the benchmarks, highlighting its superiority for advanced ISAC systems.

Mechanical vibration signal mapped into a high-dimensional space tends to exhibit a special distribution and movement characteristics, which can further reveal the dynamic behavior of the original time series. As the most natural representation of high-dimensional data, tensor can preserve the intrinsic structure of the data to the maximum extent. Thus, the tensor decomposition algorithm has broad application prospects in signal processing. High-dimensional tensor can be obtained from a one-dimensional vibration signal by using phase space reconstruction, which is called the tensorization of data. As a new signal decomposition method, tensor-based singular spectrum algorithm (TSSA) fully combines the advantages of phase space reconstruction and tensor decomposition. However, TSSA has some problems, mainly in estimating the rank of tensor and selecting the optimal reconstruction tensor. In this paper, the improved TSSA algorithm based on convex-optimization and permutation entropy (PE) is proposed. Firstly, aiming to accurately estimate the rank of tensor decomposition, this paper presents a convex optimization algorithm using non-convex penalty functions based on singular value decomposition (SVD). Then, PE is employed to evaluate the desired tensor and improve the denoising performance. In order to verify the effectiveness of proposed algorithm, both numerical simulation and experimental bearing failure data are analyzed.

CANDECOMP/PARAFAC (CP) tensor decomposition is a popular unsupervised machine learning method with numerous applications. This process involves modeling a high-dimensional, multi-modal array (a tensor) as the sum of several low-dimensional components. In order to decompose a tensor, one must solve an optimization problem, whose objective is often given by the sum of the squares of the tensor and decomposition model entry differences. One algorithm occasionally utilized to solve such problems is CP-OPT-DGN, a damped Gauss-Newton all-at-once optimization method for CP tensor decomposition. However, there are currently no published results that consider the decomposition of large-scale (with up to billions of non-zeros), sparse tensors using this algorithm. This work considers the decomposition of large-scale tensors using an efficiently implemented CP-OPT-DGN method. It is observed that CP-OPT-DGN significantly outperforms CP-ALS (CP-Alternating Least Squares) and CP-OPT-QNR (a quasi-Newton-Raphson all-at-once optimization method for CP tensor decomposition), two other widely used tensor decomposition algorithms, in terms of accuracy and latent behavior detection.

Functional near-infrared spectroscopy (fNIRS) provides an effective tool in neuroscience studies of cognition in infants. fNIRS signals are normally processed by applying ANOVA analysis on the grand average of the hemodynamic responses to investigate the cognitive-related differences between experimental groups. However, this averaging approach does not account for any differences in the temporal patterns of the responses. Therefore, we propose a new approach based on a combination of tensor decomposition and ANOVA. First, a four-way tensor of the hemodynamic responses is arranged as time × frequency × channel× subject and decomposed using Canonical Polyadic Decomposition (CPD). Next, ANOVA is applied to identify significant patterns between subjects. Instead of averaging, the CPD can capture the distinct patterns between groups in all the dimensions. We used fNIRS dataset of 70 infants who participated in an experiment to investigate cortical activation to an agent (i.e., mechanical claws vs. human hands) with different events (i.e., function and non-function). In the comparison with the traditional ANOVA, CPD+ANOVA identified the same significance factors. However, CPD+ANOVA discovered new information on the temporal and spatial patterns indicating a longer interval hemodynamic responses, which was missed using the traditional ANOVA. This new analysis of hemodynamic responses as captured using fNIRS will improve neuroscience and cognitive studies.

Tensor Ring decomposition for context-aware recommendation

A new algorithm to estimate the direction of arrival (DOA) and polarization parameters of signals impinging on an array with electromagnetic (EM) vector-sensors is presented by exploiting the canonical polyadic decomposition (CPD) of tensors. In addition to spatial and temporal diversities, further information from the polarization domain is considered and used in this paper. Estimation errors of these parameters are evaluated by the Cramér-Rao lower bound (CRB) benchmark, in the presence of additive white Gaussian noise (AWGN). The superiority of the proposed algorithm is shown by comparing with the derivative algorithms of MUSIC and ESPRIT. In the proposed algorithm, the parameters can be estimated by virtue of the diversities of the spatial and polarization belonging to the factor matrices, rather than the conventional subspace which is the foundation of MUSIC and ESPRIT. Additionally, the classical CPD algorithm based on Alternating Least Squares (ALS) is introduced to verify the efficacy of the proposed CPD algorithm. Results demonstrate that when the number of snapshots is greater than 50, the proposed algorithm requires a smaller number of snapshots to achieve a high level of performance, compared against the subspace-based algorithms and the ALS-based algorithm. Furthermore, in the matter of the array with a small number of sensors, the discovered advantage concerning the Root Mean Square Error (RMSE) in estimating the DOA and the polarization state of the signal is noteworthy.

This issue of SIAM Journal on Matrix Analysis and Applications was motivated by the Workshop on Tensor Decompositions and Applications, held in Luminy, France from August 29 to September 2, 2005. The issue was announced through both the SIMAX and the workshop web sites. Though decompositions of higher-order tensors have been around for many years, the door is now opening on greater mathematical understanding and new applications. Tensor decompositions have been applied by researchers in psychometrics and chemometrics since the seventies. More recently, tensors have found their way to signal processing via the use of high-order statistics, joint matrix techniques, and applications in telecommunications. Other applications include complexity theory, (blind) system identification, biomedical engineering, numerical analysis, and data mining, among others. This volume contains 15 papers that together form a nice cross-section of current research on tensor decompositions. The papers present new algorithms, basic algebraic results including the introduction of new decompositions, and applications in signal processing, scientific computing, and large-scale problems. Many thanks are due to all the authors for their valuable contributions. We would also like to thank Henk van der Vorst, Mitch Chernoff, and the SIAM staff for their support. Since the initial submission of the papers, two of the authors have passed away. Richard Harshman and Gene Golub were both authorities in their field.

The Canonical Polyadic (CP) decomposition of R-way arrays is a powerful tool in multilinear algebra. Algorithms to compute an approximate CP decomposition from noisy observations are often based on Alternating Least Squares (ALS) which may require a large number of iterations to converge. To avoid this drawback we investigate semi-algebraic approaches that algebraically reformulate the CP decomposition into a set of simultaneous matrix diagonalization (SMD) problems. In particular, we propose a SEmi-algebraic framework for approximate CP decompositions via SImultaneous matrix diagonalization (SMD) and generalized unfoldings (SECSI-GU). SECSI-GU combines the benefits of two existing semi-algebraic approaches based on SMDs: the SECSI framework which selects the model estimate from multiple candidates obtained by solving multiple SMDs and the “Semi-Algebraic Tensor Decomposition” (SALT) algorithm which considers a “generalized” unfolding of the tensor in order to enhance the identifiability for tensors with R > 3 dimensions. The resulting SECSI-GU framework offers a large number of degrees of freedom to flexibly adapt the performance-complexity trade-off. As we show in numerical simulations, it outperforms SECSI and SALT for tensors with R > 3 dimensions.

Group analysis of ongoing EEG data based on fast double-coupled nonnegative tensor decomposition

Tensor decomposition of EEG signals: A brief review

Probabilistic inference with noisy-threshold models based on a CP tensor decomposition

There are many combined signal processing applications such as the joint processing of EEG (Electroencephalogram) and MEG (Magnetoencephalogram) data that can benefit from coupled CP (Canonical Polyadic) tensor decompositions. The coupled CP decomposition jointly decomposes tensors that have at least one factor matrix in common. The C-SECSI (Coupled — Semi-Algebraic framework for approximate CP decomposition via SImultaneaous matrix diagonalization) framework provides a semi-algebraic solution for the coupled CP decomposition of noise corrupted low-rank tensors. The C-SECSI framework efficiently computes the factor matrices even in ill-posed scenarios with an adjustable complexity-accuracy trade-off. In this paper, we present a reliability test for the C-SECSI framework that can improve the model order estimation. Moreover, we analyse the photic driving effect from simultaneously recorded EEG and MEG data using the C-SECSI framework. The EEG and MEG data used in the analysis are obtained by stimulating volunteers with flickering light at different frequencies that are multiples of the individual alpha frequency of each volunteer.

The Canonical Polyadic Decomposition (CPD) is a fundamental tensor decomposition which has widespread use in signal processing due to its ability to extract component information. A popular for algorithm CPD is the generalized eigenvalue decomposition (GEVD) which is based on the generalized eigenvectors of a subpencil of a tensor. GEVD plays an important role in applications as it provides strong algebraic initializations for optimization routines for CPD computation. In fact, using GEVD initializations can improve final accuracy and reduce computation time. However, despite GEVD’s success, the algorithm underperforms in some settings and exhibits pencil-based instability.We present a recursive generalized eigenspace decomposition (GESD) for CPD computation. Rather using one sub-pencil, GESD combines generalized eigenspace information from many subpencils to compute a CPD. GESD is more accurate than GEVD and thereby improves the reliability of the components extracted by CPD. We also give a Cramér-Rao based analysis for the accuracy of GESD.

Semi-symmetric three-way arrays are essential tools in blind source separation (BSS) particularly in independent component analysis (ICA). These arrays can be built by resorting to higher order statistics of the data. The canonical polyadic (CP) decomposition of such semi-symmetric three-way arrays allows us to identify the so-called mixing matrix, which contains the information about the intensities of some latent source signals present in the observation channels. In addition, in many applications, such as the magnetic resonance spectroscopy (MRS), the columns of the mixing matrix are viewed as relative concentrations of the spectra of the chemical components. Therefore, the two loading matrices of the three-way array, which are equal to the mixing matrix, are nonnegative. Most existing CP algorithms handle the symmetry and the nonnegativity separately. Up to now, very few of them consider both the semi-nonnegativity and the semi-symmetry structure of the three-way array. Nevertheless, like all the methods based on line search, trust region strategies, and alternating optimization, they appear to be dependent on initialization, requiring in practice a multi-initialization procedure. In order to overcome this drawback, we propose two new methods, called JD LU + and JD QR + , to solve the problem of CP decomposition of semi-nonnegative semi-symmetric three-way arrays. Firstly, we rewrite the constrained optimization problem as an unconstrained one. In fact, the nonnegativity constraint of the two symmetric modes is ensured by means of a square change of variable. Secondly, a Jacobi-like optimization procedure is adopted because of its good convergence property. More precisely, the two new methods use LU and QR matrix factorizations, respectively, which consist in formulating high-dimensional optimization problems into several sequential polynomial and rational subproblems. By using both LU and QR matrix factorizations, we aim at studying the influence of the used matrix factorization. Numerical experiments on simulated arrays emphasize the advantages of the proposed methods especially the one based on LU factorization, in the presence of high-variance model error and of degeneracies such as bottlenecks. A BSS application on MRS data confirms the validity and improvement of the proposed methods.