Computing Low-Rank Approximations of Large-Scale Matrices with the Tensor Network Randomized SVD

Low rank approximation of matrices is an important tool in machine learning. Given a data matrix, low rank approximation helps to find factors, patterns, and provides concise representations for the data. Research on low rank approximation usually focuses on real matrices. However, in many applications data are binary (categorical) rather than continuous. This leads to the problem of low rank approximation of binary matrices. Here we are given a d x n binary matrix A and a small integer k < d. The goal is to find two binary matrices U and V of sizes d x k and k x n respectively, so that the Frobenius norm of A - U V is minimized. There are two models of this problem, depending on the definition of the dot product of binary vectors: The GF(2) model and the Boolean semiring model. Unlike low rank approximation of a real matrix which can be efficiently solved by Singular Value Decomposition, we show that approximation of a binary matrix is NP-hard, even for k=1.\nIn this paper, our main concern is the problem of Column Subset Selection (CSS), in which the low rank matrix U must be formed by k columns of the data matrix, and we are interested in the approximation ratio achievable by CSS for binary matrices. For the GF(2) model, we show that CSS has approximation ratio bounded by k/2+1+k/(2(2^k-1)) and this is asymptotically tight. For the Boolean model, it turns out that CSS is no longer sufficient to obtain a bound. We then develop a Generalized CSS (GCSS) procedure in which the columns of U are generated from Boolean formulas operating bitwise on selected columns of the data matrix. We show that the approximation ratio achieved by GCSS is bounded by 2^(k-1)+1, and argue that an exponential dependency on k is seems inherent.

In this paper, a unified algebraic transformation approach is presented for designing parallel recursive and adaptive digital filters and singular value decomposition (SVD) algorithms. The approach is based on the explorations of some algebraic properties of the target algorithms' representations. Several typical modern digital signal processing examples are presented to illustrate the applications of the technique. They include the cascaded orthogonal recursive digital filter, the Givens rotation-based adaptive inverse QR algorithm for channel equalization, and the QR decomposition-based SVD algorithms. All three examples exhibit similar throughput constraints. There exist long feedback loops in the algorithms' signal flow graph representation, and the critical path is proportional to the size of the problem. Applying the proposed algebraic transformation techniques, parallel architectures are obtained for all three examples. For cascade orthogonal recursive filter, retiming transformation and orthogonal matrix decompositions (or pseudo-commutativity) are applied to obtain parallel filter architectures with critical path of five Givens rotations. For adaptive inverse QR algorithm, the commutativity and associativity of the matrix multiplications are applied to obtain parallel architectures with critical path of either four Givens rotations or three Givens rotations plus two multiply-add operations, whichever turns out to be larger. For SVD algorithms, retiming and associativity of the matrix multiplications are applied to derive parallel architectures with critical path of eight Givens rotations. The critical paths of all parallel architectures are independent of the problem size as compared with being proportional to the problem size in the original sequential algorithms. Parallelism is achieved at the expense of slight increase (or the same for the SVD case) in the algorithms' computational complexity.

Singular Value Decomposition (SVD) algorithm is a good method in Collaborative Filtering (CF) algorithm. But the traditional SVD method faces the matrix sparse problem, people solve this problem by Latent Factor Model (LMF). The data imbalances problem also troubles us. The algorithm of this paper combines Slope One and SVD algorithm to solve the data imbalances problem. First, we use Slope One algorithm to deal with the users who have few rating records. Then this kind of user will have more rating records. And we can use new data set to train the SVD model to predict the final results. The experiment results show that our algorithm is better than the traditional Slope One and SVD algorithm.

Ideas and algorithms from numerical linear algebra are important in several areas of data mining. We give an overview of linear algebra methods in text mining (information retrieval), pattern recognition (classification of handwritten digits), and PageRank computations for web search engines. The emphasis is on rank reduction as a method of extracting information from a data matrix, low-rank approximation of matrices using the singular value decomposition and clustering, and on eigenvalue methods for network analysis.

Multicarrier differential chaos shift keying (MC-DCSK) systems transmit the reference chaotic signal for information recovery to remove the complex synchronization circuit. However, transmission errors in the reference chaotic signal would induce the error propagations when directly used for demodulations. In order to address this issue, we propose a low-rank approximation of matrices (LRAM) detection method to improve the reliability performances. In our design, we exploit the low-rank characteristic of MC-DCSK signals sharing the reference chaotic signal. Instead of directly using the received reference chaotic signal for correlated demodulation, we propose to apply the LRAM method to jointly estimate the reference signal and the information-bearing signal. We use the singular value decomposition and generalized LRAM methods to minimize the distances between the estimates and transmitted data. Thus, the signal-to-noise ratio (SNR) can be improved to achieve lower bit error rate (BER). Moreover, theoretical BER expression is derived and LRAM-aided system is proved to be capable of proving the maximum-likelihood detection for received signals. Simulation results demonstrate the BER performances of our design outperform counterparts over additive white Gaussian noise and fading channels.

This paper concerns accurate computation of the singular value decomposition (SVD) of an $m\times n$ matrix $A$ . As is well known, cross-product matrix based SVD algorithms compute large singular values accurately but generally deliver poor small singular values. A new novel cross-product matrix based SVD method is proposed: (a) Use a backward stable algorithm to compute the eigenpairs of $A^{\rm T}A$ and take the square roots of the large eigenvalues of it as the large singular values of $A$ ; (b) form the Rayleigh quotient of $A^{\rm T}A$ with respect to the matrix consisting of the computed eigenvectors associated with the computed small eigenvalues of $A^{\rm T}A$ ; (c) compute the eigenvalues of the Rayleigh quotient and take the square roots of them as the small singular values of $A$ . A detailed quantitative error analysis is conducted on the method. It is proved that if small singular values are well separated from the large ones then the method can compute the small ones accurately up to the order of the unit roundoff $\epsilon$ . An algorithm is developed that is not only cheaper than the standard Golub–Reinsch and Chan SVD algorithms but also can update or downdate a new SVD by adding or deleting a row and compute certain refined Ritz vectors for large matrix eigenproblems at very low cost. Several variants of the algorithm are proposed that compute some or all parts of the SVD. Typical numerical examples confirm the high accuracy of our algorithm.

We propose singular value decomposition (SVD) algorithms for very large-scale matrices based on a low-rank tensor decomposition technique called the tensor train (TT) format. By using the proposed algorithms, we can compute several dominant singular values and corresponding singular vectors of large-scale structured matrices given in a low-rank TT format. We propose a large-scale trace optimization problem, and in the proposed methods, the large-scale optimization problem is reduced to sequential small-scale optimization problems. We show that the computational complexity of the proposed algorithms scales logarithmically with the matrix size if the TT-ranks are bounded. Numerical simulations based on very large-scale Hilbert matrix demonstrate the effectiveness of the proposed methods.

Singular value decomposition is the key tool in the analysis and understanding of linear regularization methods in Hilbert spaces. Besides simplifying computations it allows to provide a good understanding of properties of the forward problem compared to the prior information introduced by the regularization methods. In the last decade nonlinear variational approaches such as ` or total variation regularizations became quite prominent regularization techniques with certain properties being superior to standard methods. In the analysis of those, singular values and vectors did not play any role so far, for the obvious reason that these problems are nonlinear, together with the issue of defining singular values and singular vectors in the first place. In this paper however we want to start a study of singular values and vectors for nonlinear variational regularization of linear inverse problems, with particular focus on singular onehomogeneous regularization functionals. A major role is played by the smallest singular value, which we define as the ground state of an appropriate functional combining the (semi)norm introduced by the forward operator and the regularization functional. The optimality condition for the ground state further yields a natural generalization to higher singular values and vectors involving the subdifferential of the regularization functional, although we shall see that the Rayleigh principle may fail for higher singular values. Using those definitions of singular values and vectors, we shall carry over two main properties from the world of linear regularization. The first one is gaining information about scale, respectively the behavior of regularization techniques at different scales. This also leads to novel estimates at different scales, generalizing the estimates for the coefficients in the linear singular value expansion. The second one is to provide classes of exact solutions for variational regularization methods. We will show that all singular vectors can be reconstructed up to a scalar factor by the standard Tikhonov-type regularization approach even in the presence of (small) noise. Moreover, we will show that they can even be reconstructed without any bias by the recently popularized inverse scale space method.

Low-rank approximation of large and/or sparse matrices is important in many applications, and the singular value decomposition (SVD) gives the best low-rank approximations with respect to unitarily-invariant norms. In this paper we show that good low-rank approximations can be directly obtained from the Lanczos bidiagonalization process applied to the given matrix without computing any SVD. We also demonstrate that a so-called one-sided reorthogonalization process can be used to maintain an adequate level of orthogonality among the Lanczos vectors and produce accurate low-rank approximations. This technique reduces the computational cost of the Lanczos bidiagonalization process. We illustrate the efficiency and applicability of our algorithm using numerical examples from several applications areas.

A fast-convergence singular value decomposition (SVD) algorithm is developed for tracking time-varying channels in massive MIMO precoding/beamforming systems. Since only strong eigen-modes are selected for data transmission in these systems, our SVD algorithm exploits the properties of partial decomposition and temporal correlation. Besides, the proposed self-adjusting inverse power method can achieve fast convergence by modifying the shift according to the intermediate result during each iteration. Furthermore, the singular vectors and values of the desired eigenmodes can be computed simultaneously. Thus, parallel processing is possible to facilitate high-throughput implementation. Compared to the self-power method with super linear convergence, the self-adjusting inverse power method has better convergence and lower complexity. Good channel tracking capability is also demonstrated.

Change point detection is essential to understand the time-evolving structure of dynamic networks. Recent research shows that a latent semantic indexing (LSI)-based algorithm effectively detects the change points of a dynamic network. The LSI-based method involves a singular value decomposition (SVD) on the data matrix. In a dynamic scenario, recomputing the SVD of a large matrix each time new data arrives is prohibitively expensive and impractical. A more efficient approach is to incrementally update the decomposition. However, in the classical incremental SVD (incSVD) algorithm, the information of the newly added columns is not fully considered in updating the right singular space, resulting in an approximation error which cannot be ignored. This paper proposes an enhanced incSVD (EincSVD) algorithm, in which the right singular matrix is calculated in an alternative way. An adaptive EincSVD (AEincSVD) algorithm is also proposed to further reduce the computational complexity. Theoretical analysis proves that the approximation error of the EincSVD is smaller than that of the incSVD. Simulation results demonstrate that the EincSVD and the AEincSVD perform much better than the incSVD on change point detection, and the performance of the EincSVD is comparable to the batch SVD algorithm.

Estimates of the past thermal state of the land surface are crucial to assess the magnitude of current anthropogenic climate change, as well as to assess the ability of Earth System Models to forecast the evolution of the climate near the ground, not included in standard meteorological records. Subsurface temperature data are able to retrieve long-term changes in surface energy balance –from decadal to millennial time scales, thus constituting an important record of the dynamics of the climate system that contributes low-frequency information to proxy-based paleoclimatic reconstructions. A broadly used technique to retrieve past temperature and heat flux histories from subsurface temperature profiles based on a Singular Value Decomposition (SVD) algorithm was able to take into account a limited number of sources of uncertainty, with recent works attempting to increase the number of factors considered in uncertainty estimates. Nevertheless, the SVD methodology did not define a statistical framework for aggregating inversions of individual profiles to derive global results, which lead to estimates of global and regional uncertainties that are difficult to interpret. To alleviate the lack of a conceptual framework for estimating uncertainties in past temperature and heat flux histories at regional and global scales, we combine a new bootstrapping sampling strategy with the broadly used SVD algorithm, and assess its performance against the original SVD technique and another technique based on generating perturbed parameter ensembles of inversions. The new bootstrap approach is able to reproduce the prescribed surface temperature series used to derive an artificial profile. Bootstrap results are also in agreement with the global mean surface temperature history and the global mean heat flux history retrieved in previous studies. Furthermore, the new bootstrap technique provides with a meaningful uncertainty range for the inversion of large sets of subsurface temperature profiles. We suggest the use of this new approach particularly for aggregating results from a number of individual profiles, and to this end, we release the programs used to derive all inversions in this study as a suite of codes labelled CIBOR 1.0: Codes for Inverting BORholes, version 1.0.

In this paper we present a new type of restarted Krylov method for calculating low-rank approximations of large matrices. In contrast to former Krylov methods, our approach does not apply the Lanczos algorithm, or Lanczos bidiagonalization. This simplifies the basic iteration and allows the introduction of several innovations.An advantage of the proposed method is that it requires a minimal amount of computer storage. The Krylov matrix which is built at each iteration is using an improved starting vector and a new three-term recurrence relation. These modifications lead to fast rate of convergence. Numerical experiments illustrate the usefulness of the proposed approach.

A combination of a multiresolution (MR) preconditioner and matrix decomposition algorithm – singular value decomposition (MDA-SVD) algorithm is proposed for the analysis of densely discretised electromagnetic (EM) scatterers. In realistic EM scattering problems, the discretisation may be very dense to accurately describe the geometric shape of targets. However, dense discretisations will generate two major drawbacks: (i) the badly conditioned method of moments (MoM) matrix which makes iterative solvers difficult to converge; (ii) the dense near-field MoM matrix for which the computational cost is large, even multilevel fast multipole algorithm (MLFMA) is applied. To remedy the first shortcoming of dense discretisations, an MR preconditioner can be applied to improve the condition of MoM matrices. To reduce the computational cost in solving the MoM matrix equation, an MDA-SVD algorithm can be applied to accelerate the matrix−vector multiplication. Compared with the traditional MLFMA, the MDA-SVD algorithm has no limit on the size of octree boxes which makes the MDA-SVD algorithm suitable for the analysis of dense discretisations. Therefore an MR preconditioner is combined with the MDA-SVD algorithm for fast analysis of dense discretisations. Numerical results indicate that the combination of the MR preconditioner and the MDA-SVD algorithm is efficient.

Multiple-input multiple-output (MIMO) wireless communication systems with orthogonal frequency-division multiplexing (OFDM) achieve high spectral efficiency high channel capacity, and many MIMO-OFDM systems use the spatial multiplexing technique to improve the system performance. The MIMO-OFDM systems require the singular values and the corresponding singular vectors of the channel matrix by exploiting singular value decomposition (SVD). The information of the right singular-vector matrix can be fed back to the transmitter for linear precoding to improve the error performance when facing the channel matrix with rank deficiency problem. This work proposes a SVD algorithm with fast convergence speed, which is suitable for the MIMO channels with short coherent time. The proposed SVD algorithm has the following features: (1) low total computational complexity, (2) fast convergence speed, (3) the ability of reconfigurable to different numbers of transmitter and receiver antennas, and (4) insensitive to the dynamic range problem, which is suitable for hardware implementation.