Low Rank Approximation of Binary Matrices: Column Subset Selection and Generalizations

We propose a new algorithm for the computation of a singular value decomposition (SVD) low-rank approximation of a matrix in the matrix product operator (MPO) format, also called the tensor train matrix format. Our tensor network randomized SVD (TNrSVD) algorithm is an MPO implementation of the randomized SVD algorithm that is able to compute dominant singular values and their corresponding singular vectors. In contrast to the state-of-the-art tensor-based alternating least squares SVD (ALS-SVD) and modified alternating least squares SVD (MALS-SVD) matrix approximation methods, TNrSVD can be up to 13 times faster while achieving better accuracy. In addition, our TNrSVD algorithm also produces accurate approximations in particular cases where both ALS-SVD and MALS-SVD fail to converge. We also propose a new algorithm for the fast conversion of a sparse matrix into its corresponding MPO form, which is up to 509 times faster than the standard tensor train SVD method while achieving machine precision accuracy. The efficiency and accuracy of both algorithms are demonstrated in numerical experiments.

Low-rank approximation of large and/or sparse matrices is important in many applications, and the singular value decomposition (SVD) gives the best low-rank approximations with respect to unitarily-invariant norms. In this paper we show that good low-rank approximations can be directly obtained from the Lanczos bidiagonalization process applied to the given matrix without computing any SVD. We also demonstrate that a so-called one-sided reorthogonalization process can be used to maintain an adequate level of orthogonality among the Lanczos vectors and produce accurate low-rank approximations. This technique reduces the computational cost of the Lanczos bidiagonalization process. We illustrate the efficiency and applicability of our algorithm using numerical examples from several applications areas.

In this paper, we introduce a method for multivariate function approximation using function evaluations, Chebyshev polynomials, and tensor-based compression techniques via the Tucker format. We develop novel randomized techniques to accomplish the tensor compression, provide a detailed analysis of the computational costs, provide insight into the error of the resulting approximations, and discuss the benefits of the proposed approaches. We also apply the tensor-based function approximation to develop low-rank matrix approximations to kernel matrices that describe pairwise interactions between two sets of points; the resulting low-rank approximations are efficient to compute and store (the complexity is linear in the number of points). We present an adaptive version of the function and kernel approximation that determines an approximation that satisfies a user-specified relative error over a set of random points. We extend our approach to the case where the kernel requires repeated evaluations for many values of (hyper)parameters that govern the kernel. We give detailed numerical experiments on example problems involving multivariate function approximation, low-rank matrix approximations of kernel matrices involving well-separated clusters of sources and target points, and a global low-rank approximation of kernel matrices with an application to Gaussian processes. We observe speedups up to 18X over standard matrix-based approaches.

Detecting anomalous traffic is a critical task for advanced Internet management. The traditional approaches based on Principal Component Analysis (PCA) are effective only when the corruption is caused by small additive i.i.d. Gaussian noise. The recent Direct Robust Matrix Factorization (DRMF) is proven to be more robust and accurate in anomaly detection, but it incurs a high computation cost due to its need of singular value decomposition (SVD) for low-rank matrix approximation and the iterative use of SVD execution to find the final solution. To enable the anomaly detection for large traffic matrix with the use of DRMF, we formulate the low-rank matrix approximation problem as a problem of searching for the subspace to project the traffic matrix with the minimum error. We propose a novel approach, LSH-subspace, for fast low-rank matrix approximation. To facilitate the matrix partition for the quick search of the subspace, we propose several novel techniques: a multi-layer locality sensitive hashing (LSH) table to reorder the OD pairs based on LSH function, a partition principle to guide the partition to minimize the projection error, and a lightweight algorithm to exploit the sparsity of the outlier matrix to update the LSH table at low overhead. Our extensive simulations based on real trace data demonstrate that our LSH-subspace is 3 times faster than DRMF with high anomaly detection accuracy.

We propose new approximate alternating projection methods, based on randomized sketching, for the low-rank nonnegative matrix approximation problem: find a low-rank approximation of a nonnegative matrix that is nonnegative, but whose factors can be arbitrary. We calculate the computational complexities of the proposed methods and evaluate their performance in numerical experiments. The comparison with the known deterministic alternating projection methods shows that the randomized approaches are faster and exhibit similar convergence properties.

Least Squares Support Vector Machine (LS-SVM) has been recently applied to non-parametric identification of Linear Parameter Varying (LPV) systems, described by the AutoRegressive with eXogenous input (ARX). However, the online application of LPV-ARX system in the LS-SVM setting requires high computational time, related to the number of training data used to compute the coefficients of the identified model, limiting the possibility to use the method to real-time applications. In this paper, the authors propose the Low-Rank (LR) matrix approximation and a pruning based approach to compute a sparse solution. In particular, the pruning algorithm is considered to compute off-line a sparse solution of Lagrangian multipliers and then speed up the testing stage, whereas the LR matrix approximation allows to speed up the training stage. The proposed approach has been tested by identifying a subsystem of a vehicle powertrain model by the input/output data collected from the simulation model. The proposed approach has been compared with respect to the standard approach based on LS-SVM. The methods are tested on the considered real-world problem and the proposed approach permits to reduce the execution time of about 77% on average in the considered identification problem, corresponding to a degradation of the identification result less than 0.2% with respect to the standard solution.

We study a weighted low-rank approximation that is inspired by a problem of constrained low-rank approximation of matrices as initiated by the work of Golub, Hoffman, and Stewart [Linear Algebra Appl., 88/89 (1987), pp. 317--327]. Our results reduce to that of Golub, Hoffman, and Stewart in the limiting cases. We also propose an algorithm based on the alternating direction method to solve our weighted low-rank approximation problem and compare it with the state-of-art general algorithms such as the weighted total alternating least squares algorithm and the expectation maximization algorithm.

We describe a local low rank approximation method based on super-rays for light field compression. Super-rays can be seen as a set of super-pixels that are coherent across all light field views. A super-ray based disparity estimation method is proposed using a low rank prior, in order to be able to align all the super-pixels forming each super-ray. A dedicated super-ray construction method is described that constrains the super-pixels forming a given super-ray to be all of the same shape and size, dealing with occlusions. This constraint is needed so that the super-rays can be used as a support of angular dimensionality reduction based on low rank matrix approximation. A low rank matrix approximation is then computed on the disparity compensated super-rays using a singular value decomposition (SVD). A coding algorithm is then described for the different components of the resulting low rank approximation. Experimental results show performance gains compared with two reference light field coding schemes (HEVC-based scheme and JPEG-Pleno VM 1.1).

Matrix low rank approximation including the classical PCA and the robust PCA (RPCA) method have been applied to solve the background modeling problem in video analysis. Recently, it has been demonstrated that a special weighted low rank approximation of matrices can be made robust to the outliers similar to the $\ell_1$-norm in RPCA method. In this work, we propose a new algorithm that can speed up the existing algorithm for solving the special weighted low rank approximation and demonstrate its use in background estimation problem.

Acoustic modeling using mixtures of multivariate Gaussians is the prevalent approach for many speech processing problems. Computing likelihoods against a large set of Gaussians is required as a part of many speech processing systems and it is the computationally dominant phase for LVCSR systems. We express the likelihood computation as a multiplication of matrices representing augmented feature vectors and Gaussian parameters. The computational gain of this approach over traditional methods is by exploiting the structure of these matrices and efficient implementation of their multiplication. In particular, we explore direct low-rank approximation of the Gaussian parameter matrix and indirect derivation of low-rank factors of the Gaussian parameter matrix by optimum approximation of the likelihood matrix. We show that both the methods lead to similar speedups but the latter leads to far lesser impact on the recognition accuracy. Experiments on a 1138 word vocabulary RM1 task using Sphinx 3.7 system show that, for a typical case the matrix multiplication approach leads to overall speedup of 46%. Both the low-rank approximation methods increase the speedup to around 60%, with the former method increasing the word error rate (WER) from 3.2% to 6.6%, while the latter increases the WER from 3.2% to 3.5%.

Subspace algorithms based on higher-order cumulants were developed to achieve high-resolution separation in non-Gaussian processes. However, singular value decomposition (SVD) of a huge matrix is an unavoidable step of these algorithms. The memory space and running time required by the decomposition are super-linear with respect to the size of the matrix, which is prohibitive in terms of practical applications. Thus, in this paper, a fast raypath separation algorithm based on low-rank matrix approximation is proposed in a shallow-water waveguide. The experimental results illustrate that the proposed algorithm dramatically reduces the consumption of time and space, with arbitrarily small error, compared to conventional higher-order cumulant-based algorithms.

In this paper we present a new type of restarted Krylov method for calculating low-rank approximations of large matrices. In contrast to former Krylov methods, our approach does not apply the Lanczos algorithm, or Lanczos bidiagonalization. This simplifies the basic iteration and allows the introduction of several innovations.An advantage of the proposed method is that it requires a minimal amount of computer storage. The Krylov matrix which is built at each iteration is using an improved starting vector and a new three-term recurrence relation. These modifications lead to fast rate of convergence. Numerical experiments illustrate the usefulness of the proposed approach.

Observations from hyperspectral imaging sensors lead to high dimensional data sets from hundreds of images taken at closely spaced narrow spectral bands. High storage and transmission requirements, computational complexity, and statistical modeling problems combined with physical insight motivate the idea of hyperspectral dimensionality reduction using band subset selection. Many algorithms are described in the literature to solve supervised and unsupervised band subset selection problems. This paper explores the use of unsupervised band subset selection methods using column subset selection (CSS). Column subset selection is the problem (CSSP) of selecting the most independent columns of a matrix. A recent variant of this problem is the positive column subset selection problem (pCSSP) which restricts column subset selection to only consider positive linear combinations. Many algorithms have been proposed in the literature for the solution of the CSSP. However, the pCSSP is less studied. This paper will present a comparison of different algorithms to solve the CSSP and the pCSSP for band subset selection. The performance of classifiers using the algorithms as a dimensionality reduction stage will be used to evaluate the usefulness of these algorithms in hyperspectral image exploitation.

Fixed-precision randomized quaternion singular value decomposition algorithm for low-rank quaternion matrix approximations

Convolution neural networks(CNNs) have an impressive performance in image processing and other machine learning tasks, meanwhile, colossal computation and memory requirements existing in most classical models restrict the deployment in portable and power-limited devices. The efficient approaches to circumventing the above hindrances stem from shrinking the network scale, which can be divided into two categories, i.e., the network pruning and low-rank approximation of kernel matrices. As compared to the pruning scheme, the low-rank approximation method has lower compression ratio, but is much friendlier to parallelism. In this paper, by analyzing the sensitivity of the rank of each layer to the network accuracy, we proposed a sensitivity-based layer-wise low-rank approximation algorithm. As compared with the traditional rank reduce methods, the acceleration ratio of our proposal is improved by 20%. When deploying our method on VGGNet-16 model, 2.7x compression/acceleration ratio on convolution layers and 10.9x compression/acceleration ratio on FC layers are achieved with 0.05% top-1 accuracy loss and 0.01% top-5 accuracy loss.