Efficient randomized tensor-based algorithms for function approximation and low-rank kernel interactions

Low rank approximation of matrices is an important tool in machine learning. Given a data matrix, low rank approximation helps to find factors, patterns, and provides concise representations for the data. Research on low rank approximation usually focuses on real matrices. However, in many applications data are binary (categorical) rather than continuous. This leads to the problem of low rank approximation of binary matrices. Here we are given a d x n binary matrix A and a small integer k < d. The goal is to find two binary matrices U and V of sizes d x k and k x n respectively, so that the Frobenius norm of A - U V is minimized. There are two models of this problem, depending on the definition of the dot product of binary vectors: The GF(2) model and the Boolean semiring model. Unlike low rank approximation of a real matrix which can be efficiently solved by Singular Value Decomposition, we show that approximation of a binary matrix is NP-hard, even for k=1.\nIn this paper, our main concern is the problem of Column Subset Selection (CSS), in which the low rank matrix U must be formed by k columns of the data matrix, and we are interested in the approximation ratio achievable by CSS for binary matrices. For the GF(2) model, we show that CSS has approximation ratio bounded by k/2+1+k/(2(2^k-1)) and this is asymptotically tight. For the Boolean model, it turns out that CSS is no longer sufficient to obtain a bound. We then develop a Generalized CSS (GCSS) procedure in which the columns of U are generated from Boolean formulas operating bitwise on selected columns of the data matrix. We show that the approximation ratio achieved by GCSS is bounded by 2^(k-1)+1, and argue that an exponential dependency on k is seems inherent.

An algorithm for low-rank approximation of bivariate functions using splines

Low-rank approximation of large and/or sparse matrices is important in many applications, and the singular value decomposition (SVD) gives the best low-rank approximations with respect to unitarily-invariant norms. In this paper we show that good low-rank approximations can be directly obtained from the Lanczos bidiagonalization process applied to the given matrix without computing any SVD. We also demonstrate that a so-called one-sided reorthogonalization process can be used to maintain an adequate level of orthogonality among the Lanczos vectors and produce accurate low-rank approximations. This technique reduces the computational cost of the Lanczos bidiagonalization process. We illustrate the efficiency and applicability of our algorithm using numerical examples from several applications areas.

We improve a Monte Carlo algorithm which computes accurate approximations of smooth functions on multidimensional Tchebychef polynomials by using quasi-random sequences. We first show that the convergence of the previous algorithm is twice faster using these sequences. Then, we slightly modify this algorithm to make it work from a single set of random or quasi-random points. This especially leads to a Quasi-Monte Carlo method with an increased rate of convergence for numerical integration.

We describe a local low rank approximation method based on super-rays for light field compression. Super-rays can be seen as a set of super-pixels that are coherent across all light field views. A super-ray based disparity estimation method is proposed using a low rank prior, in order to be able to align all the super-pixels forming each super-ray. A dedicated super-ray construction method is described that constrains the super-pixels forming a given super-ray to be all of the same shape and size, dealing with occlusions. This constraint is needed so that the super-rays can be used as a support of angular dimensionality reduction based on low rank matrix approximation. A low rank matrix approximation is then computed on the disparity compensated super-rays using a singular value decomposition (SVD). A coding algorithm is then described for the different components of the resulting low rank approximation. Experimental results show performance gains compared with two reference light field coding schemes (HEVC-based scheme and JPEG-Pleno VM 1.1).

Acoustic modeling using mixtures of multivariate Gaussians is the prevalent approach for many speech processing problems. Computing likelihoods against a large set of Gaussians is required as a part of many speech processing systems and it is the computationally dominant phase for LVCSR systems. We express the likelihood computation as a multiplication of matrices representing augmented feature vectors and Gaussian parameters. The computational gain of this approach over traditional methods is by exploiting the structure of these matrices and efficient implementation of their multiplication. In particular, we explore direct low-rank approximation of the Gaussian parameter matrix and indirect derivation of low-rank factors of the Gaussian parameter matrix by optimum approximation of the likelihood matrix. We show that both the methods lead to similar speedups but the latter leads to far lesser impact on the recognition accuracy. Experiments on a 1138 word vocabulary RM1 task using Sphinx 3.7 system show that, for a typical case the matrix multiplication approach leads to overall speedup of 46%. Both the low-rank approximation methods increase the speedup to around 60%, with the former method increasing the word error rate (WER) from 3.2% to 6.6%, while the latter increases the WER from 3.2% to 3.5%.

Detecting anomalous traffic is a critical task for advanced Internet management. The traditional approaches based on Principal Component Analysis (PCA) are effective only when the corruption is caused by small additive i.i.d. Gaussian noise. The recent Direct Robust Matrix Factorization (DRMF) is proven to be more robust and accurate in anomaly detection, but it incurs a high computation cost due to its need of singular value decomposition (SVD) for low-rank matrix approximation and the iterative use of SVD execution to find the final solution. To enable the anomaly detection for large traffic matrix with the use of DRMF, we formulate the low-rank matrix approximation problem as a problem of searching for the subspace to project the traffic matrix with the minimum error. We propose a novel approach, LSH-subspace, for fast low-rank matrix approximation. To facilitate the matrix partition for the quick search of the subspace, we propose several novel techniques: a multi-layer locality sensitive hashing (LSH) table to reorder the OD pairs based on LSH function, a partition principle to guide the partition to minimize the projection error, and a lightweight algorithm to exploit the sparsity of the outlier matrix to update the LSH table at low overhead. Our extensive simulations based on real trace data demonstrate that our LSH-subspace is 3 times faster than DRMF with high anomaly detection accuracy.

It has been known in potential theory that, for some kernels matrices corresponding to well-separated point sets, fast analytical low-rank approximation can be achieved via the use of proxy points. This proxy point method gives a surprisingly convenient way of explicitly writing out approximate basis matrices for a kernel matrix. However, this elegant strategy is rarely known or used in the numerical linear algebra community. It still needs clear algebraic understanding of the theoretical background. Moreover, rigorous quantifications of the approximation errors and reliable criteria for the selection of the proxy points are still missing. In this work, we use contour integration to clearly justify the idea in terms of a class of important kernels. We further provide comprehensive accuracy analysis for the analytical compression and show how to choose nearly optimal proxy points. The analytical compression is then combined with fast rank-revealing factorizations to get compact low-rank approximations and also to select certain representative points. We provide the error bounds for the resulting overall low-rank approximation. This work thus gives a fast and reliable strategy for compressing those kernel matrices. Furthermore, it provides an intuitive way of understanding the proxy point method and bridges the gap between this useful analytical strategy and practical low-rank approximations. Some numerical examples help to further illustrate the ideas.

Matrix low rank approximation including the classical PCA and the robust PCA (RPCA) method have been applied to solve the background modeling problem in video analysis. Recently, it has been demonstrated that a special weighted low rank approximation of matrices can be made robust to the outliers similar to the $\ell_1$-norm in RPCA method. In this work, we propose a new algorithm that can speed up the existing algorithm for solving the special weighted low rank approximation and demonstrate its use in background estimation problem.

In this paper, a pattern search (PS)-based solution is proposed for nonconvex multiband cooperative sensing (NCMCS) problem in cognitive radio systems. This problem consists of maximizing cumulative throughput subject to constraints on cumulative interference, probability of detection, and probability of false alarm. Initially in existing literature, this problem was solved under constraints that make it convex. However, removing the conditions for convexity and solving the NCMCS problem have been shown to improve performance. A two-step PS-based solution is presented: The first step uses uniformly distributed random sets of input points to find a solution. The set of points that gives the maximum throughput is chosen as input to the PS algorithm. Numerical examples show the improvement of the proposed method over existing genetic-algorithm-based solution, as well as PS-algorithm-based solution that uses a single set of random points as inputs. The proposed two-step solution gives higher cumulative throughput and is not sensitive to the choice of input, unlike the PS-based solution using a single set of random points as input.

We propose new approximate alternating projection methods, based on randomized sketching, for the low-rank nonnegative matrix approximation problem: find a low-rank approximation of a nonnegative matrix that is nonnegative, but whose factors can be arbitrary. We calculate the computational complexities of the proposed methods and evaluate their performance in numerical experiments. The comparison with the known deterministic alternating projection methods shows that the randomized approaches are faster and exhibit similar convergence properties.

Frieze et al. [17] proved that a small sample of rows of a given matrix A contains a low-rank approximation D that minimizes ||A - D||F to within small additive error, and the sampling can be done efficiently using just two passes over the matrix [12]. In this paper, we generalize this result in two ways. First, we prove that the additive error drops exponentially by iterating the sampling in an adaptive manner. Using this result, we give a pass-efficient algorithm for computing low-rank approximation with reduced additive error. Our second result is that using a natural distribution on subsets of rows (called volume sampling), there exists a subset of k rows whose span contains a factor (k + 1) relative approximation and a subset of k + k(k + 1)/e rows whose span contains a 1+e relative approximation. The existence of such a small certificate for multiplicative low-rank approximation leads to a PTAS for the following projective clustering problem: Given a set of points P in Rd, and integers k, j, find a set of j subspaces F1, . . ., Fj, each of dimension at most k, that minimize Σp∈Pminid(p, Fi)2.

Subspace algorithms based on higher-order cumulants were developed to achieve high-resolution separation in non-Gaussian processes. However, singular value decomposition (SVD) of a huge matrix is an unavoidable step of these algorithms. The memory space and running time required by the decomposition are super-linear with respect to the size of the matrix, which is prohibitive in terms of practical applications. Thus, in this paper, a fast raypath separation algorithm based on low-rank matrix approximation is proposed in a shallow-water waveguide. The experimental results illustrate that the proposed algorithm dramatically reduces the consumption of time and space, with arbitrarily small error, compared to conventional higher-order cumulant-based algorithms.

Convolution neural networks(CNNs) have an impressive performance in image processing and other machine learning tasks, meanwhile, colossal computation and memory requirements existing in most classical models restrict the deployment in portable and power-limited devices. The efficient approaches to circumventing the above hindrances stem from shrinking the network scale, which can be divided into two categories, i.e., the network pruning and low-rank approximation of kernel matrices. As compared to the pruning scheme, the low-rank approximation method has lower compression ratio, but is much friendlier to parallelism. In this paper, by analyzing the sensitivity of the rank of each layer to the network accuracy, we proposed a sensitivity-based layer-wise low-rank approximation algorithm. As compared with the traditional rank reduce methods, the acceleration ratio of our proposal is improved by 20%. When deploying our method on VGGNet-16 model, 2.7x compression/acceleration ratio on convolution layers and 10.9x compression/acceleration ratio on FC layers are achieved with 0.05% top-1 accuracy loss and 0.01% top-5 accuracy loss.

Least Squares Support Vector Machine (LS-SVM) has been recently applied to non-parametric identification of Linear Parameter Varying (LPV) systems, described by the AutoRegressive with eXogenous input (ARX). However, the online application of LPV-ARX system in the LS-SVM setting requires high computational time, related to the number of training data used to compute the coefficients of the identified model, limiting the possibility to use the method to real-time applications. In this paper, the authors propose the Low-Rank (LR) matrix approximation and a pruning based approach to compute a sparse solution. In particular, the pruning algorithm is considered to compute off-line a sparse solution of Lagrangian multipliers and then speed up the testing stage, whereas the LR matrix approximation allows to speed up the training stage. The proposed approach has been tested by identifying a subsystem of a vehicle powertrain model by the input/output data collected from the simulation model. The proposed approach has been compared with respect to the standard approach based on LS-SVM. The methods are tested on the considered real-world problem and the proposed approach permits to reduce the execution time of about 77% on average in the considered identification problem, corresponding to a degradation of the identification result less than 0.2% with respect to the standard solution.