Distributed O(N) Linear Solver for Dense Symmetric Hierarchical Semi-Separable Matrices

Symmetric positive definite (SPD) matrices used as feature descriptors in image recognition are usually high dimensional. Traditional manifold learning is only applicable for reducing the dimension of high-dimensional vector-form data.For high-dimensional SPD matrices, directly using manifold learning algorithms to reduce the dimension of matrix-form data is impossible. The SPD matrix must first be transformed into a long vector, and then the dimension of this vector must be reduced. However, this approach breaks the spatial structure of the SPD matrix space. To overcome this limitation, we propose a new dimension reduction algorithm on SPD matrix space to transform high-dimensional SPD matrices into low-dimensional SPD matrices. Our work is based on the fact that the set of all SPD matrices with the same size has a Lie group structure, and we aim to transform the manifold learning to the SPD matrix Lie group. We use the basic idea of the manifold learning algorithm called locality preserving projection (LPP) to construct the corresponding Laplacian matrix on the SPD matrix Lie group. Thus, we call our approach Lie-LPP to emphasize its Lie group character. We present a detailed algorithm analysis and show through experiments that Lie-LPP achieves effective results on human action recognition and human face recognition.

<p><b>Metric Learning has proved valuable in information retrieval and classification problems, with many algorithms using statistical formulations to their advantage. Optimal Transport, a powerful framework for computing distances between probability distributions, has seen significant uptake in Machine Learning and some applications in Metric Learning. The Bures-Wasserstein distance, a particular formulation of Optimal Transport, has seen limited application in Metric Learning despite being well suited to the field. In this thesis Linear Metric Learning algorithms incorporating the Bures-Wasserstein distance are developed, utilising its novel properties. The first set of algorithms we contribute use the Bures-Wasserstein distance for learning linear metrics using Cyclic Projections, Riemannian barycenters and gradient-based algorithms. The convergence properties and classification performance of these algorithms are then compared to benchmark Linear Metric Learning algorithms. Our algorithms achieve competitive classification performance on feature and image datasets and converge reliability.</b></p> <p>The second set of contributions extend Linear Metric Learning with the Bures-Wasserstein distance to a low-rank context to address issues related to scalability. An extension of the previous Cyclic Projections algorithm and a Riemannian optimisation method are developed. Convergence and classification experiments analogous to the full-rank case are conducted to examine how the reduced rank affects both of these performance metrics. Both algorithms converge reliably under a variety of synthetic conditions. Reduction in rank generally decreased the training time for algorithms until effects caused by the size of the training data dominated. Reducing rank either left classification error unchanged or increased it. With respect to other low-rank Linear Metric Learning algorithms our algorithms again performed competitively with respect to classification error. </p> <p>The last set of contributions consider Linear Metric Learning on Symmetric Positive Definite (SPD) matrices. Some datasets are expressed as or are suited to a SPD representation. As such, Metric Learning algorithms that respect the structure of these matrices have found success when adapted to this kind of data. Another extension of the Cyclic Projections algorithm is developed to use the Generalised Bures-Wasserstein distance on SPD matrices. Two other methods based on a ground metric formulation of the Bures-Wasserstein distance for learning linear metrics on SPD data are also introduced. One ground metric formulation uses an approximation (inspired by the Sinkhorn distance) to increase its scalability. Convergence experiments on synthetic problems adapted to SPD matrices are conducted on the ground metric algorithms to establish their convergence properties. Classification experiments are also conducted with respect to several types of SPD data. While the Cyclic Projections method is competitive, the ground metric algorithms have poor performance. Despite their poor classification performance, the approximation method does evaluate faster than the Generalised Bures-Wasserstein distance, potentially offering some scalability improvements.</p>

Representing images and videos with Symmetric Positive Definite (SPD) matrices, and considering the Riemannian geometry of the resulting space, has been shown to yield high discriminative power in many visual recognition tasks. Unfortunately, computation on the Riemannian manifold of SPD matrices -especially of high-dimensional ones- comes at a high cost that limits the applicability of existing techniques. In this paper, we introduce algorithms able to handle high-dimensional SPD matrices by constructing a lower-dimensional SPD manifold. To this end, we propose to model the mapping from the high-dimensional SPD manifold to the low-dimensional one with an orthonormal projection. This lets us formulate dimensionality reduction as the problem of finding a projection that yields a low-dimensional manifold either with maximum discriminative power in the supervised scenario, or with maximum variance of the data in the unsupervised one. We show that learning can be expressed as an optimization problem on a Grassmann manifold and discuss fast solutions for special cases. Our evaluation on several classification tasks evidences that our approach leads to a significant accuracy gain over state-of-the-art methods.

Symmetric Positive Definite (SPD) matrices have been used in many fields of medical data analysis. Many Riemannian metrics have been defined on this manifold but the choice of the Riemannian structure lacks a set of principles that could lead one to choose properly the metric. This drives us to introduce the principle of balanced metrics that relate the affine-invariant metric with the Euclidean and inverse-Euclidean metric, or the Bogoliubov-Kubo-Mori metric with the Euclidean and log-Euclidean metrics. We introduce two new families of balanced metrics, the mixed-power-Euclidean and the mixed-power-affine metrics and we discuss the relation between this new principle of balanced metrics and the concept of dual connections in information geometry.

The geometry of mixed-Euclidean metrics on symmetric positive definite matrices

The region covariance descriptors have recently become a popular method for detection and tracking of objects in an image. However, these descriptors are not suitable for classification of images with heterogeneous contents. In this paper, we present an image-level descriptor obtained using an affine-invariant geodesic mean of region covariance descriptors on the Riemannian manifold of symmetric positive definite (SPD) matrices. The resulting image descriptors are also SPD matrices, lending themselves to tractable geodesic distance based k-nearest neighbour classification using efficient kernels. We show that the proposed descriptor yields high classification accuracy on a challenging problem of nuclear pleomorphism scoring in breast cancer histology images.

Symmetric positive-definite (SPD) matrices are ubiquitous in Computer Vision, Machine Learning and Medical Image Analysis. Finding the center/average of a population of such matrices is a common theme in many algorithms such as clustering, segmentation, principal geodesic analysis, etc. The center of a population of such matrices can be defined using a variety of distance/divergence measures as the minimizer of the sum of squared distances/divergences from the unknown center to the members of the population. It is well known that the computation of the Karcher mean for the space of SPD matrices which is a negatively-curved Riemannian manifold is computationally expensive. Recently, the LogDet divergence-based center was shown to be a computationally attractive alternative. However, the LogDet-based mean of more than two matrices can not be computed in closed form, which makes it computationally less attractive for large populations. In this paper we present a novel recursive estimator for center based on the Stein distance - which is the square root of the LogDet divergence - that is significantly faster than the batch mode computation of this center. The key theoretical contribution is a closed-form solution for the weighted Stein center of two SPD matrices, which is used in the recursive computation of the Stein center for a population of SPD matrices. Additionally, we show experimental evidence of the convergence of our recursive Stein center estimator to the batch mode Stein center. We present applications of our recursive estimator to K-means clustering and image indexing depicting significant time gains over corresponding algorithms that use the batch mode computations. For the latter application, we develop novel hashing functions using the Stein distance and apply it to publicly available data sets, and experimental results have shown favorable comparisons to other competing methods.

Personal authentication using electroencephalographic (EEG) signals, is one of the important applications in brain computer interface (BCI). In this work we investigate the use of EEG signals as a biometric trait. Multidimensional EEG signals were represented as symmetric positive-definite (SPD) matrices on a Riemannian manifold. Two experiments are performed in the first; we use minimum distance to Riemannian mean (MDRM) as a classifier. In the second; SPD matrices are vectorized, and the generated vectors are used to train various machine learning (ML) classifiers. MDRM classifier achieved a correct recognition rate (CRR) of 96.92% , while ML classifiers achieved CRR from 95.39% to 99.45%.

For general symmetric positive definite (SPD) matrices, we present a framework for designing effective and robust black-box preconditioners via structured incomplete factorization. In a scaling-and-compression strategy, off-diagonal blocks are first scaled on both sides (by the inverses of the factors of the corresponding diagonal blocks) and then compressed into low-rank approximations. ULV-type factorizations are then computed. A resulting prototype preconditioner is always positive definite. Generalizations to practical hierarchical multilevel preconditioners are given. Systematic analysis of the approximation error, robustness, and effectiveness is shown for both the prototype preconditioner and the multilevel generalization. In particular, we show how local scaling and compression control the approximation accuracy and robustness and how aggressive compression leads to efficient preconditioners that can significantly reduce the condition number and improve the eigenvalue clustering. A result is also ...

We present GOFMM (geometry-oblivious FMM), a novel method that creates a hierarchical low-rank approximation, or compression, of an arbitrary dense symmetric positive definite (SPD) matrix. For many applications, GOFMM enables an approximate matrix-vector multiplication in N log N or even N time, where N is the matrix size. Compression requires N log N storage and work. In general, our scheme belongs to the family of hierarchical matrix approximation methods. In particular, it generalizes the fast multipole method (FMM) to a purely algebraic setting by only requiring the ability to sample matrix entries. Neither geometric information (i.e., point coordinates) nor knowledge of how the matrix entries have been generated is required, thus the term geometry-oblivious. Also, we introduce a shared-memory parallel scheme for hierarchical matrix computations that reduces synchronization barriers. We present results on the Intel Knights Landing and Haswell architectures, and on the NVIDIA Pascal architecture for a variety of matrices.

We propose a new kernel entropy component analysis approach to object recognition with image set by fusing the information entropy. Since the geometry of Symmetric Positive Definite (SPD) matrices, we model the image set with its covariance matrix (nonsingular). Thus the object recognition with image set can be model as classifying problem on the riemannian manifold space. Given the proper kernel function derived from efficient metric for SPD matrices, points lie on manifold space represented by covariance matrices can be cast into high dimensional euclidean space. In this euclidean space, traditional methods used for dimensional reduction can be applied directly. Accounting for the measurability to information of the Renyi entropy, we fuse the information entropy to the dimensional reduction progress of our method. Similar to Kernel Principal Component Analysis (KPCA), the method accomplish data dimensionality reduction by projection onto a subset of entropy preserving KPCA axes. But this subset does not need to correspond to the top eigenvalues of the kernel matrix. The positive results of experiment on datasets demonstrated effectiveness of our method.

Cross-covariance matrix: Time-shifted correlations for 3D action recognition

The use of regional covariance descriptors to generate feature data represented by Symmetric Positive Definite (SPD) matrices from images or videos has become increasingly common in machine learning. However, SPD data itself does not constitute a vector space, and dictionary learning involves a large number of linear operations, so dictionary learning cannot be performed directly on SPD data. For this reason, a more common method is to map the SPD data to the Reproducing Kernel Hilbert Space (RKHS). The so-called kernel learning is to find the most suitable RKHS for specific tasks. RKHS can be uniquely generated by a kernel function. Therefore, RKHS learning can also be considered as kernel learning. In this article, there are two main contributions. The first contribution is to propose a framework which based on Kernel Learning and Riemannian Metric (KLRM). Usually the learnable kernel function framework is to learn some parameters in the kernel function. The second contribution is dictionary learning by applying KLRM to SPD data. The SPD data is transformed into the RKHS generated by KLRM, and RKHS after training provides the most suitable working space for dictionary learning. Under the proposed framework, we design a positive definite kernel function, which is defined by the Log-Euclidean metric. This function can be transformed into a corresponding Riemannian kernel. The experimental results provided in this paper is compared with other state-of-the-art algorithms for SPD data dictionary learning and show that the proposed algorithm achieves better results.

SUMMARYIn this paper, an inner–outer preconditioner for symmetric positive definite matrices based on hierarchically semi‐separable (HSS) matrix representation is presented. A sequence of new HSS algorithms are developed, including some ULV‐type HSS methods and triangular HSS methods. During the construction of this preconditioner, off‐diagonal blocks are compressed in parallel, and an approximate HSS form is used as the outer HSS representation. In the meantime, the internal dense diagonal blocks are approximately factorized into HSS Cholesky factors. Such inner factorizations are guaranteed to exist for any approximation accuracy. The inner–outer preconditioner combines the advantages of both direct HSS and triangular HSS methods, and are both scalable and robust. Systematic complexity analysis and discussions of the errors are presented. Various tests on some practical numerical problems are used to demonstrate the efficiency and robustness. In particular, the effectiveness of the preconditioner in iterative solutions has been shown for some ill‐conditioned problems. This work also gives a practical way of developing inner–outer preconditioners using direct rank structured factorizations (other than iterative methods). Copyright © 2012 John Wiley & Sons, Ltd.

A derivation of the Levinson algorithm for solving linear systems with symmetric positive definite Toeplitz matrix