A Schur complement approach to preconditioning sparse linear least-squares problems with some dense rows

Adaptive in-time local grid refinement techniques use multilevel local discretizations designed to achieve local accuracy. The changing nature of the matrix structure of the linear systems arising from the multilevel local discretizations requires flexible approximate factorizations that focus on local components and coordinate their interaction. The solution of these composite grid systems with Krylov solvers is considered. The selection of an adequate preconditioner is crucial. The incomplete Cholesky (IC) factorization of the composite matrix and the inexact BEPS preconditioner are two such potential preconditioners. The inexact BEPS preconditioner can be constructed and applied with significantly more flexibility than the IC factorizations of the composite multilevel grid matrix. An extension of the IC factorizations for matrices arising from discretizations of self-adjoint PDEs on multilevel composite grids is proposed. The resulting factorization is referred to as the incomplete multilevel Cholesky (IMC) factorization. The IMC factorization is spectrally equivalent to the IC factorization of the matrix of the composite grid system. IMC factorization can be constructed with the same flexibility as the inexact BEPS preconditioner. The application of IMC factorization is achieved via a multilevel LLT-cycle consisting of a forward elimination pass proceeding downward on the grids from fine to coarse followed by an reverse-order upward back substitution pass. The application of multilevel factorization as a preconditioner requires roughly one-half as many operations as the inexact BEPS preconditioner. The numerical results illustrate the potential of the method.

In an extended element-free Galerkin method (X-EFG), the essential and natural boundary conditions can be imposed by the collocation based method. However, a coefficient matrix of linear systems obtained by X-EFG become asymmetric, although a symmetric structure exists in a part of the coefficient matrix. In fact, the structure of the coefficient matrices almost becomes symmetric when the size of linear systems is large. Hence, efficient effects may be obtained by using preconditioning for symmetric matrices. The purpose of the present study is to investigate effects of preconditioning for symmetric matrices to linear systems obtained by X-EFG. To this end, the incomplete Cholesky factorization (IC) is applied to the linear systems by regarding the coefficient matrix as symmetric one. In numerical experiments, it is found that the linear systems obtained by X-EFG can be efficiently solved by using IC as preconditioning for GMRES(m) and Bi-CGSTAB. In some cases, the efficiency of IC is superior than that of the incomplete LDU factorization as preconditioning for these iterative methods.

Iterative pre-conditioning for expediting the distributed gradient-descent method: The case of linear least-squares problem

Inspired by Polyak’s heavy-ball method, this paper proposes an adaptive deterministic block coordinate descent method with momentum (mADBCD) for efficiently solving large-scale linear least-squares problems. The proposed method introduces a novel column selection criterion based on the Euclidean norm of the residual vector of the normal equation. In contrast to classical block coordinate descent methods, mADBCD does not require a fixed pre-partitioning of the column indices of the coefficient matrix and avoids the expensive computation of Moore Penrose pseudoinverses of submatrices at each iteration. The method adaptively updates the block index set at each step, thereby improving both flexibility and scalability. When the coefficient matrix is of full column rank, we prove that mADBCD converges linearly to the unique solution of the least-squares problem. Numerical experiments are conducted to show that mADBCD outperforms several recent block coordinate descent methods in terms of iteration count and CPU time. In particular, when solving extremely sparse least-squares problems, mADBCD is the first block coordinate descent method reported to achieve CPU time nearly comparable to that of the classical least squares QR (LSQR) method [Paige and Saunders, ACM Trans. Math. Softw., 8 (1982)].

We propose and analyze a new tool to help solve sparse linear least-squares problems $\min_{x}\|Ax-b\|_{2}$. Our method is based on a sparse $QR$ factorization of a low-rank perturbation $\hat{A}$ of $A$. More precisely, we show that the $R$ factor of $\hat{A}$ is an effective preconditioner for the least-squares problem $\min_{x}\|Ax-b\|_{2}$, when solved using LSQR. We propose applications for the new technique. When $A$ is rank deficient, we can add rows to ensure that the preconditioner is well conditioned without column pivoting. When $A$ is sparse except for a few dense rows, we can drop these dense rows from $A$ to obtain $\hat{A}$. Another application is solving an updated or downdated problem. If $R$ is a good preconditioner for the original problem $A$, it is a good preconditioner for the updated/downdated problem $\hat{A}$. We can also solve what-if scenarios, where we want to find the solution if a column of the original matrix is changed/removed. We present a spectral theory that analyzes the generalized spectrum of the pencil $(A^{*}A,R^{*}R)$ and analyze the applications.

Block preconditionings for the conjugate gradient method are investigated for solving positive definite block tridiagonal systems of linear equations arising from discretization of boundary value problems for elliptic partial differential equations. The preconditionings rest on the use of sparse approximate matrix inverses to generate incomplete block Cholesky factorizations. Carrying out of the factorizations can be guaranteed under suitable conditions. Numerical experiments on test problems for two dimensions indicate that a particularly attractive preconditioning, which uses special properties of tridiagonal matrix inverses, can be computationally more efficient for the same computer storage than other preconditionings, including the popular point incomplete Cholesky factorization.

Incomplete factorization has been shown to be a good preconditioner for the conjugate gradient method on a wide variety of problems. It is well known that allowing some fill-in during the incomplete factorization can significantly reduce the number of iterations needed for convergence. Allowing fill-in, however, increases the time for the factorization and for the triangular system solutions. Additionally, it is difficult to predict a priori how much fill-in to allow and how to allow it. The unpredictability of the required storage/work and the unknown benefits of the additional fill-in make such strategies impractical to use in many situations. In this article we motivate, and then present, two “black-box” strategies that significantly increase the effectiveness of incomplete Cholesky factorization as a preconditioner. These strategies require no parameters from the user and do not increase the cost of the triangular system solutions. Efficient implementations for these algorithms are described. These algorithms are shown to be successful for a variety of problems from the Harwell-Boeing sparse matrix collection.

A coupled fully implicit method for solving drag-adjoint equations of steady incompressible flows

Graph spectral sparsification-based preconditioning technique has shown promising results for power grid analysis. However, the conventional methods converge slowly for high-accuracy requirement. In this work, we propose an efficient approach to address this issue. Instead of using the Cholesky factorization, we employ the incomplete Cholesky factorization to factorize the spectral sparsifier. We also propose a concept of graph spectral pattern, which can further reduce the preconditioned conjugate gradient (PCG) iterations using less number of nonzeros. Experiments show that under 10−6 relative tolerance, our proposed preconditioning technique achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.17\times $ </tex-math></inline-formula> speedup compared to AMGPCG in average; compared to the conventional spectral sparsification-based preconditioning techniques, our proposed approach achieves up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.53\times $ </tex-math></inline-formula> speedup of the factorization, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.74\times $ </tex-math></inline-formula> speedup of the PCG iteration, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.6\times $ </tex-math></inline-formula> speedup of the total time. Moreover, the speedup of the total time continues to enlarge for higher-accuracy requirement, e.g., 10−12. Finally, but not least, our method is compatible with existing graph spectral sparsification algorithms for power grid analysis.

Wilkinson defined a sparse matrix as one with enough zeros that it pays to take advantage of them. 1 This informal yet practical definition captures the essence of the goal of direct methods for solving sparse matrix problems. They exploit the sparsity of a matrix to solve problems economically: much faster and using far less memory than if all the entries of a matrix were stored and took part in explicit computations. These methods form the backbone of a wide range of problems in computational science. A glimpse of the breadth of applications relying on sparse solvers can be seen in the origins of matrices in published matrix benchmark collections (Duff and Reid 1979 a , Duff, Grimes and Lewis 1989 a , Davis and Hu 2011). The goal of this survey article is to impart a working knowledge of the underlying theory and practice of sparse direct methods for solving linear systems and least-squares problems, and to provide an overview of the algorithms, data structures, and software available to solve these problems, so that the reader can both understand the methods and know how best to use them.

The solution of the unconstrained weighted linear least-squares problem is known to be a convex combination of the basic solutions formed by the nonsingular subsystems if the weight matrix is diagonal and positive definite. In particular, this implies that the norm of this solution is uniformly bounded for any diagonal and positive definite weight matrix. In addition, the solution set is known to be the relative interior of a finite set of polytopes if the weight matrix varies over the set of positive definite diagonal matrices. In this paper, these results are reviewed and generalized to the set of weight matrices that are symmetric, positive semidefinite, and diagonally dominant and that give unique solution to the least-squares problem. This is done by means of a particular symmetric diagonal decomposition of the weight matrix, giving a finite number of diagonally weighted problems but in a space of higher dimension. Extensions to equality-constrained weighted linear least-squares problems are given. A discussion of why the boundedness properties do not hold for general symmetric positive definite weight matrices is given. The motivation for this research is from interior methods for optimization.

Fast kernel extreme learning machine for ordinal regression

This work describes a novel physics‐based deflation preconditioner approach for solving porous media flow problems characterized by highly heterogeneous media. The approach relies on high‐conductivity block solutions after rearranging the linear system coefficients into high‐conductive and low condutive blocks from a given physically driven threshold value. This rearranging relies on the Hoshen‐Kopelman (H‐K) algorithm that is commonly used to determine percolation clusters. The resulting preconditioner may alternatively be combined with a deflation preconditioning stage. The proposed approach is coined as a physics‐based 2‐stage deflation preconditioner (P2SDP). Numerical experiments on different permeability distributions reveal that P2SDP is a powerful means to solve pressure systems when compared to more conventional algebraic approaches such as the incomplete Cholesky factorization. (© 2008 WILEY‐VCH Verlag GmbH &amp; Co. KGaA, Weinheim)

A parallel preconditioned conjugate gradient package for solving sparse linear systems on a Cray Y-MP

The main purpose of this work is to present a new parallel direct solver: Dissection solver. It is based on LU factorization of the sparse matrix of the linear system and allows to detect automatically and handle properly the zero‐energy modes, which are important when dealing with DDM. A performance evaluation and comparisons with other direct solvers (MUMPS, DSCPACK) are also given for both sequential and parallel computations. Results of numerical experiments with a two‐level parallelization of large‐scale structural analysis problems are also presented: FETI is used for the global problem parallelization and Dissection for the local multithreading. In this framework, the largest problem we have solved is of an elastic solid composed of 400 subdomains running on 400 computation nodes (3200 cores) and containing about 165 millions dof. The computation of one single iteration consumes less than 20 min of CPU time. Several comparisons to MUMPS are given for the numerical computation of large‐scale linear systems on a massively parallel cluster: performances and weaknesses of this new solver are highlighted. Copyright © 2011 John Wiley &amp; Sons, Ltd.