Work-Preserving Speed-Up of Parallel Matrix Computations

Enhancements to array processors in the form of broadcast buses have been proposed for improving the speeds of algorithms in linear algebra, image processing and computational geometry. In this paper, we consider certain practical issues in such arrays which include fewer processors connected to the broadcast buses and finite time, namely log N time for broadcasting data to N processors. We propose a modified broadcast bus VLSI architecture which consists of a √N × √N mesh connected structure and a hierarchy of broadcast buses in each row and in each column such that every bus has k PE's. That is, in any row or column, in the first level there are \(\frac{{\sqrt N }}{k}\)buses with groups of k PE's connected to each bus. One PE from each of this group is connected to second level of buses in a similar manner. This is recursively done until there are only k PE's left which are connected by a broadcast bus. With this architecture, maximum, minimum, or sum of N values can be found in O(log N), median row of a binary picture in O(log N), shortest distance between two points in O(log N). This architecture is well suited for parallel processing of applications in Linear Algebra, Image Processing, Computational Geometry and Numerical Computations. These restricted connections to buses reduces the I/O ports significantly, and therefore, is more suitable for VLSI implementation.

This book focuses the solutions of linear algebra and matrix analysis problems, with the exclusive use of MATLAB. The topics include representations, fundamental analysis, transformations of matrices, matrix equation solutions as well as matrix functions. Attempts on matrix and linear algebra applications are also explored.

Scientific and engineering research is becoming increasingly dependent upon the development and implementation of efficient parallel algorithms on modern high-performance computers. Numerical linear algebra is an indispensable tool in such research and this paper attempts to collect and describe a selection of some of its more important parallel algorithms. The purpose is to review the current status and to provide an overall perspective of parallel algorithms for solving dense, banded, or block-structured problems arising in the major areas of direct solution of linear systems, least squares computations, eigenvalue and singular value computations, and rapid elliptic solvers. A major emphasis is given here to certain computational primitives whose efficient execution on parallel and vector computers is essential in order to obtain high performance algorithms.

The Society for Industrial and Applied Mathematics (SIAM) and the SIAM Activity Group in Linear Algebra in conjunction with the International Linear Algebra Society (ILAS) held the SIAM Conference on Linear Algebra and its applications on October 23-26, 2000 at the McKimmon Conference Center on the campus of North Carolina State University in Raleigh, North Carolina. The goals of this conference were to highlight the central role of linear algebra in many problems of mathematics and the applied sciences, including engineering problems in systems and control, signal processing and coding, economic and business problems, and problems from biology and geophysics. Particular consideration in this conference was given to applications in image processing, information retrieval and management (such as the performance of search engines on the Internet), aircraft manufacturing and design, industrial optimization problems, and assessing the economic cost of linear algebra in industry. With the development of high performance computers and new parallel architectures, computational linear algebra is in a state of rapid development. There are grand challenges requiring the development of efficient methods that solve truly large-scale problems by exploiting the ever-increasing computational power. One of the primary goals of this conference was to bring researchers and practitioners in these various areas together for exchange of information and ideas. In particular, the collaboration with ILAS was an important factor in bringing about fruitful interaction among researchers in theory, computation, and applications. There were 250 total attendees with 17% coming from industry and government. In addition, there were 27 students who attended.

For pt.I, see Kibernetika, vol.13, no.6, p.28 (1977). Considerable effort was devoted in the surveyed period to automatic decomposition of sequential algorithms, or rather of procedures or subprograms written in the algorithmic languages ALGOL or FORTRAN. The authors do not consider this body of research, they only note that, on the one hand, the available linear algebra subprograms included in Eispack provide convenient objects for testing various approaches to automatic construction of parallel programs and, on the other, an important state in this activity is the development of methods for fast and efficient solution of linear recurrences, which reduce to solving systems of linear equations with band-triangular matrix (in particular, of sufficiently small width). This article reflects the penetration of the parallelism ideas into the computational methods of linear algebra in recent years. 74 references.

Numerical linear algebra, digital signal processing, and parallel algorithms are three disciplines with a great deal of activity in the last few years. The interaction between them has been growing to a level that merits an Advanced Study Institute dedicated to the three areas together. This volume gives an account of the main results in this interdisciplinary field. The following topics emerged as major themes of the meeting: - Singular value and eigenvalue decompositions, including applications, - Toeplitz matrices, including special algorithms and architectures, - Recursive least squares in linear algebra, digital signal processing and control, - Updating and downdating techniques in linear algebra and signal processing, - Stability and sensitivity analysis of special recursive least squares problems, - Special architectures for linear algebra and signal processing. This book contains tutorials on these topics given by leading scientists in each of the three areas. A consider- able number of new research results are presented in contributed papers. The tutorials and papers will be of value to anyone interested in the three disciplines.

Our purpose is to describe a promising linear algebraic attack on the AAFG1 braid group cryptosystem proposed in [2] employing parameters suggested by the authors. Our method employs the well known Burau matrix representation of the braid group and techniques from computational linear algebra and provide evidence which shows that at least a certain class of keys are weak. We argue that if AAFG1 is to be viable the parameters must be fashioned to defend against this attack.KeywordsBraid GroupMapping Class GroupConjugacy ProblemArtin GroupConjugacy SearchThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we analyse the design of polylibraries, where the programs call for routines from different libraries according to the characteristics of the problem and of the system used to solve it. An architecture for this type of library is proposed. Our aim is to develop a methodology which can be used in the design of parallel libraries. To evaluate the viability of the proposed method, the typical linear algebra libraries hierarchy has been considered. Experiments have been performed in different systems and with linear algebra routines from different levels of the hierarchy. The results confirm the design of polylibraries as a good technique for speeding up computations.

Parametrization of Newton's iteration for computations with structured matrices and applications

We present an experimental study of parallel biconnected components algorithms employing several fundamental parallel primitives, e.g., prefix sum, list ranking, sorting, connectivity, spanning tree, and tree computations. Previous experimental studies of these primitives demonstrate reasonable parallel speedups. However, when these algorithms are used as subroutines to solve higher-level problems, there are two factors that hinder fast parallel implementations. One is parallel overhead, i.e., the large constant factors hidden in the asymptotic bounds; the other is the discrepancy among the data structures used in the primitives that brings non-negligible conversion cost. We present various optimization techniques and a new parallel algorithm that significantly improve the performance of finding biconnected components of a graph on symmetric multiprocessors (SMPs). Finding biconnected components has application in fault-tolerant network design, and is also used in graph planarity testing. Our parallel implementation achieves speedups up to 4 using 12 processors on a Sun E4500 for large, sparse graphs, and the source code is freely-available at our Web site.

This paper derives tradeoffs between three basic costs of a parallel algorithm: synchronization, data movement, and computational cost. These tradeoffs are lower bounds on the execution time of the algorithm which are independent of the number of processors, but dependent on the problem size. Therefore, they provide lower bounds on the parallel execution time of any algorithm computed by a system composed of any number of homogeneous components, each with associated computational, communication, and synchronization payloads. We employ a theoretical model counts the amount of work and data movement as a maximum of any execution path during the parallel computation. By considering this metric, rather than the total communication volume over the whole machine, we obtain new insights into the characteristics of parallel schedules for algorithms with non-trivial dependency structures. We also present reductions from BSP and LogP algorithms to our execution model, extending our lower bounds to these two models of parallel computation. We first develop our results for general dependency graphs and hypergraphs based on their expansion properties, then we apply the theorem to a number of specific algorithms in numerical linear algebra, namely triangular substitution, Gaussian elimination, and Krylov subspace methods. Our lower bound for LU factorization demonstrates the optimality of Tiskin's LU algorithm answering an open question posed in his paper, as well as of the 2.5D LU algorithm which has analogous costs. We treat the computations in a general manner by noting that the computations share a similar dependency hypergraph structure and analyzing the communication requirements of lattice hypergraph structures.

In this paper, we consider the synthesis of control laws for piecewise-affine hybrid systems on simplices. The construction is based on the solution to the control-to-facet problem at the continuous level, and on dynamic programming at the discrete level. The construction is given as an explicit algorithm using only linear algebra and reach-set computations for automata; no numerical integration is required. The method is conservative, in that it may fail to find a control law where one exists, but one cannot hope for a sharp algorithm for control synthesis since reachability for piecewise-affine hybrid systems is undecidable.

Unified Parallel C is a parallel language extension to standard C. Data in UPC are communicated through shared arrays, which are physically distributed. Thus, data regions have locality, or affinity, to particular threads of execution. This affinity concept engenders a non-uniformity in shared memory accesses by a particular thread. Affinity should be considered when building data structures in algorithms and applications, but UPC provides limited tools for data locality management. We propose the creation of an application support layer to support a wide variety of common data decompositions and programming idioms. We present here a first step for this layer with a selection of mapping functions and packages for numerical computation and dense matrix operations. These are driven by specific algorithms from linear algebra and numerical computation, and could be readily incorporated in such an application support layer.KeywordsMapping FunctionShared MemorySandia National LaboratoryIterative SolverGhost CellThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

If I think about the genesis of this paper, my semester at Stanford in 1995 comes to mind right away. For the mathematical orphan that I was, since my advisor, Peter Henrici died before I completed my PhD, the opportunity to spend a semester at Stanford was a truly unique opportunity. As Gene and I team-taught a seminar course on quadrature rules, linear algebra, and large-scale computations, I discovered more charming aspects of quadrature rules than I thought there could be. I must admit that when Gene first talked about the importance of Gauss–Kronrod rules in estimating errors in numerical integration, I thought that, for sure, I would never be caught working on such a dull topic. Well, soon enough I started seeing quadrature rules everywhere, and their elegance and secret connections with nearly everything began to get hold of me. Even Gauss–Kronrod quadrature rules, until then quickly dismissed even from my teaching, started to show their discrete charm. Well, never say never!

In this dissertation we develop numerical algorithms and software tools to facilitate parallel solutions of nonlinear programming (NLP) problems. In particular, we address large-scale, block-structured problems with an intrinsic decomposable configuration. These problems arise in a great number of engineering applications, including parameter estimation, optimal control, network optimization, and stochastic programming. The structure of these problems can be leveraged by optimization solvers to accelerate solutions and overcome memory limitations, and we propose variants to two classes of optimization algorithms: augmented Lagrangian (AL) schemes and Schur-complement interior-point methods. The convergence properties of augmented Lagrangian decomposition schemes like the alternating direction method of multipliers (ADMM) and progressive hedging (PH) are well established for convex optimization but convergence guarantees in non-convex settings are still poorly understood. In practice, however, ADMM and PH often perform satisfactorily in complex non-convex NLPs. In this work, we study connections between the method of multipliers (MM), ADMM, and PH to derive benchmarking metrics that explain why PH and ADMM work in practice. We illustrate the concepts using challenging dynamic optimization problems. Our exposition seeks to establish more formalism in benchmarking ADMM, PH, and AL schemes and to motivate algorithmic improvements. The effectiveness of nonlinear interior-point solvers for solving large-scale problems relies quite heavily on the solution of the underlying linear algebra systems. The schur-complement decomposition is very effective for parallelizing the solution of linear systems with modest coupling. However, for systems with large number of coupling variables the schur-complement method does not scale favorably. We implement an approach that uses a Krylov solver (GMRES) preconditioned with ADMM to solve block-structured linear systems that arise in the interior-point method. We show that this ADMM-GMRES approach overcomes the well-known scalability issues of Schur decomposition. One important drawback of using decomposition approaches like ADMM and PH is their convergence rate. Unlike Schur-complement interior-point algorithms that have super-linear convergence, augmented Lagrangian approaches typically exhibit linear and sublinear rates. We exploit connections between ADMM and the Schur-complement decomposition to derive an accelerated version of ADMM. Specifically, we study the effectiveness of performing a Newton-Raphson algorithm to compute multiplier estimates for augmented Lagrangian methods. We demonstrate using two-stage stochastic programming problems that our multiplier update achieves convergence in fewer iterations for MM on general nonlinear problems. In the case of ADMM, the newton update significantly reduces the number of subproblem solves for convex quadratic programs (QPs). Moreover, we show that using newton multiplier updates makes the method robust to the selection of the penalty parameter. Traditionally, state-of-the-art optimization solvers are implemented in low-level programming languages. In our experience, the development of decomposition algorithms in these frameworks is challenging. They present a steep learning curve and can slow the development and testing of new numerical algorithms. To mitigate these challenges, we developed PyNumero, a new open source framework implemented in Python and C++. The package seeks to facilitate development of optimization algorithms for large-scale optimization within a high-level programming environment while at the same time minimizing the computational burden of using Python. The efficiency of PyNumero is illustrated by implementing algorithms for problems arising in stochastic programming and optimal control. Timing results are presented for both serial and parallel implementations. Our computational studies demonstrate that with the appropriate balance between compiled code and Python, efficient implementations of optimization algorithms are achievable in these high-level languages.