Graph Transformation and Designing Parallel Sparse Matrix Algorithms beyond Data Dependence Analysis

A class of novel parallel algorithms for the solution of tridiagonal systems

Loops are the main source of parallelism in scientific programs. Hence, several techniques were developed to detect parallelism in these loops and transform them into parallel forms. In this dissertation, compile time transformations and efficient parallel execution of loops with various type of dependencies are investigated. First, Doacross loops with uniform dependencies are considered for execution on distributed memory parallel machines (multicomputers). Most known Doacross loop execution techniques can be applied efficiently only to shared memory parallel machines. In this thesis, code reordering technique, improvements to partitioning strategies, and finding a balance between communication and parallelism are presented to reduce the execution time of Doacross loops on multicomputers. As with most parallelizing transformation techniques, only single loopnests are considered in the first part of this dissertation. However, parallelizing each loopnest in a program separately, even if an optimal execution can be obtained for each loopnest, may not result in an efficient execution of all the code in the program because of communication overhead across the loops in a multicomputer environment. Hence, across loop data dependence representation and analysis are introduced in this work to improve the parallel execution of the whole code. Our contribution consists of finding and representing data dependencies whose sources and destinations are subspaces of the iteration space mainly common across the loops. This type of dependence information is used in this thesis to improve global iteration space partitioning, automatic generation of communication statements across loops, and index alignment. The final part of this dissertation presents new parallelizing techniques for loops with irregular and complex dependencies. Various data dependence analysis algorithms can be found in the literature even for loops with complex array indexing. However, the improvement in data dependence testing has not been followed by similar amelioration in restructuring transformations for loops with complex dependencies. Such loops are mostly executed in serial mode. Our parallelizing techniques for these loops consists of identifying regions of the iteration space where all iterations can be executed in parallel. The advantages of all the transformations presented in this dissertation are: (1) they significantly reduce the execution time of loops with various types of dependencies as shown in this work using the MasPar machine (2) they can be implemented at compile, time which makes the task of parallel programming easier.

Revisiting parallel cyclic reduction and parallel prefix-based algorithms for block tridiagonal systems of equations

The cyclic reduction algorithm is specialized to palindromic matrix polynomials and a complete analysis of applicability and convergence is provided. The resulting iteration is then related to other algorithms as the evaluation/interpolation at the roots of unity of a certain Laurent matrix polynomial, the trapezoidal rule for a certain integral and an algorithm based on the finite sections of a tridiagonal block Toeplitz matrix.

A modified data dependency graph (DDG) for modeling loop-optimizing restrictions is proposed. By using it, pipelining, loop-carried data dependencies and folding of individual multicycle operations over iteration boundaries can be modeled. Properties of semantically correct software pipelining transformations are first defined in the acyclic DDG of the unrolled loop. The case when an operation in a loop body is dependent on that same operation in some previous iteration is analyzed in the acyclic DDG of the unrolled loop. The algorithm for finding a real machine optimal schedule is split into two independent tasks: finding an optimal acyclic data dependency graph of one loop body and then finding an optimal schedule for a real machine by implementing basic block methods. A method for loop optimizing named 'pipelining of strongly connected components' is proposed. It is proved that the method is semantically correct. The loop body optimization is converted into sub-tasks of optimizing individual strongly connected components of the modified data dependency graph of a loop. >

Data dependence analysis is the foundation to many reordering related compiler optimizations and loop parallelization. Traditional data dependence analysis algorithms are developed primarily for Fortran-like subscripted array variables. They are not very effective for pointer-based references in C or C++. With more advanced hardware support for speculative execution, such as the advanced load instructions in Intel’s IA64 architecture, some data dependences with low probability can be speculatively ignored. However, such speculative optimizations must be carefully applied to avoid excessive cost associated with potential mis-speculations. Data dependence profiling is one way to provide probabilistic information on data dependences to guide such speculative optimizations and speculative thread generation. Software-based data dependence profiling requires detailed tracing of memory accesses, therefore, could be very time consuming. In this paper, we examine issues related to data dependence profiling, and propose various techniques to improve the efficiency of data dependence profiling. We use the Open Research Compiler (ORC) [15,16] to test the efficiency of our data profiling techniques. We also study the effectiveness of data dependence profiling on data speculative optimizations on Itanium systems. Our results show that efficient data dependence profiling could improve the performance for data speculative optimizations.

Legacy programs are normally monolithic (that is, all code runs in a single process and is not partitioned), and a bug in a program may result in the entire program being vulnerable and therefore untrusted. Program partitioning can be used to separate a program into multiple partitions, so as to isolate sensitive data or privileged operations. Manual program partitioning requires programmers to rewrite the entire source code, which is cumbersome, error-prone, and not generic. Automatic program partitioning tools can separate programs according to the dependency graph constructed based on data or programs. However, programmers still need to manually implement remote service interfaces for inter-partition communication. Therefore, in this paper, we propose AutoSlicer, whose purpose is to partition a program more automatically, so that the programmer is only required to annotate sensitive data. AutoSlicer constructs accurate data dependency graphs (DDGs) by enabling execution flow graphs, and the DDG-based partitioning algorithm can compute partition information based on sensitive annotations. In addition, the code refactoring toolchain can automatically transform the source code into sensitive and insensitive partitions that can be deployed on the remote procedure call framework. The experimental evaluation shows that AutoSlicer can effectively improve the accuracy (13%-27%) of program partitioning by enabling EFG, and separate real-world programs with a relatively smaller performance overhead (0.26%-9.42%).

Dynamic programming is an efficient technique to solve combinatorial search and optimization problem. There have been many parallel dynamic programming algorithms. The purpose of this paper is to study a family of dynamic programming algorithm where data dependence appear between non-consecutive stages, in other words, the data dependence is non-uniform. This kind of dynnamic programming is typically called nonserial polyadic dynamic programming. Owing to the non-uniform data dependence, it is harder to optimize this problem for parallelism and locality on parallel architectures. In this paper, we address the chanllenge of exploiting fine grain parallelism and locality of nonserial polyadic dynamic programming on a multi-core architecture. We present a programming and execution model for multi-core architectures with memory hierarchy. In the framework of the new model, the parallelism and locality benifit from a data dependence transformation. We propose a parallel pipelined algorithm for filling the dynamic programming matrix by decomposing the computation operators. The new parallel algorithm tolerates the memory access latency using multi-thread and is easily improved with tile technique. We formulate and analytically solve the optimization problem determing the tile size that minimizes the total execution time. The experiments on a simulator give a validation of the proposed model and show that the fine grain parallel algorithm achieves sub-linear speedup and that a potential high scalability on multi-core arichitecture.

Data dependence testing is the basic step in detecting loop level parallelism in numerical programs. The problem is undecidable in the general case. Therefore, work has been concentrated on a simplified problem, affine memory disambiguation. In this simpler domain, array references and loops bounds are assumed to be linear integer functions of loop variables. Dataflow information is ignored. For this domain, we have shown that in practice the problem can be solved accurately and efficiently.(1) This paper studies empirically the effectiveness of this domain restriction, how many real references are affine and flow insensitive. We use Larus's llpp system(2) to find all the data dependences dynamically. We compare these to the results given by our affine memory disambiguation system. This system is exact for all the cases we see in practice. We show that while the affine approximation is reasonable, memory disambiguation is not a sufficient approximation for data dependence analysis. We propose extensions to improve the analysis.

The data dependence of the delete-and-insert sort algorithm for median filtering is analyzed in this paper. It is shown that because of data dependence, the fastest throughput rate and the most efficient pipeline scheme cannot be used concurrently. A modified delete-and insert sort algorithm avoiding the above dilemma and its bit-level systolic array implementation are proposed in this paper. The throughput rate of the proposed architecture is equal to one-half (output/clocks) the maximum throughput allowed by the delete-and-insert sort algorithm, and the clock cycle time is equal to the propagation delay of a simple combinational circuit. Its speed is about 1.5 times faster than the existing bit-level systolic array designed by using the same delete-and-insert sort algorithm. The proposed architecture can be designed to operate at different word lengths and different window sizes. It is modular, regular, and of local interconnections and therefore amenable for VLSI implementation.

A new parallel direct algorithm for solving general linear systems of equations ist proposed in this paper. For sparse systems our algorithm requires less computations than the classical Jordan algorithm. Particularly we have also derived two related algorithms for linear recurrence problems of order 1 and tridiagonal systems. Each of the two algorithms has the same computational complexity as that of the corresponding recursive doubling algorithm or Even/Odd elimination algorithm, but requires half of the processors required by the corresponding algorithm.

The solution of tridiagonal system of equations using graphic processing units (GPU) is assessed. The parallel-Thomas-algorithm (PTA) is developed and the solution of PTA is compared to two known parallel algorithms, i.e. cyclic-reduction (CR) and parallel-cyclic-reduction (PCR). Lid-driven cavity problem is considered to assess these parallel approaches. This problem is also simulated using the classic Thomas algorithm that runs on a central processing unit (CPU). Runtimes and physical parameters of the mentioned GPU and CPU algorithms are compared. The results show that the speedup of CR, PCR and PTA against the CPU runtime is 4.4x,5.2x and 38.5x, respectively. Furthermore, the effect of coalesced and uncoalesced memory access to GPU global memory is examined for PTA, and a 2x-speedup is achieved for the coalesced memory access. Additionally, the PTA performance in a time dependent problem, the unsteady flow over a square, is assessed and a 9x-speedup is obtained against the CPU.

Tridiagonal system solver is an important kernel in many scientific and engineering applications. Even though quite a few parallel algorithms and implementations have been addressed in recent years, challenges still remain when solving large-scale tridiagonal system on heterogenous supercomputers. In this paper, a hierarchical algorithm framework SPIKE (pronounced 'SPIKE squared') is proposed to minimize the parallel overhead and to achieve the best utilization of CPU-GPU hybrid systems. In these systems, a layered and adaptive partitioning is presented based on the SPIKE algorithm to effectively control the sequential parts while efficiently exploiting the computation and communication overlapping in heterogeneous computing node. Moreover, the SPIKE algorithm is reformulated to reduce the matrix computations to only 1/3 in our hierarchical algorithm framework. Meanwhile, an improved implementation of the tiled-PCR-pThomas algorithm is employed for the GPU architecture, and the shared memory usage on the GPU can be reduced by 1/3 using careful dependence analysis on solving unit vector tridiagonal systems. Our experiments on Tianhe-1A show ideal weak scalability on up to 128 nodes when solving a tridiagonal system with a size of 1920M in the largest run and good strong scalability (70%) from 32 nodes to 256 nodes when solving a tridiagonal system with a size of 480M. Furthermore, the adaptive task partition across the CPU and GPU can get over 10% performance improvement in the strong scaling test with 256 nodes. In one computing node of Tianhe-1A, our GPU-only code can outperform the CUSPARSE version (non-pivoting tridiagonal solver) by 30%, and our hybrid code is about 6.7 times faster than the Intel SPIKE multi-process version for tridiagonal systems having a size of 3M, 5M, and 15M.

Graph models have been useful abstraction for examining the programmer's perspective of the static structure of imperative programs. A number of Graph models referring to control flow and data dependencies have appeared in the literature. The primary objective of this research is to develop a more general graph-theoretic model for describing programs written in arbitrary imperative languages and to compute a structural measure based on this abstraction. The proposed model is a synthesis of the control flow graph and the data dependency graph. This new unified abstraction, called the Generalized Program Graph (GPG), is structurally equivalent to the string from of a program. However, because of its graph structure, the new model is better suited for structural analysis of a program than original string form.

This paper presents several theoretical and fundamental results on the register need in periodic schedules, also known as MAXLIVE. Our first contribution is a novel formula for computing the exact number of registers needed by a scheduled loop. This formula has two advantages: Its computation can be done by using a polynomial algorithm with O(n lg n) complexity (n is the number of instructions in the loop) and it allows the generalization of a previous result [13]. Second, during software pipelining, we show that the minimal number of registers needed may increase when incrementing the initiation interval (II), which is contrary to intuition. For the case of zero architectural delays in accessing registers, we provide a sufficient condition for keeping the minimal number of registers from increasing when incrementing the II. Third, we prove an interesting property that enables us to optimally compute the minimal periodic register sufficiency of a loop for all its valid periodic schedules, irrespective of II. Fourth and last, we prove that the problem of optimal stage scheduling under register constraints is polynomially solvable for a subclass of data dependence graphs, whereas this problem is known to be NP-complete for arbitrary dependence graphs [7]. Our latter result generalizes a previous achievement [13] which addressed data dependence trees and forest of trees. In this study, we consider cyclic data dependence graphs without taking into account any resource constraints. The aim of our theoretical results on the periodic register need is to help current and future software pipeliners achieve significant performance improvements by making better (if not the best) use of the available resources.