Data-parallel flattening by expansion

Optimizing communication is a key issue in generating efficient SPMD codes in compiling distributed arrays on data parallel languages, such as High Performance Fortran. In HPF, the array distribution may involve alignment and cyclic(k)-distribution such that the enumeration of the local set and the enumeration of the communication set exhibit regular patterns which can be modeled as integer lattices. In the special case of unit-strided alignment, many techniques of the communication set enumeration have been proposed, while in the general case of the non-unit-strided alignment, inspector-like run-time codes are needed to build repeating pattern table or to scan over local elements such that the communication set can be constructed. Unlike other works on this problem of the general alignment and cyclic(k) distribution, our approach derives an algebraic solution for such an integer lattice that models the communication set by using the Smith-Normal-Form analysis, therefore, efficient enumeration of the communication set can be generated. Based on the integer lattice, we also present our algorithm for the SPMD code generation. In our approach, when the parameters are known, the SPMD program can be efficiently constructed without any inspector-like run-time codes.

This paper introduces a multi-threaded, portable, parallel programming runtime system, called Athapascan-Ob. It is designed to support the efficient processing of large irregular problems. Parallel machines are considered as generic networks of heterogeneous symmetric multiprocessors, with remote thread creation, message exchange and remote memory accesses. Athapascan-Ob can dynamically map fine granularity computations on such machines and overlap communication delays with computation. These features should help to perform better load sharing and achieve better performance.

There are well defined methods for supporting regular problems with scalable performance, typified by the HPF language and the BSP model. Less well understood is the solution of more irregular problems, supporting complex shared data structures and task dependencies, and typically requiring dynamic load balancing to sustain high performance. It is demonstrated how the use of Shared Abstract Data Types (SADTs), together with an extended BSP cost model, can support irregular problems in a structured manner An SADT is an extension of a serial ADT which allows the concurrent invocation of its methods. A number of SADTs are used to implement a solution of the travelling salesman problem on the Cray T3D machine, and a description of the restructuring of a parallel CFD code using SADTs is provided, with initial results given for the Cray T3E.

High Performance Fortran is a dataparallel language that allows the user to specify the parallelism in his program. It is not always easy to extract the parallelism in a given program. To help the user, an automatic loop parallelizer has been developed: Bouclettes.Bouclettes has been written to validate some scheduling and mapping techniques that are mentioned in this paper. A Fortran 77 loop nest is rewritten into a HPF loop nest with explicit parallel loops (INDEPENDENT loops) and explicit data distribution (DISTRIBUTE and ALIGN directives).The different stages of parallelization (analysis, dependence analysis, scheduling, mapping and code generation) and their implementation issues are described here.KeywordsData DependenceLoop NestIteration DomainIndependent LoopAlign DirectiveThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Already our technology is able to accommodate hundreds of cores on a single chip. The embedded market is already embracing this but the wider market is back-peddling. There is no doubt however that these markets will converge and that the processor will become the new building block of our chips and systems. The big problem problem however, will be in turning this technology into large complex systems. There are no lack of contenders for concurrent programming languages but the problem with most paradigms is that they mix algorithmic and concurrency engineering and do little to promote safe and composable parallel programs. The question that must be asked is whether there are programming models or combinations of models that are ubiquitous enough to form the basis for a co-ordinated solution to the problems of programming MPSOC. This special session has succeeded in bringing bringing together a number of key researchers working in the area of programming models for the next generation of complex systems based on multi- and many-core chips. Collectively they are developing models, languages and compilers to enable a paradigm shift in this important area. The scope of the contributions will cover deterministic approaches, such as functional languages and data-parallel languages as well as extracting concurrency from sequential code. In addition, a number of new co-ordination languages will be presented that range from static to dynamic mapping of components to resources. These are challenging problems but we are pleased to present some significant progress from invited and contributed papers in this session.

Most data-parallel languages use arrays to support parallelism. This regular data structure allows a natural development of regular parallel algorithms. The implementation of irregular algorithms requires a programming effort to project the irregular data structures onto regular structures. We first propose in this paper a classification of existing data-parallel languages. We briefly describe their irregular and dynamic aspects, and derive different levels where irregularity and dynamicity may be introduced. We propose then a new irregular and dynamic data-parallel programming model, called Idole. Finally we discuss its integration in the C++ language, and present an overview of the Idole extension of C++.

This paper describes a compilation technique used to accelerate dataflow computations, common in deep neural network computing, onto Coarse Grained Reconfigurable Array (CGRA) architectures. This technique has been demonstrated to automatically compile dataflow programs onto a commercial massively parallel CGRA-based dataflow processor (DPU) containing 16000 processing elements. The DPU architecture overcomes the von Neumann bottleneck by spatially flowing and reusing data from local memories, and provides higher computation efficiency compared to temporal parallel architectures such as GPUs and multi-core CPUs. However, existing software development tools for CGRAs are limited to compiling domain specific programs to processing elements with uniform structures, and are not effective on complex micro architectures where latencies of memory access vary in a nontrivial fashion depending on data locality. A primary contribution of this paper is to provide a general algorithm that can compile general dataflow graphs, and can efficiently utilize processing elements with rich micro-architectural features such as complex instructions, multi-precision data paths, local memories, register files, switches etc. Another contribution is a uniquely innovative application of Boolean Satisfiability to formally solve this complex, and irregular optimization problem and produce high-quality results comparable to hand-written assembly code produced by human experts. A third contribution is an adaptive windowing algorithm that harnesses the complexity of the SAT-based approach and delivers a scalable and robust solution.

In this paper, we present the rapid development of efficient multi-core parallel code for the blocked Floyd-Warshall algorithm, using the C++ library gae.h [4], on top of the Intel C++ Threading Building Blocks (TBB) library. We demonstrate that developing parallel multi-core code using gae.h is easy and fast. The parallel code developed with gae.h is also efficient. The efficiency of the parallel code developed for the blocked Floyd-Warshall algorithm on an 80-core HP ProLiant DL980 G7 multi-core machine is above 70% if less than 70 cores are used and above 90% if less than 20 cores are used.

Integrating user knowledge with information from parallelisation tools to facilitate the automatic generation of efficient parallel FORTRAN code

Recently, the first commercial High Performance Fortran (HPF) subset compilers have appeared. This article reports on our experiences with the xHPF compiler of Applied Parallel Research, version 1.2, for the Intel Paragon. At this stage, we do not expect very High Performance from our HPF programs, even though performance will eventually be of paramount importance for the acceptance of HPF. Instead, our primary objective is to study how to convert large Fortran 77 (F77) programs to HPF such that the compiler generates reasonably efficient parallel code. We report on a case study that identifies several problems when parallelizing code with HPF; most of these problems affect current HPF compiler technology in general, although some are specific for the xHPF compiler. We discuss our solutions from the perspective of the scientific programmer, and presenttiming results on the Intel Paragon. The case study comprises three programs of different complexity with respect to parallelization. We use the dense matrix‐matrix product to show that the distribution of arrays and the order of nested loops significantly influence the performance of the parallel program. We use Gaussian elimination with partial pivoting to study the parallelization strategy of the compiler. There are various ways to structure this algorithm for a particular data distribution. This example shows how much effort may be demanded from the programmer to support the compiler in generating an efficient parallel implementation. Finally, we use a small application to show that the more complicated structure of a larger program may introduce problems for the parallelization, even though all subroutines of the application are easy to parallelize by themselves. The application consists of a finite volume discretization on a structured grid and a nested iterative solver. Our case study shows that it is possible to obtain reasonably efficient parallel programs with xHPF, although the compiler needs substantial support from the programmer.

This paper describes an approach for automatically generating optimized parallel code from serial Fortran programs annotated with high level directives. A preprocessor analyzes both the program and the directives and generates efficient parallel Fortran code that runs on a number of parallel architectures, such as clusters or SMPs. The unique aspect of this approach is that the directives and optimizations can be customized and extended by the expert programmers who would be using them in their applications. This approach enables the creation of parallel extensions to Fortran that are specific to individual applications or science domains.

In this paper, we present a mathematical model for evaluating codes and primitives in optical character recognition. The model is based on code efficiency, calculated from its average length and its transmitted information. This efficiency is obtained from entropies and conditional entropies which are estimated from probabilities of character recognition depending on the issued codes. This method is used to evaluate two sets of primitives in numeral recognition. Then, we propose a method of constructing binary decision trees for the recognition of handwritten numerals. This method is based on the mathematical model previously stated, which is used to process the transmitted information about the primitives. We demonstrate the performance of our system with experiments using real data.

We describe the algorithms we have developed for linear‐scaling plane wave density functional calculations on parallel computers as implemented in the onetep program. We outline how onetep achieves plane wave accuracy with a computational cost which increases only linearly with the number of atoms by optimising directly the single‐particle density matrix expressed in a psinc basis set. We describe in detail the novel algorithms we have developed for computing with the psinc basis set the quantities needed in the evaluation and optimisation of the total energy within our approach. For our parallel computations we use the general Message Passing Interface (MPI) library of subroutines to exchange data between processors. Accordingly, we have developed efficient schemes for distributing data and computational load to processors in a balanced manner. We describe these schemes in detail and in relation to our algorithms for computations with a psinc basis. Results of tests on different materials show that onetep is an efficient parallel code that should be able to take advantage of a wide range of parallel computer architectures. (© 2006 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)

A systolic array parallelizing compiler

The approach taken in the Triton project is to let a high-level machine-independent parallel programming language drive the design of parallel hardware. This approach permits machine-independent parallel programs to be compiled into efficient machine code. The main results are as follows: (1) The parallel programming language Modula-2* extends Modula-2 with constructs for expressing a wide range of parallel algorithms in a high-level, portable, and readable way. (2) Techniques are used for efficiently translating Modula-2* programs to several modern parallel architectures and deriving recommendations for future parallel machine architectures. (3) Triton/1 is a scalable, mixed-mode SIMD/MIMD parallel computer with a highly efficient communications network. It overcomes several deficiencies of current parallel hardware and adequately supports high-level parallel languages. >