PROBLEMS WITH DATA PARALLELISM

This paper introduces the ideas that underly the data-parallel language High Performance Fortran (HPF) and the new ideas in version 2 of HPF. It first reviews HPF's key language elements. It discusses the meaning of data parallelism and the limitations of HPF version 1 as a data-parallel programming language. The second part of the paper is a review of the development of version 2 of HPF. The extended language, under development in 1996, includes a richer data mapping capability; an extension to the independent loop that allows reduction operations in the loop range; a means for directing the mapping of computation as well as data; and a way to specify concurrent execution of several parallel tasks on disjoint subsets of processors.

The synthesis of consecutive array operations or array expressions into a composite access function of the source arrays at compile time can reduce the redundant data movement, temporary storage usage and loop synchronization overhead on flat shared-memory parallel machines with uniform memory accesses. However, it remains open how the synthesis scheme can be incorporated into optimizing HPF (High-Performance Fortran) programs on distributed-memory machines by taking into account communication costs. In this paper, we propose solutions to address this open problem. We first apply the array operation synthesis (developed for Fortran 90 programs) to HPF programs and demonstrate its performance benefits on distributed-memory machines. In addition, to prevent a situation we call the "synthesis performance anomaly", we derive a cost model and present an optimal solution based on this model to guide the array synthesis process on distributed-memory machines. We also show that the optimal problem is NP-hard. Therefore, we develop a practical heuristic algorithm for compilers to devise a synthesis strategy on distributed-memory machines with HPF programs. Experimental results show a significant performance improvement over the base codes for HPF code fragments from real applications on a DEC Alpha processor farm by incorporating our proposed optimizations.

This paper presents a compiler algorithm that automatically detects the appropriate loop indices of a given nested loop and applies loop interchange and tiling in order to overlap communication with computation. It also describes method of generating communication for the tiled loop on distributed memory machines. The algorithm presented here has been implemented in our High Performance Fortran (HPF) compiler, and experimental results have shown its effectiveness on the RISC System/6000 Scalable POWERparallel System.

Since the invention of the transistor, clock frequency increase was the primary method of improving computing performance. As the reach of Moore's law came to an end, however, technology driven performance gains became increasingly harder to achieve, and the research community was forced to come up with innovative system architectures. Today increasing parallelism is the primary method of improving performance: single processors are being replaced by multiprocessor systems and multicore architectures. The challenge faced by computer architects is to increase performance while limited by cost and power consumption. The appearance of cheap and fast interconnection networks has promoted designs based on distributed memory computing. Most modern massively parallel computers, as reflected by the Top 500 list, are clusters of workstations using commodity processors connected by high speed interconnects. Today's massively parallel systems consist of hundreds of thousands of processors. Software technology to program these large systems is still in its infancy. Optimizing communication has become a key to overall system performance. To cope with the increasing burden of communication, the following methods have been explored: (i) Scalability in the messaging system: The messaging system itself needs to scale up to the 100K processor range. (ii) Scalable algorithms reducing communication: As the machine grows in size the amount of communication also increases, and the resulting overhead negatively impacts performance. New programming models and algorithms allow programmers to better exploit locality and reduce communication. (iii) Speed up communication: reducing and hiding communication latency, and improving bandwidth. Following the three items described above, this thesis contributes to the improvement of the communication system (i) by proposing a scalable memory management of the communication system, that guarantees the correct reception of data and control-data, (ii) by proposing a language extension that allows programmers to better exploit data locality to reduce inter-node communication, and (iii) by presenting and evaluating a cache of remote addresses that aims to reduce control-data and exploit the RDMA native network capabilities, resulting in latency reduction and better overlap of communication and computation. Our contributions are analyzed in two different parallel programming models: Message Passing Interface (MPI) and Unified Parallel C (UPC). Many different programing models exist today, and the programmer usually needs to choose one or another depending on the problem and the machine architecture. MPI has been chosen because it is the de facto standard for parallel programming in distributed memory machines. UPC was considered because it constitutes a promising easy-to-use approach to parallelism. Since parallelism is everywhere, programmability is becoming important and languages such as UPC are gaining attention as a potential future of high performance computing. Concerning the communication system, the languages chosen are relevant because, while MPI offers two-sided communication, UPC relays on a one-sided communication model. This difference potentially influences the communication system requirements of the language. These requirements as well as our contributions are analyzed and discussed for both programming models and we state whether they apply to both programming models.

Particle-in-Cell (PIC) plasma simulation codes model the interaction of charged particles with surrounding electrostatic and magnetic fields. Its computational requirements made it to be classified as one of the grand-challenge problems facing the high performance community. In this paper we present the implementation of 1-D and 2-D electrostatic PIC codes in High Performance Fortran(HPF) on a IBM SP-2. HPF expands Fortran 90 with data distribution and alignment directives and data parallel statements. It is a powerful language for writing portable and high performance programs across many platforms. We used one of the most successful commerical HPF compilers currently available in the market and augmented the compiler's missing HPF functions with extrinsic routines when necessary. We obtained near linear speed-up in all of our test cases. The performance of the HPF programs is comparable to the native message passing implementations of the same codes on the SP-2.

High Performance Fortran (HPF) is a data-parallel programming language that allows the programmer to specify the data decomposition onto the processors while the compiler takes care of the tedious tasks of communication generation and computation partitioning. Shifting some of the complex tasks from the user to the compiler should encourage programmers to write and port code to parallel machines especially if the compiler implements these tasks efficiently. In this paper, performance results and analysis of a subset of the SPEC92 is presented for the XL HPF compiler on IBM SP2 machines. In addition to obtaining good performance from the compiler, one of the the main concerns of HPF users is portability. Experimental results and analysis are presented in this paper to investigate performance portability (consistency) first across multiprocessor architectures and then across compilers. For performance portability across multiprocessor machines, the same XL HPF compiler used for the IBM SP2 distributed memory machine experiment is also used to compile and execute the same applications but on IBM SMP machines. The comparable speedup and behaviour obtained for both machines indicates that HPF compilers can be portable across different architectures. For performance portability across compilers, various HPF programming techniques and recommendations are introduced to increase the chances of obtaining performance consistency with different HPF compilers.

Fortran D and High Performance Fortran are languages designed to support efficient data-parallel programming on a variety of parallel architectures. The goal of the D Editor is to provide a tool that allows scientists to use these languages efficiently. The D Editor combines analyses for shared memory machines and compiler optimizations for distributed memory machines. By cooperating with the underlying compiler, it can provide novel information on partitioning, parallelism, and communication based on compile time analysis at the level of the original Fortran program. The D Editor uses color coding and a collection of graphical displays to help the user to zoom in on portions of the program containing sequentialized code or expensive communication. The prototype implementation is useful for interactively displaying the results of compile time analysis; however, it has a number of shortcomings that must be addressed. Future enhancements will provide additional advice and transformation capabilities. We believe the D Editor is representative of a new generation of tools that will be needed to assist scientists to fully exploit languages such as High Performance Fortran. >

We introduce the use of the task migration paradigm for compiling and executing fine grain parallel applications on distributed memory computers. With this model, the execution of each iteration of a parallel loop is a task. When a task needs to access non local data, it migrates on the processing element which owns this data, and its execution resumes on this processor. The parallel data are never moved unlike the owner compute rule paradigm where the data is explicitly sent to the processor using it or in SVM systems where the data is implicitly requested. Our experiments show that good performance on irregular codes can be obtained on a distributed memory computer even when the application exhibits poor memory locality.

Efficient Computation of Address Sequences in Data Parallel Programs Using Closed Forms for Basis Vectors

Portable QCD codes for massively parallel processors

Distributed memory parallel machines are increasingly being used by the scientific community to provide computational power for solving problems. Building a distributed memory parallel machine offers a tremendous advantage in terms of cost and scalability over building a shared memory parallel machine. Unfortunately, distributed memory machines are much more difficult to program, mainly due to the lack of a single global address space. Communication between processors in a distributed memory machine is accomplished by message passing. The time cost for acquiring data from another processor's memory is significantly greater than the time cost of acquiring data from local memory. To assist the programmer in using a distributed memory machine, compilers have been developed which alleviate the need for the programmer to manage the lack of a global address space. These compilers will generate semantically correct code containing communication routines to produce a logically shared memory machine on a physically distributed memory machine. However, the runtime performance of the code emitted from some of these compilers can be further improved. This thesis is primarily concerned with advanced compilation techniques which will provide better runtime performance of programs written in imperative languages for regular scientific programs. The optimization of communication generation for distributed memory machines is used to produce this result. We look at two "state of the art" compilation systems and find that improvement in the communication code generated by these compilers is still possible. We have simplified existing descriptions of the various formalisms found in these compilers, while still providing enough formalism to study the problems in depth. We have extended some existing optimization techniques by enhancing the elimination of masks, by extending the notion of relaxing the owner computes rule, by extending the notion of replicating computation, and by extending the notion of replicating data. This thesis has discussed a novel technique to split loops so that they do not contain any guarded statements. Finally, we have suggested avenues for future research, as it is clear from this thesis that more work is required before compilation systems for distributed memory parallel machines will be widely used for general purpose computing.

High Performance Fortran for aerospace applications

The data alignment phase in compiling programs for distributed-memory machines

A technique for structuring compute-aggregate-broadcast algorithms on distributed memory computers is presented. The compute-aggregate-broadcast paradigm provides an abstraction of the problem for the programmer, allowing for separation of computation and synchronization. Such algorithms are well suited for application on distributed memory computers. The structuring technique assists the parallel programmer with synchronization, allowing the programmer to concentrate more on developing code for computation. Two examples are presented. >

A critical component of many data-parallel programming languages are operations that manipulate aggregate data structures as a whole—this includes Fortran 90, Nesl, and languages based on BMF. These operations are commonly implemented by a library whose routines operate on a distributed representation of the aggregate structure; the compiler merely generates the control code invoking the library routines and all machine-dependent code is encapsulated in the library. While this approach is convenient, we argue that by breaking the abstraction enforced by the library and by presenting some of internals in the form of a new intermediate language to the compiler back-end, we can optimize on al levels of the memory hierarchy and achieve more flexible data distribution. The new intermediate language allows us to present these optimisations elegantly as program transformations. We report on first results obtained by our approach in the implementation of nested data parallelism on distributed-memory machines.