The data alignment phase in compiling programs for distributed-memory machines

The issue of data movement between processors due to cross-references between multiple distributed arrays is addressed. The problem of index domain alignment is formulated as finding a set of suitable alignment functions that map the index domains of the arrays into a common index domain so as to minimize the cost of data movement due to cross-references between the arrays. The cost function and the machine model used are abstractions of the current generation of distributed-memory machines. The problem as formulated is shown to be NP-complete. A heuristic algorithm is devised and shown to be efficient and to provide excellent results. >

Distributed memory parallel machines are increasingly being used by the scientific community to provide computational power for solving problems. Building a distributed memory parallel machine offers a tremendous advantage in terms of cost and scalability over building a shared memory parallel machine. Unfortunately, distributed memory machines are much more difficult to program, mainly due to the lack of a single global address space. Communication between processors in a distributed memory machine is accomplished by message passing. The time cost for acquiring data from another processor's memory is significantly greater than the time cost of acquiring data from local memory. To assist the programmer in using a distributed memory machine, compilers have been developed which alleviate the need for the programmer to manage the lack of a global address space. These compilers will generate semantically correct code containing communication routines to produce a logically shared memory machine on a physically distributed memory machine. However, the runtime performance of the code emitted from some of these compilers can be further improved. This thesis is primarily concerned with advanced compilation techniques which will provide better runtime performance of programs written in imperative languages for regular scientific programs. The optimization of communication generation for distributed memory machines is used to produce this result. We look at two "state of the art" compilation systems and find that improvement in the communication code generated by these compilers is still possible. We have simplified existing descriptions of the various formalisms found in these compilers, while still providing enough formalism to study the problems in depth. We have extended some existing optimization techniques by enhancing the elimination of masks, by extending the notion of relaxing the owner computes rule, by extending the notion of replicating computation, and by extending the notion of replicating data. This thesis has discussed a novel technique to split loops so that they do not contain any guarded statements. Finally, we have suggested avenues for future research, as it is clear from this thesis that more work is required before compilation systems for distributed memory parallel machines will be widely used for general purpose computing.

The synthesis of consecutive array operations or array expressions into a composite access function of the source arrays at compile time can reduce the redundant data movement, temporary storage usage and loop synchronization overhead on flat shared-memory parallel machines with uniform memory accesses. However, it remains open how the synthesis scheme can be incorporated into optimizing HPF (High-Performance Fortran) programs on distributed-memory machines by taking into account communication costs. In this paper, we propose solutions to address this open problem. We first apply the array operation synthesis (developed for Fortran 90 programs) to HPF programs and demonstrate its performance benefits on distributed-memory machines. In addition, to prevent a situation we call the "synthesis performance anomaly", we derive a cost model and present an optimal solution based on this model to guide the array synthesis process on distributed-memory machines. We also show that the optimal problem is NP-hard. Therefore, we develop a practical heuristic algorithm for compilers to devise a synthesis strategy on distributed-memory machines with HPF programs. Experimental results show a significant performance improvement over the base codes for HPF code fragments from real applications on a DEC Alpha processor farm by incorporating our proposed optimizations.

Practical Mechanisms for Reducing Processor–Memory Data Movement in Modern Workloads

The parallel realization of adaptive finite element methods (FEM) has to deal with several irregular and dynamic algorithmic properties caused by adaptive mesh refinement (AMR). For an implementation on distributed memory machines irregular communication behavior results from dynamically growing data structures and statically unknown communication partners. An efficient parallel implementation has to provide appropriate mechanisms to cope with the flexibility of the adaptive finite element approach at runtime. We have implemented a data management and communication layer for an adaptive, 3-dimensional, hexahedral FEM on distributed memory machines and use it to parallelize an existing sequential code. The data management and communication layer realizes duplicated data structures for boundaries of distributed data, additional hierarchical data structures to deal with uneven refinement, and coherence protocols to guarantee correctness of communication partners and messages. An easy to use interface provides access to the functionality of the layer.

The gradual evolution of language features and approaches used for the programming of distributed memory machines underwent substantial advances in the 1990s. One of the most promising and widely praised approaches was based on data parallelism and resulted in High Performance Fortran. This paper reports on an experiment using that approach based on a commercial distributed memory machine, available compilers and simple test programs. The results are disappointing and not encouraging. The variety of components involved and the lack of detailed knowledge available for the compilers compound the difficulties of obtaining results and doing comparisons. The results show great variation and question the premise that communication is the decisive factor in performance determination. The results are also a contribution towards the difficult tasks of predicating performance on a distributed memory computer.

Recently, use of parallel machines in reservoir simulation has received considerable attention from the petroleum industry. This paper presents parallelization of a 3D compositional, equation-of-state reservoir simulator on the CM-5 and KSR-1. To the best of our knowledge, this is the first time that the parallelization of a compositional reservoir simulator has been performed on both the CM-5 and KSR-1. For new users of the CM-5 machines, the software and hardware of CM-5 architecture is presented, as well as details of the parallelization techniques. For example, domain decomposition, I/O's, phase equilibrium computations, and well model are described. The parallelism techniques on the KSR-1 are presented with the emphasis on the porting of the phase equilibrium calculation. The performance of each machine is evaluated by showing the speedup on different sets of processing nodes. Two test problems were used to explore the capability of the parallelized version of the code; one is a waterflood problem and the other is a CO2 multiple contact miscible flood, both in a West Texas oil field. These field problems were run on 1, 2, 4, 8, 16, and 32 processors to get insight into the locations of communication bottlenecks, generally occurring in the programming with distributed memory machines. The problems of latency and bandwidth which are associated with communication efficiency of the CM-5 are also addressed.

Task Scheduling is one of the key elements in any distributed-memory machine (DMM), and an efficient algorithm can help reduce the interprocessor communication time. As optimal scheduling of tasks to DMMs is a strong NP-hard problem, many heuristic algorithms have been introduced in the literature. This paper presents a Task Duplication based Scheduling (TDS) algorithm which can schedule directed acyclic graphs (DAGs) with a complexity of O(|V|/sup 2/), where |V| is the number of tasks in the DAG. This algorithm generates an optimal schedule for a class of DAGs which satisfy a simple cost relationship. The performance of the algorithm has been observed by its application to some practical DAGs, and by comparing it with other existing scheduling schemes in terms of the schedule length and algorithm complexity.

Extract-Transform-Load (ETL) processes play an important role in data warehousing. Typically, design work on ETL has focused on performance as the sole metric to make sure that the ETL process finishes within an allocated time window. However, other quality metrics are also important and need to be considered during ETL design. In this paper, we address ETL design for performance plus fault-tolerance and freshness. There are many reasons why an ETL process can fail and a good design needs to guarantee that it can be recovered within the ETL time window. How to make ETL robust to failures is not trivial. There are different strategies that can be used and they each have different costs and benefits. In addition, other metrics can affect the choice of a strategy; e.g., higher freshness reduces the time window for recovery. The design space is too large for informal, ad-hoc approaches. In this paper, we describe our QoX optimizer that considers multiple design strategies and finds an ETL design that satisfies multiple objectives. In particular, we define the optimizer search space, cost functions, and search algorithms. Also, we illustrate its use through several experiments and we show that it produces designs that are very near optimal.

A critical issue in achieving the performance of data parallel programs is how to efficiently decompose data across processors. On distributed-memory machines, a good data decomposition should increase processor workload balance and reduce interprocessor communication. Data decomposition consists of data distribution and data alignment. In this paper, we propose a trapezoid data distribution pattern and new data alignment algorithms using alignment graph (AG). Our AG-based alignment framework is unique from other related work because it takes advantage of the effect of optimal expression evaluation with regard to multiple assignment statements.

High-performance applications running on current and future architectures are mostly performance-limited by the cost of data movement, vertically through the memory hierarchy of a node or between CPU host and accelerator, and horizontally across nodes. In this talk, we discuss the role of packed data layout in optimizing data movement across a variety of different architectures (CPU and GPU) and memory systems. Conceptually, a high performance application is defined as operations on a unit of data and parallelism that is designed for efficient vertical and horizontal data movement, and can be tailored to match the requirements of an application and target architecture. Integrating data layout abstractions and optimizations into programming systems provides critical technology for achieving performance portability across the diversity of high-performance architectures arising in the exascale era.

The scheduling of multiple interacting tasks of a single parallel program is considered the most important issue to exploit the true performance of the multiprocessor system. The problem is to find a schedule that will minimize the execution time (Make_Span) of a program. On the other hand, task scheduling on a multiprocessor system with and without communication delays is known to be NP-complete problem. Consequently, many heuristic algorithms have been developed, each of which may find optimal scheduling under different circumstances. One of the well known iterative algorithms is the hill-climbing. This algorithm starts with a complete solution and searches to improve this solution by choosing a better neighbor based on a cost function. This will lead to a local optimum which is considered the main drawback of this algorithm. The research in this study concerns to develop an efficient iterative algorithm for scheduling problem based on the hill-climbing. Present algorithm satisfies a local optimum that is very close to the global one in a reasonable amount of time. In most experiments, it satisfies the actual global optimum.

The paper describes a parallel algorithm for the LU factorization of sparse matrices on distributed memory machines by using SPMD as programming model and PVM as message passing interface. We address all the difficulties arising in sparse codes, as the fill-in or the dynamic movement of data inside the matrix. The cyclic distribution has been used to evenly distribute the elements onto a mesh of processors, whereas two local storage schemes are proposed: A semi-ordered and two-dimensional linked list, which fulfils better the requirements of the algorithm, and a compressed storage by rows, which behaves better in the use of memory. The properties of the code are extensively analyzed and execution times on the CRAY T3D are presented to illustrate the overall efficiency achieved by our methods.KeywordsExecution TimeNonzero ElementSparse MatrixSparse CodeSparse MatriceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Community detection is an advanced graph operation that is used to reveal tightly-knit groups of vertices (aka. communities) in real-world networks. Given the intractability of the problem, efficient heuristics are used in practice. Yet, even the best of these state-of-the-art heuristics can become computationally demanding over large inputs and can generate workloads that exhibit inherent irregularity in data movement on manycore platforms. In this paper, we posit that effective acceleration of the graph community detection operation can be achieved by reducing the cost of data movement through a combined innovation at both software and hardware levels. More specifically, we first propose an efficient software-level parallelization of community detection that uses approximate updates to cleverly exploit a diminishing returns property of the algorithm. Secondly, as a way to augment this innovation at the software layer, we design an efficient Wireless Network on Chip (WiNoC) architecture that is suited to handle the irregular on-chip data movements exhibited by the community detection algorithm under both unicast- and broadcast-heavy cache coherence protocols. Experimental results show that our resulting WiNoC-enabled manycore platform achieves on average 52% savings in execution time, without compromising on the quality of the outputs, when compared to a traditional manycore platform designed with a wireline mesh NoC and running community detection without employing approximate updates.

Algorithms exist for compiling Fortran D for MIMD (multiple-instruction multiple-data) distributed-memory machines, but they are significantly restricted in the presence of procedure calls. The authors present interprocedural analysis, optimization, and code generation algorithms for Fortran D that limit compilation to only one pass over each procedure. This is accomplished by collecting summary information after edits, and then compiling procedures in reverse topological order to propagate necessary information. Delaying instantiation of the computation partition, communication, and dynamic data decomposition is key to enabling interprocedural optimization. Recompilation analysis preserves the benefits of separate compilation. Empirical results show that interprocedural optimization is crucial in achieving acceptable performance for a common application. >