On parallel programming environments and multilevel optimization

Genericity of parallel programming environments, enabling development of portable parallel programs, is expected to result in performance penalties. Furthermore, programmability and tool support of programming environments are important issues if a choice between programming environments has to be made. In this paper we propose a methodology to compare native and generic parallel programming environments, taking into account such competing issues as portability and performance. As a case study, this paper compares the Iserver-Occam, Parix, Express and PVM parallel programming environments on a 512-node Parasytec GCel. Furthermore, we apply our methodology to compare Parix and PVM on a new architecture, a 32-node Parsytec PowerXplorer, which is based on the PowerPC chip. In our approach we start with a representative application and isolate the basic (environment)-dependent building blocks. These basic building blocks, which depend on floating-point performance and communication capabilities of the environments, are analysed independently. We have measured point-to-point communication times, global communication times and floating-point performance. All information is combined into a time complexity analysis, allowing comparison of the environments on different degrees of functionality. Together with demands for portability of the code and development time (i.e. programmability), an overall judgement of the environments is given.

In this paper we present a new methodology for the performance prediction of parallel programs on parallel platforms ranging from shared-memory to distributed-memory (vector) machines. The methodology comprises a procedural program and machine specification paradigm based on PAMELA (PerformAnce ModEling LAnguage), along with a performance calculus, called “serialization analysis”. This calculus extends conventional parallel program analysis technology by explicitly accounting for resource contention, yet at the low evaluation cost typical for static techniques. It is shown that, where conventional techniques introduce fundamental errors, predictions from serialization analysis remain realistic. Apart from the merits of the methodology itself, this high reliability/cost ratio makes PAMELA an attractive candidate for compile-time application within the performance prediction hierarchy often found in parallel programming environments.

While multicore architectures are used in the whole product range from server systems to handheld computers, the deployed software still undergoes the slow transition from sequential to parallel. This transition, however, is gaining more and more momentum due to the increased availability of more sophisticated parallel programming environments. Combined with the ever increasing complexity of multicore architectures, this results in a scheduling problem that is different from what it has been, because concurrently executing parallel programs and features such as non-uniform memory access, shared caches, or simultaneous multithreading have to be considered. In this paper, we compare different ways of scheduling multiple parallel applications on multicore architectures. Due to emerging parallel programming environments, we primarily consider applications where the parallelism degree can be changed on the fly. We propose TACO, a topology-aware scheduling scheme that combines equipartitioning and coscheduling, which does not suffer from the drawbacks of the individual concepts. Additionally, TACO is conceptually compatible with contention-aware scheduling strategies. We find that topology-awareness increases performance for all evaluated workloads. The combination with coscheduling is more sensitive towards the executed workloads and NUMA effects. However, the gained versatility allows new use cases to be explored, which were not possible before.

JCSPre is a highly reduced version of the JCSP (Communicating Sequential Processes for Java) parallel programming environment. JCSPre has been implemented on a LEGO Mindstorms NXT brick using the LeJOS Java runtime environment. The LeJOS environment provides an abstraction for the NXT Robot in terms of Sensors, Sensor Ports and Motors, amongst others. In the implementation described these abstractions have been converted into the equivalent active component that is much easier to incorporate into a parallel robot controller. Their use in a simple line following robot is described, thereby demonstrating the ease with which robot controllers can be built using parallel programming principles. As a further demonstration we show how the line following robot controls a slave robot by means of Bluetooth communications.

For better utilization of computing resources, it is important to consider parallel programming environments in which the number of available processors varies at runtime. In this paper, we discuss runtime support for data parallel programming in such an adaptive environment. Executing data parallel-programs in an adaptive environment requires redistributing data when the number of processors changes, and also requires determining new loop bounds and communication patterns for the new set of processors. We have developed a runtime library to provide this support. We also present performance results for a multiblock Navier-Stokes solver run on a network of workstations using PVM for message passing. Our experiments show that if the number of processors is not varied frequently, the cost of data redistribution is not significant compared to the time required for the actual computations. >

The application of parallel computers to reservoir simulation is studied with the aid of two recently introduced Multiple Instruction, Multiple Data (MIMD) computers. The problems posed by reservoir simulators are shown to be particularly well suited for such parallel computers* Fundamental parallel programming techniques using Fortran are presented for the Hetrogeneous Element Processor (HEP) produced by Denelcor and the iPSC Hypercube produced by Intel. These techniques are then applied to two of the most time consuming tasks in reservoir simulation: forming the matrix coefficients and producing a sparse matrix solution. The problem of parallel formation of sparse matrix coefficients is addressed for the single and multiphase cases using both black oil and compositional fluid models. Parallel sparse matrix solution is addressed by considering D4 ordered Gaussian elimination, and the multigrid, conjugate gradient, and Successive Over-Relaxation (SOR) methods. Three parallel algorithms based on SOR are developed to illustrate the possible diversity of parallel algorithms. The red-black and multicolored SOR methods which are often used on vector machines are shown to be easily adaptable to parallel MIMD machine. Matrix partitioning and a new method of iteration pipelining are also presented as parallel SOR methods. The various SOR algorithms, in parallel form, are mapped onto the HEP to illustrate the speedups currently possible on an MIMD machine. Projections are made on the future importance of these types of machines to reservoir simulation.

Over the years many interesting and efficient parallel algorithms have been developed to solve a wide variety of problems, but not much attention has been paid to studying the inherent parallelism in sequential algorithms---i.e., understanding the depth of their dependence structure, and how shallow dependence structures might beused to develop efficient parallel implementations. In this talk I will describe recent work on analyzing the dependence depth of iterative sequential algorithms---ones that loop over a collection of elements. Many of these algorithms have deep dependence chains in the worst case, but shallow chains (polylog w.h.p.) if the elements are randomly ordered. Examples include many fundamental algorithms: the Knuth shuffle for random permutations, sorting by insertion into a binary search tree, greedy maximal independent set (MIS), greedy maximal matching, greedy graph-coloring, counting cycles in a permutation, incremental k-dimensional linear programming, and incremental 2d Delaunay triangulation. An advantage of the approach is that it can lead to very simple and efficient parallel algorithms. Our MIS algorithm, for example can be coded in a dozen or so lines, and is significantly faster than Luby's algorithm on modern multicore machines. Also the approach encourages snapping the view that sequential and parallel algorithms are distinct, and instead thinking of algorithms, in general, as collections of instructions with dependences among them.

Presents a portable parallel programming environment for Modula-2*, an explicitly parallel machine-independent extension of Modula-2. Modula-2* offers synchronous and asynchronous parallelism, a global single address space, and automatic data and process distribution. The Modula-2* system consists of a compiler, a debugger, a cross-architecture make, graphical X Windows control panel, run-time systems for different machines, and sets of scalable parallel libraries. The existing implementation targets the MasPar MP series of massively parallel processors (SIMD), the KSR-1 parallel computer (MIMD), heterogeneous LANs of workstations (MIMD), and single workstations (SISD). We describe the important components of the Modula-2* environment, and discuss selected implementation issues. We focus on how we achieve a high degree of portability for our system, while at the same time ensuring efficiency.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Parsec is a parallel programming environment whose goal is to simplify the development of multicomputer programs without, as is often the case, sacrificing performance. We have reconciled these objectives by “compiling” the structure of parallel applications into information to configure each of a small set of communication primitives on a context sensitive basis. In this chapter we show how Parsec can be used to implement a high-performance processor farm and compare Parsec and hand-optimized implementations to demonstrate that Parsec can achieve a similar level of performance. Extensive static analysis and optimization is necessary to achieve these results. We discuss both the tools which perform these tasks as well as the user interface that provides the necessary declarative structural information. Using the processor farm, we show how Parsec simplifies the task of specifying the structure of a parallel application and improves the result by supporting abstraction, reuse and scalability.

Binary steering in discrete tomography reconstruction with sequential and simultaneous iterative algorithms

In this paper, we present a parallel iterative solution for large banded systems of linear equations based on incomplete LU-factorization (ILU). A master—workers parallel computing scheme is used. The proposed algorithm incurs reduced storage and communication overhead as compared to previous methods. The reduction in communication overhead has been achieved by pipelining the inter-workers message passing and limiting the communication between master and workers to initial work distribution and final results collection.The performance of the proposed algorithm is evaluated analytically and experimentally using Parallel Virtual Machine (PVM) as a parallel programming environment on a cluster of Linux work-stations. The obtained performance results show that the proposed algorithm exhibits higher efficiency for coarse grain computations corresponding to large matrix sizes and reduced number of processors.

Due to the development of several new technologies concerning high speed networks, one can get PC clusters for intensive computing at a low price. Parallel programming environments must take into account the heterogeneity of workstation networks into their communication kernels. New technologies must be easily integrated into such environments. The LANDA-HSN project has been designed for an efficient use of the available resources (network, processor, memory). The paper describes the mechanisms and the implementation of its communication kernel on SMP hosts and through interconnection networks. The contribution of the paper is to propose a flexible and efficient communication kernel.

Programmers of sequential and parallel computers share the objective of developing fast, reliable programs as efficiently as possible. Already, considerable effort has been made to develop tools and techniques for sequential computers. Several approaches have been devised to meet the parallel programming challenge. Sequential languages have been adapted for parallel computers; new parallel languages designed; and tools developed which aid the programmer in detecting parallel code. A parallel programming assistant can aid the user in several ways: it can carry out tasks that are simple but mundane to the user, it can make comprehension easier for the user by summarizing program information; and it can cooperate with the user in doing a complex task, thus combining its own knowledge with that of the user's.

With multi-core processors in virtually every segment of the computing market, mainstream programmers need to routinely write parallel software. While progress has been made with automatic parallelization (see Chapter 9), experience with parallel computing has shown that in most cases, programmers will need to create parallel software “by hand.” This transition from serial to parallel software is a huge challenge for the software industry [Hwu08]; perhaps the biggest it has ever faced. While parallel computing is new to mainstream programmers, it has a long his-tory in scientific computing. In the 1980s, the scientific computing community was faced with a transition from vector supercomputers supported by vectorizing compilers to explicitly parallel massively parallel processors and clusters. Computer scientists responded by creating countless parallel programming languages, but these for the most part failed. Only a tiny fraction of programmers in the sciences took upparallel programming; and those that did tended to use low-level message passing libraries [MPI] that emphasized portability and control over elegance. New high-level languageswere not enough to solve the parallel programming prob-lem in the 1980s and 1990s, and there is no reason to believe it will be different this time as we help general purpose programmers adopt parallel programming. It is my hypothesis that the key to solving this problem is to focus on how experi-enced programmers think about parallel programming. Experienced parallel programmers, once they understand how the concurrency in their problem supports parallel execution, can use any sufficiently complete parallel programming language to create parallel software. Therefore, to solve the parallel programming problem for mainstream programmers, we need to capture how expert parallel programmers think and provide that understanding to mainstream programmers. In other words, we do not need new languages or sophisticated tools (though they help); we need to help mainstream programmers learn to “think parallel.” Learning to reason about parallel algorithms is a daunting task. Fortunately, wehave a few decades of experience to draw on. All we need to do is distill this collective wisdom into a form that we can put down in writing and communicate to people new to parallel computing. We believe design patterns are an effective tool to help us accomplish this task.

Mainly designed for simulating the fluid motion, Smooth Particle Hydrodynamics (SPH) SPH has been widely applied to many fields, such as aerospace aircraft design and ship design. First, this paper presents an implementation of serial SPH method algorithm using the reflection method as the boundary condition. This paper describes entire process of SPH and gives detailed descriptions of some important procedures. Second, this paper realizes the corresponding parallel algorithm of SPH, which includes neighbor particle searching and building, physical attributes computation and other main SPH method procedures. Third, details of mapping different parts of the SPH serial algorithm to parallel algorithm on GPU with CUDA C as parallel programming language are also shown in this paper. Taking advantage of powerful computation ability of GPU, the experiment approximately achieves a 8X speedup that shows the high efficiency of GPU-based programming. The parallel algorithm not only achieves a high performance, but also promises a reliability result with a small error rate as the experiment result shows.