From transformations to methodology in parallel program development: A case study

Homomorphisms are functions that match the divide-and-conquer paradigm and thus can be computed in parallel. Two problems are studied for homomorphisms on lists: (1) parallelism extraction: finding a homomorphic representation of a given function; (2) parallelism implementation: deriving an efficient parallel program that computes the function. A systematic approach to parallelism extraction proceeds by generalization of two sequential representations based on traditional cons lists and dual snoc lists. For some non-homomorphic functions, e.g., the maximum segment sum problem, our method provides an embedding into a homomorphism. The implementation is addressed by introducing a subclass of distributable homomorphisms and deriving for them a parallel program schema, which is time optimal on the hypercube architecture. The derivation is based on equational reasoning in the Bird-Meertens formalism, which guarantees the correctness of the parallel target program. The approach is illustrated with function scan (parallel prefix), for which the combination of our two systematic methods yields the “folklore” hypercube algorithm, usually presented ad hoc in the literature.

Multicore environments are rapidly emerging and are widely used in SoC, but accompanying parallelism programming and debugging impact the ordinary sequential world. Unfortunately, according to Heisenberg's uncertainty principle, the instrument trying to probe the target will cause probe effects. Therefore, current intrusive debugging methodologies for sequential programs cannot be used directly in parallel programs in a multicore environment. This work developed a non-intrusive run-time assertion (RunAssert) for parallel program development based on a novel non-uniform debugging architecture. Our approaches are as follows: (a) a current language extension for parallel program debugging (b) corresponding non-intrusive hardware configuration logic and checking methodologies and (c) several reality cases using the extensions mentioned above. In general, the target program can be executed at its original speed without altering the parallel sequences, thereby eliminating the possibility of probe effect. The net hardware cost is relatively low, the reconfigurable logic for RunAssert is 0.6%-2.5% in a NUDA cluster with 8 cores, such that RunAssert can readily scale up for increasingly complex multicore systems.

A large variety of multiple instruction, multiple data (MIMD) parallel machines-some with global shared memory and some without, some with few processors to some with a few thousand processors-are becoming commercially available, and more are being proposed, designed, or developed. Programming such machines for fast execution of desired applications remains a challenging problem. An examination is made of some broad approaches towards this goal, identifying relevant issues and presenting a few guidelines that should be helpful in parallel programming research and development. The Chare Kernel parallel programming system is proposed as a solution to the parallel programming problem.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Computations on two-dimensional arrays such as matrices and images are one of the most fundamental and ubiquitous things in computational science and its vast application areas, but development of efficient parallel programs on two-dimensional arrays is known to be hard. In this paper, we propose a compositional framework that supports users, even with little knowledge about parallel machines, to develop both correct and efficient parallel programs on dense two-dimensional arrays systematically. The key feature of our framework is a novel use of the abide-tree representation of two-dimensional arrays. The presentation not only inherits the advantages of tree representations of matrices where recursive blocked algorithms can be defined to achieve better performance, but also supports transformational development of parallel programs and architecture-independent implementation owing to its solid theoretical foundation - the theory of constructive algorithmics.

Chapter 7 - Parallel program development

A set of metalinguistic features and their applications in formal parallel-program transformation are described. Program transformation is an effective methodology for developing correct and efficient parallel programs. However, manually carrying out the transformation is usually very cumbersome and error-prone. These metalinguistic features provide convenient language constructs for expressing and automating transformation steps. These features include a rationalized version of quoting and unquoting, a set a constructors, selectors, and predicates, and a collection of semantics preserving operators. The authors describe the metalinguistic features, illustrate how Crystal programs are conveniently manipulated with these meta constructs, and present a formal denotational semantics of the rationalized quoting and unquoting. They demonstrate the applications of these metalinguistic features by formally deriving a class of parallel palindrome recognition programs.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The behavioral correctness of parallel programs has a pivotal role in computational sciences and engineering applications as researchers draw scientific conclusions from the results generated by parallel applications. Moreover, with the advent of multicore processors, the development of parallel programs should be facilitated for the mainstream developers. While numerous programming models and APIs exist for parallel programming, we pose the view that more emphasis should be placed on designing the synchronization mechanisms of parallel programs independent from the design of their functional behaviors. More importantly, programs' behaviors evolve (due to new requirements and change of configuration), thereby creating a need for techniques and tools that enable developers to reason about the behavioral evolution of parallel programs. With such motivations, we introduce a framework for automated design/evolution of the synchronization mechanisms of parallel programs. © 2009 IEEE.

Genericity of parallel programming environments, enabling development of portable parallel programs, is expected to result in performance penalties. Furthermore, programmability and tool support of programming environments are important issues if a choice between programming environments has to be made. In this paper we propose a methodology to compare native and generic parallel programming environments, taking into account such competing issues as portability and performance. As a case study, this paper compares the Iserver-Occam, Parix, Express and PVM parallel programming environments on a 512-node Parasytec GCel. Furthermore, we apply our methodology to compare Parix and PVM on a new architecture, a 32-node Parsytec PowerXplorer, which is based on the PowerPC chip. In our approach we start with a representative application and isolate the basic (environment)-dependent building blocks. These basic building blocks, which depend on floating-point performance and communication capabilities of the environments, are analysed independently. We have measured point-to-point communication times, global communication times and floating-point performance. All information is combined into a time complexity analysis, allowing comparison of the environments on different degrees of functionality. Together with demands for portability of the code and development time (i.e. programmability), an overall judgement of the environments is given.

The teaching environment described here is introduced as an effective way for learning how to deal with the problems related with parallel/distributed real-time systems. The educational process has three main domains: the cognitive; the affective; and the psychomotor. Normally, the student dominates the first, but lacks more interest for the subject and a familiarisation with the challenges of real problems in the area. The environment has a visual interface with the user whose aim is to facilitate the learning process in the generation and debugging of source code of applications developed for the parallel kernels and in particular for Virtuoso (Virtual Single Processor Programming System (TM)). The environment is composed of four tools: a parallel programs generator; a worst case execution time analyser; a scheduling analyser; and a parallel debugger. The first tool was released as a teaching environment for Virtuoso (TEV). The second of these tools is already under implementation and the last two tools will be available in the near future. Roughly, this visual environment should be able to provide continuity in the development of projects using the most common development methods (traditional or object-oriented), by offering support during the phases of implementation, debugging and testing.

It has been attracting much attention to make use of list homomorphisms in parallel programming because they ideally suit the divide-and-conquer parallel paradigm. However, they have been usually treated rather informally and ad hoc in the development of efficient parallel programs. What is worse is that some interesting functions, e.g., the maximum segment sum problem, are basically not list homomorphisms. In this article, we propose a systematic and formal way for the construction of a list homomorphism for a given problem so that an efficient parallel program is derived. We show, with several well-known but nontrivial problems, how a straightforward, and “obviously” correct, but quite inefficient solution to the problem can be successfully turned into a semantically equivalent “almost list homomorphism.” The derivation is based on two transformations, namely tupling and fusion, which are defined according to the specific recursive structures of list homomorphisms.

The development of efficient parallel programs requires expertise in the application domain as well as deep knowledge of parallel algorithms, languages and tools for the construction and execution of parallel programs. We present a method to make such expertise available in an domain specific tool set. Its construction is based on extensive use of a variety of powerful reuse methods. It automates a large amount of the software construction process, such that users need not know about parallelism.

The authors propose a repetitive fault tolerance (RFT) model, which provides an environment for the systematic development of fault tolerant parallel programs. RFT programs can tolerate processor failures without sacrificing performance. The system gives an optimal performance when all the processors are working while continuing to work, though at a lower performance, when failure occurs. Also, the system works as long as there is at least one working processor. Thus, it not only provides a software solution to achieve a highly reliable parallel computation environment but also provides an elegant solution for constructing reliable nonrepairable systems. The model is applied to three examples to illustrate the construction procedure and to evaluate the performance of repetitive fault tolerant programs as well as to demonstrate the applicability of this model. >

Computations on two-dimensional arrays such as matrices and images are one of the most fundamental and ubiquitous things in computational science and its vast application areas, but development of efficient parallel programs on two-dimensional arrays is known to be hard. To solve this problem, we have proposed a skeletal framework on two-dimensional arrays based on the theory of constructive algorithmics. It supports users, even with little knowledge about parallel machines, to develop systematically both correct and efficient parallel programs on two-dimensional arrays. In this paper, we apply our framework to the matrix-convolutions often used in image filters and difference methods. We show the efficacy of the framework by giving a general parallel program for the matrix-convolutions described with the skeletons, and a theorem that optimizes the general program into an application-specific one.KeywordsBinary OperatorParallel ProgramGeneral ProgramConstructive AlgorithmicsParallel ComplexityThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The development of parallel applications usually require the use of different techniques for each of the phases of development. The paper reports on the process of moving from the design of a parallel real time application into its implementation using the Parallel Program Generator. This tool is part of a visual environment for the development of parallel real time programs, an integrated environment for the development of applications executed in a parallel machine. The aim of the tool being described here is to facilitate the generation and debugging of source code of parallel real time applications. As the aim of the environment is to provide continuity in the development of projects using the most common development methods (traditional or object oriented), an example is provided in order to show the potential of the tool, whose first version is available for downloading at http://www.dc.ufscar.br//spl sim/tev/tev.html.

We present a technique for developing efficient functional programs by using optimal synchronizations among function calls. This technique is based on the application of the tupling strategy and it works for a wide class of functions defined by recursive equations. The derived programs are shown to have minimal redundancy, in the sense that repeated computations of identical recursive calls are never performed and identical subexpressions may be recomputed a number of times which is linearly bounded w.r.t. the depth of the recursion. The derived programs also have minimal spatial synchronization, in the sense that only a minimal number of computations are synchronized together, without increasing the parallel computation time.