- Book Chapter
3
- 10.1016/s0927-5452(04)80006-6
On compiler support for mixed task and data parallelism
- Jan 01, 2004
- Advances in Parallel Computing
- Thomas Rauber + 2 more +2
On compiler support for mixed task and data parallelism
An integrated performance analysis tool for SPMD data-parallel programs
On compiler support for mixed task and data parallelism
On compiler support for mixed task and data parallelism
Data parallelism and functional programming
Data parallelism is often seen as a form of explicit parallelism for SIMD and vector machines, and data parallel programming as an explicit programming paradigm for these architectures. Data parallel languages possess certain software qualities as well, which justifies their use in higher level programming and specification closer to the algorithm domain. Thus, it is interesting to study how the data parallel paradigm can be best realized in a declarative setting, since declarative languages offer a pure view of computation which is good for these purposes. For numerical computing the functional programming paradigm is especially attractive, since numerical algorithms often are specified by recursion equations and thus can be translated more or less directly into recursive functional programs. Merging the data parallel and functional paradigms then yields languages and formalisms where many algorithms can be expressed in a very succinct fashion. In this paper we review data parallelism, functional programming, and existing approaches to the integration of the two paradigms. We then proceed to describe a formalism for data parallel functional programming, allowing very simple languages, where the view of aggregate data is particularly abstract. We explain how various data parallel operations can be expressed in this formalism. Finally, we conclude with a discussion of issues for languages based directly on the formalism.KeywordsData FieldSystolic ArrayFunctional ProgrammingFunctional LanguageData ParallelismThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Read moreAdaptor: A Compilation System for Data Parallel Fortran Programs
Data parallel programming stands for single threaEnd, global name space, and loosely synchronous parallel computation. This kind of parallel programming has been proven to be very user-friendly, easy to Enbug and easy to use. But this programming moEnl is not available for most message passing multiprocessor architectures. Adaptor (Automatic Data Parallelism Translator) is a compilation system that transforms data parallel programs written in Fortran with array extensions, parallel loops, and layout directives to parallel programs with explicit message passing. Therefore it does automatic partitioning directed by user specified data distributions . The current version supports especially the translation of Connection Machine Fortran and High Performance Fortran programs to message passing programs. After a short Enscription of the system it will be shown how efficient the compilation is. Therefore many results about speed-ups and efficiencies of real coEns are presented. Furthermore, the automatically generated message passing programs are compared with hand written message passing programs. The results will show that data parallel programs will be competitive to hand written message passing programs if the data parallelism in the program can be recognized and utilized by the compiler.
Read moreTuning Task Granularity and Data Locality of Data Parallel GPH Programs
The performance of data parallel programs often hinges on two key coordination aspects: the computational costs of the parallel tasks relative to their management overhead — task granularity; and the communication costs induced by the distance between tasks and their data — data locality. In data parallel programs both granularity and locality can be improved by clustering, i.e. arranging for parallel tasks to operate on related sub-collections of data.The GPH parallel functional language automatically manages most coordination aspects, but also allows some high-level control of coordination using evaluation strategies. We study the coordination behavior of two typical data parallel programs, and find that while they can be improved by introducing clustering evaluation strategies, further performance improvements can be achieved by restructuring the program.We introduce a new generic Cluster class that allows clustering to be systematically introduced, and improved by program transformation. In contrast to many other parallel program transformation approaches, we transform realistic programs and report performance results on a 32-processor Beowulf cluster. The cluster class is highly-generic and extensible, amenable to reasoning, and avoids conflating computation and coordination aspects of the program.
Read moreData parallelism in Haskell
The implicit data parallelism in collective operations on aggregate data structures constitutes an attractive parallel programming model for functional languages. Beginning with our work on integrating nested data parallelism into Haskell, we explored a variety of different approaches to array-centric data parallel programming in Haskell, experimented with a range of code generation and optimisation strategies, and targeted both multicore CPUs and GPUs. In addition to practical tools for parallel programming, the outcomes of this research programme include more widely applicable concepts, such as Haskell's type families and stream fusion.In this talk, I will contrast the different approaches to data parallel programming that we explored. I will discuss their strengths and weaknesses and review what we have learnt in the course of exploring the various options. This includes our experience of implementing these approaches in the Glasgow Haskell Compiler as well the experimental results that we have gathered so far. Finally, I will outline the remaining open challenges and our plans for the future.This talk is based on joint work with Gabriele Keller, Sean Lee, Roman Leshchinskiy, Ben Lippmeier, Trevor L. McDonell, and Simon Peyton Jones.
Read moreBraid: integrating task and data parallelism
Archetype data parallel or task parallel applications are well served by contemporary languages. However, for applications containing a balance of task and data parallelism the choice of language is less clear. While there are languages that enable both forms of parallelism, e.g., one can write data parallel programs using a task parallel language, there are few languages which support both. We present a set of data parallel extensions to the Mentat Programming Language (MPL) which allow us to integrate task parallelism, data parallelism, and nested task and data parallelism within a single language on top of a single run time system. The result is an object-oriented language, Braid, that supports both task and data parallelism on MIMD machines. In addition, the data parallel extensions define a language in and of itself which makes a number of contributions to the data parallel programming style. These include subset-level operations (a more general notion of element-level operations), compiler provided iteration within a data parallel data set and the ability to define complex data parallel operations.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>
Read moreAutomatic data distribution for massively parallel processors
Massively Parallel Processor systems provide the required computational power to solve most large scale High Performance Computing applications. Machines with physically distributed memory allow a cost-effective way to achieve this performance, however, these systems are very diffcult to program and tune. In a distributed-memory organization each processor has direct access to its local memory, and indirect access to the remote memories of other processors. But the cost of accessing a local memory location can be more than one order of magnitude faster than accessing a remote memory location. In these systems, the choice of a good data distribution strategy can dramatically improve performance, although different parts of the data distribution problem have been proved to be NP-complete.<br/>The selection of an optimal data placement depends on the program structure, the program's data sizes, the compiler capabilities, and some characteristics of the target machine. In addition, there is often a trade-off between minimizing interprocessor data movement and load balancing on processors. Automatic data distribution tools can assist the programmer in the selection of a good data layout strategy. These use to be source-to-source tools which annotate the original program with data distribution directives.<br/>Crucial aspects such as data movement, parallelism, and load balance have to be taken into consideration in a unified way to efficiently solve the data distribution problem.<br/>In this thesis a framework for automatic data distribution is presented, in the context of a parallelizing environment for massive parallel processor (MPP) systems. The applications considered for parallelization are usually regular problems, in which data structures are dense arrays. The data mapping strategy generated is optimal for a given problem size and target MPP architecture, according to our current cost and compilation model.<br/>A single data structure, named Communication-Parallelism Graph (CPG), that holds symbolic information related to data movement and parallelism inherent in the whole program, is the core of our approach. This data structure allows the estimation of the data movement and parallelism effects of any data distribution strategy supported by our model. Assuming that some program characteristics have been obtained by profiling and that some specific target machine features have been provided, the symbolic information included in the CPG can be replaced by constant values expressed in seconds representing data movement time overhead and saving time due to parallelization. The CPG is then used to model a minimal path problem which is solved by a general purpose linear 0-1 integer programming solver. Linear programming techniques guarantees that the solution provided is optimal, and it is highly effcient to solve this kind of problems.<br/>The data mapping capabilities provided by the tool includes alignment of the arrays, one or two-dimensional distribution with BLOCK or CYCLIC fashion, a set of remapping actions to be performed between phases if profitable, plus the parallelization strategy associated. <br/>The effects of control flow statements between phases are taken into account in order to improve the accuracy of the model. The novelty of the approach resides in handling all stages of the data distribution problem, that traditionally have been treated in several independent phases, in a single step, and providing an optimal solution according to our model.
Read moreMapping strategies in data parallel programming models; the projection methods
We Analyze data parallel programming of some general purpose methods in linear algebra. Specifically, the projection methods to solve very large linear system and/or eigenproblem. The expensive parts of these methods are their projection phases. This portion of these algorithms has generally, a very simple structure for such a programming model. It is composed essentially of matrix-vector multiplications and inner-products. Then, we simply need to find a good data distribution in order to obtain a well adapted communication pattern and not loose too much storage space. We begin with a survey of data parallel behavior of some projection methods such as Arnoldi, GMRES, Lanczos, PRR,... After analyzing some methods of data mapping onto virtual processors we point out that for a fixed number of physical processors, the performances are a function of the mapping method. We will see also that the maximum size of the problems which can be solved on the architectures supporting the data parallel programming model is a function of the data mapping method. In conclusion we present the performances obtained on a Connection Machine 5 (CM5).
Read moreChapter 5 - Collective Communication Support
Chapter 5 - Collective Communication Support
Programming abstractions, compilation, and execution techniques for massively parallel data analysis
Aufgrund fallender Preise zur Speicherung von Daten kann man derzeit eine explosionsartige Zunahme in der Menge der verfügbaren Daten beobachten. Diese Entwicklung gibt Unternehmen und wissenschaftliche Institutionen die Möglichkeit empirische Daten in ungekannter Größenordnung zu analysieren. Für viele Firmen ist die Analyse der gesammelten Daten aus ihrem operationalen Geschäft längst zu einem zentralen strategischen Aspekt geworden. Im Gegensatz zu der seit längerem schon betriebenen Business Intelligence, bestehen diese Analysen nicht mehr nur aus traditionellen relationalen Anfragen. In zunehmendem Anteil kommen komplexe Algorithmen aus den Bereichen Data Mining und Maschinelles Lernen hinzu, um versteckte Muster in den Daten zu erkennen, oder Vorhersagemodelle zu trainieren. Mit zunehmender Datenmenge und Komplexität der Analysen wird jedoch eine neue Generation von Systemen benötigt, die diese Kombination aus Anfragekomplexität und Datenvolumen gewachsen sind. Relationale Datenbanken waren lange Zeit das Zugpferd der Datenanalyse im großen Stil. Grund dafür war zum großen Teil ihre deklarativen Anfragesprache, welche es ermöglichte die logischen und physischen Aspekte der Datenspeicherung und Verarbeitung zu trennen, und Anfragen automatisch zu optimieren. Das starres Datenmodell und ihre beschränkte Menge von möglichen Operationen schränken jedoch die Anwendbarkeit von relationalen Datenbanken für viele der neueren analytischen Probleme stark ein. Diese Erkenntnis hat die Entwicklung einer neuen Generation von Systemen und Architekturen eingeläutet, die sich durch sehr generische Abstraktionen für parallelisierbare analytische Programme auszeichnen; MapReduce kann hier beispielhaft genannt werden, als der zweifelsohne prominenteste Vertreter dieser Systeme. Zwar vereinfachte und erschloss diese neue Generation von Systemen die Datenanalyse in diversen neuen Anwendungsfeldern, sie ist jedoch nicht in der Lage komplexe Anwendungen aus den Bereichen Data Mining und Maschinelles Lernen effizient abzubilden, ohne sich dabei extrem auf spezifische Anwendungen zu spezialisieren. Verglichen mit den relationalen Datenbanken haben MapReduce und vergleichbare Systeme außerdem die deklarative Abstraktion aufgegeben und zwingen den Anwender dazu systemnahe Programme zu schreiben und diese manuell zu optimieren. In dieser Dissertation werden verschiedene Techniken vorgestellt, die es ermöglichen etliche der zentralen Eigenschaften von relationalen Datenbanken im Kontext dieser neuen Generation von daten-parallelen Analysesystemen zu realisieren. Mithilfe dieser Techniken ist es möglich ein Analysesystem zu beschreiben, dessen Programme gleichzeitig sowohl generische und ausdrucksstark, als auch prägnant und deklarativ sind. Im einzelnen stellen wir folgende Techniken vor: Erstens, eine Programmierabstraktion die generisch ist und mit komplexen Datenmodellen umgehen kann, aber gleichzeitig viele der deklarativen Eigenschaften der relationalen Algebra erhält. Programme, die gegen dies Abstraktion entwickelt werden können ähnlich optimiert werden wie relationale Anfragen. Zweitens stellen wir eine Abstraktion für iterative daten-parallele Algorithmen vor. Die Abstraktion unterstützt inkrementelle (delta-basierte) Berechnungen und geht mit zustandsbehafteteten Berechnungen transparent um. Wir beschreiben wie man einen relationalen Anfrageoptimierer erweitern kann so dass dieser iterative Anfragen effektiv optimiert. Wir zeigen dabei dass der Optimierer dadurch in die Lage versetzt wird automatisch Ausführungspläne zu erzeugen, die wohlbekannten, manuell erstellten Programmen entsprechen. Die Abstraktion subsumiert dadurch spezialisierte Systeme (wie Pregel) und bietet vergleichbare Performanz. Drittens stellen wir Methoden vor, um die Programmierabstraktion in eine funktionale Sprachen einzubetten. Diese Integration ermögliche es prägnante Programme zu schreiben und einfach wiederzuverwendenden Komponenten und Bibliotheken, sowie Domänenspezifische Sprachen, zu erstellen. Wir legen dar wie man die Übersetzung und Optimierung des daten-parallelen Programms mit dem Sprachübersetzer der funktionalen Sprache so integriert, dass maximales Optimierungspotenzial besteht.
Read moreDeFT
While multicore processors promise large performance benefits for parallel applications, writing these applications is notoriously difficult. Tuning a parallel application to achieve good performance, also known as performance debugging, is often more challenging than debugging the application for correctness. Parallel programs have many performance-related issues that are not seen in sequential programs. An increase in cache misses is one of the biggest challenges that programmers face. To minimize these misses, programmers must not only identify the source of the extra misses, but also perform the tricky task of determining if the misses are caused by interthread communication (i.e., coherence misses) and if so, whether they are caused by true or false sharing (since the solutions for these two are quite different). In this article, we propose a new programmer-centric definition of false sharing misses and describe our novel algorithm to perform coherence miss classification. We contrast our approach with existing data-centric definitions of false sharing. A straightforward implementation of our algorithm is too expensive to be incorporated in real hardware. Therefore, we explore the design space for low-cost hardware support that can classify coherence misses on-the-fly into true and false sharing misses, allowing existing performance counters and profiling tools to expose and attribute them. We find that our approximate schemes achieve good accuracy at only a fraction of the cost of the ideal scheme. Additionally, we demonstrate the usefulness of our work in a case study involving a real application.
Read moreOnline Scalability Characterization of Data-Parallel Programs on Many Cores
We present an accurate online scalability prediction model for data-parallel programs on NUMA many-core systems. Memory contention is considered to be the major limiting factor of program scalability as data parallelism limits the amount of synchronization or data dependencies between parallel work units. Reflecting the architecture of NUMA systems, contention is modeled at the last-level caches of the compute nodes and the memory nodes using a two-level queuing model to estimate the mean service time of the individual memory nodes. Scalability predictions for individual or co-located parallel applications are based solely on data obtained during a short sampling period at runtime; this allows the presented model to be employed in a variety of scenarios. The proposed model has been implemented into an open-source OpenCL and the GNU OpenMP runtime and evaluated on a 64-core AMD system. For a wide variety of parallel workloads and configurations, the evaluations show that the model is able to predict the scalability of data-parallel kernels with high accuracy.
Read moreDAME: an environment for preserving the efficiency of data-parallel computations on distributed systems
DAME (Data Migration Environment) uses transparent supports to overcome inefficiencies in data parallel programming. These supports hide irregular network topology, dynamically adapt the data distribution to platform conditions, and mask the consequent nonuniform distribution to the programmer. The authors compare DAME's performance with that of some popular frameworks. They begin by discussing DAME's three main design goals: efficiency, transparency, and scalability. Next, they describe the five supports that DAME gives the programmer: virtual topology, data distribution, data management, interprocess communication, and workload reconfiguration. Then, they present the results they obtained from experiments using 10 workstations that provide a hardware heterogeneous, data homogeneous, nonuniform platform. The results show that DAME provides a virtual single program, multiple data machine that overcomes most of the differences that distinguish a parallel virtual machine from an ideal SPMD machine.
Read moreA performance visualization paradigm for data parallel computing
Observing the activities of a complex parallel computer system is no small feat, and relating these observations to program behavior is even harder. This paper presents a general measurement approach that is applicable to a large class of scalable programs and machines, specifically data parallel programs executing on distributed memory computer systems. The combined instrumentation and visualization paradigm, called VISTA (which stands for Visualization and Instrumentation of Scalable mulTicomputer Applications), is based on the author's experiences of programming and monitoring applications running on an nCUBE 2 computer and a MasPar MP-1 computer. The key is that performance data are treated similarly to any distributed data in the context of the data parallel programming model. Because of the data-parallel mapping of the program onto the machine, one can view the performance as it relates to each processor, processor cluster or processor ensemble and as it relates to the data structures of the program. The author illustrates the utility of VISTA by way of an example.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>
Read moreCompiling data parallel programs to message passing programs for massively parallel MIMD systems
S.100-107