Understanding the performance of parallel symbolic programs

The performance of data parallel programs often hinges on two key coordination aspects: the computational costs of the parallel tasks relative to their management overhead — task granularity; and the communication costs induced by the distance between tasks and their data — data locality. In data parallel programs both granularity and locality can be improved by clustering, i.e. arranging for parallel tasks to operate on related sub-collections of data.The GPH parallel functional language automatically manages most coordination aspects, but also allows some high-level control of coordination using evaluation strategies. We study the coordination behavior of two typical data parallel programs, and find that while they can be improved by introducing clustering evaluation strategies, further performance improvements can be achieved by restructuring the program.We introduce a new generic Cluster class that allows clustering to be systematically introduced, and improved by program transformation. In contrast to many other parallel program transformation approaches, we transform realistic programs and report performance results on a 32-processor Beowulf cluster. The cluster class is highly-generic and extensible, amenable to reasoning, and avoids conflating computation and coordination aspects of the program.

MPI is the most important parallel programming tool in cluster currently. It implements communication in parallel program by message. Implementing load balance in MPI parallel program is very important. It may reduce running time and improve performance of MPI parallel program, aiming at solving the dynamic balancing problem in homogeneous cluster system. This paper proposes an implementing method in MPI parallel program that can transfer tasks between nodes effectively by nodepsilas load. The experiments prove the availability and practicability of the algorithm in parallel computing task.

In recent years, even though the chip density can still increase, it is difficult to increase the main frequency. Performance improvements for single processors may be close to their limits. So multi-core processors are coming on the scene. In order to make full use of multi-core platforms, we must find the inherent parallelism of programs and write programs that can execute in parallel. OpenMP standard is widely used in parallel programming because of its good portability and ease of use. For OpenMP programs generated by parallel compilers and OpenMP programs which only have simple parallelism, they belong to programs with insufficient optimization. In OpenMP parallel execution, synchronization control is one of the main overheads. Its unnecessary barrier synchronization reduces the performance of parallel program. This paper discusses an optimization technology of OpenMP program. Firstly, the parallel area of OpenMP program is merged and expanded to reduce the overhead of parallel and serial switching in execution, and at the same time, it is convenient for next steps to optimize the program. Then explicit the implicit synchronization in OpenMP program. Finally, data dependency analysis is carried out for the context of each synchronization, and unnecessary synchronizations are deleted. The program is tested with the program running time as an index to evaluate the performance of the test program. The experimental results show that the optimization strategy proposed in this paper correctly reconstructs the parallel regions and reduces its execution overhead; it reduces the number of redundant synchronizations and effectively improve the performance of OpenMP program.

A mixed method combining formal and auto-tuning approaches and aimed at maximizing efficiency of parallel programs (in terms of execution time) is proposed. The formal approach is based on algorithmic algebra and the usage of tools for automated design and synthesis of programs based on high-level algorithm specifications (schemes). Parallel software auto-tuning is the method of adjusting some structural parameters of a program to a target hardware platform to speed-up computation as much as possible. Previously, we have developed a framework intended to automate the generation of an auto-tuner from a program source code. However, auto-tuning for complex and nontrivial parallel systems is usually time-consuming due to empirical evaluation of huge amount of parameter values combinations of an initial parallel program in a target environment. In this paper, we extend our approach with statistical modeling and neural network algorithms that allow to reduce significantly the space of possible parameter combinations. The improvement consists in automatic training of a neural network model on results of “traditional” tuning cycles and the subsequent replacement of some auto-tuner calls with an evaluation from the statistical model. The method allows, particularly, transferring knowledge about the influence of parameters on program performance between “similar” (in terms of hardware architecture) computing environments for the same applications. The idea is to reuse a model trained on data from a similar environment. The use of the method is illustrated by an example of tuning a parallel sorting program which combines several sorting methods.

Our experiments with Qlisp have led to two ways of improving the performance of parallel Lisp programs. One is to reduce the cost of process creation and scheduling; the other is to use a dynamic partitioning and scheduling method. We describe these techniques and present the results of several experiments that use them. We also present an analysis of the dynamic partitioning method to explain the reasons for its success.

Lattice Boltzmann method (LBM) is a new attractive computational approach for simulating isothermal multi-phase flows in computational fluid dynamics (CFD). It is based on the kinetic theory and easy to be parallelized. This study aims to analyze the performance of parallel LBM programming for the incompressible two-phase flows at high density and viscosity ratio. For this purpose, a liquid drop impact on a wetted wall with a pre-existing thin film of the same liquid is simulated by using the parallel LBM code. During the simulations, the domain decomposition, data communication and parallelization of the LBM code using the message passing interface (MPI) library have been investigated. The computational results show that the parallel LBM code exhibits a good high performance computing (HPC) on the parallel speed-up.

Efficient and scalable performance modeling is essential to high-performance cluster computing. The critical path based performance analysis has been widely used as it provides valuable insights into the performance of parallel programs, but it is also expensive, inefficient, and inflexible due to its strong reliance on trace-driven simulation. This paper presents an innovative performance modeling framework based on a novel concept of critical-path candidates. The critical-path candidates refer to a group of paths that could potentially be the critical path. Using the instruction and communication counts as the metrics, the critical-path candidate captures the intrinsic computation and communication dependencies, and hence can be reused for exploring multiple design options. Using real-world MPI workloads, we show that the proposed framework achieves a modeling accuracy within 10% compared with the measured runtime for up to 16K MPI ranks. This framework provides an efficient and scalable platform for performance analysis as well as load imbalance analysis.

Automatic decomposition is a compile technique that maps computation and data onto different processors, and array is one of the main targets processed. Certain arrays, whose migration is the most expensive, are termed as dominant arrays. Since every computing node has its own memory on distributed memory parallel computers (DMPCs), decompositions of dominant arrays have directly impact on the performance of parallel program. To avoid remote data accessing, each definition and use of arrays needs to be distributed consistently, so as that there are too many partition constraints to increase decomposition choices of dominant arrays. We propose an automatic computation and data decomposition algorithm of prioritized dominant array in this paper. Our algorithm ranks arrays according to their potential communication costs, and then finds data decomposition for arrays in the decreasing order of rank. We serialize low rank arrays to enhance the decomposition priority of high rank ones. Finally, serial arrays are to be partitioned if maintaining parallel benefits of previous results. The experimental results show that this algorithm can improve the performance of parallel programs.

We describe new features of FG that are designed to improve performance and extend the range of computations that fit into its framework. FG (short for framework generator) is a programming environment for parallel programs running on clusters. It was originally designed to mitigate latency in accessing data by running a program as a series of asynchronous stages that operate on buffers in a linear pipeline. To improve performance, FG now allows stages to be replicated, either statically by the programmer or dynamically by FG itself. FG also now alters thread priorities to use resources more efficiently; again, this action may be initiated by either the programmer or FG. To extend the range of computations that fit into its framework, FG now incorporates fork-join and DAG structures. Not only do fork-join and DAG structures allow for more programs to be designed for FG, but they also can enable significant performance improvements over linear pipeline structures.

In this paper, we report a performance gap betweeen a schedule with small makespan on the task scheduling model and the corresponding parallel program on distributed memory parallel machines. The main reason of the gap is the software overhead in the interprocessor communication. Therefore, speedup ratios of schedules on the model do not approximate well to those of parallel programs on the machines. The purpose of the paper is to get a task scheduling algorithm that generates a schedule with good approximation to the corresponding parallel program and with small makespan. For this purpose, we propose algorithm BCSH that generates only bulk synchronous schedules. In those schedules, no-communication phases and communication phases appear alternately. All interprocessor communications are done only in the latter phases, and thus the corresponding parallel programs can make better use of the message packaging technique easily. It reduces many software overheads of messages form a source processor to the same destination processor to almost one software overhead, and improves the performance of a parallel program significantly. Finally, we show some experimental results of performance gaps on BCSH, Kruatrachue's algorithm DSH, and Ahmad et al's algorithm ECPFD. The schedules by DSH and ECPFD are famous for their small makespans, but message packaging can not be effectively applied to the corresponding program. The results show that a bulk synchronous schedule with small makespan has advantages that the gap is small and the corresponding program is a high performance parallel one.

Multicore processors serve as target platforms in a broad variety of applications ranging from high-performance computing to embedded mobile computing and automotive. But, the required parallel programming opens up a huge design space of parallelization strategies each with potential bottlenecks. Therefore, an early estimation of an application's performance is a desirable development tool. However, out-of-order execution, superscalar instruction pipelines, as well as communication costs and (shared-) cache effects essentially influence the performance of parallel programs. While offering a good modeling and simulation speed, analytic models provide moderate prediction results so far. Virtual prototyping requires a time-consuming simulation, but produces better accuracy. Furthermore, even existing statistical methods often require detailed knowledge of the hardware for characterization. In this work, we present a concept and its evaluation for a statistical approach for performance prediction based on abstract runtime parameters, which describe an application's scalability behavior and can be extracted from profiles without user input. These scalability parameters not only include information on the interference of software demands and hardware capabilities, but indicate bottlenecks as well. Depending on the database setup, we achieve a competitive accuracy of 20 % mean prediction error (11 % median), which we also proof in a case study.

ParaGraph, a software tool that provides a detailed, dynamic, graphical animation of the behavior of message-passing parallel programs and graphical summaries of their performance, is presented. ParaGraph animates trace information from actual runs to depict behavior and obtain the performance summaries. It provides twenty-five perspectives on the same data, lending insight that might otherwise be missed. ParaGraph's features are described, its use is explained, its software design is briefly discussed, and its displays are examined in some detail. Future work on ParaGraph is indicated.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

In this paper, a parallel implementation of an elliptic solver in solving 1D Boussinesq model is presented. Numerical solution of Boussinesq model is obtained by implementing a staggered grid scheme to continuity, momentum, and elliptic equation of Boussinesq model. Tridiagonal system emerging from numerical scheme of elliptic equation is solved by cyclic reduction algorithm. The parallel implementation of cyclic reduction is executed on multicore processors with shared memory architectures using OpenMP. To measure the performance of parallel program, large number of grids is varied from 28 to 214. Two test cases of numerical experiment, i.e. propagation of solitary and standing wave, are proposed to evaluate the parallel program. The numerical results are verified with analytical solution of solitary and standing wave. The best speedup of solitary and standing wave test cases is about 2.07 with 214 of grids and 1.86 with 213 of grids, respectively, which are executed by using 8 threads. Moreover, the best efficiency of parallel program is 76.2% and 73.5% for solitary and standing wave test cases, respectively.

The performance of SPMD parallel programs is strongly affected by dynamic load imbalancing factors. The use of a suitable load balancing strategy is essential in overcoming the effects of these imbalancing factors. This chapter deals with concepts and experiments related to load balancing in SPMD applications. Initially, we discuss a set of classification criteria for load balancing algorithms designed for SPMD applications. In addition, we define a load imbalancing index in order to measure the load imbalance of a parallel application execution. In the experimental part of this chapter, we describe the development of an SPMD parallel application which computes the macroscopic thermal dispersion in porous media. Nine versions of this scientific application were developed, each one adopting a different load balancing strategy. We evaluate and compare the performance of these nine versions and show the importance of using an appropriate load balancing strategy for the characteristics of a specific SPMD parallel application.

Reuse distance (RD) analysis is a powerful memory analysis tool that can potentially help architects study multicore processor scaling. One key obstacle though is multicore RD analysis requires measuring concurrent reuse distance (CRD) profiles across thread-interleaved memory reference streams. Sensitivity to memory interleaving makes CRD profiles architecture dependent, preventing them from analyzing different processor configurations. For loop-based parallel programs, CRD profiles shift coherently to larger CRD values with core count scaling because interleaving threads are symmetric. Simple techniques can predict such shifting, making the analysis of numerous multicore configurations from a small set of CRD profiles feasible. Given the ubiquity and scalability of loop-level parallelism, such techniques will be extremely valuable for studying future large multicore designs. This paper investigates using RD analysis to efficiently analyze multicore cache performance for loop-based parallel programs, making several contributions. First, we provide in depth analysis on how CRD profiles change with core count scaling. Second, we develop techniques to predict CRD profile scaling, in particular employing reference groups to predict coherent shift, and evaluate prediction accuracy. Third, we show core count scaling only degrades performance for last level caches (LLCs) below 16MB for our benchmarks and problem sizes, increasing to 64 -- 128MB if problem size scales by 64x. Finally, we apply CRD profiles to analyze multicore cache performance. When combined with existing problem scaling prediction, our techniques can predict LLC MPKI to within 11.1% of simulation across 1,728 configurations using only 36 measured CRD profiles.