Dynamic Load Balancing Algorithm for MPI Parallel Computing

Parallel loop self‐scheduling on parallel and distributed systems has been a critical problem and it is becoming more difficult to deal with in the emerging heterogeneous cluster computing environments. In the past, some self‐scheduling schemes have been proposed as applicable to heterogeneous cluster computing environments. In recent years, multicore computers have been widely included in cluster systems. However, previous researches into parallel loop self‐scheduling did not consider certain aspects of multicore computers; for example, it is more appropriate for shared‐memory multiprocessors to adopt Open Multi‐Processing (OpenMP) for parallel programming. In this paper, we propose a performance‐based approach using hybrid OpenMP and MPI parallel programming, which partition loop iterations according to the performance weighting of multicore nodes in a cluster. Because iterations assigned to one MPI process are processed in parallel by OpenMP threads run by the processor cores in the same computational node, the number of loop iterations allocated to one computational node at each scheduling step depends on the number of processor cores in that node. Experimental results show that the proposed approach performs better than previous schemes. Copyright © 2010 John Wiley & Sons, Ltd.

The performance of data parallel programs often hinges on two key coordination aspects: the computational costs of the parallel tasks relative to their management overhead — task granularity; and the communication costs induced by the distance between tasks and their data — data locality. In data parallel programs both granularity and locality can be improved by clustering, i.e. arranging for parallel tasks to operate on related sub-collections of data.The GPH parallel functional language automatically manages most coordination aspects, but also allows some high-level control of coordination using evaluation strategies. We study the coordination behavior of two typical data parallel programs, and find that while they can be improved by introducing clustering evaluation strategies, further performance improvements can be achieved by restructuring the program.We introduce a new generic Cluster class that allows clustering to be systematically introduced, and improved by program transformation. In contrast to many other parallel program transformation approaches, we transform realistic programs and report performance results on a 32-processor Beowulf cluster. The cluster class is highly-generic and extensible, amenable to reasoning, and avoids conflating computation and coordination aspects of the program.

Parallel programs have traditionally been implemented on distributed systems using conventional programming languages, such as Fortran or C, plus a library with support for communication, such as, for instance, PVM [11]. Also, some effort has been put into developing parallel programming languages [13]. Usually, the different development platforms are compared as to application execution time or parallel efficiency. However, as pointed out in [4], when measuring the performance of a parallel application or programming tool, different metrics may need to be taken into consideration. The ease of construction of an application and the time a programmer spends to implement it may in some cases be more important than the final execution time. Working under this assumption, this work argues that the use of interpreted languages as a basis for parallel programming tools should be further investigated. As support for our argument, we show an example of use of an interpreted language in a classical application in parallel distributed programming, namely, that of detecting program termination. Two different algorithms for termination detection are implemented in Lua, an extension language developed at PUC-Rio [5].

Parallel symbolic programs usually exhibit certain characteristics: dynamic, data-dependent behavior patterns; complex run-time interactions that cross modularity boundaries; and control structures characterized by recursion and hierarchy. Studying the performance of such programs requires flexible tools that can adapt to the many different execution patterns that can occur.

With the advent of Gigabit Ethernet network technology, a new contender for the next-generation cluster interconnect is on the horizon. We benchmark Gigabit Ethernet and compare it to other cluster interconnects, namely Fast Ethernet, Myrinet and Scalable Coherent Interface. For a meaningful comparison, benchmark experiments are carried out at two levels: TCP/IP networking and MPI parallel programming. Using high-end PCs (Pentium III 450 MHz) and standard system software (Linux and MPICH), our results show that with Gigabit Ethernet, end-to-end throughputs of up to 44 MBytes per second can be achieved. A change from Fast Ethernet to Gigabit Ethernet resulted in performance gains of up to a factor of 5.05 for ftp, up to a factor of 3.10 for MPI point-to-point communication and up to a factor of 2.58 for some NAS parallel benchmarks, depending on communication granularity and message size. Further performance gains seem to require improved protocol processing and node hardware (memory bandwidth, file system, PCI bus).

A mixed method combining formal and auto-tuning approaches and aimed at maximizing efficiency of parallel programs (in terms of execution time) is proposed. The formal approach is based on algorithmic algebra and the usage of tools for automated design and synthesis of programs based on high-level algorithm specifications (schemes). Parallel software auto-tuning is the method of adjusting some structural parameters of a program to a target hardware platform to speed-up computation as much as possible. Previously, we have developed a framework intended to automate the generation of an auto-tuner from a program source code. However, auto-tuning for complex and nontrivial parallel systems is usually time-consuming due to empirical evaluation of huge amount of parameter values combinations of an initial parallel program in a target environment. In this paper, we extend our approach with statistical modeling and neural network algorithms that allow to reduce significantly the space of possible parameter combinations. The improvement consists in automatic training of a neural network model on results of “traditional” tuning cycles and the subsequent replacement of some auto-tuner calls with an evaluation from the statistical model. The method allows, particularly, transferring knowledge about the influence of parameters on program performance between “similar” (in terms of hardware architecture) computing environments for the same applications. The idea is to reuse a model trained on data from a similar environment. The use of the method is illustrated by an example of tuning a parallel sorting program which combines several sorting methods.

We describe new features of FG that are designed to improve performance and extend the range of computations that fit into its framework. FG (short for framework generator) is a programming environment for parallel programs running on clusters. It was originally designed to mitigate latency in accessing data by running a program as a series of asynchronous stages that operate on buffers in a linear pipeline. To improve performance, FG now allows stages to be replicated, either statically by the programmer or dynamically by FG itself. FG also now alters thread priorities to use resources more efficiently; again, this action may be initiated by either the programmer or FG. To extend the range of computations that fit into its framework, FG now incorporates fork-join and DAG structures. Not only do fork-join and DAG structures allow for more programs to be designed for FG, but they also can enable significant performance improvements over linear pipeline structures.

In recent years, even though the chip density can still increase, it is difficult to increase the main frequency. Performance improvements for single processors may be close to their limits. So multi-core processors are coming on the scene. In order to make full use of multi-core platforms, we must find the inherent parallelism of programs and write programs that can execute in parallel. OpenMP standard is widely used in parallel programming because of its good portability and ease of use. For OpenMP programs generated by parallel compilers and OpenMP programs which only have simple parallelism, they belong to programs with insufficient optimization. In OpenMP parallel execution, synchronization control is one of the main overheads. Its unnecessary barrier synchronization reduces the performance of parallel program. This paper discusses an optimization technology of OpenMP program. Firstly, the parallel area of OpenMP program is merged and expanded to reduce the overhead of parallel and serial switching in execution, and at the same time, it is convenient for next steps to optimize the program. Then explicit the implicit synchronization in OpenMP program. Finally, data dependency analysis is carried out for the context of each synchronization, and unnecessary synchronizations are deleted. The program is tested with the program running time as an index to evaluate the performance of the test program. The experimental results show that the optimization strategy proposed in this paper correctly reconstructs the parallel regions and reduces its execution overhead; it reduces the number of redundant synchronizations and effectively improve the performance of OpenMP program.

Current and planned computer systems present challenges for scientific programming. Memory capacity and bandwidth are limiting performance as floating point capability increases due to more cores per processor and wider vector units. Effectively using hardware requires finding greater parallelism in programs while using relatively less memory. In this poster, we present how we tuned the Livermore Unstructured Lagrange Explicit Shock Hydrodynamics proxy application for on-node performance resulting in 62% fewer memory reads, a 19% smaller memory footprint, 770% more floating point operations vectorizing and less than 0.1% serial section runtime. Tests show serial code version runtime decreases of up to 57% and parallel runtime reductions of up to 75%. We are also applying these optimizations to GPUs and a subset of ALE3D, from which the proxy application was derived. So far we achieve up to a 1.9x speedup on GPUs, and a 13% runtime reduction in the application for the same problem.

Our experiments with Qlisp have led to two ways of improving the performance of parallel Lisp programs. One is to reduce the cost of process creation and scheduling; the other is to use a dynamic partitioning and scheduling method. We describe these techniques and present the results of several experiments that use them. We also present an analysis of the dynamic partitioning method to explain the reasons for its success.

Lattice Boltzmann method (LBM) is a new attractive computational approach for simulating isothermal multi-phase flows in computational fluid dynamics (CFD). It is based on the kinetic theory and easy to be parallelized. This study aims to analyze the performance of parallel LBM programming for the incompressible two-phase flows at high density and viscosity ratio. For this purpose, a liquid drop impact on a wetted wall with a pre-existing thin film of the same liquid is simulated by using the parallel LBM code. During the simulations, the domain decomposition, data communication and parallelization of the LBM code using the message passing interface (MPI) library have been investigated. The computational results show that the parallel LBM code exhibits a good high performance computing (HPC) on the parallel speed-up.

In this paper, we report a performance gap betweeen a schedule with small makespan on the task scheduling model and the corresponding parallel program on distributed memory parallel machines. The main reason of the gap is the software overhead in the interprocessor communication. Therefore, speedup ratios of schedules on the model do not approximate well to those of parallel programs on the machines. The purpose of the paper is to get a task scheduling algorithm that generates a schedule with good approximation to the corresponding parallel program and with small makespan. For this purpose, we propose algorithm BCSH that generates only bulk synchronous schedules. In those schedules, no-communication phases and communication phases appear alternately. All interprocessor communications are done only in the latter phases, and thus the corresponding parallel programs can make better use of the message packaging technique easily. It reduces many software overheads of messages form a source processor to the same destination processor to almost one software overhead, and improves the performance of a parallel program significantly. Finally, we show some experimental results of performance gaps on BCSH, Kruatrachue's algorithm DSH, and Ahmad et al's algorithm ECPFD. The schedules by DSH and ECPFD are famous for their small makespans, but message packaging can not be effectively applied to the corresponding program. The results show that a bulk synchronous schedule with small makespan has advantages that the gap is small and the corresponding program is a high performance parallel one.

Efficient and scalable performance modeling is essential to high-performance cluster computing. The critical path based performance analysis has been widely used as it provides valuable insights into the performance of parallel programs, but it is also expensive, inefficient, and inflexible due to its strong reliance on trace-driven simulation. This paper presents an innovative performance modeling framework based on a novel concept of critical-path candidates. The critical-path candidates refer to a group of paths that could potentially be the critical path. Using the instruction and communication counts as the metrics, the critical-path candidate captures the intrinsic computation and communication dependencies, and hence can be reused for exploring multiple design options. Using real-world MPI workloads, we show that the proposed framework achieves a modeling accuracy within 10% compared with the measured runtime for up to 16K MPI ranks. This framework provides an efficient and scalable platform for performance analysis as well as load imbalance analysis.

Automatic decomposition is a compile technique that maps computation and data onto different processors, and array is one of the main targets processed. Certain arrays, whose migration is the most expensive, are termed as dominant arrays. Since every computing node has its own memory on distributed memory parallel computers (DMPCs), decompositions of dominant arrays have directly impact on the performance of parallel program. To avoid remote data accessing, each definition and use of arrays needs to be distributed consistently, so as that there are too many partition constraints to increase decomposition choices of dominant arrays. We propose an automatic computation and data decomposition algorithm of prioritized dominant array in this paper. Our algorithm ranks arrays according to their potential communication costs, and then finds data decomposition for arrays in the decreasing order of rank. We serialize low rank arrays to enhance the decomposition priority of high rank ones. Finally, serial arrays are to be partitioned if maintaining parallel benefits of previous results. The experimental results show that this algorithm can improve the performance of parallel programs.

Achieving reproducibility of scientific results in parallel computing is both a challenge and a source of active research. A significant contribution to non-reproducibility is rounding error introduced into calculations by the non-associativity of floating point addition. Scientific applications that rely on accumulation of many small values, such as climate and N-body simulations, are susceptible to this type of error. This paper proposes a variant of an existing fixed-point method for real number summation that yields sums with perfect precision, and which are invariant to summation order and system architecture. The new method improves upon the existing technique by exhibiting improved performance for large numbers of summands, introducing tunable fractional precision to place precision where it is needed, and eliminating the aliasing problem of the original method. The proposed technique is described and its performance is demonstrated in the OpenMP, MPI, CUDA, and Xeon Phi parallel programming environments. In particular, the proposed method outperforms the previous state-of-the-art for larger problems involving over one million summands at high precision. With the anticipated convergence of exascale high-performance computing and big data analytics on hybrid architectures, computational reproducibility will become an even more difficult problem than it is today. Use of numerical techniques such as the method proposed here can help to mitigate the impact of error and variation within simulations at these large scales.