Adaptive Performance Modeling on Hierarchical Grid Computing Environments

A few years ago, there was a huge development of new parallel and distributed systems. For many reasons, such as the inherent heterogeneity, the diversity and the continuous evolution of such computational supports, it is very hard to solve efficiently a target problem by using a single algorithm or to write portable programs that perform well on any architecture. Towards this goal, we propose a generic framework combining communication models and adaptive approaches to deal with the performance modelling problem associated with the design of efficient parallel algorithms in grid computing environments, and we apply this methodology to collective communication operations. Experiments performed on a grid platform prove that the framework provides significant performances while determining the best combination of model and algorithm depending on the problem and architecture parameters.

Parallel computing utilizes heterogeneous parallel computing environments in which multiple processors of various kinds are connected by communication links. Efficient collective communication operations are needed to realize efficient parallel computing in parallel computing environments. Although methods for realizing efficient collective communication operations have previously been proposed for homogeneous parallel computing environments, these methods applied as they are to heterogeneous parallel computing environments cannot realize efficient collective communication operations. In this paper, we consider design of an efficient broadcast, which is one of the most important collective communication operations. We show that we can extend the known sufficient condition for Fastest‐Node‐First (FNF) to output an optimal schedule, where FNF is an efficient heuristic algorithm for the broadcast. We show that we can reduce the time cost of the branch‐and‐bound search for the optimal schedule. © 2004 Wiley Periodicals, Inc. Syst Comp Jpn, 35(5): 44–54, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10533

PARALLEL AND LARGE SCALE MATRIX COMPUTATIONS IN CONTROL: SOME IDEAS

Efficient parallel algorithms and software for compressed octrees with applications to hierarchical methods

The performance of the MPI’s collective communications is critical in most MPI-based applications. A general algorithm for a given collective communication operation may not give good performance on all systems due to the differences in architectures, network parameters and the buffering scheme of the underlying MPI implementation. In this paper, we discuss an approach in which the collective communications are tuned for any given system by conducting a series of experiments on the system. We also discuss a dynamic topology method that uses the tuned static topology shape, but re-orders the logical addresses to compensate for changing run time variations. A series of ex-periments were conducted comparing our tuned MPI_Bcast to various native vendor MPI implementations. The results obtained were encouraging, and show that our implementations of collective algorithms can significantly improve the performance of current MPI implementations.KeywordsBuffer SizeMessage SizeSegment SizeCollective CommunicationCollective OperationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Chapter 6 - Multiprocessor Machines

Graph algorithms are notorious for not getting good speedup on parallel architectures. These algorithms tend to suffer from irregular dependencies and a high synchronization cost that prevent an efficient execution on distributed memory machines. Hence such algorithms are mostly parallelized on shared memory machines. However, current commodity shared memory machines do not typically offer enough parallelism to process these problems. In this paper, we are presenting an early investigation of the scalability of such algorithms on Intel's upcoming Many Integrated Core (Intel MIC) architecture which, when it will be released in 2012, is expected to provide more than 50 physical cores with SMT capability. The Intel MIC architecture can be programmed through many programming models, here we investigate the three most popular of these models namely OpenMP, Cilk Plus and Intel's TBB. We present scalability results of a parallel graph coloring algorithm, three variations of a breadth-first search algorithm and a micro benchmark for irregular computations using these three programming models. Our results on a prototype board show that the multi-threaded architecture of Intel MIC can be effectively used for hiding latencies in irregular applications to achieve almost perfect speedup.

Parallel graph generation algorithms for shared and distributed memory machines

Among elliptic boundary value problems, the class of problems with separable variables can be solved fast and directly. Elliptic problems with separable variables usually assume that the computational domains are simple e.g., rectangle or circle, and constant coefficients [1]. This kind of problems can serve to generate preconditioners in iterative procedures for more complex methods. They can also be used in low-accuracy models similar to the ones used in Numerical Weather Simulations. A straightforward implementation of solvers for such problems makes them considered as too simple to pay much attention to them. For instance, NETLIB* contains the codes of various solvers for problems with separable variables. We are not aware if such solvers are provided as separate solvers in other software packages except NETLIB Fishpack* and Intel$\textcircled{\tiny{\,R}}$ Math Kernel Library (Intel$\textcircled{\tiny{\,R}}$ MKL) together with Intel$\textcircled{\tiny{\,R}}$ Cluster Poisson Solver Library (Intel$\textcircled{\tiny{\,R}}$ CPSL). The problems with the separable variables can be suboptimal in the sense that they require slightly more arithmetic operations (up to logarithmic factor) than the number of unknowns to compute the solution to the problem. This statement is true if, for example, the sizes of the discretized problems are powers of 2. Computational Mathematics suggests that we take into consideration not only arithmetic operations, but also the cost of memory operations as well. Modern computers can perform computations at a very high speed, while lacking the ability to deliver data to the computational units. Keeping in mind that a memory operation can easily be dozen to hundred times slower than an arithmetic one, a computationally optimal algorithm could compute the solution slower than memory optimal algorithm. The recent developments in processor industry resulted in multicore processors become standard processors not only for powerful clusters, but also for home computers and laptops. Therefore, the algorithm can also be non-optimal from the parallelization point of view. Optimality here can be understood in terms of the number of synchronization points and/or in terms of the amount of data that needs to be transferred between different cores/processors. In summary, the modern computational algorithm should focus on 3 key aspects, namely, parallelization, memory operations, and arithmetic costs. The purpose of this paper is to demonstrate on a simple 2D problem with separable variables that taking into account modern model the solution can be computed efficiently and fast. This would also help developers to compute solution to the problem with separable variables really negligible with respect to other computations. To complete our goal, we will evaluate software provided by Intel Corporation. In particular, we focus on the comparison (where possible) of NETLIB Fishpack* and Intel$\textcircled{\tiny{\,R}}$ CPSL provided at [2] as well as Intel$\textcircled{\tiny{\,R}}$ MKL provided at [3]. This paper is organized as follows. First we are solving a 2D Helmholtz problem using single precision on a shared memory machine with two 4-Core Intel$\textcircled{\tiny{\,R}}$ Xeon$\textcircled{\tiny{\,R}}$ Processors E5462. Next we do the same but use double precision arithmetic. Lastly we use double precision on a distributed memory machine using MPI.

Gradient boosting decision tree (GBDT) is one of the most popular machine learning models widely used in both academia and industry. Although GBDT has been widely supported by existing systems such as XGBoost, LightGBM, and MLlib, one system bottleneck appears when the dimensionality of the data becomes high. As a result, when we tried to support our industrial partner on datasets of the dimension up to 330K, we observed suboptimal performance for all these aforementioned systems. In this paper, we ask Can we build a scalable GBDT training system whose performance scales better with respect to dimensionality of the data? The first contribution of this paper is a careful investigation of existing systems by developing a performance model with respect to the dimensionality of the data. We find that the collective communication operations in many existing systems only implement the algorithm designed for small messages. By just fixing this problem, we are able to speed up these systems by up to 2X. Our second contribution is a series of optimizations to further optimize the performance of collective communications. These optimizations include a task scheduler, a two-phase split finding method, and low-precision gradient histograms. Our third contribution is a sparsity-aware algorithm to build gradient histograms and a novel index structure to build histograms in parallel. We implement these optimizations in DimBoost and show that it can be 2-9X faster than existing systems.

Distributed memory parallel machines are increasingly being used by the scientific community to provide computational power for solving problems. Building a distributed memory parallel machine offers a tremendous advantage in terms of cost and scalability over building a shared memory parallel machine. Unfortunately, distributed memory machines are much more difficult to program, mainly due to the lack of a single global address space. Communication between processors in a distributed memory machine is accomplished by message passing. The time cost for acquiring data from another processor's memory is significantly greater than the time cost of acquiring data from local memory. To assist the programmer in using a distributed memory machine, compilers have been developed which alleviate the need for the programmer to manage the lack of a global address space. These compilers will generate semantically correct code containing communication routines to produce a logically shared memory machine on a physically distributed memory machine. However, the runtime performance of the code emitted from some of these compilers can be further improved. This thesis is primarily concerned with advanced compilation techniques which will provide better runtime performance of programs written in imperative languages for regular scientific programs. The optimization of communication generation for distributed memory machines is used to produce this result. We look at two "state of the art" compilation systems and find that improvement in the communication code generated by these compilers is still possible. We have simplified existing descriptions of the various formalisms found in these compilers, while still providing enough formalism to study the problems in depth. We have extended some existing optimization techniques by enhancing the elimination of masks, by extending the notion of relaxing the owner computes rule, by extending the notion of replicating computation, and by extending the notion of replicating data. This thesis has discussed a novel technique to split loops so that they do not contain any guarded statements. Finally, we have suggested avenues for future research, as it is clear from this thesis that more work is required before compilation systems for distributed memory parallel machines will be widely used for general purpose computing.

The efficient parallel algorithms proposed for many fundamental problems, such as list ranking, computing preorder numberings on trees and integer sorting, are very sensitive to processor failures. The requirements of efficiency (commonly formalized using {\em Parallel-time x Processors} as a cost measure) has led to the design of highly tuned PRAM algorithms which, given the additional constraint of simple processor failures, unfortunately become inefficient or even incorrect. We propose a new notion of {\em robustness} that combines efficiency with fault tolerance. For the common case of fail-stop errors, we develop a general and easy-to-implement technique to make robust many efficient parallel algorithms, e.g., algorithms for all the problems listed above. More specifically,{\em for any dynamic pattern of fail-stop errors on} a CRCW PRAM {\em with at least one surviving processor}, our method increases the original algorithm''s cost by at most a {\em polylog} multiplicative factor. We also show that at least a {\em log/loglog} multiplicative overhead will be incurred for certain patterns of failures by any algorithm that implements robust solutions to the problems listed.

Nonblocking data structures are an essential part of many parallel applications in that they can help to improve fault tolerance and performance. Although there are scores of nonblocking data structures, such as stacks, queues, double-ended queues (deques), lists, widely used in practice, most of them are designed to be used on shared-memory machines only, and cannot be used in a distributed-memory setting. Several recent studies focus on the development of novel tailor-made distributed nonblocking data structures and omit the potential for adapting a great wealth of existing shared-memory nonblocking ones for the distributed-memory case. Hence, we propose a general approach to bridge the gap between most existing nonblocking data structures and distributed-memory machines in this work. Several challenges, such as safe memory reclamation and solving the ABA problem, must be overcome. In this paper, we present a global memory management scheme to address these issues. The scheme takes advantage of hazard pointers which are widely used to tackle the problems in shared-memory environments. To demonstrate our general approach, we take stacks as a typical example of nonblocking data structures. This work also provides a survey of well-known nonblocking stack algorithms along with our analysis and evaluation in distributed-memory environments. Moreover, this paper depicts how to improve performance of a stack algorithm by making use of node information.

In this paper, we present two new parallel formulations of the Barnes-Hut method. These parallel formulations are especially suited for simulations with irregular particle densities. We first present a parallel formulation that uses a static partitioning of the domain and assignment of subdomains to processors. We demonstrate that this scheme delivers acceptable load balance, and coupled with two collective communication operations, it yields good performance. We present a second parallel formulation which combines static decomposition of the domain with an assignment of subdomains to processors based on Morton ordering. This alleviates the load imbalance inherent in the first scheme. The second parallel formulation is inspired by two currently best known parallel algorithms for the Barnes-Hut method. We present an experimental evaluation of these schemes on a 256 processor nCUBE2 parallel computer for an astrophysical simulation.

A comparative study of MPI implementations on a cluster of SMP workstations