Simulating Spatially Explicit Problems on High Performance Architectures

Fine-Grained Parallel and Distributed Spatial Stochastic Simulation of Biological Reactions

In an optimistic (time-warp) parallel simulation, local clocks with different logical processes must advance at the same rate in order to reduce the number of rollbacks. In this paper, we propose two algorithms for dynamic load balancing which reduce the number of rollbacks in an optimistic parallel discrete event simulation (PDES) system. The first algorithm is based on the load transfer mechanism between logical processes, while the second algorithm, which is based on the principle of an evolutionary strategy, migrates logical processors between several pairs of physical processors. We have implemented both of these algorithms on a cluster of heterogeneous workstations and studied their performance. The experimental results show that the algorithm based on the load transfer is effective when the grain size is larger than 10 ms, and the algorithm based on the process migration yields good performance for grain sizes of 20 ms or larger. In both of these cases, the average speed-up ranges between 1 and 2 using four processors, when the computation grain-size is within the range 7 to 50 ms. The reduction in rollback messages as a percentage of the total number of messages due to the algorithms is, however, around 4 to 8%.

In this report we present parallel solvers for large linear systems arising from the finite-element discretization of the three-dimensional steady-state groundwater flow problem. Our solvers are based on multigrid and Krylov subspace methods. The parallel implementation is based on a domain decomposition strategy with explicit message passing using NX and MPI libraries. We have tested our parallel implementations on the Intel Paragon XP/S 150 supercomputer using up to 1024 parallel processors and on other parallel platforms such as SGI/Power Challenge Array, Cray/SGI Origin 2000, Convex Exemplar SPP-1200, and IBM SP using up to 64 processors. We show that multigrid can be a scalable algorithm on distributed memory machines. We demonstrate the effectiveness of parallel multigrid based solvers by solving problems requiring more than 70 million nodes in less than a minute. This is more than 25 times faster than the diagonal preconditioned conjugate gradient method which is one of the more popular methods for large sparse linear systems. Our results also show that multigrid as a stand alone solver works best for problems with smooth coefficients, but for rough coefficients it is best used as a preconditioner for a Krylov subspace method such as the conjugate gradient method. We show that even for extremely heterogeneous systems the multigrid pre-conditioned conjugate gradient method is at least 10 times faster than the diagonally preconditioned conjugate gradient method.

The difficulty of handling out-of-core data limits the potential of parallel machines and high-end supercomputers. Since writing an efficient out-of-core version of a program is a difficult task and since virtual memory systems do not perform well on scientific computations, we believe that there is a clear need for compiler-directed explicit I/O approach for out-of-core computations. In this paper, we present a compiler algorithm to optimize locality of disk accesses in out-of-core codes by choosing a good combination of file layouts on disks and loop transformations. The transformations change the access order of array data. Experimental results obtained on IBM SP-2 and Intel Paragon provide encouraging evidence that our approach is successful at optimizing programs which depend on disk-resident data in distributed-memory machines.

This report explicates the communication overlapping capabilities of three distributed-memory machines, SGI/Cray T3E, IBM SP-2 with wide nodes, and the ETL EM-X. Bitonic sorting and Fast Fourier Transform are selected for experiments. Various message sizes are used to determine when, where, how much and why the overlapping takes place. Experimental results with up to 64 processors indicated that the communication performance of EM-X is insensitive to various message sizes while SP-2 is the most sensitive. T3E stayed in between. The EM-X gave the highest communication overlapping capability while T3E did the lowest. The experimental results are compared with the analytical results based on LogP and LogGP communication models.

Parallel simulation is a technique to accelerate microarchitecture simulation of CMPs by exploiting the inherent parallelism of CMPs. In this paper, we explore the simulation paradigm of simulating each core of a target CMP in one thread and then spreading the threads across the hardware thread contexts of a host CMP. We start with cycle-by-cycle simulation and then relax the synchronization condition in various schemes, which we call slack simulations. In slack simulations, the Pthreads simulating different simulated cores do not synchronize after each simulated cycle, but rather they are given some slack. The slack is the difference in cycle between the simulated times of any two target cores. Small slacks, such as a few cycles, greatly improve the efficiency of parallel CMP simulations, with no or negligible simulation error. We have developed a simulation framework called SlackSim to experiment with various slack simulation schemes. Unlike previous attempts to parallelize multiprocessor simulations on distributed memory machines, SlackSim takes advantage of the efficient sharing of data in the host CMP architecture. We demonstrate the efficiency and accuracy of some well-known slack simulation schemes and of some new ones on SlackSim running on a state-of-the-art CMP platform.

Exploring the Role of Domain Partitioning on Efficiency of Parallel Distributed Hydrologic Model Simulations Spatially distributed hydrologic models of watersheds and river basins are data and computation intensive because of the combined nature of hydrodynamics, complex forcings and heterogeneous parameter fields. Application of these models at fine temporal and spatial resolutions, and on large problem domains, is facilitated by parallel computation on multi-processor clusters. Notably, the computation efficiency of parallel simulations is crucially determined by the efficiency with which data are divided-and-distributed in a multiprocessor environment and how the information is shared between processors. While numerous data partitioning algorithms exist and have been extensively studied in computer science literature, detailed elucidation of the role of hydrologic model structure on data partitioning has not been presented yet. In addition, the relative role of computational load balance and interprocessor communication on parallel computation efficiency of a hydrologic model is not known. Considering the unstructured domain discretization scheme used in PIHM hydrologic model as an example, the paper first presents a generic methodology for incorporating hydrologic factors in optimal domain partitioning algorithms. The partitions are then used to explore the isolated role of computation load balance and interprocessor communication on parallel efficiency. Results confirm that parallel simulations on partitions that minimize interprocessor communication and divide the computational load equally are the most efficient. More importantly, load balance between processors is observed to be a more sensitive control on parallel efficiency than minimization of interprocessor communication. Further analyses of the efficiency and scalability of the parallel code for different partitioning configurations reveal a direct correspondence between parallel efficiency and theoretical metrics such as load balance ratio and communication to computation ratio. Results indicate that theoretical metrics can be used for the selection of best partitions before computationally intensive parallel simulations are performed. The study serves as a proof-of-concept evaluation of the impact of computation and communication on the efficiency of parallelized distributed hydrologic models at multiple resolutions.

The implementation and performance of a parallel spatial direct numerical simulation (PSDNS) approach on the Intel iPSC/860 hypercube and IBM SP1 and SP2 parallel computers is documented. Spatially evolving disturbances associated with laminar-to-turbulent transition in boundary-layer flows are computed with the PSDNS code. The feasibility of using the PSDNS to perform transition studies on these computers is examined. The results indicate that PSNDS approach can effectively be parallelized on a distributed memory parallel machine by remapping the distributed data structure during the course of the calculation. The performance of the PSDNS indicates that 32 nodes on the IBM SP1 or SP2 are required to match a comparable computation on the CRAY C-90 single processor; the hypercube was found to be two times slower than the CRAY Y/MP for this application.

At first thought, one may not realize the need for a special issue of Parallel and Distributed Computing Practices focused on Cluster Computing when our community has already invested years of research and spent a wealth of resources on traditional Big Iron supercomputers, for instance, the SGI/Cray Origin and IBM SP. Academia, industry, and national laboratories are still using, and for the foreseeable future, will continue to use, supercomputers to solve both grand-challenge scale and high-throughput applications. Then why the interest and new popularity of research into clusters for high-performance parallel and distributed applications? Clusters incorporate the tremendous successes that are attributable to our parallel and distributed processing community. Over the next decade, clusters will span the entire range of high-performance computing platforms. Instead of being restricted solely to supercomputers for achieving high-performance computing, the spectrum of available platforms will spread out to Beowulf-class clusters, to superclusters, and on to supercomputers. Moving along this continuum represents increasing computational resources, which in turn allows the capability for solving larger classes of applications or instances of problems. Compute-bound applications on workstations often may take advantage of the extra horsepower provided by symmetric multiprocessors. Applications with even larger computational requirements and a high-degree of concurrency may use Beowulf clusters, that is, clusters that are home-built from mass-market, commercial off-the-shelf workstations and networks, for instance, PC's interconnected by Fast Ethernet. A supercluster may contain commodity SMP nodes interconnected with scalable, low-latency, high-bandwidth networks, and is integrated by a system vendor rather than the computational scientist. Applications with greater complexity, for example, grand challenges from the sciences, demand scalable systems with high-performance, interconnects, perhaps coupled with terascale I/O subsystems, and thus, are appropriate for these superclusters. Finally, supercomputers, the largest of the clusters, may provide the ultimate platform for the highest-fidelity models and simulations and data-intensive applications. At each step up this ladder of cluster architectures, the system cost increases approximately by an order of magnitude, but having the high-performance capability to solve the problem of interest may require an entry point at any of these levels, the Beowulf, or supercluster, or even supercomputer platform. Cluster computing with Beowulf- or superclusters, thus, may be one or two orders of magnitude less expensive than a supercomputer, and provide a cost-effective solution and capability that was not available on a workstation. But this cost comparison measures just the basic system price. What about the applications? As recently as a few years ago, the lack of a canonical high-performance architectural paradigm meant that migrating to a new computer system, even from the same vendor, typically required redesigning applications to efficiently use the high-performance system. Thus, the application must be redesigned with parallel algorithms and libraries that are optimized for each high-performance system. This process of porting, redesigning, optimizing, debugging, and analyzing, the application may in fact be prohibitively expensive, and certainly frustrating when the process is completed just in time to greet the next generation system! That was then, this is now. With the successful standardization of system-level interfaces, many of these issues no longer hinder our expedited use of these new, high-performance clusters. The message-passing interface (MPI) is a clear example of a standard that has contributed significantly to the success of cluster computing. Supercomputer vendors working together with the high-performance computing community embarked on a process to standardize the user-level interface and behavior for passing messages between concurrent tasks. Now, using ordinary sequential compilers and vendor-supplied or freely-available implementations of this message-passing interface, applications can be implemented efficiently on one supercomputer and ported with little or no effort to other parallel machines. The message-passing paradigm, thus, provides a reasonably good canonical high-performance model and methodology for implementing high-performance applications. An algorithm that exhibits efficient execution and scaling on one machine then is likely to perform well on another with little or no additional programming effort. Using this cluster computing paradigm, algorithms typically are optimized by minimizing the number of remote memory accesses (reducing the amount of messages passed), and carefully organizing the data placement to increase memory locality. Thus, high-performance algorithms and applications that have been optimized for supercomputers, using the same cluster paradigm, may be appropriate for Beowulf and superclusters. Cluster computing, realizing the importance of a standard high-performance programming environment that applies equally well to both ends of the architectural spectrum, highly leverages these and other successes from parallel and distributed processing research and development. This is in an exciting, new era for high-performance computing where the canonical architectural paradigm stretches from home-built Beowulf clusters, to superclusters, and on to supercomputers. But cluster computing faces some challenges ahead of it. Continued research is required for designing 1) scalable, high-performance algorithms, applications, and methodologies, that can tolerate the communication overheads, and adapt to hybrid, SMP, and heterogeneous, nodes; 2) high-performance computer architectures, scalable interconnection networks, and I/O subsystems; and 3) scalable systems software, compilers, and monitoring tools. Cluster computing has come of age, and it is high time for the parallel and distributed processing community to lay the groundwork for our own legacy of success stories. David A. Bader University of New Mexico

Null message algorithm (NMA) is one of the efficient conservative time management algorithms that use null messages to provide synchronization between the logical processes (LPs) in a parallel discrete event simulation (PDES) system. However, the performance of a PDES system could be severely degraded if a large number of null messages need to be generated by LPs to avoid deadlock. In this paper, we present a mathematical model based on the quantitative criteria specified in (Rizvi et al., 2006) to optimize the performance of NMA by reducing the null message traffic. Moreover, the proposed mathematical model can be used to approximate the optimal values of some critical parameters such as frequency of transmission, Lookahead (L) values, and the variance of null message elimination. In addition, the performance analysis of the proposed mathematical model incorporates both uniform and non-uniform distribution of L values across multiple output lines of an LP. Our simulation and numerical analysis suggest that an optimal NMA offers better scalability in PDES system if it is used with the proper selection of critical parameters.

Introduces and evaluates a new efficient dynamic load-balancing scheme for parallel molecular dynamics simulation on distributed memory machines. It decomposes a spatial domain of particles into disjoint parts, each of which corresponds with a processor and dynamically changes its shape to keep almost the same number of particles throughout simulation. In contrast to other similar schemes, ours requires no long-distance inter-processor communications but only those among adjacent processors (and, thus, little communication overhead), whereas it still guarantees fast reduction of load imbalance among the processors. It owes these advantages mainly to the following features. (1) The sufficiently correct global load information is effectively obtained with the stepwise propagation of appropriate information via nearest-neighbor communication. (2) In addition to the global load balancing, another load-balancing procedure is also invoked on each processor without global load information in order to suppress the rapid increase or decrease of loads. Thus, information from remote processors can provide reliable values even after a certain period of delay. To evaluate the effectiveness of our scheme, we have integrated our load balancer into the publicly-available NAMD simulation system, through replacing its built-in load-balancing component. Preliminary experiments on a cluster of workstations connected through Myrinet switches shows that it successfully reduces load imbalance and improves the simulation performance.

We present a system for functional parallel computing on distributed memory machines, which does dynamic load balancing and is totally symmetric. This means that every node can perform any computation and is client and server at the same time. Furthermore, the calling convention is similar to threads on shared memory machines to allow a uniform usage of parallel and distributed procedure calls. Finally the communication and computation is strictly separated for easy maintenance of the user program.

Parallel discrete-event simulation (PDES) is an important tool in the codesign of extreme-scale systems because PDES provides a cost-effective way to evaluate designs of high-performance computing systems. Optimistic synchronization algorithms for PDES, such as Time Warp, allow events to be processed without global synchronization among the processing elements. A rollback mechanism is provided when events are processed out of timestamp order. Although optimistic synchronization protocols enable the scalability of large-scale PDES, the performance of the simulations must be tuned to reduce the number of rollbacks and provide an improved simulation runtime. To enable efficient large-scale optimistic simulations, one has to gain insight into the factors that affect the rollback behavior and simulation performance. We developed a tool for ROSS model developers that gives them detailed metrics on the performance of their large-scale optimistic simulations at varying levels of simulation granularity. Model developers can use this information for parameter tuning of optimistic simulations in order to achieve better runtime and fewer rollbacks. In this work, we instrument the ROSS optimistic PDES framework to gather detailed statistics about the simulation engine. We have also developed an interactive visualization interface that uses the data collected by the ROSS instrumentation to understand the underlying behavior of the simulation engine. The interface connects real time to virtual time in the simulation and provides the ability to view simulation data at different granularities. We demonstrate the usefulness of our framework by performing a visual analysis of the dragonfly network topology model provided by the CODES simulation framework built on top of ROSS. The instrumentation needs to minimize overhead in order to accurately collect data about the simulation performance. To ensure that the instrumentation does not introduce unnecessary overhead, we perform a scaling study that compares instrumented ROSS simulations with their noninstrumented counterparts in order to determine the amount of perturbation when running at different simulation scales.

Cycle-based simulation at register-transfer- and gate level realized by a Levelized Compiled Code (LCC) technique represents a well established method for functional verification in processor design. Due to rapidly increasing design sizes, simulation acceleration is an important issue. We present a parallel LCC simulation system that has been developed to run on loosely-coupled systems. It comprises three parallel simulators and a complex model partitioning environment. An essential idea of our parallelization approach is to valuate circuit model partitions with respect to the expected parallel simulation run-time and to integrate corresponding cost functions into partitioning algorithms. This is done on the basis of a formal model of parallel LCC simulation. In this paper, we focus on the strong relation between model partitioning and the parallel simulation technique chosen. The components of the simulation system are outlined and experimental results with respect to simulation acceleration are given for IBM processor models of different size.

Particle-in-Cell (PIC) plasma simulation codes model the interaction of charged particles with surrounding electrostatic and magnetic fields. Its computational requirements made it to be classified as one of the grand-challenge problems facing the high performance community. In this paper we present the implementation of 1-D and 2-D electrostatic PIC codes in High Performance Fortran(HPF) on a IBM SP-2. HPF expands Fortran 90 with data distribution and alignment directives and data parallel statements. It is a powerful language for writing portable and high performance programs across many platforms. We used one of the most successful commerical HPF compilers currently available in the market and augmented the compiler's missing HPF functions with extrinsic routines when necessary. We obtained near linear speed-up in all of our test cases. The performance of the HPF programs is comparable to the native message passing implementations of the same codes on the SP-2.