Efficiently unifying parallel simulation techniques

Fine-Grained Parallel and Distributed Spatial Stochastic Simulation of Biological Reactions

Simulation is becoming an increasingly important technique for what-if analysis in the context of (real-time) decision making applications. Consequently, quick delivery of simulation outputs to end-users (or applications) is a core objective. One approach for high performance simulation consists of exploiting parallel techniques, where the simulation model is partitioned into objects (or logical processes), concurrently executing events on different CPUs and/or multiple CPU-cores. For this type of simulation systems,a further run-time improvement arose from the exploitation of event uncertainty, both in time and space, which has lead to more flexible synchronization protocols. Although this approach can provide significant performance gains, one drawback is the risk of less reliable simulation results due to potential bias induced by the mechanisms for resolving the uncertainty, which are sometimes exclusively targeted to run-time effectiveness. In this article we focus on space uncertain simulation events in optimistic parallel simulation and introduce a mechanism that, compared to previous approaches, allows trading-off execution speed vs reliability of simulation results. In other words, our target is the achievement of high performance while controlling, at the same time, the bias introduced by space uncertainty on the simulation output.

Three parallel discrete-event simulators-synchronous, conservative and optimistic-implemented on an Intel Paragon multicomputer are comparatively evaluated. Parallelism is achieved by model decomposition, distributing the simulation among a set of collaborative logical processes. The three simulators differ in the way those processes synchronize to obey causal restrictions in the simulation of events. Message passing network models are used to study these simulation alternatives. A set of experiments are carried out to understand how model parameters influence simulator performance. Experimental evidence leads to the conclusion that the optimistic simulator is not a viable tool for the analysis of this kind of models. The opposite conclusion applies to the other two: if the workload assigned to each logical process is above a certain threshold then the synchronization overhead is comparatively low and the simulators perform and scale well up to a large number of processors. The performance threshold is influenced by some parameters of the simulated model (size of the network, load level and message length), as well as by the number of processors used by the simulators.

Global Virtual Time (GVT) is the fundamental synchronization concept in optimistic simulations. It is defined as the earliest time tag within the set of unprocessed pending events in distributed simulation. A number of techniques for determining GVT have been proposed in recent years, each having their own intrinsic properties. However, most of these techniques either focus on specific types of simulation problems or assume specific hardware support. This paper specifically addresses the GVT problem in the context of the following area A new GVT algorithm, called SPEEDES GVT, has been developed in the Synchronous Parallel Environment for Emulation and Discrete-Event Simulation (SPEEDES) framework. The new algorithm runs periodically but does not disrupt event processing. It provides flow control by processing events risk-free while flushing out messages during the GVT computation. SPEEDES GVT is built from a set of global reduction operations that are easily implementable on any hardware system.

The authors address the problem of developing parallel simulation techniques to analyze stochastic Petri net (SPN) models. The approach of parallel simulation is to divide a general SPN spatially into several connected subnets. The rich and complex structure of Petri nets necessitates the development of an algorithm which can handle general forms of network partitions. The authors present an algorithm based on the time warp strategy for optimistic parallel simulation. The various subnetworks are simulated in parallel by several logical processes (LPs) which synchronize by rollbacks. Each LP simulates a subnetwork and advances its local simulation time as far as it can. When a token is needed by other subnetworks, a message with the local simulation time (called token time) will be sent to the proper LPs. When a simulation error is detected in a LP, such as receiving a message with a small token time, the LP will roll back to the simulation time indicated by the token time of the received message and then resimulate the firing process from that point on. >

최근 모바일 장치의 사용의 증가와 복잡한 응용 프로그램의 사용이 증가하면서 MPSoC의 사용이 증가하고 있다. 이러한 MPSoC의 성능을 향상시키기 위해 프로세서의 수가 늘어나고 있는 추세이다. 다수의 프로세서 구조에서 장점이 있는 분산 메모리 구조의 효율적인 데이터 전달하기 위해서 표준 MPI를 이용한다. 표준 MPI는 소프트웨어로 제공되지만, 하드웨어로 구현하면 보다 높은 성능을 얻을 수 있다. 하드웨어로 구현된 MPI의 메시지 전송 방식으로 기존의 동기 방식(Synchronous Mode), 준비 방식(Ready Mode), 버퍼 방식(Buffered Mode)과 이 방식들을 혼합한 형태인 표준 방식(Standard Mode)가 있다. 본 논문에는 기존의 MPI 하드웨어 유닛에서 사용되던 구조에 작은 크기의 데이터를 선별하여 버퍼 방식으로 전송함으로써 전송율을 극대화 하였다. 기존의 구조에서 사용된 3개의 큐(Queue)는 그대로 같은 기능을 하고, 본 논문에서 추가된 2개의 큐(작은 준비 큐와 작은 요청 큐)을 추가하여 임계점보다 작은 크기의 데이터에 대한 처리와 저장을 담당하도록 하여 성능을 향상하였다. 제안된 구조에서 임계점을 32byte로 제한하였을 때 임계점 이하의 데이터에서 20%의 성능 개선 효과를 볼 수 있었다. The use of MPSoC has been increasing because of a rise of use of mobile devices and complex applications. For improving the performance of MPSoC, number of processor has been increasing. Standard MPI is used for efficiently sending data in distributed memory architecture that has advantage in multi processor. Standard In this paper, we propose a scalable distributed memory system with a low cost hardware message passing interface(MPI). The proposed architecture improves transfer rate with buffered send for small size packet. Three queues, Ready Queue, Request Queue, and Reservation Queue, work as previous architecture, and two queues, Small Ready Queue and Small Request Queue, are added to send small size packet. When the critical point is set 8 bytes, the proposed architecture takes more than 2 times the performance improvement in the data that below the critical point.

Global virtual time (GVT) is used in parallel discrete event simulations to reclaim memory, commit output, detect termination, and handle errors. Mattern 's [I] has proposed G VT approximation with distributed termination detection algorithm. This algorithm works fine and gives optimal performance in terms of accurate GVT computation at the expense of slower execution rate. This slower execution rate results a high GVT latency. Due to the high GVT latency, the processors involve in communication remain idle during that period of time. As a result, the overall throughput of a discrete event parallel simulation system degrades significantly. Thus, the high G VT latency prevents the widespread use of this algorithm in discrete event parallel simulation system. However, if we could improve the latency of GVT computation, most of the discrete event parallel simulation system would likely take advantage of this technique in terms of accurate G VT computation. In this paper, we examine the potential use of tress and butterflies barriers with the Mattern's GVT structure using a ring. Simulation results demonstrate that the use of tree barriers with the Mattern's GVT structure can significantly improve the latency time and thus increase the overall throughput of the parallel simulation system. The performance measure adopted in this paper is the achievable latency for a fixed number of processors and the number of message transmission during the G VT computation.

Parallel discrete-event simulation (PDES) is an important tool in the codesign of extreme-scale systems because PDES provides a cost-effective way to evaluate designs of high-performance computing systems. Optimistic synchronization algorithms for PDES, such as Time Warp, allow events to be processed without global synchronization among the processing elements. A rollback mechanism is provided when events are processed out of timestamp order. Although optimistic synchronization protocols enable the scalability of large-scale PDES, the performance of the simulations must be tuned to reduce the number of rollbacks and provide an improved simulation runtime. To enable efficient large-scale optimistic simulations, one has to gain insight into the factors that affect the rollback behavior and simulation performance. We developed a tool for ROSS model developers that gives them detailed metrics on the performance of their large-scale optimistic simulations at varying levels of simulation granularity. Model developers can use this information for parameter tuning of optimistic simulations in order to achieve better runtime and fewer rollbacks. In this work, we instrument the ROSS optimistic PDES framework to gather detailed statistics about the simulation engine. We have also developed an interactive visualization interface that uses the data collected by the ROSS instrumentation to understand the underlying behavior of the simulation engine. The interface connects real time to virtual time in the simulation and provides the ability to view simulation data at different granularities. We demonstrate the usefulness of our framework by performing a visual analysis of the dragonfly network topology model provided by the CODES simulation framework built on top of ROSS. The instrumentation needs to minimize overhead in order to accurately collect data about the simulation performance. To ensure that the instrumentation does not introduce unnecessary overhead, we perform a scaling study that compares instrumented ROSS simulations with their noninstrumented counterparts in order to determine the amount of perturbation when running at different simulation scales.

The data distribution problem is very complex, because it involves trade-off decisions between minimizing communication and maximizing parallelism. A common approach towards solving this problem is to break the data mapping into two stages: an alignment stage and a distribution stage. The alignment stage attempts to increase parallelism, while the distribution stage attempts to decrease communication overhead. As opposed to previous approaches, we consider the alignment and distribution problems in a unified framework, and attempt to simultaneously maximize parallelism and minimize communication overhead. The problem becomes harder if dynamic remapping, multi-dimensional distributions, array replications and control flow are taken into account. This paper formulates the full data decomposition problem that addresses all these issues and presents a simple new algorithm to find the optimal solution of the dynamic data distribution problem, given the number of processors and a partitioning of the input program into phases. The algorithm runs efficiently for small search spaces (several hundreds of data distributions).

Global virtual time (GVT) is used in the Time Warp synchronization mechanism to perform irrevocable operations such as I/O and to reclaim storage. Most existing algorithms for computing GVT assume a message-passing programming model. Here, GVT computation is examined in the context of a shared-memory model. We observe that computation of GVT is much simpler in shared-memory multiprocessors because these machines normally guarantee that no two processors will observe a set of memory operations as occurring in different orders. Exploiting this fact, we propose an efficient, asynchronous, shared-memory GVT algorithm and prove its correctness. This algorithm does not require message acknowledgments, special GVT messages, or FIFO delivery of messages, and requires only a minimal number of shared variables and data structures. The algorithm only requires one round of interprocessor communication to compute GVT, in contrast to many message-based algorithms that require two. An efficient implementatin is described that eliminates the need for a processor to explicitly compute a local minimum for time warp systems using a lowest-timestamp-first scheduling policy in each processor. In addition, we propose a new mechanism called on-the-fly fossil collection that enables efficient storage reclamation for simulations containing large numbers, e.g., hundreds of thousand or even millions of simulator objects. On-the-fly fossil collection can be used in time warp systems executing on either shared-memory or message-based machines. Performance measurements of the GVT algorithm and the on-the-fly fossil collection mechanism on a Kendall Square Research KSR-2 machine demonstrate that these techniques enable frequent GVT and fossil collections, e.g., every millisecond, without incurring a significant performance penalty

In this paper, we investigate the performance of Parallel Discrete Event Simulation (PDES) on a cluster of many-core Intel KNL processors. Specifically, we analyze the impact of different Global Virtual Time (GVT) algorithms in this environment and contribute three significant results. First, we show that it is essential to isolate the thread performing MPI communications from the task of processing simulation events, otherwise the simulation is significantly imbalanced and performs poorly. This applies to both synchronous and asynchronous GVT algorithms. Second, we demonstrate that synchronous GVT algorithm based on barrier synchronization is a better choice for communication-dominated models, while asynchronous GVT based on Mattern's algorithm performs better for computation-dominated scenarios. Third, we propose Controlled Asynchronous GVT (CA-GVT) algorithm that selectively adds synchronization to Mattern-style GVT based on simulation conditions. We demonstrate that CA-GVT outperforms both barrier and Mattern's GVT and achieves about 8% performance improvement on mixed computation-communication models. This is a reasonable improvement for a simple modification to a GVT algorithm.

Recent studies investigated the performance of Parallel Discrete Event Simulation (PDES) on Intel Xeon Phi many-core processors, but generally reported underwhelming performance results, especially at high scales when all cores and thread contexts are fully loaded. While the lack of scalability in an earlier study on a Knights Corner (KC) processor is an artifact of physical limitations of the KC system, performance challenges on a Knights Landing (KNL) system partially stem from a slower global virtual time (GVT) computation algorithm used in that study. In this paper, we re-examine PDES performance on KNL under more efficient GVT algorithms to alleviate the GVT bottleneck. Specifically, we compare a synchronous GVT algorithm based on barrier synchronization, and two asynchronous GVT implementations: a modified Mattern's algorithm for shared memory systems and a recently-proposed wait-free algorithm. Using the ROSS simulator, we demonstrate that minimizing the GVT bottleneck results in significant improvement in scalability, allowing the simulation to scale with performance all the way to 250 threads (per chip). Interestingly, we observe that while for the balanced models the wait-free algorithm is a clear winner, barrier-based GVT provides significantly better results for imbalanced models executed at high scale. We also perform detailed simulation profiling to understand the underlying reasons for these performance trends.

Previous implementation of distributed Gauss-Newton (DGN) optimization algorithm runs multiple optimization threads in parallel, employing a synchronous running mode (S-DGN). As a result, it waits for all simulations submitted in each iteration to complete, which may significantly degrade performance because a few simulations may run much longer than others, especially for time-consuming real-field cases. To overcome this limitation and thus improve the DGN optimizer's execution, we propose two asynchronous DGN (A-DGN) optimization algorithms in this paper. The A-DGN optimizer is a well-parallelized and efficient derivative-free (DFO) method. The A-DGN optimizer generates multiple initial guesses by sampling from the prior probability distribution of uncertain parameters in the first iteration. It then runs multiple simulations on high-performance-computing (HPC) clusters in parallel. A checking time interval is introduced to control the optimization process. The A-DGN optimizer checks the status of all running simulations after every checking time frame. A new simulation case is proposed immediately once the simulation of an optimization thread is completed, without waiting for the completion of other simulations. Thus, each A-DGN optimization thread becomes independent. The two A-DGN optimization algorithms are 1) the local-search algorithm to locate multiple maximum-a-posteriori (MAP) estimates and 2) the integrated global-search algorithm with the randomized-maximum-likelihood (RML) method to generate hundreds of RML samples in parallel for uncertainty quantification. We modified the training-data data set updating algorithm using the iteration index for each thread to implement the asynchronous running mode. The sensitivity matrix at the best solution of each optimization thread is estimated by linear interpolation of a subset of the training data closest to the best solution, using the modified QR decomposition method. A new simulation case (or search point) is generated by solving the Gauss-Newton trust-region subproblem (GNTRS), together with the estimated sensitivity matrix, using the more efficient and robust GNTRS solver that we developed recently. The proposed A-DGN optimization method is tested and validated on a synthetic problem and then applied to a real-field deep-water reservoir model. Numerical tests confirm that the proposed A-DGN optimization method can converge to solutions with matching quality comparable to those obtained by the S-DGN optimizer, saving on the time required for the optimizer to converge by a factor ranging from 1.3 to 2 when compared to the S-DGN optimizer depending on the problem. The new A-DGN optimization algorithm presented in this paper helps improve efficiency and robustness in solving history-matching or inversion problems, especially for uncertainty quantification of subsurface model parameters and production forecasts of real-field reservoirs by conditioning to production data.

Global virtual time (GVT) is used in distributed simulations to reclaim memory, commit output, detect termination, and handle errors. It is a global function that is computed many times during the course of a simulation. A small GVT latency (delay between its occurrence and detection) allows for more efficient use of resources. We present an algorithm which minimizes the latency, and we prove its correctness. The algorithm is unique in that a target virtual time (TVT) is predetermined by an initiator who then detects when GVT ≥ TVT. This approach eliminates the avalanche effect because the collection phase is spread out over time, and it allows for regular and timely GVT updates. The algorithm does not require messages to be acknowledged, which significantly reduces the message overhead of the simulation. One possible application is with interactive simulators, where regular and timely updates would produce output that is up to date and appears smooth.

We investigate a slow-wave THz free-electron laser consisting of axially or helically corrugated waveguides driven by an annular electron beam. When the device is driven by a 10-mA electron beam at 30 keV, we numerically demonstrated backward-wave oscillation at 0.2212 and 0.2651 THz in axially and helically corrugated waveguides, respectively, without a solenoid field within a submillimeter aperture. We developed a unified theoretical framework for the two types of waveguides and successfully explained the observed backward-wave resonances. In our theory, azimuthal-symmetry breaking in a helically corrugated waveguide couples the synchronous transverse-magnetic radiation mode to a transverse-electric mode, resulting in radiation loss for the oscillation. Therefore, this study concludes superior radiation performance for the axially corrugated waveguide as a slow-wave THz free-electron laser. Furthermore, we show the opportunity of exciting self-started cyclotron resonance maser oscillations at THz frequencies from a helically corrugated waveguide without a solenoid field. Published by the American Physical Society 2025