Comment on gmd-2021-382

Abstract. Lagrangian models are fundamental tools to study atmospheric transport processes and for practical applications such as dispersion modeling for anthropogenic and natural emission sources. However, conducting large-scale Lagrangian transport simulations with millions of air parcels or more can become rather numerically costly. In this study, we assessed the potential of exploiting graphics processing units (GPUs) to accelerate Lagrangian transport simulations. We ported the Massive-Parallel Trajectory Calculations (MPTRAC) model to GPUs using the open accelerator (OpenACC) programming model. The trajectory calculations conducted within the MPTRAC model were fully ported to GPUs, i.e., except for feeding in the meteorological input data and for extracting the particle output data, the code operates entirely on the GPU devices without frequent data transfers between CPU and GPU memory. Model verification, performance analyses, and scaling tests of the Message Passing Interface (MPI) – Open Multi-Processing (OpenMP) – OpenACC hybrid parallelization of MPTRAC were conducted on the Jülich Wizard for European Leadership Science (JUWELS) Booster supercomputer operated by the Jülich Supercomputing Centre, Germany. The JUWELS Booster comprises 3744 NVIDIA A100 Tensor Core GPUs, providing a peak performance of 71.0 PFlop s−1. As of June 2021, it is the most powerful supercomputer in Europe and listed among the most energy-efficient systems internationally. For large-scale simulations comprising 108 particles driven by the European Centre for Medium-Range Weather Forecasts' fifth-generation reanalysis (ERA5), the performance evaluation showed a maximum speed-up of a factor of 16 due to the utilization of GPUs compared to CPU-only runs on the JUWELS Booster. In the large-scale GPU run, about 67 % of the runtime is spent on the physics calculations, conducted on the GPUs. Another 15 % of the runtime is required for file I/O, mostly to read the large ERA5 data set from disk. Meteorological data preprocessing on the CPUs also requires about 15 % of the runtime. Although this study identified potential for further improvements of the GPU code, we consider the MPTRAC model ready for production runs on the JUWELS Booster in its present form. The GPU code provides a much faster time to solution than the CPU code, which is particularly relevant for near-real-time applications of a Lagrangian transport model.

<p>The European Centre for Medium-Range Weather Forecasts’ (ECMWF’s) next-generation reanalysis ERA5 provides many improvements, but it also confronts the community with a "big data" challenge. Data storage requirements for ERA5 increase by a factor of ∼80 compared with the ERA-Interim reanalysis, introduced a decade ago. Considering the significant increase in resources required for working with the new ERA5 data set, it is important to assess its impact on Lagrangian transport simulations. To quantify the differences between transport simulations using ERA5 and ERA-Interim data, we analyzed comprehensive global sets of 10-day forward trajectories for the free troposphere and the stratosphere for the year 2017. The new ERA5 data have a considerable impact on the simulations. Spatial transport deviations between ERA5 and ERA-Interim trajectories are up to an order of magnitude larger than those caused by parameterized diffusion and subgrid-scale wind fluctuations after 1 day and still up to a factor of 2–3 larger after 10 days. Depending on the height range, the spatial differences between the trajectories map into deviations as large as 3 K in temperature, 30 % in specific humidity, 1.8 % in potential temperature, and 50 % in potential vorticity after 1 day. Part of the differences between ERA5 and ERA-Interim is attributed to the better spatial and temporal resolution of the ERA5 reanalysis, which allows for a better representation of convective updrafts, gravity waves, tropical cyclones, and other meso- to synoptic-scale features of the atmosphere. Another important finding is that ERA5 trajectories exhibit significantly improved conservation of potential temperature in the stratosphere, pointing to an improved consistency of ECMWF’s forecast model and observations that leads to smaller data assimilation increments. We conducted a number of downsampling experiments with the ERA5 data, in which we reduced the numbers of meteorological time steps, vertical levels, and horizontal grid points. Significant differences remain present in the transport simulations, if we downsample the ERA5 data to a resolution similar to ERA-Interim. This points to substantial changes of the forecast model, observations, and assimilation system of ERA5 in addition to improved resolution. A comparison of two Lagrangian trajectory models allowed us to assess the readiness of the codes and workflows to handle the comprehensive ERA5 data and to demonstrate the consistency of the simulation results. Our results will help to guide future Lagrangian transport studies attempting to navigate the increased computational complexity and leverage the considerable benefits and improvements of ECMWF’s new ERA5 data set.</p><p>Reference: Hoffmann, L., Günther, G., Li, D., Stein, O., Wu, X., Griessbach, S., Heng, Y., Konopka, P., Müller, R., Vogel, B., and Wright, J. S.: From ERA-Interim to ERA5: the considerable impact of ECMWF's next-generation reanalysis on Lagrangian transport simulations, Atmos. Chem. Phys., 19, 3097–3124, https://doi.org/10.5194/acp-19-3097-2019, 2019.</p><p> </p>

There are two inherent obstacles to effectively using Graphics Processing Units (GPUs) for query processing in search engines: (a) the highly restricted GPU memory space, and (b) the CPU-GPU transfer latency. Previously, Ao et al. presented a GPU method for lists intersection, an essential component in AND-based query processing. However, this work assumes the whole inverted index can be stored in GPU memory and does not address document ranking. In this paper, we describe and analyze a GPU query processing method which incorporates both lists intersection and top-k ranking. We introduce a parameterized pruned posting list GPU caching method where the parameter determines how much GPU memory is used for caching. This method allows list caching for large inverted indexes using the limited GPU memory, thereby making a qualitative improvement over previous work. We also give a mathematical model which can identify an approximately optimal choice of the parameter. Experimental results indicate that this GPU approach under the pruned list caching policy achieves better query throughput than its CPU counterpart, even when the inverted index size is much larger than the GPU memory space.

The graphics processing unit (GPU) is extensively used in diverse domains, such as finance, machine learning, and image processing. The GPU can be underutilized as multiple applications may not share the same GPU concurrently owing to a memory oversubscription issue. For example, when applications that require fewer computational resources but a larger GPU memory are running instantaneously, the GPU memory may be insufficient; consequently, the number of GPU applications running simultaneously is restricted, decreasing GPU utilization. Further, it can even stop the execution of applications that are running on the GPU. To this end, we propose FlexGPU, which schedules the kernels of the GPU applications that run on the same GPU according to their features. This framework 1) schedules the kernel at the launching time according to its features to improve GPU utilization and 2) temporarily checkpoints and restores non-dependent content in the GPU memory to/from the host memory, which avoids oversubscription of the GPU when out-of-memory failure occurs and allows more kernels to run concurrently on the GPU. The experimental results show that compared to existing methods, our approach demonstrates a 7 times improvement in performance in terms of execution time and enables a 2.5 times increase in the concurrent execution of applications.

Multi-phase SPH modelling of violent hydrodynamics on GPUs

Purpose: To develop a Graphics Processing Unit (GPU) based Monte Carlo (MC) code that uses a dual‐GPU system to accelerate radiographic simulation and CT imaging dose calculations. Methods: We considered two clinical cases, a chest x‐ray radiography and an abdominal CT scan. In the first case, a voxelized VIP‐Man phantom with detailed 3D anatomical information was used and an x‐ray beam of 120kVp was simulated. In the second case, a voxelized abdomen phantom derived from 120 CT slices was used, and a GE LightSpeed 16‐MDCT scanner was modeled. The CPU version of the MC code was written in C++ and run on Intel Xeon X5660 2.8GHz CPU, then translated into GPU code written in CUDA C and tested on a dual Tesla m2090 GPU system. The code was featured with automatic assignment of simulation task to multiple GPUs, as well as accurate calculation of energy‐ and material‐dependent cross‐sections. Results: Double‐precision floating point format was used for accuracy. In the first case, radiograph formation was simulated and doses to the organs listed in ICRP‐60 were calculated. When running on a single GPU, the MC GPU code was found to be x13 times faster than the CPU code and x29 times faster than MCNPX. In the second case, doses to the rectum, prostate, bladder and femoral heads were calculated. A speedup of x19 was observed compared to CPU code. These speedup factors were doubled on the dual‐GPU system. The imaging dose was benchmarked against MCNPX and a maximum difference of 1% was observed when the relative error is kept below 0.1%. Conclusions: A GPU‐based MC code was developed to simulate radiography and calculate imaging dose using detailed patient and CT scanner models. Efficiency and accuracy were both guaranteed in this code. Scalability of the code was confirmed on the dual‐GPU system.

Accelerating the discontinuous Galerkin method for seismic wave propagation simulations using the graphic processing unit (GPU)—single-GPU implementation

<p>Sulfur dioxide (SO<sub>2</sub>) is one of the most abundant gases released by volcanic eruptions. It is of concern due to its potential to influence climate on the global scale. Next to advection and diffusion, various chemical and geophysical processes need to be considered to properly represent volcanic SO<sub>2</sub> plumes in Lagrangian transport simulations. The chemical reaction of SO<sub>2</sub> with the hydroxyl radical (OH) and wet deposition are major processes depleting SO<sub>2</sub> from volcanic plumes. In this study, we implemented and revised new modules for OH chemistry and wet deposition in the Massive Parallel Trajectory Calculations (MPTRAC) Lagrangian transport model. In MPTRAC, the OH chemistry module adopts the newest JPL data evaluation to calculate the temperature- and pressure-dependent reaction coefficient of the termolecular reaction of SO<sub>2</sub> and OH. Similar to other Lagrangian chemistry transport models, monthly mean zonal mean OH fields are applied in the MPTRAC simulations. However, here we propose to introduce a correction factor depending on the solar zenith angle in order to represent the diurnal variations of the OH concentrations. The wet deposition module of MPTRAC mainly uses cloud ice and liquid water content retrieved from ECMWF’s meteorological data as well as an effective Henry constant, considering both, the dissociation and dissolution of SO<sub>2</sub> in cloud water. The revised and improved MPTRAC model is evaluated in the case study on the July 2018 eruption of Ambae, Vanuatu, the most voluminous volcano of the New Hebrides archipelago. It is reported that during the eruption, the island of Ambae suffered from acid rain, which means that the volcanic plume encountered clouds and significant wet deposition of SO<sub>2</sub> due to precipitation. A series of sensitivity tests was conducted to assess the two new chemistry modules of MPTRAC. An inspection of the time series of SO<sub>2</sub> total mass revealed that the diurnal variations due to the OH chemistry are now properly represented in the model. The trajectories of the SO<sub>2</sub> parcels have large influence on the wet deposition because they determine whether they pass through cloudy regions or not at a certain time and location. Therefore, it is important to use the most accurate meteorological data and source information. With the revised MPTRAC model, we found that a significant part of the loss of SO<sub>2</sub> total mass from the Ambae eruption is represented in the simulations. However, satellite observations reveal even shorter tropospheric lifetimes of the volcanic SO<sub>2</sub> plume from the Ambae eruption, which indicates that future work needs to focus on including additional chemical and geophysical processes in the simulations.</p>

In data-parallel computing that uses a graphic processing unit (GPU), processing of large data requires that multiple GPUs be used in the computer to increase its execution performance. Increasing processing performance by using multiple computing resources has been enabled by the development of computing frameworks based on a directed acyclic graph (DAG). However, their performance degrades in out-of-core processing, which often occurs in processing of large data on GPUs with limited memory capacity. The GPU data input/output (I/O) for data swapping between host memory and GPU memory during the execution of a user DAG is usually a performance bottleneck. A computing framework called is proposed to overcome this drawback. It uses a novel scheduler that involves two methods to minimize the total amount of GPU data I/O of data swapping. First, it performs locality-aware scheduling. When it schedules a task, it selects one that requires the minimum amount of data swapping and reuses as much of the data residing in GPU memory as possible. Second, it extends the locality-aware scheduling so that GPUs can execute data prefetching. Prefetching data that are swapped out from a GPU enables efficient use of bottleneck GPU I/O resources. To prefetch the input data of future tasks, it is required to determine the schedule of future tasks. Victream's scheduler (hereafter, the Victream scheduler) extends the locality-aware scheduling so that it can schedule future tasks to enable data prefetching that is executed in the way that minimizes the amount of data I/O of data swapping. Evaluation of a Victream prototype showed that the performance of Victream is better than that of conventional frameworks by up to 117%.

Feedback control has become a crucial tool in the research on magnetic confinement of plasmas for achieving controlled nuclear fusion. This thesis presents a novel plasma feedback control system that, for the first time, employs a Graphics Processing Unit (GPU) for microsecond-latency, real-time control computations. This novel application area for GPU computing is opened up by a new system architecture that is optimized for low-latency computations on less than kilobyte sized data samples as they occur in typical plasma control algorithms. In contrast to traditional GPU computing approaches that target complex, high-throughput computations with massive amounts of data, the architecture presented in this thesis uses the GPU as the primary processing unit rather than as an auxiliary of the CPU, and data is transferred from A-D/D-A converters directly into GPU memory using peer-to-peer PCI Express transfers. The described design has been implemented in a new, GPU-based control system for the High-Beta Tokamak – Extended Pulse (HBT-EP) device. The system is built from commodity hardware and uses an NVIDIA GeForce GPU and D-TACQ A-D/D-A converters providing a total of 96 input and 64 output channels. The system is able to run with sampling periods down to 4 μs and latencies down to 8 μs. The GPU provides a total processing power of 1.5 x 1012 floating point operations per second. To illustrate the performance and versatility of both the general architecture and concrete implementation, a new control algorithm has been developed. The algorithm is designed for the control of multiple rotating magnetic perturbations in situations where the plasma equilibrium is not known exactly and features an adaptive system model: instead of requiring the rotation frequencies and growth rates embedded in the system model to be set a priori, the adaptive algorithm derives these parameters from the evolution of the perturbation amplitudes themselves. This results in non-linear control computations with high computational demands, but is handled easily by the GPU based system. Both digital processing latency and an arbitrary multi-pole response of amplifiers and control coils is fully taken into account for the generation of control signals. To separate sensor signals into perturbed and equilibrium components without knowledge of the equilibrium fields, a new separation method based on biorthogonal decomposition is introduced and used to derive a filter that performs the separation in real-time. The control algorithm has been implemented and tested on the new, GPU-based feedback control system of the HBT-EP tokamak. In this instance, the algorithm was set up to control four rotating n = 1 perturbations at different poloidal angles. The perturbations were treated as coupled in frequency but independent in amplitude and phase, so that the system effectively controls a helical n = 1 perturbation with unknown poloidal spectrum. Depending on the plasma's edge safety factor and rotation frequency, the control system is shown to be able to suppress the amplitude of the dominant 8 kHz mode by up to 60% or amplify the saturated amplitude by a factor of up to two. Intermediate feedback phases combine suppression and amplification with a speed up or slow down of the mode rotation frequency. Increasing feedback gain results in the excitation of an additional, slowly rotating 1.4 kHz mode without further effects on the 8 kHz mode. The feedback performance is found to exceed previous results obtained with an FPGA- and Kalman-filter based control system without requiring any tuning of system model parameters. Experimental results are compared with simulations based on a combination of the Boozer surface current model and the Fitzpatrick-Aydemir model. Within the subset of phenomena that can be represented by the model as well as determined experimentally, qualitative agreement is found.

SummaryAn OpenACC directive‐based graphics processing unit (GPU) parallel scheme is presented for solving the compressible Navier–Stokes equations on 3D hybrid unstructured grids with a third‐order reconstructed discontinuous Galerkin method. The developed scheme requires the minimum code intrusion and algorithm alteration for upgrading a legacy solver with the GPU computing capability at very little extra effort in programming, which leads to a unified and portable code development strategy. A face coloring algorithm is adopted to eliminate the memory contention because of the threading of internal and boundary face integrals. A number of flow problems are presented to verify the implementation of the developed scheme. Timing measurements were obtained by running the resulting GPU code on one Nvidia Tesla K20c GPU card (Nvidia Corporation, Santa Clara, CA, USA) and compared with those obtained by running the equivalent Message Passing Interface (MPI) parallel CPU code on a compute node (consisting of two AMD Opteron 6128 eight‐core CPUs (Advanced Micro Devices, Inc., Sunnyvale, CA, USA)). Speedup factors of up to 24× and 1.6× for the GPU code were achieved with respect to one and 16 CPU cores, respectively. The numerical results indicate that this OpenACC‐based parallel scheme is an effective and extensible approach to port unstructured high‐order CFD solvers to GPU computing. Copyright © 2015 John Wiley & Sons, Ltd.

Nowadays, Graphics Processing Unit (GPU) is essential for general-purpose high-performance computing, because of its dominant performance in parallel computing compare to that of CPU. There have been many successful trials on the use of GPU in virtualized environment. Especially, NVIDIA Docker obtained a most practical way to bring GPU into the container-based virtualized environment. However, most of these trials did not consider sharing GPU among multiple containers. Without the above consideration, a system will experience a program failure or a deadlock situation in the worst case. In this paper, we propose ConVGPU, a solution to share the GPU in multiple containers. With ConVGPU, the system can guarantee the required GPU memory which the container needs to execute. To achieve it, we introduce four scheduling algorithms that manage the GPU memory to be taken by the containers. These algorithms can prevent the system from falling into deadlock situations between containers during execution.

Graphics processing units (GPUs) have been widely adopted to accelerate the training of deep neural networks (DNNs). Although the computational performance of GPUs has been improving steadily, the memory size of modern GPUs is still quite limited, which restricts the sizes of DNNs that can be trained on GPUs, and hence raises serious challenges. This paper introduces a framework, referred to as moDNN (memory optimal DNN training on GPUs), to optimize the memory usage in DNN training. moDNN supports automatic tuning of DNN training code to match any given memory budget (not smaller than the theoretical lower bound). By taking full advantage of overlapping computations and data transfers, we develop new heuristics to judiciously schedule data offloading and prefetching transfers, together with convolution algorithm selection, to optimize memory usage. We further devise a new sub-batch size selection method which also greatly reduces memory usage. moDNN can save memory usage up to 59×, compared with an ideal case which assumes that the GPU memory is sufficient to hold all data. When executing moDNN on a GPU with 12 GB memory, the training time is increased by only 3 percent, which is much shorter than that incurred by the best known approach, vDNN. Furthermore, we propose an optimization strategy for moDNN on multiple GPUs again by utilizing the idea of overlapping data transfers and GPU computations. The results show that 3.7× speedup is attained on four GPUs.

Graphics Processing Units (GPUs) have been successfully used to accelerate scientific applications due to their computation power and the availability of programming languages that make more approachable writing scientific applications for GPUs. However, since the programming model of GPUs requires offloading all the data to the GPU memory, the memory footprint of the application is limited to the size of the GPU memory. Multi-GPU systems can make memory limited problems tractable by parallelizing the computation and data among the available GPUs. Parallelizing applications written for running on single-GPU systems can be done (i) at runtime through an environment that captures the memory operations and kernel calls and distributes among the available GPUs, and (ii) at compile time through a pre-compiler that transforms the application for decomposing the data and computation among the available GPUs. In this paper we propose a framework and implement a tool that transforms an OpenCL application written to run on single-GPU systems into one that runs on multi-GPU systems. Based on data dependencies and data usage analysis, the application is transformed to decompose data and computation among the available GPUs. To reduce the data transfer overhead, computation-communication overlapping techniques are utilized. We tested our tool using two applications with different data transfer requirements, for the application with no data transfer requirements, a linear speedup is achieved, while for the application with data transfers, the computation-communication overlapping reduces the communication overhead by 40%.

Graphics processing units (GPUs) are important components of modern computing devices for not only graphics rendering, but also efficient parallel computations. However, their security problems are ignored despite their importance and popularity. In this paper, we first perform an in-depth security analysis on GPUs to detect security vulnerabilities. We observe that contemporary, widely-used GPUs, both NVIDIA's and AMD's, do not initialize newly allocated GPU memory pages which may contain sensitive user data. By exploiting such vulnerabilities, we propose attack methods for revealing a victim program's data kept in GPU memory both during its execution and right after its termination. We further show the high applicability of the proposed attacks by applying them to the Chromium and Firefox web browsers which use GPUs for accelerating webpage rendering. We detect that both browsers leave rendered webpage textures in GPU memory, so that we can infer which web pages a victim user has visited by analyzing the remaining textures. The accuracy of our advanced inference attack that uses both pixel sequence matching and RGB histogram matching is up to 95.4%.