Experience Report: Writing a Portable GPU Runtime with OpenMP 5.1

The Monte Carlo radiation transport method is considered the most accurate approach for absorbed dose calculations in external beam radiation therapy. In this study, an efficient and accurate source model of the Varian TrueBeam 6X STx Linac is developed and integrated with a fast Monte Carlo photon-electron transport absorbed dose engine, ARCHER-RT, which is capable of being executed on CPUs, NVIDIA GPUs, and AMD GPUs. This capability of fast yet accurate radiation dose calculation is essential for clinical utility of this new technology. This paper describes the software and algorithmic developments made to the ARCHER-RT absorbed dose engine. AMD's Heterogeneous-Compute Interface for Portability (HIP) was implemented in ARCHER-RT to allow for device independent execution on NVIDIA and AMD GPUs. Architecture-specific atomic-add algorithms have been identified and both more accurate single-precision and double-precision computational absorbed dose calculation methods have been added to ARCHER-RT and validated through a test case to evaluate the accuracy and performance of the algorithms. The validity of the source model and the radiation transport physics were benchmarked against Monte Carlo simulations performed with EGSnrc. Secondary dose-check physics plans, and a clinical prostate treatment plan were calculated to demonstrate the applicability of the platform for clinical use. Absorbed dose difference maps and gamma analyses were conducted to establish the accuracy and consistency between the two Monte Carlo models. Timing studies were conducted on a CPU, an NVIDIA GPU, and an AMD GPU to evaluate the computational speed of ARCHER-RT. Percent depth doses were computed for different field sizes ranging from 1.5cm2 ×1.5cm2 to 22cm2 ×40cm2 and the two codes agreed for all points outside high gradient regions within 3%. Axial profiles computed for a 10cm2 ×10cm2 field for multiple depths agreed for all points outside high gradient regions within 2%. The test case investigating the impact of native single-precision compared to double-precision showed differences in voxels as large as 71.47% and the implementation of KAS single-precision reduced the difference to less than 0.01%. The 3%/3mm gamma pass rates for an MPPG5a multileaf collimator (MLC) test case and a clinical VMAT prostate plan were 94.2% and 98.4% respectively. Timing studies demonstrated the calculation of a VMAT plan was completed in 50.3, 187.9, and 216.8s on an NVIDIA GPU, AMD GPU, and Intel CPU, respectively. ARCHER-RT is capable of patient-specific VMAT external beam photon absorbed dose calculations and its potential has been demonstrated by benchmarking against a well validated EGSnrc model of a Varian TrueBeam. Additionally, the implementation of AMD's HIP has shown the flexibility of the ARCHER-RT platform for device independent calculations. This work demonstrates the significant addition of functionality added to ARCHER-RT framework which has marked utility for both research and clinical applications and demonstrates further that Monte Carlo-based absorbed dose engines like ARCHER-RT have the potential for widespread clinical implementation.

In this paper, we show that OpenMP 4.5 based implementation of TestSNAP, a proxy-app for the Spectral Neighbor Analysis Potential (SNAP) in LAMMPS, can be ported across the NVIDIA, Intel, and AMD GPUs. Roofline analysis is employed to assess the performance of TestSNAP on each of the architectures. The main contributions of this paper are two-fold: 1) Provide OpenMP as a viable option for application portability across multiple GPU architectures, and 2) provide a methodology based on the roofline analysis to determine the performance portability of OpenMP implementations on the target architectures. The GPUs used for this work are Intel Gen9, AMD Radeon Instinct MI60, and NVIDIA Volta V100.

A merge tree is a topological descriptor of a real-valued function. Merge trees are used in visualization and topological data analysis, either directly or as a means to another end: computing a 0-dimensional persistence diagram, identifying connected components, performing topological simplification, etc. Scientific computing relies more and more on GPUs to achieve fast, scalable computation. For efficiency, data analysis should take place at the same location as the main computation, which motivates interest in parallel algorithms and portable software for merge trees that can run not only on a CPU, but also on a GPU, or other types of accelerators. The SYCL standard defines a programming model that allows the same code, written in standard C++, to compile targets for multiple parallel backends (CPUs via OpenMP or TBB, NVIDIA GPUs via CUDA, AMD GPUs via ROCm, Intel GPUs via Level Zero, FPGAs). In this paper, we adapt the triplet merge tree algorithm to SYCL and compare our implementation with the VTK-m implementation, which is the only other implementation of merge trees for GPUs that we know of.

To attain scalable performance efficiently, the HPC community expects future exascale systems to consist of multiple nodes, each with different types of hardware accelerators. In addition to GPUs and Intel MICs, additional candidate accelerators include embedded multiprocessors and FPGAs. End users need appropriate tools to efficiently use the available compute resources in such systems, both within a compute node and across compute nodes. As such, we present MetaMorph, a library framework designed to (automatically) extract as much computational capability as possible from HPC systems. Its design centers around three core principles: abstraction, interoperability, and adaptivity. To demonstrate its efficacy, we present a case study that uses the structured grids design pattern, which is heavily used in computational fluid dynamics. We show how MetaMorph significantly reduces the development time, while delivering performance and interoperability across an array of heterogeneous devices, including multicore CPUs, Intel MICs, AMD GPUs, and NVIDIA GPUs.

In-place data manipulation is very desirable in many-core architectures with limited on-board memory. This paper deals with the in-place implementation of a class of primitives that perform data movements in one direction. We call these primitives Data Sliding (DS) algorithms. Notable among them are relational algebra primitives (such as select and unique), padding to insert empty elements in a data structure, and stream compaction to reduce memory requirements. Their in-place implementation in a bulk synchronous parallel model, such as GPUs, is specially challenging due to the difficulties in synchronizing threads executing on different compute units. Using a novel adjacent work-group synchronization technique, we propose two algorithmic schemes for regular and irregular DS algorithms. With a set of 5 benchmarks, we validate our approaches and compare them to the state-of-the-art implementations of these benchmarks. Our regular DS algorithms demonstrate up to 9.11x and 73.25x on NVIDIA and AMD GPUs, respectively, the throughput of their competitors. Our irregular DS algorithms outperform NVIDIA Thrust library by up to 3.24x on the three most recent generations of NVIDIA GPUs.

Modern High Performance Computing (HPC) sys-tems are built with innovative system architectures and novel programming models to further push the speed limit of computing. The increased complexity poses challenges for performance portability and performance evaluation. The Standard Perfor-mance Evaluation Corporation (SPEC) has a long history of producing industry-standard benchmarks for modern computer systems. SPEC's newly released SPEChpc 2021 benchmark suites, developed by the High Performance Group, are a bold attempt to provide a fair and objective benchmarking tool designed for state-of-the-art HPC systems. With the support of multiple host and accelerator programming models, the suites are portable across both homogeneous and heterogeneous architectures. Different workloads are developed to fit system sizes ranging from a few compute nodes to a few hundred compute nodes. In this work we present our first experiences in performance benchmarking the new SPEChpc2021 suites and evaluate their portability and basic performance characteristics on various popular and emerging HPC architectures, including x86 CPU, NVIDIA GPU, and AMD GPU. This study provides a first-hand experience of executing the SPEChpc 2021 suites at scale on production HPC systems, discusses real-world use cases, and serves as an initial guideline for using the benchmark suites.

Efficiently processing sparse matrices is a central and performance-critical part of many scientific simulation codes. Recognizing the adoption of manycore accelerators in HPC, we evaluate in this paper the performance of the currently best sparse matrix-vector product (SpMV) implementations on high-end GPUs from AMD and NVIDIA. Specifically, we optimize SpMV kernels for the CSR, COO, ELL, and HYB format taking the hardware characteristics of the latest GPU technologies into account. We compare for 2,800 test matrices the performance of our kernels against AMD’s hipSPARSE library and NVIDIA’s cuSPARSE library, and ultimately assess how the GPU technologies from AMD and NVIDIA compare in terms of SpMV performance.

Portability is critical to ensuring high productivity in developing and maintaining scientific software as the diversity in on-node hardware architectures increases. While several programming models provide portability for diverse GPU systems, they don't make any guarantees about performance portability. In this work, we explore several programming models -- CUDA, HIP, Kokkos, RAJA, OpenMP, OpenACC, and SYCL, to assess the consistency of their performance across NVIDIA and AMD GPUs. We use five proxy applications from different scientific domains, create implementations where missing, and use them to present a comprehensive comparative evaluation of the performance portability of these programming models. We provide a Spack scripting-based methodology to ensure reproducibility of experiments conducted in this work. Finally, we analyze the reasons for why some programming models underperform in certain scenarios and in some cases, present performance optimizations to the proxy applications.

Software stewardship and advancement of a high-performance computing scientific application: QMCPACK

SummaryThis work explores the performance of single‐ and multi‐GPU computing on state‐of‐the‐art NVIDIA‐ and AMD‐based server‐class hardware using various programming interfaces to accelerate a real‐world scientific application for solidification modeling based on the phase‐field method. The main computations of this memory‐bound application correspond to 20 stencils computed across grid nodes. We investigate the application's scalability for two basic schemes of organizing computation: without and with hiding data transfers behind computation, combined with using either peer‐to‐peer inter‐GPU data transfers through NVIDIA NVLink and AMD Infinity interconnects or communication over the PCIe and main memory. Among the studied programming interfaces is CUDA, HIP, and OpenMP Accelerator Model. While the first two are designed to write the codes for a specific hardware platform, OpenMP enables code portability between NVIDIA and AMD GPUs. The resulting performance is experimentally assessed on computing platforms containing NVIDIA V100 (up to 8 GPUs) and A100 (one GPU), as well as AMD MI210 (one device) and MI250 (up to 8 logical GPUs).

We describe our approach in augmenting the beagle library for high-performance statistical phylogenetic inference to support concurrent computation of independent partial likelihoods arrays. Our solution involves identifying independent likelihood estimates in analyses of partitioned datasets and in proposed tree topologies, and configuring concurrent computation of these likelihoods via cuda and opencl frameworks. We evaluate the effect of each increase in concurrency on throughput performance for our partial likelihoods kernel for a four-state nucleotide substitution model on a variety of parallel computing hardware, such as nvidia and amd gpus, and Intel multicore cpus, observing up to 16-fold speedups over our previous implementation. Finally, we evaluate the effect of these gains on an domain application program, mrbayes. For a partitioned nucleotide-model analysis we observe an average speedup for the overall run time of 2.1-fold over our previous parallel implementation, and 10-fold over the native mrbayes with sse.

Recent GPUs enable Peer-to-Peer Direct Memory Access ( p 2 p ) from fast peripheral devices like NVMe SSDs to exclude the CPU from the data path between them for efficiency. Unfortunately, using p 2 p to access files is challenging because of the subtleties of low-level non-standard interfaces, which bypass the OS file I/O layers and may hurt system performance. Developers must possess intimate knowledge of low-level interfaces to manually handle the subtleties of data consistency and misaligned accesses. We present SPIN , which integrates p 2 p into the standard OS file I/O stack, dynamically activating p 2 p where appropriate, transparently to the user. It combines p 2 p with page cache accesses, re-enables read-ahead for sequential reads, all while maintaining standard POSIX FS consistency, portability across GPUs and SSDs, and compatibility with virtual block devices such as software RAID. We evaluate SPIN on NVIDIA and AMD GPUs using standard file I/O benchmarks, application traces, and end-to-end experiments. SPIN achieves significant performance speedups across a wide range of workloads, exceeding p 2 p throughput by up to an order of magnitude. It also boosts the performance of an aerial imagery rendering application by 2.6× by dynamically adapting to its input-dependent file access pattern, enables 3.3× higher throughput for a GPU-accelerated log server, and enables 29% faster execution for the highly optimized GPU-accelerated image collage with only 30 changed lines of code.

Application management in Linux based operating systems is managed by package managers. The package repositories of most Linux distributions provide software through central servers. While having a central server provides a unique verified and maintained version, servers often separate official and community packages, and users frequently use community packages. However, some highly demanded community packages may not be stored in official repositories. While central servers are open to a single point of failure, using community packages without any security check may be insecure. In this paper, we introduce the idea of a distributed package management architecture for Linux systems based on blockchain technology. In the proposed system, package providers are considered as miners, building a package is mining, and proof-of-work is reproducible packaging by the miners. The blockchain keeps the hash of the package contents. Creating and testing a package will return to the packager as reliability scores. This system will have the following advantages. Packages will be verified through consensus, single points of failure will be eliminated ensuring availability, packages are stored in a distributed network. The goal is to keep the operating system alive with a decentralized, reliable, and transparent blockchain-based package management system.

Package managers are a very important part of Linux distributions but we have noticed two weaknesses in them: They use pre-built packages that are not optimised for specific hardware and often they are too heavy for a specific need, or packagesmay require plenty of time and resources to be compiled. In this paper, we present a novel Linux package manager which uses cloud computing features to compile and distribute Linux packages without impacting the end user's performance. We also show how Portage, Gentoo's package manager can be optimised for customisation and performance, along with the cloud computing features to compile Linux packages more efficiently. All of this resulting in a new cloud-based Linux package manager that is built for better computing performance.

Efficiently simulating solid mechanics is vital across various engineering applications. As constitutive models grow more complex and simulations scale up in size, harnessing the capabilities of modern computer architectures has become essential for achieving timely results. This paper presents advancements in running parallel simulations of solid mechanics on multi-core CPUs and GPUs using a single-code implementation. This portability is made possible by the C++ matrix and array (MATAR) library, which interfaces with the C++ Kokkos library, enabling the selection of fine-grained parallelism backends (e.g., CUDA, HIP, OpenMP, pthreads, etc.) at compile time. MATAR simplifies the transition from Fortran to C++ and Kokkos, making it easier to modernize legacy solid mechanics codes. We applied this approach to modernize a suite of constitutive models and to demonstrate substantial performance improvements across different computer architectures. This paper includes comparative performance studies using multi-core CPUs along with AMD and NVIDIA GPUs. Results are presented using a hypoelastic–plastic model, a crystal plasticity model, and the viscoplastic self-consistent generalized material model (VPSC-GMM). The results underscore the potential of using the MATAR library and modern computer architectures to accelerate solid mechanics simulations.