Multi-GPU Support on Single Node Using Directive-Based Programming Model

GPUs are being widely used to accelerate different workloads and multi-GPU systems can provide higher performance with multiple discrete GPUs interconnected together. However, there are two main communication bottlenecks in multi-GPU systems -- accessing remote GPU memory and the communication between GPU and the host CPU. Recent advances in multi-GPU programming, including unified virtual addressing and unified memory from NVIDIA, has made programming simpler but the costly remote memory access still makes multi-GPU programming difficult. In order to overcome the communication limitations, we propose to leverage the memory network based on hybrid memory cubes (HMCs) to simplify multi-GPU memory management and improve programmability. In particular, we propose scalable kernel execution (SKE) where multiple GPUs are viewed as a single virtual GPU as a single kernel can be executed across multiple GPUs without modifying the source code. To fully enable the benefits of SKE, we explore alternative memory network designs in a multi-GPU system. We propose a GPU memory network (GMN) to simplify data sharing between the discrete GPUs while a CPU memory network (CMN) is used to simplify data communication between the host CPU and the discrete GPUs. These two types of networks can be combined to create a unified memory network (UMN) where the communication bottleneck in multi-GPU can be significantly minimized as both the CPU and GPU share the memory network. We evaluate alternative network designs and propose a sliced flattened butterfly topology for the memory network that scales better than previously proposed alternative topologies by removing local HMC channels. In addition, we propose an overlay network organization for unified memory network to minimize the latency for CPU access while providing high bandwidth for the GPUs. We evaluate trade-offs between the different memory network organization and show how UMN significantly reduces the communication bottleneck in multi-GPU systems.

Optimizing makespan and resource utilization for multi-DNN training in GPU cluster

A parallel distributed CUDA implementation of a Lattice Boltzmann Method for multiphase flows with large density ratios is described in this paper. Validation runs studying the terminal velocity of a rising bubble under the effect of gravity show good agreement with the expected theoretical values. The code is benchmarked against the performance of a typical CPU implementation of the same algorithm on both AMD and Intel platforms, and a single GPU is observed to perform up to 10X faster than a quad-core CPU socket, a 40X speedup with respect to a single core. The code is shown to scale well when executed on multiple GPUs, which makes the port to CUDA valuable even when compared to parallel CPU implementations.

Influence Maximization (IM) is a popular problem focusing on finding a seed vertex set in a graph that maximizes the expected number of vertices affected via diffusion under a given, usually probabilistic model. For most diffusion models used in practice, finding an optimal seed set of a given size is NP-Hard. Hence, approximation algorithms and heuristics are often proposed and used. The Greedy approach is one of the most frequently applied approximation approach employed for IM. Indeed, this Monte-Carlo-based approach performs remarkably well in terms of seed set quality, i.e., the number of affected vertices. However, it is impractical for real-life networks containing tens of millions of vertices due to its expensive simulation costs. Recently, parallel IM kernels running on CPUs and GPUs have been proposed in the literature. In this work, we propose SUPERFUSER, a blazing-fast, sketch-based Influence Maximization algorithm developed for multiple GPUs. SUPERFUSER uses hash-based fused sampling to process multiple simulations at the same time with minimal overhead. In addition, we propose a Sampling-Aware Sample-Space Split approach to partition the edges to multiple GPUs efficiently by exploiting the unique characteristics of the sampling process. Based on our experiments, SUPERFUSER is up to 6.31× faster than its nearest competitor on a single GPU. Furthermore, we achieve 6.8× speed-up on average using 8 GPUs over a single GPU performance, and thanks to our novel partitioning scheme, we can process extremely large-scale graphs in practice without sacrificing quality too much. As an example, SUPERFUSER can generate a high-quality seed set with 50 vertices for a graph having 1.8B edges in less than 15 seconds on 2 GPUs.

Efficient breadth first search on multi-GPU systems

Numerical simulation of underground CO2 storage requires accurate, fast and field-scale calculations of saturation and reservoir pressures, yet high-fidelity physics-based simulators are computationally expensive to do a rapid analysis. Recent advances in deep learning, especially Graph Neural Networks (GNNs), offer promising alternatives by approximating simulation outputs while dramatically reducing computational costs by minimizing the total reliance on traditional simulators. However, training GNN models on large-scale, spatiotemporal datasets remains a significant bottleneck when constrained to a single GPU. To address the computational bottleneck of training GNN on large-scale and high-resolution datasets, we propose to use multi-GPU scalable training of GNNs. The U-Net Enhanced Graph Neural Network (U-GCN) specifically was adopted to predict the spatiotemporal evolution of gas saturation and pressure buildup in CO2 storage in saline aquifers. The UGCN model was trained across multiple GPUs, allowing for concurrent gradient computations and parameter updates on separate GPU processes. The model operates natively on regular reservoir meshes to learn spatiotemporal evolution of pressure and saturation from thousands of simulation cases, and the training pipeline couples process-per-GPU data parallelism with a node-local caching strategy that deamplifies the source data file (in Hierarchical Data Format HDF5 format) Input-Output (I/O) by performing a single deserialization per node and broadcasting preprocessed graphs to sibling ranks. This eliminates multi-node startup contention and renders runs compute bound. Scaling experiments from 1 to 16 GPUs demonstrate substantial wall-clock reductions while preserving validation behavior. With per-GPU batch size 100, total training time decreased from 955.46 s (1 GPU) to 145.17 s (16 GPUs), a 6.58× speedup (≈41% parallel efficiency). With per-GPU batch size 8, time decreased from 5519.00 s to 1047.17 s, a 5.27× speedup (≈33% efficiency). The speedups are reported per epoch. Cross-node overheads relative to ideal doubling were ∼11–35%, more pronounced at small per-GPU batches where collective latency dominates. Training and validation losses rose modestly as GPU count increased under weak scaling—consistent with the larger global batch and fewer optimizer updates per epoch—yet validation tracked training across all settings, indicating stable generalization and no data leakage. These effects are mitigated by standard large-batch remedies (e.g., linear learning-rate scaling with warmup, longer schedules, optional gradient accumulation). By integrating domain-aware batching, multi-GPU data parallelism, mixed precision, and node-local caching with a GNN tailored to structured meshes, the proposed approach delivers a reproducible, high-throughput storage surrogate training regime that scales to multi-node clusters without compromising model validity. The resulting surrogates enable rapid scenario screening and outer-loop workflows (optimization and UQ) at orders-of-magnitude lower training cost compared to conventional GNN training on single a GPU baseline.

Self-supervised pretraining has shown impressive performance in recent years, matching or even outperforming ImageNet weights on a broad range of downstream tasks. Unfortunately, existing methods require massive amounts of computing power with large batch sizes and batch norm statistics synchronized across multiple GPUs. This effectively excludes substantial parts of the computer vision community from the benefits of self-supervised learning who do not have access to extensive computing resources.To address that, we develop FastSiam with the aim of matching ImageNet weights given as little computing power as possible. We find that a core weakness of previous methods like SimSiam is that they compute the training target based on a single augmented crop (or “view”), leading to target instability. We show that by using multiple views per image instead of one, the training target can be stabilized, allowing for faster convergence and substantially reduced runtime. We evaluate FastSiam on multiple challenging downstream tasks including object detection, instance segmentation and keypoint detection and find that it matches ImageNet weights after 25 epochs of pretraining on a single GPU with a batch size of only 32.KeywordsResource-efficientSelf-supervised learning

The quantum mechanics/molecular mechanics (QM/MM) approach is an essential and well-established tool in computational chemistry that has been widely applied in a myriad of biomolecular problems in the literature. In this publication, we report the integration of the QUantum Interaction Computational Kernel (QUICK) program as an engine to perform electronic structure calculations in QM/MM simulations with AMBER. This integration is available through either a file-based interface (FBI) or an application programming interface (API). Since QUICK is an open-source GPU-accelerated code with multi-GPU parallelization, users can take advantage of “free of charge” GPU-acceleration in their QM/MM simulations. In this work, we discuss implementation details and give usage examples. We also investigate energy conservation in typical QM/MM simulations performed at the microcanonical ensemble. Finally, benchmark results for two representative systems, the N-methylacetamide (NMA) molecule and the photoactive yellow protein (PYP) in bulk water, show the performance of QM/MM simulations with QUICK and AMBER using a varying number of CPU cores and GPUs. Our results highlight the acceleration obtained from a single or multiple GPUs; we observed speedups of up to 38x between a single GPU vs. a single CPU core and of up to 2.6x when comparing four GPUs to a single GPU. Results also reveal speedups of up to 3.5x when the API is used instead of FBI.

Hyperspectral images (HSI) classification has been a research hotspot in the remote sensing field. Deep learning methods such as Cube-CNN have been applied to address the HSI classification problem. However, mainstream frameworks exist a performance gap to train Cube-CNN, since they are not designed for processing high dimensional data like HSI. To close this gap, we propose a Multi-GPU Training Framework (MTFC) for Cube-CNN-based HSI classification. We first design a Parallel Neighbor Pixel Extraction (PNPE) algorithm for efficiently generating 3-dimensional cube samples from raw data. Then, to fully exploit massive GPU parallelism and realize unique characteristics of HSI and Cube-CNN, we employ optimizations in MTFC such as task division, fine-grained mapping between tasks and GPU thread blocks, shared memory usage reduction, etc. Finally, to further improve training speed, we take advantage of CUDA streams and multiple GPUs to train a mini-batches of data samples simultaneously. An extensive set of experiments highlights that MTFC constantly outperforms the two baselines Caffe and Theano for all measured metrics across all system configurations, while offering the same level of classification accuracy. The speedup is up to <inline-formula><tex-math notation="LaTeX">$3.6x$</tex-math></inline-formula> when using a single GPU and MTFC can achieve a rough linear scaling on multiple GPUs.

High-level domain-specific languages for array processing on the GPU are increasingly common, but they typically only run on a single GPU. As computational power is distributed across more devices, languages must target multiple devices simultaneously. To this end, we present a compositional translation that fissions data-parallel programs in the Accelerate language, allowing subsequent compiler and runtime stages to map computations onto multiple devices for improved performance---even programs that begin as a single data-parallel kernel.

Atmospheric radiation physical process plays an important role in climate simulations. As a radiative transfer scheme, the rapid radiative transfer model for general circulation models (RRTMG) is widely used in weather forecasting and climate simulation systems. However, its expensive computational overhead poses a severe challenge to system performance. Therefore, improving the radiative transfer model’s computational performance has significant scientific research and practical value. Numerous radiative transfer models have benefited from a widely used and powerful GPU. Nevertheless, few of them have exploited CPU/GPU cluster resources within heterogeneous high-performance computing platforms. In this paper, we endeavor to demonstrate an approach that runs a large-scale, computationally intensive, longwave radiative transfer model on a GPU cluster. First, a CUDA-based acceleration algorithm of the RRTMG longwave radiation scheme (RRTMG_LW) on multiple GPUs is proposed. Then, a heterogeneous, hybrid programming paradigm (MPI+CUDA) is presented and utilized with the RRTMG_LW on a GPU cluster. After implementing the algorithm in CUDA Fortran, a multi-GPU version of the RRTMG_LW, namely GPUs-RRTMG_LW, was developed. The experimental results demonstrate that the multi-GPU acceleration algorithm is valid, scalable, and highly efficient when compared to a single GPU or CPU. Running the GPUs-RRTMG_LW on a K20 cluster achieved a $$77.78 \times$$ speedup when compared to a single Intel Xeon E5-2680 CPU core.

Efficient graph processing is essential for a wide range of applications. Scalability and memory access patterns are still a challenge, especially with the Breadth-First Search algorithm. This work focuses on leveraging HPC systems with multiple GPUs available in a single node with peer-to-peer functionality of the Intel oneAPI implementation of SYCL. We propose three GPU-based load-balancing methods: work-group localisation for efficient data access, even workload distribution for higher GPU occupancy, and a hybrid strided-access approach for heuristic balancing. These methods ensure performance, portability, and productivity with a unified codebase. Our proposed methodologies outperform state-of-the-art single-GPU implementations based on CUDA on synthetic RMAT graphs. We analysed BFS performance across NVIDIA A100, Intel Max 1550, and AMD MI300X GPUs, achieving a peak performance of 153.27 GTEPS on an RMAT25-64 graph using 8 GPUs on the NVIDIA A100. Furthermore, our work demonstrates the capability to handle RMAT graphs up to scale 29, achieving superior performance on synthetic graphs and competitive results on real-world datasets.

We present an efficient tensor-network-based approach for simulating large-scale quantum circuits exemplified by quantum support vector machines (QSVMs). Experimentally, leveraging the cuTensorNet library on multiple GPUs, our method effectively reduces the exponential runtime growth to near-quadratic scaling with respect to the number of qubits in practical scenarios. Traditional state-vector simulations become computationally infeasible beyond approximately 50 qubits; in contrast, our simulator successfully handles QSVMs with up to 784 qubits, executing simulations within seconds on a single high-performance GPU. Furthermore, utilizing the message passing interface for multi-GPU environments, our method demonstrates strong linear scalability, effectively decreasing computation time as dataset sizes increase. We validate our framework using the MNIST and Fashion MNIST datasets, achieving successful multiclass classification and highlighting the potential of QSVMs for high-dimensional data analysis. By integrating tensor-network techniques with advanced high-performance computing resources, this work demonstrates both the feasibility and scalability of simulating large-qubit quantum machine learning models, providing a valuable validation tool within the emerging Quantum-HPC ecosystem.

Spread spectrum (SS) signal acquisition in satellite communication is a very computation intensive technique, which hinders the development of real-time spread spectrum signal acquisition. In this paper, in order to achieve real-time acquisition, we propose a multi-GPU based SS signal acquisition algorithm. First, sliding correlation, the computation kernel, is formulated and efficiently parallelized by CUDA. Second, a CUDA-enabled SS signal acquisition algorithm is implemented by adopting the CUDA-enabled sliding correlation. Third, a multi-GPU based algorithm is implemented by using multi-GPU programming. The performance is evaluated in a real SS telemetry system. Real-time acquisition is achieved in all cases by using a single K40 GPU. Furthermore, an average of 374.7 $\times$ speedup (max 473 $\times$ ) in 6 datasets is achieved when using four K40 GPUs. Good scalability is observed when varying the parameters.

We present a new two-stage fine-grained air Particulate Matter (PM) prediction system using a variety of deep memory networks. Our model is significantly simpler than traditional weather report systems, which rely heavily on the atmospheric reaction equations and pollutant emissions inventories. Pollutant inventories are notoriously difficult to obtain and are often packed with declination for multiple reasons, while reaction equations are impossible to exhaust. These traditional models also tend to perform poorly when affected by strong convective weather. In contrast, our model does not need precise hand collected inventories of pollution sources. It can utilize the potential of Deep-Neural-Networks (DNN) to reveal the relationship among different locations and even find relations between known social events and air quality. Both potentially provide a valid path to air pollution control. The key to our approach is a well-tuned sophisticated attention based network that uses multiple GPUs, allowing us to transform a traditional sparse prediction problem into a sequence-to-sequence learning problem and train it end-to-end. By evaluating the model on over 1,625 instances of data of Northern China collected by our team, we show that it is not only computationally efficient but also accurately feasible compared with other optional models.