Efficient breadth first search on multi-GPU systems

Optimizing makespan and resource utilization for multi-DNN training in GPU cluster

GPUs are being widely used to accelerate different workloads and multi-GPU systems can provide higher performance with multiple discrete GPUs interconnected together. However, there are two main communication bottlenecks in multi-GPU systems -- accessing remote GPU memory and the communication between GPU and the host CPU. Recent advances in multi-GPU programming, including unified virtual addressing and unified memory from NVIDIA, has made programming simpler but the costly remote memory access still makes multi-GPU programming difficult. In order to overcome the communication limitations, we propose to leverage the memory network based on hybrid memory cubes (HMCs) to simplify multi-GPU memory management and improve programmability. In particular, we propose scalable kernel execution (SKE) where multiple GPUs are viewed as a single virtual GPU as a single kernel can be executed across multiple GPUs without modifying the source code. To fully enable the benefits of SKE, we explore alternative memory network designs in a multi-GPU system. We propose a GPU memory network (GMN) to simplify data sharing between the discrete GPUs while a CPU memory network (CMN) is used to simplify data communication between the host CPU and the discrete GPUs. These two types of networks can be combined to create a unified memory network (UMN) where the communication bottleneck in multi-GPU can be significantly minimized as both the CPU and GPU share the memory network. We evaluate alternative network designs and propose a sliced flattened butterfly topology for the memory network that scales better than previously proposed alternative topologies by removing local HMC channels. In addition, we propose an overlay network organization for unified memory network to minimize the latency for CPU access while providing high bandwidth for the GPUs. We evaluate trade-offs between the different memory network organization and show how UMN significantly reduces the communication bottleneck in multi-GPU systems.

Atmospheric radiation physical process plays an important role in climate simulations. As a radiative transfer scheme, the rapid radiative transfer model for general circulation models (RRTMG) is widely used in weather forecasting and climate simulation systems. However, its expensive computational overhead poses a severe challenge to system performance. Therefore, improving the radiative transfer model’s computational performance has significant scientific research and practical value. Numerous radiative transfer models have benefited from a widely used and powerful GPU. Nevertheless, few of them have exploited CPU/GPU cluster resources within heterogeneous high-performance computing platforms. In this paper, we endeavor to demonstrate an approach that runs a large-scale, computationally intensive, longwave radiative transfer model on a GPU cluster. First, a CUDA-based acceleration algorithm of the RRTMG longwave radiation scheme (RRTMG_LW) on multiple GPUs is proposed. Then, a heterogeneous, hybrid programming paradigm (MPI+CUDA) is presented and utilized with the RRTMG_LW on a GPU cluster. After implementing the algorithm in CUDA Fortran, a multi-GPU version of the RRTMG_LW, namely GPUs-RRTMG_LW, was developed. The experimental results demonstrate that the multi-GPU acceleration algorithm is valid, scalable, and highly efficient when compared to a single GPU or CPU. Running the GPUs-RRTMG_LW on a K20 cluster achieved a $$77.78 \times$$ speedup when compared to a single Intel Xeon E5-2680 CPU core.

K-Means is the clustering algorithm which is widely used in many areas such as information retrieval, computer vision and pattern recognition. With the recent advance in General Purpose Graphics Processing Unit (GPGPU), we can use a modern GPU which is capable to do computation up to Tflops to calculate K-Means clustering on average problems. However, due to the exponential growth of data, the K-Means clustering on a single GPU will not be adequate for large datasets in the near future. In this paper, we present the design and implementation of an efficient large-scale parallel K-Means on GPU clusters. We utilize the massive parallelism in GPUs to speed up the most time consuming part of K-Means clustering in each node. We employ the dynamic load balancing to distribute workload equally on different GPUs installed in the clusters so as to improve the performance of the parallel K-Means at the inter-node level. We also take advantage from software distributed shared memory to simplify the communication and collaboration among nodes. The result of the evaluation shows the performance improvement of the parallel K-Means by maintaining load balance on GPU clusters.

Searching protein sequence database is a fundamental and often repeated task in computational biology and bioinformatics. However, the high computational cost and long runtime of many database scanning algorithms on sequential architectures heavily restrict their applications for large-scale protein databases, such as GenBank. The continuing exponential growth of sequence databases and the high rate of newly generated queries further deteriorate the situation and establish a strong requirement for time-efficient scalable database searching algorithms. In this paper, we demonstrate how GPU clusters, powered by the Compute Unified Device Architecture (CUDA), OpenMP, and MPI parallel programming models can be used as an efficient computational platform to accelerate the popular BLASTP algorithm. Compared to GPU-BLAST 1.0-2.2.24, our implementation achieves speedups up to 1.6 on a single GPU and up to 6.6 on the 6 GPUs of a Tesla S1060 quad-GPU computing system. The source code is available at: http://sites.google.com/site/liuweiguohome/mpicuda-blastp

Nowadays, driven by the exponential growth of parameters and training data of AI applications and Large Language Models, a single GPU is no longer sufficient in terms of computing power and storage capacity. Building high-performance multi-GPU systems or a GPU cluster via vertical scaling (scale-up) has thus become an effective approach to break the bottleneck and has further emerged as a key research focus. Given that traditional inter-GPU communication technologies fail to meet the requirement of GPU interconnection in vertical scaling, a variety of high-performance inter-GPU communication protocols tailored for the scale-up domain have been proposed recently. Notably, due to the emerging nature of these demands and technologies, academic research in this field remains scarce, with limited deep participation from the academic community. Inspired by this trend, this article identifies the challenges and requirements of a scale-up network, analyzes the bottlenecks of traditional technologies like PCIe in a scale-up network, and surveys the emerging scale-up targeted technologies, including NVLink, OISA, UALink, SUE, and other X-Links. Then, an in-depth comparison and discussion is conducted, and we express our insights in protocol design and related technologies. We also highlight that existing emerging protocols and technologies still face limitations, with certain technical mechanisms requiring further exploration. Finally, this article presents future research directions and opportunities. As the first review article fully focusing on intra-node GPU interconnection in a scale-up network, this article aims to provide valuable insights and guidance for future research in this emerging field, and we hope to establish a foundation that will inspire and direct subsequent studies.

Existing studies show that using single GPU can lead to obtaining significant performance gains. We should be able to achieve further performance speedup if we use more than one GPU. Heterogeneous processors consisting of multiple CPUs and GPUs offer immense potential and are often considered as a leading candidate for porting complex scientific applications. Unfortunately programming heterogeneous systems requires more effort than what is required for traditional multicore systems. Directive-based programming approaches are being widely adopted since they make it easy to use/port/maintain application code. OpenMP and OpenACC are two popular models used to port applications to accelerators. However, neither of the models provides support for multiple GPUs. A plausible solution is to use combination of OpenMP and OpenACC that forms a hybrid model; however, building this model has its own limitations due to lack of necessary compilers’ support. Moreover, the model also lacks support for direct device-to-device communication. To overcome these limitations, an alternate strategy is to extend OpenACC by proposing and developing extensions that follow a task-based implementation for supporting multiple GPUs. We critically analyze the applicability of the hybrid model approach and evaluate the proposed strategy using several case studies and demonstrate their effectiveness.

GPUs have been accelerating a wide range of algorithms and applications with their massively parallel computing capabilities. Today, as GPU programs work under the close supervision of a CPU, computing platforms are usually equipped with both CPUs and GPUs. Most existing GPU programs underutilize the computing capabilities of the CPU since the CPU is frequently idle while GPUs are executing. Additionally, existing work primarily focuses on using a single GPU, thus failing to utilize the computing power of multi-GPU systems. In this dissertation, we propose a new heterogeneous computing paradigm--Collaborative Heterogeneous Computing. Collaborative Heterogeneous Computing leverages fine-grained CPU-GPU communication mechanisms in computation with the use of CPUs and multiple GPUs simultaneously. Collaborative Heterogeneous Computing can be divided into CPU-GPU collaborative computing and Multi-GPU collaborative computing. For CPU-GPU collaborative computing, we first categorize 7 CPU-GPU collaborative execution patterns. With the guidance of the summarized CPU-GPU execution patterns, we implement Hetero-Mark, a benchmark suite that enables users to explore CPU-GPU collaborative execution patterns. We also design and develop Multi2Sim-HSA, an HSA system emulator that emulates behaviors of CPUs and GPUs in CPU-GPU collaborative execution applications. In addition, we observe that CPU-GPU communication over the PCIe network causes congestion and can become a major performance bottleneck. To address this issue, we propose a priority-based PCIe scheduling algorithm that can improve device utilization and satisfy Quality of Service (QoS) requirements. For multi-GPU collaborative computing, we develop MGPUMark, a benchmark suite that explores multi-GPU collaborative execution and fine-grained inter-GPU communication. Benchmarks in MGPUMark enable exploration of inter-GPU communication patterns. We also design and develop MGPUSim, a high-performance, high-flexibility, and high-accuracy GPU simulator. MGPUSim can faithfully model behaviors of GPUs in multi-GPU collaborative computing. We identify that the main performance bottlenecks occur due to long-latency, low-bandwidth, inter-GPU interconnects. Thus, reducing inter-GPU communication traffic can improve multi-GPU system performance. We develop the Locality APIs, a set of GPU programming APIs that enables programmers to place the computing threads and associated data on a multi-GPU system to reduce inter-GPU traffic. We also introduce Progressive Page-Splitting Migration (PASI), a hardware-based solution that allows GPUs to adjust data placement to avoid inter-GPU traffic.--Author's abstract

Heterogeneous acceleration algorithms for shallow cumulus convection scheme over GPU clusters

A parallel distributed CUDA implementation of a Lattice Boltzmann Method for multiphase flows with large density ratios is described in this paper. Validation runs studying the terminal velocity of a rising bubble under the effect of gravity show good agreement with the expected theoretical values. The code is benchmarked against the performance of a typical CPU implementation of the same algorithm on both AMD and Intel platforms, and a single GPU is observed to perform up to 10X faster than a quad-core CPU socket, a 40X speedup with respect to a single core. The code is shown to scale well when executed on multiple GPUs, which makes the port to CUDA valuable even when compared to parallel CPU implementations.

Influence Maximization (IM) is a popular problem focusing on finding a seed vertex set in a graph that maximizes the expected number of vertices affected via diffusion under a given, usually probabilistic model. For most diffusion models used in practice, finding an optimal seed set of a given size is NP-Hard. Hence, approximation algorithms and heuristics are often proposed and used. The Greedy approach is one of the most frequently applied approximation approach employed for IM. Indeed, this Monte-Carlo-based approach performs remarkably well in terms of seed set quality, i.e., the number of affected vertices. However, it is impractical for real-life networks containing tens of millions of vertices due to its expensive simulation costs. Recently, parallel IM kernels running on CPUs and GPUs have been proposed in the literature. In this work, we propose SUPERFUSER, a blazing-fast, sketch-based Influence Maximization algorithm developed for multiple GPUs. SUPERFUSER uses hash-based fused sampling to process multiple simulations at the same time with minimal overhead. In addition, we propose a Sampling-Aware Sample-Space Split approach to partition the edges to multiple GPUs efficiently by exploiting the unique characteristics of the sampling process. Based on our experiments, SUPERFUSER is up to 6.31× faster than its nearest competitor on a single GPU. Furthermore, we achieve 6.8× speed-up on average using 8 GPUs over a single GPU performance, and thanks to our novel partitioning scheme, we can process extremely large-scale graphs in practice without sacrificing quality too much. As an example, SUPERFUSER can generate a high-quality seed set with 50 vertices for a graph having 1.8B edges in less than 15 seconds on 2 GPUs.

A GPU cluster is a cluster equipped with GPU devices. Excellent acceleration is achievable for computation-intensive tasks (e. g. matrix multiplication and LINPACK) and bandwidth-intensive tasks with data locality (e. g. finite-difference simulation). Bandwidth-intensive tasks such as large-scale FFTs without data locality are harder to accelerate, as the bottleneck often lies with the PCI between main memory and GPU device memory or the communication network between workstation nodes. That means optimizing the performance of FFT for a single GPU device will not improve the overall performance. This paper uses large-scale FFT as an example to show how to achieve substantial speedups for these more challenging tasks on a GPU cluster. Three GPU-related factors lead to better performance: firstly the use of GPU devices improves the sustained memory bandwidth for processing large-size data; secondly GPU device memory allows larger subtasks to be processed in whole and hence reduces repeated data transfers between memory and processors; and finally some costly main-memory operations such as matrix transposition can be significantly sped up by GPUs if necessary data adjustment is performed during data transfers. This technique of manipulating array dimensions during data transfer is the main technical contribution of this paper. These factors (as well as the improved communication library in our implementation) attribute to 24.3x speedup with respect to FFTW and 7x speedup with respect to Intel MKL for 4096 3D single-precision FFT on a 16-node cluster with 32 GPUs. Around 5x speedup with respect to both standard libraries are achieved for double precision.

Numerical simulation of underground CO2 storage requires accurate, fast and field-scale calculations of saturation and reservoir pressures, yet high-fidelity physics-based simulators are computationally expensive to do a rapid analysis. Recent advances in deep learning, especially Graph Neural Networks (GNNs), offer promising alternatives by approximating simulation outputs while dramatically reducing computational costs by minimizing the total reliance on traditional simulators. However, training GNN models on large-scale, spatiotemporal datasets remains a significant bottleneck when constrained to a single GPU. To address the computational bottleneck of training GNN on large-scale and high-resolution datasets, we propose to use multi-GPU scalable training of GNNs. The U-Net Enhanced Graph Neural Network (U-GCN) specifically was adopted to predict the spatiotemporal evolution of gas saturation and pressure buildup in CO2 storage in saline aquifers. The UGCN model was trained across multiple GPUs, allowing for concurrent gradient computations and parameter updates on separate GPU processes. The model operates natively on regular reservoir meshes to learn spatiotemporal evolution of pressure and saturation from thousands of simulation cases, and the training pipeline couples process-per-GPU data parallelism with a node-local caching strategy that deamplifies the source data file (in Hierarchical Data Format HDF5 format) Input-Output (I/O) by performing a single deserialization per node and broadcasting preprocessed graphs to sibling ranks. This eliminates multi-node startup contention and renders runs compute bound. Scaling experiments from 1 to 16 GPUs demonstrate substantial wall-clock reductions while preserving validation behavior. With per-GPU batch size 100, total training time decreased from 955.46 s (1 GPU) to 145.17 s (16 GPUs), a 6.58× speedup (≈41% parallel efficiency). With per-GPU batch size 8, time decreased from 5519.00 s to 1047.17 s, a 5.27× speedup (≈33% efficiency). The speedups are reported per epoch. Cross-node overheads relative to ideal doubling were ∼11–35%, more pronounced at small per-GPU batches where collective latency dominates. Training and validation losses rose modestly as GPU count increased under weak scaling—consistent with the larger global batch and fewer optimizer updates per epoch—yet validation tracked training across all settings, indicating stable generalization and no data leakage. These effects are mitigated by standard large-batch remedies (e.g., linear learning-rate scaling with warmup, longer schedules, optional gradient accumulation). By integrating domain-aware batching, multi-GPU data parallelism, mixed precision, and node-local caching with a GNN tailored to structured meshes, the proposed approach delivers a reproducible, high-throughput storage surrogate training regime that scales to multi-node clusters without compromising model validity. The resulting surrogates enable rapid scenario screening and outer-loop workflows (optimization and UQ) at orders-of-magnitude lower training cost compared to conventional GNN training on single a GPU baseline.

Self-supervised pretraining has shown impressive performance in recent years, matching or even outperforming ImageNet weights on a broad range of downstream tasks. Unfortunately, existing methods require massive amounts of computing power with large batch sizes and batch norm statistics synchronized across multiple GPUs. This effectively excludes substantial parts of the computer vision community from the benefits of self-supervised learning who do not have access to extensive computing resources.To address that, we develop FastSiam with the aim of matching ImageNet weights given as little computing power as possible. We find that a core weakness of previous methods like SimSiam is that they compute the training target based on a single augmented crop (or “view”), leading to target instability. We show that by using multiple views per image instead of one, the training target can be stabilized, allowing for faster convergence and substantially reduced runtime. We evaluate FastSiam on multiple challenging downstream tasks including object detection, instance segmentation and keypoint detection and find that it matches ImageNet weights after 25 epochs of pretraining on a single GPU with a batch size of only 32.KeywordsResource-efficientSelf-supervised learning

The quantum mechanics/molecular mechanics (QM/MM) approach is an essential and well-established tool in computational chemistry that has been widely applied in a myriad of biomolecular problems in the literature. In this publication, we report the integration of the QUantum Interaction Computational Kernel (QUICK) program as an engine to perform electronic structure calculations in QM/MM simulations with AMBER. This integration is available through either a file-based interface (FBI) or an application programming interface (API). Since QUICK is an open-source GPU-accelerated code with multi-GPU parallelization, users can take advantage of “free of charge” GPU-acceleration in their QM/MM simulations. In this work, we discuss implementation details and give usage examples. We also investigate energy conservation in typical QM/MM simulations performed at the microcanonical ensemble. Finally, benchmark results for two representative systems, the N-methylacetamide (NMA) molecule and the photoactive yellow protein (PYP) in bulk water, show the performance of QM/MM simulations with QUICK and AMBER using a varying number of CPU cores and GPUs. Our results highlight the acceleration obtained from a single or multiple GPUs; we observed speedups of up to 38x between a single GPU vs. a single CPU core and of up to 2.6x when comparing four GPUs to a single GPU. Results also reveal speedups of up to 3.5x when the API is used instead of FBI.