Scalable and accurate multi-GPU-based image reconstruction of large-scale ptychography data

This paper presents a parallel system capable of accelerating biological sequence alignment on the graphics processing unit (GPU) grid. The GPU grid in this paper is a desktop grid system that utilizes idle GPUs and CPUs in the office and home. Our parallel implementation employs a master-worker paradigm to accelerate an OpenGL-based algorithm that runs on a single GPU. We integrate this implementation into a screensaver-based grid system that detects idle resources on which the alignment code can run. We also show some experimental results comparing our implementation with three different implementations running on a single GPU, a single CPU, or multiple CPUs. As a result, we find that a single non-dedicated GPU can provide us almost the same throughput as two dedicated CPUs in our laboratory environment, where GPU-equipped machines are ordinarily used to develop GPU applications. In a dedicated environment, the GPU-accelerated code achieves five times higher throughput than the CPU-based code. Furthermore, a linear speedup of 30.7X is observed on a 32-node cluster of dedicated GPUs. We also implement a compute unified device architecture (CUDA) based algorithm to demonstrate further acceleration.

Large-scale simulations on multiple Graphics Processing Units (GPUs) for the direct simulation Monte Carlo method

The Generalized Minimum Residual (GMRES) method is one of the most widely-used iterative methods for solving nonsymmetric linear systems of equations. In recent years, techniques to avoid communication in GMRES have gained attention because in comparison to floating-point operations, communication is becoming increasingly expensive on modern computers. Since graphics processing units (GPUs) are now becoming crucial component in computing, we investigate the effectiveness of these techniques on multicore CPUs with multiple GPUs. While we present the detailed performance studies of a matrix powers kernel on multiple GPUs, we particularly focus on orthogonalization strategies that have a great impact on both the numerical stability and performance of GMRES, especially as the matrix becomes sparser or ill-conditioned. We present the experimental results on two eight-core Intel Sandy Bridge CPUs with three NDIVIA Fermi GPUs and demonstrate that significant speedups can be obtained by avoiding communication, either on a GPU or between the GPUs. As part of our study, we investigate several optimization techniques for the GPU kernels that can also be used in other iterative solvers besides GMRES. Hence, our studies not only emphasize the importance of avoiding communication on GPUs, but they also provide insight about the effects of these optimization techniques on the performance of the sparse solvers, and may have greater impact beyond GMRES.

Multi-phase SPH modelling of violent hydrodynamics on GPUs

Nowadays, Graph Traversal (GT) is a fundamental procedure for developing a measure in computing systems and algorithms. Graph traversal algorithms usually iterate input datasets of a graph to provide a logical convergence in every iteration. Graphic Processing Units (GPUs) have been utilized broadly as accelerators of GT due to their massive parallelism and accessibility of high-transfer speed memory. Furthermore, graph-based computation is pervasive and challenging for memory systems implementation in a specific time period. This paper provides quadratic and single GPUs, allowing developers to extend graph-based GPU computation effortlessly. The paper concentrates on the creditability and validation of sparse graphs in GPUs. Since the benchmark characteristic of Breadth-First Search (BFS) plays a critical role in graphs, this paper considers a primary characteristic benchmark for BFS computation usage. This paper explores the effectiveness and applicability of BFS for analyzing graph systems by utilizing the mentioned characteristics in BFS computation. This paper also utilizes the penalization of GPU BFS during the analysis and simulation results. In addition, using a comparative simulation result, this paper concludes that improved traversal rates speed up the computation and reduce memory usage for the single and quadratic GPU structures.

A novel parallel algorithm is presented for generating random scale-free networks using the preferential attachment model. The algorithm, named cuPPA, is custom-designed for “single instruction multiple data (SIMD)” style of parallel processing supported by modern processors such as graphical processing units (GPUs). To the best of our knowledge, our algorithm is the first to exploit GPUs, and also the fastest implementation available today, to generate scale-free networks using the preferential attachment model. A detailed performance study is presented to understand the scalability and runtime characteristics of the cuPPA algorithm. Also another version of the algorithm called cuPPA-Hash tailored for multiple GPUs is presented. On a single GPU, the original cuPPA algorithm delivers the best performance, but is challenging to port to multi-GPU implementation. For multi-GPU implementation, cuPPA-Hash has been used as the parallel algorithm to achieve a perfect linear speedup up to 4 GPUs. In one of the best cases, when executed on an NVidia GeForce 1080 GPU, the original cuPPA generates a scale-free network of two billion edges in less than 3 s. On multi-GPU platforms, cuPPA-Hash generates a scale-free network of 16 billion edges in less than 7 s using a machine consisting of 4 NVidia Tesla P100 GPUs.

The Lorentz–Drude model incorporated Maxwell equations are simulated by using the three-dimensional finite difference time domain (FDTD) method and the method is parallelized on multiple graphics processing units (GPUs) for plasmonics applications. The compute unified device architecture (CUDA) is used for GPU parallelization. The Lorentz–Drude (LD) model is used to simulate the dispersive nature of materials in plasmonics domain and the auxiliary differential equation (ADE) approach is used to make it consistent with time domain Maxwell equations. Different aspects of multiple GPUs for the FDTD method are presented such as comparison of different numbers of GPUs, transfer time in between them, synchronous, and asynchronous passing. It is shown that by using multiple GPUs in parallel fashion, significant reduction in the simulation time can be achieved as compared to the single GPU.

Graphics processing units (GPUs) have been widely adopted to accelerate the training of deep neural networks (DNNs). Although the computational performance of GPUs has been improving steadily, the memory size of modern GPUs is still quite limited, which restricts the sizes of DNNs that can be trained on GPUs, and hence raises serious challenges. This paper introduces a framework, referred to as moDNN (memory optimal DNN training on GPUs), to optimize the memory usage in DNN training. moDNN supports automatic tuning of DNN training code to match any given memory budget (not smaller than the theoretical lower bound). By taking full advantage of overlapping computations and data transfers, we develop new heuristics to judiciously schedule data offloading and prefetching transfers, together with convolution algorithm selection, to optimize memory usage. We further devise a new sub-batch size selection method which also greatly reduces memory usage. moDNN can save memory usage up to 59×, compared with an ideal case which assumes that the GPU memory is sufficient to hold all data. When executing moDNN on a GPU with 12 GB memory, the training time is increased by only 3 percent, which is much shorter than that incurred by the best known approach, vDNN. Furthermore, we propose an optimization strategy for moDNN on multiple GPUs again by utilizing the idea of overlapping data transfers and GPU computations. The results show that 3.7× speedup is attained on four GPUs.

The calculation of pairwise correlation coefficient on a dataset, known as the correlation matrix, is often used in data analysis, signal processing, pattern recognition, image processing, and bioinformatics. With the state-of-the-art Graphic Processing Units (GPUs) that consist of massive cores capable to do processing up to several Gflops, the calculation of correlation matrix can be accelerated several times over traditional CPUs. However, due to the rapid growth of the data in the digital era, the correlation matrix calculation becomes computing intensive which needs to be executed on multiple GPUs. As of now, GPUs are common components in data center at many institutions. Their GPU deployment tends towards a GPU cluster which each node is equipped with GPUs. In this paper, we propose a parallel computing based on the hybrid MPI/CUDA programming for fast and efficient Pearson correlation matrix calculation on GPU clusters. At coarse grain parallelism, the correlation matrix is partitioned into tiles which are distributed to execute concurrently on many GPUs using MPI. At fine grain level, the CUDA kernel function on each node performs massively parallel computing on a GPU. To balance load across all GPUs, we adopt the work pool model which there is a master node that manages tasks in the work pool and dynamically assign tasks to worker nodes. The result of the evaluation shows that the proposed work can ensure the load balance across different GPUs and thus gives better execution time than using a simple static data partitioning.

Programmable graphics processing units (GPUs) nowadays offer very high performance computing power at relatively low hardware cost and power consumption. In this paper, we present the implementation of the dynamics routine of the HIRLAM weather forecast model on the NVIDIA GeForce 9800 GX2 GPU card using the Compute Unified Device Architecture (CUDA) as parallel programming model. We converted the original Fortran to C and CUDA by hand, straightforwardly, without much concern about optimization. On a single GPU, we observe speed-ups by an order of magnitude over our hosting CPU (Intel quad core, 1998 MHz). This includes the relatively very costly copying of data between GPU and CPU memories. Calculation times proper decreased by a factor of 2000. A single GPU, however, has not enough memory for practical use. Therefore, we investigated a parallel implementation on 4 GPUs. We found a parallel speed-up of 3.6, which is not very promising if memory limitations force the use of many GPUs in parallel. We discuss several options to solve this issue.

Purpose: To develop a Graphics Processing Unit (GPU) based Monte Carlo (MC) code that uses a dual‐GPU system to accelerate radiographic simulation and CT imaging dose calculations. Methods: We considered two clinical cases, a chest x‐ray radiography and an abdominal CT scan. In the first case, a voxelized VIP‐Man phantom with detailed 3D anatomical information was used and an x‐ray beam of 120kVp was simulated. In the second case, a voxelized abdomen phantom derived from 120 CT slices was used, and a GE LightSpeed 16‐MDCT scanner was modeled. The CPU version of the MC code was written in C++ and run on Intel Xeon X5660 2.8GHz CPU, then translated into GPU code written in CUDA C and tested on a dual Tesla m2090 GPU system. The code was featured with automatic assignment of simulation task to multiple GPUs, as well as accurate calculation of energy‐ and material‐dependent cross‐sections. Results: Double‐precision floating point format was used for accuracy. In the first case, radiograph formation was simulated and doses to the organs listed in ICRP‐60 were calculated. When running on a single GPU, the MC GPU code was found to be x13 times faster than the CPU code and x29 times faster than MCNPX. In the second case, doses to the rectum, prostate, bladder and femoral heads were calculated. A speedup of x19 was observed compared to CPU code. These speedup factors were doubled on the dual‐GPU system. The imaging dose was benchmarked against MCNPX and a maximum difference of 1% was observed when the relative error is kept below 0.1%. Conclusions: A GPU‐based MC code was developed to simulate radiography and calculate imaging dose using detailed patient and CT scanner models. Efficiency and accuracy were both guaranteed in this code. Scalability of the code was confirmed on the dual‐GPU system.

Graphics Processing Units (GPUs) are having a transformational effect on numerical lattice quantum chromodynamics (LQCD) calculations of importance in nuclear and particle physics. The QUDA library provides a package of mixed precision sparse matrix linear solvers for LQCD applications, supporting single GPUs based on NVIDIA's Compute Unified Device Architecture (CUDA). This library, interfaced to the QDP++/Chroma framework for LQCD calculations, is currently in production use on the "9g" cluster at the Jefferson Laboratory, enabling unprecedented price/performance for a range of problems in LQCD. Nevertheless, memory constraints on current GPU devices limit the problem sizes that can be tackled. In this contribution we describe the parallelization of the QUDA library onto multiple GPUs using MPI, including strategies for the overlapping of communication and computation. We report on both weak and strong scaling for up to 32 GPUs interconnected by InfiniBand, on which we sustain in excess of 4 Tflops.

Most Computational Fluid Dynamics (CFD) applications solve the Naiver-Stokes equations using various discretization methods like Finite Difference Method (FDM), Finite Element Method (FEM), Finite Volume Method (FVM), etc. The most compute-intensive process in CFD algorithms is the pressure Poisson solver. Solving the pressure Poisson equation requires the solution of a set of linear simultaneous equations. As the problem size/complexity increases, so does the effort and time invested in solving the equations. This study presents fast and robust iterative solvers and the acceleration of the same using the Graphics Processing Unit (GPU) architecture. Serial versions of the codes are implemented in the C language and the parallelization on GPU is achieved using Compute Unified Device Architecture (CUDA). For this study, a two-dimensional heat conduction problem is considered. FDM is used for discretization. Iterative solvers Conjugate Gradient and Multi Grid methods have been implemented in serial and parallel. The performance enhancement with single GPU and CPU system over a single CPU in terms of computing time is reported. It is seen that multigrid algorithms are superior in convergence and nearly 13 times speed-up has been obtained through GPU acceleration using CUDA, for a grid size of 2048 X 2048. The tests have been done on a cluster with Intel(R) Xeon(R) CPU E5-2670 CPU and NVIDIA K10 GPU.

We present a spectral crystal plasticity (CP) solver for graphics processing unit (GPU) architectures that achieves a tenfold increase in efficiency over prior GPU solvers. The approach makes use of a database containing a spectral decomposition of CP simulations performed using a conventional iterative solver over a parameter space of crystal orientations and applied velocity gradients. The key improvements in efficiency come from reducing global memory transactions, exposing more instruction-level parallelism, reducing integer instructions and performing fast range reductions on trigonometric arguments. The scheme also makes more efficient use of memory than prior work, allowing for larger problems to be solved on a single GPU. We illustrate these improvements with a simulation of 390 million crystal grains on a consumer-grade GPU, which executes at a rate of 2.72 s per strain step.

In data-parallel computing that uses a graphic processing unit (GPU), processing of large data requires that multiple GPUs be used in the computer to increase its execution performance. Increasing processing performance by using multiple computing resources has been enabled by the development of computing frameworks based on a directed acyclic graph (DAG). However, their performance degrades in out-of-core processing, which often occurs in processing of large data on GPUs with limited memory capacity. The GPU data input/output (I/O) for data swapping between host memory and GPU memory during the execution of a user DAG is usually a performance bottleneck. A computing framework called is proposed to overcome this drawback. It uses a novel scheduler that involves two methods to minimize the total amount of GPU data I/O of data swapping. First, it performs locality-aware scheduling. When it schedules a task, it selects one that requires the minimum amount of data swapping and reuses as much of the data residing in GPU memory as possible. Second, it extends the locality-aware scheduling so that GPUs can execute data prefetching. Prefetching data that are swapped out from a GPU enables efficient use of bottleneck GPU I/O resources. To prefetch the input data of future tasks, it is required to determine the schedule of future tasks. Victream's scheduler (hereafter, the Victream scheduler) extends the locality-aware scheduling so that it can schedule future tasks to enable data prefetching that is executed in the way that minimizes the amount of data I/O of data swapping. Evaluation of a Victream prototype showed that the performance of Victream is better than that of conventional frameworks by up to 117%.