NMF-mGPU: non-negative matrix factorization on multi-GPU systems

A parallel Back-Propagation(BP) neural network training technique using Compute Unified Device Architecture (CUDA) on multiple Graphics Processing Units(GPUs) is proposed. To exploit the maximum performance of GPUs, we propose to implement batch mode BP training by building input neurons, hidden neurons and output neurons into matrix form. The implementation includes CUDA Basic Linear Algebra Subroutines (cuBLAS) function to perform matrix and vector operations and CUDA kernel. The proposed technique utilizes multiple GPUs to achieve further acceleration. Each GPU has the same neural network structure and weight parameter. The number of training samples are distributed to multiple GPUs. Each GPU calculates local training error and the gradient at each layer then transferred to the first GPU to calculate the summations. The summations are transferred back to each GPU to update the local weights until the training goal is achieved. A cavity microwave bandpass filter example is used to illustrate the validity of this technique.

Programmable graphics processing units (GPUs) nowadays offer very high performance computing power at relatively low hardware cost and power consumption. In this paper, we present the implementation of the dynamics routine of the HIRLAM weather forecast model on the NVIDIA GeForce 9800 GX2 GPU card using the Compute Unified Device Architecture (CUDA) as parallel programming model. We converted the original Fortran to C and CUDA by hand, straightforwardly, without much concern about optimization. On a single GPU, we observe speed-ups by an order of magnitude over our hosting CPU (Intel quad core, 1998 MHz). This includes the relatively very costly copying of data between GPU and CPU memories. Calculation times proper decreased by a factor of 2000. A single GPU, however, has not enough memory for practical use. Therefore, we investigated a parallel implementation on 4 GPUs. We found a parallel speed-up of 3.6, which is not very promising if memory limitations force the use of many GPUs in parallel. We discuss several options to solve this issue.

The Lorentz–Drude model incorporated Maxwell equations are simulated by using the three-dimensional finite difference time domain (FDTD) method and the method is parallelized on multiple graphics processing units (GPUs) for plasmonics applications. The compute unified device architecture (CUDA) is used for GPU parallelization. The Lorentz–Drude (LD) model is used to simulate the dispersive nature of materials in plasmonics domain and the auxiliary differential equation (ADE) approach is used to make it consistent with time domain Maxwell equations. Different aspects of multiple GPUs for the FDTD method are presented such as comparison of different numbers of GPUs, transfer time in between them, synchronous, and asynchronous passing. It is shown that by using multiple GPUs in parallel fashion, significant reduction in the simulation time can be achieved as compared to the single GPU.

In this paper, we propose a hybrid parallel programming approach for a numerical solution of a two-dimensional acoustic wave equation using an implicit difference scheme for a single computer. The calculations are carried out in an implicit finite difference scheme. First, we transform the differential equation into an implicit finite-difference equation and then using the alternating direction implicit (ADI) method, we split the equation into two sub-equations. Using the cyclic reduction algorithm, we calculate an approximate solution. Finally, we change this algorithm to parallelize on graphics processing unit (GPU), GPU + Open Multi-Processing (OpenMP), and Hybrid (GPU + OpenMP + message passing interface (MPI)) computing platforms. The special focus is on improving the performance of the parallel algorithms to calculate the acceleration based on the execution time. We show that the code that runs on the hybrid approach gives the expected results by comparing our results to those obtained by running the same simulation on a classical processor core, Compute Unified Device Architecture (CUDA), and CUDA + OpenMP implementations.

Graphics Processing Units (GPUs) are having a transformational effect on numerical lattice quantum chromodynamics (LQCD) calculations of importance in nuclear and particle physics. The QUDA library provides a package of mixed precision sparse matrix linear solvers for LQCD applications, supporting single GPUs based on NVIDIA's Compute Unified Device Architecture (CUDA). This library, interfaced to the QDP++/Chroma framework for LQCD calculations, is currently in production use on the "9g" cluster at the Jefferson Laboratory, enabling unprecedented price/performance for a range of problems in LQCD. Nevertheless, memory constraints on current GPU devices limit the problem sizes that can be tackled. In this contribution we describe the parallelization of the QUDA library onto multiple GPUs using MPI, including strategies for the overlapping of communication and computation. We report on both weak and strong scaling for up to 32 GPUs interconnected by InfiniBand, on which we sustain in excess of 4 Tflops.

The implementation of time-domain Green’s functions (TDGFs) in the graphics processing unit (GPU) and the central processing unit (CPU) using a finite-difference scheme is shown. The TDGFs represent the transient electric scalar and magnetic vector potentials due to a horizontal electric dipole (HED) in open-layered media. The layered media is bounded with a perfectly matched layer (PML), symmetry axis, and perfect electric conductor (PEC). We adopted four different parallel approaches as follows: 1) open multiprocessing (OpenMP) CPU implementation; 2) message passing interface (MPI) CPU implementation; 3) open accelerators (OpenACC) GPU implementation; and 4) compute unified device architecture (CUDA) GPU implementation. The accuracy and efficiency of the utilized programming models are validated by comparing and verifying the obtained results using a sequential CPU implementation. Compared to single threaded CPU implementation, speed-ups obtained by the OpenMP, MPI, OpenACC, and CUDA programming models are $4.8\times $ , $6.12\times $ , $45.97\times $ , and $96.53\times $ higher, respectively. The final result shows that GPU implementation leads to a considerable speed-up while the solution’s accuracy is fixed.

In this work we present a programming philosophy and a template code for achieving computational scalability when using multiple graphics processing units (GPUs) in the numerical solution of any mathematical system of equations found in continuum dynamic simulations. The programming philosophy exploits the principal characteristics of the GPU hardware, with emphasis in the delivering of threads with massive memory fetches, intense calculations using local registers and limited writes to global memory. The philosophy requires explicit formulas for calculations for which domain decomposition is trivial. The domains are decomposed in regions that use the local central processing unit (CPU) to communicate common interfaces using the message passing interface (MPI). A template code for the heat equation is established and tested for scalability. The novelty is that we show a series of codes, constructed from the basic template, that solve all the basic model equations found in continuum dynamics, and present illustrative results. The model equations are the heat equation, the Poisson equation, the shallow-water equations, the flow in porous media equations and the vorticity equations.

Graphic Processing Units (GPUs) have become the accelerator of choice for improving the performance of many of the most demanding applications. While performance of these devices continues to improve generation after generation, reliability of these devices has not been studied rigorously. Several sources of errors can undermine the reliability of these devices, including radiation-induced transient faults, environmental perturbations, and process, temperature or voltage variations. In particular, transient faults in GPU execution have become a significant threat to high performance computing (HPC) and safety-critical applications. HPC systems experience transient faults every few tens of hours and the trend is expected to become worse. A key point in the study of transient faults and their effects on user programs is that some faults do not cause undesirable results in the affected programs. This is especially true for GPU applications. Past efforts to study and understand GPU vulnerability to transient faults have demonstrated reliability can vary greatly across different applications. A significant amount of resilience resides intrinsically in some GPU applications. Transient faults in these applications are not likely to affect the results they produce. Other applications are highly sensitive to transient faults and are likely to crash or produce incorrect results when they are affected by transient faults. While it is generally a good idea to protect the GPU hardware from transient faults, the penalties incurred in terms of performance, power and area are not always justifiable, depending on the applications utilizing the hardware resources. Understanding the relationship between the underlying program characteristics and their implications on vulnerability is crucial. The inherent resilience in applications should be carefully considered when making decisions about designing protection mechanisms to guard against nefarious transient faults. In this thesis, we focus on program characteristics that contribute to their vulnerability. We offer several methodologies that aim at alleviating the prohibitively expensive process of quantifying and estimating the vulnerability of GPU applications, as this is the first step toward improving the reliability of GPUs. Our analyses also demonstrate that beyond the variability between the vulnerabilities of different applications, the vulnerability of GPU applications also varies during their runtime, and present a phase behavior. This phase behavior opens new opportunities for not only more efficient vulnerability estimation, but also more efficient fault mitigation approaches. We demonstrate a methodology for application designers to reduce the cost of protection against transient faults.

The Generalized Minimum Residual (GMRES) method is one of the most widely-used iterative methods for solving nonsymmetric linear systems of equations. In recent years, techniques to avoid communication in GMRES have gained attention because in comparison to floating-point operations, communication is becoming increasingly expensive on modern computers. Since graphics processing units (GPUs) are now becoming crucial component in computing, we investigate the effectiveness of these techniques on multicore CPUs with multiple GPUs. While we present the detailed performance studies of a matrix powers kernel on multiple GPUs, we particularly focus on orthogonalization strategies that have a great impact on both the numerical stability and performance of GMRES, especially as the matrix becomes sparser or ill-conditioned. We present the experimental results on two eight-core Intel Sandy Bridge CPUs with three NDIVIA Fermi GPUs and demonstrate that significant speedups can be obtained by avoiding communication, either on a GPU or between the GPUs. As part of our study, we investigate several optimization techniques for the GPU kernels that can also be used in other iterative solvers besides GMRES. Hence, our studies not only emphasize the importance of avoiding communication on GPUs, but they also provide insight about the effects of these optimization techniques on the performance of the sparse solvers, and may have greater impact beyond GMRES.

Application programming for GPUs (Graphics Processing Units) is complex and error-prone, because the popular approaches - CUDA and OpenCL - are intrinsically low-level and offer no special support for systems consisting of multiple GPUs. The SkelCL library presented in this paper is built on top of the OpenCL standard and offers pre-implemented recurring computation and communication patterns (skeletons) which greatly simplify programming for multi-GPU systems. The library also provides an abstract vector data type and a high-level data (re)distribution mechanism to shield the programmer from the low-level data transfers between the system's main memory and multiple GPUs. In this paper, we focus on the specific support in SkelCL for systems with multiple GPUs and use a real-world application study from the area of medical imaging to demonstrate the reduced programming effort and competitive performance of SkelCL as compared to OpenCL and CUDA. Besides, we illustrate how SkelCL adapts to large-scale, distributed heterogeneous systems in order to simplify their programming.

Application of graphics processing unit (GPU) to software in elementary particle/high energy physics field

<strong class="journal-contentHeaderColor">Abstract.</strong> With semiconductor technology gradually approaching its physical and thermal limits, Graphics processing unit (GPU) is becoming an attractive solution in many scientific applications due to their high performance. This paper presents an application of GPU accelerators in air quality model. We endeavor to demonstrate an approach that runs a PPM solver of horizontal advection (HADVPPM) for air quality model CAMx on GPU clusters. Specifically, we first convert the HADVPPM to a new Compute Unified Device Architecture C (CUDA C) code to make it computable on the GPU (GPU-HADVPPM). Then, a series of optimization measures are taken, including reducing the CPU-GPU communication frequency, increasing the size of data computation on GPU, optimizing the GPU memory access, and using thread and block indices in order to improve the overall computing performance of CAMx model coupled with GPU-HADVPPM (named as CAMx-CUDA model). Finally, a heterogeneous, hybrid programming paradigm is presented and utilized with the GPU-HADVPPM on GPU clusters with Massage Passing Interface (MPI) and CUDA. Offline experiment results show that running GPU-HADVPPM on one NVIDIA Tesla K40m and NVIDIA Tesla V100 GPU can achieve up to 845.4x and 1113.6x acceleration. By implementing a series of optimization schemes, the CAMx-CUDA model resulted in a 29.0x and 128.4x improvement in computational efficiency using a GPU accelerator card on a K40m and V100 cluster, respectively. In terms of the single-module computational efficiency of GPU-HADVPPM, it can achieve 1.3x and 19.4x speedup on NVIDIA Tesla K40m GPU and NVIDA Tesla V100 GPU respectively. The multi-GPU acceleration algorithm enables 4.5x speedup with 8 CPU cores and 8 GPU accelerators on V100 cluster.

<strong class="journal-contentHeaderColor">Abstract.</strong> With semiconductor technology gradually approaching its physical and thermal limits, Graphics processing unit (GPU) is becoming an attractive solution in many scientific applications due to their high performance. This paper presents an application of GPU accelerators in air quality model. We endeavor to demonstrate an approach that runs a PPM solver of horizontal advection (HADVPPM) for air quality model CAMx on GPU clusters. Specifically, we first convert the HADVPPM to a new Compute Unified Device Architecture C (CUDA C) code to make it computable on the GPU (GPU-HADVPPM). Then, a series of optimization measures are taken, including reducing the CPU-GPU communication frequency, increasing the size of data computation on GPU, optimizing the GPU memory access, and using thread and block indices in order to improve the overall computing performance of CAMx model coupled with GPU-HADVPPM (named as CAMx-CUDA model). Finally, a heterogeneous, hybrid programming paradigm is presented and utilized with the GPU-HADVPPM on GPU clusters with Massage Passing Interface (MPI) and CUDA. Offline experiment results show that running GPU-HADVPPM on one NVIDIA Tesla K40m and NVIDIA Tesla V100 GPU can achieve up to 845.4x and 1113.6x acceleration. By implementing a series of optimization schemes, the CAMx-CUDA model resulted in a 29.0x and 128.4x improvement in computational efficiency using a GPU accelerator card on a K40m and V100 cluster, respectively. In terms of the single-module computational efficiency of GPU-HADVPPM, it can achieve 1.3x and 19.4x speedup on NVIDIA Tesla K40m GPU and NVIDA Tesla V100 GPU respectively. The multi-GPU acceleration algorithm enables 4.5x speedup with 8 CPU cores and 8 GPU accelerators on V100 cluster.

Based on the principle of QR loop iterations, this paper implements a parallel algorithm based on the hardware of GPU (Graphic Process Unit) by using routines from CUDA (Computer Unified Device Architecture) to find the eigenvalues of general matrices. CPU and GPU computing card form a client-server computing framework. Here, CPU can be regarded as a client and GPU card is considered as a computing server. For the experiment environment, this paper chooses a GPU card with the model of NVIDIA GeForce GTX460 as a server side and a CPU with the model of Intel Core i5-760 quad-core as a client side. Win7 64-bit is selected as the operating system. The parallel implementation consists of two parts:PA H and PA QR. PA H is a procedure that transforms a matrix A into the Hessenberg matrix B. PA QR is the actual parallel algorithm of the QR iterations that is imposed on the Hessenberg matrix B for finding eigenvalues of the general matrix A. The speedup ratio of the proposed algorithm is jarless when the number of iterations becomes greater. The experimental results show that the parallel implementation with CUDA on GPU only makes use of less running time than traditional sequential algorithms. The speedup ratio of PA H is between 1.79 and 7.81. The speedup ratio of PA QR is between 3.24 and 118.9. Especially, when the order of general matrix equals 8192, the amount of iterations becomes 10000. The speedup ratio of the PA-H and the PA QR can run up to 7.81 and 118.9 respectively.

The small slop approximation (SSA) is an accurate method to calculate the electromagnetic (EM) scattering properties of rough surfaces. However, its computational complexity restricts its application to smaller domains and there is always the need for speedup in very large cases using pure central processing units (CPUs) hardware. With the development of graphics processing units (GPUs), more processors are dedicated to perform independent calculations. In addition, NVIDIA introduced a parallel computing platform, compute unified device architecture (CUDA), which provides researchers an easy way to use processors on GPU. To calculate EM scattering properties on GPU, we reformulate the SSA method with CUDA to take advantage of GPU threads. Because each thread executes synchronously and deals with a corresponding point data of rough surface, the CUDA-based SSA method calculates faster than the pure-CPU equivalent. To overcome memory limitations, the data of large rough surface are stored on hard disk. Moreover, a subsidiary thread is used to deal with the process of data transmission between the memory and the hard disk and reduce transmitting time further. The factors, block size, data transfers, and register, are also discussed in the optimization of the CUDA application. Test cases running on a NVIDIA GTX 460 GPU indicate that two orders of magnitude speedup, including file input and output, is obtained with our new formulation.