Generating Portable High-Performance Code via Multi-Dimensional Homomorphisms

Homomorphisms (traditionally defined on lists) are functions that can be parallelized by the divide-and-conquer paradigm. In this paper, we introduce an extension of the traditional homomorphism concept—multi-dimensional homomorphisms (MDHs)—which capture parallelism on multi-dimensional arrays. We propose md_hom—a new parallel pattern (a.k.a. algorithmic skeleton), based on the MDH concept, to simplify parallel programming for a broad class of applications. The md_hom pattern is general enough to subsume common parallel patterns such as map and reduce, and also more complex functions built by composing and nesting several patterns. We present a generic implementation schema for md_hom in form of an efficient, correct-by-construction OpenCL pseudocode that targets various parallel architectures such as multi-core CPU and graphics processing unit (GPU). We develop our pseudocode schema as parametrized in tuning parameters: these allow to optimize the code for different devices and input sizes by performing an automated search on the parameter space. We evaluate the schematically generated, executable OpenCL code using the example of general matrix–vector multiplication (GEMV)—an important linear algebra routine which has gained more attention recently due to its use in the application area of deep learning—on two parallel architectures—Intel CPU and NVIDIA GPU. Our performance results are competitive and in some cases even better than the hand-tuned GEMV implementations provided by the state-of-the-art libraries Intel MKL and NVIDIA cuBLAS, as well as the auto-tunable OpenCL BLAS library CLBlast.

Watershed analysis, as a fundamental component of digital terrain analysis, is based on the Digital Elevation Model (DEM), which is a grid (raster) model of the Earth surface and topography. Watershed analysis consists of computationally and data intensive computing algorithms that need to be implemented by leveraging parallel and high-performance computing methods and techniques. In this paper, the Multiple Flow Direction (MFD) algorithm for watershed analysis is implemented and evaluated on multi-core Central Processing Units (CPU) and many-core Graphics Processing Units (GPU), which provides significant improvements in performance and energy usage. The implementation is based on NVIDIA CUDA (Compute Unified Device Architecture) implementation for GPU, as well as on OpenACC (Open ACCelerators), a parallel programming model, and a standard for parallel computing. Both phases of the MFD algorithm (i) iterative DEM preprocessing and (ii) iterative MFD algorithm, are parallelized and run over multi-core CPU and GPU. The evaluation of the proposed solutions is performed with respect to the execution time, energy consumption, and programming effort for algorithm parallelization for different sizes of input data. An experimental evaluation has shown not only the advantage of using OpenACC programming over CUDA programming in implementing the watershed analysis on a GPU in terms of performance, energy consumption, and programming effort, but also significant benefits in implementing it on the multi-core CPU.

In this paper we investigate the dissimilar multithreading programming paradigms on x86 CPU architectures, where the recently released Intel Xeon Phi Coprocessor and commonly used Intel Xeon processors were studied, as well as the NVIDIA K20 GPU, which represents the cutting-edge general purpose graphics processing unit. The relevant numerical algorithm selected to address the problem is power method, which is widely used to compute the dominant eigenvalue of a matrix. This work focuses on dense linear algebra. The frequently used multi-core or many-core processor parallelism techniques include OpenMP, Intel Cilk Plus, Intel Threading Building Blocks, i.e. TBB, along with the optimized computing libraries such as Intel Math Kernel Library(MKL) or the NVIDIA CUDA Basic Linear Algebra Subroutines(cuBLAS) library. Optimized implementations of these techniques were separately applied to the aforementioned architectures. For the reason that a unitary programming model may not satisfy the growing performance demand, we also explored some possible mix of these languages. The study shows that the hybrid pattern of multithreading and data parallelism via explicit vectorization maximizes the performance on x86 architectures, which allows us to obtain 80% of the sustainable peak performance in double precision on the Intel Many Integrated Core(MIC) Architecture. In the case of single precision, this number reaches even 96%. In addition, this approach enables a reasonable performance by requiring least developing time. The numbers of iterations till convergence are roughly the same in both architectures of CPU and GPU. The GPU performs better in small matrix sizes. However, the Intel Xeon Phi coprocessor excels for large sizes with a better scalability.

Static timing analysis (STA) is an essential yet time-consuming task during the circuit design flow to ensure the correctness and performance of the design. Thanks to the advancement of general-purpose computing on graphics processing units (GPUs), new possibilities and challenges have arisen for boosting the performance of STA. In this work, we present an efficient and holistic GPU-accelerated STA engine. We accelerate major STA tasks, including levelization, delay computation, graph propagation, and multicorner analysis, by developing high-performance GPU kernels and data structures. By dividing the STA workloads into CPU–GPU concurrent tasks with managed dependencies, our acceleration framework supports versatile incremental updates. Furthermore, we have extended our approach to multicorner analysis by exploring a large amount of corner-level data parallelism using GPU computing. Our implementation based on the open-source STA engine OpenTimer has achieved up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.07\times $ </tex-math></inline-formula> speed-up on single corner analysis, and up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$25.67\times $ </tex-math></inline-formula> speed-up on multicorner analysis on TAU 2015 contest designs and a 14-nm technology.

Most Computational Fluid Dynamics (CFD) applications solve the Naiver-Stokes equations using various discretization methods like Finite Difference Method (FDM), Finite Element Method (FEM), Finite Volume Method (FVM), etc. The most compute-intensive process in CFD algorithms is the pressure Poisson solver. Solving the pressure Poisson equation requires the solution of a set of linear simultaneous equations. As the problem size/complexity increases, so does the effort and time invested in solving the equations. This study presents fast and robust iterative solvers and the acceleration of the same using the Graphics Processing Unit (GPU) architecture. Serial versions of the codes are implemented in the C language and the parallelization on GPU is achieved using Compute Unified Device Architecture (CUDA). For this study, a two-dimensional heat conduction problem is considered. FDM is used for discretization. Iterative solvers Conjugate Gradient and Multi Grid methods have been implemented in serial and parallel. The performance enhancement with single GPU and CPU system over a single CPU in terms of computing time is reported. It is seen that multigrid algorithms are superior in convergence and nearly 13 times speed-up has been obtained through GPU acceleration using CUDA, for a grid size of 2048 X 2048. The tests have been done on a cluster with Intel(R) Xeon(R) CPU E5-2670 CPU and NVIDIA K10 GPU.

Both size and computational activities of Vehicular Area Network (VANET) are growing. Simulation of VANETs not only requires the simulation of network standards, but also requires the mobility of nodes. Such a dynamic system involves computations of node distances, routing protocols, application layers, data send, data receive, etc. The simulation model of VANET requires both hardware and software supports to deal with massive computational problems. Currently available network simulators, like Network Simulator 3 (NS-3), are not adequate for simulating large-scale VANET systems. In this work, we propose a Compute Unified Device Architecture (CUDA)-assisted VANET simulation model for multicore Central Processing Unit (CPU) / manycore Graphics Processing Unit (GPU) platform to increase computational throughput. The proposed VANET/GPU simulator uses NS-3 as the core engine and improves throughput by exploiting massively parallel processing on the GPU. Experimental results show that the overall computation speedup can be increased up to 129x by using the proposed VANET/GPU simulator.

This paper presents a study of exact string matching algorithms and their performance behavior when executed on dynamic parallelism enabled Kepler Graphics Processing Unit (GPU) by Nvidia. The algorithms considered in this paper are Quick search (QS), Horspool (HP), and Brute force (BF) string matching. Their efficient implementation on Kepler gives a remarkable improvement over their respective multi-core CPU and generic GPU implementations. In addition, the proposed work further optimizes the algorithm by exploiting the fundamental architectural aspects of the GPU like the memory hierarchy, lightweight threads and avoiding strided access. The optimization associated with the memory is called Binning and the one using memory alignment is named Chunking. As a result of the significant boost obtained by extracting the most out of these features, the newer methods are named as SWIFT. SWIFT algorithms give performance benefit of about 1.7X on an average and up to 5X in some cases, which will be discussed in the paper. Also, the paper proposes a hybrid algorithm that employs both regular GPU implementation and SWIFT based on a predefined condition that gives benefit for all pattern sizes. The experimental results for different pattern sizes using benchmark datasets are presented in the paper.

Summary form only given. Numerical analysis of electromagnetic scattering has been widely studied in the past. The method of moments (MoM) is suitable to simulate scattering from an object in an open region. In order to solve problems with complicated geometries and materials, the finite element method (FEM) is preferred. However, when solving unbounded problems with FEM, the use of absorbing boundary conditions or perfectly matched layers will result in truncation errors. To combine the advantages of both MoM and FEM, the finite element-boundary integral (FE-BI) method was developed and widely used (J. M. Jin, The Finite Element Method in Electromagnetics, Second Edition. New York: Wiley, 2002). This algorithm is not only able to deal with highly complicated geometry and material, but also able to eliminate the truncation error. However, since the MoM has O(N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2)</sup> computational and storage complexities, the computational cost of the BI-related algorithm is still prohibitively high when solving large-scale scattering problems. To alleviate this problem, the multilevel fast multipole algorithm (MLFMA) has been incorporated into the FE-BI method (J. Liu and J. M. Jin, IEEE Trans. Antennas Propag., vol. 51, pp.1157-1167, Jun. 2003). Another important technique to accelerate the computation is to resort to parallel computers. Recently, the graphics processing unit (GPU), as one of many-core computing systems, has received intensive attention from the scientific computing community due to its low price, a massive number of processors, and the resulting high computational efficiency. In this work, we discuss the GPU implementation of the FE-BI algorithm for solving electromagnetic scattering problems. First, the assembly of the BI-related matrix is parallelized on GPU by NVIDIA's compute unified device architecture (CUDA). A massive number of threads are assigned to carry out the assembly, and each thread computes one element in the BI-related matrix. Second, both the direct and iterative solvers are implemented on GPU. For the direct solver, the Intel's math kernel library (MKL) is used for LU decomposition, and the CUSPARSE (a set of basic linear algebra subroutines used for handling sparse matrices) is employed to speedup forward and backward substitutions. For the iterative solver, the high performance CUDA Basic Linear Algebra Subprograms (BLAS) is utilized for the matrix-vector multiplication and the vector-vector multiplication. Extensive numerical experiments are conducted to validate the implementation and demonstrate the achievable computational speedup.

Performance and efficiency became recently key requirements of computer architectures. Modern computers incorporate Graphics Processing Units (GPUs) into running data mining algorithms, as well as other general purpose computations. In this paper, different parallelization methods are analyzed and compared in order to understand their applicability. From multi-threading on shared memory to using NVIDIA’s GPU accelerators for increasing performance and efficiency on parallel computing, this work discusses the parallelization of data mining algorithms considering performance and efficiency issues. The performance is compared on both many-core systems and GPU accelerators on a distance measure algorithm using a relatively big data set. We optimize the way we deal with GPUs in heterogeneous systems to make them more suitable for big data mining applications with heavy distance calculations. Moreover, we focus on achieving a higher utilization of GPU resources and a better reuse of data. Our implementation of the content-based similarity algorithm SQFD on the GPU outperforms by up to 50× CPU counterparts, and up to 15× CPU multi-threaded implementations.

Over the last decade, many-core Graphics Processing Units (GPUs) have been widely used to accelerate a variety of applications.Meanwhile, Intel has released its Xeon Phi Coprocessor, which is equipped with more than fifty x86 cores, each supporting four hardware threads.Despite their widespread use, many-core processors are still considered relatively difficult to program, in that they require the programmer to be familiar with both parallel programming and the hardware features of these devices.

Machine learning (ML) and high-performance computing (HPC) applications increasingly combine dense and sparse memory access computations to maximize storage efficiency. However, existing central processing units (CPUs) and graphics processing units (GPUs) struggle to flexibly handle these heterogeneous workloads with consistently high compute efficiency. We present Occamy, a 432-core, 768-DP-GFLOP/s, dual-HBM2E, dual-chiplet RISC-V system with a latency-tolerant hierarchical interconnect and in-core streaming units (SUs) designed to accelerate dense and sparse FP8-to-FP64 ML and HPC workloads. We implement Occamy’s compute chiplets in 12-nm FinFET and its passive interposer, Hedwig, in a 65-nm node. On dense linear algebra (LA), Occamy achieves a competitive floating-point unit (FPU) utilization of 89%. On stencil codes, Occamy reaches an FPU utilization of 83% and a technology-node-normalized compute density of 11.1 DP-GFLOP/s/mm2, leading state-of-the-art (SoA) processors by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.7\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.2\times $ </tex-math></inline-formula>, respectively. On sparse-dense LA, it achieves 42% FPU utilization and a normalized compute density of 5.95 DP-GFLOP/s/mm2, surpassing the SoA by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.2\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11\times $ </tex-math></inline-formula>, respectively. On sparse–sparse LA, Occamy reaches a throughput of up to 187 GCOMP/s at 17.4 GCOMP/s/W and a compute density of 3.63 GCOMP/s/mm2. Finally, we reach up to 75% and 54% FPU utilization on and dense (large language model) and graph-sparse (graph convolutional network) ML inference workloads. Occamy’s register transfer level (RTL) description is freely available under a permissive open-source license.

We present an implementation of general FFTs for graphics processing units (GPUs). Unlike most existing GPU FFT implementations, we handle both complex and real data of any size that can fit in a texture. The basic building block for our algorithms is a radix-2 Stockham formulation of the FFT for power-of-two data sizes that avoids expensive bit reversals and exploits the high GPU memory band-width efficiently. We implemented our algorithms using the DirectX9 API, which enables our routines to be used on many of the existing GPUs today. We have performed comparisons against optimized CPU-based and GPU-based FFT libraries (Intel Math Kernel Library and NVIDIA CUFFT, respectively). Our results on a NVIDIA GeForce 8800 GTX GPU indicate a significant performance improvement over the existing libraries for many input cases.

Large scale artificial neural networks (ANNs) have been widely used in data processing applications. The recurrent neural network (RNN) is a special type of neural network equipped with additional recurrent connections. Such a unique architecture enables the recurrent neural network to remember the past processed information and makes it an expressive model for nonlinear sequence processing tasks. However, the large computation complexity makes it difficult to effectively train a recurrent neural network and therefore significantly limits the research on the recurrent neural network in the last 20 years. In recent years, the use of graphics processing units (GPUs) becomes a significant advance to speed up the training process of large scale neural networks by taking advantage of the massive parallelism capabilities of GPUs. In this paper, we propose an efficient GPU implementation of the large scale recurrent neural network and demonstrate the power of scaling up the recurrent neural network with GPUs. We first explore the potential parallelism of the recurrent neural network and propose a fine-grained two-stage pipeline implementation. Experiment results show that the proposed GPU implementation can achieve 2 ~ 11 x speed-up compared with the basic CPU implementation with the Intel Math Kernel Library. We then use the proposed GPU implementation to scale up the recurrent neural network and improve its performance. The experiment results of the Microsoft Research Sentence Completion Challenge demonstrate that the large scale recurrent network without class layer is able to beat the traditional class-based modest-size recurrent network and achieve an accuracy of 47%, the best result achieved by a single recurrent neural network on the same dataset.

SummaryThe stencil pattern is important in many scientific and engineering domains, spurring great interest from researchers and industry. In recent years, various optimizations have been proposed for parallel stencil applications running on graphics processing units (GPUs). In particular, tiling is a technique that can significantly enhance application performance by improving data locality and by reducing the volume of communication between host memory and GPU. In addition, tiling enables stencil applications to process inputs that are larger than the physical GPU memory. However, implementing tiling efficiently is complex, time‐consuming, and error‐prone. In this paper, we propose transparently optimized automatic stencil tiling (TOAST), an automatic tiling mechanism for iterative stencil computations running on GPUs; TOAST has 3 main benefits: (1) It incorporates an optimization model that seeks to maximize data reuse within tiles while respecting the amount of dynamically available GPU memory; (2) it offers a virtualized GPU memory for stencil computations, allowing for large input data; and (3) it performs optimal tiling transparently to the developer of the parallel stencil application. The current implementation of TOAST augments the PSkel framework with an internal solver based on genetic algorithms. Our experimental results show that TOAST improves the performance of iterative stencil applications by up to 13 × compared with their multithreaded (central processing unit–based) optimized versions and up to 48 × compared with a naive tiling approach on GPU. The TOAST mechanism is able to automatically achieve a low percentual overhead of data management compared with actual stencil computation.

Thanks to their massive computational power and their SIMT computational model, Graphics Processing Units (GPUs) have been successfully used to accelerate a wide variety of regular applications (linear algebra, stencil computations, image processing and bioinformatics algorithms, among others). However, many established and emerging problems are based on irregular data structures, such as graphs. Examples can be drawn from different application domains: networking, social networking, machine learning, electrical circuit modeling, discrete event simulation, compilers, and computational sciences. It has been shown that irregular applications based on large graphs do exhibit runtime parallelism; moreover, the amount of available parallelism tends to increase with the size of the datasets. In this work, we explore an implementation space for deploying a variety of graph algorithms on GPUs. We show that the dynamic nature of the parallelism that can be extracted from graph algorithms makes it impossible to find an optimal solution. We propose a runtime system able to dynamically transition between different implementations with minimal overhead, and investigate heuristic decisions applicable across algorithms and datasets. Our evaluation is performed on two graph algorithms: breadth-first search and single-source shortest paths. We believe that our proposed mechanisms can be extended and applied to other graph algorithms that exhibit similar computational patterns.