Review on Sparse Matrix Storage Formats With Space Complexity Analysis

Numerous applications in science and engineering rely on sparse linear algebra. The efficiency of a fundamental kernel such as the Sparse Matrix-Vector multiplication (SpMV) is crucial for solving increasingly complex computational problems. However, the SpMV is notorious for its extremely low arithmetic intensity and irregular memory patterns, posing a challenge for optimization. Over the last few years, an extensive amount of literature has been devoted to implementing SpMV on Graphic Processing Units (GPUs), with the aim of exploiting the available fine-grain parallelism and memory bandwidth. In this paper, we propose to efficiently combine adaptivity and compression into an ELL-based sparse format in order to improve the state-of-the-art of the SpMV on Graphic Processing Units (GPUs). The foundation of our work is AdELL, an efficient sparse data structure based on the idea of distributing working threads to rows according to their computational load, creating balanced hardware-level blocks (warps) while coping with the irregular matrix structure. We designed a lightweight index compression scheme based on delta encoding and warp granularity that can be transparently embedded into AdELL, leading to an immediate performance benefit associated with the bandwidth-limited nature of the SpMV. The proposed integration provides a highly-optimized novel sparse matrix format known as Compressed Adaptive ELL (CoAdELL). We evaluated the effectiveness of our approach on a large set of benchmarks from heterogeneous application domains. The results show consistent improvements for double-precision SpMV calculations over the AdELL baseline. Moreover, we assessed the general relevance of CoAdELL with respect to other optimized GPU-based sparse matrix formats. We drew a direct comparison with clSpMV and BRO-HYB, obtaining sufficient experimental evidence (33% geometric average improvement over clSpMV and 43% over BRO-HYB) to propose our research work as the novel state-of-the-art.

This paper presents an efficient strategy to perform the assembly stage of finite element analysis (FEA) on general purpose graphics processing units (GPUs). This strategy involves dividing the assembly task using symbolic and numeric kernels, and thereby reducing the complexity of the standard single-kernel assembly approach. Two sparse storage formats based on the proposed strategy are also developed by modifying the existing sparse storage formats with the intention of removing the degrees-of-freedom-based redundancies in the global matrix. The inherent problem of race condition is resolved through the implementation of coloring and atomics. The proposed strategy is compared with the state-of-the-art GPU-based and central processing unit (CPU)-based assembly techniques. These comparisons reveal a significant number of benefits in terms of reducing storage space requirements and execution time and increasing performance (GFLOPS). Moreover, using the proposed strategy, it is found that the coloring method is more effective compared to the atomics-based method for the existing as well as the modified storage formats.

Sparse Matrix-Vector Multiplication (SpMV) kernel dominates the computing cost in numerous scientific applications. Many implementations based on different sparse formats were proposed recently for this kernel on the GPU side. Since the performance of these sparse formats varies significantly according to the sparsity characteristics of the input matrix and the hardware specifications, no one of them can be considered as the best one to use for every sparse matrix. In this paper, we address the problem of selecting the best representation for a given sparse matrix on GPU by using a machine learning approach. First, we present some interesting and easy to compute features for characterizing the sparse matrices on GPU. Second, we use a multiclass Support Vector Machine (SVM) classifier to select the best format for each input matrix. We consider in this paper four popular formats (COO, CSR, ELL, and HYB), but our work can be extended to support more sparse representations. Experimental results on two different GPUs (Fermi GTX 580 and Maxwell GTX 980 Ti) show that we achieved more than 98% of the performance possible with a perfect selection.

Parallel processing with minicomputers

For the time-domain linear system with diagonal sparse matrices, based on the popular preconditioned generalized minimum residual method (GMRES), we proposed an efficient solving algorithm on the graphics processing unit (GPU), which is called T-GMRES. In the proposed T-GMRES, three are the following novelties: (1) a new sparse storage format BRCSD is presented to alleviate the drawback of the diagonal format (DIA) that a large number of zeros are filled to maintain the diagonal structure when many diagonals are far away from the main diagonal; (2) an efficient sparse matrix-vector multiplication on GPU for BRCSD is proposed; and (3) for assembling the sparse matrix for BRCSD and the vector efficiently on GPU, a new kernel is suggested. The experimental results have validated the high efficiency and good performance of our proposed algorithm.

An acceleration strategy for TV-ADM reconstruction algorithm in Compton scattering tomography (CST) is proposed. By analyzing the sparse characteristic of CST projection matrixes, firstly, the sparse matrix vector CSR format and ELL format are used to store them, which greatly reduce the memory consumption. Then, a Sparse Matrix Vector multiplication (SpMV) method is utilized to accelerate the projector and back projector process. Finally, based on the parallel features, the TV-ADM is computed with Graphics Processing Unit (GPU). Numerical experiments show that the TV-ADM with the presented acceleration strategy could achieve a 96 times speedup ratio and 224 times memory compression ratio without precision loss.

Matrix computation is a key technology in various data processing tasks including data mining, machine learning, and information retrieval. Size of matrices has been increasing with the development of computational resources and dissemination of big data. Huge matrices are memory- and computational-time-consuming. Therefore, reducing the size and computational time of huge matrices is a key challenge in the data processing area. We develop MOARLE, a novel matrix computation framework that saves memory space and computational time. In contrast to conventional matrix computational methods that target to sparse matrices, MOARLE can efficiently handle both sparse matrices and dense matrices. Our experimental results show that MOARLE can reduce the memory usage to 2% of the original usage and improve the computational performance by a factor of 124x.

The Multiple Associative Computing (MASC) parallelmodel is a generalization model of an AssociativeComputing (ASC) parallel model designed to supportmultiple ASC data parallel threads by using controlparallelism.

The authors present a software tool for mapping neural network computations onto mesh-connected SIMD (single instruction stream multiple data stream) computers. The tool, called NetMap, contains complete mapping programs for built-in network models and a library of elemental routing routines. For built-in models the user is required to provide only minimum information about the network. For example, for a backpropagation network with fully connected layers, it is enough to provide the number of neurons in each layer of the network. If the network has been pruned or structured in some other way, so that the layers are no longer fully interconnected, a complete connectivity matrix needs to be provided. Thus, NetMap can be used for networks of arbitrary topology. >

Due to the high-sped development in semiconductor technology, general-purpose computers can process digital video, audio and graphics easily. Nowadays, the similar functionality has shifted from the general-purpose computer to the so-called media-processor one. Since the processor is targeted on mass-produced consumer electronic devices, the price and the ease of use are the main concern. The latest MPEG-4 video coding standard, which consumes relatively low bit-rate but yields acceptable quality, meets the target exactly. In this paper, by realizing an MPEG-4 codec using both general-purpose processor and media-processor, we investigated some issues related to the use of a media-processor for satisfying various multimedia compression and processing requirements. The investigated issues include the speedup of the MPEG-4 decoder based on a media-processor DSP chip, a multimedia co-processor or hardware accelerator, the very long instruction word (VLIW) and the single instruction multiple data stream (SIMD) programming techniques. Moreover, a media-processor based MPEG-4 solution for set-top box application is also included.

A prototype OCR is constructed using a very compact parallel processing unit. This unit, designed for interactive character recognition applications, is equipped with a hand-scanner for input and a personal computer for word and/or image processing. The heart of this unit is a bit-serial Single Instruction Multiple Data stream (SIMD) array processor constructed with four identical cellular array LSIs (AAP2). The processor is fully programmable and the complex pro-cess of Japanese character recognition can be carried out with a single program package. Its architecture permits flexible and high-speed SIMD operations to process bitline data such as local fields of scanned documents. The processor components were integrated into one board and confirmed to be more than ten times faster than present image processors of the same size through various image processing tests. High character recognition performance is obtained at a reading speed of 8 Japanese characters per second which is sufficient for hand-scanning data input operations. The recognition rate is higher than 98% for about 3,300 Japanese characters.

In this paper, we present a new sparse matrix data format that leads to improved memory coalescing and more efficient sparse matrix-vector multiplication for a wide range of problems on high-throughput architectures such as a GPU. The sparse matrix structure is constructed by sorting the rows based on the row length (defined as the number of non-zero elements in a matrix row) followed by a partition into two ranges, short rows and long rows. Based on this partition, the matrix entries are then transformed into ELLPACK or vectorized compressed sparse row format. In addition, the number of threads are adaptively selected by their row length, in order to balance the workload for each graphics processing unit thread. Several computational experiments are presented to support this approach and the results suggest a notable improvement over a wide range of matrix structures.

Integrating batched sparse iterative solvers for the collision operator in fusion plasma simulations on GPUs

The calculation of pairwise correlation coefficient on a dataset, known as the correlation matrix, is often used in data analysis, signal processing, pattern recognition, image processing, and bioinformatics. With the state-of-the-art Graphic Processing Units (GPUs) that consist of massive cores capable to do processing up to several Gflops, the calculation of correlation matrix can be accelerated several times over traditional CPUs. However, due to the rapid growth of the data in the digital era, the correlation matrix calculation becomes computing intensive which needs to be executed on multiple GPUs. As of now, GPUs are common components in data center at many institutions. Their GPU deployment tends towards a GPU cluster which each node is equipped with GPUs. In this paper, we propose a parallel computing based on the hybrid MPI/CUDA programming for fast and efficient Pearson correlation matrix calculation on GPU clusters. At coarse grain parallelism, the correlation matrix is partitioned into tiles which are distributed to execute concurrently on many GPUs using MPI. At fine grain level, the CUDA kernel function on each node performs massively parallel computing on a GPU. To balance load across all GPUs, we adopt the work pool model which there is a master node that manages tasks in the work pool and dynamically assign tasks to worker nodes. The result of the evaluation shows that the proposed work can ensure the load balance across different GPUs and thus gives better execution time than using a simple static data partitioning.

The aim of this paper is to explore the use of the Graphic Processing Unit (GPU) for mask design using inverse lithography technique (ILT). We extend a newly proposed ILT algorithm called cost-function-reduction method (CFRM) to general partial-coherence image systems. To release heavy computational cost in this algorithm such as intensity computation, the algorithm is modified for GPU implementation. The scalability of the GPU implementation is demonstrated using different sizes of matrix in incoherence image system and partial-coherence image system. The total GPU optimization time for a 25μm×25μm mask in partial-coherence image model is about 8.2 second. About 15X performance increase have been achieved than that of an algorithm solely implemented on a high-end CPU. The maximum mask size is limited by GPU card memory.