An Analytical Model for Matrix Multiplication on Many Threaded Vector Processors

Parallel matrix multiplication is one of the most studied fundamental problems in distributed and high performance computing. We obtain a new parallel algorithm that is based on Strassen's fast matrix multiplication and minimizes communication. The algorithm outperforms all known parallel matrix multiplication algorithms, classical and Strassen-based, both asymptotically and in practice. A critical bottleneck in parallelizing Strassen's algorithm is the communication between the processors. Ballard, Demmel, Holtz, and Schwartz (SPAA '11) prove lower bounds on these communication costs, using expansion properties of the underlying computation graph. Our algorithm matches these lower bounds, and so is communication-optimal. It exhibits perfect strong scaling within the maximum possible range.

Describes the important role of decomposition strategy in parallel computing, and its influence is illustrated with the example of a very frequently used parallel algorithm for matrix multiplication. On the basis of the performed analysis of the parallel computers used worldwide divides them into the two basic groups which are considered from the programmer‐developer point of view to be very different. They also introduce the typical principal structures for both these groups of parallel computers and also their models. The paper then in an illustrative way describes the development of concrete parallel algorithm for matrix multiplication on various parallel systems. For each individual practical implementation of matrix multiplication there is introduced the derivation of its calculation complexity. The individual ways of developing parallel matrix multiplication and their implementations are compared, analysed and discussed from the point of view of the programmer‐developer and user. The very important role of the decomposition strategies that are mainly of the class of asynchronous parallel computers is highlighted.

In order to solve the problem of parallel algorithm for matrix multiplication,according to the characteristic of Biswapped network,a parallel algorithm for matrix multiplication based on Biswapped network was developed. The algorithm is simple in operation and easy to implement by using a new suggested mapping scheme. It is showed in theoretic analysis and experimental test,the proposed algorithm equals to Cannon's approximately.

For many scientific applications, dense matrix multiplication is one of the most important and computation intensive linear algebra operations. An efficient matrix multiplication on high performance and parallel computers requires optimizations on how matrices are decomposed and exchanged between computational nodes to reduce communication and synchronization overhead, as well as to efficiently exploit the memory hierarchy within a node to improve both spatial and temporal data locality. In this paper, we presented our studies of performance, cache behavior, and energy efficiency of multiple parallel matrix multiplication algorithms on a multicore desktop computer and a medium-size shared memory machine, both being considered as referenced sizes of nodes to create a medium- and largescale computational clusters for high performance computing used in industry and national laboratories. Our results highlight both the performance and energy efficiencies, and also provide implications on the memory and resources pressures of those algorithms. We hope this could help users choose the appropriate implementations according to their specific data sets when composing larger-scale scientific applications that use parallel matrix multiplication kernels on a node.

The author presents a new parallel matrix multiplication algorithm on distributed memory concurrent computers, which is fast and scalable, and whose performance is independent of data distribution on processors, and calls it DIMMA (Distribution-Independent Matrix Multiplication Algorithm). The algorithm is based on two new ideas; it uses a modified pipelined communication scheme to overlap computation and communication effectively and exploits the LCM block concept to obtain the maximum performance of the sequential BLAS routine in each processor even when the block size is very small as well as very large. The algorithm is implemented and compared with SUMMA on the Intel Paragon computer.

Utilizing all CPU cores available for numerical computations is a topic of considerable interest in HPC. This paper analyzes and compares four different parallel algorithms for matrix multiplication without block partitioning using OpenMP. The comparison of the algorithms is based on the achieved speed, memory bandwidth and efficient use of the cache of the algorithms.

We present a novel heterogeneous parallel matrix multiplication algorithm that utilizes both central processing units (CPUs) and graphics processing units (GPUs) for large-scale matrices. Based on Strassen’s method, we represent matrix multiplication work as a set of matrix addition and multiplication tasks among their sub-matrices. Then, we distribute the tasks to CPUs and GPUs while considering the characteristics of the tasks and computing resources to minimize the data communication overhead and fully utilize the available computing power. To handle a large matrix efficiently with limited GPU memory, we also propose a block-based work decomposition method. We then further improve the performance of our method by exploiting the concurrent execution abilities of a heterogeneous parallel computing system. We implemented our method on five different heterogeneous systems and applied it to matrices of various sizes. Our method generally shows higher performance than the prior GPU-based matrix multiplication methods. Moreover, compared with the state-of-the-art GPU matrix multiplication library (i.e., CUBLAS), our method achieved up to 1.97 times higher performance using the same GPUs and CPU cores. In some cases, our method using a low-performance GPU (e.g., GTX 1060, 3 GB) achieved performance comparable to that of CUBLAS using a high-performance GPU (e.g., RTX 2080, 8 GB). Also, our method continually improves performance as we use more computing resources like additional CPU cores and GPUs. We could achieve such high performance because our approach fully utilized the capacities of the given heterogeneous parallel computing systems while employing the Strassen’s method, which has a lower asymptotic complexity. These results demonstrate the efficiency and robustness of our algorithm.

Summary form only given. We propose a new distribution scheme for a parallel Strassen's matrix multiplication algorithm on heterogeneous clusters. In the heterogeneous clustering environment, appropriate data distribution is the most important factor for achieving maximum overall performance. However, Strassen's algorithm reduces the total operation count to about 7/8 times per one recursion and, hence, the recursion level has an effect on the total operation count. Thus, we need to consider not only load balancing but also the recursion level in Strassen's algorithm. Our scheme achieves both load balancing and reduction of the total operation count. As a result, we achieve a speedup of nearly 21.7% compared to the conventional parallel Strassen's algorithm in a heterogeneous clustering environment.

Winograd’s algorithm to multiply two n × n matrices reduces the asymptotic operation count from O(n 3) of the traditional algorithm to O(n 2.81), hence on distributed memory computers, the combination of Winograd’s algorithm and the parallel matrix multiplication algorithms always gives remarkable results. Within this combination, the application of Winograd’s algorithm at the inter-processor level requires us to solve more difficult problems but it leads to more effective algorithms. In this paper, a general formulation of these algorithms will be presented. We also introduce a scalable method to implement these algorithms on distributed memory computers. This work also opens a new direction to parallelize Winograd’s algorithm based on the generalization of Winograd’s formula for the case where the matrices are partitioned into 2 k parts (the case k = 2 gives us the original formula).

This paper presents a natural and efficient implementation for the classical broadcast message passing routine which optimizes performance of Ethernet based clusters. A simple algorithm for parallel matrix multiplication is specifically designed to take advantage of both, parallel computing facilities (CPUs) provided by clusters, and optimized performance of broadcast messages on Ethernet based clusters. Also, this simple parallel algorithm proposed for matrix multiplication takes into account the possibly heterogenous computing hardware and maintains a balanced workload of computers according to their relative computing power. Performance tests are presented on a heterogenous cluster as well as on a homogeneous cluster, where it is compared with the parallel matrix multiplication provided by the ScaLAPACK library. Another simple parallel algorithm is proposed for LU matrix factorization (a general method to solve dense systems of equations) following the same guidelines used for the parallel matrix multiplication algorithm. Some performance tests are presented over a homogenous cluster.

Graphics processing units (GPUs) have shown increased popularity and play an important role as kind of coprocessor in heterogeneous co-processing environment. Heavily data parallel problems can be solved efficiently due to tens of thousands threads collaborative work in parallel GPU architecture. The achieved performance, therefore, depends on the capability of multiple threads in parallel collaboration. This paper, a static analytical kernel performance model (SAKP) was proposed to estimate the execution time of GPU kernel. Especially, a set of kernel and device features for target GPU is generated in the proposed model. We determine the performance limiting factors and generate an estimation of kernel execution time with this model. Matrix multiplication (MM) and histogram generation (HG) in NVIDIA GTX680 GPU card were performed to verify our proposed model and showed an absolute error in prediction less than 6.8%.

We design a parallel huge matrix multiplication algorithm on a cluster of GPU nodes. Since input matrices are too big to accommodate in the memory, the algorithm repeats the loading, computing, storing partial matrix data from/to disk and GPU buffer. The key to achieve the best speedup is not only to use GPU with full performance, but to reduce the overhead in data movement between disk and GPU buffer. We devise an efficient way to lower the latency of supplying the matching pair of the partial matrices to the GPU buffer, and to optimize the data partition, distribution, and disk access using the pipelined way. Experimental results show our algorithm outperforms a generic algorithm, resulting in the computing time reduction by 45%. Also, the scalability of the algorithm enhances with more GPU nodes.

Analytical modeling is applied to the automated design of application-specific superscalar processors. Using an analytical method bridges the gap between the size of the design space and the time required for detailed cycle-accurate simulations. The proposed design framework takes as inputs the design targets (upper bounds on execution time, area, and energy), design alternatives, and one or more application programs. The output is the set of out-of-order superscalar processors that are Pareto-optimal with respect to performance-energy-area. The core of the new design framework is made up of analytical performance and energy activity models, and an analytical model-based design optimization process.For a set of benchmark programs and a design space of 2000 designs, the design framework arrives at all performance-energy-area Pareto-optimal design points within 16 minutes on a 2 GHz Pentium-4. In contrast, it is estimated that a naive cycle-accurate simulation-based exhaustive search would require at least two months to arrive at the Pareto-optimal design points for the same design space.

We present efficient parallel matrix multiplication algorithms for linear arrays with reconfigurable pipelined bus systems (LARPBS). Such systems are able to support a large volume of parallel communication of various patterns in constant time. An LARPBS can also be reconfigured into many independent subsystems and, thus, is able to support parallel implementations of divide-and-conquer computations like Strassen's algorithm. The main contributions of the paper are as follows. We develop five matrix multiplication algorithms with varying degrees of parallelism on the LARPBS computing model; namely, MM/sub 1/, MM/sub 2/, MM/sub 3/, and compound algorithms C/sub 1/(/spl epsiv/)and C/sub 2/(/spl delta/). Algorithm C/sub 1/(/spl epsiv/) has adjustable time complexity in sublinear level. Algorithm C/sub 2/(/spl delta/) implies that it is feasible to achieve sublogarithmic time using /spl sigma/(N/sup 3/) processors for matrix multiplication on a realistic system. Algorithms MM/sub 3/, C/sub 1/(/spl epsiv/), and C/sub 2/(/spl delta/) all have o(/spl Nscr//sup 3/) cost and, hence, are very processor efficient. Algorithms MM/sub 1/, MM/sub 3/, and C/sub 1/(/spl epsiv/) are general-purpose matrix multiplication algorithms, where the array elements are in any ring. Algorithms MM/sub 2/ and C/sub 2/(/spl delta/) are applicable to array elements that are integers of bounded magnitude, or floating-point values of bounded precision and magnitude, or Boolean values. Extension of algorithms MM/sub 2/ and C/sub 2/(/spl delta/) to unbounded integers and reals are also discussed.

In this paper, we give a practical high performance parallel implementation of Strassen’s algorithm for matrix multiplication. We show how under restricted conditions, this algorithm can be implemented plug compatible with standard parallel matrix multiplication algorithms. Results obtained on a large Intel Paragon system show a 10– 20% reduction in execution time compared to what we believe to be the fastest standard parallel matrix multiplication implementation available at this time.