Parallelizing Strassen's method for matrix multiplication on distributed-memory MIMD architectures

Prior implementations of Strassen's matrix multiplication algorithm on GPUs traded additional workspace in the form of global memory or registers for time. Although Strassen's algorithm offers a reduction in computational complexity as compared to the classical algorithm, the memory overhead associated with the algorithm limits its practical utility. While there were past attempts at reducing the memory footprint of Strassen's algorithm by compromising parallelism, no prior implementation, to our knowledge, was able to hide the workspace requirement successfully. This paper presents an implementation of Strassen's matrix multiplication in eUDA, titled Multi-Stage Memory Efficient Strassen (MSMES), that eliminates additional workspace requirements by reusing and recovering input matrices. MSMES organizes the steps involved in Strassen's algorithm into five stages where multiple steps in the same stage can be executed in parallel. Two additional stages are also discussed in the paper that allows the recovery of the input matrices. Unlike previous works, MSMES has no additional memory requirements irrespective of the level of recursion of Strassen's algorithm. Experiments performed with MSMES (with the recovery stages) on NVIDIA Tesla V100 GPU and NVIDIA GTX 1660ti GPU yielded higher compute performance and lower memory requirements as compared to the NVIDIA library function for double precision matrix multiplication, cublasDgemm.

Summary form only given. We propose a new distribution scheme for a parallel Strassen's matrix multiplication algorithm on heterogeneous clusters. In the heterogeneous clustering environment, appropriate data distribution is the most important factor for achieving maximum overall performance. However, Strassen's algorithm reduces the total operation count to about 7/8 times per one recursion and, hence, the recursion level has an effect on the total operation count. Thus, we need to consider not only load balancing but also the recursion level in Strassen's algorithm. Our scheme achieves both load balancing and reduction of the total operation count. As a result, we achieve a speedup of nearly 21.7% compared to the conventional parallel Strassen's algorithm in a heterogeneous clustering environment.

Fast algorithms for matrix multiplication, namely those that perform asymptotically fewer scalar operations than the classical algorithm, have been considered primarily of theoretical interest. Apart from Strassen's original algorithm, few fast algorithms have been efficiently implemented or used in practical applications. However, there exist many practical alternatives to Strassen's algorithm with varying performance and numerical properties. Fast algorithms are known to be numerically stable, but because their error bounds are slightly weaker than the classical algorithm, they are not used even in cases where they provide a performance benefit. We argue in this paper that the numerical sacrifice of fast algorithms, particularly for the typical use cases of practical algorithms, is not prohibitive, and we explore ways to improve the accuracy both theoretically and empirically. The numerical accuracy of fast matrix multiplication depends on properties of the algorithm and of the input matrices, and we consider both contributions independently. We generalize and tighten previous error analyses of fast algorithms and compare their properties. We discuss algorithmic techniques for improving the error guarantees from two perspectives: manipulating the algorithms, and reducing input anomalies by various forms of diagonal scaling. Finally, we benchmark performance and demonstrate our improved numerical accuracy.

This paper firstly gives a description of the traditional distribution schemes of parallel matrix multiplication and an analysis of their efficiencies. In order to reduce communication expenses between multiple processors, two solutions are offered in this paper, and the corresponding specific implementation models are also listed. Then a method of how to calculate the n-th power of massive matrix is proposed. Finally, the experimental results show that this method of calculating the n-th power of massive matrix can reduce the execution time and improve the efficiency.

Parallel matrix multiplication is one of the most studied fundamental problems in distributed and high performance computing. We obtain a new parallel algorithm that is based on Strassen's fast matrix multiplication and minimizes communication. The algorithm outperforms all known parallel matrix multiplication algorithms, classical and Strassen-based, both asymptotically and in practice. A critical bottleneck in parallelizing Strassen's algorithm is the communication between the processors. Ballard, Demmel, Holtz, and Schwartz (SPAA '11) prove lower bounds on these communication costs, using expansion properties of the underlying computation graph. Our algorithm matches these lower bounds, and so is communication-optimal. It exhibits perfect strong scaling within the maximum possible range.

The presence of rounding errors is frequently inevitable when performing arithmetic operations in computers due to the use of floating-point number system whose elements have finite precision. Consequently, error-free transformation algorithms are proposed as solutions, and a particular instance of these is the error-free transformation algorithm for real matrix multiplication. Based on the previously mentioned algorithm, this study proposes two error-error free transformation algorithms for complex matrix multiplication. By conducting the numerical experiment, one finds that the combination of both proposed algorithms with an accurate summation algorithm generates the most accurate results, but it also suffers from a notable increase in the computing time ratio. On the other hand, the second proposed algorithm, which is simply another form of the first one with the change of order in summation, combined with the pure floating-point addition produces results that are nearly as accurate as those generated by the former combination. Since these results are achieved with significantly smaller computing time ratio, then the study concludes that the last combination is the best method for complex matrix multiplication in terms of accuracy and computing time efficiency.

Strassen's algorithm is an efficient method for multiplying large matrices. We explore various ways of mapping Strassen's algorithm into reconfigurable hardware that contains one or more customisable instruction processors. Our approach has been implemented using Nios processors with custom instructions and with custom-designed coprocessors, taking advantage of the additional logic and memory blocks available on a reconfigurable platform.

On practical algorithms for accelerated matrix multiplication

Strassen's algorithm for matrix multiplication gains its lower arithmetic complexity at the expense of reduced locality of reference, which makes it challenging to implement the algorithm efficiently on a modern machine with a hierarchical memory system. We report on an implementation of this algorithm that uses several unconventional techniques to make the algorithm memory-friendly. First, the algorithm internally uses a non- standard array layout known as Morton order that is based on a quad-tree decomposition of the matrix. Second, we dynamically select the recursion truncation point to minimize padding without affecting the performance of the algorithm, which we can do by virtue of the cache behavior of the Morton ordering. Each technique is critical for performance, and their combination as done in our code multiplies their effectiveness. Performance comparisons of our implementation with that of competing implementations show that our implementation often outperforms the alternative techniques (up to 25%). However, we also observe wide variability across platforms and across matrix sizes, indicating that at this time, no single implementation is a clear choice for all platforms or matrix sizes. We also note that the time required to convert matrices to/from Morton order is a noticeable amount of execution time (5% to 15%). Eliminating this overhead further reduces our execution time.

We present efficient parallel matrix multiplication algorithms for linear arrays with reconfigurable pipelined bus systems (LARPBS). Such systems are able to support a large volume of parallel communication of various patterns in constant time. An LARPBS can also be reconfigured into many independent subsystems and, thus, is able to support parallel implementations of divide-and-conquer computations like Strassen's algorithm. The main contributions of the paper are as follows. We develop five matrix multiplication algorithms with varying degrees of parallelism on the LARPBS computing model; namely, MM/sub 1/, MM/sub 2/, MM/sub 3/, and compound algorithms C/sub 1/(/spl epsiv/)and C/sub 2/(/spl delta/). Algorithm C/sub 1/(/spl epsiv/) has adjustable time complexity in sublinear level. Algorithm C/sub 2/(/spl delta/) implies that it is feasible to achieve sublogarithmic time using /spl sigma/(N/sup 3/) processors for matrix multiplication on a realistic system. Algorithms MM/sub 3/, C/sub 1/(/spl epsiv/), and C/sub 2/(/spl delta/) all have o(/spl Nscr//sup 3/) cost and, hence, are very processor efficient. Algorithms MM/sub 1/, MM/sub 3/, and C/sub 1/(/spl epsiv/) are general-purpose matrix multiplication algorithms, where the array elements are in any ring. Algorithms MM/sub 2/ and C/sub 2/(/spl delta/) are applicable to array elements that are integers of bounded magnitude, or floating-point values of bounded precision and magnitude, or Boolean values. Extension of algorithms MM/sub 2/ and C/sub 2/(/spl delta/) to unbounded integers and reals are also discussed.

The intersection of climate change and human activities exacerbates the prominent contradiction between environmental resources and economic development. The synergistic development of water resources, economy and society, and ecological environment is particularly urgent. Therefore, accurately identifying its influencing factors and related pathways holds significant practical importance in resolving the contradiction between resources and economic development. This study constructs a system of influencing factors for the synergistic development of “water resources, economic society and ecological environment” from three dimensions, reveals the influencing relationship between indicators through the Decision Making Trial and Evaluation Laboratory (DEMATEL), determines the hierarchical structure of the influencing factors and the related paths through the Interpretive Structural Model (ISM), and classifies the influencing factors through the Matrix Multiplication Method of Cross Influence Matrix Multiplication (MICMAC). The results show that water resource factors are important cause factors and economic, social and ecological factors are essential effect factors. The factors influencing the synergistic development of “water resources, economic society and ecological environment” can be divided into four levels, of which the total water resources, precipitation and NDVI are at the bottom and are the most fundamental factors. The percentage of groundwater supply, total water supply, water resource utilization rate and energy consumption of 10,000 yuan of GDP are the deep core elements. Total grain output, water resources per capita, and water use per capita are the direct influencing factors. Finally, taking the Central Plains Urban Agglomeration as an example, selecting the identified important influencing factors and analyzing the level of synergistic development of its coupled system found that the degree of coupling coordination increased from 0.324 in 2011 to 0.978 in 2020. The water resources system showed a fluctuating change, and it is still the key to synergistic development. Analyzing the relationship between subsystems and clarifying the key factors for synergistic development of coupled systems are important for policy formulation, improving environmental governance and promoting high-quality economic development.

The paper presents exact analytical solutions of one-dimensional periodic heat conduction through an inhomogeneous slab for a certain class of thermal conductivity profiles (including linear and exponential). The exact analytical solutions for some of these profiles have been compared with those obtained by considering the slab to be made up of a number of homogeneous layers with different thermal conductivities varying from layer to layer and using the layered structure (or matrix multiplication) method. The numerical results arrived at by the layered-structure method converge rapidly (with increasing number of layers considered) to the values obtained from the exact analytical solutions. This gives confidence in the application of the layered-structure method to periodic heat conduction through inhomogeneous slabs. The numerical results have been presented in the form of elements of a 2 × 2 matrix, relating the sinusoidal steady-state temperature and heat flux on the two sides of the slab.

The communication cost of algorithms (also known as I/O-complexity) is shown to be closely related to the expansion properties of the corresponding computation graphs. We demonstrate this on Strassen's and other fast matrix multiplication algorithms, and obtain first lower bounds on their communication costs. In the sequential case, where the processor has a fast memory of size $M$, too small to store three $n$-by-$n$ matrices, the lower bound on the number of words moved between fast and slow memory is, for many of the matrix multiplication algorithms, $\Omega((\frac{n}{\sqrt M})^{\omega_0}\cdot M)$, where $\omega_0$ is the exponent in the arithmetic count (e.g., $\omega_0 = \lg 7$ for Strassen, and $\omega_0 = 3$ for conventional matrix multiplication). With $p$ parallel processors, each with fast memory of size $M$, the lower bound is $p$ times smaller. These bounds are attainable both for sequential and for parallel algorithms and hence optimal. These bounds can also be attained by many fast algorithms in linear algebra (e.g., algorithms for LU, QR, and solving the Sylvester equation).

Developing an efficient algorithm for solving a large linear system in a parallel computing environment is the major problem associated with the application of parallel processing to the numerical solution of large-scale engineering problems. This paper presents a new algorithm called Multiple Sequential Staging of Tasks (MSST) to speed up the solution of a large linear system. The technique of Sequential Staging of Tasks (SST) is a highly efficient approach to the parallel solution of a large linear system, but it is not suitable for middle- and large-scale parallel computers due to the idle periods of processors. The MSST technique partitions processors into groups and makes each group start its operation from a different row of a large linear system to remove the idle period. Therefore, MSST can be performed effectively on middle- and large-scale parallel computers and achieves a higher speed-up. Numerical results were obtained from computer experiments performed with a numerical solution method of the Poisson equation on a Dawning-1000 supercomputer (a distributed-memory MIMD architecture). The parallel speed-up is satisfactory. Copyright © 1999 John Wiley & Sons, Ltd.

We present a novel heterogeneous parallel matrix multiplication algorithm that utilizes both central processing units (CPUs) and graphics processing units (GPUs) for large-scale matrices. Based on Strassen’s method, we represent matrix multiplication work as a set of matrix addition and multiplication tasks among their sub-matrices. Then, we distribute the tasks to CPUs and GPUs while considering the characteristics of the tasks and computing resources to minimize the data communication overhead and fully utilize the available computing power. To handle a large matrix efficiently with limited GPU memory, we also propose a block-based work decomposition method. We then further improve the performance of our method by exploiting the concurrent execution abilities of a heterogeneous parallel computing system. We implemented our method on five different heterogeneous systems and applied it to matrices of various sizes. Our method generally shows higher performance than the prior GPU-based matrix multiplication methods. Moreover, compared with the state-of-the-art GPU matrix multiplication library (i.e., CUBLAS), our method achieved up to 1.97 times higher performance using the same GPUs and CPU cores. In some cases, our method using a low-performance GPU (e.g., GTX 1060, 3 GB) achieved performance comparable to that of CUBLAS using a high-performance GPU (e.g., RTX 2080, 8 GB). Also, our method continually improves performance as we use more computing resources like additional CPU cores and GPUs. We could achieve such high performance because our approach fully utilized the capacities of the given heterogeneous parallel computing systems while employing the Strassen’s method, which has a lower asymptotic complexity. These results demonstrate the efficiency and robustness of our algorithm.