Discovering faster matrix multiplication algorithms with reinforcement learning

Interactions of zeros of polynomials and multiplicity matrices

Coded Distributed Matrix Multiplication (CDMM) is a distributed matrix multiplication (DMM) for large-scale matrices through a coding scheme such that any <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</i> worker nodes among all <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> worker nodes can recover the final product, where <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> corresponds to the length of the code and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</i> ≤ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> is called the recovery threshold. The state-of-art CDMM schemes, such as EP codes for Single DMM and GCSA codes for batch DMM, are defined over a Galois field GF(<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</i>) of size <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</i> ≥ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i>. These are inefficient for small Galois fields such as GF(2) and the integer residue ring Z<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>p</i></sub><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ</sup> due to the lack of invertible elements for interpolation. DMM over Z<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">_p</i><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ</sup> (such as Z<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">64</sup> ) is well-motivated in practice due to their direct compatibility with hardware. In this work, we construct efficient CDMM over the Galois ring GR(p<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ</sup>, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</i>) which is an extension ring over Z<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>p</i></sub><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ</sup> of degree <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</i>, particularly, GR(<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p, d</i>) = GF(<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sup>) is the Galois field and GR(<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>p</i></sub><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ</sup>, 1) = Z<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">_p</i><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ</sup>. We first give a general CDMM framework for the batch of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> matrix multiplications via the famous RMFE (Cascudo et al. Crypto’18). Compared with GCSA, our construction has a smaller recovery threshold by a factor of 1/<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i>. Next, we optimize EP codes via batch preprocessing of the input matrices. We give two types of Single CDMM, which can achieve almost the same performance as EP codes over a Galois field with size <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</i> ≥ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i>. Finally, we present the experimental analysis of our CDMM on Galois rings.

An $m \times (n+1)$ multiplicity matrix is a matrix $M = ( \mu_{i,j} )$ with rows enumerated by $i \in \{ 1,\ 2, \ldots, m \}$ and columns enumerated by $j \in \{ 0,1,\ldots, n \}$ whose coordinates are nonnegative integers satisfying the following two properties: (1) If $\mu_{i,j} \geq 1$, then $j \leq n-1$ and $\mu_{i,j+1} = \mu_{i,j}-1$, and (2) the $j$th column sum of $M$ satisfies the inequality $\sum_{i=1}^{m} \mu_{i, j} \leq n-j$ for all $j$. Let $K$ be a field of characteristic 0 and let $f(x)$ be a polynomial of degree $n$ with coefficients in $K$. Let $f^{(j)}(x)$ be the $j$th derivative of $f(x)$. Let $\Lambda = ( \lambda_1,\ldots, \lambda_{m})$ be a sequence of distinct elements of $K$. For $i \in \{1, 2, \ldots, m \}$ and $j \in \{1,2,\ldots, n\}$, let $ \mu_{i,j}$ be the multiplicity of $\lambda_i$ as a zero of the polynomial $f^{(j)}(x)$. The $m \times (n+1)$ matrix $M_f(\Lambda) = ( \mu_{i,j} )$ is called the multiplicity matrix of the polynomial $f(x)$ with respect to $\Lambda$. An open problem is to classify the multiplicity matrices that are multiplicity matrices of polynomials in $K[x]$ and to construct multiplicity matrices that are not multiplicity matrices of polynomials.

Transformer and its variants, built based on attention mechanisms, have recently achieved remarkable performance in many NLP tasks. Most existing works on Transformer explanation tend to reveal and utilize the attention matrix with human subjective intuitions in a qualitative manner. However, the huge size of dimensions directly challenges these methods to quantitatively analyze the attention matrix. Therefore, in this paper, we propose a novel reinforcement learning (RL) based framework for Transformer explanation via attention matrix, namely AttExplainer. The RL agent learns to perform step-by-step masking operations by observing the change in attention matrices. We have adapted our method to two scenarios, perturbation-based model explanation and text adversarial attack. Experiments on three widely used text classification benchmarks validate the effectiveness of the proposed method compared to state-of-the-art baselines. Additional studies show that our method is highly transferable and consistent with human intuition. The code of this paper is available at https://github.com/niuzaisheng/AttExplainer .

Matrix-vector multiplication (MVM) is the core operation of many important algorithms. Crosspoint resistive memory array enables naturally calculating MVM in one operation, thus representing a highly promising computing accelerator for various applications. To evaluate computing performance as well as scalability of in-memory MVM, the fundamental issue of time complexity of the circuit shall be elaborated. Based on the most common MVM circuit that uses transimpedance amplifiers to read out current product in crosspoint array, we analyze its dynamic response and the corresponding time complexity. The result shows that the computing time is governed by the maximal row sum of the implemented matrix, which leads to an explicit time complexity for a specific dataset, e.g., O( N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sup> ) and O(ln N) for discrete cosine transformation and Toeplitz matrix, respectively. By changing accordingly feedback conductance of transimpedance amplifier for different matrix sizes, it is possible to reduce the time complexity to O(1). Impact of non-ideal factors of the circuit on computing time is also studied. This work provides an insight into the performance and its improvement of MVM computation for efficient in-memory computing accelerators.

We improve the performance of lattice-based cryptosystems Dilithium on Cortex-M3 with expensive multiplications. Our contribution is two-fold: (i) We generalize Barrett multiplication and show that the resulting shape-independent modular multiplication performs comparably to long multiplication on some platforms without special hardware when precomputation is free. We call a modular multiplication “shape-independent” if its correctness and efficiency depend only on the magnitude of moduli and not the shapes of the moduli. This was unknown in the literature even though modular multiplication has been studied for more than 40 years. In the literature, shape-independent modular multiplications often perform several times slower than long multiplications even if we ignore the cost of the precomputation. (ii) We show that polynomial multiplications based on Nussbaumer fast Fourier transform and Toom–Cook over Z2k perform the best when modular multiplications are expensive and k is not very close to the arithmetic precision.For practical evaluation, we implement assembly programs for the polynomial arithmetic used in the digital signature Dilithium on Cortex-M3. For the modular multiplications in Dilithium, our generalized Barrett multiplications are 1.92 times faster than the state-of-the-art assembly-optimized Montgomery multiplications, leading to 1.38−1.51 times faster Dilithium NTT/iNTT. Along with the improvement in accumulating products, the core polynomial arithmetic matrix-vector multiplications are 1.71−1.77 times faster. We further apply the FFT-based polynomial multiplications over Z2k to the challenge polynomial multiplication ct0, leading to 1.31 times faster computation for ct0.We additionally apply the ideas to Saber on Cortex-M3 and demonstrate their improvement to Dilithium and Saber on our 8-bit AVR environment. For Saber on Cortex-M3, we show that matrix-vector multiplications with FFT-based polynomial multiplications over Z2k are 1.42−1.46 faster than the ones with NTT-based polynomial multiplications over NTT-friendly coefficient rings. When moving to a platform with smaller arithmetic precision, such as 8-bit AVR, we improve the matrix-vector multiplication of Dilithium with our Barrett-based NTT/iNTT by a factor of 1.87−1.89. As for Saber on our 8-bit AVR environment, we show that matrixvector multiplications with NTT-based polynomial multiplications over NTT-friendly coefficient rings are faster than polynomial multiplications over Z2k due to the large k in Saber.

Convolutional neural networks (CNNs) have demonstrated promising results in various applications such as computer vision, speech recognition, and natural language processing. One of the key computations in many CNN applications is matrix multiplication, which accounts for a significant portion of computation. Therefore, hardware accelerators to effectively speed up the computation of matrix multiplication have been proposed, and several studies have attempted to design hardware accelerators to perform better matrix multiplications in terms of both speed and power consumption. Typically, accelerators with either a two-dimensional (2D) systolic array structure or a single instruction multiple data (SIMD) architecture are effective only when the input matrix has shapes that are close to or similar to a square. However, several CNN applications require multiplications of non-squared matrices with various shapes and dimensions, and such irregular shapes lead to poor utilization efficiency of the processing elements (PEs). This study proposes a configurable engine for neural network acceleration, called CONNA, whose computation engine can conduct matrix multiplications with highly utilized computing units, regardless of the access patterns, shapes, and dimensions of the input matrices by changing the shape of matrix multiplication conducted in the physical array. To verify the functionality of the CONNA accelerator, we implemented CONNA as an SoC platform that integrates a RISC-V MCU with CONNA on Xilinx VC707 FPGA. SqueezeNet on CONNA achieved an inference performance of 100 frames per second (FPS) with 2.36 mm2 and 83.55 mW in a 65 nm process, improving efficiency by up to 34.1 times better than existing accelerators in terms of FPS, silicon area, and power consumption.

Matrix multiplication is required for a wide variety of applications, including data mining, linear algebra, graph transformations, etc. Most of the existing works to accelerate matrix multiplication have focused on matrices with floating point elements. In this work, we propose for the first time an FPGA based accelerator architecture for binary matrix multiplication. It consists of processing elements laid out in regular tiled manner. The communication structure used is a torus. We undertook detailed experimental study of the proposed architecture. The architecture shows excellent scalability with increase in number of processing elements, with minimal drop in operating frequency. The proposed system achieves maximum throughput of 1120 Gops for $4 \times 4$ network size with $2048 \times 2048$ matrix size. The performance achieved by the system is considerably higher than existing works of floating point multiplication on FPGAs, due to optimized PE design for binary matrix multiplication.

Strassen's algorithm for matrix multiplication gains its lower arithmetic complexity at the expense of reduced locality of reference, which makes it challenging to implement the algorithm efficiently on a modern machine with a hierarchical memory system. We report on an implementation of this algorithm that uses several unconventional techniques to make the algorithm memory-friendly. First, the algorithm internally uses a non- standard array layout known as Morton order that is based on a quad-tree decomposition of the matrix. Second, we dynamically select the recursion truncation point to minimize padding without affecting the performance of the algorithm, which we can do by virtue of the cache behavior of the Morton ordering. Each technique is critical for performance, and their combination as done in our code multiplies their effectiveness. Performance comparisons of our implementation with that of competing implementations show that our implementation often outperforms the alternative techniques (up to 25%). However, we also observe wide variability across platforms and across matrix sizes, indicating that at this time, no single implementation is a clear choice for all platforms or matrix sizes. We also note that the time required to convert matrices to/from Morton order is a noticeable amount of execution time (5% to 15%). Eliminating this overhead further reduces our execution time.

In-memory computing (IMC) is gaining momentum for low-power acceleration of data intensive problems, particularly in the frame of artificial intelligence (AI). Analogue matrix-vector multiplication (MVM) in crosspoint arrays allows to dramatically improve energy efficiency in neural networks for inference and training [1]. Recently, it has been shown that a feedback configuration of MVM can accelerate the solution of generic linear matrix problems, such as linear systems, matrix inversion and eigenvector calculation, with a single computational step [2,3]. This is achieved by the high parallelism of MVM in the memory array, combined with the physical iteration executed naturally in the feedback loop. Eigenvector extraction allows to address general problems, such as PageRank of internet webpages, with complexity O(1), i.e., the time to solve a ranking problem remains fixed irrespective of the size of the coefficient matrix [4]. In this work, we address two specific problems of AI acceleration, namely regression and optimization.Linear regression can be viewed as one of the most basic problems of machine learning: given a sequence of points, one must predict the value of the next point in the sequence. This problem can be solved in just one computational step by using the closed-loop crosspoint array of Fig. 1, where the independent variables x_i are physically written within the two crosspoint arrays, while the dependent variables y_i are applied externally as input currents. The crosspoint arrays are connected within a closed-loop configuration by using operational amplifiers (OAs) in the rows and the columns of the arrays. The OA output yields the coefficients of the best fitting line along the sequence, i.e., the coefficients a and b in the line equation y= ax + b providing the least mean squares (LMS) regression of the sequence. This problem can be extended to any arbitrary dimension, i.e., not only data on x-y plane, but also data on x-y-z space and any hyperspace of n dimensions. The problem can also be extended to logistic regression, which enables classification in one computational step. In particular, the crosspoint regression circuit allows the training of a 2-layer perceptron in just one step, by assuming random weights in the first layer according to the extreme learning machine (ELM) approach. In-memory computing thus appears as a promising option for accelerating neural network training with complexity O(1).The continuous-time, analogue closed-loop approach can be extended to a discrete-time, digital recurrent neural network (RNN) for addressing optimization problems [5]. The RNN allows the efficient solution of constraint satisfaction problems (CSPs), such as Sudoku, by taking advantage of analogue MVM in the crosspoint array. In the RNN, stochastic iterations can allow the minimization of the cost function, hence the number of violated constraints, thus leading to the global minimum. Noise can be injected as spike jitter by phase change memory devices as integrate-and-fire neurons. The convergence to the solution can be efficiently accelerated by a 2-layer RNN, where the first layer is programmed to prevent the violation of constraints in the CSP. In-memory solution of CSPs can dramatically improve the energy efficiency and speed of data intensive optimization in transport, industry, and society in general.

&lt;p style="text-indent: 1.27cm; margin-bottom: 0.35cm; line-height: 115%;" align="justify"&gt;&lt;span style="font-family: Arial,serif;"&gt;&lt;span style="font-size: small;"&gt;&lt;em&gt;Deep Neural Network training algorithms consumes long training time, especially when the number of hidden layers and nodes is large. Matrix multiplication is the key operation carried out at every node of each layer for several hundreds of thousands of times during the training of Deep Neural Network. Blocking is a well-proven optimization technique to improve the performance of matrix multiplication. Blocked Matrix multiplication algorithms can easily be parallelized to accelerate the performance further. This paper proposes a novel approach of implementing Parallel Blocked Matrix multiplication algorithms to reduce the long training time. The proposed approach was implemented using a parallel programming model OpenMP with collapse() clause for the multiplication of input and weight matrices of Backpropagation and Boltzmann Machine Algorithms for training Deep Neural Network and tested on multi-core processor system. Experimental results showed that the proposed approach achieved approximately two times speedup than classic algorithms.&lt;/em&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt;

We consider the techniques behind the current best algorithms for matrix multiplication. Our results are threefold. (1) We provide a unifying framework, showing that all known matrix multiplication running times since 1986 can be achieved from a single very natural tensor - the structural tensor T_q of addition modulo an integer q. (2) We show that if one applies a generalization of the known techniques (arbitrary zeroing out of tensor powers to obtain independent matrix products in order to use the asymptotic sum inequality of Schonhage) to an arbitrary monomial degeneration of T_q, then there is an explicit lower bound, depending on q, on the bound on the matrix multiplication exponent omega that one can achieve. We also show upper bounds on the value alpha that one can achieve, where alpha is such that n * n^alpha * n matrix multiplication can be computed in n^{2+o(1)} time. (3) We show that our lower bound on omega approaches 2 as q goes to infinity. This suggests a promising approach to improving the bound on omega: for variable q, find a monomial degeneration of T_q which, using the known techniques, produces an upper bound on omega as a function of q. Then, take q to infinity. It is not ruled out, and hence possible, that one can obtain omega=2 in this way.

Due to the recent progress of the DNA microarray technology, a large number of gene expression profile data are being produced. How to analyze gene expression data is an important topic in computational molecular biology. Several studies have been done using the Boolean network as a model of a genetic network. This paper proposes efficient algorithms for identifying Boolean networks of bounded indegree and related biological networks, where identification of a Boolean network can be formalized as a problem of identifying many Boolean functions simultaneously. For the identification of a Boolean network, an O(mnD+1) time naive algorithm and a simple O (mnD) time algorithm are known, where n denotes the number of nodes, m denotes the number of examples, and D denotes the maximum in degree. This paper presents an improved O(momega-2nD + mnD+omega-3) time Monte-Carlo type randomized algorithm, where omega is the exponent of matrix multiplication (currently, omega < 2.376). The algorithm is obtained by combining fast matrix multiplication with the randomized fingerprint function for string matching. Although the algorithm and its analysis are simple, the result is nontrivial and the technique can be applied to several related problems.

Multicore processors are marking the beginning of a new era of computing where massive parallelism is available and necessary. Slightly slower but easy to parallelize kernels are becoming more valuable than sequentially faster kernels that are unscalable when parallelized. In this paper, we focus on the multiplication of sparse matrices (SpGEMM). We first present the issues with existing sparse matrix representations and multiplication algorithms that make them unscalable to thousands of processors. Then, we develop and analyze two new algorithms that overcome these limitations. We consider our algorithms first as the sequential kernel of a scalable parallel sparse matrix multiplication algorithm and second as part of a polyalgorithm for SpGEMM that would execute different kernels depending on the sparsity of the input matrices. Such a sequential kernel requires a new data structure that exploits the hypersparsity of the individual submatrices owned by a single processor after the 2D partitioning. We experimentally evaluate the performance and characteristics of our algorithms and show that they scale significantly better than existing kernels.

Efficient parallel factorization and solution of structured and unstructured linear systems