Selective Triplication for Fault-Tolerant Systolic Array Processing Elements

Research on artificial neural networks (ANNs) has been carried out for more than five decades. A renewed interest appeared in the 80's with the finding of powerful models like J. Hopfield's recurrent networks, T. Kohonen's self-organizing feature maps, and the back-propagation rule. At that time, there was no platform that was at the same time versatile enough for any ANN model to be implemented and fast enough to solve large problems. Super-computers were the sole exception to this rule, but were prohibitively expensive for most applications. However, both research scientists and application engineers clearly identified the need for such a computing power. This triggered many projects in the field. In parallel, research on multi-processor systems started during the 60's. Systolic arrays have been proposed in 1979 as a means to fully exploit the possibilities of VLSI. Two previous theses, by F. Blayo and C. Lehmann, have studied the use of bi-dimensional systolic arrays for neural computation. At first, the presented system, called GENES, has been designed for the Hopfield model. Extensions to other ANNs have also been proposed. The goal of the present thesis is to study, design, build, and analyze an efficient accelerator for neural computation. In a first step, the GENES architecture has been extended towards generality and efficiency. This includes a thorough analysis of ANN models, of other neural computers, and of previous GENES implementations. The result of this work is the GENES IV integrated circuit, whose architecture has been co-designed by P. Ienne and the author. The main part of this thesis discusses the architecture, the design and the analysis of the MANTRA I machine, a neural computer based on a GENES IV array with up to 40 × 40 processing elements (PEs). The delta rule (and hence the Perceptron and Adaline rules), the back-propagation rule, the Hopfield model, and the Kohonen model can be implemented on this system. Although not a generic system, such a machine may be regarded as a multi-model neural computer. A prototype has been running for a year and is used daily by software designers. Several novel features distinguish the MANTRA I machine from other neural systems. First, it belongs to the few existing neural computers, contrary to the majority of implementations, which are specific to an application or to an algorithm. The machine does not hard-wire any algorithm, but provides the necessary primitives to implement the target models. This is a key feature for research, since several algorithms or versions of an algorithm can be tested on a problem. It is an important aspect for applications as well, because different ANN models are often cascaded to solve a problem. The GENES IV array — that is, the computing core of the MANTRA I machine — features synapse-level parallelism (i.e., one real or virtual PE is allocated per synapse or neural connection), while most other systems exploit only neuron-level parallelism (i.e., one PE per neuron). Hence, this system aims at a much finer parallelism grain and is well suited for massively parallel architectures. The problem size that can be computed by a neural accelerator should not be limited by the hardware (except for memory size). Therefore, it is essential to support time-sharing of PEs. On the MANTRA I machine, this is achieved by the concept of virtual arrays. Matrices are divided into sub-matrices that can be mapped onto the physical array, which is then time-shared among them. An efficient mechanism has been implemented to swap sub-matrices in background, while some other computation is performed. Since systolic arrays are pipelined systems, it is important to avoid emptying and re-filling them too often, in order to keep the hardware utilization rate high. Therefore, a systolic instruction flow has been implemented, so that each instruction follows the data for which it has been issued. Like any SIMD system, the MANTRA I machine is composed of a parallel or SIMD module and a control module. The SIMD module consists of the GENES IV array and a set of dedicated units designed for the computation that scales with the number of neurons and would poorly fit on a bi-dimensional array. A complex system of FIFOs and memories sustains the required input/output streams for the systolic array. The control module is a complete SISD system. Its tasks are (1) to control the SIMD module by dispatching instructions, (2) to manage data input and output, (3) to communicate with the external world, and (4) to perform data pre- and post-processing. The SIMD instructions are of the very long instruction word (VLIW) type. Synchronization between the two modules is achieved by an instruction FIFO. The performance of the MANTRA I machine has been analyzed using the delta rule. Measurements show that the sustained performance is very close to its peak value, as long as the problem fits in the memory banks connected to the GENES IV array. Experiments have also been run to investigate the impact of the constraints imposed by the hardware on the convergence of algorithms. Finally, the use of systolic arrays as neural accelerators is discussed in the light of the experience acquired with the GENES IV array and the MANTRA I machine. The weaknesses of the machine are analyzed, and several solutions are proposed to avoid them in a future design. A general discussion of the future of neural computers concludes this thesis.

Automatic systolic array generation has long been an interesting topic since manual designs usually require huge implementation efforts. Conventional automatic systolic array generation builds dependency graphs from algorithms, and iteratively maps computation nodes in the graph into processing elements (PEs) with time stamps that specify the sequences for nodes to operate within the PE. Sometimes this mapping will generate an array with I/O operations being distributed to all PEs, which causes complex design scheme and area overheads. The process to eliminate such I/O is called interior I/O elimination. This paper presents our approach for interior I/O elimination. We propose an automated compiler to map algorithms to systolic arrays. For any mapped array with interior I/O, the compiler analyzes the communication patterns and determines if interior I/O can be eliminated in the design. If so, the compiler eliminates the I/O and adds necessary modules in the design for data transfer. We demonstrate our approach on two important applications'matrix multiplication and convolutional neural network. With the help of interior I/O elimination, the designs generated by the compiler are able to achieve 591.4GFLOPs and 405.2GFLOPs, respectively.

The design of optimal planar systolic arrays for matrix multiplication

The systolic array implementation of artificial neural networks is one of the ideal solutions for communication problems generated by highly interconnected neurons. A systolic array is an arrangement of processors in an array where data flows synchronously across the array between neighbours, usually with different data flowing in different directions. The simulation of systolic array for matrix multiplication is the practical application in order to evaluate the performance of systolic array. In this paper, a two-dimensional orthogonal systolic array for matrix multiplication is presented. Perl scripting language is used to simulate a two-dimensional orthogonal systolic array compared to conventional matrix multiplication in terms of average execution time. The comparison is made using matrices of size 5xM versus Mx5 which M ranges from 1 to 10, 10 to 100 and 100 to 1000. The orthogonal systolic array results show better average execution time when M is more than 30 compared to conventional matrix multiplication when the size of the matrix multiplication is increased.

Exploiting data encoding and reordering for low-power streaming in systolic arrays

Since neural networks (NNs) require an enormous amount of learning time, various kinds of dedicated parallel computers have been developed. In the paper a 2-D systolic array (SA) of dedicated processing elements (PEs) also called systolic cells (SCs) is presented as the heart of a multimodel neural-network accelerator. The instruction set of the SA allows the implementation of several neural algorithms, including error back propagation and a self organizing feature map algorithm. Several special architectural facilities are presented in the paper in order to improve the 2-D SA performance. A swapping mechanism of the weight matrix allows the implementation of NNs larger than 2-D SA. A systolically propagated instruction word accompanying each input vector inside the 2-D SA allows the operating mode to be changed progressively, avoiding intermediate inactive cycles inside the 2-D SA. An FPGA implementation of the proposed 2-D SA is presented.

Matrix multiplication (MM) is a basic operation for many Digital Signal Processing applications. A Systolic Array (SA) is often considered as one of the most favorable architecture to achieve high performance for matrix multiplication. In this paper, the design exploration for an approximate SA is pursued; three design schemes are proposed by introducing approximation in multiple sub-modules. An approximation factor $\alpha$ is introduced; it is related to the inexact columns in the SA to explore the accuracy-efficiency trade-off present in the proposed designs. In the evaluation, an 8-bit input operand matrix multiplication is considered; the Synopsys Design Compiler at 45nm technology node is used to establish hardware-related metrics. The Error Rate (ER), Normalized Mean Error Distance (NMED) and Mean Relative Error Distance (MRED) are used as figures of merit for error analysis. Results show that the proposed architecture for 8-bit matrix multiplication with an approximation factor $\alpha=7$ has the lower power consumption compared to existing inexact designs found in the technical literature with comparable NMED. In addition, a power delay product vs NMED analysis shows the proposed designs have a lower PDP so applicable to low power applications. The practicality of the proposed architecture is established by computing the Discrete Cosine Transform.

A modular architecture for very fast digital signal processing (DSP) elements are presented. The computation is performed over finite rings (or fields) and is able to emulate processing over the integer ring using residue number systems. The computations are restricted to closed operations (ring or field binary operators) with the ability to perform limited scaling operations. Computations naturally defined over finite mathematical systems are also easily implemented using this approach. The technique evolves from the decomposition of each closed calculation using the ring/field associativity property. Linear systolic arrays, formed with multiple elements, each of a single generic form, are used for all calculations. The pipeline cycle is determined from the generic cell and is predicted to be very fast by a critical path analysis. The cells are matched to the VLSI medium, and the resulting array structures are very dense. Examples of DSP applications are given to illustrate the technique, and example cell and array VLSI layouts are presented for a 3- mu m CMOS process. >

This paper deals with fast implementation of the direct kinematics equations of robotic manipulators, using systolic arrays that incorporate one CORDIC rotor in their processing elements. The rotation or the translation motions of the links are implemented by using a processing element of general form. The processing elements used for the revolute joints incorporate one CORDIC rotor and at most an adder and one inverter. The processing elements used for the prismatic joints incorporate at most two adders and one inverter. The overall systolic architecture is derived by cascading the processing elements that correspond to the successive links and pipelining the unidirectional linear array. The use of systolic arrays with CORDIC processors results into fast pipelined VLSI architectures, which may be used as peripheral devices to a host digital computer and are suitable for computation of the kinematic equations, in run-time operation.

Convolutional neural network (CNN) inference on mobile devices demands efficient hardware acceleration of low-precision (INT8) general matrix multiplication (GEMM). The systolic array (SA) is a pipelined 2D array of processing elements (PEs), with very efficient local data movement, well suited to accelerating GEMM, and widely deployed in industry. In this letter, we describe two significant improvements to the traditional SA architecture, to specifically optimize for CNN inference. First, we generalize the traditional scalar PE, into a Tensor-PE, which gives rise to a family of new Systolic Tensor Array (STA) microarchitectures. The STA family increases intra-PE operand reuse and datapath efficiency, resulting in circuit area and power dissipation reduction of as much as 2.08× and 1.36× respectively, compared to the conventional SA at iso-throughput with INT8 operands. Second, we extend this design to support a novel block-sparse data format called density-bound block (DBB). This variant (STA-DBB) achieves a 3.14× and 1.97× improvement over the SA baseline at iso-throughput in area and power respectively, when processing specially-trained DBB-sparse models, while remaining fully backwards compatible with dense models.

Compact convolutional neural networks have become a hot research topic. However, we find that the hardware accelerator with systolic arrays processing compact models is extremely performance-inefficient, especially when processing depthwise convolutional layers in the networks. To make systolic arrays efficient for compact convolutional neural networks, we propose the heterogeneous systolic array (HeSA) architecture. It introduces heterogeneous processing elements that support multiple modes of dataflow, which can further exploit the reuse data chance of depthwise convolutional layers and without changing the architecture of the naïve systolic array. By increasing the utilization rate of processing elements in the array, HeSA improves the performance, throughput, and energy efficiency compared to the standard baseline. Based on our evaluation with typical workloads, HeSA improves the utilization rate of the computing resource in depthwise convolutional layers by 4.5×-5.5× and acquires 1.5-2.2× total performance speedup compared to the standard systolic array architecture. HeSA also improves the on-chip data reuse chance and saves over 20% of energy consumption. Meanwhile, the area of HeSA is basically unchanged compared to the baseline due to its simple design.

The problem of detection and identification of a faulty processing element in a systolic array is addressed. A method for designing processing elements with concurrent error detection is presented. The mod gAN mod /sub M/ code is shown to be an effective code for encoding the operands in a systolic array. It is shown that the mod g3N mod /sub M/ code is equivalent to a residue code with the check and information bits interchanged, for an odd number of information bits. This allows arithmetic to be performed separately on the information and check bits while the output can be checked by an AN checker. An architecture and rules for designing a self-checking processing element (PE) for systolic arrays are presented. Both redundancy and extra delay of the self-checking PE are shown to be low.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The requirement for efficient digital signal processing (DSP) systems and its solutions increases highly over the last few years. DSP techniques as well as its algorithms are necessary for day to day life tool namely smartphone, set-top unit (STU), video and acoustic players, digital cameras, digital television, etc. This paper describes a comparative study on knowledge-based digital filter structure namely transposed form, direct form, and systolic array. To implement the parallel processing the systolic array structure is often used. A Goertzel algorithm is implemented in the field-programmable gate array (FPGA) Altera EP4CE115F29C7 device which is used to determine the discrete Fourier transform (DFT). Finally, filtering technique is implemented efficiently by dividing filter coefficients and data into odd and even which increases the computational speed. The computational speed and functional verification results are achieved by using MATLAB. The Altera DSP builder and Simulink are used for the implementation of Goertzel algorithm. The 64-tap systolic low-pass filter structures offer 213.97-μs-less delay than the transposed form, 1700 μs, and direct form, 3900 μs, structures. Similarly for 2048-tap low-pass filter using sub-filter offers 49.397-ms computational speed and direct filtering offers 53.257 ms.

Various artificial intelligence (AI) algorithms are proposed in recent years, the demand for computing complexity has also increased. Matrix multiplication is a computing unit commonly used in AI calculations. This paper proposes a novel sparse matrix multiplication optimized scalable systolic array multiplier, which has three characteristics compared to traditional matrix multipliers: 1. This paper deals with a sparse matrix multiplication optimization multiplier, which is achieved by the critical path. 2. This paper focuses on high-dimensional convolutional neural network computation and proposed a multiplier with an expandable feature, which can calculate the matrices that dimension below the multiplier itself. Compared with the inextensible multiplier, this improvement reduces power consumption when calculating low-dimensional data. 3. The matrix multiplier of the traditional systolic array has a pulsation relationship between columns, which introduced delay registers between adjacent columns. Therefore, we proposed a new structure based on the traditional systolic array which removes the delay module. We designed a 4*4 matrix multiplication and deployed it at the PYNQ-Z7020 field-programmable gate array (FPGA). The result shows that the proposed structure reduces the 9.2% calculation delay, saving 13.3% slice logic power consumption than the traditional multiplication.

Energy consumption in Digital Signal Processing (DSP) application is a vital parameter of consideration. Mostly DSP applications deplete power. Battery constraint is important in many mobiles, processors, and sensor-related devices. The major critical operations in DSP architecture are multiplication and addition. Multiplication process is performed using repetitive addition. Hence, adder is the basic component used in digital signal processor. DSP architecture is proposed with the energy-efficient goal. Traditional carry select adder contains ripple carry adder block is replaced by parallel prefix adder. By replacing that block and introducing folded tree, we can reduce power used by DSP processors. This architecture can be used in all power constrained DSP applications. In the existing system, folded tree is used in architecture to reduce the number of processing elements, and they used carry look-ahead adder to perform the internal processing element operations. In the proposed system, carry look-ahead adder is replaced by LFA and PA to harness the energy depleted in an application (e.g., DSP application). Power consumption is reduced by 12–15% as compared with the existing algorithms using Cyclone III (EP3C16F484C6).