Automatic Interior I/O Elimination in Systolic Array Architecture

Convolutional neural networks (CNNs) have demonstrated promising results in various applications such as computer vision, speech recognition, and natural language processing. One of the key computations in many CNN applications is matrix multiplication, which accounts for a significant portion of computation. Therefore, hardware accelerators to effectively speed up the computation of matrix multiplication have been proposed, and several studies have attempted to design hardware accelerators to perform better matrix multiplications in terms of both speed and power consumption. Typically, accelerators with either a two-dimensional (2D) systolic array structure or a single instruction multiple data (SIMD) architecture are effective only when the input matrix has shapes that are close to or similar to a square. However, several CNN applications require multiplications of non-squared matrices with various shapes and dimensions, and such irregular shapes lead to poor utilization efficiency of the processing elements (PEs). This study proposes a configurable engine for neural network acceleration, called CONNA, whose computation engine can conduct matrix multiplications with highly utilized computing units, regardless of the access patterns, shapes, and dimensions of the input matrices by changing the shape of matrix multiplication conducted in the physical array. To verify the functionality of the CONNA accelerator, we implemented CONNA as an SoC platform that integrates a RISC-V MCU with CONNA on Xilinx VC707 FPGA. SqueezeNet on CONNA achieved an inference performance of 100 frames per second (FPS) with 2.36 mm2 and 83.55 mW in a 65 nm process, improving efficiency by up to 34.1 times better than existing accelerators in terms of FPS, silicon area, and power consumption.

Convolutional neural network (CNN) inference on mobile devices demands efficient hardware acceleration of low-precision (INT8) general matrix multiplication (GEMM). The systolic array (SA) is a pipelined 2D array of processing elements (PEs), with very efficient local data movement, well suited to accelerating GEMM, and widely deployed in industry. In this letter, we describe two significant improvements to the traditional SA architecture, to specifically optimize for CNN inference. First, we generalize the traditional scalar PE, into a Tensor-PE, which gives rise to a family of new Systolic Tensor Array (STA) microarchitectures. The STA family increases intra-PE operand reuse and datapath efficiency, resulting in circuit area and power dissipation reduction of as much as 2.08× and 1.36× respectively, compared to the conventional SA at iso-throughput with INT8 operands. Second, we extend this design to support a novel block-sparse data format called density-bound block (DBB). This variant (STA-DBB) achieves a 3.14× and 1.97× improvement over the SA baseline at iso-throughput in area and power respectively, when processing specially-trained DBB-sparse models, while remaining fully backwards compatible with dense models.

Convolutional neural networks (CNN) is a widely known deep learning architecture and achieves higher accuracies in speech processing and computer vision applications. CNN has arisen as best option in the fields such as video surveillance, mobile robot vision, smart factories, and medical diagnostics. CNN has shown its tremendous potential for image understanding in Cardio-Vascular (CV) systems. But, the tremendous potential of CNNs came up with additional computational burden. To get an outstanding performance of CNN-based algorithms, it requires large amount of enormous applications and memory resources. Matrix multiplication (MM) is the most fundamental computational operation taking place in CNN. Researchers in this field found that multiplication operation is the most resource-intensive and power-hungry operation in CNN. This work aims to design an FPGA-based CNN using systolic array which can improve both the accuracy and hardware efficiency of convolutional neural network. Here, we made design optimisation in multiplier unit which brings a significant impact on overall performance of convolutional neural networks.

Convolutional Neural Network (CNN) accelerators are popular specialized platforms for efficient CNN processing. As semiconductor manufacturing technology scales down to nano scale, process variation dramatically affects the chip's quality. Process variation causes delay variation within the chip due to transistor parameter differences. CNN accelerators adopt a large number of Processing Elements (PEs) for parallel computing, which are highly susceptible to process variation effects. Fast CNN processing desires consistent performance among PEs, otherwise the processing speed is limited by the slowest PE within the chip. In this work, we first quantitatively model and analyze the impact of process variation on CNN accelerator's operating frequency. We further analyze the utilization of CNN accelerator and the characteristics of CNN models. We then leverage the PE underutilization to propose a sub-matrix reformation mechanism and leverage the pixel similarity of images to propose a weight transfer technique. Both techniques are able to tolerate the low-frequency PEs, and achieve performance improvement at chip level. Evaluation results show our techniques are able to achieve significant processing speed improvement with negligible accuracy loss.

Systolic arrays are widely used for high-performance digital computations, enabling efficient and parallel execution of operations such as matrix multiplication and convolution. They are deployed in digital signal processing and embedded systems, and they have been more recently adopted to accelerate deep neural networks. However, due to manufacturing defects, aging, and harsh environments, systolic arrays are vulnerable to faults, which can compromise computation reliability. These vulnerabilities call for reliability analysis and hardening methods. Since full triplication induces a high hardware overhead, this work introduces an automatic method that applies redundancy only to the most impactful flip-flops in the processing elements. FPGA simulations show a wide Pareto trade-off between reliability and hardware resources, offering flexible protection levels for fault-resilient systems.

The design of optimal planar systolic arrays for matrix multiplication

Convolutional Neural Networks (CNNs) are popular in Machine Learning and CNNs have rich parallelism. Many efficient CNN accelerators have been designed recently. Especially, the Tiling structure accelerators are widely used. However, we observe that the Tiling architecture may not be efficient when the network layers change. This situation will result in a waste of processing elements (PEs). In order to achieve a high-utilization of PEs, we propose a new architecture called Single-Channel to 1) ameliorate the Tiling architecture without increasing the hardware complexity while 2) improve the utilization of PEs. We evaluate the Single-Channel architecture with four typical CNN networks. The hardware achieves the 1.2-5× speedup and the 40%-60% utilization improvement compared with the Tiling architectures.

Convolutional neural networks (CNNs) require high throughput hardware accelerators for real time applications owing to their huge computational cost. Most traditional CNN accelerators rely on single core, linear processing elements (PEs) in conjunction with 1D dataflows for accelerating convolution operations. This limits the maximum achievable ratio of peak throughput per PE count to unity. Most of the past works optimize their dataflows to attain close to a 100% hardware utilization to reach this ratio. In this paper, we introduce a high throughput, multi-threaded, log-based PE core. The designed core provides a 200% increase in peak throughput per PE count while only incurring a 6% increase in area overhead compared to a single, linear multiplier PE core with same output bit precision. We also present a 2D weight broadcast dataflow which exploits the multi-threaded nature of the PE cores to achieve a high hardware utilization per layer for various CNNs. The entire architecture, which we refer to as NeuroMAX, is implemented on Xilinx Zynq 7020 SoC at 200 MHz processing clock. Detailed analysis is performed on throughput, hardware utilization, area and power breakdown, and latency to show performance improvement compared to previous FPGA and ASIC designs.

This paper deals with fast implementation of the direct kinematics equations of robotic manipulators, using systolic arrays that incorporate one CORDIC rotor in their processing elements. The rotation or the translation motions of the links are implemented by using a processing element of general form. The processing elements used for the revolute joints incorporate one CORDIC rotor and at most an adder and one inverter. The processing elements used for the prismatic joints incorporate at most two adders and one inverter. The overall systolic architecture is derived by cascading the processing elements that correspond to the successive links and pipelining the unidirectional linear array. The use of systolic arrays with CORDIC processors results into fast pipelined VLSI architectures, which may be used as peripheral devices to a host digital computer and are suitable for computation of the kinematic equations, in run-time operation.

Power-efficient data processing subsystems performing millions of complex concurrent arithmetic operations per second form part of today’s essential solution required to meet the growing demand of edge computing applications, given the volume of data collected by real-time Internet-Of-Things (IoT) sensors. Adding to it, the in-memory computation designed as memory and processing elements on a single wafer has enabled promising performance improvement in terms of computational power savings by avoiding the memory wall created while accessing the memory array. The Resistive RAM (RRAM), with its simple metal-insulator-metal (MIM) structure, proves to be a very appealing candidate for in-memory computation given its ultralow switching power and its Complementary Metal Oxide Semiconductor (CMOS) process fabrication compatibility. However, despite all advantages, the resistive switching phenomenon in RRAM has an inherent stochastic variability. On the algorithmic side, convolution neural networks (CNN) have gained popularity in image classification applications, and the network’s architecture is memory-intense in nature for memorizing the trained weights. Hence, an RRAM-based CNN system will pave way for a power-efficient image classification system on the edge. Accounting however for the inherent variability in RRAM (inter-device and intra-device), the accuracy of CNN’s prediction is surely expected to drop. This motivates us to quantify the impact of RRAM variability on the CNN trained weights and classification accuracy (prediction loss). In this study, we have constructed a Look-Up-Table (LUT) based model for encoding wide current compliance (2μA to 250μA) 65nm CMOS 1T1R OxRAM’s (TiN/HfO2/Hf/TiN) resistive variability into CNN’s trained weight in a digital regime. The RRAM resistance encoded trained weights are in turn used here to simulate the two extreme CNN architectures, namely, Fully Serial System (FSS) and Fully Parallel System (FPS). The architectures’ prediction variability trends are quantified given its current compliance, RRAM resistive variability, CNN’s convolution matrix sizes (5×5, 3×3, 1×1, and 1×1 max pool), the total number of layers in the CNN as well as the input image pixel size.

Research on artificial neural networks (ANNs) has been carried out for more than five decades. A renewed interest appeared in the 80's with the finding of powerful models like J. Hopfield's recurrent networks, T. Kohonen's self-organizing feature maps, and the back-propagation rule. At that time, there was no platform that was at the same time versatile enough for any ANN model to be implemented and fast enough to solve large problems. Super-computers were the sole exception to this rule, but were prohibitively expensive for most applications. However, both research scientists and application engineers clearly identified the need for such a computing power. This triggered many projects in the field. In parallel, research on multi-processor systems started during the 60's. Systolic arrays have been proposed in 1979 as a means to fully exploit the possibilities of VLSI. Two previous theses, by F. Blayo and C. Lehmann, have studied the use of bi-dimensional systolic arrays for neural computation. At first, the presented system, called GENES, has been designed for the Hopfield model. Extensions to other ANNs have also been proposed. The goal of the present thesis is to study, design, build, and analyze an efficient accelerator for neural computation. In a first step, the GENES architecture has been extended towards generality and efficiency. This includes a thorough analysis of ANN models, of other neural computers, and of previous GENES implementations. The result of this work is the GENES IV integrated circuit, whose architecture has been co-designed by P. Ienne and the author. The main part of this thesis discusses the architecture, the design and the analysis of the MANTRA I machine, a neural computer based on a GENES IV array with up to 40 × 40 processing elements (PEs). The delta rule (and hence the Perceptron and Adaline rules), the back-propagation rule, the Hopfield model, and the Kohonen model can be implemented on this system. Although not a generic system, such a machine may be regarded as a multi-model neural computer. A prototype has been running for a year and is used daily by software designers. Several novel features distinguish the MANTRA I machine from other neural systems. First, it belongs to the few existing neural computers, contrary to the majority of implementations, which are specific to an application or to an algorithm. The machine does not hard-wire any algorithm, but provides the necessary primitives to implement the target models. This is a key feature for research, since several algorithms or versions of an algorithm can be tested on a problem. It is an important aspect for applications as well, because different ANN models are often cascaded to solve a problem. The GENES IV array — that is, the computing core of the MANTRA I machine — features synapse-level parallelism (i.e., one real or virtual PE is allocated per synapse or neural connection), while most other systems exploit only neuron-level parallelism (i.e., one PE per neuron). Hence, this system aims at a much finer parallelism grain and is well suited for massively parallel architectures. The problem size that can be computed by a neural accelerator should not be limited by the hardware (except for memory size). Therefore, it is essential to support time-sharing of PEs. On the MANTRA I machine, this is achieved by the concept of virtual arrays. Matrices are divided into sub-matrices that can be mapped onto the physical array, which is then time-shared among them. An efficient mechanism has been implemented to swap sub-matrices in background, while some other computation is performed. Since systolic arrays are pipelined systems, it is important to avoid emptying and re-filling them too often, in order to keep the hardware utilization rate high. Therefore, a systolic instruction flow has been implemented, so that each instruction follows the data for which it has been issued. Like any SIMD system, the MANTRA I machine is composed of a parallel or SIMD module and a control module. The SIMD module consists of the GENES IV array and a set of dedicated units designed for the computation that scales with the number of neurons and would poorly fit on a bi-dimensional array. A complex system of FIFOs and memories sustains the required input/output streams for the systolic array. The control module is a complete SISD system. Its tasks are (1) to control the SIMD module by dispatching instructions, (2) to manage data input and output, (3) to communicate with the external world, and (4) to perform data pre- and post-processing. The SIMD instructions are of the very long instruction word (VLIW) type. Synchronization between the two modules is achieved by an instruction FIFO. The performance of the MANTRA I machine has been analyzed using the delta rule. Measurements show that the sustained performance is very close to its peak value, as long as the problem fits in the memory banks connected to the GENES IV array. Experiments have also been run to investigate the impact of the constraints imposed by the hardware on the convergence of algorithms. Finally, the use of systolic arrays as neural accelerators is discussed in the light of the experience acquired with the GENES IV array and the MANTRA I machine. The weaknesses of the machine are analyzed, and several solutions are proposed to avoid them in a future design. A general discussion of the future of neural computers concludes this thesis.

Exploiting data encoding and reordering for low-power streaming in systolic arrays

The systolic array implementation of artificial neural networks is one of the ideal solutions for communication problems generated by highly interconnected neurons. A systolic array is an arrangement of processors in an array where data flows synchronously across the array between neighbours, usually with different data flowing in different directions. The simulation of systolic array for matrix multiplication is the practical application in order to evaluate the performance of systolic array. In this paper, a two-dimensional orthogonal systolic array for matrix multiplication is presented. Perl scripting language is used to simulate a two-dimensional orthogonal systolic array compared to conventional matrix multiplication in terms of average execution time. The comparison is made using matrices of size 5xM versus Mx5 which M ranges from 1 to 10, 10 to 100 and 100 to 1000. The orthogonal systolic array results show better average execution time when M is more than 30 compared to conventional matrix multiplication when the size of the matrix multiplication is increased.

Compact convolutional neural networks have become a hot research topic. However, we find that the hardware accelerator with systolic arrays processing compact models is extremely performance-inefficient, especially when processing depthwise convolutional layers in the networks. To make systolic arrays efficient for compact convolutional neural networks, we propose the heterogeneous systolic array (HeSA) architecture. It introduces heterogeneous processing elements that support multiple modes of dataflow, which can further exploit the reuse data chance of depthwise convolutional layers and without changing the architecture of the naïve systolic array. By increasing the utilization rate of processing elements in the array, HeSA improves the performance, throughput, and energy efficiency compared to the standard baseline. Based on our evaluation with typical workloads, HeSA improves the utilization rate of the computing resource in depthwise convolutional layers by 4.5×-5.5× and acquires 1.5-2.2× total performance speedup compared to the standard systolic array architecture. HeSA also improves the on-chip data reuse chance and saves over 20% of energy consumption. Meanwhile, the area of HeSA is basically unchanged compared to the baseline due to its simple design.

Convolutional Neural Networks (CNNs) have been shown to be very useful in image recognition and other AI applications. CNNs are usually computationally intensive. To address the challenge of overwhelming calculation requirements, researchers have proposed network compression methods to reduce the number of synaptic weights and the amount of computations. In this paper, we propose an input row based sparse convolution neural network accelerator on FPGAs that performs sparse CNN computing efficiently. Similar to the DNNWEAVER architecture, our accelerator also uses two-level architecture hierarchy, with multiple Processing Units (PUs) and each PU comprises a set of basic Processing Elements (PEs). The number of PEs in a single PU and the number of PUs in a design are reconfigurable for different CNNs for best performance. Our architecture does not require the large multiplexer for data selection as needed in Cambricon-X, thus, is more suitable for larger accelerator designs for high performance. Besides, we propose a weight merging method to balance the computation load on different PUs to maximize the overall computation efficiency. For evaluation, we implement our design with 32 PUs and each with 14 PEs. When compared with the DNNWEAVER implementation for non-sparse VGG16 network, we get an overall performance of 3.6x speedup running at 100MHz on a Xilinx ZC706 board and reach the speed of 297 GOPS.