Timeloop: A Systematic Approach to DNN Accelerator Evaluation

This article presents an energy-harvester-powered ultra-low power (ULP) vibration-based condition monitoring (VbCM) chip with a digital compute-in-memory (CIM)-based deep neural network (DNN) accelerator. The VbCM chip achieves end-to-end signal processing, including a piezoelectric (PZ)-based accelerometer sensor and its readout circuit (RoC), a digital CIM-based DNN accelerator and a ULP radio system. To perform the targeted rotating bearing anomaly detection task with best-in-class accelerator power consumption and energy efficiency, the chip integrates several technologies from the algorithm level to the hardware level, including: 1) Compressing deep neural network (C-DNN) to reduce the network size through a feature compressor module (FCM); 2) Signal chain characteristic adaptation from direct training of the time-domain feature extractor (TD-FEx); 3) ULP 13T SRAM CIM bitcell to perform high energy efficiency in-cell multiplication; and 4) ripple counter (RCNT)-based accumulation scheme to improve the overall energy efficiency of the accelerator under a 0.35 V supply voltage. The VbCM chip achieves 90.7 nW total power with an inference FR of 20 frames/s, and the DNN accelerator’s energy efficiency achieves 18.8 fJ/MAC. With all the weight parameters statically stored in the CIM macros, the accelerator can operate with an inference accuracy of 91.3% while consuming 24.7 nW.

The conventional Deep Neural Networks (DNNs) accelerator is usually designed based on array-based processing element computing. By using the array-based computation, memory access can be reduced efficiently based on a specific dataflow. However, the computing flexibility of the contemporary DNN accelerators is usually restricted because they support the monotonous computing dataflow. Hence, the computing efficiency will be degraded under the different hyperparameters of the target DNN (e.g., kernel size, layers, etc.) Because of the high flexibility and scalability of Network on Chip (NoC) interconnection, the NoC-based DNN design methodology becomes an attractive design paradigm, which reduces the design complexity of the DNN accelerator implementation. However, the current NoC-based DNN designs usually assume that the entire DNN model can be mapped to the target NoC. In this way, the area overhead will be larger with respect to the increasing DNN scale. To solve this problem, we propose a dynamic mapping algorithm, called: dense mapping. The dense mapping is used to map the neuron operations to the NoC as long as the available computing resources are enough for the neuron operations. Besides, an input sharing mechanism is proposed to reuse input data. In this way, we can not only process a DNN model on a small-scale NoC but also decrease the number of memory access by using the proposed input sharing mechanism. Compared with the related work, the proposed approaches help to reduce 60.5% number of memory access and improve 96.5% throughput due to fewer memory accesses.

Energy-efficient Resistive RAM (ReRAM) based deep neural network (DNN) accelerator suffers from severe Stuck-At-Fault (SAF) problem that drastically degrades the inference accuracy. The SAF problem gets even worse in realistic ReRAM devices with low cell resolution. To address the issue, we propose a fault-resilient DNN accelerator based on realistic ReRAM devices. We first analyze the SAF problem in a realistic ReRAM device and propose a 3-stage offline fault-resilient compilation and lightweight online compensation. The proposed work enables the reliable execution of DNN with only 5% area and 0.8% energy overhead from the ideal ReRAM-based DNN accelerator.

This paper presents an optimization framework named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NeuroSpector</i> that systematically analyzes the dataflow of deep neural network (DNN) accelerators and rapidly identifies optimal execution methods. The proposed methodology is demonstrated to work effectively with a variety of accelerator architectures and DNN workloads. It has been a baffling challenge to devise scheduling schemes for neural accelerators to maximize energy efficiency and performance. The challenge lies in that hardware specifications associated with multi-dimensional DNN data create an enormous number of possible scheduling options that can be exerted on accelerators. Related work suggested various techniques to solve the challenge encompassing brute-force search of massive solution spaces pruned by user constraints, solving the objective functions of system models, learning-based optimization, etc. However, each suggested technique was devised only for a specific accelerator model. Therefore, we find that they are not adaptively applicable to different accelerators and DNN workloads in that they produce hit-or-miss results with 100.1% greater energy and cycles on average compared to optimal scheduling schemes obtained from fully comprehensive brute-force searches. In contrast, NeuroSpector identifies efficient execution methods for various accelerators and workloads with only 1.5% differences on average to the optimal scheduling solutions. The optimization strategy of NeuroSpector is based on an observation that optimal executions are strongly correlated with minimizing data movements to the lower-level memory hierarchy of accelerators rather than maximizing the utilization of processing elements. Thus, NeuroSpector prioritizes optimizing lower-level components in the accelerator hierarchy, which is proven highly effective for various accelerators and DNN workloads.

FPGAs have been rapidly adopted for acceleration of Deep Neural Networks (DNNs) with improved latency and energy efficiency compared to CPU and GPU-based implementations. High-level synthesis (HLS) is an effective design flow for DNNs due to improved productivity, debugging, and design space exploration ability. However, optimizing large neural networks under resource constraints for FPGAs is still a key challenge. In this paper, we present a series of effective design techniques for implementing DNNs on FPGAs with high performance and energy efficiency. These include the use of configurable DNN IPs, performance and resource modeling, resource allocation across DNN layers, and DNN reduction and re-training. We showcase several design solutions including Long-term Recurrent Convolution Network (LRCN) for video captioning, Inception module for FaceNet face recognition, as well as Long Short-Term Memory (LSTM) for sound recognition. These and other similar DNN solutions are ideal implementations to be deployed in vision or sound based IoT applications.

Deep convolutional neural networks (DCNNs) have achieved state-of-the-art performance in classification, natural language processing (NLP), and regression tasks. However, there is still a great gap between DCNNs and the human brain in terms of computation efficiency. Inspired by neural synaptic plasticity and stochastic computing (SC), we propose neural synaptic plasticity-inspired computing (NSPC) to simulate the human brain's neural network activity for inference tasks with simple logic gates. The multiplication and accumulation (MAC) is transformed by the wire connectivity in NSPC, which only requires bundles of wires and small width adders. To this end, the NSPC imitates the structure of neural synaptic plasticity from a circuit wires connection perspective. Furthermore, from the principle of NSPC, we use a data mapping method to convert the convolution operations to matrix multiplications. Based on the methodology of NSPC, fully-pipelined and low latency architecture is designed. The proposed NSPC accelerator exhibits high hardware efficiency while maintaining a comparable network accuracy level. The NSPC based DCNN accelerator (NSPC-CNN) processes DCNN at 1.5625M images/s with a power dissipation of 15.42 W and an area of 36.4 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . The NSPC based deep neural network (DNN) accelerator (NSPC-DNN) that implements three fully connected layers DNN consumes only 6.6 mm2 area and 2.93 W power, and achieves a throughput of 400M images/s. Compared with conventional fixed-point implementations, the NSPC-CNN achieves 2.77× area efficiency, 2.25× power efficiency; the proposed NSPC-DNN exhibits 2.31× area efficiency and 2.09× power efficiency.

Deep Neural Networks (DNNs) are the current state of the art for various tasks such as object detection, natural language processing and semantic segmentation. These networks are massively parallel, hierarchical models with each level of hierarchy performing millions of operations on a single input. The enormous amount of parallel computation makes these DNNs suitable for custom acceleration. Custom accelerators can provide real time inference of DNNs at low power thus enabling widespread embedded deployment. In this paper, we present Snowflake, a high efficiency, low power accelerator for DNNs. Snowflake was designed to achieve optimum occupancy at low bandwidths and it is agnostic to the network architecture. Snowflake was implemented on the Xilinx Zynq XC7Z045 APSoC and achieves a peak performance of 128 G-ops/s. Snowflake is able to maintain a throughput of 98 FPS on AlexNet while averaging 1.2 GB/s of memory bandwidth.

Deep neural networks (DNN) have demonstrated highly promising results across computer vision and speech recognition, and are becoming foundational for ubiquitous AI. The computational complexity of these algorithms and a need for high energy-efficiency has led to a surge in research on hardware accelerators. % for this paradigm. To reduce the latency and energy costs of accessing DRAM, most DNN accelerators are spatial in nature, with hundreds of processing elements (PE) operating in parallel and communicating with each other directly. DNNs are evolving at a rapid rate, and it is common to have convolution, recurrent, pooling, and fully-connected layers with varying input and filter sizes in the most recent topologies.They may be dense or sparse. They can also be partitioned in myriad ways (within and across layers) to exploit data reuse (weights and intermediate outputs). All of the above can lead to different dataflow patterns within the accelerator substrate. Unfortunately, most DNN accelerators support only fixed dataflow patterns internally as they perform a careful co-design of the PEs and the network-on-chip (NoC). In fact, the majority of them are only optimized for traffic within a convolutional layer. This makes it challenging to map arbitrary dataflows on the fabric efficiently, and can lead to underutilization of the available compute resources. DNN accelerators need to be programmable to enable mass deployment. For them to be programmable, they need to be configurable internally to support the various dataflow patterns that could be mapped over them. To address this need, we present MAERI, which is a DNN accelerator built with a set of modular and configurable building blocks that can easily support myriad DNN partitions and mappings by appropriately configuring tiny switches. MAERI provides 8-459% better utilization across multiple dataflow mappings over baselines with rigid NoC fabrics.

Data multiplexed and hardware reused architecture for deep neural network accelerator

With the advent of Deep Neural Network (DNN) accelerators, permanent faults are increasingly becoming a serious challenge for DNN hardware accelerator, as they can severely degrade DNN inference accuracy. The State-of-the-art works address this issue by adding homogeneous redundant Processing Elements (PEs) to the DNN accelerator’s central computing array, or bypassing faulty PEs directly. However, such designs induce inference loss, extra hardware cost, and performance overhead. Moreover, current designs are able to only deal with a limited number of faults due to costs. In this paper, we propose FSA, a Fault-tolerant Systolic Array-based DNN accelerator with the goal of maintaining DNN inference accuracy in the presence of permanent faults. The key feature of the proposed FSA is a unified re-computing module (RCM) that dynamically recalculates the required DNN computations that are supposed to be accomplished by faulty PEs with minimal latency and power consumption. Simulation results show that the proposed FSA reduces inference accuracy loss by 46%, improves execution time by 23%, and reduces energy consumption by 35% on average, as compared to existing designs.

Voltage overscaling, or undervolting, is an enticing approximate technique in the context of energy-efficient Deep Neural Network (DNN) acceleration, given the quadratic relationship between power and voltage. Nevertheless, its very high error rate has thwarted its general adoption. Moreover, recent undervolting accelerators rely on 8-bit arithmetic and cannot compete with state-of-the-art low-precision (<8b) architectures. To overcome these issues, we propose a new technique called Guarded Aggressive underVolting (GAV), which combines the ideas of undervolting and bit-serial computation to create a flexible approximation method based on aggressively lowering the supply voltage on a select number of least significant bit combinations. Based on this idea, we implement GAVINA (GAV mIxed-Precision Accelerator), a novel architecture that supports arbitrary mixed precision and flexible undervolting, with an energy efficiency of up to 89 TOP/sW in its most aggressive configuration. By developing an error model of GAVINA, we show that GAV can achieve an energy efficiency boost of 20% via undervolting, with negligible accuracy degradation on ResNet-18.

Off-chip memory, such as DRAM, its access energy cost is orders of magnitude higher than other operations such as multiply and accumulate, thereby dominating the system energy consumption. Therefore, optimizing the access of the off-chip memory is crucial to further improve the energy efficiency of the deep neural network (DNN) accelerator. Towards this, this brief proposed an adaptive scheduling algorithm to minimize the DRAM access. Compared with the previous works, it can not only dynamically determine the data partition and the data type that will be reused, but also considered the constraints between adjacent layer, that is, if the output feature map of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$i$ </tex-math></inline-formula> th layer is divided into N parts, the output feature map of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$i+1$ </tex-math></inline-formula> th layer can only be divided into N parts or write back to the off-chip memory. Therefore, a minimize and realizable memory access solution can be obtained. Choosing 3 popular networks UNet, VGG-16 and MobileNet as benchmarks, the experiment results show that our scheduling algorithm can achieve a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$34.46\% \sim 93.42\%$ </tex-math></inline-formula> reduction in energy consumption of DRAM access and a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$34.34\% \sim 93.37\%$ </tex-math></inline-formula> reduction in DRAM access latency when compared to previous works.

The acceleration of deep neural networks (DNNs) on field-programmable gate arrays (FPGAs) has gained widespread adoption due to their high performance and energy efficiency. However, the shared hardware nature of FPGAs introduces new security vulnerabilities, especially in multi-tenant cloud and IoT environments. Among these, remote voltage side-channel attacks remain largely underexplored. In this paper, we present a novel attack paradigm that enables remote extraction of sensitive information through voltage fluctuation analysis on Xilinx 7-series FPGAs. Unlike traditional side-channel approaches requiring physical access or external sensors, our method leverages the embedded XADC module to non-invasively capture power-induced voltage variations caused by victim DNN computations. To effectively extract useful information from voltage fluctuations, we develop a customized processing pipeline tailored to the inherently low signal-to-noise ratio present in remote measurements. This pipeline employs dynamic thresholding, image-centric quantization, and multistage denoising to extract meaningful patterns from raw voltage traces. Applying this method, we successfully reconstruct input images with an average Normalized Cross-Correlation (NCC) of 73.86% against the original MNIST inputs, demonstrating high visual similarity and validating the feasibility of the attack. Our findings reveal critical security flaws in the shared power delivery and sensing infrastructure of modern FPGAs. This work underscores the need for hardware-level isolation and mitigation strategies, and expands the attack surface model for future FPGA-based cloud and edge deployments.

DeepEdgeSoC: End-to-end deep learning framework for edge IoT devices

The Resistive-RAM (RRAM) based deep neural network (DNN) accelerators have shown great potential as they are good at solving matrix-vector multiplication (MVM). However, this computing paradigm does not benefit other NN operations like activation, which may be built upon various transcendental functions and require customized circuit as in current RRAM-based NN accelerators. In this paper, we propose the RRAM-CORDIC algorithm and crossbar design which enable various transcendental activation calculations on a RRAM crossbar just like MVM. By applying encoding and multi-iteration transformation, the RRAM-CORDIC can exploit higher MAC (multiply-and-accumulation) parallelism that is traditionally uneconomic in CMOS but now efficient in RRAM crossbar. In addition, it can work in a pipelined manner with high computing throughput. Experiment results show that the RRAM-CORDIC algorithm can sustain high accuracy on different transcendental functions, and deliver less than 0.5% NN accuracy loss on typical DNN inference. The elimination of CMOS circuit in turn can trade more computing resources for MVM in the same area budget that improves the performance up to 47% for different networks.