Dynamic Mapping Mechanism to Compute DNN Models on a Resource-limited NoC Platform

As Deep Neural Networks (DNN) are becoming essential, Network-on-Chip (NoC)-based DNN accelerators gained increasing popularity. To save link power in NoC, many researchers focus on reducing the Bit Transition (BT). We propose ’1’-bit count-based ordering method to reduce BT for DNN workloads. We provide a mathematical proof of the efficacy of proposed ordering. We evaluate our method through experiments without NoC and with NoC. Without NoC, our proposed ordering method achieves up to 20.38% BT reduction for floating-point-32 data and 55.71% for fixed-point-8 data, respectively. We propose two data ordering methods, affiliated-ordering and separated-ordering to process weight and input jointly or individually and apply them to run full DNNs in NoC-based DNN accelerator. We evaluate our approaches under various configurations, including different DNN models such as LeNet and DarkNet, various NoC sizes with different numbers of memory controllers, random weights and trained weights, and different data precision. Our approach efficiently reduces the link power by achieving up to 32.01% BT reduction for floating-point-32 data and 40.85% BT reduction for fixed-point-8 data.

This paper presents Timeloop, an infrastructure for evaluating and exploring the architecture design space of deep neural network (DNN) accelerators. Timeloop uses a concise and unified representation of the key architecture and implementation attributes of DNN accelerators to describe a broad space of hardware topologies. It can then emulate those topologies to generate an accurate projection of performance and energy efficiency for a DNN workload through a mapper that finds the best way to schedule operations and stage data on the specified architecture. This enables fair comparisons across different architectures and makes DNN accelerator design more systematic. This paper describes Timeloop's underlying models and algorithms in detail and shows results from case studies enabled by Timeloop, which provide interesting insights into the current state of DNN architecture design. In particular, they reveal that dataflow and memory hierarchy co-design plays a critical role in optimizing energy efficiency. Also, there is currently still not a single architecture that achieves the best performance and energy efficiency across a diverse set of workloads due to flexibility and efficiency trade-offs. These results provide inspiration into possible directions for DNN accelerator research.

Deep neural networks (DNN) have demonstrated highly promising results across computer vision and speech recognition, and are becoming foundational for ubiquitous AI. The computational complexity of these algorithms and a need for high energy-efficiency has led to a surge in research on hardware accelerators. % for this paradigm. To reduce the latency and energy costs of accessing DRAM, most DNN accelerators are spatial in nature, with hundreds of processing elements (PE) operating in parallel and communicating with each other directly. DNNs are evolving at a rapid rate, and it is common to have convolution, recurrent, pooling, and fully-connected layers with varying input and filter sizes in the most recent topologies.They may be dense or sparse. They can also be partitioned in myriad ways (within and across layers) to exploit data reuse (weights and intermediate outputs). All of the above can lead to different dataflow patterns within the accelerator substrate. Unfortunately, most DNN accelerators support only fixed dataflow patterns internally as they perform a careful co-design of the PEs and the network-on-chip (NoC). In fact, the majority of them are only optimized for traffic within a convolutional layer. This makes it challenging to map arbitrary dataflows on the fabric efficiently, and can lead to underutilization of the available compute resources. DNN accelerators need to be programmable to enable mass deployment. For them to be programmable, they need to be configurable internally to support the various dataflow patterns that could be mapped over them. To address this need, we present MAERI, which is a DNN accelerator built with a set of modular and configurable building blocks that can easily support myriad DNN partitions and mappings by appropriately configuring tiny switches. MAERI provides 8-459% better utilization across multiple dataflow mappings over baselines with rigid NoC fabrics.

This paper presents an optimization framework named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NeuroSpector</i> that systematically analyzes the dataflow of deep neural network (DNN) accelerators and rapidly identifies optimal execution methods. The proposed methodology is demonstrated to work effectively with a variety of accelerator architectures and DNN workloads. It has been a baffling challenge to devise scheduling schemes for neural accelerators to maximize energy efficiency and performance. The challenge lies in that hardware specifications associated with multi-dimensional DNN data create an enormous number of possible scheduling options that can be exerted on accelerators. Related work suggested various techniques to solve the challenge encompassing brute-force search of massive solution spaces pruned by user constraints, solving the objective functions of system models, learning-based optimization, etc. However, each suggested technique was devised only for a specific accelerator model. Therefore, we find that they are not adaptively applicable to different accelerators and DNN workloads in that they produce hit-or-miss results with 100.1% greater energy and cycles on average compared to optimal scheduling schemes obtained from fully comprehensive brute-force searches. In contrast, NeuroSpector identifies efficient execution methods for various accelerators and workloads with only 1.5% differences on average to the optimal scheduling solutions. The optimization strategy of NeuroSpector is based on an observation that optimal executions are strongly correlated with minimizing data movements to the lower-level memory hierarchy of accelerators rather than maximizing the utilization of processing elements. Thus, NeuroSpector prioritizes optimizing lower-level components in the accelerator hierarchy, which is proven highly effective for various accelerators and DNN workloads.

Machine learning has surged in popularity in recent years, with deep neural networks (DNNs) becoming a cornerstone of applications such as autonomous driving, augmented reality, and natural language processing, owing to their remarkable accuracy and scalability. As the demand for greater portability and flexibility grows, there is a strong push to optimize these applications for resource-constrained edge devices like smartphones and IoT devices.However, deploying DNN models on such devices presents significant challenges, primarily due to the large size of these models, which is a key contributor to their high performance. For instance, the DeepSpeech 2 model outperforms its predecessor, DeepSpeech 1, but its size is eight times larger, encompassing approximately 68 million parameters. Similarly, other networks designed for various tasks, including convolutional neural networks like VGG and transformer-based networks like BERT and GPT, also exhibit substantial sizes. For example, the GPT-3 model, a predecessor to ChatGPT, consists of an astonishing 175 billion parameters. Consequently, fitting DNN models onto resource-limited edge devices is a formidable challenge. My thesis addresses these challenges with a three-part approach: * High-Performance DNN Accelerators: I will develop DNN accelerators that excel in high performance, low latency, and energy efficiency through an algorithm and hardware co-design across diverse FPGA architectures (Chapter 2). * Real-Time DNN Execution: I aim to achieve exceptional accuracy and real-time execution for various large-scale DNNs on off-the-shelf mobile devices, IoT devices, and other platforms (Chapter 3). * CompVQC Framework: I will explore a systematic framework, CompVQC, to reduce the circuit length of quantum neural networks (QNNs) through model compression, which can achieve up to a 20% improvement in accuracy. This reveals that compilation-aware compression can enhance the robustness of QNNs on near-term noisy quantum devices (Chapter 4). The field of AI has recently undergone a significant transformation, ushering in the era of generative AI, where models like ChatGPT and Stable Diffusion create realistic and creative content for a wide range of applications. As generative AI becomes more prevalent, it is crucial to adapt algorithm-hardware co-design strategies to meet the specific needs of these models, particularly on edge devices. In the final chapter (Chapter 5), I will introduce my ongoing and future work focused on activating generative AI on the edge.--Author's abstract

Deep Neural Networks (DNNs) tend to go deeper and wider, which poses a significant challenge to the training of DNNs, due to the limited memory capacity of DNN accelerators. Existing solutions for memory-efficient DNN training are densely coupled with the application features of DNN workloads, e.g., layer structures or computational graphs of DNNs are necessary for these solutions. This would result in weak versatility for DNNs with sophisticated layer structures or complicated computation graphs. These schemes usually need to be re-implemented or re-adapted due to the new layer structures or the unusual operators in the computational graphs introduced by these DNNs. In this article, we review the memory pressure issues of DNN training from the perspective of runtime systems and model the memory access behaviors of DNN workloads. We identify the iterative, regularity , and extremalization properties of memory access patterns for DNN workloads. Based on these observations, we propose AppObMem, an application-oblivious memory scheduling system. AppObMem automatically traces the memory behaviors of DNN workloads and schedules the memory swapping to reduce the memory pressure of the device accelerators without the perception of high-level information of layer structures or computation graphs. Evaluations on a variety of DNN models show that, AppObMem obtains 40–60% memory savings with acceptable performance loss. AppObMem is also competitive with other open sourced SOTA schemes.

Energy-efficient Resistive RAM (ReRAM) based deep neural network (DNN) accelerator suffers from severe Stuck-At-Fault (SAF) problem that drastically degrades the inference accuracy. The SAF problem gets even worse in realistic ReRAM devices with low cell resolution. To address the issue, we propose a fault-resilient DNN accelerator based on realistic ReRAM devices. We first analyze the SAF problem in a realistic ReRAM device and propose a 3-stage offline fault-resilient compilation and lightweight online compensation. The proposed work enables the reliable execution of DNN with only 5% area and 0.8% energy overhead from the ideal ReRAM-based DNN accelerator.

Advanced Artificial Intelligence (AI) systems that process vast feature sets can be designed using Deep Neural Networks (DNN). The popularity and prevalence of DNN based tasks are mainly due to their ability to achieve state of the art accuracy. However, DNN based systems are computationally intensive with heavy on-chip and off-chip data movement demands. Efficient hardware accelerators with systematized interconnection platform support handling this insistent computation and fast communication requirements. With rise in accelerator complexity, it is important to identify and solve communication overheads imposed due to heavy data movement in the system. Need for increased data transfers come as a consequence of complex computations and structure of data storage hierarchy in different DNN models. Multi-core system with underlying Network on Chip (NoC) framework is a promising design choice for implementing DNN based systems with diverse layers. Data movement plays a vital role in deciding performance of such systems. Analysing metrics that affect latency and energy/power efficiency of system is thus crucial in deciding performance enhancement of these DNN inference accelerators. This paper quantifies the communication overheads associated with accelerating different DNN models which use NoC as its basic communication platform.

This article presents an energy-harvester-powered ultra-low power (ULP) vibration-based condition monitoring (VbCM) chip with a digital compute-in-memory (CIM)-based deep neural network (DNN) accelerator. The VbCM chip achieves end-to-end signal processing, including a piezoelectric (PZ)-based accelerometer sensor and its readout circuit (RoC), a digital CIM-based DNN accelerator and a ULP radio system. To perform the targeted rotating bearing anomaly detection task with best-in-class accelerator power consumption and energy efficiency, the chip integrates several technologies from the algorithm level to the hardware level, including: 1) Compressing deep neural network (C-DNN) to reduce the network size through a feature compressor module (FCM); 2) Signal chain characteristic adaptation from direct training of the time-domain feature extractor (TD-FEx); 3) ULP 13T SRAM CIM bitcell to perform high energy efficiency in-cell multiplication; and 4) ripple counter (RCNT)-based accumulation scheme to improve the overall energy efficiency of the accelerator under a 0.35 V supply voltage. The VbCM chip achieves 90.7 nW total power with an inference FR of 20 frames/s, and the DNN accelerator’s energy efficiency achieves 18.8 fJ/MAC. With all the weight parameters statically stored in the CIM macros, the accelerator can operate with an inference accuracy of 91.3% while consuming 24.7 nW.

Deep learning is being incorporated in many modern software systems. Deep learning approaches train a deep neural network (DNN) model using training examples, and then use the DNN model for prediction. While the structure of a DNN model as layers is observable, the model is treated in its entirety as a monolithic component. To change the logic implemented by the model, e.g. to add/remove logic that recognizes inputs belonging to a certain class, or to replace the logic with an alternative, the training examples need to be changed and the DNN needs to be retrained using the new set of examples. We argue that decomposing a DNN into DNN modules— akin to decomposing a monolithic software code into modules—can bring the benefits of modularity to deep learning. In this work, we develop a methodology for decomposing DNNs for multi-class problems into DNN modules. For four canonical problems, namely MNIST, EMNIST, FMNIST, and KMNIST, we demonstrate that such decomposition enables reuse of DNN modules to create different DNNs, enables replacement of one DNN module in a DNN with another without needing to retrain. The DNN models formed by composing DNN modules are at least as good as traditional monolithic DNNs in terms of test accuracy for our problems.

Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

The growing complexity and diversity of deep neural network (DNN) applications have inspired intensive research on specialized DNN accelerators and also the design automation frameworks. Previous specialized NN acceleratos roughly fall into two categories of implementation, either the no-pipelined architecture that relies on a generic processing unit (PU) to sequentially execute the DNN layers in a layer-wise way, or the fully-pipelined architecture that dedicates interconnected customized PUs to the corresponding DNN layers in the model. Thus, such designs often suffer from either the resource under-utilization issue faced by no-pipelined accelerators or the resource scalability problem brought by the over-deep pipeline designs. In this work, we propose a novel class of design solution for DNN acceleration, segment-grained pipeline architecture (SPA). In the SPA accelerator, the targeted workload of DNN models will be divided into many segments and each segment will be sequentially executed on the shared interconnected PUs in a pipeline manner, so that they will benefit from both the efficiency of pipelined execution and also the flexibility of sharing PUs across different model layers. Particularly, we found that the efficiency of the implemented SPA accelerator significantly depends on the segmentation strategies of the models and the hardware resources assignment policy for PUs. Therefore, we introduce an automated design framework, AutoSeg, that includes a parameterized SPA accelerator template and a co-design engine that will generate the efficient model segmentation solution and hardware pipeline design parameters for the acceleration workload. Experimental results show that the SPA solutions generated by the AutoSeg framework achieve $1.2\times to 6.3\times$ speedup when compared to ASIC-based general DNN processors, and the FPGA designs implemented by AutoSeg also achieve as high as $3.4\times$ DSP efficiency and $3.6\times$ throughput improvement.

FPGAs have been rapidly adopted for acceleration of Deep Neural Networks (DNNs) with improved latency and energy efficiency compared to CPU and GPU-based implementations. High-level synthesis (HLS) is an effective design flow for DNNs due to improved productivity, debugging, and design space exploration ability. However, optimizing large neural networks under resource constraints for FPGAs is still a key challenge. In this paper, we present a series of effective design techniques for implementing DNNs on FPGAs with high performance and energy efficiency. These include the use of configurable DNN IPs, performance and resource modeling, resource allocation across DNN layers, and DNN reduction and re-training. We showcase several design solutions including Long-term Recurrent Convolution Network (LRCN) for video captioning, Inception module for FaceNet face recognition, as well as Long Short-Term Memory (LSTM) for sound recognition. These and other similar DNN solutions are ideal implementations to be deployed in vision or sound based IoT applications.

As training a high-performance deep neural network (DNN) model requires a large amount of data, powerful computing resources and expert knowledge, protecting well-trained DNN models from intellectual property (IP) infringement has raised serious concerns in recent years. Most existing methods using DNN watermarks to verify the ownership of the models after IP infringement occurs, which is reactive in the sense that they cannot prevent unauthorized users from using the model in the first place. Different from these methods, in this article, we propose an active authorization control and user’s fingerprint tracking method for the IP protection of DNN models by utilizing sample-specific backdoor attack. The proposed method inversely and multiplely exploits sample-specific trigger as the key to implement authorization control for DNN model, in which the generated triggers are imperceptible and sample-specific for clean images. Specifically, a U-Net model is used to generate backdoor instances. Then, the target model is trained on the clean images and backdoor instances, which are inversely labeled as wrong classes and correct classes, respectively. Only authorized users can use the target model normally by pre-processing the clean images through the U-Net model. Moreover, the images processed by the U-Net model will contain unique fingerprint that can be extracted to verify and track the corresponding user’s identity. This article is the first work that utilizes the sample-specific backdoor attack to implement active authorization control and user’s fingerprint management for DNN model under black-box scenarios. Extensive experimental results on ImageNet dataset and YouTube Aligned Face dataset demonstrate that the proposed method is effective in protecting the DNN model from unauthorized usage. Specifically, the protected model has a low inference accuracy (1.00%) for unauthorized users, while maintaining a normal inference accuracy (97.67%) for authorized users. Besides, the proposed method can achieve 100% fingerprint tracking success rates on both the ImageNet and YouTube Aligned Face datasets. Moreover, it is demonstrated that the proposed method is robust against fine-tuning attack, pruning attack, pruning attack with retraining, reverse-engineering attack, adaptive attack, and JPEG compression attack. The code is available at https://github.com/nuaaaisec/SSAT .

In this article, for the first time, we propose channel characteristic-based deep neural network (DNN) models for accurate eye-height (EH) and eye-width (EW) estimation of high bandwidth memory (HBM) silicon interposer channels. The proposed models preprocess the design parameters that are highly relevant to the characteristics of the HBM channels. By taking account of the contribution of each design parameter to the eye diagram, the proposed models can accurately estimate the EH and EW of the channels even with a limited number of datasets. For verification, the proposed DNN models were applied to the microstrip and stripline channels of an HBM silicon interposer. Only redistributed layer (RDL) was used to clearly see the effect of reflecting the channel characteristics of the proposed method. We compared the proposed DNN models with various regression methods and a conventional fully connected multilayer DNN model. As a result, the proposed DNN models reduced the EH and EW error rates by 22.7 and 43.9&#x0025; compared to the other regression methods. In addition, the proposed DNN models not only reduced the error rates by 22.0&#x2013;28.4&#x0025; but also reduced the computing cost by 8.0&#x2013;9.4&#x0025;, compared to the conventional DNN model. Moreover, we compared the proposed models with various DNN models having other preprocessing structures. By showing 26.7 and 28.8&#x0025; lower EH and EW error rates than the other DNN models, we validated that the proposed models properly consider the most dominant design factors in preprocessing.