High-performance and energy-efficient deep learning for resource-constrained devices
Driven by the rapid development of deep neural networks (DNNs) in recent years, artificial intelligence applications have been flourishing in a spectrum of fields, such as image classification, object detection, machine translation, speech recognition, and smart homes. However, the enormous number of weight parameters and computations of DNN models require resource-rich devices, resulting in tremendous power and energy consumption. The sizes of the state-of-the-art DNN models are even increasingly massive, which further impede the deployment of DNNs in resource-constrained devices. This dissertation centers around addressing this challenge, and our efforts are classified into two directions: DNN model compression and hardware accelerator design. In pursuit of high-performance and energy-efficient DNN accelerator, investigating technologies beyond the conventional binary computing paradigm is desirable and we consider Stochastic computing (SC) a highly promising candidate. First, SC is a probabilistic computing paradigm, which uses a bit-sequence to represent a probability number by counting the number of ones in the sequence. This feature makes it suitable for DNN inference, which is essentially an approximate computing application and the final decision depends on the probabilities at the output layer. Second, SC is renowned as a footprint saver since many complex arithmetic operations can be implemented with simple logic components. For example, multiplication can be conducted with AND gates in SC. Consequently, these two fascinating features of SC make it a favorable alternative to conventional binary computing. In this dissertation, we present SC-DCNN, the first SC-based DCNN inference accelerator. Specifically, (i) we propose the design of diverse function blocks for the basic operations in DCNN. (ii) We propose the novel feature extraction blocks (FEBs), which are intended for extracting features from input feature maps. (iii) We propose comprehensive techniques to reduce the area and power (energy) consumption of weight storage. (iv) We propose holistic optimizations for the overall SC-DCNN architecture, with carefully selected layer-wise FEB configurations, to minimize area and power (energy) consumption while maintaining high network accuracy. Overall, the proposed SC-DCNN achieves the lowest hardware cost and energy consumption in implementing LeNet-5 compared with the state-of-the-art prior works. Besides emerging computing technologies, structured compression techniques, aiming at reducing the number of weights, and the corresponding accelerator design also require extensive research. Therefore, we study the block-circulant weight matrix (BCM)-based compression, which is suitable for serving this goal. BCM compression partitions the original weight matrix into blocks of square sub-matrices and each sub-matrix is trained into a circulant matrix. In a circulant matrix, each row vector can be produced by shifting its prior row vector by one element. Therefore, the whole matrix can be represented by only the first row vector, achieving significant storage and computation reduction as a result. The effectiveness of the algorithm has been verified on multiple representative DNN models, including both DCNNs for image classification and LSTMs for speech recognition. Moreover, we propose an ASIC accelerator design using the compression method. Experimental results show that the proposed BCM accelerator exhibits remarkable advantages in terms of power, throughput, and energy efficiency, indicating that this method is greatly desirable for resource-constrained devices running DNNs. In order to further boost compression ratios and advance energy-efficient deep learning, we propose ADMM-NN, a model-agnostic and systematic compression framework, unifying DNN pruning and quantization. In the proposed ADMM-NN, DNN compression is formulated as an optimization problem and solved using the alternating direction method of multipliers (ADMM). ADMM-NN first decomposes the optimization problem into two sub-problems. The first sub-problem is a neural network training problem with a regularization term, which regularizes the weight parameters to approach a specific compression pattern. The second problem is to find a local optimal compression pattern, which will then be fed back to the first problem. By iteratively solving the two relatively easy-to-solve sub-problems, a solution of the original problem can be found. Thereby, a high compression ratio can be obtained. Without accuracy loss, ADMM-NN achieved 85× and 24× pruning on LeNet-5 and AlexNet models, respectively. Combining weight pruning and quantization, we achieved 1,910× and 231× reductions in overall model size on these two benchmarks. Besides, 26× and 17.4× weight pruning ratios were observed on VGG-16 and ResNet-50, respectively. Furthermore, we propose a hardware-aware compression framework. Specifically, we studied the relationship between pruning ratios and speedups of running a pruned model, and the discovered relationship curve was then integrated into the framework to guide the pruning strategy. By applying the hardware-aware framework, ASIC synthesis results showed 3.6× overall speedup on conv1-conv5 layers of AlexNet. Based on ADMM-NN, we further propose a structured pruning algorithm for two reasons. First, ADMM-NN is a problem-solving framework that integrates neural network training and compression algorithm, but itself is not fixed to any specific compression algorithm. Therefore, we need to study an effective compression algorithm. Second, the conventional irregular pruning algorithm incurs high index and decoding overhead, and thus little acceleration can be achieved. Therefore, a pruning algorithm, which can produce a regular matrix structure and meanwhile is compatible with ADMM-NN, is also imperative. Since irregular pruning has been empirically proved to be able to achieve the highest compression ratios among a diverse variety of compression techniques, it is desirable to start the study by analyzing irregular pruning masks. We figured out three characteristics in the irregular pruning mask: (i) the number of retained weights in different rows varies significantly, and maintaining this variety helps sustain accuracy; (ii) denser rows are more sensitive to pruning than sparser rows; (iii) a block-max weight masking method is proposed to effectively sustain the overall salience of the weight matrix, meanwhile producing a high regularity. By leveraging the discovered characteristics, we propose the density-adaptive regular block (DARB) pruning, which can simultaneously achieve high compression ratios and high hardware performance. DARB was evaluated on five models, across three major application domains, and it outperforms the state-of-the-art prior generally by 2.4× to 4.8×. Besides, it achieves high decoding efficiency, which is defined as the number of activations selected for the corresponding retained weights per clock cycle. The hardware synthesis results showed that DARB outperforms block pruning with block size 4 × 4 and 8 × 8 by 14.3× and 3.6×, respectively. Meanwhile, DARB outperforms them on pruning ratios by 1.8× and 2.5×, respectively.
Read more