Advanced Quantization Methods for CNNs

In this paper, we analyze the suitability of convolutional neural network (CNN) inference workloads on a phase-change memory (PCM) platform. CNN inference has an average of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$14\times$</tex> more read requests than write requests (i.e., read dominant) and a significantly low last-level cache misses per kilo instructions (LLC MPKI) of 2 on average (i.e., computation intensive). In addition, to compare the latency and energy of PCM and DRAM systems, we evaluate CNN inference workloads on two memory systems through a memory simulator. As a result, compared to DRAM, PCM can save total energy by 54% on average, but instruction per cycle (IPC) of PCM is reduced by an average of 28%. In conclusion, CNN inference is a workload suitable for PCM in terms of energy efficiency, but it must be accompanied by a scheme to improve IPC for practical use.

Convolutional neural network (CNN) inference on mobile devices demands efficient hardware acceleration of low-precision (INT8) general matrix multiplication (GEMM). The systolic array (SA) is a pipelined 2D array of processing elements (PEs), with very efficient local data movement, well suited to accelerating GEMM, and widely deployed in industry. In this letter, we describe two significant improvements to the traditional SA architecture, to specifically optimize for CNN inference. First, we generalize the traditional scalar PE, into a Tensor-PE, which gives rise to a family of new Systolic Tensor Array (STA) microarchitectures. The STA family increases intra-PE operand reuse and datapath efficiency, resulting in circuit area and power dissipation reduction of as much as 2.08× and 1.36× respectively, compared to the conventional SA at iso-throughput with INT8 operands. Second, we extend this design to support a novel block-sparse data format called density-bound block (DBB). This variant (STA-DBB) achieves a 3.14× and 1.97× improvement over the SA baseline at iso-throughput in area and power respectively, when processing specially-trained DBB-sparse models, while remaining fully backwards compatible with dense models.

For model inference of convolutional neural networks (CNNs), we nowadays witness a shift from the Cloud to the Edge. Unfortunately, deploying and inferring large, compute- and memory-intensive CNNs on Internet of Things devices at the Edge is challenging as they typically have limited resources. One approach to address this challenge is to leverage all available resources across multiple edge devices to execute a large CNN by properly partitioning it and running each CNN partition on a separate edge device. However, there currently does not exist a design and programming framework that takes a trained CNN model as input and subsequently allows for efficiently exploring and automatically implementing a range of different CNN partitions on multiple edge devices to facilitate distributed CNN inference. Therefore, in this article, we propose a novel framework that automates the splitting of a CNN model into a set of submodels as well as the code generation needed for the distributed and collaborative execution of these submodels on multiple, possibly heterogeneous, edge devices, while supporting the exploitation of parallelism among and within the edge devices. In addition, since the number of different CNN mapping possibilities on multiple edge devices is vast, our framework also features a multistage and hierarchical design space exploration methodology to efficiently search for (near-)optimal distributed CNN inference implementations. Our experimental results demonstrate that our work allows for rapidly finding and realizing distributed CNN inference implementations with reduced energy consumption and memory usage per edge device, and under certain conditions, with improved system throughput as well.

Convolutional neural networks (CNNs) have achieved significant accuracy improvement in many intelligent applications at the cost of intensive convolution operations and massive data movements. To efficiently deploy CNNs on low power embedded platforms in real time, the depthwise separable convolution has been proposed to replace the standard convolution, especially in lightweight CNNs, which remarkably reduces the computation complexity and model size. However, it is difficult for a general convolution engine to obtain the theoretical performance improvement as the decreased data dependency of depthwise convolution significantly reduces the data reuse opportunity. To address this issue, a flexible and highperformance accelerator based on FPGA is proposed to efficiently process the inference of both large-scale and lightweight CNNs. Firstly, by sharing the activation dataflow between the depthwise convolution and pooling layers, the control logic and data bus of the two layers are reused to maximize the data utilization and minimize the logic overhead. Furthermore, these two layers can be processed either directly after standard convolutions to eliminate the external memory accesses or independently to gain better flexibility. Thirdly, a performance model is proposed to automatically explore the optimal design options of the accelerator. The proposed hardware accelerator is evaluated on Intel Arria 10 SoC FPGA and demonstrates state-of-the-art performance on both large-scale CNNs, e.g., VGG, and lightweight ones, e.g., MobileNet.

This paper compares three different optimization approaches for accelerating the inference of convolutional neural networks (CNNs). We compare the techniques of separable convolution, weight pruning, and binarization. Each method is implemented and empirically compared in three aspects: preservation of accuracy, storage requirements, and achieved speed-up. Experiments are performed both on a desktop computer and on a mobile platform using a CNN model for vehicle type classification. Our experiments show that the largest speed-up is achieved by binarization, whereas pruning achieves the largest reduction in storage requirements. Both of these approaches largely preserve the accuracy of the original network.

Efficient inference of Convolutional Neural Network (CNN) is a challenging task and design choices heavily rely on the target context and size of the CNN model. Many devices are available, each one targeting a specific class of application. The most famous ones target the server-side of cloud applications and some focus on embedded applications. In this paper we show how to exploit the low-level hardware features of an embedded multicore called STxP70 ASMP, each core being equipped with a vector coprocessor. This work shows how to adapt the algorithm to the platform and vice-versa, and provides an original algo rithmic transform to optimize internal resources. Experiments are made to study the effect of numerous design parameters and CNN configurations. The results show the benefits of the proposed strategy and outline the low-level hardware features required to further optimize CNN inference.

Computing convolutional layers in frequency domain can largely reduce the computation overhead for training and inference of convolutional neural networks (CNNs). However, existing designs with such an idea require repeated spatial- and frequency-domain transforms due to the absence of nonlinear functions in the frequency domain, as such it makes the benefit less attractive for low-latency inference. This paper presents a fully spectral CNN approach by proposing a novel adaptive Rectified Linear Unit (ReLU) activation in spectral domain. The proposed design maintains the non-linearity in the network while taking into account the hardware efficiency in algorithm level. The spectral model size is further optimized by merging and fusing layers. Then, a customized hardware architecture is proposed to implement the designed spectral network on FPGA device with DSP optimizations for 8-bit fixed point multipliers. Our hardware accelerator is implemented on Intel's Arria 10 device and applied to the MNIST, SVHN, AT&T and CIFAR-10 datasets. Experimental results show a speed improvement of 6 × ~ 10 × and 4 × ~ 5.7 × compared to state-of-the-art spatial or FFT-based designs respectively, while achieving similar accuracy across the benchmark datasets.

We compare different platforms for inference of convolutional neural networks in this paper. We trained various neural networks to determine the material in the source hyperspectral cube. Then we convert them to inference format and compare the inference results. We used tools under Xilinx Vitis AI for FPGA implementation. We try to minimize the size of the proposed networks by pruning them and provide further comparisons. FPGA platforms show to be energy efficient but still slower than a graphics card in terms of performance.

Convolutional Neural Networks (CNNs) are becoming increasingly popular in deep learning applications, e.g. image classification, speech recognition, medicine, to name a few. However, the CNN inference is computationally intensive and demanding a large among of memory resources. In this work is proposed a CNN inference hardware accelerator, which was implemented in a co-processing scheme. The aim is to reduce the hardware resources and achieve the better possible throughput. The design was implemented in the Digilent Arty Z7-20 development board, which is based on System on Chip (SoC) Zynq-7000 of Xilinx. Our implementation achieved a of accuracy for the MNIST database using only 12-bits fixed-point format. The results show that the co-processing scheme operating at a conservative speed of 100 MHz can identify around 441 images per second, which is about 17% times faster than a 650 MHz - software implementation. It is difficult to compare our results against other implementations based on Field-Programmable Gate Array (FPGA), because the others implementations are not exactly like ours. However, some comparisons, regarding the logical resources used and accuracy, suggest that our work could be better than previous works.

[ES] La Inteligencia Artificial (IA) va camino de convertirse en la tecnología más transformadora del siglo XXI. La proliferación de diversas arquitecturas de redes neuronales, como CNN y LLM, ha posibilitado su integración en numerosos sectores industriales y aplicaciones. En la actualidad, la IA impulsa soluciones que incluyen: el reconocimiento del lenguaje, secuenciación de ADN, conducción autónoma, análisis de datos, entre muchos otros. Este rápido crecimiento ha impulsado a su vez la investigación y exploración de aceleradores hardware orientados a incrementar la eficiencia computacional y la capacidad de procesamiento. Entre las diferentes arquitecturas propuestas para el procesamiento eficiente de redes neuronales, se encuentran las FPGA. Las FPGAs son un tipo de circuito integrado que puede configurarse y reprogramarse posteriormente a su fabricación, lo que permite adaptar el hardware a distintos dominios de aplicación. Debido a su versatilidad, se utilizan para el procesamiento de redes neuronales en centros de datos y computación en el borde. No obstante, los limitados recursos disponibles en una única FPGA imponen restricciones a su capacidad para adaptarse al creciente tamaño de los modelos actuales. Dada la limitada capacidad de recursos, los sistemas multi-FPGA han emergido como una solución prometedora para el procesamiento de redes neuronales, ofreciendo una combinación de baja latencia, reducido consumo energético y reconfigurabilidad. Sin embargo, la implementación de sistemas multi-FPGA presenta diversos desafíos tanto a nivel de hardware como de software. En el marco de esta tesis se presenta el diseño e implementación de un sistema multi-FPGA innovador destinado al procesamiento eficiente de redes neuronales convolucionales. El sistema consta de ocho FPGAs interconectadas mediante una topología de cubo para una mejor conectividad y un mayor ancho de banda. Para abordar la elevada demanda de memoria de los modelos CNN, el sistema implementa un paradigma de memoria compartida distribuida, lo que permite a cada nodo acceder a todo el espacio de memoria del sistema. De este modo, se elimina la necesidad de realizar transferencias de memoria explicitas de datos entre nodos, simplificando el intercambio de información. Además, la arquitectura integra mecanismos de comunicación mediante streams, proporcionando colectivas de comunicación punto a punto, multicast y broadcast. La combinación de ambos paradigmas dota al sistema de una arquitectura novedosa, allanando el camino para el desarrollo de diversas estrategias de mapeo. Con el objetivo de explotar completamente los recursos del sistema, se han explorado diversas estrategias de mapeo. Inicialmente, se ha evaluado la eficiencia de distintas estrategias de particionado. Los resultados obtenidos evidencian la necesidad de estrategias más flexibles y adaptativas. En ese sentido, se ha desarrollado un algoritmo de particionado que analiza el modelo capa por capa, y selecciona la estrategia de particionado más adecuada en función de las características de cada capa. La eficiencia del algoritmo se ha evaluado con cuatro modelos de redes convolucionales: ResNet50, ResNet152, DeepLabV3 y InceptionV4. Los experimentos muestran que el algoritmo logra una aceleración máxima de 7.46 para el caso de DeepLabV4 con ocho FPGAs. A pesar de la escalabilidad obtenida con algunos modelos, los resultados ponen de manifiesto las limitaciones inherentes que tiene las estrategias de particionado. Para superar dichas limitaciones, el algoritmo ha sido extendido para incorporar un esquema de segmentación basado en comunicaciones por streaming, combinando ambas estrategias. El nuevo algoritmo mejora el rendimiento para todos los modelos, proporcionando una aceleración prácticamente lineal con hasta ocho FPGAs, demostrando la viabilidad y efectividad de los sistemas con múltiples FPGAs para acelerar cargas de trabajo con redes neuronales convolucionales.

Outsourcing Machine Learning (ML) tasks to cloud servers is a cost-effective solution when dealing with distributed data. However, outsourcing these tasks to cloud servers could lead to data breaches. Secure computing methods, such as Homomorphic Encryption (HE) and Trusted Execution Environments (TEE), have been used to protect outsourced data. Nevertheless, HE remains inefficient in processing complicated functions (e.g., non-linear functions) and TEE (e.g., Intel SGX) is not ideal for directly processing ML tasks due to side-channel attacks and parallel-unfriendly computation.In this paper, we propose a hybrid framework integrating SGX and HE, called HT2ML, to protect user's data and models. In HT2ML, HE-friendly functions are protected with HE and performed outside the enclave, while the remaining operations are performed inside the enclave obliviously. HT2ML leverages optimised HE matrix multiplications to accelerate HE computations outside the enclave while using oblivious blocks inside the enclave to prevent access-pattern-based attacks. We evaluate HT2ML using Linear Regression (LR) training and Convolutional Neural Network (CNN) inference as two instantiations. The performance results show that HT2ML is up to ∼11× faster than HE only baseline with 6-dimensional data in LR training. For CNN inference, HT2ML is ∼196× faster than the most recent approach (Xiao et al., ICDCS'21).

Deploying convolutional neural network (CNN) inference on resource-constrained devices remains a remarkable challenge for industrial Internet of Things (IIoT). Although the cloud computing shows great promise in machine learning training and prediction, outsourcing data to remote cloud always incurs privacy risk and high latency. Therefore, we design a new framework for efficient and privacy-preserving CNN inference based on cloud-edge-client collaboration <formula><tex>$(\rm{named} \, \text{PCNN}_{\text{CEC}})$</tex></formula>. In <formula><tex>$ \text{PCNN}_{\text{CEC}} $</tex></formula>, the model of cloud and the data of client in IIoT are split into two shares and sent to two non-colluded edge servers. By applying the arithmetic secret sharing and pre-computation of beaver's triplets, the two edge servers can jointly calculate the predicting results without learning anything about the model and data. To speed up the pre-computation of offline phase and not sacrifice security, the task of triplets generation is delegated to the cloud, so that the edge servers do not require frequent interactions to generate triplets themselves or introducing additional trusted party. The experimental results show the proposed private comparison protocol achieves a better tradeoff between low latency and high throughput, when it is compared with garbled circuit based protocols and other secret sharing based protocols. Additionally, the benchmarks conducted on realistic MNIST and CIFAR-10 datasets demonstrate that <formula><tex>$ \text{PCNN}_{\text{CEC}} $</tex></formula> costs less communication and runtime than two recently related schemes under the same security level.

Convolutional neural networks (CNNs) exhibit remarkable performance in various machine learning tasks. As sensor-equipped Internet of Things devices permeate into every aspect of modern life, the ability to execute CNN inference, a computationally intensive application, on resource constrained devices has become increasingly important. In this context, we present Cappuccino, a framework for synthesis of efficient inference software targeting mobile system-on-chips (SoCs). We propose techniques for efficient parallelization of CNN inference targeting mobile SoCs, and explore the underlying tradeoffs. Experiments with different CNNs on three mobile devices demonstrate the effectiveness of our approach.

With the wide adoption of AI applications, there is a pressing need of\nenabling real-time neural network (NN) inference on small embedded devices, but\ndeploying NNs and achieving high performance of NN inference on these small\ndevices is challenging due to their extremely weak capabilities. Although NN\npartitioning and offloading can contribute to such deployment, they are\nincapable of minimizing the local costs at embedded devices. Instead, we\nsuggest to address this challenge via agile NN offloading, which migrates the\nrequired computations in NN offloading from online inference to offline\nlearning. In this paper, we present AgileNN, a new NN offloading technique that\nachieves real-time NN inference on weak embedded devices by leveraging\neXplainable AI techniques, so as to explicitly enforce feature sparsity during\nthe training phase and minimize the online computation and communication costs.\nExperiment results show that AgileNN's inference latency is &gt;6x lower than the\nexisting schemes, ensuring that sensory data on embedded devices can be timely\nconsumed. It also reduces the local device's resource consumption by &gt;8x,\nwithout impairing the inference accuracy.\n

Convolutional neural networks (CNNs) have gained a huge attention for real-world artificial intelligence (AI) applications such as image classification and object detection. On the other hand, for better accuracy, the size of the CNNs’ parameters (weights) has been increasing, which in turn makes it difficult to enable on-device CNN inferences in resource-constrained edge devices. Though weight pruning and 5-bit quantization methods have shown promising results, it is still challenging to deploy large CNN models in edge devices. In this paper, we propose an encoding and hardware-based decoding technique which can be applied to 5-bit quantized weight data for on-device CNN inferences in resource-constrained edge devices. Given 5-bit quantized weight data, we employ arithmetic coding with range scaling for lossless weight compression, which is performed offline. When executing on-device inferences with underlying CNN accelerators, our hardware decoder enables a fast in-situ weight decompression with small latency overhead. According to our evaluation results with five widely used CNN models, our arithmetic coding-based encoding method applied to 5-bit quantized weights shows a better compression ratio by 9.6× while also reducing the memory data transfer energy consumption by 89.2%, on average, as compared to the case of uncompressed 32-bit floating-point weights. When applying our technique to pruned weights, we obtain better compression ratios by 57.5×–112.2× while reducing energy consumption by 98.3%–99.1% as compared to the case of 32-bit floating-point weights. In addition, by pipelining the weight decoding and transfer with the CNN execution, the latency overhead of our weight decoding with 16 decoding unit (DU) hardware is only 0.16%–5.48% and 0.16%–0.91% for non-pruned and pruned weights, respectively. Moreover, our proposed technique with 4-DU decoder hardware reduces system-level energy consumption by 1.1%–9.3%.