Accelerating Fully Spectral CNNs with Adaptive Activation Functions on FPGA

In this paper, we analyze the suitability of convolutional neural network (CNN) inference workloads on a phase-change memory (PCM) platform. CNN inference has an average of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$14\times$</tex> more read requests than write requests (i.e., read dominant) and a significantly low last-level cache misses per kilo instructions (LLC MPKI) of 2 on average (i.e., computation intensive). In addition, to compare the latency and energy of PCM and DRAM systems, we evaluate CNN inference workloads on two memory systems through a memory simulator. As a result, compared to DRAM, PCM can save total energy by 54% on average, but instruction per cycle (IPC) of PCM is reduced by an average of 28%. In conclusion, CNN inference is a workload suitable for PCM in terms of energy efficiency, but it must be accompanied by a scheme to improve IPC for practical use.

Convolutional neural network (CNN) inference on mobile devices demands efficient hardware acceleration of low-precision (INT8) general matrix multiplication (GEMM). The systolic array (SA) is a pipelined 2D array of processing elements (PEs), with very efficient local data movement, well suited to accelerating GEMM, and widely deployed in industry. In this letter, we describe two significant improvements to the traditional SA architecture, to specifically optimize for CNN inference. First, we generalize the traditional scalar PE, into a Tensor-PE, which gives rise to a family of new Systolic Tensor Array (STA) microarchitectures. The STA family increases intra-PE operand reuse and datapath efficiency, resulting in circuit area and power dissipation reduction of as much as 2.08× and 1.36× respectively, compared to the conventional SA at iso-throughput with INT8 operands. Second, we extend this design to support a novel block-sparse data format called density-bound block (DBB). This variant (STA-DBB) achieves a 3.14× and 1.97× improvement over the SA baseline at iso-throughput in area and power respectively, when processing specially-trained DBB-sparse models, while remaining fully backwards compatible with dense models.

Deep convolutional neural networks (CNNs) are widely used in many computer vision tasks. Since CNNs involve billions of computations, it is critical to reduce the resource /power consumption and improve parallelism. Compared with extensive researches on fixed point conversion for cost reduction, floating point customization has not been paid enough attention due to its higher cost than fixed point. This paper explores the customized floating point for both the training and inference of CNNs. 9-bit customized floating point is found sufficient for the training of ResNet-20 on CIFAR-10 dataset with less than 1% accuracy loss, which can also be applied to the inference of CNNs. With reduced bit-width, a computational unit (CU) based on Quad-Multiplier Packing is proposed to improve the resource efficiency of CNNs on FPGA. This design can save 87.5% DSP slices and 62.5% LUTs on Xilinx Kintex-7 platform compared to CU using 32-bit floating point. More CUs can be arranged on FPGA and higher throughput can be expected accordingly.

For model inference of convolutional neural networks (CNNs), we nowadays witness a shift from the Cloud to the Edge. Unfortunately, deploying and inferring large, compute- and memory-intensive CNNs on Internet of Things devices at the Edge is challenging as they typically have limited resources. One approach to address this challenge is to leverage all available resources across multiple edge devices to execute a large CNN by properly partitioning it and running each CNN partition on a separate edge device. However, there currently does not exist a design and programming framework that takes a trained CNN model as input and subsequently allows for efficiently exploring and automatically implementing a range of different CNN partitions on multiple edge devices to facilitate distributed CNN inference. Therefore, in this article, we propose a novel framework that automates the splitting of a CNN model into a set of submodels as well as the code generation needed for the distributed and collaborative execution of these submodels on multiple, possibly heterogeneous, edge devices, while supporting the exploitation of parallelism among and within the edge devices. In addition, since the number of different CNN mapping possibilities on multiple edge devices is vast, our framework also features a multistage and hierarchical design space exploration methodology to efficiently search for (near-)optimal distributed CNN inference implementations. Our experimental results demonstrate that our work allows for rapidly finding and realizing distributed CNN inference implementations with reduced energy consumption and memory usage per edge device, and under certain conditions, with improved system throughput as well.

Convolutional neural networks (CNNs) have achieved significant accuracy improvement in many intelligent applications at the cost of intensive convolution operations and massive data movements. To efficiently deploy CNNs on low power embedded platforms in real time, the depthwise separable convolution has been proposed to replace the standard convolution, especially in lightweight CNNs, which remarkably reduces the computation complexity and model size. However, it is difficult for a general convolution engine to obtain the theoretical performance improvement as the decreased data dependency of depthwise convolution significantly reduces the data reuse opportunity. To address this issue, a flexible and highperformance accelerator based on FPGA is proposed to efficiently process the inference of both large-scale and lightweight CNNs. Firstly, by sharing the activation dataflow between the depthwise convolution and pooling layers, the control logic and data bus of the two layers are reused to maximize the data utilization and minimize the logic overhead. Furthermore, these two layers can be processed either directly after standard convolutions to eliminate the external memory accesses or independently to gain better flexibility. Thirdly, a performance model is proposed to automatically explore the optimal design options of the accelerator. The proposed hardware accelerator is evaluated on Intel Arria 10 SoC FPGA and demonstrates state-of-the-art performance on both large-scale CNNs, e.g., VGG, and lightweight ones, e.g., MobileNet.

Convolutional Neural Networks (CNNs) are becoming increasingly popular in deep learning applications, e.g. image classification, speech recognition, medicine, to name a few. However, the CNN inference is computationally intensive and demanding a large among of memory resources. In this work is proposed a CNN inference hardware accelerator, which was implemented in a co-processing scheme. The aim is to reduce the hardware resources and achieve the better possible throughput. The design was implemented in the Digilent Arty Z7-20 development board, which is based on System on Chip (SoC) Zynq-7000 of Xilinx. Our implementation achieved a of accuracy for the MNIST database using only 12-bits fixed-point format. The results show that the co-processing scheme operating at a conservative speed of 100 MHz can identify around 441 images per second, which is about 17% times faster than a 650 MHz - software implementation. It is difficult to compare our results against other implementations based on Field-Programmable Gate Array (FPGA), because the others implementations are not exactly like ours. However, some comparisons, regarding the logical resources used and accuracy, suggest that our work could be better than previous works.

The convolutional neural networks (CNN), including AlexNet, GoogleNet, VGGNet, etc. extract features for many computer vision problems which are very discriminative. The trained CNN model over one dataset performs reasonably well whereas on another dataset of similar type the hand-designed feature descriptor outperforms the same trained CNN model. The Rectified Linear Unit (ReLU) layer discards some values in order to introduce the non-linearity. In this paper, it is proposed that the discriminative ability of deep image representation using trained model can be improved by Average Biased ReLU (AB-ReLU) at the last few layers. Basically, AB-ReLU improves the discriminative ability in two ways: 1) it exploits some of the discriminative and discarded negative information of ReLU and 2) it also neglects the irrelevant and positive information used in ReLU. The VGGFace model trained in MatConvNet over the VGG-Face dataset is used as the feature descriptor for face retrieval over other face datasets. The proposed approach is tested over six challenging, unconstrained and robust face datasets (PubFig, LFW, PaSC, AR, FERET and ExtYale) and also on a large scale face dataset (PolyUNIR) in retrieval framework. It is observed that the AB-ReLU outperforms the ReLU when used with a pre-trained VGGFace model over the face datasets. The validation error by training the network after replacing all ReLUs with AB-ReLUs is also observed to be favorable over each dataset. The AB-ReLU even outperforms the state-of-the-art activation functions, such as Sigmoid, ReLU, Leaky ReLU and Flexible ReLU over all seven face datasets.

Efficient inference of Convolutional Neural Network (CNN) is a challenging task and design choices heavily rely on the target context and size of the CNN model. Many devices are available, each one targeting a specific class of application. The most famous ones target the server-side of cloud applications and some focus on embedded applications. In this paper we show how to exploit the low-level hardware features of an embedded multicore called STxP70 ASMP, each core being equipped with a vector coprocessor. This work shows how to adapt the algorithm to the platform and vice-versa, and provides an original algo rithmic transform to optimize internal resources. Experiments are made to study the effect of numerous design parameters and CNN configurations. The results show the benefits of the proposed strategy and outline the low-level hardware features required to further optimize CNN inference.

This paper compares three different optimization approaches for accelerating the inference of convolutional neural networks (CNNs). We compare the techniques of separable convolution, weight pruning, and binarization. Each method is implemented and empirically compared in three aspects: preservation of accuracy, storage requirements, and achieved speed-up. Experiments are performed both on a desktop computer and on a mobile platform using a CNN model for vehicle type classification. Our experiments show that the largest speed-up is achieved by binarization, whereas pruning achieves the largest reduction in storage requirements. Both of these approaches largely preserve the accuracy of the original network.

This study presents a convolutional neural network (CNN)‐based response estimation model for structural health monitoring (SHM) of tall buildings subject to wind loads. In this model, the wind‐induced responses are estimated by CNN trained with previously measured sensor signals; this enables the SHM system to operate stably even when a sensor fault or data loss occurs. In the presented model, top‐level wind‐induced displacement in the time and frequency domains, and wind data in the frequency domain are configured into the input map of the CNN to reflect the resisting capacity of a tall building, the change in the dynamic characteristics of the building due to wind loads, and the relationship between wind load and the building. To evaluate stress, which is used as a safety indicator for structural members in the building, the maximum and minimum strains of columns are set as the output layer of the CNN. The CNN is trained using measured wind and wind response data to predict the column strains during a future wind load. The presented model is validated using data from a wind tunnel test of a building model. The performance of the presented model is verified through strain estimation with data that were not used in the CNN training. To assess the validity of the presented input map configuration, the estimation performance is compared with a CNN that considered only the time domain responses as input. Furthermore, the effects of the variations in the configuration of the CNN on the wind response estimation performance are examined.

Advances in deep learning, especially Convolutional Neural Networks (CNNs) have revolutionized intelligent frame-works such as Human Activity Recognition (HAR) systems by effectively and efficiently inferring human activity from various modalities of data. However, the training and inference of CNNs are often resource-intensive. Recent research developments are focused on bringing the effectiveness of CNNs in resource con-strained edge devices through Tiny Machine Learning (TinyML). However, this is extremely hard to achieve due to the limitations in memory, compute power, and energy of resource constrained edge devices. This paper provides a benchmark to understand these trade-offs among variations of CNN network architectures, different training methodologies, and different modalities of data in the context of HAR, TinyML, and edge devices. We tested and reported the performance of CNN and Depthwise Separable CNN (DSCNN) models as well as two training methodologies: Quantization Aware Training (QAT) and Post-training Quantization (PTQ) on five commonly used benchmark datasets containing image and time-series data: UP-Fall, Fall Detection Dataset (FDD), PAMAP2, UCI-HAR, and WISDM. We also deployed and tested the performance of the model-based standalone applications on multiple commonly available resource constrained edge devices in terms of inference time and power consumption. The experimental results demonstrate the effectiveness and feasibility of Tiny ML for HAR in edge devices.

Convolutional neural network (CNN) has been widely used in deep learning. However, the hardware consumption of the convolutional neural network is very large. Traditional Central Processing Units (CPUs) and Graphic Processing Units (GPUs) are inefficient and expensive for neural network, so an efficient hardware design is required. The proposed design based on Digital Signal Processor (DSP) has rapid operating speed and strong computation ability for training and inference of CNN. In this paper, the hardware architecture of convolution and softmax function is specially optimized. Winograd algorithm can reduce multiplications of convolution, thus decreases hardware complexity, since multiplication is much more complex in hardware implementation than addition. The softmax function is also simplified by replacing divider by subtractor and logarithmic function which cost fewer resources. The proposed hardware architecture dramatically decreases the complexity and hardware resources.

Maize kernel traits such as kernel length, kernel width, and kernel number determine the total kernel weight and, consequently, maize yield. Therefore, the measurement of kernel traits is important for maize breeding and the evaluation of maize yield. There are a few methods that allow the extraction of ear and kernel features through image processing. We evaluated the potential of deep convolutional neural networks and binary machine learning (ML) algorithms (logistic regression (LR), support vector machine (SVM), AdaBoost (ADB), Classification tree (CART), and the K-Neighbor (kNN)) for accurate maize kernel abortion detection and classification. The algorithms were trained using 75% of 66 total images, and the remaining 25% was used for testing their performance. Confusion matrix, classification accuracy, and precision were the major metrics in evaluating the performance of the algorithms. The SVM and LR algorithms were highly accurate and precise (100%) under all the abortion statuses, while the remaining algorithms had a performance greater than 95%. Deep convolutional neural networks were further evaluated using different activation and optimization techniques. The best performance (100% accuracy) was reached using the rectifier linear unit (ReLu) activation procedure and the Adam optimization technique. Maize ear with abortion were accurately detected by all tested algorithms with minimum training and testing time compared to ear without abortion. The findings suggest that deep convolutional neural networks can be used to detect the maize ear abortion status supplemented with the binary machine learning algorithms in maize breading programs. By using a convolution neural network (CNN) method, more data (big data) can be collected and processed for hundreds of maize ears, accelerating the phenotyping process.

For the internet of medical things (IoMT) enabled long-term health monitoring and disease prediction applications, there is a demand for an automatic photoplethysmogram (PPG) signal quality assessment (SQA) for reducing false alarms and energy consumption. This paper presents a PPG-SQA method by using raw PPG signal and convolutional neural network (CNN) with optimal parameters. The main focus of this paper is to find an optimal number of filters (16, 32, 64) and number of layers (2 and 4 layers) with rectified linear unit (ReLU) activation function and to study robustness of trained CNN models by using unseen PPG datasets and different kinds of noise sources, which are not addressed in the past studies on the CNN-based PPG-SQA methods. Evaluation results showed that the 4 layer CNN-based method had the higher accuracy of 99.58% for noise-free PPG (NF-PPG) versus wrist-cup noisy PPG signal database (MA-DB01), 99.99% for NF versus random noises added PPG (RN-PPG) signals, and 75.80% for NF-PPG versus acceleration corrupted PPG signals (MA-DB02). For the unknown dataset, the 4-layer CNN model had the higher accuracy of 96.71% for NF-PPG versus MA-DB01 and 99.04% for NF versus RN-PPG, and the 2-layer CNN model had the higher accuracy of 76.16% for NF-PPG versus MA-DB02 PPG segments. Results demonstrate that the CNN-based PPG-SQA method with optimal parameters is not only improve the accuracy but can also reduce the computational load as compared with other existing methods.

Deploying convolutional neural network (CNN) inference on resource-constrained devices remains a remarkable challenge for industrial Internet of Things (IIoT). Although the cloud computing shows great promise in machine learning training and prediction, outsourcing data to remote cloud always incurs privacy risk and high latency. Therefore, we design a new framework for efficient and privacy-preserving CNN inference based on cloud-edge-client collaboration <formula><tex>$(\rm{named} \, \text{PCNN}_{\text{CEC}})$</tex></formula>. In <formula><tex>$ \text{PCNN}_{\text{CEC}} $</tex></formula>, the model of cloud and the data of client in IIoT are split into two shares and sent to two non-colluded edge servers. By applying the arithmetic secret sharing and pre-computation of beaver's triplets, the two edge servers can jointly calculate the predicting results without learning anything about the model and data. To speed up the pre-computation of offline phase and not sacrifice security, the task of triplets generation is delegated to the cloud, so that the edge servers do not require frequent interactions to generate triplets themselves or introducing additional trusted party. The experimental results show the proposed private comparison protocol achieves a better tradeoff between low latency and high throughput, when it is compared with garbled circuit based protocols and other secret sharing based protocols. Additionally, the benchmarks conducted on realistic MNIST and CIFAR-10 datasets demonstrate that <formula><tex>$ \text{PCNN}_{\text{CEC}} $</tex></formula> costs less communication and runtime than two recently related schemes under the same security level.