Stochastic Computing for Low-Power and High-Speed Deep Learning on FPGA

Training neural networks with many processors can reduce time-to-solution; however, it is challenging to maintain convergence and efficiency at large scales. The Kroneckerfactored Approximate Curvature (K-FAC) was recently proposed as an approximation of the Fisher Information Matrix that can be used in natural gradient optimizers. We investigate here a scalable K-FAC design and its applicability in convolutional neural network (CNN) training at scale. We study optimization techniques such as layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC update decoupling to reduce training time while preserving convergence. We use residual neural networks (ResNet) applied to the CIFAR10 and ImageNet-1k datasets to evaluate the correctness and scalability of our K-FAC gradient preconditioner. With ResNet-50 on the ImageNet-1k dataset, our distributed K-FAC implementation converges to the 75.9% MLPerf baseline in 18-25% less time than does the classic stochastic gradient descent (SGD) optimizer across scales on a GPU cluster.

Convolutional Neural Network (CNN) is a powerful and successful deep learning technique for a variety of computer vision and image analysis applications. Interpreting and explaining the decisions of CNNs is one of the most challengeable tasks despite its significant success in various image analysis tasks. Topological Data Analysis (TDA) is an approach that exploits algebraic invariants from topology to analyse high dimensional and noisy datasets as well as growing challenges of big data applications. Persistent homology (PH) is an algebraic topology method for measuring topological features of shapes and/or functions at different distance or similarity resolutions. This work is an attempt to investigate the algebraic properties of pretrained CNN convolutional layer filters based on random Gaussian/Uniform distribution. We shall investigate the stability and sensitivity of the condition number of CNN filters during and post the model training with focus on class discriminability of the PH features of the convolved images. We shall demonstrate a strong link between the condition number of the CNN filters and their discriminating power of the PH representation. In particular, we shall establish that if small perturbation added to the original images then feature maps with well-conditioned filters will produce similar topological features to the original image. Our investigation and findings are based on training CNN's with Digits, MNIST and CIFAR-10 datasets. Our ultimate interest in applying the results of these findings in designing appropriate CNN models for classifications of ultrasound tumor scan images. Preliminary results for these applications are encouraging.

As the energy and hardware investments necessary for conventional high-precision digital computing continue to explode in the era of artificial intelligence (AI), a change in paradigm that can trade precision for energy and resource efficiency is being sought for many computing applications. Stochastic computing (SC) is an attractive alternative since, unlike digital computers, which require many logic gates and a high transistor volume to perform basic arithmetic operations such as addition, subtraction, multiplication, sorting, etc., SC can implement the same using simple logic gates. While it is possible to accelerate SC using traditional silicon complementary metal-oxide-semiconductor (CMOS) technology, the need for extensive hardware investment to generate stochastic bits (s-bits), the fundamental computing primitive for SC, makes it less attractive. Memristor and spin-based devices offer natural randomness but depend on hybrid designs involving CMOS peripherals for accelerating SC, which increases area and energy burden. Here, the limitations of existing and emerging technologies are overcome, and a standalone SC architecture embedded in memory and based on 2Dmemtransistors is experimentally demonstrated. The monolithic and non-von-Neumann SC architecture occupies a small hardware footprint and consumes a miniscule amount of energy (<1nJ) for both s-bit generation and arithmetic operations, highlighting the benefits of SC.

In this study, a method of predicting the seismic responses of building structures based on a convolutional neural network (CNN) is proposed. In the method, the time histories of acceleration responses previously measured in a building during earthquakes are used in the CNN input layer, with the corresponding time histories of the displacement responses being used in the CNN output layer. The correlations between the features automatically extracted from the acceleration responses by the convolution and pooling operations in the various CNN layers and the displacement responses in the CNN output layer are established through CNN training. The trained CNN predicts the displacement responses for a future earthquake event using only the measured acceleration responses. To verify the validity of the proposed method, a numerical study on the ASCE benchmark model and an experimental study on a reinforced concrete frame structure are conducted. In the numerical study, the structural responses of the ASCE model subject to artificial earthquakes are utilized for CNN training, and the performance of the trained CNN is verified through seismic response prediction. In the experimental study, shaking table tests are performed for various earthquakes to measure the seismic responses of the reinforced concrete test structure. The seismic response prediction performance of the proposed method is examined using CNN trained with the measured seismic responses. Furthermore, the performances of CNNs trained with different numbers of datasets generated by the proposed data generation method based on data overlapping with the same data pool are discussed with related to the number of CNN training iteration number.

Stochastic Number Generators (SNG) plays a significant role in designing a stochastic computing system. SNGs make the stochastic system comfortable for computing in the stochastic domain. The challenges in developing the stochastic computing system are correlation and hardware area occupancy. By considering these phenomena, we have considered Linear Feedback Shift Register (LFSR) based SNG and S-box based SNG in this work. Our contributions to this paper are stochastic computation for activation functions using the SNGs mentioned above and stochastic computation for arithmetic components in the stochastic domain. By considering the two SNG methods, the difference in the computation for accuracy has been analyzed for stochastic activation function and stochastic arithmetic computation. The better SNG method will be used as SNG for stochastic convolutional neural network design using this analysis.

This paper demonstrates image classification using deep learning. Deep learning has the inherent ability to automatically discover and extract meaningful features for specific applications. Among the popular techniques in deep learning, the convolutional neural network (CNN) stands out. CNN consists of an input layer, hidden layers, and an output layer, where meaningful features are automatically extracted from input images. This paper presents the performance and identifies the most effective CNN architectures for accurately classifying images in the CIFAR-10 dataset. Five CNN architectures were implemented, namely [Architecture 1], [Architecture 2], [Architecture 3], [Architecture 4], and [Architecture 5] using the CIFAR-10 dataset. The architectures were selected based on the need to explore variations in convolutional filter sizes, dense layers, and batch normalization to assess their impact on CIFAR-10 image classification performance. Each architecture was trained on a standard training set and evaluated on a validation set. We used specific details on data preprocessing and training settings for a consistent and fair comparison. After training and evaluation, we have obtained the following results for each architecture. Architecture 1 has a training accuracy of 74.7% and validation accuracy of 76.6%, Architecture 2 has 96.09% and 86.08%, Architecture 3 has 77% and 78.1%, Architecture 4 has 67.91% and 69.54%, and Architecture 5 has 94.64% and 87.34% of training accuracy and validation accuracy respectively. After conducting a comparative analysis, we found that Architecture 5 has achieved the highest validation accuracy in classifying images in the CIFAR-10 dataset. These findings suggest that Architecture 5 is a promising choice for image classification tasks involving the CIFAR-10 dataset.

Recently, CMOS invertible logic has been presented and is one of the new computing paradigms based on a probabilistic device model. It is designed based on stochastic computing that provides bidirectional operations between inputs and outputs and has been applied for several critical issues, such as integer factorization and machine learning (ML). This article presents an overview of CMOS invertible logic from principle to application. First, the principle is explained with a simple design example, and a design flow is introduced, as is an automatic design tool. Second, the hardware of CMOS invertible logic is designed using stochastic computing and then evaluated in two applications implemented on a field-programmable gate array (FPGA) or application-specific integrated circuits (ASICs). Finally, this article ends with future challenges.

Convolutional Neural Networks (CNNs) have been shown to have high accuracy for classification tasks in numerous applications, which has resulted in their widespread adoption. However, the high accuracy of CNNs comes at the cost of high compute and bandwidth requirements for both classification and training. In this work we discuss an FPGA-based CNN training engine: FCTE, implemented using High-Level Synthesis (HLS), targeting the Xilinx Kintex Ultrascale XCKU115 device. Furthermore, we detail custom-precision floating-point (CPFP) cores for multiplication and addition implemented using HLS, which allows for reduced area utilization. We use these cores with our engine to train networks to demonstrate that an exponent width of 6 and mantissa width of 5 achieves accuracy comparable to single-precision floating-point for the MNIST and CIFAR-10 datasets. These results are achieved using round-to-zero for the CPFP multipliers and round-to-nearest for the CPFP adders, allowing for LUT savings of 32.6% for the multipliers and 21.7% for the adders when compared to half-precision floating-point, while using the same number of DSPs.

This paper presents a fast algorithmic method to train convolutional neural network (CNN) classifiers through extreme learning which has been verified on popular datasets on classification and pedestrian detection. CNN has been one of the best classifiers for images and object recognition. However, the Backpropagation (BP) algorithm, mostly used for training CNN, suffers from slow learning, local minimum, and poor generalization. To solve these problems, a novel architecture called CNN-ELM has been proposed here. Its core architecture is based on a local image (local receptive field) version of the ELM (Extreme Learning Machine) adopting random feature learning. Using MATLAB 2015a, classification experiments using the raw image data as input, show a comparable or mostly better classification performance compared to the BP trained CNN, with its training speed up to 200 times faster on MNIST, NORB, and CIFAR-10 datasets. Pedestrian detection experiments using INRIA datasets also exhibits much faster training than the BP trained CNN without sacrificing detection performance.

Convolutional Neural Network (CNN) has achieved great success in various fields of machine vision, such as image classification and recognition, image segmentation, and video analysis. In specific applications, it is often necessary to customize the network structure by altering image size, convolution kernel size, pooling size, network architecture, and the number of network layers. These customizations pose significant challenges to the architecture of CNN accelerator especially in real-time systems, and as the number of these network variables increases, the internal convolution calculations become extremely demanding. The inference speed and energy consumption of CNN are becoming more and more important, which requires the design of the accelerator to adapt to different network architecture and run efficiently. Field-programmable gate arrays (FPGAs) is an ideal choice for a CNN accelerator due to its high programmability and low power consumption. To address the aforementioned challenges, we present NRUCA, a novel network reconfigurable universal CNN accelerator based on FPGA. On one hand, we have designed a flexible architecture that can dynamically configure CNN’s network structure parameters. These parameters can be sent to the FPGA via a configuration file, enabling the same design to run different CNN networks and allowing to modify on the fly. This forms the basis of the architecture’s adaptability presented in this paper. On the other hand, we employ a multi-channel parallel and efficient pipeline matrix multiplication architecture to implement the convolution and fully-connected layers, which constitute the majority of the computational load in CNNs. By utilizing an innovative inverted matrix multiplication algorithm and an interleaving cache data method, we reduce the internal cache required for multi-channel data. Furthermore, most of the intermediate calculation data does not need to be output to DDR, significantly improving the operation efficiency of the accelerator. We also fully leverage the FPGA chip architecture to compile multiple calculation kernels, which can be flexibly scheduled and combined to cater to various application scenarios. Based on the flexible and efficient overall design architecture, the layout of various resources of FPGA is balanced, which enables the designed FPGA project to compile four acceleration kernels with the running clock up to 300MHz in Xilinx Vitis and test them on Xilinx Alveo U250. We verified the adaptability and acceleration capabilities of our FPGA project using LeNet, AlexNet, VGG11, VGG13, VGG16, and VGG19 networks. Experiments demonstrated that our FPGA architecture achieves 33X and 35X energy savings compared to the Intel Xeon 5220R CPU, and 1.08X and 1.05X energy savings compared to the Nvidia Tesla P100 GPU, when accelerating AlexNet and VGG16, respectively.

This study presents a convolutional neural network (CNN)‐based response estimation model for structural health monitoring (SHM) of tall buildings subject to wind loads. In this model, the wind‐induced responses are estimated by CNN trained with previously measured sensor signals; this enables the SHM system to operate stably even when a sensor fault or data loss occurs. In the presented model, top‐level wind‐induced displacement in the time and frequency domains, and wind data in the frequency domain are configured into the input map of the CNN to reflect the resisting capacity of a tall building, the change in the dynamic characteristics of the building due to wind loads, and the relationship between wind load and the building. To evaluate stress, which is used as a safety indicator for structural members in the building, the maximum and minimum strains of columns are set as the output layer of the CNN. The CNN is trained using measured wind and wind response data to predict the column strains during a future wind load. The presented model is validated using data from a wind tunnel test of a building model. The performance of the presented model is verified through strain estimation with data that were not used in the CNN training. To assess the validity of the presented input map configuration, the estimation performance is compared with a CNN that considered only the time domain responses as input. Furthermore, the effects of the variations in the configuration of the CNN on the wind response estimation performance are examined.

Road detection aims at detecting the (drivable) road surface ahead vehicle and plays a crucial role in driver assistance system. To improve the accuracy and robustness of road detection approaches in complex environments, a new road detection method based on CNN (convolutional neural network) and MRF (markov random field) is proposed. The original road image is segmented into super-pixels of uniform size using simple linear iterative clustering (SLIC) algorithm. On this basis, we train the CNN which can automatically learn the features that are most beneficial to classification. Then, the trained CNN is applied to classify road region and non-road region. Finally, based on the relationship between the super-pixels neighborhood, we utilize MRF to optimize the classification results of CNN. Quantitative and qualitative experiments on the publicly datasets demonstrate that the proposed method is robust in complex environments. Furthermore, compared with state-of-the-art algorithms, the approach provides the better performance.

Attributed to the ever-increasing large image datasets, Convolutional Neural Networks (CNNs) have become popular for vision-based tasks. It is generally admirable to have larger-sized datasets for higher network training accuracies. However, the impact of dataset quality has not to be involved. It is reasonable to assume the near-duplicate images exist in the datasets. For instance, the Street View House Numbers (SVHN) dataset having cropped house plate digits from 0 to 9 are likely to have repetitive digits from the same/similar house plates. Redundant images may take up a certain portion of the dataset without consciousness. While contributing little to no accuracy improvement for the CNNs training, these duplicated images unnecessarily pose extra resource and computation consumption. To this end, this paper proposes a framework to assess the impact of the near-duplicate images on CNN training performance, called CE-Dedup. Specifically, CE-Dedup associates a hashing-based image deduplication approach with downstream CNNs-based image classification tasks. CE-Dedup balances the tradeoff between a large deduplication ratio and a stable accuracy by adjusting the deduplication threshold. The effectiveness of CE-Dedup is validated through extensive experiments on well-known CNN benchmarks. On one hand, while maintaining the same validation accuracy, CE-Dedup can reduce the dataset size by 23%. On the other hand, when allowing a small validation accuracy drop (by 5%), CE-Dedup can trim the dataset size by 75%.

Convolutional Neural Networks (CNNs) require massive parallelism due to the high-precision floating-point arithmetic operations they perform. So, demand of processing power in them is significantly higher than what a standard CPU can offer. This has traditionally made CNNs more suited for running on a Graphics Processing Unit (GPU). However, GPUs consume much more power than CPUs, rendering the former impractical for implementing CNNs in Edge AI (Artificial Intelligence), where restraining power consumption is paramount. On the other hand, FPGAs (Field Programmable Gate Arrays) are more suited for AI computing at the edge as they consume much lesser power than GPUs and even CPUs. Additionally, GPUs and CPUs are not suited for real-time AI applications, which require both high throughput and low latency at the same time and FPGAs excel at all these requirements. The purpose of this paper is to provide a study of the performance of FPGA as the most suitable platform for AI-based computing at the edge. To achieve this, we chose AlexNet, a popular CNN image classifier, for which we present a case study on four different platforms: CPU, GPU, embedded RISC core, and FPGA fabric. Then, we quantitatively measure and compare the performance in terms of inference time (time needed to classify an image) on all these platforms. Inference time using FPGA is reduced by almost 64, 1.6, and 1.1 times compared to a dual-core ARM, an i5-6400 CPU, and an Nvidia GPU, respectively.

Stochastic computing (SC) is a promising technique with advantages such as low-cost, low-power, and error-resilience. However so far SC-based CNN (convolutional neural network) accelerators have been kept to relatively small CNNs only, primarily due to the inherent precision disadvantage of SC. At the same time, previous SC architectures do not exploit the dynamic precision capability, which can be crucial in providing efficiency as well as flexibility in SC-CNN implementations. In this paper we present a DPS (dynamic precision scaling) SC-CNN that is able to exploit dynamic precision with very low overhead, along with the design methodology for it. Our experimental results demonstrate that our DPS SC-CNN is highly efficient and accurate up to ImageNet-targeting CNNs, and show efficiency improvements over conventional digital designs ranging in 50???100% in operations-per-area depending on the DNN and the application scenario, while losing less than 1% in recognition accuracy.