FPGA-based training of convolutional neural networks with a reduced precision floating-point library

Training neural networks with many processors can reduce time-to-solution; however, it is challenging to maintain convergence and efficiency at large scales. The Kroneckerfactored Approximate Curvature (K-FAC) was recently proposed as an approximation of the Fisher Information Matrix that can be used in natural gradient optimizers. We investigate here a scalable K-FAC design and its applicability in convolutional neural network (CNN) training at scale. We study optimization techniques such as layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC update decoupling to reduce training time while preserving convergence. We use residual neural networks (ResNet) applied to the CIFAR10 and ImageNet-1k datasets to evaluate the correctness and scalability of our K-FAC gradient preconditioner. With ResNet-50 on the ImageNet-1k dataset, our distributed K-FAC implementation converges to the 75.9% MLPerf baseline in 18-25% less time than does the classic stochastic gradient descent (SGD) optimizer across scales on a GPU cluster.

Information extraction and coding of free-text pathology reports is an important activity for cancer registries to support national cancer surveillance. Cancer registrars must process high volumes of pathology reports on an annual basis. In this study, we investigated an automated approach using a coarse-to-fine training of convolutional neural networks (CNNs) for extracting the primary site, histological grade and laterality from unstructured cancer pathology text reports. Our proposed training scheme consists of two stages. In the first stage, the multi-task learning (MTL) with hard parameter sharing approach is used to train a multi-task MT-CNN model for all the tasks. Then, the TM-CNN model parameters are used to initialize a CNN model for each task to be fine trained individually using its corresponding dataset. The performance of our proposed approach was compared against a state-of-the-art CNN and the commonly used SVM classifier. We observed that the proposed model consistently outperformed the base line models, especially for the less prevalent classes. Specifically, the proposed training approach achieved a micro-F score of 0.7749 over 12 ICD-O-3 topography codes which is a significant improvement as compared with state-of-the-art CNN (0.7101) and the SVM (0.6019) classifiers. Also, the results demonstrate the potential of the proposed method for handling class imbalance within each task. It significantly improves macro-F score by 24% and 12% of the primary site and histology grade tasks, respectively.

Convolutional Neural Network (CNN) is a powerful and successful deep learning technique for a variety of computer vision and image analysis applications. Interpreting and explaining the decisions of CNNs is one of the most challengeable tasks despite its significant success in various image analysis tasks. Topological Data Analysis (TDA) is an approach that exploits algebraic invariants from topology to analyse high dimensional and noisy datasets as well as growing challenges of big data applications. Persistent homology (PH) is an algebraic topology method for measuring topological features of shapes and/or functions at different distance or similarity resolutions. This work is an attempt to investigate the algebraic properties of pretrained CNN convolutional layer filters based on random Gaussian/Uniform distribution. We shall investigate the stability and sensitivity of the condition number of CNN filters during and post the model training with focus on class discriminability of the PH features of the convolved images. We shall demonstrate a strong link between the condition number of the CNN filters and their discriminating power of the PH representation. In particular, we shall establish that if small perturbation added to the original images then feature maps with well-conditioned filters will produce similar topological features to the original image. Our investigation and findings are based on training CNN's with Digits, MNIST and CIFAR-10 datasets. Our ultimate interest in applying the results of these findings in designing appropriate CNN models for classifications of ultrasound tumor scan images. Preliminary results for these applications are encouraging.

Abstract. In recent years, there has been a significant improvement in the detection, identification and classification of objects and images using Convolutional Neural Networks. To study the potential of the Convolutional Neural Network, in this paper three approaches are investigated to train classifiers based on Convolutional Neural Networks. These approaches allow Convolutional Neural Networks to be trained on datasets containing only a few hundred training samples, which results in a successful classification. Two of these approaches are based on the concept of transfer learning. In the first approach features, created by a pretrained Convolutional Neural Network, are used for a classification using a support vector machine. In the second approach a pretrained Convolutional Neural Network gets fine-tuned on a different data set. The third approach includes the design and training for flat Convolutional Neural Networks from the scratch. The evaluation of the proposed approaches is based on a data set provided by the IEEE Geoscience and Remote Sensing Society (GRSS) which contains RGB and LiDAR data of an urban area. In this work it is shown that these Convolutional Neural Networks lead to classification results with high accuracy both on RGB and LiDAR data. Features which are derived by RGB data transferred into LiDAR data by transfer learning lead to better results in classification in contrast to RGB data. Using a neural network which contains fewer layers than common neural networks leads to the best classification results. In this framework, it can furthermore be shown that the practical application of LiDAR images results in a better data basis for classification of vehicles than the use of RGB images.

Computing convolutional layers in frequency domain can largely reduce the computation overhead for training and inference of convolutional neural networks (CNNs). However, existing designs with such an idea require repeated spatial- and frequency-domain transforms due to the absence of nonlinear functions in the frequency domain, as such it makes the benefit less attractive for low-latency inference. This paper presents a fully spectral CNN approach by proposing a novel adaptive Rectified Linear Unit (ReLU) activation in spectral domain. The proposed design maintains the non-linearity in the network while taking into account the hardware efficiency in algorithm level. The spectral model size is further optimized by merging and fusing layers. Then, a customized hardware architecture is proposed to implement the designed spectral network on FPGA device with DSP optimizations for 8-bit fixed point multipliers. Our hardware accelerator is implemented on Intel's Arria 10 device and applied to the MNIST, SVHN, AT&T and CIFAR-10 datasets. Experimental results show a speed improvement of 6 × ~ 10 × and 4 × ~ 5.7 × compared to state-of-the-art spatial or FFT-based designs respectively, while achieving similar accuracy across the benchmark datasets.

Convolutional Neural Networks (CNNs) experience performance and training efficiency changes according to the selection of correct hyperparameters. The research presents WACSO which combines Crow Search Optimization with Grey Wolf Optimizer to improve Convolutional Neural Networks hyperparameter selection through a hybrid metaheuristic algorithm. The hybrid algorithm WACSO uses exploration parts from CSO together with GWO exploitation mechanics to obtain optimized performance. WACSO reaches higher classification accuracy than traditional optimization algorithms when performing tests on the MNIST and CIFAR-10 datasets along with Random Search and particle swarm optimization and genetic algorithms and standalone CSO and standalone GWO. The best classification results reached 98.9% accuracy levels on MNIST along with 91.5% accuracy levels on CIFAR-10. The final outcomes of this system depend on the combination of model structure along with dataset challenges and available computational power. The investigation demonstrates that mixing algorithms drawn from nature can lead to successful CNN hyperparameter optimization. The promising outcomes of WACSO depend on multiple variables including computation expenses and sensitive parameter adjustments and universal result adaptability between different datasets and network setups. Research into WACSO should expand to involve longer evaluations across multiple datasets and various models to confirm widespread usage.

Glass forming ability prediction of bulk metallic glasses based on fused strategy

Recently, the deep learning methods have achieved great success in pattern recognition tasks. Especially for character recognition, most of the state-of-the-art results belong to the deep learning models. Among those models, the convolutional neural network (CNN) becomes the most popular due to its outstanding performance. Therefore, many trials were made in order to make improvements on CNN. However, most of the trials only focused on the network structure or training skills, the inter-class information is usually ignored. In this paper, we have proposed a novel CNN model with two training feedbacks: the reconstruction feedback and the classification feedback. By using the reconstruction feedback, the inter-class information (for example, shape similarity) of the characters is taken into account. Consequently, without enlarging the network structure, our model can outperform those state-of-the-art improved CNN models, which is proved by the experimental results.

This paper presents a fast algorithmic method to train convolutional neural network (CNN) classifiers through extreme learning which has been verified on popular datasets on classification and pedestrian detection. CNN has been one of the best classifiers for images and object recognition. However, the Backpropagation (BP) algorithm, mostly used for training CNN, suffers from slow learning, local minimum, and poor generalization. To solve these problems, a novel architecture called CNN-ELM has been proposed here. Its core architecture is based on a local image (local receptive field) version of the ELM (Extreme Learning Machine) adopting random feature learning. Using MATLAB 2015a, classification experiments using the raw image data as input, show a comparable or mostly better classification performance compared to the BP trained CNN, with its training speed up to 200 times faster on MNIST, NORB, and CIFAR-10 datasets. Pedestrian detection experiments using INRIA datasets also exhibits much faster training than the BP trained CNN without sacrificing detection performance.

Structural damage identification using strain mode differences by the iFEM based on the convolutional neural network (CNN)

In this study, a method of predicting the seismic responses of building structures based on a convolutional neural network (CNN) is proposed. In the method, the time histories of acceleration responses previously measured in a building during earthquakes are used in the CNN input layer, with the corresponding time histories of the displacement responses being used in the CNN output layer. The correlations between the features automatically extracted from the acceleration responses by the convolution and pooling operations in the various CNN layers and the displacement responses in the CNN output layer are established through CNN training. The trained CNN predicts the displacement responses for a future earthquake event using only the measured acceleration responses. To verify the validity of the proposed method, a numerical study on the ASCE benchmark model and an experimental study on a reinforced concrete frame structure are conducted. In the numerical study, the structural responses of the ASCE model subject to artificial earthquakes are utilized for CNN training, and the performance of the trained CNN is verified through seismic response prediction. In the experimental study, shaking table tests are performed for various earthquakes to measure the seismic responses of the reinforced concrete test structure. The seismic response prediction performance of the proposed method is examined using CNN trained with the measured seismic responses. Furthermore, the performances of CNNs trained with different numbers of datasets generated by the proposed data generation method based on data overlapping with the same data pool are discussed with related to the number of CNN training iteration number.

GPUs became the reference platform for both training and inference phases of Convolutional Neural Networks (CNN) due to their tailored architecture to the CNN operators. However, GPUs are power-hungry architectures. A path to enable the deployment of CNNs in energy-constrained devices is adopting hardware accelerators for the inference phase. The design space exploration of CNNs using standard approaches, such as RTL, is limited due to their complexity. Thus, designers need frameworks enabling design space exploration that delivers accurate hardware estimation metrics to deploy CNNs. This work proposes a framework to explore CNNs design space, providing power, performance, and area (PPA) estimations. The heart of the framework is a system simulator. The system simulator front-end is TensorFlow, and the back-end is performance estimations obtained from the physical synthesis of hardware accelerators, not only from components like multipliers and adders. The first set of results evaluate the CNN accuracy using integer quantization, the accelerators PPA after physical synthesis, and the benefits of using a system simulator. These results allow a rich design space exploration, enabling selecting the best set of CNN parameters to meet the design constraints.

To solve the problem of low accuracy of the traditional monitoring method, a deep learning based monitoring method is designed. A sensor is installed at the part that may cause the circuit breaker to run in non-full phase and the state data of the circuit breaker is collected through the sensor. Filter and digital-to-analog converter are used to reduce noise and digital-to-analog conversion of the signal transmitted by the sensor. After the data are standardized and normalized, input into the convolutional neural network. The parameters and weights of convolutional neural network are determined by training the sample data. After the training of convolutional neural network, the input neural network data are processed according to the determined parameters and weights. Output the processed results to the database, in the database classification algorithm after the data classification, sent to the circuit breaker operation control module, to achieve the circuit breaker operation status monitoring. Through the comparison with the traditional monitoring methods, it is verified that the designed deep learning-based state monitoring method has higher accuracy and can make up for the shortcomings of the traditional methods.

Stochastic Computing (SC) presents a low-cost and low-power alternative to conventional binary computing. In SC, continuous values are represented by stochastically generated bit streams. By performing simple hardware-friendly bit-wise operations on these streams, complex calculations can be realized very efficiently. However, the inherent randomness and approximation used in SC can result in undesirable computational errors. As Convolutional Neural Networks (CNNs) are inherently error-tolerant, SC could be embedded in them to gain higher speed and lower power without significant accuracy loss. In this paper, we propose using SC techniques to approximate multiplication operations on fixed-point weights and biases during training of CNNs. By employing such techniques, we demonstrate near state-of-the-art learning performance for the MNIST and CIFAR-10 datasets, while achieving significant resource and speed improvements when implementing the deep networks on a Field Programmable Gate Array (FPGA). For MNIST, we demonstrate that SC compared to conventional computing, will result in almost 3 times increase in learning speed with only 1.37% degradation in validation accuracy. Similarly, for CIFAR-10, training is accelerated 3.5 times with a degradation of 3.39%. We also show that our FPGA implementations of CNNs adopting stochastic multipliers consume over 17 times less power than their GPU counterparts.

In recent years, deep learning has made good progress and has been applied to face recognition, video monitoring, image processing, and other fields. In this big data background, deep convolution neural network has also received more and more attention. In order to extract the ancient Chinese characters effectively, the paper will discuss the structure model, pool process, and network training of deep convolution neural network and compare the algorithm with the traditional machine learning algorithm. The results show that the accuracy and recall rate of the Chinese characters in the plaque of Ming Dynasty can reach the peak, 81.38% and 81.31%, respectively. When the number of training samples increases to 50, the recognition rate of MFA is 99.72%, which is much higher than other algorithms. This shows that the algorithm based on deep convolution neural network and big data analysis has excellent performance and can effectively identify the Chinese characters under different dynasties, different sample sizes, and different interference factors, which can provide a powerful reference for the extraction of ancient Chinese characters.