Logic Shrinkage: Learned Connectivity Sparsification for LUT-Based Neural Networks

Deep Neural Networks are resource-intensive learning models. To enable their training and deployment on low-end devices, decreasing their computational requirements has become an attractive research area in recent years. One of the most promising approaches in this field is designing networks which use only binary values for the network weights and activations, i.e., Binary Neural Networks (BNN). However, BNNs have robustness and stability issues, which make their training difficult and BNNs perform considerably worse than their real-valued counterparts. To improve the training and deployment performance of BNNs, in this work, we employ a tree-structured network model, which has been widely used in Deep Neural Networks (DNN). We perform experiments to analyze the accuracy, stability and robustness of the proposed tree-structured BNN model. Tree-structured network model integrates multiple classifiers on a backbone BNN resulting in a multi-branch architecture. Our experiments demonstrates that tree-structures model improves the accuracy, stability and robustness of BNNs.

To achieve an advanced Internet of Things (IoT), it is necessary to combine artificial intelligence (AI) with IoT. Compact circuits that can operate AI functions will be useful for this purpose. Therefore, we propose stochastic weights binary neural networks (SWBNN). SWBNNs are more accurate than binary neural networks (BNN) with small circuits. BNNs can be realized with small circuits since binary calculation needs simpler circuits than real number calculation. However, BNNs have lower accuracy than networks with real numbers. Thus, the proposed SWBNNs are BNNs that behave stochastically, which makes them more accurate than BNNs. Moreover, SWBNNs can still be achieved with small circuits since they execute binary calculation. As a result, the accuracy for the test data of SWBNNs is closer to the accuracy for learning data than the accuracy for the test data of BNNs is. Especially when using the CIFAR10 database, the difference in the identification accuracy rate between learning data and test data decreased from 6% for BNNs to 2% for SWBNNs. From results of a field-programmable gate array (FPGA) implementation, circuits of SWBNNs are sufficiently small although they are 10% bigger than those of BNNs. Therefore, SWBNNs are more accurate than BNNs, and the circuit costs ofintroducing stochastic weights are low.

Survey on Deep Neural Networks in Speech and Vision Systems

Based on the vigorous development of deep learning, image classification nowadays is mainly accomplished by using the Convolutional Neural Networks whose main computation tasks are usually calculated by Graphics Processing Units. Although the computing speed of Graphics Processing Units is faster than Central Processing Units when considering the neural networks, the power consumption of Graphics Processing Units is significantly huge. In this paper, we investigate the hardware acceleration of neural networks by using the Field Programmable Gate Array via High-Level Synthesis. We study the binarized neural networks with binarized weights and activation functions, which is suitable to be implemented by the Field Programmable Gate Array. The proposed design has high computing speed and low power consumption. Our method is applied to a robot vision application to classify robots from human and others. Experiment results show that our design provides good performance for image classification.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

Speaker verification based on deep neural network is one of the most prospective biometric authentication methods. But massive computation and parameters make it difficult to deploy speaker verification systems on memory and power constrained embedded devices. Binary neural networks with weights and activations compressed to 1 bit can reduce computational and memory cost significantly. However, speaker verification system based on binary neural networks suffers performance degradation problem. In this paper, to mitigate this problem and facilitate information flow across layers, we optimize binary neural network structure by adding a shortcut to connect layers. We propose an energy and memory efficient speaker system based on the optimized binary networks. Experimental results on text-dependent and text-independent datasets show that performance of proposed speaker verification system outperforms conventional binary neural networks significantly. Compared with full precision networks, the proposed system can largely maintain the performance while reducing memory and computational costs.

Binarized neural networks (BNNs) have shown exciting potential for utilising neural networks in embedded implementations where area, energy and latency constraints are paramount. With BNNs, multiply-accumulate (MAC) operations can be simplified to XnorPopcount operations, leading to massive reductions in both memory and computation resources. Furthermore, multiple efficient implementations of BNNs have been reported on field-programmable gate array (FPGA) implementations. This paper proposes a smaller, faster, more energy-efficient approximate replacement for the XnorPopcountoperation, called XNorMaj, inspired by state-of-the-art FPGAlook-up table schemes which benefit FPGA implementations. Weshow that XNorMaj is up to 2x more resource-efficient than the XnorPopcount operation. While the XNorMaj operation has a minor detrimental impact on accuracy, the resource savings enable us to use larger networks to recover the loss.

An efficient GPU-accelerated inference engine for binary neural network on mobile phones

This paper presents a field programmable gate array (FPGA)–based implementation of a deep neural network (DNN) for handwritten digit recognition. We propose the use of a fully connected four‐layer neural network with the hidden layers implementing the ReLU activation function and the output layer based on the Softmax activation function. The neural network model, including the forward propagation algorithm and the backward propagation (BP) algorithm, is entirely implemented in an FPGA system, enabling both testing and training of the network. Mathematical operations in the network are performed on 32‐bit floating‐point data. The proposed network achieves promising results in terms of system simplicity and low consumption of hardware resources while maintaining acceptable recognition accuracy and algorithm execution speed at the same time. The system is implemented on Xilinx’s ZYBO development board and achieves a precision of 92.8% digit recognition. Training and test images for system evaluation are obtained from the MNIST database. The system occupies 10,020 (57%) lookup tables, 7781 (22%) registers, 17 (21%) DSP blocks, and 3 (5%) BRAM blocks. The maximum power consumption of the system is equal to 1.602 W, at the operating frequency of the system equal to 100 MHz.

In this paper, aiming at the problems of difficult positioning, slow speed and low precision of digital printing, a detection system suitable for textile printing positioning is proposed and designed. This detection system innovatively combines a neural network and field programmable gate array (FPGA) to realize rapid and accurate positioning of printing. In the neural network part, this paper selects the backbone network Darknet19 of YOLOv2 as the backbone network, and under the premise of ensuring a certain detection accuracy, the network model is pruned and quantified to make it suitable for deployment on the embedded device FPGA. In addition, before the network training, this paper optimizes the candidate boxes by introducing k-means clustering to customize the analysis of the fabric print dataset to improve the detection accuracy. In the FPGA part, this paper optimizes the architecture on the FPGA side in two parts: data computation and data transmission. In terms of computational optimization, parallel optimization of the neural network is performed by combining FPGA optimization methods, such as pipeline and unroll. In terms of transmission optimization, we use a double-buffered design to ping-pong in the input and output modules to overlap the latency, and then use multi-port transmission to improve the overall bandwidth utilization and reduce the transmission latency caused by on-chip and off-chip interactions. The experimental results show that the detection system combining the neural network and FPGA can effectively position fabric prints and meet the needs of real-time. The design scheme has lower power compared to the graphics processing unit and is faster compared to the central processing unit.

Optimization strategies for neural network deployment on FPGA: An energy-efficient real-time face detection use case

The deep learning has become the key for artificial intelligence applications development. It was successfully used to solve computer vision tasks. But the deep learning algorithms are based on Deep Neural Networks (DNN) with many hidden layers which need a huge computation effort and a big storage space. Thus, the general-purpose graphical processing units (GPGPU) are the best candidate for DNN development and inference because of their high number of processing core and the big integrated memory. In the other side, the disadvantage of the GPGPU is high-power consumption. In a real-world application, the processing unit is an embedded system based on limited power and computation resources. In recent years, Field Programmable Gate Array (FPGA) becomes a serious solution that can outperform GPGPU because of their flexible architecture and low power consumption. The FPGA is equipped with a very small integrated memory and a low bandwidth. To make DNNs fit into FPGA we need a lot of optimization techniques at different levels such as the network level, the hardware level, and the implementation tools level. In this paper, we will cite the existing optimization techniques and evaluate them to provide a complete overview of FPGA based DNN accelerators.

Deep neural network (DNN) has powerful feature extraction and classification abilities and is extensively used in mechanical fault diagnosis. However, in the DNN-based fault diagnosis method, too many model parameters, high computational complexity, and occupying a large amount of storage space make it very difficult to optimize and retrain the DNN model. To overcome this deficiency, a binary neural network (BNN) is introduced into fault diagnosis. In the proposed BNN-based fault diagnosis method, the floating-point weights and activations are binarized as ‘+1’ and ‘−1’ to save storage space, and the floating-point multiply-accumulate operations (OPs) are replaced by bitwise OPs to reduce the computational complexity. The BNN-based fault diagnosis method effectively compresses the structure of DNNs and accelerates model inference. However, the binarization OP may bring about the quantization error and gradient error, which will accumulate with the increase of the network depth, and lead to a degradation in model performance. To solve these deficiencies in the proposed BNN-based fault diagnosis method, an improved BNN fault diagnosis method based on adaptive depth-to-width ratio (ADTWR) is proposed. For the convenience in description, the proposed method is called the BNN-ADTWR method. In the proposed BNN-ADTWR method, the network structure is reshaped by the comparison of the gains brought by the depth and width of the backbone network, and the backbone network is further optimized by the pruning technique. To evaluate the effectiveness of the proposed method, the comparative experiments, such as ablation, anti-noise, and computational complexity, had successfully completed on several rolling bearing datasets. The experiment results show that the proposed BNN-ADTWR method significantly reduces computational complexity and helps with rapid fault diagnosis. The constructed ADTWR backbone network decreases activations in the traditional backbone network and improves absolute accuracy while the computational cost is equivalent.

Deep convolutional neural networks (DCNNs) have achieved state-of-the-art performance in classification, natural language processing (NLP), and regression tasks. However, there is still a great gap between DCNNs and the human brain in terms of computation efficiency. Inspired by neural synaptic plasticity and stochastic computing (SC), we propose neural synaptic plasticity-inspired computing (NSPC) to simulate the human brain's neural network activity for inference tasks with simple logic gates. The multiplication and accumulation (MAC) is transformed by the wire connectivity in NSPC, which only requires bundles of wires and small width adders. To this end, the NSPC imitates the structure of neural synaptic plasticity from a circuit wires connection perspective. Furthermore, from the principle of NSPC, we use a data mapping method to convert the convolution operations to matrix multiplications. Based on the methodology of NSPC, fully-pipelined and low latency architecture is designed. The proposed NSPC accelerator exhibits high hardware efficiency while maintaining a comparable network accuracy level. The NSPC based DCNN accelerator (NSPC-CNN) processes DCNN at 1.5625M images/s with a power dissipation of 15.42 W and an area of 36.4 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . The NSPC based deep neural network (DNN) accelerator (NSPC-DNN) that implements three fully connected layers DNN consumes only 6.6 mm2 area and 2.93 W power, and achieves a throughput of 400M images/s. Compared with conventional fixed-point implementations, the NSPC-CNN achieves 2.77× area efficiency, 2.25× power efficiency; the proposed NSPC-DNN exhibits 2.31× area efficiency and 2.09× power efficiency.

Determining the structure of a protein given its sequence is a challenging problem. Deep learning is a rapidly evolving field which excels at problems where there are complex relationships between input features and desired outputs. Deep Neural Networks have become popular for solving problems in protein science. Various deep neural network architectures have been proposed including deep feed-forward neural networks, recurrent neural networks and more recently neural Turing machines and memory networks. This article provides a short review of deep learning applied to protein prediction problems.