BinaryDenseNet: Developing an Architecture for Binary Neural Networks

Speaker verification based on deep neural network is one of the most prospective biometric authentication methods. But massive computation and parameters make it difficult to deploy speaker verification systems on memory and power constrained embedded devices. Binary neural networks with weights and activations compressed to 1 bit can reduce computational and memory cost significantly. However, speaker verification system based on binary neural networks suffers performance degradation problem. In this paper, to mitigate this problem and facilitate information flow across layers, we optimize binary neural network structure by adding a shortcut to connect layers. We propose an energy and memory efficient speaker system based on the optimized binary networks. Experimental results on text-dependent and text-independent datasets show that performance of proposed speaker verification system outperforms conventional binary neural networks significantly. Compared with full precision networks, the proposed system can largely maintain the performance while reducing memory and computational costs.

An efficient unconstrained facial expression recognition algorithm based on Stack Binarized Auto-encoders and Binarized Neural Networks

To achieve an advanced Internet of Things (IoT), it is necessary to combine artificial intelligence (AI) with IoT. Compact circuits that can operate AI functions will be useful for this purpose. Therefore, we propose stochastic weights binary neural networks (SWBNN). SWBNNs are more accurate than binary neural networks (BNN) with small circuits. BNNs can be realized with small circuits since binary calculation needs simpler circuits than real number calculation. However, BNNs have lower accuracy than networks with real numbers. Thus, the proposed SWBNNs are BNNs that behave stochastically, which makes them more accurate than BNNs. Moreover, SWBNNs can still be achieved with small circuits since they execute binary calculation. As a result, the accuracy for the test data of SWBNNs is closer to the accuracy for learning data than the accuracy for the test data of BNNs is. Especially when using the CIFAR10 database, the difference in the identification accuracy rate between learning data and test data decreased from 6% for BNNs to 2% for SWBNNs. From results of a field-programmable gate array (FPGA) implementation, circuits of SWBNNs are sufficiently small although they are 10% bigger than those of BNNs. Therefore, SWBNNs are more accurate than BNNs, and the circuit costs ofintroducing stochastic weights are low.

Deep neural network (DNN) has powerful feature extraction and classification abilities and is extensively used in mechanical fault diagnosis. However, in the DNN-based fault diagnosis method, too many model parameters, high computational complexity, and occupying a large amount of storage space make it very difficult to optimize and retrain the DNN model. To overcome this deficiency, a binary neural network (BNN) is introduced into fault diagnosis. In the proposed BNN-based fault diagnosis method, the floating-point weights and activations are binarized as ‘+1’ and ‘−1’ to save storage space, and the floating-point multiply-accumulate operations (OPs) are replaced by bitwise OPs to reduce the computational complexity. The BNN-based fault diagnosis method effectively compresses the structure of DNNs and accelerates model inference. However, the binarization OP may bring about the quantization error and gradient error, which will accumulate with the increase of the network depth, and lead to a degradation in model performance. To solve these deficiencies in the proposed BNN-based fault diagnosis method, an improved BNN fault diagnosis method based on adaptive depth-to-width ratio (ADTWR) is proposed. For the convenience in description, the proposed method is called the BNN-ADTWR method. In the proposed BNN-ADTWR method, the network structure is reshaped by the comparison of the gains brought by the depth and width of the backbone network, and the backbone network is further optimized by the pruning technique. To evaluate the effectiveness of the proposed method, the comparative experiments, such as ablation, anti-noise, and computational complexity, had successfully completed on several rolling bearing datasets. The experiment results show that the proposed BNN-ADTWR method significantly reduces computational complexity and helps with rapid fault diagnosis. The constructed ADTWR backbone network decreases activations in the traditional backbone network and improves absolute accuracy while the computational cost is equivalent.

Binary Neural Networks (BNNs) are regarded as very effective approaches to reduce the high computational and memory cost of deep convolutional neural networks; however, most of the previous works only focus on the binarization of traditional large-scale networks. In this paper, to further decrease the computational and memory complexity, we propose to binarize the lightweight network: GhostNet, which can generate feature maps with fewer operations. We first modify the ghost block for reducing the information loss caused by the binarization. Moreover, to enhance the performance of binarized GhostNet, some training schemes are introduced including gradient approximation and new activation layers. Various experiments on CIFAR10 dataset show that, compared with other methods, the MicroNet can achieve better performance with fewer computational and memory cost.

Recent advances in deep learning have shown that Binary Neural Networks (BNNs) are capable of providing a satisfying accuracy on various image datasets with significant reduction in computation and memory cost. With both weights and activations binarized to +1 or -1 in BNNs, the high-precision multiply-and-accumulate (MAC) operations can be replaced by XNOR and bit-counting operations. In this work, we propose a RRAM synaptic architecture (XNOR-RRAM) with a bit-cell design of complementary word lines that implements equivalent XNOR and bit-counting operation in a parallel fashion. For large-scale matrices in fully connected layers or when the convolution kernels are unrolled in multiple channels, the array partition is necessary. Multi-level sense amplifiers (MLSAs) are employed as the intermediate interface for accumulating partial weighted sum. However, a low bit-level MLSA and intrinsic offset of MLSA may degrade the classification accuracy. We investigate the impact of sensing offsets on classification accuracy and analyze various design options with different sub-array sizes and sensing bit-levels. Experimental results with RRAM models and 65nm CMOS PDK show that the system with 128×128 sub-array size and 3-bit MLSA can achieve accuracies of 98.43% for MLP on MNIST and 86.08% for CNN on CIFAR-10, showing 0.34% and 2.39% degradation respectively compared to the accuracies of ideal BNN algorithms. The projected energy-efficiency of XNOR-RRAM is 141.18 TOPS/W, showing ~33X improvement compared to the conventional RRAM synaptic architecture with sequential row-by-row read-out.

The binarized neural network (BNN) is one of the most promising candidates for low-cost convolutional neural networks (CNNs). This is because of its significant reduction in memory and computational costs, and reasonable classification accuracy. Content-addressable memory (CAM) can perform binarized convolution operations efficiently since the bitwise comparison in CAM matches well with the binarized multiply operation in a BNN. However, a significant design issue in CAM-based BNN accelerators is that the operational reliability is severely degraded by process variations during match-line (ML) sensing operations. In this paper, we proposed a novel ML sensing scheme to reduce the hardware error probability. Most errors occur when the difference between the number of matches in the evaluation ML and the reference ML is small; thus, the proposed hardware identified cases that are vulnerable to process variations using dual references. The proposed dual-reference sensing structure has >49% less ML sensing errors than that of the conventional design, leading to a >1.0% accuracy improvement for Fashion MNIST image classification. In addition, owing to the parallel convolution operation of the CAM-based BNN accelerator, the proposed hardware achieved >34% processing-time improvement compared with that of the digital logic implementation.

For accelerating Binarized Neural Networks (BNNs), analog computing-based crossbar accelerators, utilizing XNOR gates and additional interface circuits, have been proposed. Such accelerators demand a large amount of analog-to-digital converters (ADCs) and registers, resulting in expensive designs. To increase the inference efficiency, the state of the art divides the interface circuit into an Analog Path (AP), utilizing (cheap) analog comparators, and a Digital Path (DP), utilizing (expensive) ADCs and registers. During BNN execution, a certain path is selectively triggered. Ideally, as inference via AP is more efficient, it should be triggered as often as possible. However, we reveal that, unless the number of weights is very small, the AP is rarely triggered. To overcome this, we propose a novel BNN inference scheme, called Local Thresholding Approximation (LTA). It approximates the global thresholdings in BNNs by local thresholdings. This enables the use of the AP through most of the execution, which significantly increases the interface circuit efficiency. In our evaluations with two BNN architectures, using LTA reduces the area by 42x and 54x, the energy by 2.7x and 4.2x, and the latency by 3.8x and 1.15x, compared to the state-of-the-art crossbar-based BNN accelerators.

Deep learning has revolutionized computer vision and other fields since its big bang in 2012. However, it is challenging to deploy Deep Neural Networks (DNNs) into real-world applications due to their high computational complexity. Binary Neural Networks (BNNs) dramatically reduce computational complexity by replacing most arithmetic operations with bitwise operations. Existing implementations of BNNs have been focusing on GPU or FPGA, and using the conventional image-to-column method that doesn't perform well for binary convolution due to low arithmetic intensity and unfriendly pattern for bitwise operations. We propose BitFlow, a gemm-operator-network three-level optimization framework for fully exploiting the computing power of BNNs on CPU. BitFlow features a new class of algorithm named PressedConv for efficient binary convolution using locality-aware layout and vector parallelism. We evaluate BitFlow with the VGG network. On a single core of Intel Xeon Phi, BitFlow obtains 1.8x speedup over unoptimized BNN implementations, and 11.5x speedup over counterpart full-precision DNNs. Over 64 cores, BitFlow enables BNNs to run 1.1x faster than counterpart full-precision DNNs on GPU (GTX 1080).

Binary neural networks have recently begun to be used as a highly energy- and computation-efficient image processing technique for computer vision tasks. This paper proposes a novel extension of existing binary neural network technology based on the use of a Hadamard transform in the input layer of a binary neural network. Previous state-of-the-art binary neural networks require floating-point arithmetic at several parts of the neural network model computation in order to maintain a sufficient level of accuracy. The Hadamard transform is similar to a Discrete Cosine Transform (used in the popular JPEG image compression method) except that it does not include expensive multiplication operations. In this paper, it is shown that the Hadamard transform can be used to replace the most expensive floating-point arithmetic portion of a binary neural network. In order to test the efficacy of this proposed method, three types of experiments were conducted: application of the proposed method to several state-of-the-art neural network models, verification of its effectiveness in a large image dataset (ImageNet), and experiments to verify the effectiveness of the Hadamard transform by comparing the performance of binary neural networks with and without the Hadamard transform. The results show that the Hadamard transform can be used to implement a highly energy-efficient binary neural network with only a miniscule loss of accuracy.

Deep Neural Networks are resource-intensive learning models. To enable their training and deployment on low-end devices, decreasing their computational requirements has become an attractive research area in recent years. One of the most promising approaches in this field is designing networks which use only binary values for the network weights and activations, i.e., Binary Neural Networks (BNN). However, BNNs have robustness and stability issues, which make their training difficult and BNNs perform considerably worse than their real-valued counterparts. To improve the training and deployment performance of BNNs, in this work, we employ a tree-structured network model, which has been widely used in Deep Neural Networks (DNN). We perform experiments to analyze the accuracy, stability and robustness of the proposed tree-structured BNN model. Tree-structured network model integrates multiple classifiers on a backbone BNN resulting in a multi-branch architecture. Our experiments demonstrates that tree-structures model improves the accuracy, stability and robustness of BNNs.

Resistive random access memory (ReRAM) has been proven capable to efficiently perform in-situ matrix-vector computations in convolutional neural network (CNN) processing. The computations are often conducted on multi-level cell (MLC) that have limited precision and hence, show significant vulnerability to noises. The binarized neural network (BNN) is a hardware-friendly model that can dramatically reduce the computation and storage overheads. However, XNOR, which is the key operation in BNNs, cannot be directly computed in-situ in ReRAM because of its nonlinear behavior. To enable real in-situ processing of BNNs in ReRAM, we modified the BNN algorithm to enable direct computation of XNOR, POPCOUNT and POOL based on ReRAM cells. We also proposed the complementary resistive cell (CRC) design to efficiently conduct XNOR operations and optimized the pipeline design with decoupled buffer and computation stages. Our results show our scheme, namely, ReBNN, improves the system performance by $$25.36\times$$ and the energy efficiency by $$4.26\times$$ compared to conventional ReRAM based accelerator, and ensures a throughput higher than state-of-the-art BNN accelerators. The correctness of the modified algorithm is also validated.

Deep Neural Networks (DNNs) are very powerful and successful but suffer from high computation and memory cost. As a useful attempt, binary neural networks represent weights and activations with binary values, which can significantly reduce resource consumption. However, the simultaneous binarization introduces the coupling effect, aggravating the difficulty of training. In this paper, we develop a novel framework named TP-ADMM that decouples the binarization process into two iteratively optimized stages. Firstly, we propose an improved target propagation method to optimize the network with binary activations in a more stable format. Secondly, we apply the alternating direction method (ADMM) with a varying penalty to get the weights binarized, making weights binarization a discretely constrained optimization problem. Experiments on three public datasets for image classification show that the proposed method outperforms the existing methods.

Binary Neural Networks (BNNs) have received significant attention due to the memory and computation efficiency recently. However, the considerable accuracy gap between BNNs and their full-precision counterparts hinders BNNs to be deployed to resource-constrained platforms. One of the main reasons for the performance gap can be attributed to the frequent weight flip, which is caused by the misleading weight update in BNNs. To address this issue, we propose a state-aware binary neural network (SA-BNN) equipped with the well designed state-aware gradient. Our SA-BNN is inspired by the observation that the frequent weight flip is more likely to occur, when the gradient magnitude for all quantization states {-1,1} is identical. Accordingly, we propose to employ independent gradient coefficients for different states when updating the weights. Furthermore, we also analyze the effectiveness of the state-aware gradient on suppressing the frequent weight flip problem. Experiments on ImageNet show that the proposed SA-BNN outperforms the current state-of-the-arts (e.g., Bi-Real Net) by more than 3% when using a ResNet architecture. Specifically, we achieve 61.7%, 65.5% and 68.7% Top-1 accuracy with ResNet-18, ResNet-34 and ResNet-50 on ImageNet, respectively.

Binary neural networks (BNNs) have received increasing attention due to their superior reductions of computation and memory. Most existing works focus on either lessening the quantization error by minimizing the gap between the full-precision weights and their binarization or designing a gradient approximation to mitigate the gradient mismatch, while leaving the "dead weights" untouched. This leads to slow convergence when training BNNs. In this paper, for the first time, we explore the influence of "dead weights" which refer to a group of weights that are barely updated during the training of BNNs, and then introduce rectified clamp unit (ReCU) to revive the "dead weights" for updating. We prove that reviving the "dead weights" by ReCU can result in a smaller quantization error. Besides, we also take into account the information entropy of the weights, and then mathematically analyze why the weight standardization can benefit BNNs. We demonstrate the inherent contradiction between minimizing the quantization error and maximizing the information entropy, and then propose an adaptive exponential scheduler to identify the range of the "dead weights". By considering the "dead weights", our method offers not only faster BNN training, but also state-of-the-art performance on CIFAR-10 and ImageNet, compared with recent methods. Code can be available at https://github.com/z-hXu/ReCU.