MicroNet: Realizing Micro Neural Network via Binarizing GhostNet

Although deep neural networks are successful for many tasks in the speech domain, the high computational and memory costs of deep neural networks make it difficult to directly deploy highperformance Neural Network systems on low-resource embedded devices. There are several mechanisms to reduce the size of the neural networks i.e. parameter pruning, parameter quantization, etc. This paper focuses on how to apply binary neural networks to the task of speaker verification. The proposed binarization of training parameters can largely maintain the performance while significantly reducing storage space requirements and computational costs. Experiment results show that, after binarizing the Convolutional Neural Network, the ResNet34-based network achieves an EER of around 5% on the Voxceleb1 testing dataset and even outperforms the traditional real number network on the text-dependent dataset: Xiaole while having a 32x memory saving.

Speaker verification based on deep neural network is one of the most prospective biometric authentication methods. But massive computation and parameters make it difficult to deploy speaker verification systems on memory and power constrained embedded devices. Binary neural networks with weights and activations compressed to 1 bit can reduce computational and memory cost significantly. However, speaker verification system based on binary neural networks suffers performance degradation problem. In this paper, to mitigate this problem and facilitate information flow across layers, we optimize binary neural network structure by adding a shortcut to connect layers. We propose an energy and memory efficient speaker system based on the optimized binary networks. Experimental results on text-dependent and text-independent datasets show that performance of proposed speaker verification system outperforms conventional binary neural networks significantly. Compared with full precision networks, the proposed system can largely maintain the performance while reducing memory and computational costs.

Deep neural network (DNN) has powerful feature extraction and classification abilities and is extensively used in mechanical fault diagnosis. However, in the DNN-based fault diagnosis method, too many model parameters, high computational complexity, and occupying a large amount of storage space make it very difficult to optimize and retrain the DNN model. To overcome this deficiency, a binary neural network (BNN) is introduced into fault diagnosis. In the proposed BNN-based fault diagnosis method, the floating-point weights and activations are binarized as ‘+1’ and ‘−1’ to save storage space, and the floating-point multiply-accumulate operations (OPs) are replaced by bitwise OPs to reduce the computational complexity. The BNN-based fault diagnosis method effectively compresses the structure of DNNs and accelerates model inference. However, the binarization OP may bring about the quantization error and gradient error, which will accumulate with the increase of the network depth, and lead to a degradation in model performance. To solve these deficiencies in the proposed BNN-based fault diagnosis method, an improved BNN fault diagnosis method based on adaptive depth-to-width ratio (ADTWR) is proposed. For the convenience in description, the proposed method is called the BNN-ADTWR method. In the proposed BNN-ADTWR method, the network structure is reshaped by the comparison of the gains brought by the depth and width of the backbone network, and the backbone network is further optimized by the pruning technique. To evaluate the effectiveness of the proposed method, the comparative experiments, such as ablation, anti-noise, and computational complexity, had successfully completed on several rolling bearing datasets. The experiment results show that the proposed BNN-ADTWR method significantly reduces computational complexity and helps with rapid fault diagnosis. The constructed ADTWR backbone network decreases activations in the traditional backbone network and improves absolute accuracy while the computational cost is equivalent.

Deep Neural Networks (DNNs) are very powerful and successful but suffer from high computation and memory cost. As a useful attempt, binary neural networks represent weights and activations with binary values, which can significantly reduce resource consumption. However, the simultaneous binarization introduces the coupling effect, aggravating the difficulty of training. In this paper, we develop a novel framework named TP-ADMM that decouples the binarization process into two iteratively optimized stages. Firstly, we propose an improved target propagation method to optimize the network with binary activations in a more stable format. Secondly, we apply the alternating direction method (ADMM) with a varying penalty to get the weights binarized, making weights binarization a discretely constrained optimization problem. Experiments on three public datasets for image classification show that the proposed method outperforms the existing methods.

Binary Neural Networks (BNNs) show promising progress in reducing computational and memory costs, but suffer from substantial accuracy degradation compared to their real-valued counterparts on large-scale datasets, e.g., ImageNet. In this work we study existing BNN architectures and revisit the commonly used technique to include scaling factors. We suggest several architectural design principles for BNNs, based on our studies on architectures. Guided by our principles we develop a novel BNN architecture BinaryDenseNet, which is the first architecture specifically created for BNNs to the best of our knowledge. In our experiments, BinaryDenseNet achieves 18.6% and 7.6% relative improvement over the well-known XNOR-Network and the current state-of-the-art Bi-Real Net in terms of top-1 accuracy on ImageNet, respectively. Further, we show the competitiveness of our BinaryDenseNet regarding memory requirements and computational complexity.

Deep Neural Networks are resource-intensive learning models. To enable their training and deployment on low-end devices, decreasing their computational requirements has become an attractive research area in recent years. One of the most promising approaches in this field is designing networks which use only binary values for the network weights and activations, i.e., Binary Neural Networks (BNN). However, BNNs have robustness and stability issues, which make their training difficult and BNNs perform considerably worse than their real-valued counterparts. To improve the training and deployment performance of BNNs, in this work, we employ a tree-structured network model, which has been widely used in Deep Neural Networks (DNN). We perform experiments to analyze the accuracy, stability and robustness of the proposed tree-structured BNN model. Tree-structured network model integrates multiple classifiers on a backbone BNN resulting in a multi-branch architecture. Our experiments demonstrates that tree-structures model improves the accuracy, stability and robustness of BNNs.

The binarized neural network (BNN) is one of the most promising candidates for low-cost convolutional neural networks (CNNs). This is because of its significant reduction in memory and computational costs, and reasonable classification accuracy. Content-addressable memory (CAM) can perform binarized convolution operations efficiently since the bitwise comparison in CAM matches well with the binarized multiply operation in a BNN. However, a significant design issue in CAM-based BNN accelerators is that the operational reliability is severely degraded by process variations during match-line (ML) sensing operations. In this paper, we proposed a novel ML sensing scheme to reduce the hardware error probability. Most errors occur when the difference between the number of matches in the evaluation ML and the reference ML is small; thus, the proposed hardware identified cases that are vulnerable to process variations using dual references. The proposed dual-reference sensing structure has >49% less ML sensing errors than that of the conventional design, leading to a >1.0% accuracy improvement for Fashion MNIST image classification. In addition, owing to the parallel convolution operation of the CAM-based BNN accelerator, the proposed hardware achieved >34% processing-time improvement compared with that of the digital logic implementation.

In pulsed Doppler (PD) radar, active jamming recognition faces two key challenges: complex jamming in scenarios with moving targets and the inefficiency of full‐precision (FP) convolutional neural networks (CNNs) for deployment. This letter proposes a binary pyramid self‐supervised reconstruction network (BPSR‐Net), a lightweight recognition algorithm based on binary neural networks (BNNs). BPSR‐Net employs a piecewise quadratic gradient approximation (PQGA) to improve training stability, a multi‐scale binary pyramid (BiPy) structure for efficient feature extraction, and a binary self‐supervised reconstruction (BiSR) module to enhance global feature learning. Experiments on 12 single and compound jamming types show that BPSR‐Net achieves 98.93% top‐1 accuracy, 99.90% top‐5 accuracy, with only 15.64 Mbit of memory usage and 43.76 × 10 6 FLOPs. The proposed model balances accuracy, memory, and computational cost, providing a practical solution for low‐power radar anti‐jamming systems.

Binary neural networks (BNNs) can substantially accelerate a neural network's inference time by substituting its costly floating-point arithmetic with bit-wise operations. Nevertheless, state-of-the-art approaches reduce the efficiency of the data flow in the BNN layers by introducing intermediate conversions from 1 to 16/32 bits. We propose a novel training scheme, denoted as BNN-Clip, that can increase the parallelism and data flow of the BNN pipeline; specifically, we introduce a clipping block that reduces the data width from 32 bits to 8. Furthermore, we decrease the internal accumulator size of a binary layer, usually kept using 32 bits to prevent data overflow, with no accuracy loss. Moreover, we propose an optimization of the batch normalization layer that reduces latency and simplifies deployment. Finally, we present an optimized implementation of the binary direct convolution for ARM NEON instruction sets. Our experiments show a consistent inference latency speed-up (up to 1.3 and 2.4× compared to two state-of-the-art BNN frameworks) while reaching an accuracy comparable with state-of-the-art approaches on datasets like CIFAR-10, SVHN, and ImageNet.

Field-programmable gate array (FPGA)–specific deep neural network (DNN) architectures using native lookup tables (LUTs) as independently trainable inference operators have been shown to achieve favorable area-accuracy and energy-accuracy trade-offs. The first work in this area, LUTNet, exhibited state-of-the-art performance for standard DNN benchmarks. In this article, we propose the learned optimization of such LUT-based topologies, resulting in higher-efficiency designs than via the direct use of off-the-shelf, hand-designed networks. Existing implementations of this class of architecture require the manual specification of the number of inputs per LUT, K . Choosing appropriate K a priori is challenging. Doing so at even high granularity, for example, per layer, is a time-consuming and error-prone process that leaves FPGAs’ spatial flexibility underexploited. Furthermore, prior works see LUT inputs connected randomly, which does not guarantee a good choice of network topology. To address these issues, we propose logic shrinkage , a fine-grained netlist pruning methodology enabling K to be automatically learned for every LUT in a neural network targeted for FPGA inference. By removing LUT inputs determined to be of low importance, our method increases the efficiency of the resultant accelerators. Our GPU-friendly solution to LUT input removal is capable of processing large topologies during their training with negligible slowdown. With logic shrinkage, we improve the area and energy efficiency of the best-performing LUTNet implementation of the CNV network classifying CIFAR-10 by 1.54× and 1.31×, respectively, while matching its accuracy. This implementation also reaches 2.71× the area efficiency of an equally accurate, heavily pruned binary neural network (BNN). On ImageNet, with the Bi-Real Net architecture, employment of logic shrinkage results in a post-synthesis area reduction of 2.67× vs. LUTNet, allowing for implementation that was previously impossible on today’s largest FPGAs. We validate the benefits of logic shrinkage in the context of real application deployment by implementing a face mask detection DNN using a BNN, LUTNet, and logic-shrunk layers. Our results show that logic shrinkage results in area gains versus LUTNet (up to 1.20×) and equally pruned BNNs (up to 1.08×), along with accuracy improvements.

Recent advances in deep learning have shown that Binary Neural Networks (BNNs) are capable of providing a satisfying accuracy on various image datasets with significant reduction in computation and memory cost. With both weights and activations binarized to +1 or -1 in BNNs, the high-precision multiply-and-accumulate (MAC) operations can be replaced by XNOR and bit-counting operations. In this work, we propose a RRAM synaptic architecture (XNOR-RRAM) with a bit-cell design of complementary word lines that implements equivalent XNOR and bit-counting operation in a parallel fashion. For large-scale matrices in fully connected layers or when the convolution kernels are unrolled in multiple channels, the array partition is necessary. Multi-level sense amplifiers (MLSAs) are employed as the intermediate interface for accumulating partial weighted sum. However, a low bit-level MLSA and intrinsic offset of MLSA may degrade the classification accuracy. We investigate the impact of sensing offsets on classification accuracy and analyze various design options with different sub-array sizes and sensing bit-levels. Experimental results with RRAM models and 65nm CMOS PDK show that the system with 128×128 sub-array size and 3-bit MLSA can achieve accuracies of 98.43% for MLP on MNIST and 86.08% for CNN on CIFAR-10, showing 0.34% and 2.39% degradation respectively compared to the accuracies of ideal BNN algorithms. The projected energy-efficiency of XNOR-RRAM is 141.18 TOPS/W, showing ~33X improvement compared to the conventional RRAM synaptic architecture with sequential row-by-row read-out.

S[formula omitted]NN: Time step reduction of spiking surrogate gradients for training energy efficient single-step spiking neural networks

An efficient GPU-accelerated inference engine for binary neural network on mobile phones

An efficient unconstrained facial expression recognition algorithm based on Stack Binarized Auto-encoders and Binarized Neural Networks

Deep learning has revolutionized computer vision and other fields since its big bang in 2012. However, it is challenging to deploy Deep Neural Networks (DNNs) into real-world applications due to their high computational complexity. Binary Neural Networks (BNNs) dramatically reduce computational complexity by replacing most arithmetic operations with bitwise operations. Existing implementations of BNNs have been focusing on GPU or FPGA, and using the conventional image-to-column method that doesn't perform well for binary convolution due to low arithmetic intensity and unfriendly pattern for bitwise operations. We propose BitFlow, a gemm-operator-network three-level optimization framework for fully exploiting the computing power of BNNs on CPU. BitFlow features a new class of algorithm named PressedConv for efficient binary convolution using locality-aware layout and vector parallelism. We evaluate BitFlow with the VGG network. On a single core of Intel Xeon Phi, BitFlow obtains 1.8x speedup over unoptimized BNN implementations, and 11.5x speedup over counterpart full-precision DNNs. Over 64 cores, BitFlow enables BNNs to run 1.1x faster than counterpart full-precision DNNs on GPU (GTX 1080).