Fault-Tolerance of Binarized and Stochastic Computing-based Neural Networks

To achieve an advanced Internet of Things (IoT), it is necessary to combine artificial intelligence (AI) with IoT. Compact circuits that can operate AI functions will be useful for this purpose. Therefore, we propose stochastic weights binary neural networks (SWBNN). SWBNNs are more accurate than binary neural networks (BNN) with small circuits. BNNs can be realized with small circuits since binary calculation needs simpler circuits than real number calculation. However, BNNs have lower accuracy than networks with real numbers. Thus, the proposed SWBNNs are BNNs that behave stochastically, which makes them more accurate than BNNs. Moreover, SWBNNs can still be achieved with small circuits since they execute binary calculation. As a result, the accuracy for the test data of SWBNNs is closer to the accuracy for learning data than the accuracy for the test data of BNNs is. Especially when using the CIFAR10 database, the difference in the identification accuracy rate between learning data and test data decreased from 6% for BNNs to 2% for SWBNNs. From results of a field-programmable gate array (FPGA) implementation, circuits of SWBNNs are sufficiently small although they are 10% bigger than those of BNNs. Therefore, SWBNNs are more accurate than BNNs, and the circuit costs ofintroducing stochastic weights are low.

Stochastic neuromorphic computation (SNC) has the potential to enable a low power, error tolerant and scalable computing platform in comparison to its deterministic counterparts. However, the hardware implementation of complementary metal oxide semiconductor (CMOS)-based stochastic circuits involves conversion blocks that cost more than the actual processing circuits. The realization of the activation function for SNCs also requires a complicated circuit that results in a significant amount of power dissipation and area overhead. The inherent probabilistic switching behavior of nanomagnets provides an advantage to overcome these complexity issues for the realization of low power and area efficient SNC systems. This paper presents magnetic tunnel junction (MTJ)-based stochastic computing methodology for the implementation of a neural network. The stochastic switching behavior of the MTJ has been exploited to design a binary to stochastic converter to mitigate the complexity of the CMOS-based design. The paper also presents the technique for realizing stochastic sigmoid activation function using an MTJ. Such circuits are simpler than existing ones and use considerably less power. An image classification system employing the proposed circuits has been implemented to verify the effectiveness of the technique. The MTJ-based SNC system shows area and energy reduction by a factor of 13.5 and 2.5, respectively, while the prediction accuracy is 86.66%. Furthermore, this paper investigates how crucial parameters, such as stochastic bitstream length, number of hidden layers and number of nodes in a hidden layer, need to be set precisely to realize an efficient MTJ-based stochastic neural network (SNN). The proposed methodology can prove a promising alternative for highly efficient digital stochastic computing applications.

In spiking neural networks, the information is conveyed by the spike times, that depend on the intrinsic dynamics of each neuron, the input they receive and on the connections between neurons. In this article we study the Markovian nature of the sequence of spike times in stochastic neural networks, and in particular the ability to deduce from a spike train the next spike time, and therefore produce a description of the network activity only based on the spike times regardless of the membrane potential process. To study this question in a rigorous manner, we introduce and study an event-based description of networks of noisy integrate-and-fire neurons, i.e. that is based on the computation of the spike times. We show that the firing times of the neurons in the networks constitute a Markov chain, whose transition probability is related to the probability distribution of the interspike interval of the neurons in the network. In the cases where the Markovian model can be developed, the transition probability is explicitly derived in such classical cases of neural networks as the linear integrate-and-fire neuron models with excitatory and inhibitory interactions, for different types of synapses, possibly featuring noisy synaptic integration, transmission delays and absolute and relative refractory period. This covers most of the cases that have been investigated in the event-based description of spiking deterministic neural networks.

Stochastic Number Generators (SNG) plays a significant role in designing a stochastic computing system. SNGs make the stochastic system comfortable for computing in the stochastic domain. The challenges in developing the stochastic computing system are correlation and hardware area occupancy. By considering these phenomena, we have considered Linear Feedback Shift Register (LFSR) based SNG and S-box based SNG in this work. Our contributions to this paper are stochastic computation for activation functions using the SNGs mentioned above and stochastic computation for arithmetic components in the stochastic domain. By considering the two SNG methods, the difference in the computation for accuracy has been analyzed for stochastic activation function and stochastic arithmetic computation. The better SNG method will be used as SNG for stochastic convolutional neural network design using this analysis.

Approximate computing (AC) is an effective energy-efficient method for error-resilient applications. The essence behind AC is to reduce energy consumption by slightly sacrificing computation accuracy purposefully while providing quality-acceptable results. On the other hand, soft error is a common problem during program execution and may cause unacceptable outputs or catastrophic failure to the system. As AC introduces errors and soft errors are mitigated by fault-tolerant mechanisms, they have conflict goals and contradictory approaches. To the best of our knowledge, there is no previous efforts to consider the two at the same time. In this paper, we study the problem of AC with soft errors in order to guarantee the safe execution of the program while reducing energy (by AC). More specifically, we propose FTApprox, a fault-tolerant approximate arithmetic computing data format, to enable the detection and correction of SEs. As an approximate data format, FTApprox can use 16 bits to approximate any 32-bit integers and fixed-point numbers, and will select only the most significant part of operands for AC at runtime. Energy saving is obtained by converting 32-bit arithmetic operations to 8-bit operations. Meanwhile, for soft errors such as random bit flips, FTApprox not only can detect all single bit flips and most 2-bit flips, it can also correct most of these errors. The experimental results show that FTApprox has significant resistance against soft errors while providing 66.4%-79.6% energy saving.

The paper tackles the stabilization of stochastic hybrid neural networks. Different from the existing work which the Brownian motion was used to stabilize neural networks for the stochastic stability, the stabilization is generalized by Lévy noise in this paper. To address the unobservable items in the hybrid neural networks, we introduce the Markov chain into the hybrid neural networks and divide the state space of Markov chain into two subsets for observable and unobservable items. Then, the sufficient conditions of stabilization and destabilization for the stochastic neural networks are obtained.

Precise segmentation of clinical target volume (CTV) is the key to stomach cancer radiotherapy. We proposed a novel stochastic width-deep neural network (SW-DNN) for better automatically contouring stomach CTV. Stochastic width-deep neural network was an end-to-end approach, of which the core component was a novel SW mechanism that employed shortcut connections between the encoder and decoder in a random manner, and thus the width of the SW-DNN was stochastically adjustable to obtain improved segmentation results. In total, 150 stomach cancer patient computed tomography (CT) cases with the corresponding CTV labels were collected and used to train and evaluate the SW-DNN. Three common quantitative measures: true positive volume fraction (TPVF), positive predictive value (PPV), and Dice similarity coefficient (DSC) were used to evaluate the segmentation accuracy. Clinical target volumes calculated by SW-DNN had significant quantitative advantages over three state-of-the-art methods. The average DSC value of SW-DNN was 2.1%, 2.8%, and 3.6% higher than that of three state-of-the-art methods. The average DSC, TPVF, and PPV values of SW-DNN were 2.1%, 4.0%, and 0.3% higher than that of the corresponding constant width DNN. Stochastic width-deep neural network provided better performance for contouring stomach cancer CTV accurately and efficiently. It is a promising solution in clinical radiotherapy planning for stomach cancer.

Hardware-aware neural architecture search for stochastic computing-based neural networks on tiny devices

In dermatology, deep learning may be applied for skin lesion classification. However, for a given input image, a neural network only outputs a label, obtained using the class probabilities, which do not model uncertainty. Our group developed a novel method to quantify uncertainty in stochastic neural networks. In this study, we aimed to train such network for skin lesion classification and evaluate its diagnostic performance and uncertainty, and compare the results to the assessments by a group of dermatologists. By passing duplicates of an image through such a stochastic neural network, we obtained distributions per class, rather than a single probability value. We interpreted the overlap between these distributions as the output uncertainty, where a high overlap indicated a high uncertainty, and vice versa. We had 29 dermatologists diagnose a series of skin lesions and rate their confidence. We compared these results to those of the network. The network achieved a sensitivity and specificity of 50% and 88%, comparable to the average dermatologist (respectively 68% and 73%). Higher confidence/less uncertainty was associated with better diagnostic performance both in the neural network and in dermatologists. We found no correlation between the uncertainty of the neural network and the confidence of dermatologists (R = -0.06, p = 0.77). Dermatologists should not blindly trust the output of a neural network, especially when its uncertainty is high. The addition of an uncertainty score may stimulate the human-computer interaction.

<p>This paper deals with the problem of passivity<br> analysis for stochastic neural networks with leakage, discrete and<br> distributed delays. By using delay partitioning technique, free<br> weighting matrix method and stochastic analysis technique, several<br> sufficient conditions for the passivity of the addressed neural<br> networks are established in terms of linear matrix inequalities<br> (LMIs), in which both the time-delay and its time derivative can be<br> fully considered. A numerical example is given to show the<br> usefulness and effectiveness of the obtained results.</p>

In this paper, the existence and stability of Weyl almost automorphic solutions in distribution sense for a class of Clifford-valued stochastic neural networks with time-varying delays are studied by using the direct method. Firstly, the existence and uniqueness of Weyl almost automorphic solutions in distribution sense for this class of neural networks are studied by using the Banach fixed point theorem and the relationship between several different senses of random almost automorphy. Then, the global exponential stability in p th mean of the unique Weyl almost automorphic solution in distribution sense is proved by inequality technique and counter proof method. Even when this class of neural networks we consider is real-valued, our results are new. Meanwhile, the method proposed in this paper can be used to study the existence of Weyl almost automorphic solutions of other types of neural networks including stochastic and deterministic neural networks. Finally, an example is given to illustrate the feasibility of our results.

Dynamics of infectious disease mathematical model through unsupervised stochastic neural network paradigm

S[formula omitted]NN: Time step reduction of spiking surrogate gradients for training energy efficient single-step spiking neural networks

Resistive random access memory (ReRAM) has been proven capable to efficiently perform in-situ matrix-vector computations in convolutional neural network (CNN) processing. The computations are often conducted on multi-level cell (MLC) that have limited precision and hence, show significant vulnerability to noises. The binarized neural network (BNN) is a hardware-friendly model that can dramatically reduce the computation and storage overheads. However, XNOR, which is the key operation in BNNs, cannot be directly computed in-situ in ReRAM because of its nonlinear behavior. To enable real in-situ processing of BNNs in ReRAM, we modified the BNN algorithm to enable direct computation of XNOR, POPCOUNT and POOL based on ReRAM cells. We also proposed the complementary resistive cell (CRC) design to efficiently conduct XNOR operations and optimized the pipeline design with decoupled buffer and computation stages. Our results show our scheme, namely, ReBNN, improves the system performance by $$25.36\times$$ and the energy efficiency by $$4.26\times$$ compared to conventional ReRAM based accelerator, and ensures a throughput higher than state-of-the-art BNN accelerators. The correctness of the modified algorithm is also validated.

In this paper, we propose a neural network algorithm--multi-start stochastic competitive Hopfield neural network (MS-SCHNN) for the p -median problem. The proposed algorithm combines two mechanisms to improve neural network's performance. First, it introduces stochastic dynamics into the competitive Hopfield neural network (CHNN) to help the network escape from local minima. Second, it adopts multi-start strategy to further improve the performance of SCHNN. Experimental results on a series of benchmark problems show that MS-SCHNN outperforms previous neural network algorithms for the p -median problem.