Can Stochastic Computing Truly Tolerate Bit Flips?

As the energy and hardware investments necessary for conventional high-precision digital computing continue to explode in the era of artificial intelligence (AI), a change in paradigm that can trade precision for energy and resource efficiency is being sought for many computing applications. Stochastic computing (SC) is an attractive alternative since, unlike digital computers, which require many logic gates and a high transistor volume to perform basic arithmetic operations such as addition, subtraction, multiplication, sorting, etc., SC can implement the same using simple logic gates. While it is possible to accelerate SC using traditional silicon complementary metal-oxide-semiconductor (CMOS) technology, the need for extensive hardware investment to generate stochastic bits (s-bits), the fundamental computing primitive for SC, makes it less attractive. Memristor and spin-based devices offer natural randomness but depend on hybrid designs involving CMOS peripherals for accelerating SC, which increases area and energy burden. Here, the limitations of existing and emerging technologies are overcome, and a standalone SC architecture embedded in memory and based on 2Dmemtransistors is experimentally demonstrated. The monolithic and non-von-Neumann SC architecture occupies a small hardware footprint and consumes a miniscule amount of energy (<1nJ) for both s-bit generation and arithmetic operations, highlighting the benefits of SC.

Deep neural networks (DNNs) have been revolutionizing machine learning research and various applications by achieving better accuracy than human perception. However, a large amount of computational resources are needed due to their complex topologies, restricting the usage of DNNs in embedded systems and portable devices with limited areas and power budgets. This chapter considers stochastic computing (SC) as an alternative to binarized computing. SC uses a random bit sequence where the signal value is computed by counting the number of ones in the bit sequence. This allows the implementation of major arithmetic units like multiplication and addition by a simple logic. Consequently, SC can implement DNNs with essentially low hardware overhead and also offers good scalability. First, the chapter presents a detailed study of processing elements used in neural networks. Then, combining several building blocks, such as stochastic approximation of activation functions in SC, saturation arithmetic, and scaling, a new scheme is proposed for DNN inference and training in SC. At last, the chapter shows that SC-compatible DNNs can be effectively deployed on low-cost hardware. Deep neural network, optimization, accelerator, custom computing, stochastic computing

Stochastic computing (SC) is a low-cost design technique that has great promise in applications such as image processing. SC enables arithmetic operations to be performed on stochastic bit-streams using ultra-small and low-power circuitry. However, accurate computations tend to require long run-times due to the random fluctuations inherent in stochastic numbers (SNs). We present novel techniques for SN generation that lead to better accuracy/run-time trade-offs. First, we analyze a property called progressive precision (PP) which allows computational accuracy to grow systematically with run-time. Second, borrowing from Monte Carlo methods, we show that SC performance can be greatly improved by replacing the usual pseudo-random number sources by low-discrepancy (LD) sequences that are predictably progressive. Finally, we evaluate the use of LD stochastic numbers in SC, and show they can produce significantly faster and more accurate results than existing stochastic designs.

Stochastic computing enables computationally complex arithmetic using binary numbers converted to stochastic bitstreams. A large number of applications have used stochastic computing due to its fault-tolerant nature. However, stochastic circuit synthesis presents a larger solution space when compared to classical logic synthesis. Previous methods synthesize stochastic circuits using a heuristic method. In this paper, a novel exact synthesis method using Boolean satisfiability (SAT) is proposed to obtain an optimal stochastic circuit represented by majority-inverter graphs (MIGs). The experimental results suggest that the proposed approach can achieve 21% area reduction, 4% delay improvement, with a 3% mean absolute error trade-off.

This chapter begins by reviewing the sources of inaccuracy in stochastic computing, focusing on correlation, that is, dependencies among stochastic bit-streams. The measurement of correlation is considered, and the SCC metric is defined. The properties of correlation are then explored including some that have only been discovered recently. Correlation can be seen in two ways: either as corrupting a function f, or as changing f to a different, but potentially useful one. Therefore, to ensure that a stochastic circuit works as expected it is important to manage correlation appropriately. This can be done with correlation-controlling units, which must be used carefully to avoid unexpected functional changes and excessive hardware area or latency overhead. There are also cases where correlation has no effect at all (correlation insensitivity). Identifying such immunity to correlation can aid the design of stochastic circuits. Finally, design of stochastic number generators to provide specified levels of correlation is discussed.

Stochastic neuromorphic computation (SNC) has the potential to enable a low power, error tolerant and scalable computing platform in comparison to its deterministic counterparts. However, the hardware implementation of complementary metal oxide semiconductor (CMOS)-based stochastic circuits involves conversion blocks that cost more than the actual processing circuits. The realization of the activation function for SNCs also requires a complicated circuit that results in a significant amount of power dissipation and area overhead. The inherent probabilistic switching behavior of nanomagnets provides an advantage to overcome these complexity issues for the realization of low power and area efficient SNC systems. This paper presents magnetic tunnel junction (MTJ)-based stochastic computing methodology for the implementation of a neural network. The stochastic switching behavior of the MTJ has been exploited to design a binary to stochastic converter to mitigate the complexity of the CMOS-based design. The paper also presents the technique for realizing stochastic sigmoid activation function using an MTJ. Such circuits are simpler than existing ones and use considerably less power. An image classification system employing the proposed circuits has been implemented to verify the effectiveness of the technique. The MTJ-based SNC system shows area and energy reduction by a factor of 13.5 and 2.5, respectively, while the prediction accuracy is 86.66%. Furthermore, this paper investigates how crucial parameters, such as stochastic bitstream length, number of hidden layers and number of nodes in a hidden layer, need to be set precisely to realize an efficient MTJ-based stochastic neural network (SNN). The proposed methodology can prove a promising alternative for highly efficient digital stochastic computing applications.

Stochastic Computing (SC) presents a low-cost and low-power alternative to conventional binary computing. In SC, continuous values are represented by stochastically generated bit streams. By performing simple hardware-friendly bit-wise operations on these streams, complex calculations can be realized very efficiently. However, the inherent randomness and approximation used in SC can result in undesirable computational errors. As Convolutional Neural Networks (CNNs) are inherently error-tolerant, SC could be embedded in them to gain higher speed and lower power without significant accuracy loss. In this paper, we propose using SC techniques to approximate multiplication operations on fixed-point weights and biases during training of CNNs. By employing such techniques, we demonstrate near state-of-the-art learning performance for the MNIST and CIFAR-10 datasets, while achieving significant resource and speed improvements when implementing the deep networks on a Field Programmable Gate Array (FPGA). For MNIST, we demonstrate that SC compared to conventional computing, will result in almost 3 times increase in learning speed with only 1.37% degradation in validation accuracy. Similarly, for CIFAR-10, training is accelerated 3.5 times with a degradation of 3.39%. We also show that our FPGA implementations of CNNs adopting stochastic multipliers consume over 17 times less power than their GPU counterparts.

Stochastic Computing (SC) is a computing paradigm that allows for the\nlow-cost and low-power computation of various arithmetic operations using\nstochastic bit streams and digital logic. In contrast to conventional\nrepresentation schemes used within the binary domain, the sequence of bit\nstreams in the stochastic domain is inconsequential, and computation is usually\nnon-deterministic. In this brief, we exploit the stochasticity during switching\nof probabilistic Conductive Bridging RAM (CBRAM) devices to efficiently\ngenerate stochastic bit streams in order to perform Deep Learning (DL)\nparameter optimization, reducing the size of Multiply and Accumulate (MAC)\nunits by 5 orders of magnitude. We demonstrate that in using a 40-nm\nComplementary Metal Oxide Semiconductor (CMOS) process our scalable\narchitecture occupies 1.55mm$^2$ and consumes approximately 167$\\mu$W when\noptimizing parameters of a Convolutional Neural Network (CNN) while it is being\ntrained for a character recognition task, observing no notable reduction in\naccuracy post-training.\n

Stochastic computing, which employs random bit streams for computations, has shown low hardware cost and high fault-tolerance compared to the computations using a conventional binary encoding. Finite state machine (FSM) based stochastic computing elements can compute complex functions, such as the exponentiation and hyperbolic tangent functions, more efficiently than those using combinational logic. However, the FSM, as a sequential logic, cannot be directly implemented in parallel like the combinational logic, so reducing the long latency of the calculation becomes difficult. Applications in the relatively higher frequency domain would require an extremely fast clock rate using FSM. This paper proposes a parallel implementation of the FSM, using an estimator and a dispatcher to directly initialize the FSM to the steady state. Experimental results show that the outputs of four typical functions using the parallel implementation are very close to those of the serial version. The parallel FSM scheme further shows equivalent or better image quality than the serial implementation in two image processing applications Edge Detection and Frame Difference.

As we approach the limits of traditional Moore’s-Law scaling, alternative computing techniques that consume energy more efficiently become attractive. Stochastic computing (SC), as a re-emerging computing technique, is a low-cost and error-tolerant alternative to conventional binary circuits in several important applications such as image processing and communications. SC allows a natural accuracy-energy tradeoff that has been exploited in the past. This article presents an accuracy-energy tradeoff technique for SC circuits that reduces their energy consumption with virtually no accuracy loss. To this end, we employ voltage or frequency scaling, which normally reduce energy consumption at the cost of timing errors. Then we show that due to their inherent error tolerance, SC circuits operate satisfactorily without significant accuracy loss even with aggressive scaling. This significantly improves their energy efficiency. In contrast, conventional binary circuits quickly fail as the supply voltage decreases. To find the most energy-efficient operating point of an SC circuit, we propose an error estimation method that allows us to quickly explore the circuit’s design space. The error estimation method is based on Markov chain and least-squares regression. Furthermore, we investigate opportunities to optimize SC circuits under such aggressive scaling. We find that logical and physical design techniques can be combined to significantly expand the already-powerful accuracy-energy tradeoff possibilities of SC. In particular, we demonstrate that careful adjustment of path delays can lead to significant error reduction under voltage and frequency scaling. We perform buffer insertion and route detouring to achieve more balanced path delays. These techniques differ from conventional path-balancing techniques whose goal is to minimize power consumption by resizing the non-critical paths. The goal of our path-balancing approach is to increase error cancellation chances in voltage-/frequency-scaled SC circuits. Our circuit optimization comprehends the tradeoff between power overheads due to inserted buffers and wires versus the energy reduction from supply voltage downscaling enabled by more balanced path delays. Simulation results show that our optimized SC circuits can tolerate aggressive voltage scaling with no significant signal-to-noise ratio (SNR) degradation. In one example, a 40% supply voltage reduction (1V to 0.6V) on the SC circuit leads to 66% energy saving (20.7pJ to 6.9pJ) and makes it more efficient than its conventional binary counterpart. In the same example, a 100% frequency boosting (400ps to 200ps) of the optimized circuits leads to no significant SNR degradation. We also show that process variation and temperature variation have limited impact on optimized SC circuits. The error change is less than 5% when temperature changes by 100°C or process condition changes from worst case to best case.

Energy efficiency presents a significant challenge for stochastic computing (SC) due to the long random binary bit streams required for accurate computation. In this paper, a type of low discrepancy (LD) sequences, the Sobol sequence, is considered for energy-efficient implementations of SC circuits. The use of Sobol sequences improves the output accuracy of a stochastic circuit with a reduced sequence length compared to the use of another type of LD sequences, the Halton sequence, and conventional linear feedback shift register (LFSR)-generated pseudorandom sequence. The use of Sobol sequences leads to a similar or higher accuracy than using Halton sequences for basic arithmetic operations. Sobol sequence generators cost less energy than the Halton counterparts when multiple random sequences are required in a circuit, thus the use of Sobol sequences can lead to a higher energy efficiency in an SC circuit than using Halton sequences.

Both binarized and stochastic computing-based neural networks exploit bit-wise operations to replace expensive full-precision multiplications with simple XNOR gates and thus, offer low-cost hardware implementation. In stochastic computing, arithmetic computations are performed on sequences of random bits which can approximate any real values. Stochastic computing-based neural networks benefit from approximate computing and promote fault-tolerant architectures against soft errors in noisy environments. On the other hand, in binarized neural networks, real values are deterministically binarized using the sign function. As a result, any bit-flip in the binarized values dramatically changes the outcome of arithmetic computations and makes binarized neural networks more vulnerable against soft errors. In this paper, we compare these two neural networks against each other in terms of fault-tolerance and hardware complexity (i.e., area and energy efficiency).

Energy efficiency has been a big concern for the IC industry. Although the CMOS technology continues to scale down, the scaling of the V dd on the other hand, has been stagnant for a long while, leading to the dark silicon dilemma, which means more cores can fit on a die, but a growing fraction cannot actually be turned on due to strict power constraint [1]. Meanwhile, the blooming development of internet-of-things (IoT) and wearable/implantable electronics has raised additional requirements on ultralow power. Thus, the community has re-started searching for alternative computing paradigms to overcome the power challenges and achieve performance improvements.Stochastic computing (SC) [2]-[4], which processes the data in form of random bit streams, is a promising alternative to conventional deterministic binary computing. It enables ultralow power, high error-tolerance and massive parallelism. In this talk, I will discuss the feasibility of SC circuits based on state-of-the-art FinFET technology. Applications of on-chip image processing [5] and stochastic computing neural networks [6] are taken as examples. Practical technical issues are carefully examined, including static and transient device variations. Design optimization methodology is also proposed for SC circuits, which can largely reduce its energy consumption with almost no accuracy loss.

Neural networks have been used in a variety of fields including speech process, image recognition and video classification. In this work, based on 16/14nm FinFET technology, the feasibility of the neuron circuit using stochastic computing is evaluated. Considering the practical technology issues, the impact of random variabilities is carefully examined under ultra-low voltage. Using handwritten digital MNIST database, a reliable validation accuracy in neural network based on stochastic computing neuron circuits can be obtained with an ultra-low V dd . The design optimization procedure of neuron circuits is also proposed, which can decrease the energy consumption without the penalty of accuracy.

Applying error recovery monotonously can either compromise the real-time constraint, or worsen the power/energy envelope. Neither of these violations can be realistically accepted in embedded system design, which expects ultra efficient realization of a given application. In this paper, we propose a HW/SW methodology that exploits both application specific characteristics and Spatial/Temporal redundancy. Our methodology combines design-time and runtime optimizations, to enable the resultant embedded processor to perform runtime adaptive error recovery operations, precisely targeting the reliability-wise critical instruction executions. The proposed error recovery functionality can dynamically 1) evaluate the reliability cost economy (in terms of execution-time and dynamic power), 2) determine the most profitable scheme, and 3) adapt to the corresponding error recovery scheme, which is composed of spatial and temporal redundancy based error recovery operations. The experimental results have shown that our methodology at best can achieve fifty times greater reliability while maintaining the execution time and power deadlines, when compared to the state of the art.