An Efficient Hardware Implementation of Activation Functions Using Stochastic Computing for Deep Neural Networks

Stochastic computing, which employs random bit streams for computations, has shown low hardware cost and high fault-tolerance compared to the computations using a conventional binary encoding. Finite state machine (FSM) based stochastic computing elements can compute complex functions, such as the exponentiation and hyperbolic tangent functions, more efficiently than those using combinational logic. However, the FSM, as a sequential logic, cannot be directly implemented in parallel like the combinational logic, so reducing the long latency of the calculation becomes difficult. Applications in the relatively higher frequency domain would require an extremely fast clock rate using FSM. This paper proposes a parallel implementation of the FSM, using an estimator and a dispatcher to directly initialize the FSM to the steady state. Experimental results show that the outputs of four typical functions using the parallel implementation are very close to those of the serial version. The parallel FSM scheme further shows equivalent or better image quality than the serial implementation in two image processing applications Edge Detection and Frame Difference.

As we approach the limits of traditional Moore’s-Law scaling, alternative computing techniques that consume energy more efficiently become attractive. Stochastic computing (SC), as a re-emerging computing technique, is a low-cost and error-tolerant alternative to conventional binary circuits in several important applications such as image processing and communications. SC allows a natural accuracy-energy tradeoff that has been exploited in the past. This article presents an accuracy-energy tradeoff technique for SC circuits that reduces their energy consumption with virtually no accuracy loss. To this end, we employ voltage or frequency scaling, which normally reduce energy consumption at the cost of timing errors. Then we show that due to their inherent error tolerance, SC circuits operate satisfactorily without significant accuracy loss even with aggressive scaling. This significantly improves their energy efficiency. In contrast, conventional binary circuits quickly fail as the supply voltage decreases. To find the most energy-efficient operating point of an SC circuit, we propose an error estimation method that allows us to quickly explore the circuit’s design space. The error estimation method is based on Markov chain and least-squares regression. Furthermore, we investigate opportunities to optimize SC circuits under such aggressive scaling. We find that logical and physical design techniques can be combined to significantly expand the already-powerful accuracy-energy tradeoff possibilities of SC. In particular, we demonstrate that careful adjustment of path delays can lead to significant error reduction under voltage and frequency scaling. We perform buffer insertion and route detouring to achieve more balanced path delays. These techniques differ from conventional path-balancing techniques whose goal is to minimize power consumption by resizing the non-critical paths. The goal of our path-balancing approach is to increase error cancellation chances in voltage-/frequency-scaled SC circuits. Our circuit optimization comprehends the tradeoff between power overheads due to inserted buffers and wires versus the energy reduction from supply voltage downscaling enabled by more balanced path delays. Simulation results show that our optimized SC circuits can tolerate aggressive voltage scaling with no significant signal-to-noise ratio (SNR) degradation. In one example, a 40% supply voltage reduction (1V to 0.6V) on the SC circuit leads to 66% energy saving (20.7pJ to 6.9pJ) and makes it more efficient than its conventional binary counterpart. In the same example, a 100% frequency boosting (400ps to 200ps) of the optimized circuits leads to no significant SNR degradation. We also show that process variation and temperature variation have limited impact on optimized SC circuits. The error change is less than 5% when temperature changes by 100°C or process condition changes from worst case to best case.

Deep learning has become an increasingly heated topic as artificial intelligence is on the rise. At the same time, hardware restrictions in real applications have driven the investigation of combining another rising technique, stochastic computing (SC), with deep learning systems to achieve low power costs. By far, operations successfully implemented include addition, multiplication, inner product, and other more complicated nonlinear functions such as hyperbolic tangent (tanh) with linear finite state machines (FSM). The inner product implementation realizes convolution, a core of neural networks, therefore encouraging SC- based deep learning neural network implementations. Meanwhile, extremely long bitstream lengths are needed to achieve satisfying accuracy, especially for large-scale deep learning systems, causing latency issues. The integration of parallelism is thus considered in an attempt to alleviate the latency issue. In this paper, an optimization to stochastic computing based deep learning system is proposed by introducing parallel FSM implementations to replace serial ones generally used in previous works. Substituting serial linear FSMs with several parallel linear FSMs of the same size yet with shorter bitstream length, parallel FSM aims at trading hardware for processing latency. The accuracy performance of a sample parallel FSM unit is evaluated against its counterpart in serial implementation before a case study verifies that the replacement sacrifices little accuracy, while reducing computing time exponentially in actual deep learning system realizations.

We consider the design of stochastic computing (SC) hardware based on spintronic devices. SC offers low-cost implementations of arithmetic operations and high degrees of error tolerance. When compared to charge-based devices, spin-based devices could be lower energy, nonvolatile, etc. However, spin-based devices can be fundamentally more error prone than charge-based devices. The marriage of SC architectures and spin-based devices has the potential to produce information processing systems that are robust, low energy, and nonvolatile. In this paper, we investigate SC hardware comprised of nanomagnetic logic (NML) devices. NML is a “beyond-CMOS” technology that uses bistable magnets to store, process, and move binary information. We introduce new NML circuit structures that exploit unique features of the technology to efficiently realize hardware components required for an SC system (e.g., random number generation). We also benchmark NML-based SC circuits against other implementations, and illustrate how features that are unique to NML (e.g., inherent pipelining) can lead to improved performance. Our results indicate NML SC implementations achieve smaller area footprints with reduced energy and delay when compared to CMOS equivalents.

Many leading technology companies currently offer Machine Learning as a Service Platform, enabling developers and organizations to access the inference capabilities of pre-trained models via API calls. However, due to concerns over user data privacy, inter-enterprise competition, and legal and regulatory constraints, directly utilizing pre-trained models in the cloud for inference faces security challenges. In this paper, we propose communication-efficient secure three-party protocols for recurrent neural network (RNN) inference. First, we design novel three-party secret-sharing protocols for digit decomposition, B2A conversion, enabling efficient transformation of secret shares between Boolean and arithmetic rings. Then, we propose the lookup table-based secure three-party protocol. Unlike the intuitive way of directly looking up tables to obtain results, we compute the results by utilizing the inherent mathematical properties of binary lookup tables, and the communication complexity of the lookup table protocol is only related to the output bit width. We also design secure three-party protocols for key functions in the RNN, including matrix multiplication, sigmoid function, and Tanh function. Our protocol divides the computation into online and offline phase, and places most of the computations locally. The theoretical analysis shows that the communication round of our work was reduced from four rounds to one round. The experiment results show that compared with the current SOTA-SIRNN, the online communication overhead of sigmoid and tanh functions decreased by 80.39% and 79.94%, respectively.

To enable local or edge AI like family-based robots and self-driving cars without using cloud computing, the power consumption for any available computing system is still four to six orders of magnitude higher than needed. Among all proposed spintronic devices for computing, magnetic tunnel junctions (MTJs) [1] hold the brightest future with large practical impacts considering the maturity of manufacturing MTJs array. Magnetic tunnel junctions (MTJs) were proved to provide high endurance and low-cost solutions for non-volatile, stochastic and nonlinear activation functions and the combinations [2]. Those enable both the stochastic computing and probabilistic computing. This talk will report a new computing architecture based on MTJs: computational random-access memory (CRAM) [3,4], which will be a key build block for different AI accelerators.

When can environmental variability benefit population growth? Counterintuitive effects of nonlinearities in vital rates

Stochastic computing (SC) acts on data encoded by bit-streams, and is an attractive, low-cost and error-tolerant alternative to conventional binary circuits in some important applications such as image processing and communications. We study the use of energy reduction techniques such as voltage or frequency scaling in SC circuits. We show that due to their inherent error-tolerance, SC circuits operate satisfactorily without significant accuracy loss even with aggressive scaling that improves their energy efficiency by orders of magnitude. To find the minimum-energy operating point of an SC circuit, we propose a Markov chain model that allows us to quickly explore the space of operating points. We also investigate opportunities to optimize SC circuits under such aggressive scaling. We find that logical and physical design techniques can be used to significantly expand the already powerful accuracy-energy tradeoff possibilities in SC circuits. Our simulation results show that our optimized SC circuits can tolerate aggressive voltage scaling with no significant SNR degradation after 40% supply voltage reduction (1V to 0.6V), leading to 66% energy saving (20.7pJ to 6.9pJ). Similarly, a 100% frequency boosting (400ps to 200ps) of the optimized circuits leads to no significant SNR degradation for several representative circuits.

Stochastic computing (SC) is a method of expressing a numerical value by the appearance ratio of 1 in a bitstream to perform an operation. Rapid single flux quantum (RSFQ) circuits can handle SC naturally and operate at ultra-high speed. RSFQ SC circuits are suitable for applications tolerating error. This paper presents a design method of RSFQ SC circuits to calculate polynomials. Using the proposed design method, an internal circuit is designed to feed bitstreams of coefficients from bitstreams of constant to reduce large stochastic number generators. The method inserts storage elements to reduce correlations of signals, and thereby reduce the calculation error of the circuit. Additionally, it minimizes the number of inserted D flip-flops utilizing the latching function of gate inputs of RSFQ logic gates.

Energy efficiency presents a significant challenge for stochastic computing (SC) due to the long random binary bit streams required for accurate computation. In this paper, a type of low discrepancy (LD) sequences, the Sobol sequence, is considered for energy-efficient implementations of SC circuits. The use of Sobol sequences improves the output accuracy of a stochastic circuit with a reduced sequence length compared to the use of another type of LD sequences, the Halton sequence, and conventional linear feedback shift register (LFSR)-generated pseudorandom sequence. The use of Sobol sequences leads to a similar or higher accuracy than using Halton sequences for basic arithmetic operations. Sobol sequence generators cost less energy than the Halton counterparts when multiple random sequences are required in a circuit, thus the use of Sobol sequences can lead to a higher energy efficiency in an SC circuit than using Halton sequences.

The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> -out-of- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> : G(F) majority voter consists of n components (or modules) and a number of the components are required to be operating correctly for the overall system to be correct. As per the state discretization of the components, such a system is usually classified as either a binary system or a multi-state system. In practice, the operating conditions of different components may contribute differently to the operation of the entire system. In this manuscript, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> -out-of- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> : G(F) majority voter is generalized as a consecutive-weighted- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> -out-of- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> : G(F) voter with either binary states or multiple states. To overcome the drawbacks of existing approaches, a stochastic analysis is proposed for assessing the system reliability. In the stochastic analysis, the input signal probabilities are encoded into non-Bernoulli sequences with fixed numbers of 0s and 1s for the Boolean case, or randomly permuted sequences for the multi-state scenario. By using stochastic logic, the reliability of a general system consisting of consecutive-weighted- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> -out-of- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> majority voters is efficiently and accurately predicted. The results are validated by an analysis of several case studies. Although the accuracy of the stochastic analysis is closely related with the employed sequence length, it is shown that a stochastic approach is more efficient than a universal generating function (UGF) method, while still retaining an acceptable accuracy.

Spider monkey optimization (SMO) algorithm is a recently developed optimizer that is stimulated by the extraordinary social activities of spider monkeys known as fission–fusion social structure. The SMO is developed to find solution of difficult optimization problems in real world, which are difficult to solve by the available deterministic strategies. During the solution search process in SMO, perturbation rate plays very important role. The convergence rate of SMO is highly affected by it. Usually, perturbation rate is defined by a simple function that is linearly in nature. But some application has nonlinear nature, thus a nonlinear function may improve the outcomes of SMO. For that reason, a non linear function, namely sigmoidal function used to decide perturbation in SMO and proposed strategy named as sigmoidal SMO. The investigational outcomes show the superiority of the anticipated technique over other meta-heuristics.

A conventional proportional integral (PI) controller is modified with a nonlinear activation function (sigmoid function) applied directly to the controller output in order to improve the stability and target fidelity of the system response to large variational inputs in both state and internal controller gain variables. The controller is implemented in a simulated water management role applied to a major subregional pumping station between Lake Okeechobee, the Loxahatchee National Wildlife Refuge (WCA1), and the West Palm Beach supply canal (C51) in the South Florida geographic region. The simulation consists of an integrated hydrological numerical model implemented in the Regional Simulation Model (RSM), which is currently under development at the South Florida Water Management District. Analysis of the modified controller in the Laplace domain establishes the expected control behavior, and subsequent results of the simulation for the conventional PI and the sigmoid modified controller are presented and compared. The modified controller achieves significant improvement in stability while simultaneously reducing the control signal energy.

The limitations of conventional model-based control mechanisms for flexible manipulator systems have stimulated the development of intelligent control mechanisms incorporating fuzzy logic and neural networks. Problems have been encountered in applying the traditional PD-, PI-, and PID-type fuzzy controllers to flexible-link manipulators. A PD-PI-type fuzzy controller has been developed where the membership functions are adjusted by tuning the scaling factors using a neural network. Such a network needs a sufficient number of neurons in the hidden layer to approximate the nonlinearity. A simple realisable network is desirable and hence a single neuron network with a nonlinear activation function is used. It has been demonstrated that the sigmoidal function and its shape can represent the nonlinearity of the system. A genetic algorithm is used to learn the weights, biases and shape of the sigmoidal function of the neural network.

This is a comparison between two different digit recognition models, with being almost similar in structure except the last layer of the multi-layered neural network. One model has the Sigmoid function while the other model has the SoftMax function as the last non-linear activation function.