- Video Transcripts
- 10.48448/xywb-cr81
Running Deep Neural Networks on Superparamagnetic Fluctuations
- Mar 30, 2021
- Underline Science Inc.
- Philippe Talatchian + 4 more +4
Magnetic tunnel junctions offer significant and near-term improvements to new computer chips, most immediately from non-volatile memory applications [1]. These devices consist of two magnetic layers whose relative alignment – parallel or antiparallel – determine the device's two-terminal resistance. When the energy barrier between these two states is sufficiently small, thermal fluctuations can cause the device to exhibit random telegraph noise in its resistive state. In this case, the device is called a superparamagnetic tunnel junction [2].Computing with random telegraph noise can offer favorable properties such as noise-tolerance, massive parallelism, and – for certain operations – low power and area footprints. The field of stochastic computing addresses questions of engineering such systems but has mostly been restricted to the use of low-quality pseudorandom number generators. The low periodicity and high cross-correlations between these devices have restricted traditional stochastic computing by forcing frequent regeneration of stochastic bitstreams. Crossing in and out of the stochastic domain can incur significant energy penalties [3].In our work [4], we introduce a low-energy circuit for extracting programmable randomness from superparamagnetic tunnel junctions. The required probability is determined via a digital input signal, rather than an analog current, so ohmic losses are minimized (Figure 1). Using detailed analog circuit simulation, we find that our device requires only half the energy of the traditional all-CMOS solution, while also offering infinite-period truly random bitstreams.To demonstrate the value of truly random bitstreams, we designed a deep neural network modeled after LeNet5 [5], a famous convolutional neural network used for handwriting recognition. Traditional neural networks multiply neuron outputs by a synaptic weight, add these products together at the input of another neuron, and then apply a nonlinear operation to that input sum. In our implementation, the values passed between neurons are encoded in the expected value of the voltage on any given wire. To multiply two signals together, we need only use a Boolean AND gate to construct the joint distribution of the wires. Feeding many wires into a Boolean OR gate generates a polynomial function which looks like linear addition near the origin, but acquires high-order nonlinear corrections when the inputs are strong. However, the OR gate only functions in this way because the inputs can be regarded as statistically independent, which would be impossible with pseudorandom number generators absent an unacceptable increase in energy consumption.We implement LeNet5 using superparamagnetic tunnel junctions together with these simple logic gates and train a simulation of this deep neural network. We are able to achieve 97 % accuracy on handwritten digit recognition using only 150 nJ per inference. This energy efficiency represents a factor of 1.4 to 7.7 improvement over comparable proposals that do not use tunnel junctions. However, after systematic optimization, the best CMOS-only solution is able to achieve 99.1 % accuracy [6]. Questions about the best way to train OR-gate neurons for optimal performance is a question for future research. Nevertheless, our demonstration shows that taking a digital approach to utilizing SMTJ fluctuations can unlock extremely energy-efficient circuit designs with the potential for offering stochastic neural networks computing in near-term application specific integrated circuits. **
Read more