Fully-Integrated Spiking Neural Network Using SiOx-Based RRAM as Synaptic Device

This paper presents, to the best of the authors' knowledge, the first complete integration of a Spiking Neural Network, combining analog neurons and Resistive RAM (RRAM)-based synapses. The implemented topology is a perceptron, aimed at performing MNIST classification. An existing framework was tailored for offline learning and weight quantization. The test chip, fabricated in 130nm CMOS, shows well-controlled integration of synaptic currents and no RRAM read disturb issue during inference tasks (at least 750M spikes). The classification accuracy is 84%, with a 3.6 pJ energy dissipation per spike at the synapse and neuron level (up to 5x lower vs. similar chips using formal coding).

Spiking neural networks (SNNs) are quickly gaining traction as a viable alternative to deep neural networks (DNNs). Compared to DNNs, SNNs are computationally more powerful and energy efficient. The design metrics (synaptic weights, membrane threshold, etc.) chosen for such SNN architectures are often proprietary and constitute confidential intellectual property (IP). Our study indicates that SNN architectures implemented using conventional analog neurons are susceptible to side channel attack (SCA). Unlike the conventional SCAs that are aimed to leak private keys from cryptographic implementations, SCANN (SCA̲ of spiking n̲eural n̲etworks) can reveal the sensitive IP implemented within the SNN through the power side channel. We demonstrate eight unique SCANN attacks by taking a common analog neuron (axon hillock neuron) as the test case. We chose this particular model since it is biologically plausible and is hence a good fit for SNNs. Simulation results indicate that different synaptic weights, neurons/layer, neuron membrane thresholds, and neuron capacitor sizes (which are the building blocks of SNN) yield distinct power and spike timing signatures, making them vulnerable to SCA. We show that an adversary can use templates (using foundry-calibrated simulations or fabricating known design parameters in test chips) and analysis to identify the specifications of the implemented SNN.

Spiking Neural Network (SNN) is the third generation of Artificial Neural Network (ANN), which is potentially an efficient way to reduce the computation load as well as the power consumption on hardware due to the sparse activation and event-driven behavior of its neuron. In this paper, we have trained an SNN by using Spike Timing Dependent Plasticity in an unsupervised manner. The training process is optimized by using the quantization method on its weights, performed at the regular intervals during training. From the simulation results, the SNN with the 4-bit weights received an accuracy drop of 0.09%. The behavioral simulation result with the same bit-width weight quantization received an accuracy drop of 0.28%.

Spiking neural networks (SNNs) represent a promising frontier in artificial intelligence (AI), offering event-driven, energy-efficient computation that mimics rich neural dynamics in the brain. However, running large-scale SNNs on mainstream computing hardware faces significant challenges to efficiently emulate these dynamical processes using synchronized and logical chips. Memristor based systems have recently demonstrated great potential for AI acceleration, sparking speculations and explorations of using these emerging devices for SNN tasks. This paper reviews the promises and challenges of memristive devices in SNN implementations, and our discussions are focused on the scaling and integration of neuronal and synaptic devices. We survey recent progress in device and circuit development, discuss possible pathways for chip-level integration, and finally probe into hardware-oriented algorithm designs. This review offers a system-level perspective on implementing scalable memristor based SNN platforms.

Artificial Neural Networks (ANNs) have shown remarkable performance in various fields. However, ANN relies on the von-Neumann architecture, which consumes a lot of power. Hardware-based spiking neural networks (SNNs) inspired by a human brain have become an alternative with significantly low power consumption. In this paper, we propose on-chip trainable SNNs using a time-to-first-spike (TTFS) method. We modify the learning rules of conventional SNNs using TTFS to be suitable for on-chip learning. Vertical NAND flash memory cells fabricated by a device manufacturer are used as synaptic devices. The entire learning process considering the hardware implementation is also demonstrated. The performance of the proposed network is evaluated through the MNIST classification in system-level simulation using Python. The proposed SNNs show an accuracy of 96% for a network size of 784 – 400 – 10. We also investigate the effect of non-ideal cell characteristics (such as pulse-to-pulse and device-to-device variations) on inference accuracy. Our networks demonstrate excellent immunity for various device variations compared with the networks using off-chip training.

In this work, we implement hardware-based spiking neural network (SNN) using the thin-film transistor (TFT)-type flash synaptic devices. A hardware-based SNN architecture with synapse arrays and integrate-and-fire (I&F) neuron circuits is presented for executing reinforcement learning (RL). Two problems were used to evaluate the applicability of the proposed hardware-based SNNs to off-chip RL: the Cart Pole balancing problem and the Rush Hour problem. The neural network was trained using a deep Q-learning algorithm. The proposed hardware-based SNNs using the synapse model with measured characteristics successfully solve the two problems and show high performance, implying that the networks are suitable for executing RL. Furthermore, the effect of variations in non-ideal synaptic devices and neurons on the performance was investigated.

Spiking neurons are essential for building energy-efficient biomimetic spatiotemporal systems because they communicate with other neurons using sparse and binary signals. However, the achievable high density of artificial neurons having a capacitor for emulating the integrate function of biological neurons has a limit. Furthermore, a low-voltage operation (<1.0V) is essential for connecting with modern complementary metal-oxide-semiconductor-field-effect-transistor-based (C-MOSFET-based) integrated circuits. Here, a capacitorless memristive-neural integrated chip (MnIC) based on the negative differential resistance of the electrochemical metallization cell designed using a 28-nm C-MOSFET process in a foundry is reported. The fabricated MnIC exhibits extremely low-voltage operation (<0.7V) via the rupture dynamics of Ag filaments formed in the GeS2 chalcogenide layer, with a nonlinear increase in the action potential in a manner similar to a human sensory system. Moreover, to construct a fully-structured spiking neural network (SNN), an oxygenated amorphous carbon-based (α-COx-based) synaptic device having 32 multi-level conductance states is designed. The designed MnIC and α-COx-based synaptic device demonstrate real-time unsupervised learning via a spike-timing-dependent plasticity learning rule with an SNN. Using the trained SNN, the real-time hand-written digit image of a cell phone obtained from a live webcam is successfully classified, which suggests practical applications for brain-like neuromorphic chips.

Building accurate and efficient deep neural network (DNN) models for intelligent sensing systems to process data locally is essential. Spiking neural networks (SNNs) have gained significant popularity in recent years because they are more biological-plausible and energy-efficient than DNNs. However, SNNs usually have lower accuracy than DNNs. In this paper, we propose to use SNNs for image sensing applications. Moreover, we introduce the DNN-SNN knowledge distillation algorithm to reduce the accuracy gap between DNNs and SNNs. Our DNNSNN knowledge distillation improves the accuracy of an SNN by transferring knowledge between a DNN and an SNN. To better transfer the knowledge, our algorithm creates two learning paths from a DNN to an SNN. One path is between the output layer and another path is between the intermediate layer. DNNs use real numbers to propagate information between neurons while SNNs use 1-bit spikes. To empower the communication between DNNs and SNNs, we utilize a decoder to decode spikes into real numbers. Also, our algorithm creates a learning path from an SNN to a DNN. This learning path better adapts the DNN to the SNN by allowing the DNN to learn the knowledge from the SNN. Our SNN models are deployed on Loihi, which is a specialized chip for SNN models. On the MNIST dataset, our SNN models trained by the DNN-SNN knowledge distillation achieve better accuracy than the SNN models on GPU trained by other training algorithms with much lower energy consumption per image.

Developmental plasticity plays a prominent role in shaping the brain's structure during ongoing learning in response to dynamically changing environments. However, the existing network compression methods for deep artificial neural networks (ANNs) and spiking neural networks (SNNs) draw little inspiration from brain's developmental plasticity mechanisms, thus limiting their ability to learn efficiently, rapidly, and accurately. This paper proposed a developmental plasticity-inspired adaptive pruning (DPAP) method, with inspiration from the adaptive developmental pruning of dendritic spines, synapses, and neurons according to the "use it or lose it, gradually decay" principle. The proposed DPAP model considers multiple biologically realistic mechanisms (such as dendritic spine dynamic plasticity, activity-dependent neural spiking trace, and local synaptic plasticity), with additional adaptive pruning strategy, so that the network structure can be dynamically optimized during learning without any pre-training and retraining. Extensive comparative experiments show consistent and remarkable performance and speed boost with the extremely compressed networks on a diverse set of benchmark tasks for deep ANNs and SNNs, especially the spatio-temporal joint pruning of SNNs in neuromorphic datasets. This work explores how developmental plasticity enables complex deep networks to gradually evolve into brain-like efficient and compact structures, eventually achieving state-of-the-art (SOTA) performance for biologically realistic SNNs.

Rethinking the performance comparison between SNNS and ANNS

Spiking neural networks (SNNs), inspired by the neuronal network in the brain, provide biologically relevant and low-power consuming models for information processing. Existing studies either mimic the learning mechanism of brain neural networks as closely as possible, for example, the temporally local learning rule of spike-timing-dependent plasticity (STDP), or apply the gradient descent rule to optimize a multilayer SNN with fixed structure. However, the learning rule used in the former is local and how the real brain might do the global-scale credit assignment is still not clear, which means that those shallow SNNs are robust but deep SNNs are difficult to be trained globally and could not work so well. For the latter, the nondifferentiable problem caused by the discrete spike trains leads to inaccuracy in gradient computing and difficulties in effective deep SNNs. Hence, a hybrid solution is interesting to combine shallow SNNs with an appropriate machine learning (ML) technique not requiring the gradient computing, which is able to provide both energy-saving and high-performance advantages. In this article, we propose a HybridSNN, a deep and strong SNN composed of multiple simple SNNs, in which data-driven greedy optimization is used to build powerful classifiers, avoiding the derivative problem in gradient descent. During the training process, the output features (spikes) of selected weak classifiers are fed back to the pool for the subsequent weak SNN training and selection. This guarantees HybridSNN not only represents the linear combination of simple SNNs, as what regular AdaBoost algorithm generates, but also contains neuron connection information, thus closely resembling the neural networks of a brain. HybridSNN has the benefits of both low power consumption in weak units and overall data-driven optimizing strength. The network structure in HybridSNN is learned from training samples, which is more flexible and effective compared with existing fixed multilayer SNNs. Moreover, the topological tree of HybridSNN resembles the neural system in the brain, where pyramidal neurons receive thousands of synaptic input signals through their dendrites. Experimental results show that the proposed HybridSNN is highly competitive among the state-of-the-art SNNs.

Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Compared with counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8×/7.4×. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNN research, with a great balance between effectiveness and energy efficiency.

Spiking neural networks (SNNs) have attracted many researchers’ interests due to its biological plausibility and event-driven characteristic. In particular, recently, many studies on high-performance SNNs comparable to the conventional analog-valued neural networks (ANNs) have been reported by converting weights trained from ANNs into SNNs. However, unlike ANNs, SNNs have an inherent latency that is required to reach the best performance because of differences in operations of neuron. In SNNs, not only spatial integration but also temporal integration exists, and the information is encoded by spike trains rather than values in ANNs. Therefore, it takes time to achieve a steady-state of the performance in SNNs. The latency is worse in deep networks and required to be reduced for the practical applications. In this work, we propose a pre-charged membrane potential (PCMP) for the latency reduction in SNN. A variety of neural network applications (e.g., classification, autoencoder using MNIST and CIFAR-10 datasets) are trained and converted to SNNs to demonstrate the effect of the proposed approach. The latency of SNNs is successfully reduced without accuracy loss. In addition, we propose a delayed evaluation method (DE), by which the errors during the initial transient are discarded. The error spikes occurring in the initial transient is removed by DE, resulting in the further latency reduction. DE can be used in combination with PCMP for further latency reduction. Finally, we also show the advantages of the proposed methods in improving the number of spikes required to reach a steady-state of the performance in SNNs for energy-efficient computing.

Filamentary and interface switching of CMOS-compatible Ta2O5 memristor for non-volatile memory and synaptic devices

Neuron circuits are the fundamental building blocks in the modern neuromorphic system. Designing compact and low‐power neuron circuits can significantly improve the overall area and energy efficiencies of a neuromorphic chip architecture. Here, practical neuron circuits must overcome the variations arising from nonideal behaviors of synaptic devices, such as stuck‐at‐fault and conductance deviation. In this study, a compact leaky integrate‐and‐fire neuron circuit has been designed, with resilience to synaptic device state variations, for hardware implementation of spiking neural networks (SNNs). The proposed neuron circuit is simulated on the 0.35‐ μ m Si complementary metal‐oxide‐semiconductor technology node by a series of circuit simulations based on HSPICE. The proposed circuit occupies a reduced area and exhibits low power consumption (14.7 µ W per spike). Furthermore, the optimized circuit design results in a high degree of tolerance toward input‐current variations arising from conductance‐state variations in the synapse array. Hence, the proposed neuron circuit would be capable of substantially improving the area efficiency and reliability in the realization of the hardware‐oriented SNN architectures.