Analytical performance assessment and high-throughput low-latency spike routing algorithm for spiking neural network systems

Spiking neural networks (SNNs) are artificial neural network models that more closely mimic biological neural networks. In addition to neuronal and synaptic state, SNNs incorporate the variant time scale into their computational model. Since each neuron in these networks is connected to thousands of others, high bandwidth is required. Moreover, since the spike times are used to encode information in SNN, very low communication latency is also needed. The 2D-NoC was used as a solution to provide a scalable interconnection fabric in large-scale parallel SNN systems. The 3D-ICs have also attracted a lot of attention as a potential solution to resolve the interconnect bottleneck. The combination of these two emerging technologies provides a new horizon for IC designs to satisfy the high requirements of low power and small footprint in emerging AI applications. In this work, we first present a comprehensive analytical model to analyze the performance of 3D mesh NoC over variants of different SNN topologies and communications protocols. Second, we present an architecture and a low-latency spike routing algorithm, named shortest path K-means based multicast (SP-KMCR), for three-dimensional NoC of spiking neurons (3DNoC-SNN). The proposed system was validated based on an RTL-level implementation, while area/power analysis was performed using 45nm CMOS technology.

Rethinking the performance comparison between SNNS and ANNS

Monitoring heart conditions through electrocardiography (ECG) has been the cornerstone of identifying cardiac irregularities. Cardiologists often rely on a detailed analysis of ECG recordings to pinpoint deviations that are indicative of heart anomalies. This traditional method, while effective, demands significant expertise and is susceptible to inaccuracies due to its manual nature. In the realm of computational analysis, Artificial Neural Networks (ANNs) have gained prominence across various domains, which can be attributed to their superior analytical capabilities. Conversely, Spiking Neural Networks (SNNs), which mimic the neural activity of the brain more closely through impulse-based processing, have not seen widespread adoption. The challenge lies primarily in the complexity of their training methodologies. Despite this, SNNs offer a promising avenue for energy-efficient computational models capable of displaying a high-level performance. This paper introduces an innovative approach employing SNNs augmented with an attention mechanism to enhance feature recognition in ECG signals. By leveraging the inherent efficiency of SNNs, coupled with the precision of attention modules, this model aims to refine the analysis of cardiac signals. The novel aspect of our methodology involves adapting the learned parameters from ANNs to SNNs using leaky integrate-and-fire (LIF) neurons. This transfer learning strategy not only capitalizes on the strengths of both neural network models but also addresses the training challenges associated with SNNs. The proposed method is evaluated through extensive experiments on two publicly available benchmark ECG datasets. The results show that our model achieves an overall accuracy of 93.8% on the MIT-BIH Arrhythmia dataset and 85.8% on the 2017 PhysioNet Challenge dataset. This advancement underscores the potential of SNNs in the field of medical diagnostics, offering a path towards more accurate, efficient, and less resource-intensive analyses of heart diseases.

Joint A-SNN: Joint training of artificial and spiking neural networks via self-Distillation and weight factorization

Spiking neural networks (SNNs) more closely mimic the human brain than artificial neural networks (ANNs). For SNNs, time-to-first-spike (TTFS) encoding, which represents the output values of neurons based on the timing of a single spike, has been proposed as a promising model to reduce power consumption. Adversarial attacks that can lead ANNs to misrecognize images have been reported in many studies. However, the characteristics of TTFS-based SNNs trained using a backpropagation algorithm against adversarial attacks have not yet been clarified. In particular, the dependence of the robustness against adversarial attacks on spike timings has not been investigated. In this brief, we investigated the robustness of SNNs against adversarial attacks and compared it with that of an ANN. We found that SNNs trained with the appropriate temporal penalty settings are more robust against adversarial images than ANNs.

Neuromorphic computing is a promising technology for future machine learning research, which is envisioned to provide lower power consumption, fault tolerance, and massive parallelism. However, the neuromorphic network, i.e., spike neural network (SNN), suffers from a broader range of its application compared with artificial neural networks (ANN). In this work, we propose a stochastic computation SC-based hybrid ANN and SNN scheme, which supports both ANN and SNN computation in SC domain. The ANN computation is converted to SC, which is implemented with simple logic gates. Moreover, we propose a new two-stage stochastic computation that is compatible with the sequence computation style of SNN. Consequently, the logic resource for the hybrid SNN and ANN can be shared. Compared to the state-of-the-art hybrid scheme, the proposed SC-based ANN and SNN exhibit much lower hardware costs and higher hardware efficiency.

Spiking neural networks (SNNs) are potentially highly efficient models for inference on fully parallel neuromorphic hardware, but existing training methods that convert conventional artificial neural networks (ANNs) into SNNs are unable to exploit these advantages. Although ANN-to-SNN conversion has achieved state-of-the-art accuracy for static image classification tasks, the following subtle but important difference in the way SNNs and ANNs integrate information over time makes the direct application of conversion techniques for sequence processing tasks challenging. Whereas all connections in SNNs have a certain propagation delay larger than zero, ANNs assign different roles to feed-forward connections, which immediately update all neurons within the same time step, and recurrent connections, which have to be rolled out in time and are typically assigned a delay of one time step. Here, we present a novel method to obtain highly accurate SNNs for sequence processing by modifying the ANN training before conversion, such that delays induced by ANN rollouts match the propagation delays in the targeted SNN implementation. Our method builds on the recently introduced framework of streaming rollouts, which aims for fully parallel model execution of ANNs and inherently allows for temporal integration by merging paths of different delays between input and output of the network. The resulting networks achieve state-of-the-art accuracy for multiple event-based benchmark datasets, including N-MNIST, CIFAR10-DVS, N-CARS, and DvsGesture, and through the use of spatio-temporal shortcut connections yield low-latency approximate network responses that improve over time as more of the input sequence is processed. In addition, our converted SNNs are consistently more energy-efficient than their corresponding ANNs.

Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Compared with counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8×/7.4×. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNN research, with a great balance between effectiveness and energy efficiency.

Spiking Self-organizing Maps for Classification Problem

Developmental plasticity plays a prominent role in shaping the brain's structure during ongoing learning in response to dynamically changing environments. However, the existing network compression methods for deep artificial neural networks (ANNs) and spiking neural networks (SNNs) draw little inspiration from brain's developmental plasticity mechanisms, thus limiting their ability to learn efficiently, rapidly, and accurately. This paper proposed a developmental plasticity-inspired adaptive pruning (DPAP) method, with inspiration from the adaptive developmental pruning of dendritic spines, synapses, and neurons according to the "use it or lose it, gradually decay" principle. The proposed DPAP model considers multiple biologically realistic mechanisms (such as dendritic spine dynamic plasticity, activity-dependent neural spiking trace, and local synaptic plasticity), with additional adaptive pruning strategy, so that the network structure can be dynamically optimized during learning without any pre-training and retraining. Extensive comparative experiments show consistent and remarkable performance and speed boost with the extremely compressed networks on a diverse set of benchmark tasks for deep ANNs and SNNs, especially the spatio-temporal joint pruning of SNNs in neuromorphic datasets. This work explores how developmental plasticity enables complex deep networks to gradually evolve into brain-like efficient and compact structures, eventually achieving state-of-the-art (SOTA) performance for biologically realistic SNNs.

Artificial Neural Networks (ANNs) have shown remarkable performance in various fields. However, ANN relies on the von-Neumann architecture, which consumes a lot of power. Hardware-based spiking neural networks (SNNs) inspired by a human brain have become an alternative with significantly low power consumption. In this paper, we propose on-chip trainable SNNs using a time-to-first-spike (TTFS) method. We modify the learning rules of conventional SNNs using TTFS to be suitable for on-chip learning. Vertical NAND flash memory cells fabricated by a device manufacturer are used as synaptic devices. The entire learning process considering the hardware implementation is also demonstrated. The performance of the proposed network is evaluated through the MNIST classification in system-level simulation using Python. The proposed SNNs show an accuracy of 96% for a network size of 784 – 400 – 10. We also investigate the effect of non-ideal cell characteristics (such as pulse-to-pulse and device-to-device variations) on inference accuracy. Our networks demonstrate excellent immunity for various device variations compared with the networks using off-chip training.

Artificial neural network (ANN) research is inspired by how information is dynamically and massively processed in parallel by biological neural networks. Spiking neural networks (SNNs) are versions of artificial neural networks that are more biologically realistic than commonly used static models. As in actual brains, neurons signal each other through current spikes (rather than constant inputs, as in conventional ANNs), and spike timing plays a key role in SNN functioning. In this work, we give an overview of SNNs, and we describe and implement three different mathematical models (integrate and fire, leaky integrate and fire, conductance-based) using the Brian2 software simulator. We also describe the training of SNNs using the spike-timing-dependent plasticity (STDP) algorithm, and discuss an experiment by Diehl and Cook that shows the ability of SNNs to learn to distinguish handwritten digits. Lastly, we describe a few available software SNN simulators and hardware SNN implementations with possible practical applications of SNNs.

Deep neural networks such as convolutional neural networks (CNNs) have achieved a great success in a broad range of fields. Spiking neural networks (SNNs) are designed as a solution for realizing ultra-low power consumption using spikebased neuromorphic hardware. To enable the mapping between conventional artificial neural networks (ANNs) and spike-based neuromorphic hardware, direct conversion from conventional ANNs into SNNs has recently been proposed. However, performance loss is hard to avoid after conversion. To reduce the loss, we analyze the encoding methods of SNNs and the optimization methods for converting. We use the rate coding and approximate the output of an activation function of CNNs by the number of spikes produced by a neuron within a given time window in SNNs. We propose a method for generating fixed uniform spike train whose number of spikes is as exact as we expect, and present an optimization method to reduce the loss by rescaling the threshold of each layer in SNNs. For evaluation, we train three different CNNs with MNIST and SVHN datasets separately and convert all of them into SNNs. The networks achieve a max accuracy of 99.17% on MNIST and of 93.36% on additional training dataset of SVHN. The results show the proposed fixed and uniform spike train not only outperforms the Poisson distributed spike train but also costs less time to generate. Our threshold rescaling method greatly improves the performance of SNNs.

Spiking Neural Networks (SNNs) mimic the behavior of biological neurons. Unlike traditional Artificial Neural Networks (ANNs) that operate in a continuous time domain and use activation functions to process information, SNNs operate discrete event-driven, where data is encoded and communicated through spikes or discrete events. This unique approach offers several advantages, such as efficient computation and lower power consumption, making SNNs particularly attractive for energy-constrained and neuromorphic applications. However, training SNNs poses significant challenges due to the discrete nature of spikes and the non-differentiable behavior they exhibit. As a result, converting pre-trained ANNs into SNNs has gained attention as a convenient approach. While this approach simplifies the training process, it introduces certain drawbacks, including high latency. The conversion of ANNs to SNNs typically leads to a loss of accuracy, which can be attributed to various factors, including quantization, clipping, and timing errors. Previous studies have proposed techniques to mitigate quantization and clipping errors during the conversion process. However, they do not consider timing errors, degrading SNN accuracies at low latency conditions. This work introduces the MiCE conversion method, which offers a comprehensive joint optimization strategy to simultaneously alleviate quantization, clipping, and timing errors. At a moderate latency of 8 time-steps, our converted ResNet-20 achieves classification accuracies of 79.02% and 95.74% on the CIFAR-100 and CIFAR-10 datasets, respectively.

Efficient fusion of spiking neural networks and FET-type gas sensors for a fast and reliable artificial olfactory system