Alleviating the Communication Bottleneck in Neuromorphic Computing with Custom-Designed Spiking Neural Networks

The ultra-wideband, frequency agile and high duty cycle features of Low Probability of Intercept (LPI) radar pose serious concerns for passive radar detection systems that must efficiently identify LPI radar modulation schemes with very low latency and high accuracy. For these tasks, conventional deep learning techniques have often been employed, though this comes at the cost of increased model complexity and high energy consumption when implemented in tactically relevant scenarios with significant channel deterioration and size, weight, and power (SWaP) constrained platforms. In this paper, we propose a neuromorphic approach based on energy-efficient and robust Spiking Neural Networks (SNNs). To demonstrate the effectiveness of our proposed approach, we perform a challenging LPI radar waveform recognition task with a Spiking Convolutional Neural Network (SCNN). Simulation results show that the SCNN achieves a 13-waveform recognition accuracy of 99.1% at OdB SNR with 1.51µJ of total energy per inference. It is found that our system outperforms non-spiking models with a reduction in model size and three orders of magnitude lower energy consumption. To the best of our knowledge, this work presents the first spike-based implementation of an LPI radar waveform recognition system. Moreover, this paper highlights the potential for extremely low SWaP neuromorphic hardware deployment to enable intelligent electronic support systems that demand high throughput efficiency and ultra-low power.

Neurons in the brain form thousands of connections, or synapses, with one another, allowing signals to pass from one cell to the next. To activate a neuron, a high enough activating signal or ‘action potential’ must be reached. However, the accepted view of signal transmission assumes that the great majority of synapses are too weak to activate neurons. This means that often simultaneous inputs from many neurons are required to trigger a single neuron’s activation. However, such coordination is likely unreliable as neurons can react differently to the same stimulus depending on the circumstances. An alternative way of transmitting signals has been reported in turtle brains, where impulses from a single neuron can trigger activity across a network of connections. Furthermore, these responses are reliably repeatable, activating the same neurons in the same order. Riquelme et al. set out to understand the mechanism that underlies this type of neuron activation using a mathematical model based on data from the turtle brain. These data showed that the neural network in the turtle’s brain had many weak synapses but also a few, rare, strong synapses. Simulating this neural network showed that those rare, strong synapses promote the signal’s reliability by providing a consistent route for the signal to travel through the network. The numerous weak synapses, on the other hand, have a regulatory role in providing flexibility to how the activation spreads. This combination of strong and weak connections produces a system that can reliably promote or stop the signal flow depending on the context. Riquelme et al.’s work describes a potential mechanism for how signals might travel reliably through neural networks in the brain, based on data from turtles. Experimental work will need to address whether strong connections play a similar role in other animal species, including humans. In the future, these results may be used as the basis to design new systems for artificial intelligence, building on the success of neural networks.

Neuromorphic computing is a promising technology for future machine learning research, which is envisioned to provide lower power consumption, fault tolerance, and massive parallelism. However, the neuromorphic network, i.e., spike neural network (SNN), suffers from a broader range of its application compared with artificial neural networks (ANN). In this work, we propose a stochastic computation SC-based hybrid ANN and SNN scheme, which supports both ANN and SNN computation in SC domain. The ANN computation is converted to SC, which is implemented with simple logic gates. Moreover, we propose a new two-stage stochastic computation that is compatible with the sequence computation style of SNN. Consequently, the logic resource for the hybrid SNN and ANN can be shared. Compared to the state-of-the-art hybrid scheme, the proposed SC-based ANN and SNN exhibit much lower hardware costs and higher hardware efficiency.

The application of spiking neural networks (SNNs) for processing visual and auditory data necessitate the conversion of traditional neural network datasets into a format suitable for spike-based computations. Existing datasets designed for conventional neural networks are incompatible with SNNs due to their reliance on spike timing and specific preprocessing requirements. This paper introduces a comprehensive pipeline that enables the conversion of common datasets into rate-coded spikes, meeting processing demands of SNNs. The proposed solution is evaluated on Spike-CNN trained on Time-to-First-Spike encoded MNIST and compared with the similar system trained on the neuromorphic dataset (N-MNIST). Both systems have comparative precision; however the proposed solution is more energy efficient than the system based on neuromorphic computing. Since, the proposed solution is not limited to any specific data form and can be applied to various types of audio/visual content. By providing a means to adapt existing datasets, this research facilitates the exploration and advancement of SNNs across different domains.

Spiking Neural Networks (SNNs) are brain-inspired computational networks that promise an efficient solution for real-life applications such as audio processing and pattern recognition. An SNN is a complex network of neurons interconnected with synapses, whose spike times and synaptic strength is essential for information processing. SNNs consume low power and perform computations in parallel, making them attractive for hardware implementation of applications in neuromorphic engineering. For hardware implementation of SNNs, we need devices that mimic the behavior of neurons and synapses as well as their computational models to build and characterize large-scale SNNs. Earlier, a Pr <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0.7</inf> Ca <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0.3</inf> MnO <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</inf> (PCMO) material based Resistive-RAM (RRAM) has been used to experimentally demonstrate an Integrate and Fire (IF) Neuron and Izhikevich Neuron. Here, we present an experimentally validated Verilog-A model of PCMO RRAM, which behaves like a neuron. The model captures the conductance change of the RRAM for different applied voltages and can mimic an IF Neuron and an Izhikevich Neuron. The model enables the design of large-scale SNNs and studies their behavior in a simulation domain.

Spiking neural networks (SNNs) have attracted many researchers’ interests due to its biological plausibility and event-driven characteristic. In particular, recently, many studies on high-performance SNNs comparable to the conventional analog-valued neural networks (ANNs) have been reported by converting weights trained from ANNs into SNNs. However, unlike ANNs, SNNs have an inherent latency that is required to reach the best performance because of differences in operations of neuron. In SNNs, not only spatial integration but also temporal integration exists, and the information is encoded by spike trains rather than values in ANNs. Therefore, it takes time to achieve a steady-state of the performance in SNNs. The latency is worse in deep networks and required to be reduced for the practical applications. In this work, we propose a pre-charged membrane potential (PCMP) for the latency reduction in SNN. A variety of neural network applications (e.g., classification, autoencoder using MNIST and CIFAR-10 datasets) are trained and converted to SNNs to demonstrate the effect of the proposed approach. The latency of SNNs is successfully reduced without accuracy loss. In addition, we propose a delayed evaluation method (DE), by which the errors during the initial transient are discarded. The error spikes occurring in the initial transient is removed by DE, resulting in the further latency reduction. DE can be used in combination with PCMP for further latency reduction. Finally, we also show the advantages of the proposed methods in improving the number of spikes required to reach a steady-state of the performance in SNNs for energy-efficient computing.

Deep neural networks such as convolutional neural networks (CNNs) have achieved a great success in a broad range of fields. Spiking neural networks (SNNs) are designed as a solution for realizing ultra-low power consumption using spikebased neuromorphic hardware. To enable the mapping between conventional artificial neural networks (ANNs) and spike-based neuromorphic hardware, direct conversion from conventional ANNs into SNNs has recently been proposed. However, performance loss is hard to avoid after conversion. To reduce the loss, we analyze the encoding methods of SNNs and the optimization methods for converting. We use the rate coding and approximate the output of an activation function of CNNs by the number of spikes produced by a neuron within a given time window in SNNs. We propose a method for generating fixed uniform spike train whose number of spikes is as exact as we expect, and present an optimization method to reduce the loss by rescaling the threshold of each layer in SNNs. For evaluation, we train three different CNNs with MNIST and SVHN datasets separately and convert all of them into SNNs. The networks achieve a max accuracy of 99.17% on MNIST and of 93.36% on additional training dataset of SVHN. The results show the proposed fixed and uniform spike train not only outperforms the Poisson distributed spike train but also costs less time to generate. Our threshold rescaling method greatly improves the performance of SNNs.

Efficiently solving combinatorial optimization problems (COPs) such as Max-Cut is challenging because the resources required increase exponentially with the problem size. This study proposes a hardware-friendly method for solving the Max-Cut problem by implementing a spiking neural network (SNN)-based Boltzmann machine (BM) in neuromorphic hardware systems. To implement the hardware-oriented version of the spiking Boltzmann machine (sBM), the stochastic dynamics of leaky integrate-and-fire (LIF) neurons with random walk noise are analyzed, and an innovative algorithm based on overlapping time windows is proposed. The simulation results demonstrate the effective convergence and high accuracy of the proposed method for large-scale Max-Cut problems. The proposed method is validated through successful hardware implementation on a 6-transistor/2-resistor (6T2R) neuromorphic chip with phase change memory (PCM) synapses. In addition, as an expansion of the algorithm, several annealing techniques and bias split methods are proposed to improve convergence, along with circuit design ideas for efficient evaluation of sampling convergence using cell arrays and spiking systems. Overall, the results of the proposed methods demonstrate the potential of energy-efficient and hardware-implementable approaches using SNNs to solve COPs. To the best of the author's knowledge, this is the first study to solve the Max-Cut problem using an SNN neuromorphic hardware chip.

With recent advances in learning algorithms, recurrent networks of spiking neurons are achieving performance that is competitive with vanilla recurrent neural networks. However, these algorithms are limited to small networks of simple spiking neurons and modest-length temporal sequences, as they impose high memory requirements, have difficulty training complex neuron models and are incompatible with online learning. Here, we show how the recently developed Forward-Propagation Through Time (FPTT) learning combined with novel liquid time-constant spiking neurons resolves these limitations. Applying FPTT to networks of such complex spiking neurons, we demonstrate online learning of exceedingly long sequences while outperforming current online methods and approaching or outperforming offline methods on temporal classification tasks. The efficiency and robustness of FPTT enable us to directly train a deep and performant spiking neural network for joint object localization and recognition, demonstrating the ability to train large-scale dynamic and complex spiking neural network architectures. Memory efficient online training of recurrent spiking neural networks without compromising accuracy is an open challenge in neuromorphic computing. Yin and colleagues demonstrate that training a recurrent neural network consisting of so-called liquid time-constant spiking neurons using an algorithm called Forward-Propagation Through Time allows for online learning and state-of-the-art performance at a reduced computational cost compared with existing approaches.

In recent years, spiking neural networks (SNNs), which originated from the theoretical basis of neuroscience, have attracted neuromorphic computing and brain-like computing due to their advantages, such as neural dynamics and coding mechanism, which are similar to biological neurons. SNNs have become one of the mainstream frameworks in the field of brain-like computing. However, most of the Leaky Integrate-and-Fire (LIF) neuron models currently used by SNNs based on direct training of backpropagation (BP) do not consider the changes in the recurrent connections and the dynamic strength of neuron connections over time. This study presented the LIF neuron model with recurrent connections and a method for dynamically changing the presynaptic currents. Recurrent LIF neurons have an additional cyclic connection compared with classic LIF neurons. Their postsynaptic current stimulates a change in membrane potential at the next time point. Their dynamics were more similar to the activities of biological neurons. We also proposed an efficient and flexible BP training method for recurrent LIF neurons. On the basis of the above methods, we proposed the recurrent SNN with dynamic presynaptic currents based on backpropagation (RDS-BP). We test the proposed RDS-BP on three image data sets (MNIST, Fashion-MNIST and CIFAR-10) and two text data sets (IMDB and TREC). The results showed that the performance of RDS-BP not only exceeded the naive SNN models based on BP but also exceeded the SNN methods proposed in previous studies in recent years, which had excellent performance in previous experiments. Our work provides a new LIF neuron model with a recurrent connection and dynamic presynaptic current and a BP training arrangement for the proposed neuron, which could merit developments with neuromorphic and brain-like computing.

In spiking neural networks, the information is conveyed by the spike times, that depend on the intrinsic dynamics of each neuron, the input they receive and on the connections between neurons. In this article we study the Markovian nature of the sequence of spike times in stochastic neural networks, and in particular the ability to deduce from a spike train the next spike time, and therefore produce a description of the network activity only based on the spike times regardless of the membrane potential process. To study this question in a rigorous manner, we introduce and study an event-based description of networks of noisy integrate-and-fire neurons, i.e. that is based on the computation of the spike times. We show that the firing times of the neurons in the networks constitute a Markov chain, whose transition probability is related to the probability distribution of the interspike interval of the neurons in the network. In the cases where the Markovian model can be developed, the transition probability is explicitly derived in such classical cases of neural networks as the linear integrate-and-fire neuron models with excitatory and inhibitory interactions, for different types of synapses, possibly featuring noisy synaptic integration, transmission delays and absolute and relative refractory period. This covers most of the cases that have been investigated in the event-based description of spiking deterministic neural networks.

Spiking neural networks (SNNs), inspired by the neuronal network in the brain, provide biologically relevant and low-power consuming models for information processing. Existing studies either mimic the learning mechanism of brain neural networks as closely as possible, for example, the temporally local learning rule of spike-timing-dependent plasticity (STDP), or apply the gradient descent rule to optimize a multilayer SNN with fixed structure. However, the learning rule used in the former is local and how the real brain might do the global-scale credit assignment is still not clear, which means that those shallow SNNs are robust but deep SNNs are difficult to be trained globally and could not work so well. For the latter, the nondifferentiable problem caused by the discrete spike trains leads to inaccuracy in gradient computing and difficulties in effective deep SNNs. Hence, a hybrid solution is interesting to combine shallow SNNs with an appropriate machine learning (ML) technique not requiring the gradient computing, which is able to provide both energy-saving and high-performance advantages. In this article, we propose a HybridSNN, a deep and strong SNN composed of multiple simple SNNs, in which data-driven greedy optimization is used to build powerful classifiers, avoiding the derivative problem in gradient descent. During the training process, the output features (spikes) of selected weak classifiers are fed back to the pool for the subsequent weak SNN training and selection. This guarantees HybridSNN not only represents the linear combination of simple SNNs, as what regular AdaBoost algorithm generates, but also contains neuron connection information, thus closely resembling the neural networks of a brain. HybridSNN has the benefits of both low power consumption in weak units and overall data-driven optimizing strength. The network structure in HybridSNN is learned from training samples, which is more flexible and effective compared with existing fixed multilayer SNNs. Moreover, the topological tree of HybridSNN resembles the neural system in the brain, where pyramidal neurons receive thousands of synaptic input signals through their dendrites. Experimental results show that the proposed HybridSNN is highly competitive among the state-of-the-art SNNs.

Neuromorphic computers emulate the human brain while being extremely power efficient for computing tasks. In fact, they are poised to be critical for energy-efficient computing in the future. Neuromorphic computers are primarily used in spiking neural network–based machine learning applications. However, they are known to be Turing-complete, and in theory can perform all general-purpose computation. One of the biggest bottlenecks in realizing general-purpose computations on neuromorphic computers today is the inability to efficiently encode data on the neuromorphic computers. To fully realize the potential of neuromorphic computers for energy-efficient general-purpose computing, efficient mechanisms must be devised for encoding numbers. Current encoding mechanisms (e.g., binning, rate-based encoding, and time-based encoding) have limited applicability and are not suited for general-purpose computation. In this paper, we present the virtual neuron abstraction as a mechanism for encoding and adding integers and rational numbers by using spiking neural network primitives. We evaluate the performance of the virtual neuron on physical and simulated neuromorphic hardware. We estimate that the virtual neuron could perform an addition operation using just 23 nJ of energy on average with a mixed-signal, memristor-based neuromorphic processor. We also demonstrate the utility of the virtual neuron by using it in some of the μ-recursive functions, which are the building blocks of general-purpose computation.

Comparison of spiking neural networks with different topologies based on anti-disturbance ability under external noise

Spiking Neural Networks (SNN) is one special implementation of Artificial Neural Networks (ANN), where the input signals are encoded in the temporal relationship between consecutive spikes (spike trains) instead of real-numbered values. Nevertheless, SNN is believed to be a closer representation of the biological neural system, because it imitates the current spikes that are transmitted between neurons in real biological systems. Practical and simplified SNN models include the Leaky Integrate-and-Fire (LIF) function of the neurons and the Spiking Timing Dependent Plasticity (STDP) function of the synapses. While most ANN architectures can be implemented with digital logic gates, SNN is more suitable to be implemented in analog circuits. In this paper, we propose revised analog circuit implementations of SNN neurons with LIF and STDP functions. Compared with previous works by other researchers, our proposed analog designs use fewer components and can be cascaded to form a complete neural system. The circuits are designed and simulated with SMIC 55nm CMOS LP process. The simulated results demonstrate that the analog neural system can work under a very small current (less than 10 µA) and voltage supply (1.0 Volts), and consumes less power consumption than digital implementations.