Enhanced Spiking Neural Network with forgetting phenomenon based on electronic synaptic devices

In recent years, scientists and researchers have paid special attention to the implementation of Spiking Neural Networks (SNN), for approaching simulations of the human brain mechanisms, or to solve practical problems, such as epilepsy and seizure detection [1]. Nevertheless, large-scale SNN simulations are expensive from the computational point of view. These models quite often require the usage of parallel computing implementations on several devices[2], such as FPGAs [3], supercomputers [4], and recently on GPUs. Particularly on these last class of devices, a variety of techniques have been posed, like OpenGL pixel shade rendering [5], as well OpenCL and CUDA C/C++ [1], [6]. In this paper, we introduce a configurable large-scale fully parallel implementation that could well model a wide range of SNN using NVIDIA's GPU Common unified device architecture. Because of the achieved scalability, and flexibility in the definition of synaptic connections and neuron population, we show that our implementation shall be easily adapted as a back-end system for a wide variety of purposes. To examine some advantages of this implementation, we pose a comparative between Izhikevich's serialized program and our implementation using a large amount of neurons and synaptic connections. Also, as an additional experimental data, we report execution times, generated spikes and firing rates in a huge amount of neurons and a comparative between different neuron-synapse relations. We also achieved the simulation of 3.8×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">6</sup> synaptic connections in real time (which is 1,000 time steps, one millisecond each, executed in one second). Finally, we demonstrated the increased performance on different hardware environments.

Artificial neural systems for computation were first proposed three quarters of a century ago and the concepts developed by the pioneers still shape the field today. The first generation of neural systems was developed in the nineteen forties in the context of analogue electronics and the theoretical research in logic and mathematics that led to the first digital computers in nineteen forties and fifties. The second generation of neural systems implemented on digital computers was born in the nineteen fifties and great progress was made in the subsequent half century with neural networks being applied to many problems in pattern recognition and machine learning. Through this history there has been an interplay between biologically inspired neural systems and their implementation by engineers on digital machines. This thesis concerns the third generation of neural networks, Spiking Neural Networks, which is making possible the creation of new kinds of brain inspired computing architectures that offer the potential to increase the level of realism and sophistication in terms of autonomous machine behaviour and cognitive computing. This thesis presents the development and demonstration of a new theoretical architecture for third generation neural systems, the Integrate-and-Fire based Spiking Neural Model with extended Neuro-modulated Spike Timing Dependent Plasticity capabilities. This proposed architecture overcomes the limitation of the homosynaptic architecture underlying existing implementations of spiking neural networks that it lacks a natural spike timing dependent plasticity regulation mechanism, and this results in ‘run away’ dynamics. To overcome this ad hoc procedures have been implemented to overcome the ‘run away’ dynamics that emerge from the use of spike timing dependent plasticity among other hebbian-based plasticity rules. The new heterosynaptic architecture presented, explicitly abstracts the modulation of complex biochemical mechanisms into a simplified mechanism that is suitable for the engineering of artificial systems with low computational complexity. Neurons work by receiving input signals from other neurons through synapses. The difference between homosynaptic and heterosynaptic plasticity is that, in the former the change in the properties of a synapse (e.g. synaptic efficacy) depends on the point to point activity in either of the sending and receiving neurons, in contrast for heterosynaptic plasticity the change in the properties of a synapse can be elicited by neurons that are not necessary presynaptic or postsynaptic to the synapse in question. The new architecture is tested by a number of implementations in simulated and real environments. This includes experiments with a simulation environment implemented in Netlogo, and an implementation using Lego Mindstorms as the physical robot platform. These experiments demonstrate the problems with the traditional Spike timing dependent plasticity homosynaptic architecture and how the new heterosynaptic approach can overcome them. It is concluded that the new theoretical architecture provides a natural, theoretically sound, and practical new direction for research into the role of modulatory neural systems applied to spiking neural networks.

Neuron circuits are the fundamental building blocks in the modern neuromorphic system. Designing compact and low‐power neuron circuits can significantly improve the overall area and energy efficiencies of a neuromorphic chip architecture. Here, practical neuron circuits must overcome the variations arising from nonideal behaviors of synaptic devices, such as stuck‐at‐fault and conductance deviation. In this study, a compact leaky integrate‐and‐fire neuron circuit has been designed, with resilience to synaptic device state variations, for hardware implementation of spiking neural networks (SNNs). The proposed neuron circuit is simulated on the 0.35‐ μ m Si complementary metal‐oxide‐semiconductor technology node by a series of circuit simulations based on HSPICE. The proposed circuit occupies a reduced area and exhibits low power consumption (14.7 µ W per spike). Furthermore, the optimized circuit design results in a high degree of tolerance toward input‐current variations arising from conductance‐state variations in the synapse array. Hence, the proposed neuron circuit would be capable of substantially improving the area efficiency and reliability in the realization of the hardware‐oriented SNN architectures.

Neuromorphic computing research is being actively pursued to address the challenges posed by the need for energy-efficient processing of big data. One of the promising approaches to tackle the challenges is the hardware implementation of spiking neural networks (SNNs) with bio-plausible learning rules. Numerous research works have been done to implement the SNN hardware with different synaptic plasticity rules to emulate human brain operations. While a standard spike-timing-dependent-plasticity (STDP) rule is emulated in many SNN hardware, the various STDP rules found in the biological brain have rarely been implemented in hardware. This study proposes a CMOS-memristor hybrid synapse circuit for the hardware implementation of a Ca ion-based plasticity model to emulate the various STDP curves. The memristor was adopted as a memory device in the CMOS synapse circuit because memristors have been identified as promising candidates for analog non-volatile memory devices in terms of energy efficiency and scalability. The circuit design was divided into four sub-blocks based on biological behavior, exploiting the non-volatile and analog state properties of memristors. The circuit was designed to vary weights using an H-bridge circuit structure and PWM modulation. The various STDP curves have been emulated in one CMOS-memristor hybrid circuit, and furthermore a simple neural network operation was demonstrated for associative learning such as Pavlovian conditioning. The proposed circuit is expected to facilitate large-scale operations for neuromorphic computing through its scale-up.

FPGA devices have witnessed popularity in their use for the rapid prototyping of biological Spiking Neural Network (SNNs) applications, as they offer the key requirement of reconfigurability. However, FPGAs do not efficiently realise the biological neuron/synaptic models. Also their routing structures cannot accommodate the high levels of neuron inter-connectivity inherent in complex SNNs. This paper highlights and discusses the current challenges of implementing large scale SNNs on reconfigurable FPGAs. The paper presents a novel Field Programmable Neural Network (FPNN) architecture incorporating low power analogue synapse and a network on chip architecture for SNN routing and configuration. Initial results are presented.

Spiking Neural Networks (SNN) is one special implementation of Artificial Neural Networks (ANN), where the input signals are encoded in the temporal relationship between consecutive spikes (spike trains) instead of real-numbered values. Nevertheless, SNN is believed to be a closer representation of the biological neural system, because it imitates the current spikes that are transmitted between neurons in real biological systems. Practical and simplified SNN models include the Leaky Integrate-and-Fire (LIF) function of the neurons and the Spiking Timing Dependent Plasticity (STDP) function of the synapses. While most ANN architectures can be implemented with digital logic gates, SNN is more suitable to be implemented in analog circuits. In this paper, we propose revised analog circuit implementations of SNN neurons with LIF and STDP functions. Compared with previous works by other researchers, our proposed analog designs use fewer components and can be cascaded to form a complete neural system. The circuits are designed and simulated with SMIC 55nm CMOS LP process. The simulated results demonstrate that the analog neural system can work under a very small current (less than 10 µA) and voltage supply (1.0 Volts), and consumes less power consumption than digital implementations.

Spike-timing dependent plasticity (STDP)-based spiking neural network (SNN) is a promising choice to realize unsupervised intelligent systems with a limited power budget. In addition to STDP, another two bio-inspired mechanisms of lateral inhibition and homeostasis are always implemented in the unsupervised training procedure of STDP-based SNNs. However, the existing methods to achieve lateral inhibition necessitate a great number of connections that are proportional to the square of the number of learning neurons, and the existing hardware solution of homeostasis demands complex circuits for each learning neuron, both of which challenge the hardware implementation of STDP-based SNNs. In this brief, we propose a novel SNN using memristor-based inhibitory synapses to realize the mechanisms of lateral inhibition and homeostasis with low hardware complexity. The proposed SNN can improve the network scalability by reducing the connection number for lateral inhibition from $N^{2}$ to $N$ and reduce the hardware overhead by leveraging the circuit of lateral inhibition to achieve homeostasis. Software simulations on the recognition task on MNIST dataset show that the proposed SNN achieves a ~ 2 times higher learning efficiency with comparable accuracy. In addition, the challenging properties of realistic memristor devices, including limited number of resistive states, intrinsic parameter variation, and permanent open device, are added in the simulation to evaluate the robustness of our proposed approach.

Spiking neural networks (SNNs), inspired by the neuronal network in the brain, provide biologically relevant and low-power consuming models for information processing. Existing studies either mimic the learning mechanism of brain neural networks as closely as possible, for example, the temporally local learning rule of spike-timing-dependent plasticity (STDP), or apply the gradient descent rule to optimize a multilayer SNN with fixed structure. However, the learning rule used in the former is local and how the real brain might do the global-scale credit assignment is still not clear, which means that those shallow SNNs are robust but deep SNNs are difficult to be trained globally and could not work so well. For the latter, the nondifferentiable problem caused by the discrete spike trains leads to inaccuracy in gradient computing and difficulties in effective deep SNNs. Hence, a hybrid solution is interesting to combine shallow SNNs with an appropriate machine learning (ML) technique not requiring the gradient computing, which is able to provide both energy-saving and high-performance advantages. In this article, we propose a HybridSNN, a deep and strong SNN composed of multiple simple SNNs, in which data-driven greedy optimization is used to build powerful classifiers, avoiding the derivative problem in gradient descent. During the training process, the output features (spikes) of selected weak classifiers are fed back to the pool for the subsequent weak SNN training and selection. This guarantees HybridSNN not only represents the linear combination of simple SNNs, as what regular AdaBoost algorithm generates, but also contains neuron connection information, thus closely resembling the neural networks of a brain. HybridSNN has the benefits of both low power consumption in weak units and overall data-driven optimizing strength. The network structure in HybridSNN is learned from training samples, which is more flexible and effective compared with existing fixed multilayer SNNs. Moreover, the topological tree of HybridSNN resembles the neural system in the brain, where pyramidal neurons receive thousands of synaptic input signals through their dendrites. Experimental results show that the proposed HybridSNN is highly competitive among the state-of-the-art SNNs.

Memristive technologies are attractive due to their non-volatility, high-density, low-power and compatibility with CMOS. For memristive devices, a model corresponding to practical behavioral characteristics is highly favorable for the realization of its neuromorphic system and applications. This paper presents a novel flexible memristor model with electronic resistive switching memory behavior. Firstly, the Ag-Au / MoSe2-doped Se / Au-Ag memristor is prepared using hydrothermal synthesis method and magnetron sputtering method, and its performance test is conducted on an electrochemical workstation. Then, the mathematical model and SPICE circuit model of the Ag-Au / MoSe2-doped Se / Au-Ag memristor are constructed. The model accuracy is verified by using the electrochemical data derived from the performance test. Furthermore, the proposed model is applied to the circuit implementation of spiking neural network with biological mechanism. Finally, computer simulations and analysis are carried out to verify the validity and effectiveness of the entire scheme.

The outbreak of a global pandemic called coronavirus has created unprecedented circumstances resulting into a large number of deaths and risk of community spreading throughout the world. Desperate times have called for desperate measures to detect the disease at an early stage via various medically proven methods like chest computed tomography (CT) scan, chest X-Ray, etc., in order to prevent the virus from spreading across the community. Developing deep learning models for analysing these kinds of radiological images is a well-known methodology in the domain of computer based medical image analysis. However, doing the same by mimicking the biological models and leveraging the newly developed neuromorphic computing chips might be more economical. These chips have been shown to be more powerful and are more efficient than conventional central and graphics processing units. Additionally, these chips facilitate the implementation of spiking neural networks (SNNs) in real-world scenarios. To this end, in this work, we have tried to simulate the SNNs using various deep learning libraries. We have applied them for the classification of chest CT scan images into COVID and non-COVID classes. Our approach has achieved very high F1 score of 0.99 for the potential-based model and outperforms many state-of-the-art models. The working code associated with our present work can be found here.

Spiking neural networks (SNNs) have emerged as a hardware efficient architecture for classification tasks. The challenge of spike-based encoding has been the lack of a universal training mechanism performed entirely using spikes. There have been several attempts to adopt the powerful backpropagation (BP) technique used in non-spiking artificial neural networks (ANNs): (1) SNNs can be trained by externally computed numerical gradients. (2) A major advancement towards native spike-based learning has been the use of approximate BP using spike-time dependent plasticity with phased forward/backward passes. However, the transfer of information between such phases for gradient and weight update calculation necessitates external memory and computational access. This is a challenge for standard neuromorphic hardware implementations. In this paper, we propose a stochastic SNN based back-prop (SSNN-BP) algorithm that utilizes a composite neuron to simultaneously compute the forward pass activations and backward pass gradients explicitly with spikes. Although signed gradient values are a challenge for spike-based representation, we tackle this by splitting the gradient signal into positive and negative streams. The composite neuron encodes information in the form of stochastic spike-trains and converts BP weight updates into temporally and spatially local spike coincidence updates compatible with hardware-friendly resistive processing units. Furthermore, we characterize the quantization effect of discrete spike-based weight update to show that our method approaches BP ANN baseline with sufficiently long spike-trains. Finally, we show that the well-performing softmax cross-entropy loss function can be implemented through inhibitory lateral connections enforcing a winner take all rule. Our SNN with a two-layer network shows excellent generalization through comparable performance to ANNs with equivalent architecture and regularization parameters on static image datasets like MNIST, Fashion-MNIST, Extended MNIST, and temporally encoded image datasets like Neuromorphic MNIST datasets. Thus, SSNN-BP enables BP compatible with purely spike-based neuromorphic hardware.

In this paper, we present an hardware implementation of spiking neural networks based on analog integrated circuits. These ICs compute in real-time biologically realistic cortical neuron models. Each integrated circuit includes five neurons and analog memory cells to set and store the conductance model parameters. The system allows switching on-line the model of cortical neuron. Circuits are embedded in a multi-board system all connected to a backplane with daisy-chain facilities. Each action potential computed by analog neuromimetic chips is time-stamped when detected by digital device (FPGA). These FPGAs are also in charge of the real-time plasticity computation and of controlling inter-boards communication. The implemented neural plasticity is also biological relevant thanks to its time dependent computation. The whole system is designed to compute programmable models and connectivity schemes in biological real-time. It will allow extending the hybrid technique (connection between biological and artificial neurons) to Micro Electrode Array.

Spiking neural networks (SNNs) represent a promising frontier in artificial intelligence (AI), offering event-driven, energy-efficient computation that mimics rich neural dynamics in the brain. However, running large-scale SNNs on mainstream computing hardware faces significant challenges to efficiently emulate these dynamical processes using synchronized and logical chips. Memristor based systems have recently demonstrated great potential for AI acceleration, sparking speculations and explorations of using these emerging devices for SNN tasks. This paper reviews the promises and challenges of memristive devices in SNN implementations, and our discussions are focused on the scaling and integration of neuronal and synaptic devices. We survey recent progress in device and circuit development, discuss possible pathways for chip-level integration, and finally probe into hardware-oriented algorithm designs. This review offers a system-level perspective on implementing scalable memristor based SNN platforms.

A forecast-based STDP rule suitable for neuromorphic implementation

Spiking Neural Networks (SNNs), widely known as the third generation of neural networks, encode input information temporally using sparse spiking events, which can be harnessed to achieve higher computational efficiency for cognitive tasks. However, considering the rapid strides in accuracy enabled by state-of-the-art Analog Neural Networks (ANNs), SNN training algorithms are much less mature, leading to accuracy gap between SNNs and ANNs. In this paper, we propose different SNN training methodologies, varying in degrees of biofidelity, and evaluate their efficacy on complex image recognition datasets. First, we present biologically plausible Spike Timing Dependent Plasticity (STDP) based deterministic and stochastic algorithms for unsupervised representation learning in SNNs. Our analysis on the CIFAR-10 dataset indicates that STDP-based learning rules enable the convolutional layers to self-learn low-level input features using fewer training examples. However, STDP-based learning is limited in applicability to shallow SNNs (≤4 layers) while yielding considerably lower than state-of-the-art accuracy. In order to scale the SNNs deeper and improve the accuracy further, we propose conversion methodology to map off-the-shelf trained ANN to SNN for energy-efficient inference. We demonstrate 69.96% accuracy for VGG16-SNN on ImageNet. However, ANN-to-SNN conversion leads to high inference latency for achieving the best accuracy. In order to minimize the inference latency, we propose spike-based error backpropagation algorithm using differentiable approximation for the spiking neuron. Our preliminary experiments on CIFAR-10 show that spike-based error backpropagation effectively captures temporal statistics to reduce the inference latency by up to 8× compared to converted SNNs while yielding comparable accuracy