Exploring the link between temporal difference learning and spike-timing-dependent plasticity

To understand the mechanism of information processing by a biological neural network, computer simulation of a large-scale spiking neural network is an important method. However, because of a high computation cost of the simulation of a large-scale spiking neural network, the simulation requires high performance computing implemented by a supercomputer or a computer cluster. Recently, hardware for parallel computing such as a multi-core CPU and a graphics card with a graphics processing unit (GPU) is built in a gaming computer and a workstation. Thus, parallel computing using this hardware is becoming widespread, allowing us to obtain powerful computing power for simulation of a large-scale spiking neural network. However, it is not clear how much increased performance the parallel computing method using a new GPU yields in the simulation of a large-scale spiking neural network. In this study, we compared computation time between the computing methods with CPUs and GPUs in a simulation of neuronal models. We developed computer programs of neuronal simulations for the computing systems that consist of a gaming graphics card with new architecture (the NVIDIA GTX 1080) and an accelerator board using a GPU (the NVIDIA Tesla K20C). Our results show that the computing systems can perform a simulation of a large number of neurons faster than CPU-based systems. Furthermore, we investigated the accuracy of a simulation using single precision floating point. We show that the simulation results of single precision were accurate enough compared with those of double precision, but chaotic neuronal response calculated by a GPU using single precision is prominently different from that calculated by a CPU using double precision. Furthermore, the difference in chaotic dynamics appeared even if we used double precision of a GPU. In conclusion, the GPU-based computing system exhibits a higher computing performance than the CPU-based system, even if the GPU system includes data transfer from a graphics card to host memory.

A workstation has been implemented for development of Brain Machine Interfaces (BMIs) operating with spike-based models of biological neurons. The workstation has been designed to provide real-time interactions between biological and simulated spiking neurons during in vivo BMI experiments. Parallel execution of online spike sorting, real-time spiking neural network (SNN) simulation and experimental environment controlling tasks are supported by the system to fulfill the requirements of closed-loop BMI experiments. The modular software architecture of the system enables easy replacement of SNN simulation and other computation modules required in BMI studies. In this paper, we describe the software structure and computational infrastructure of the workstation, and illustrate the results of a performance test run on the system with an implemented real-time SNN simulator.

Spike timing dependent plasticity (STDP) is a synaptic learning rule where the relative timing between the presynaptic and postsynaptic action potentials determines the sign and strength of synaptic plasticity. In its basic form STDP has an asymmetric form which incorporates both persistent increases and persistent decreases in synaptic strength. The basic form of STDP, however, is not a fixed property and depends on the dendritic location. An asymmetric curve is observed in the distal dendrites, whereas a symmetrical one is observed in the proximal ones. A recent computational study has shown that the transition from the asymmetry to symmetry is due to inhibition under certain conditions. Synapses have also been observed to be unreliable at generating plasticity when excitatory postsynaptic potentials and single spikes are paired at low frequencies. Bursts of spikes, however, are reliably signaled because transmitter release is facilitated. This article presents a two-compartment model of the CA1 pyramidal cell. The model is neurophysiologically plausible with its dynamics resulting from the interplay of many ionic and synaptic currents. Plasticity is measured by a deterministic Ca(2+) dynamics model which measures the instantaneous calcium level and its time course in the dendrite and change the strength of the synapse accordingly. The model is validated to match the asymmetrical form of STDP from the pairing of a presynaptic (dendritic) and postsynaptic (somatic) spikes as observed experimentally. With the parameter set unchanged the model investigates how pairing of bursts with single spikes and bursts in the presence or absence of inhibition shapes the STDP curve. The model predicts that inhibition strength and frequency are not the only factors of the asymmetry-to-symmetry switch of the STDP curve. Burst interspike interval is another factor. This study is an important first step towards understanding how STDP is affected under natural firing patterns in vivo.

Substantial evidence indicates that the time structure of neuronal spike trains is relevant in neuronal signal processing. Bio-inspired spiking neural networks are taking these results into account. Applications of these networks to low vision problems, e.g. segmentation, requires that the simulation of large-scale networks must be performed in a reasonable time. On this basis, we investigated the achievable performance of existing hardware platforms for the simulation of spiking neural networks with sizes from 8k neurons up to 512k neurons/50M synapses. We present results for workstations (Sparc-Ultra), digital signal processors (TMS-C8x), neurocomputers (CNAPS, SYNAPSE), small- and large-scale parallel-computers (4xPentium, CM-2, SP2) and discuss the specific implementation issues. According to our investigation, only supercomputers like CM-2 can match the performance requirements for the simulation of very large-scale spiking neural networks. Therefore, there is still. the need for low-cost hardware accelerators.

Brain functions such as learning and memory rely on synaptic plasticity. Many studies have shown that synaptic plasticity can be driven by the timings between pre- and postsynaptic spikes, also known as spike-timing-dependent plasticity (STDP). In most of the modeling studies exploring STDP functions, presynaptic spikes have been postulated to be Poisson (random) spikes and postsynaptic neurons have been described using an integrate-and-fire model, for simplicity. However, experimental data suggest this is not necessarily true. In this study, we investigated how STDP worked in synaptic competition if more neurophysiologically realistic properties for pre- and postsynaptic dynamics were incorporated; presynaptic (input) spikes obeyed a gamma process and the postsynaptic neuron was a multi-timescale adaptive threshold model. Our results showed that STDP strengthened specific combinations of pre- and postsynaptic properties; regular spiking neurons favored regular input spikes whereas random spiking neurons did random ones, suggesting neural information coding utilizes both the properties.

Spike timing dependent plasticity (STDP) learning rule is one of hot topics in neurobiology since it's been widely believed that synaptic plasticity mainly contribute to learning and memory in brain. Up to now, STDP has been observed in a wide variety of areas of brain, hippocampus, cortex and so on. Competition among synapses is an important behavior for this learning rule. In present study, we propose a single layer spiking neural network model using STDP learning rule in inhibitory synapses to investigate the competitive behavior. The experiments show that the synapses among neurons are both strengthened on the whole training process. Thus neurons inhibit the activities of one another, eventually the neuron with the highest input spike rate win the competition. We have found that the behavior is efficient when the differences of firing rates of input neurons without STDP are great than 5Hz, otherwise the winner neuron is random. In order to use the principle to artificial intelligent system, we use a mechanism of dynamic learning rate to let the neuron with the highest input to be selected by the competitive behavior as the winner. Therefore, a robust competitive spiking neural network is obtained.

Neural coupled oscillators are a useful building block in numerous models and applications. They were analyzed extensively in theoretical studies and more recently in biologically realistic simulations of spiking neural networks. The advent of mixed-signal analog/digital neuromorphic electronic circuits provides new means for implementing neural coupled oscillators on compact, low-power, spiking neural network hardware platforms. However, their implementation on this noisy, low-precision and inhomogeneous computing substrate raises new challenges with regards to stability and controllability. In this work, we present a robust, spiking neural network model of neural coupled oscillators and validate it with an implementation on a mixed-signal neuromorphic processor. We demonstrate its robustness showing how to reliably control and modulate the oscillator's frequency and phase shift, despite the variability of the silicon synapse and neuron properties. We show how this ultra-low power neural processing system can be used to build an adaptive cardiac pacemaker modulating the heart rate with respect to the respiration phases and compare it with surface ECG and respiratory signal recordings from dogs at rest. The implementation of our model in neuromorphic electronic hardware shows its robustness on a highly variable substrate and extends the toolbox for applications requiring rhythmic outputs such as pacemakers.

Several analog and digital brain-inspired electronic systems have been recently proposed as dedicated solutions for fast simulations of spiking neural networks. While these architectures are useful for exploring the computational properties of large-scale models of the nervous system, the challenge of building low-power compact physical artifacts that can behave intelligently in the real-world and exhibit cognitive abilities still remains open. In this paper we propose a set of neuromorphic engineering solutions to address this challenge. In particular, we review neuromorphic circuits for emulating neural and synaptic dynamics in real-time and discuss the role of biophysically realistic temporal dynamics in hardware neural processing architectures; we review the challenges of realizing spike-based plasticity mechanisms in real physical systems and present examples of analog electronic circuits that implement them; we describe the computational properties of recurrent neural networks and show how neuromorphic Winner-Take-All circuits can implement working-memory and decision-making mechanisms. We validate the neuromorphic approach proposed with experimental results obtained from our own circuits and systems, and argue how the circuits and networks presented in this work represent a useful set of components for efficiently and elegantly implementing neuromorphic cognition.

Spike-timing dependent plasticity (STDP), a synaptic modification depending on a relative timing of presynaptic and postsynaptic spikes, has fascinated researchers in the fields of neurophysiology and computational neuroscience, because it is not only conceptually simple or biologically reasonable but is also versatile in neural network simulations. The STDP rule may be valid only under specific conditions, however. We propose herein a method that could find more natural and potent rules of synaptic plasticity.

A major challenge in computational neuroscience is to achieve high performance for real-time simulations of full size brain networks. Recent advances in GPU technology provide massively parallel, low-cost and efficient hardware that is widely available on the computer market. However, the comparatively low-level programming that is necessary to create an efficient GPU-compatible implementation of neuronal network simulations can be challenging, even for otherwise experienced programmers. To resolve this problem a number of tools for simulating spiking neural networks (SNN) on GPUs have been developed [1,2], but using a particular simulator usually comes with restrictions to particular supported neuron models, synapse models or connectivity schemes. Besides being inconvenient, this can unduly influence the path of scientific enquiry. Here we present GeNN (GPU enhance neuronal networks), which builds on NVIDIA's common unified device architecture (CUDA) to enable a more flexible framework. CUDA allows programmers to write C-like code and execute it on NVIDIA’s massively parallel GPUs. However, in order to achieve good performance, it is critical but not trivial to make the right choices on how to parallelize a computational problem, organize its data in memory and optimize the memory access patterns. GeNN is based on the idea that much of this optimization can be cast into heuristics that allow the GeNN meta-compiler to generate optimized GPU code from a basic description of the neuronal network model in a minimal domain specific language of C function calls. For further simplification, this description may also be obtained by translating variables, dynamical equations and parameters from an external simulator into GeNN input files. We are developing this approach for the Brian 2 [3] and SpineCreator/SpineML [4] systems. Using a code generation approach in GeNN has important advantages: 1. A large number of different neuron and synapse models can be provided without performance losses in the final simulation code. 2. The generated simulator code can be optimized for the available GPU hardware and for the specific model. 3. The framework is intrinsically extensible: New GPU optimization strategies, including strategies of other simulators, can be added in the generated code for situations where they are effective. The first release version of GeNN is available at http://sourceforge.net/projects/genn. It has been built and optimized for simulating neuronal networks with an anatomical structure (separate neuron populations with sparse or dense connection patterns with the possibility to use some common learning rules). We have executed performance and scalability tests on an NVIDIA Tesla C2070 GPU with an Intel Xeon(R) E5-2609 CPU running Ubuntu 12.04 LTS. Our results show that as the network size increases, GPU simulations never fail to outperform CPU simulations. But we are also able to demonstrate the performance limits of using GPUs with GeNN under different scenarios of network connectivity, learning rules and simulation parameters, confirming the that GPU acceleration can differ largely depending on the particular details of the model of interest.

The major bottleneck in simulation of large-scale neural networks is the communication problem due to one-to-many neuron connectivity. Network-on-Chip concept has been proposed to address the problem. This work explores the drawback that is introduced by interconnection networks - a delay jitter. The preliminary experiment is held in the spiking neural network simulator introducing variable communicational delay to the simulation. The performance degradation is reported.

A forecast-based STDP rule suitable for neuromorphic implementation

Almost all animal behaviors can be seen as sequences of actions towards achieving certain goals. How the association cortices learn to link sensory stimuli to a correct sequence of motor responses is not well understood, especially when only a correct sequence of responses is rewarding. We present a biologically plausible neuronal network model that can be trained to perform a large variety of tasks when only stimuli and reward contingencies are varied. The model’s aim is to learn action values in a feedforward neuronal network and we present mechanisms to overcome the structural and temporal credit assignment problems. The temporal credit assignment problem is solved by a form of Q-learning [1]. The structural credit assignment problem is solved by a form of ‘attentional’ feedback from motor cortex to association cortex that delineates the units that should change connectivity to improve behavior [2]. Moreover, the model has a new mechanism to store traces of relevant sensory stimuli in working memory. During learning, the sensory stimuli, in combination with traces of previous stimuli in working memory become associated with a unique set of action values. Learning in the model is biologically realistic as model units have Hebbian plasticity that is gated by two factors [2]. Firstly, reinforcers or increases in reward expectancy cause the global release of neuromodulatory signals that inform all synapses of the network if the outcome of a trial was better or worse than expected [3]. Selective attention is the second factor that gates plasticity. Attentional feedback highlights the chain of neurons between sensory and motor cortex responsible for the selected action. Only neurons that are causally linked to the action receive attentional feedback, and change the strength of their connections. Selective attention thereby solves the structural credit assignment problem. The resulting learning rule is a form of AGREL [2], which was previously shown to be on average equivalent to error-backpropagation in classification tasks with immediate reward. The present generalization of the learning scheme is based on temporal difference learning and it can train multilayer feedforward networks to perform delayed reward tasks with multiple epochs that require multiple behavioral responses. Importantly, the generalization MQ-AGREL learns to store in working memory information that is relevant at a later stage during a task. This memory is maintained by persistent activity of units at the intermediate network layers. We show that MQ-AGREL can be trained in many tasks that are in use in neurophysiology, including (1) (delayed) saccade-antisaccade tasks; (2) categorization tasks; and (3) probabilistic classification tasks. Neurons at intermediate levels of the network acquire visual responses and memory responses as the result of training that resemble the tuning of neurons in association areas of the cerebral cortex of animals that are trained in these same tasks. We conclude that MQ-AGREL is a powerful and biologically realistic learning rule that accounts for learning in delayed reward tasks that involve non-linear mappings from sensory stimuli and working memory onto motor responses.

This paper demonstrates a single transistor synapse that stores a weight in a non-volatile manner, computes a biological EPSP, and also demonstrates biological learning rules such as LTP, LTD and STDP. It also describes a highly scalable architecture of an array of synapses that can implement the described learning rules. Parameters for weight update in a 0.35μm process were extracted and used to predict changes in weight based on the time difference between pre-synaptic and post-synaptic spike times.

A novel integrative learning architecture based on a reinforcement learning schemata model (RLSM) with a spike timing-dependent plasticity (STDP) network is described. This architecture models operant conditioning with discriminative stimuli in an autonomous agent engaged in multiple reinforcement learning tasks. The architecture consists of two constitutional learning architectures: RLSM and STDP. RLSM is an incremental modular reinforcement learning architecture, and it makes an autonomous agent acquire several behavioral concepts incrementally through continuous interactions with its environment and/or caregivers. STDP is a learning rule of neuronal plasticity found in cerebral cortices and the hippocampus of the human brain. STDP is a temporally asymmetric learning rule that contrasts with the Hebbian learning rule. We found that STDP enabled an autonomous robot to associate auditory input with its acquired behaviors and to select reinforcement learning modules more effectively. Auditory signals interpreted based on the acquired behaviors were revealed to correspond to 'signs' of required behaviors and incoming situations. This integrative learning architecture was evaluated in the context of on-line modular learning.