Simulating spiking neural networks on massively parallel graphical processing units using a code generation approach with GeNN

To understand the mechanism of information processing by a biological neural network, computer simulation of a large-scale spiking neural network is an important method. However, because of a high computation cost of the simulation of a large-scale spiking neural network, the simulation requires high performance computing implemented by a supercomputer or a computer cluster. Recently, hardware for parallel computing such as a multi-core CPU and a graphics card with a graphics processing unit (GPU) is built in a gaming computer and a workstation. Thus, parallel computing using this hardware is becoming widespread, allowing us to obtain powerful computing power for simulation of a large-scale spiking neural network. However, it is not clear how much increased performance the parallel computing method using a new GPU yields in the simulation of a large-scale spiking neural network. In this study, we compared computation time between the computing methods with CPUs and GPUs in a simulation of neuronal models. We developed computer programs of neuronal simulations for the computing systems that consist of a gaming graphics card with new architecture (the NVIDIA GTX 1080) and an accelerator board using a GPU (the NVIDIA Tesla K20C). Our results show that the computing systems can perform a simulation of a large number of neurons faster than CPU-based systems. Furthermore, we investigated the accuracy of a simulation using single precision floating point. We show that the simulation results of single precision were accurate enough compared with those of double precision, but chaotic neuronal response calculated by a GPU using single precision is prominently different from that calculated by a CPU using double precision. Furthermore, the difference in chaotic dynamics appeared even if we used double precision of a GPU. In conclusion, the GPU-based computing system exhibits a higher computing performance than the CPU-based system, even if the GPU system includes data transfer from a graphics card to host memory.

In this paper, we present a methodology for effi-ciently mapping neural networks over a neuromorphic computing architecture. The target architecture is a globally asynchronous locally synchronous (GALS) multi-core designed for simulating spiking neural networks (SNN) in real-time, that is spike timings should be the same as in the human brain. The SNN is implemented as a set of concurrent tasks modelling the behaviour of biological neurons, which are executed on the processing cores and communicate through spikes travelling on a network-on-chip. The problem of neuron-to-core mapping is relevant as a non-efficient allocation may impact real-time and reliability of the neural network execution. We designed a task placement pipeline capable of analysing the network of neurons and producing a placement configuration that enables a reduction of communication between computational nodes. The neuron-to-core mapping problem has been formalised as a problem of minimisation of synaptic elongation. Intuitively, this metric represents the cumulative distance that spikes generated by neurons running on a specific core have to travel to reach their destination core. The proposed placement methodology allows using different techniques to solve the problem. In this work Spectral Analysis, Multilevel Static Mapping, and Simulated Annealing were compared evaluating the overall post-placement synaptic elongation. Results point out that mapping solutions taking into account the directionality of the SNN provide a better placement and quantify this impact. Between all techniques considered only the Simulated Annealing was able to overcome an improvement of 25% compared to a random placement.

Simulation of spiking neural networks — architectures and implementations

Spike-timing-dependent plasticity enhanced synchronization transitions induced by autapses in adaptive Newman-Watts neuronal networks

Graphics Processing Units (GPUs) have been widely used to accelerate artificial intelligence, physics simulation, medical imaging, and information visualization applications. To improve GPU performance, GPU hardware designers need to identify performance issues by inspecting a huge amount of simulator‐generated traces. Visualizing the execution traces can reduce the cognitive burden of users and facilitate making sense of behaviors of GPU hardware components. In this paper, we first formalize the process of GPU performance analysis and characterize the design requirements of visualizing execution traces based on a survey study and interviews with GPU hardware designers. We contribute data and task abstraction for GPU performance analysis. Based on our task analysis, we propose Daisen, a framework that supports data collection from GPU simulators and provides visualization of the simulator‐generated GPU execution traces. Daisen features a data abstraction and trace format that can record simulator‐generated GPU execution traces. Daisen also includes a web‐based visualization tool that helps GPU hardware designers examine GPU execution traces, identify performance bottlenecks, and verify performance improvement. Our qualitative evaluation with GPU hardware designers demonstrates that the design of Daisen reflects the typical workflow of GPU hardware designers. Using Daisen, participants were able to effectively identify potential performance bottlenecks and opportunities for performance improvement. The open‐sourced implementation of Daisen can be found at gitlab.com/akita/vis. Supplemental materials including a demo video, survey questions, evaluation study guide, and post‐study evaluation survey are available at osf.io/j5ghq.

Event Abstract Back to Event Roles played by depressing synapses in neural circuits Joanna Jędrzejewska-Szmek1* and Jarosław Żygierewicz2 1 University of Warsaw Faculty of Physics, Poland 2 University of Warsaw Faculty of Physics, Poland Synapses exhibiting short-term synaptic depression can be found in different locations in neural circuits [1], e.g. pyramid-pyramid and interneuron-pyramid connections. Based on these findings, two Wilson-Cowan type models with depressing synapses were constructed, following formalism presented in [2]. First one comprised a depressing positive feedback, the second one comprised a depressing negative feedback (synapse is between the inhibitory and excitatory population). The models' parameters were provided by [1,3]. Both models generated oscillatory signals as predicted by [2,3]. In case of both models a regime exists with high frequency oscillations (higher than 80 Hz) and high amplitude of oscillations corresponding to 'high-gamma' characteristics [4]. The model with depressing positive feedback showed three regimes: 1) oscillations in the theta band almost independent of external input of the excitatory population, 2) high-frequency oscillations with an envelope frequency in the theta band. In the third regime we observed oscillations in theta and 'high-gamma band'. An increase of the external input can abruptly change the oscillation frequency from the theta band to 'high-gamma'. Excitatory population firing rate is shown in the Figure below. One can see, that generated activity is similar to the theta/gamma oscillations [5] linked to short-term memory [5,6]. Short-term synaptic depression plays important role in generation of not only slow oscillations activity [3] and theta waves, but also higher frequency phenomena such as 'high-gamma'. It might also serve as a basis for coupling between the high- and low-frequency bands of ongoing electrical activity in the human brain [5,6,7]. [1] Bannister AP, Thomson AM. Cerebral Cortex 2007 [2] Tsodyks MV, Pawelzik K, Markram H. Neural Computation 1998 [3] Melamed O, Barak O, Silberberg G, Markram H, Tsodyks M. Journal of Computational Neuroscience 2008 [4] Ray S, Maunsell JHR. PloS Biology 2011 [5] Lisman J. Hippocampus 2005 [6] Lisman J, Idiart MA. Science 1995 [7] Canolty RT, Edwards E, Dalal SS, Soltani M, Nagarajan SS, Kirsch HE, Berger MS, Barbaro NM, Knight RT. Science 2006 Figure 1 Keywords: computational neuroscience, neural circuits, Synapses, electrical activity, human brain Conference: 5th INCF Congress of Neuroinformatics, Munich, Germany, 10 Sep - 12 Sep, 2012. Presentation Type: Poster Topic: Neuroinformatics Citation: Jędrzejewska-Szmek J and Żygierewicz J (2014). Roles played by depressing synapses in neural circuits. Front. Neuroinform. Conference Abstract: 5th INCF Congress of Neuroinformatics. doi: 10.3389/conf.fninf.2014.08.00020 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 21 Mar 2013; Published Online: 27 Feb 2014. * Correspondence: Dr. Joanna Jędrzejewska-Szmek, University of Warsaw Faculty of Physics, Warszawa, Poland, asia@in.waw.pl Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Joanna Jędrzejewska-Szmek Jarosław Żygierewicz Google Joanna Jędrzejewska-Szmek Jarosław Żygierewicz Google Scholar Joanna Jędrzejewska-Szmek Jarosław Żygierewicz PubMed Joanna Jędrzejewska-Szmek Jarosław Żygierewicz Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

In this work, we implement hardware-based spiking neural network (SNN) using the thin-film transistor (TFT)-type flash synaptic devices. A hardware-based SNN architecture with synapse arrays and integrate-and-fire (I&F) neuron circuits is presented for executing reinforcement learning (RL). Two problems were used to evaluate the applicability of the proposed hardware-based SNNs to off-chip RL: the Cart Pole balancing problem and the Rush Hour problem. The neural network was trained using a deep Q-learning algorithm. The proposed hardware-based SNNs using the synapse model with measured characteristics successfully solve the two problems and show high performance, implying that the networks are suitable for executing RL. Furthermore, the effect of variations in non-ideal synaptic devices and neurons on the performance was investigated.

Complex ideas are best conveyed through well-designed illustrations. Up to now, computational neuroscientists have mostly relied on box-and-arrow diagrams of even complex neuronal networks, often using ad hoc notations with conflicting use of symbols from paper to paper. This significantly impedes the communication of ideas in neuronal network modeling. We present here Connectivity Pattern Tables (CPTs) as a clutter-free visualization of connectivity in large neuronal networks containing two-dimensional populations of neurons. CPTs can be generated automatically from the same script code used to create the actual network in the NEST simulator. Through aggregation, CPTs can be viewed at different levels, providing either full detail or summary information. We also provide the open source ConnPlotter tool as a means to create connectivity pattern tables.

Neuroscientists study the brain at many levels of organization - from single molecules to the entire brain. Over the past 30 odd years, advances molecular biology, genetics and biophysical methods have revolutionized studies at the small scale - ion channels, synapses and subcellular compartments. At the other end, advances macroscopic brain mapping have ignited a wave of studies at the large scale. However, the computations carried out by brains are best understood terms of the collective behavior of neuronal populations. Although we have known the value of circuit-level analysis for some time now, limitations experimental and analytical tools have been significant barriers for progress. Brains are made up of staggering numbers of neurons (~105 fruit flies, ~1011 neurons humans); we don't know the detailed connectivity, we don't have a detailed understanding of the input and output, and we don't know the algorithms. We are often probing the dark. Things are starting to change, and mesoscale studies involving circuits of neurons are now ready to take centre stage. Optical and genetic techniques now allow us to visualize and perturb activity of identified populations of neurons at the microscopic scale; serial electron microscopy and clever molecular circuit-tracing techniques are on the verge of giving us detailed information on connectivity among large numbers of neurons; and theoretical neuroscientists have embraced the messiness of biological hardware and started to think about how networks actually compute. These new tools and techniques are poised to produce a torrent of new experimental results and theoretical insights. Neural Systems & Circuits will offer a forum for the seamless integration of the resulting body of knowledge. We welcome both theoretical and experimental studies, and everything between, as long as they relate to circuit or systems-level analysis. In fact, we anticipate an increasing number of these in studies, as the boundaries between theory and experiment blur. Too often, theoretical papers become mathematical exercises for specialists, with little attempt to engage or stimulate experimentalists (who turn, simply ignore most theory papers). Conversely, many experimental studies read like catalogues, offering little incentive to others wanting to integrate them into a bigger picture or a theory. By offering a mix of papers a single locus, we aim to foster communication among theorists and experimentalists, and so catalyze cross-pollination. Neural Systems & Circuits is a fully open-access journal. Articles will be published online as soon as they are accepted, and it is anticipated that they will also be indexed PubMed due course. Besides research articles, we will publish reviews and opinion pieces. We hope to report, on occasion, dialogs between those with opposing points of view on important and controversial topics. Studies of disease that have particular emphasis on circuits are also welcome, as are technical advances. The diverse and expert editorial board will ensure excellence the review process. This inaugural issue of Neural Systems & Circuits presents a selection of articles on different topics and aims to illustrate some of the topics we would like to include as the journal grows. As our understanding of the field changes, we expect the journal to grow to accommodate the latest advances. We are committed to making Neural Systems & Circuits a success, and sincerely hope that you will share our vision and will consider contributing to this new journal.

Simplicial complexes are mathematical constructions that describe higher-order interactions within the interconnecting elements of a network. Such higher-order interactions become increasingly significant in neuronal networks since biological backgrounds and previous outcomes back them. In light of this, the current research explores a higher-order network of the memristive Rulkov model. To that end, the master stability functions are used to evaluate the synchronization of a network with pure pairwise hybrid (electrical and chemical) synapses alongside a network with two-node electrical and multi-node chemical connections. The findings provide good insight into the impact of incorporating higher-order interaction in a network. Compared to two-node chemical synapses, higher-order interactions adjust the synchronization patterns to lower multi-node chemical coupling parameter values. Furthermore, the effect of altering higher-order coupling parameter value on the dynamics of neurons in the synchronization state is researched. It is also shown how increasing network size can enhance synchronization by lowering the value of coupling parameters whereby synchronization occurs. Except for complete synchronization, cluster synchronization is detected for higher electrical coupling strength values wherein the neurons are out of the completed synchronization state.

Parallel computing of layer-based method for generating hologram of 3D objects is introduced. 3D MAX is used to model 3D object. The hologram of 3D model with depth information is calculated by Fresnel diffraction algorithm. The computational hologram generated by computer is reconstructed photoelectric to verify the correctness of the algorithm. This paper expounds the hardware architecture of GPU and CPU, briefly introduces the bottleneck and solution of CPU and GPU acceleration, and describes the optimization of thread and storage bandwidth in parallel processing. We use GPU hardware parallel computing and optimize the calculation process of 3D object hologram by using MKL and CUDA computing environment to improve the efficiency of computing. After analysis, the results show that the parallel computing speed of GPU hardware is 63 times faster than CPU alone. The parallel acceleration method can greatly shorten the computing time of generating hologram with layer-based method.

GPU accelerators have become an Important backbone for scientific high performance-computing, and the performance advances obtained from adopting new GPU hardware are significant. In this paper we take a first look at NVIDIA's newest server-line GPU, the A100 architecture, part of the Ampere generation. Specifically, we assess its performance for sparse and batch computations, as these routines are relied upon in many scientific applications, and compare to the performance achieved on NVIDIA's previous server-line GPU.

A fundamental question in computational neuroscience is how to connect a network of spiking neurons to produce desired macroscopic or mean field dynamics. One possible approach is through the Neural Engineering Framework (NEF). The NEF approach requires quantities called decoders which are solved through an optimization problem requiring large matrix inversion. Here, we show how a decoder can be obtained analytically for type I and certain type II firing rates as a function of the heterogeneity of its associated neuron. These decoders generate approximants for functions that converge to the desired function in mean-squared error like 1/N, where N is the number of neurons in the network. We refer to these decoders as scale-invariant decoders due to their structure. These decoders generate weights for a network of neurons through the NEF formula for weights. These weights force the spiking network to have arbitrary and prescribed mean field dynamics. The weights generated with scale-invariant decoders all lie on low dimensional hypersurfaces asymptotically. We demonstrate the applicability of these scale-invariant decoders and weight surfaces by constructing networks of spiking theta neurons that replicate the dynamics of various well known dynamical systems such as the neural integrator, Van der Pol system and the Lorenz system. As these decoders are analytically determined and non-unique, the weights are also analytically determined and non-unique. We discuss the implications for measured weights of neuronal networks.

Simulating large networks of spiking neurons is a very common task in the areas of Neuroinformatics and Computational Neurosciences. These simulations are time-consuming but also often intrinsically parallel. The recent advent of powerful and programmable graphic cards seems to be a pertinent solution to the problem: they offer a cheap but efficient possibility to serve as very fast co-processors for the parallel computing that spiking neural networks need. We describe our implementation of three different problems on such a card: two image-segmentation algorithms using spiking neural networks and one multi-purpose spiking neural-network simulator. Using these examples we show the benefits, the challenges and the limits of such an implementation.

A core question in systems and computational neuroscience is how the brain represents information. Identifying principles of information coding in neural circuits is critical to understanding brain organization and function in sensory, motor, and cognitive neuroscience. This provides a conceptual bridge between the underlying biophysical mechanisms and the ultimate behavioral goals of the organism. Central to this framework is the question of computation: what are the relevant representations of input and output, and what algorithms govern the input-output transformation? Remarkably, evidence suggests that certain canonical computations exist across different circuits, brain regions, and species. Such computations are implemented by different biophysical and network mechanisms, indicating that the unifying target of conservation is the algorithmic form of information processing rather than the specific biological implementation. A prime candidate to serve as a canonical computation is divisive normalization, which scales the activity of a given neuron by the activity of a larger neuronal pool. This nonlinear transformation introduces an intrinsic contextual modulation into information coding, such that the selective response of a neuron to features of the input is scaled by other input characteristics. This contextual modulation allows the normalization model to capture a wide array of neural and behavioral phenomena not captured by simpler linear models of information processing. The generality and flexibility of the normalization model arises from the normalization pool, which allows different inputs to directly drive and suppress a given neuron, effectively separating information that drives excitation and contextual modulation. Originally proposed to describe responses in early visual cortex, normalization has been widely documented in different brain regions, hierarchical levels, and modalities of sensory processing; furthermore, recent work shows that the normalization extends to cognitive processes such as attention, multisensory integration, and decision making. This ubiquity reinforces the canonical nature of the normalization computation and highlights the importance of an algorithmic framework in linking biological mechanism and behavior.