Extending the Functional Subnetwork Approach to a Generalized Linear Integrate-and-Fire Neuron Model

The problem we address in this paper is that of finding effective and parsimonious patterns of connectivity in sparse associative memories. This problem must be addressed in real neuronal systems, so that results in artificial systems could throw light on real systems. We show that there are efficient patterns of connectivity and that these patterns are effective in models with either spiking or non-spiking neurons. This suggests that there may be some underlying general principles governing good connectivity in such networks. We also show that the clustering of the network, measured by Clustering Coefficient, has a strong negative linear correlation to the performance of associative memory. This result is important since a purely static measure of network connectivity appears to determine an important dynamic property of the network.

With recent advances in learning algorithms, recurrent networks of spiking neurons are achieving performance that is competitive with vanilla recurrent neural networks. However, these algorithms are limited to small networks of simple spiking neurons and modest-length temporal sequences, as they impose high memory requirements, have difficulty training complex neuron models and are incompatible with online learning. Here, we show how the recently developed Forward-Propagation Through Time (FPTT) learning combined with novel liquid time-constant spiking neurons resolves these limitations. Applying FPTT to networks of such complex spiking neurons, we demonstrate online learning of exceedingly long sequences while outperforming current online methods and approaching or outperforming offline methods on temporal classification tasks. The efficiency and robustness of FPTT enable us to directly train a deep and performant spiking neural network for joint object localization and recognition, demonstrating the ability to train large-scale dynamic and complex spiking neural network architectures. Memory efficient online training of recurrent spiking neural networks without compromising accuracy is an open challenge in neuromorphic computing. Yin and colleagues demonstrate that training a recurrent neural network consisting of so-called liquid time-constant spiking neurons using an algorithm called Forward-Propagation Through Time allows for online learning and state-of-the-art performance at a reduced computational cost compared with existing approaches.

We investigated spontaneous activity and excitability in large networks of artificial spiking neurons. We compared three different spiking neuron models: integrate-and-fire (IF), regular-spiking (RS), and resonator (RES). First, we show that different models have different frequency-dependent response properties, yielding large differences in excitability. Then, we investigate the responsiveness of these models to a single afferent inhibitory/excitatory spike and calibrate the total synaptic drive such that they would exhibit similar peaks of the postsynaptic potentials (PSP). Based on the synaptic calibration, we build large microcircuits of IF, RS, and RES neurons and show that the resonance property favors homeostasis and self-sustainability of the network activity. On the other hand, integration produces instability while it endows the network with other useful properties, such as responsiveness to external inputs. We also investigate other potential sources of stable self-sustained activity and their relation to the membrane properties of neurons. We conclude that resonance and integration at the neuron level might interact in the brain to promote stability as well as flexibility and responsiveness to external input and that membrane properties, in general, are essential for determining the behavior of large networks of neurons.

Non-spiking neurons use graded potentials for processing neural information, unlike spiking neurons. These non-spiking neurons struggle to transmit signals distally due to membrane potential decay. Consequently, they possibly perform unique computations different from spiking neurons by using local processing with graded potential. Invertebrates, such as insects, possess both non-spiking and spiking neurons in their central nervous system. However, how the subcellular local regions in non-spiking neurons process sensory information remains unclear. We examined intracellular Ca2+ dynamics in the wind-sensitive non-spiking neurons of crickets. Directional information about airflow is processed by a local circuit within the terminal abdominal ganglion, where several identified non-spiking interneurons are involved. In this study, we measured the membrane potential and Ca2+ responses to airflow from different angles in three local non-spiking interneurons. These neurons exhibited spatial heterogeneities in their Ca2+ responses. In two non-spiking interneurons, the similarity in the time course of Ca2+ responses between subcellular regions decreased with increasing intracellular distance. However, the similarity in directional selectivity was not always linked to the intracellular distance. In addition, the Ca2+ signals induced by current injection did not necessarily couple with the membrane potential changes. The temporal dynamics of Ca2+ signals were possibly influenced by partially uneven electrical propagation, and their directional selectivity, primarily determined by the spatial presynaptic arrangement, might depend less on intracellular distance. Therefore, local non-spiking interneurons likely perform local processing affected by both presynaptic and postsynaptic factors, enabling a single neuron to substitute for functions otherwise distributed across multiple spiking neurons.

Neuromorphic computing [1], inspired by the energy-efficient and parallelized architecture of the human brain, represents a paradigm shift in computing. This approach not only drives innovative hardware designs but also inspires novel network architectures like spiking neural networks (SNNs), which process information as discrete spike events. However, scalability in neuromorphic systems is constrained by the large hardware footprint. Realizing multimodal functionality using the same set of hardware primitives can offer a promising solution to these limitations.In this work, we present a dual encoder-neuron circuit based on a nanoporous graphene (NPG) memristor device [2]. The graphene leaky integrate-and-fire (GLIF) circuit has a versatile multimodal design that combines the roles of a LIF neuron and a spike encoder within the same architecture. This design mimics the dual functionality of biological neurons, which encode and process signals locally. The spike encoding capability of the circuit is further revealed using single and double layer SNN networks for pattern recognition tasks using the Modified National Institute of Standards and Technology (MNIST) dataset.Inset of Figure 1(a) shows the schematic of the fabricated NPG device with a channel length of 100 µm. The device has a lateral structure with NPG channel and gold (Au) electrodes. Figure 1(a) shows the current-voltage (I-V) characteristics of the fabricated NPG device which shows a threshold switching behavior with a wide hysteresis window and a threshold voltage (Vth) of 4.9 V. A behavioral SPICE model is developed based on a voltage-controlled switch model. It is seen from Figure 1(a) that the I-V curves simulated using the SPICE model matches the experimentally observed device switching characteristics. This reveals that the simplified SPICE model can be a good substitute for complex Verilog based models. Based on the SPICE model, we designed a simple neuron circuit with a resistor (Rs, 12 kΩ) in series with a parallel combination of a capacitor (Cm, 10 nF) and the NPG device (SPICE model). Figure 1(b) shows the variation of the membrane potential (Vm) and the output spike current in response to the input pulse voltage (Vin, 6 V). Vm shows a clear leaky integration behavior and full reset to zero voltage after the spike response. These are essential attributes of a bioplausible LIF neuron circuit. In order to develop the multimodal GLIF circuit, the output spike frequency of the LIF neuron circuit is plotted as a function of the Rs values. This mapping allows pixel values from MNIST input images to be directly translated into normalized conductance values (G/G0). Hence, each pixel is represented as a normalized conductance value, establishing a clear correlation between pixel intensity and spike frequency. This core idea is further used to develop the GLIF encoder based on the same neuron circuit as before. Based on the dual encoder-neuron circuit, we designed a single layer SNN for recognising the handwritten digits from the MNIST dataset. Figure 1(c) shows the spike-encoded images for the MNIST digits “2” and “5”. The probability of spiking in each timestep is proportional to the spiking frequency. The spike encoded images at each time step reveal the contours of the original input images using sparse representation. These sparse matrices highlight the energy efficiency and computational advantages of the GLIF system. The performance of the dual encoder-neuron system is evaluated by training the single layer SNN using both the GLIF encoder and neuron circuit as integral components. The single layer SNN recorded a high pattern recognition accuracy of 90.77% which is comparable to the purely software-based implementation (92.69%). To explore the scalability of the GLIF system, we implemented a double-layer SNN using the GLIF system, achieving a high recognition accuracy of 97.37%. The GLIF circuit exemplifies how reconfigurable primitive components can drive the development of scalable neuromorphic hardware.

Electrical activity of excitable cells results from ion exchanges through cell membranes, so that genetic or epigenetic changes in genes encoding ion channels are likely to affect neuronal electrical signaling throughout the brain. There is a large literature on the effect of variations in ion channels on the dynamics of spiking neurons that represent the main type of neurons found in the vertebrate nervous systems. Nevertheless, non-spiking neurons are also ubiquitous in many nervous tissues and play a critical role in the processing of some sensory systems. To our knowledge, however, how conductance variations affect the dynamics of non-spiking neurons has never been assessed. Based on experimental observations reported in the biological literature and on mathematical considerations, we first propose a phenotypic classification of non-spiking neurons. Then, we determine a general pattern of the phenotypic evolution of non-spiking neurons as a function of changes in calcium and potassium conductances. Furthermore, we study the homeostatic compensatory mechanisms of ion channels in a well-posed non-spiking retinal cone model. We show that there is a restricted range of ion conductance values for which the behavior and phenotype of the neuron are maintained. Finally, we discuss the implications of the phenotypic changes of individual cells at the level of neuronal network functioning of the C. elegans worm and the retina, which are two non-spiking nervous tissues composed of neurons with various phenotypes.

One developing approach for robotic control is the use of networks of dynamic neurons connected with conductance-based synapses, also known as Synthetic Nervous Systems (SNS). These networks are often developed using cyclic topologies and heterogeneous mixtures of spiking and non-spiking neurons, which is a difficult proposition for existing neural simulation software. Most solutions apply to either one of two extremes, the detailed multi-compartment neural models in small networks, and the large-scale networks of greatly simplified neural models. In this work, we present our open-source Python package SNS-Toolbox, which is capable of simulating hundreds to thousands of spiking and non-spiking neurons in real-time or faster on consumer-grade computer hardware. We describe the neural and synaptic models supported by SNS-Toolbox, and provide performance on multiple software and hardware backends, including GPUs and embedded computing platforms. We also showcase two examples using the software, one for controlling a simulated limb with muscles in the physics simulator Mujoco, and another for a mobile robot using ROS. We hope that the availability of this software will reduce the barrier to entry when designing SNS networks, and will increase the prevalence of SNS networks in the field of robotic control.

We show that networks of relatively realistic mathematical models for biological neurons in principle can simulate arbitrary feedforward sigmoidal neural nets in a way that has previously not been considered. This new approach is based on temporal coding by single spikes (respectively by the timing of synchronous firing in pools of neurons) rather than on the traditional interpretation of analog variables in terms of firing rates. The resulting new simulation is substantially faster and hence more consistent with experimental results about the maximal speed of information processing in cortical neural systems. As a consequence we can show that networks of noisy spiking neurons are "universal approximators" in the sense that they can approximate with regard to temporal coding any given continuous function of several variables. This result holds for a fairly large class of schemes for coding analog variables by firing times of spiking neurons. This new proposal for the possible organization of computations in networks of spiking neurons systems has some interesting consequences for the type of learning rules that would be needed to explain the self-organization of such networks. Finally, the fast and noise-robust implementation of sigmoidal neural nets by temporal coding points to possible new ways of implementing feedforward and recurrent sigmoidal neural nets with pulse stream VLSI.

A theoretical model for analogue computation in networks of spiking neurons with temporal coding is introduced and tested through simulations in GENESIS. It turns out that the use of multiple synapses yields very noise robust mechanisms for analogue computations via the timing of single spikes in networks of detailed compartmental neuron models.In this way, one arrives at a method for emulating arbitrary Hopfield nets with spiking neurons in temporal coding, yielding new models for associative recall of spatio-temporal firing patterns. We also show that it suffices to store these patterns in the efficacies of excitatory synapses.A corresponding layered architecture yields a refinement of the synfire-chain model that can assume a fairly large set of different stable firing patterns for different inputs.

Spiking Irregularity and Frequency Modulate the Behavioral Report of Single-Neuron Stimulation

The paper presents a modification of the structure of a biological neural network (BNN) based on spiking neuron models. The proposed modification allows to influence the level of the stimulus response of particular neurons in the BNN. We consider an extended, three-dimensional Hodgkin-Huxley model of the neural cell. A typical BNN composed of such neural cells have been expanded by addition of resistors in each branch point. The resistors can be treated as the weights in such BNN. We demonstrate that adding these elements to the BNN significantly affects the waveform of the potential on the membrane of the neuron, causing an uncontrolled excitation. This provides a better description of processes that take place in nervous cell. Such BNN enables an easy adaptation of the learning rules used in artificial or spiking neural networks. The modified BNN has been implemented on Graphics Processing Unit (GPU) in the CUDA C language. This platform enables a parallel data processing, which is an important feature in such applications.

Spiking Neuron Networks (SNNs) are often referred to as the third generation of neural networks. Highly inspired by natural computing in the brain and recent advances in neurosciences, they derive their strength and interest from an accurate modeling of synaptic interactions between neurons, taking into account the time of spike firing. SNNs overcome the computational power of neural networks made of threshold or sigmoidal units. Based on dynamic event-driven processing, they open up new horizons for developing models with an exponential capacity to memorize and a strong ability to do fast adaptation. Today, the main challenge is to discover efficient learning rules that might take advantage of the specific features of SNNs while keeping the nice properties (general-purpose, easy-to-use, available simulators, etc.) of traditional connectionist models. This chapter relates the history of the “spiking neuron” in Sect. 1 and summarizes the most currently-in-use models of neurons and synaptic plasticity in Sect. 2. The computational power of SNNs is addressed in Sect. 3 and the problem of learning in networks of spiking neurons is tackled in Sect. 4, with insights into the tracks currently explored for solving it. Finally, Sect. 5 discusses application domains, implementation issues and proposes several simulation frameworks.

Unlike spiking neurons which compress continuous inputs into digital signals for transmitting information via action potentials, non-spiking neurons modulate analog signals through graded potential responses. Such neurons have been found in a large variety of nervous tissues in both vertebrate and invertebrate species, and have been proven to play a central role in neuronal information processing. If general and vast efforts have been made for many years to model spiking neurons using conductance-based models (CBMs), very few methods have been developed for non-spiking neurons. When a CBM is built to characterize the neuron behavior, it should be endowed with generalization capabilities (i.e. the ability to predict acceptable neuronal responses to different novel stimuli not used during the model’s building). Yet, since CBMs contain a large number of parameters, they may typically suffer from a lack of such a capability. In this paper, we propose a new systematic approach based on multi-objective optimization which builds general non-spiking models with generalization capabilities. The proposed approach only requires macroscopic experimental data from which all the model parameters are simultaneously determined without compromise. Such an approach is applied on three non-spiking neurons of the nematode Caenorhabditis elegans (C. elegans), a well-known model organism in neuroscience that predominantly transmits information through non-spiking signals. These three neurons, arbitrarily labeled by convention as RIM, AIY and AFD, represent, to date, the three possible forms of non-spiking neuronal responses of C. elegans.

A fundamental question in computational neuroscience is how to connect a network of spiking neurons to produce desired macroscopic or mean field dynamics. One possible approach is through the Neural Engineering Framework (NEF). The NEF approach requires quantities called decoders which are solved through an optimization problem requiring large matrix inversion. Here, we show how a decoder can be obtained analytically for type I and certain type II firing rates as a function of the heterogeneity of its associated neuron. These decoders generate approximants for functions that converge to the desired function in mean-squared error like 1/N, where N is the number of neurons in the network. We refer to these decoders as scale-invariant decoders due to their structure. These decoders generate weights for a network of neurons through the NEF formula for weights. These weights force the spiking network to have arbitrary and prescribed mean field dynamics. The weights generated with scale-invariant decoders all lie on low dimensional hypersurfaces asymptotically. We demonstrate the applicability of these scale-invariant decoders and weight surfaces by constructing networks of spiking theta neurons that replicate the dynamics of various well known dynamical systems such as the neural integrator, Van der Pol system and the Lorenz system. As these decoders are analytically determined and non-unique, the weights are also analytically determined and non-unique. We discuss the implications for measured weights of neuronal networks.

Networks of spiking neurons with adaption have been shown to be able to reproduce a wide range of neural activities, including the emergent population bursting and spike synchrony that underpin brain disorders and normal function. Exact mean-field models derived from spiking neural networks are extremely valuable, as such models can be used to determine how individual neurons and the network they reside within interact to produce macroscopic network behaviours. In the paper, we derive and analyze a set of exact mean-field equations for the neural network with spike frequency adaptation. Specifically, our model is a network of Izhikevich neurons, where each neuron is modeled by a two dimensional system consisting of a quadratic integrate and fire equation plus an equation which implements spike frequency adaptation. Previous work deriving a mean-field model for this type of network, relied on the assumption of sufficiently slow dynamics of the adaptation variable. However, this approximation did not succeed in establishing an exact correspondence between the macroscopic description and the realistic neural network, especially when the adaptation time constant was not large. The challenge lies in how to achieve a closed set of mean-field equations with the inclusion of the mean-field dynamics of the adaptation variable. We address this problem by using a Lorentzian ansatz combined with the moment closure approach to arrive at a mean-field system in the thermodynamic limit. The resulting macroscopic description is capable of qualitatively and quantitatively describing the collective dynamics of the neural network, including transition between states where the individual neurons exhibit asynchronous tonic firing and synchronous bursting. We extend the approach to a network of two populations of neurons and discuss the accuracy and efficacy of our mean-field approximations by examining all assumptions that are imposed during the derivation. Numerical bifurcation analysis of our mean-field models reveals bifurcations not previously observed in the models, including a novel mechanism for emergence of bursting in the network. We anticipate our results will provide a tractable and reliable tool to investigate the underlying mechanism of brain function and dysfunction from the perspective of computational neuroscience.