Bio-inspired sensory systems with integrated capabilities of sensing, data storage, and processing

In recent years, spiking neural networks (SNNs), which originated from the theoretical basis of neuroscience, have attracted neuromorphic computing and brain-like computing due to their advantages, such as neural dynamics and coding mechanism, which are similar to biological neurons. SNNs have become one of the mainstream frameworks in the field of brain-like computing. However, most of the Leaky Integrate-and-Fire (LIF) neuron models currently used by SNNs based on direct training of backpropagation (BP) do not consider the changes in the recurrent connections and the dynamic strength of neuron connections over time. This study presented the LIF neuron model with recurrent connections and a method for dynamically changing the presynaptic currents. Recurrent LIF neurons have an additional cyclic connection compared with classic LIF neurons. Their postsynaptic current stimulates a change in membrane potential at the next time point. Their dynamics were more similar to the activities of biological neurons. We also proposed an efficient and flexible BP training method for recurrent LIF neurons. On the basis of the above methods, we proposed the recurrent SNN with dynamic presynaptic currents based on backpropagation (RDS-BP). We test the proposed RDS-BP on three image data sets (MNIST, Fashion-MNIST and CIFAR-10) and two text data sets (IMDB and TREC). The results showed that the performance of RDS-BP not only exceeded the naive SNN models based on BP but also exceeded the SNN methods proposed in previous studies in recent years, which had excellent performance in previous experiments. Our work provides a new LIF neuron model with a recurrent connection and dynamic presynaptic current and a BP training arrangement for the proposed neuron, which could merit developments with neuromorphic and brain-like computing.

Spiking Neural Networks (SNN) is one special implementation of Artificial Neural Networks (ANN), where the input signals are encoded in the temporal relationship between consecutive spikes (spike trains) instead of real-numbered values. Nevertheless, SNN is believed to be a closer representation of the biological neural system, because it imitates the current spikes that are transmitted between neurons in real biological systems. Practical and simplified SNN models include the Leaky Integrate-and-Fire (LIF) function of the neurons and the Spiking Timing Dependent Plasticity (STDP) function of the synapses. While most ANN architectures can be implemented with digital logic gates, SNN is more suitable to be implemented in analog circuits. In this paper, we propose revised analog circuit implementations of SNN neurons with LIF and STDP functions. Compared with previous works by other researchers, our proposed analog designs use fewer components and can be cascaded to form a complete neural system. The circuits are designed and simulated with SMIC 55nm CMOS LP process. The simulated results demonstrate that the analog neural system can work under a very small current (less than 10 µA) and voltage supply (1.0 Volts), and consumes less power consumption than digital implementations.

In recent years, Deep Convolutional Neural Networks (DCNNs) have outreached the performance of classical algorithms for image restoration tasks. However most of these methods are not suited for computational efficiency and are therefore too expensive to be executed on embedded and mobile devices. In this work we investigate Spiking Neural Networks (SNNs) for Gaussian denoising, with the goal of approaching the performance of conventional DCNN while reducing the computational load. We propose a formal analysis of the information conversion processing carried out by the Leaky Integrate and Fire (LIF) neurons and we compare its performance with the classical rate-coding mechanism. The neural coding schemes are then evaluated through experiments in terms of denoising performance and computation efficiency for a state-of-the-art deep convolutional neural network. Our results show that SNNs with LIF neurons can provide competitive denoising performance but at a reduced computational cost.

Spiking neural networks (SNNs) aim to simulate real neural networks in the human brain with biologically plausible neurons. The leaky integrate-and-fire (LIF) neuron is one of the most widely studied SNN architectures. However, it has the vanishing gradient problem when trained with backpropagation. Additionally, its neuronal parameters are often manually specified and fixed, in contrast to the heterogeneity of real neurons in the human brain. This paper proposes a gated parametric neuron (GPN) to process spatio-temporal information effectively with the gating mechanism. Compared with the LIF neuron, the GPN has two distinguishing advantages: (1) it copes well with the vanishing gradients by improving the flow of gradient propagation; and, (2) it learns spatio-temporal heterogeneous neuronal parameters automatically. Additionally, we use the same gate structure to eliminate initial neuronal parameter selection and design a hybrid recurrent neural network-SNN structure. Experiments on two spike-based audio datasets demonstrated that the GPN network outperformed several state-of-the-art SNNs, could mitigate vanishing gradients, and had spatio-temporal heterogeneous parameters. Our work shows the ability of SNNs to handle long-term dependencies and achieve high performance simultaneously.

Cortical networks are capable of unsupervised learning and spontaneous replay of complex temporal sequences. Endowing artificial spiking neural networks with similar learning abilities remains a challenge. In particular, it is unresolved how different plasticity rules can contribute to both learning and the maintenance of network stability during learning. Here we introduce a biologically inspired form of GABA-Modulated Spike Timing-Dependent Plasticity (GMS) and demonstrate its ability to permit stable learning of complex temporal sequences including natural language in recurrent spiking neural networks. Motivated by biological findings, GMS utilizes the momentary level of inhibition onto excitatory cells to adjust both the magnitude and sign of Spike Timing-Dependent Plasticity (STDP) of connections between excitatory cells. In particular, high levels of inhibition in the network cause depression of excitatory-to-excitatory connections. We demonstrate the effectiveness of this mechanism during several sequence learning experiments with character- and token-based text inputs as well as visual input sequences. We show that GMS maintains stability during learning and spontaneous replay and permits the network to form a clustered hierarchical representation of its input sequences. Overall, we provide a biologically inspired model of unsupervised learning of complex sequences in recurrent spiking neural networks.

Speech trigger and speech command recognition technologies have become pivotal human-machine interfaces for AIoT applications. Typically, a speech-related system consists of a mel-filter-based handcrafted feature extractor and an artificial neuron-based classifier. Recent advancements [1–3] have demonstrated promising results by training the feature extractor jointly with the neural network classifier. Moreover, the emergence of spiking neuron network training algorithms and hardware designs has showcased their competitive edge over traditional artificial neural networks. Consequently, the design and training of learnable audio feature extractors and spiking neural network classifiers are the key focuses of this thesis. The first focus of this thesis is the design of an ultra-low-power, learnable audio feature extractor to support end-to-end training. Conventional low-power approaches, such as extreme pruning and quantization in the audio feature extractor (AuFEx) design, often result in decreased accuracy. To address this, the thesis proposes a hardware-aware, end-to-end training framework to co-optimize the AuFEx design, achieving high performance with minimal hardware cost. Two types of AuFEx designs are explored: a time-domain (TD) filter-based learnable audio feature extractor (AuFEx-TD) and a frequency-domain (FD) filter-based learnable audio feature extractor (AuFEx-FD). Using this framework, the AuFEx-TD achieves high accuracy (97.8%) for one wake word detection, with low power consumption (496 nW) and low latency (375 μs). The optimized AuFEx-FD demonstrates noise-robust 10-keyword spotting accuracy (87.5%@5dB – 92.2%@20dB) with low power consumption (1.26 μW). Both AuFEx-TD and AuFEx-FD designs are implemented using UMC 40nm CMOS technology to validate the framework's effectiveness. For the speech trigger or wake word detection (WWD) task, power and latency are critical design considerations. This thesis presents a sub-μW spiking neural network accelerator with an 8 ms latency. The core WWD engine employs a spiking convolutional neural network (SCNN) model, leveraging sparse activation and addition-only operations within spiking neurons. The proposed SCNN model enhances the existing framewise incremental computation flow [4] by incorporating a spike processing unit (SPU), reducing system power and latency by 16.5% and 43.2%, respectively. Extensive quantization further reduces weight and activation precisions to 4-bit and 1-bit. Additionally, a power-gating mechanism, driven by an energy-based voice activity detection (VAD) module, further minimizes power consumption in random and sparse event (RSE) scenarios. Full chip simulation results show that the chip consumes only 110 nW, with a 2.15% false alarm rate and a 3.00% false reject rate in a 10% voice event stream test. It achieves state-of-the-art recognition accuracy of 99% and 96% for one and two wake word detection tasks, respectively. For speech command recognition or keyword spotting (KWS) tasks, robustness against noise and speaker accents is essential for user experience. This thesis explores both end-to-end training and on-device transfer learning to enhance KWS chip robustness with minimal power overhead. The proposed chip comprises three main blocks: the Learnable Audio Feature Extractor (AuFEx), the Spiking-DSCNN Core (SC), and the Transfer Learning Core (TLC). The AuFEx is an optimized AuFEx-FD, with the on-chip memory for storing FD-filter parameters replaced by an FD-filter parameter generator, reducing power consumption to 1.225 μW. The SC implements a spiking neural network (SNN) model using a Depthwise Separable Convolutional Neural Network (DSCNN) topology. Multi-level encoding in spiking neurons reduces simulation time, while the inherent activation sparsity of these neurons decreases power consumption and latency. The TLC supports both forward and backward propagation for a fully connected layer and softmax layer, enabling the KWS chip to adapt to environmental and user variations using minimal labeled data and weight updates. Fabricated in UMC 40nm CMOS technology, the chip achieves 10-keyword spotting accuracy ranging from 87.5% to 92.2% under background noise levels of 5dB to 20dB. Through on-device transfer learning, the chip can improve KWS accuracy on personalized datasets by 11.73%, using only a few examples per keyword and limited training epochs. The simulated power consumption of the chip in streaming inference mode is 10.26 μW at a 250 kHz clock frequency and 0.85V supply voltage. In conclusion, this thesis presents an end-to-end spiking neural network-based speech system for speech trigger (WWD) and speech command recognition (KWS) tasks. By integrating the AuFEx-TD and the SCNN accelerator, as described in the first and second parts of the thesis, a low-latency, sub-μW wake word detection system has been developed. In the final part, a noise-robust, personalized, few-μW-level speech command recognition ASIC is introduced, demonstrating the feasibility of high-performance, low-power AIoT speech interfaces.

This brief presents an innovative reconfigurable parametric model of a digital Spiking Neuron (SN). The proposed neuron model is based on the classical Leaky Integrate and Fire (LIF) model with some modifications, allowing neuron configuration using seven different leak modes and three activation functions with a dynamic threshold setting. A complementary online learning model based on adjustable Spike Timing Dependent Plasticity (STDP) learning rules has been developed as part of the proposed neuron architecture. Efficient hardware implementation of the proposed SN significantly reduces area and power costs. The proposed SN model consumes less than 10nW and requires only 700 ASIC 2-input NAND gates for implementation using ten neuron-inputs. Simulation results show an average power consumption of about 3.5 mW/cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . Simulation of the proposed digital SN demonstrates its ability to replicate accurately the behavior of a biological neuron model.

An adaptive threshold mechanism for accurate and efficient deep spiking convolutional neural networks

As Moore's law ends, the conventional von Neumann computer architecture with binary-coded data representation has reached its bottleneck because of having separate computing and memory modules. In this architecture, continuous power is required due to sequential processing, making it challenging to improve efficiency further. The human brain can be regarded as the most energy-efficient computer architecture. In the brain, data is represented as small voltage pulses as spikes. That is why the neural units are called spiking neural networks (SNN). In SNNs, energy is required only when there is a spike, making it more energy efficient than the von-Neumann computer architecture. Therefore, in recent decades, researchers in this field have been interested in mimicking the data processing of the human brain in electronic circuits. In biological SNNs, data is propagated from one neuron to another via a synapse. When there is an incoming spike, the chemical weight changes in the synapse, and the next neuron receives the signal. In the electronic neural network, a memristor, a two-terminal nonvolatile memory element, can emulate the function of a synapse by changing its conductance while receiving a pulse. The most common learning rule in the spiking neural network is Spike-Timing-Dependent Plasticity (STDP), which refers to the change of synaptic weight with respect to the time difference between pre- and post-synaptic neural spikes. Although electronic SNNs comprised of memristors with the STDP learning rule have shown promising performance in various event-driven tasks, circuit complexity limitations and a lack of third-order parametric inclusion exist. A solution can be using a simple circuit element to modulate the memristor response.

A convolutional spiking neural network with adaptive coding for motor imagery classification

A forecast-based STDP rule suitable for neuromorphic implementation

Spiking neural networks (SNNs), inspired by the neuronal network in the brain, provide biologically relevant and low-power consuming models for information processing. Existing studies either mimic the learning mechanism of brain neural networks as closely as possible, for example, the temporally local learning rule of spike-timing-dependent plasticity (STDP), or apply the gradient descent rule to optimize a multilayer SNN with fixed structure. However, the learning rule used in the former is local and how the real brain might do the global-scale credit assignment is still not clear, which means that those shallow SNNs are robust but deep SNNs are difficult to be trained globally and could not work so well. For the latter, the nondifferentiable problem caused by the discrete spike trains leads to inaccuracy in gradient computing and difficulties in effective deep SNNs. Hence, a hybrid solution is interesting to combine shallow SNNs with an appropriate machine learning (ML) technique not requiring the gradient computing, which is able to provide both energy-saving and high-performance advantages. In this article, we propose a HybridSNN, a deep and strong SNN composed of multiple simple SNNs, in which data-driven greedy optimization is used to build powerful classifiers, avoiding the derivative problem in gradient descent. During the training process, the output features (spikes) of selected weak classifiers are fed back to the pool for the subsequent weak SNN training and selection. This guarantees HybridSNN not only represents the linear combination of simple SNNs, as what regular AdaBoost algorithm generates, but also contains neuron connection information, thus closely resembling the neural networks of a brain. HybridSNN has the benefits of both low power consumption in weak units and overall data-driven optimizing strength. The network structure in HybridSNN is learned from training samples, which is more flexible and effective compared with existing fixed multilayer SNNs. Moreover, the topological tree of HybridSNN resembles the neural system in the brain, where pyramidal neurons receive thousands of synaptic input signals through their dendrites. Experimental results show that the proposed HybridSNN is highly competitive among the state-of-the-art SNNs.

Spiking Neural Networks (SNNs), widely known as the third generation of neural networks, encode input information temporally using sparse spiking events, which can be harnessed to achieve higher computational efficiency for cognitive tasks. However, considering the rapid strides in accuracy enabled by state-of-the-art Analog Neural Networks (ANNs), SNN training algorithms are much less mature, leading to accuracy gap between SNNs and ANNs. In this paper, we propose different SNN training methodologies, varying in degrees of biofidelity, and evaluate their efficacy on complex image recognition datasets. First, we present biologically plausible Spike Timing Dependent Plasticity (STDP) based deterministic and stochastic algorithms for unsupervised representation learning in SNNs. Our analysis on the CIFAR-10 dataset indicates that STDP-based learning rules enable the convolutional layers to self-learn low-level input features using fewer training examples. However, STDP-based learning is limited in applicability to shallow SNNs (≤4 layers) while yielding considerably lower than state-of-the-art accuracy. In order to scale the SNNs deeper and improve the accuracy further, we propose conversion methodology to map off-the-shelf trained ANN to SNN for energy-efficient inference. We demonstrate 69.96% accuracy for VGG16-SNN on ImageNet. However, ANN-to-SNN conversion leads to high inference latency for achieving the best accuracy. In order to minimize the inference latency, we propose spike-based error backpropagation algorithm using differentiable approximation for the spiking neuron. Our preliminary experiments on CIFAR-10 show that spike-based error backpropagation effectively captures temporal statistics to reduce the inference latency by up to 8× compared to converted SNNs while yielding comparable accuracy

Neuromorphic speech recognition systems that use spiking neural networks (SNNs) and memristors are progressing in hardware development. The conventional manual preprocessing of audio signals is shifting toward event-based recognition with convolutional SNNs. Despite achieving high accuracy in classification, the efficient extraction of spatiotemporal features from audio events continues to be a substantial challenge. In this study, we introduce dynamic time-surface neurons (DTSNs) using volatile memristors featuring an adjustable temporal kernel decay, enabled by series-connected transistors with an Au/LiCoO 2 /Au configuration. DTSNs act as feature descriptors, enhancing the spatiotemporal feature extraction from event audio data. A two-layer SNN classifier, fully connected and incorporating a 1T1R nonvolatile memristor array, is trained to recognize the spatiotemporal features of the audio data. Our findings show classification accuracies of up to 95.91%, substantial improvements in computational efficiency, and increased noise resilience, confirming the promise of our memristor-based speech recognition system for practical applications.

Spike timing dependent plasticity (STDP) learning rule is one of hot topics in neurobiology since it's been widely believed that synaptic plasticity mainly contribute to learning and memory in brain. Up to now, STDP has been observed in a wide variety of areas of brain, hippocampus, cortex and so on. Competition among synapses is an important behavior for this learning rule. In present study, we propose a single layer spiking neural network model using STDP learning rule in inhibitory synapses to investigate the competitive behavior. The experiments show that the synapses among neurons are both strengthened on the whole training process. Thus neurons inhibit the activities of one another, eventually the neuron with the highest input spike rate win the competition. We have found that the behavior is efficient when the differences of firing rates of input neurons without STDP are great than 5Hz, otherwise the winner neuron is random. In order to use the principle to artificial intelligent system, we use a mechanism of dynamic learning rate to let the neuron with the highest input to be selected by the competitive behavior as the winner. Therefore, a robust competitive spiking neural network is obtained.