Line-based event preprocessing: towards low-energy neuromorphic computer vision

Machine learning applications that are implemented with spike-based computation model, e.g., Spiking Neural Network (SNN), have a great potential to lower the energy consumption when they are executed on a neuromorphic hardware. However, compiling and mapping an SNN to the hardware is challenging, especially when compute and storage resources of the hardware (viz. crossbar) need to be shared among the neurons and synapses of the SNN. We propose an approach to analyze and compile SNNs on a resource-constrained neuromorphic hardware, providing guarantee on key performance metrics such as execution time and throughput. Our approach makes the following three key contributions. First, we propose a greedy technique to partition an SNN into clusters of neurons and synapses such that each cluster can fit on to the resources of a crossbar. Second, we exploit the rich semantics and expressiveness of Synchronous Dataflow Graphs (SDFGs) to represent a clustered SNN and analyze its performance using Max-Plus Algebra, considering the available compute and storage capacities, buffer sizes, and communication bandwidth. Third, we propose a self-timed execution-based fast technique to compile and admit SNN-based applications to a neuromorphic hardware at run-time, adapting dynamically to the available resources on the hardware. We evaluate our approach with standard SNN-based applications and demonstrate a significant performance improvement compared to current practices.

Spiking neural network (SNN) has drawn research interests as it mimics dynamic activities of human brain and has the potential to perform real-time cognitive tasks. However, latency, throughput and flexibility of existing hardware implemented SNNs are limited. The conventional rate coding is inefficient in terms of accuracy and latency. Oversimplified SNN models adopted by neuromorphic hardware discard characteristics such as neuron dynamics and filter effects etc., which are critical for neural information processing. Recent research advancements show that the potential of SNN can be better utilized by moving beyond rate-based model and considering temporal information embedded in the spike sequences. However, these works employ complex biologically realistic SNN models, posing challenges to hardware complexity. Furthermore, most existing neuromorphic hardware are developed for specific SNN models, or aiming at replicating biological behaviors. There is a lack of general methodology for SNN design optimization. Novel hardware architecture and systematic optimization techniques are required for efficient FPGA implementation and support flexible SNN models. To address above issues, in this work we proposed a holistic optimization framework for encoder, model, and architecture design of FPGA based neuromorphic hardware. We present an efficient neural coding scheme and training algorithm, which can optimize encoder parameters to enable fast inference. A flexible and hardware-friendly model is proposed, in which SNNs are represented as a network of Infinite Impulse Response (IIR) filters. Finally, an end-to-end framework is developed to optimize and deploy FPGA implementation. Experimental results show our work achieves state-of-the-art accuracy in various classification tasks, and outperforms various platforms including CPU, GPU and dedicated neuromorphic processors in terms of latency and throughput.

To improve the robustness and to reduce the energy consumption of facial expression recognition, this study proposed a facial expression recognition method based on improved deep residual networks (ResNets). Residual learning has solved the degradation problem of deep Convolutional Neural Networks (CNNs); therefore, in theory, a ResNet can consist of infinite number of neural layers. On the one hand, ResNets benefit from better performance on artificial intelligence (AI) tasks, thanks to its deeper network structure; meanwhile, on the other hand, it faces a severe problem of energy consumption, especially on mobile devices. Hence, this study employs a novel activation function, the Noisy Softplus (NSP), to replace rectified linear units (ReLU) to get improved ResNets. NSP is a biologically plausible activation function, which was first proposed in training Spiking Neural Networks (SNNs); thus, NSP-trained models can be directly implemented on ultra-low-power neuromorphic hardware. We built an 18-layered ResNet using NSP to perform facial expression recognition across datasets Cohn-Kanade (CK+), Karolinska Directed Emotional Faces (KDEF) and GENKI-4K. The results achieved better anti-noise ability than ResNet using the activation function ReLU and showed low energy consumption running on neuromorphic hardware. This study not only contributes a solution for robust facial expression recognition, but also consolidates the low energy cost of their implementation on neuromorphic devices, which could pave the way for high-performance, noise-robust and energy-efficient vision applications on mobile hardware.

As the desire for biologically realistic spiking neural networks (SNNs) increases, tuning the enormous number of open parameters in these models becomes a difficult challenge. SNNs have been used to successfully model complex neural circuits that explore various neural phenomena such as neural plasticity, vision systems, auditory systems, neural oscillations, and many other important topics of neural function. Additionally, SNNs are particularly well-adapted to run on neuromorphic hardware that will support biological brain-scale architectures. Although the inclusion of realistic plasticity equations, neural dynamics, and recurrent topologies has increased the descriptive power of SNNs, it has also made the task of tuning these biologically realistic SNNs difficult. To meet this challenge, we present an automated parameter tuning framework capable of tuning SNNs quickly and efficiently using evolutionary algorithms (EA) and inexpensive, readily accessible graphics processing units (GPUs). A sample SNN with 4104 neurons was tuned to give V1 simple cell-like tuning curve responses and produce self-organizing receptive fields (SORFs) when presented with a random sequence of counterphase sinusoidal grating stimuli. A performance analysis comparing the GPU-accelerated implementation to a single-threaded central processing unit (CPU) implementation was carried out and showed a speedup of 65× of the GPU implementation over the CPU implementation, or 0.35 h per generation for GPU vs. 23.5 h per generation for CPU. Additionally, the parameter value solutions found in the tuned SNN were studied and found to be stable and repeatable. The automated parameter tuning framework presented here will be of use to both the computational neuroscience and neuromorphic engineering communities, making the process of constructing and tuning large-scale SNNs much quicker and easier.

Hardware implementation of neuromorphic computing can significantly improve performance and energy efficiency of machine learning tasks implemented with spiking neural networks (SNNs), making these hardware platforms particularly suitable for embedded systems and other energy-constrained environments. We observe that the long bitlines and wordlines in a crossbar of the hardware create significant current variations when propagating spikes through its synaptic elements, which are typically designed with non-volatile memory (NVM). Such current variations create a thermal gradient within each crossbar of the hardware, depending on the machine learning workload and the mapping of neurons and synapses of the workload to these crossbars. This thermal gradient becomes significant at scaled technology nodes and it increases the leakage power in the hardware leading to an increase in the energy consumption. We propose a novel technique to map neurons and synapses of SNN-based machine learning workloads to neuromorphic hardware. We make two novel contributions. First, we formulate a detailed thermal model for a crossbar in a neuromorphic hardware incorporating workload dependency, where the temperature of each NVM-based synaptic cell is computed considering the thermal contributions from its neighboring cells. Second, we incorporate this thermal model in the mapping of neurons and synapses of SNN-based workloads using a hill-climbing heuristic. The objective is to reduce the thermal gradient in crossbars. We evaluate our neuron and synapse mapping technique using 10 machine learning workloads for a state-of-the-art neuromorphic hardware. We demonstrate an average 11.4K reduction in the average temperature of each crossbar in the hardware, leading to a 52% reduction in the leakage power consumption (11% lower total energy consumption) compared to a performance-oriented SNN mapping technique.KeywordsNeuromorphic computingSpiking Neural NetworkNon-Volatile Memory (NVM)Phase-Change Memory (PCM)TemperatureLeakage power consumptionCrossbar

Recent advancements in spiking neural networks (SNNs) have highlighted their advantages, including energy efficiency, real-time processing, and compatibility with neuromorphic hardware. These features make SNNs particularly well-suited for human-robot interaction (HRI) in rehabilitative wearable robotics, where real-time adaptability and low power consumption are essential. However, there is still a lack of comprehensive reviews on SNNs’ application to HRI. This paper addresses this gap by providing a detailed overview of the latest advancements in SNNs from the perspective of embodied intelligence in rehabilitative wearable robots. We systematically examine recent progress in SNNs, including spiking neuron models, encoding methods, and learning mechanisms. These advancements are then analyzed with a focus on HRI, addressing specific challenges in rehabilitative wearable robots from three key perspectives: human motion decoding, robotic control, and neuromorphic implementation for embedded systems. By reviewing current research, this paper highlights the potential benefits and limitations of SNNs in achieving embodied intelligence and identifies crucial areas for further investigation, offering new insights and directions for their future applications in rehabilitative wearable robotics.

Spiking neural networks (SNNs) are the basis for many energy-efficient neuromorphic hardware systems. While there has been substantial progress in SNN research, artificial SNNs still lack many capabilities of their biological counterparts. In biological neural systems, memory is a key component that enables the retention of information over a huge range of temporal scales, ranging from hundreds of milliseconds up to years. While Hebbian plasticity is believed to play a pivotal role in biological memory, it has so far been analyzed mostly in the context of pattern completion and unsupervised learning in artificial and SNNs. Here, we propose that Hebbian plasticity is fundamental for computations in biological and artificial spiking neural systems. We introduce a novel memory-augmented SNN architecture that is enriched by Hebbian synaptic plasticity. We show that Hebbian enrichment renders SNNs surprisingly versatile in terms of their computational as well as learning capabilities. It improves their abilities for out-of-distribution generalization, one-shot learning, cross-modal generative association, language processing, and reward-based learning. This suggests that powerful cognitive neuromorphic systems can be built based on this principle.

Epilepsy is a global chronic disease that brings pain and inconvenience to patients, and an electroencephalogram (EEG) is the main analytical tool. For clinical aid that can be applied to any patient, an automatic cross-patient epilepsy seizure detection algorithm is of great significance. Spiking neural networks (SNNs) are modeled on biological neurons and are energy-efficient on neuromorphic hardware, which can be expected to better handle brain signals and benefit real-world, low-power applications. However, automatic epilepsy seizure detection rarely considers SNNs. In this article, we have explored SNNs for cross-patient seizure detection and discovered that SNNs can achieve comparable state-of-the-art performance or a performance that is even better than artificial neural networks (ANNs). We propose an EEG-based spiking neural network (EESNN) with a recurrent spiking convolution structure, which may better take advantage of temporal and biological characteristics in EEG signals. We extensively evaluate the performance of different SNN structures, training methods, and time settings, which builds a solid basis for understanding and evaluation of SNNs in seizure detection. Moreover, we show that our EESNN model can achieve energy reduction by several orders of magnitude compared with ANNs according to the theoretical estimation. These results show the potential for building high-performance, low-power neuromorphic systems for seizure detection and also broaden real-world application scenarios of SNNs.

Spiking neural network (SNN) distinguish themselves from artificial neural network (ANN) because of their inherent temporal processing and spike-based computations, enabling a power-efficient implementation in neuromorphic hardware. In this study, we demonstrate that data processing with spiking neurons can be enhanced by co-learning the synaptic weights with two other biologically inspired neuronal features: (1) a set of parameters describing neuronal adaptation processes and (2) synaptic propagation delays. The former allows a spiking neuron to learn how to specifically react to incoming spikes based on its past. The trained adaptation parameters result in neuronal heterogeneity, which leads to a greater variety in available spike patterns and is also found in the brain. The latter enables to learn to explicitly correlate spike trains that are temporally distanced. Synaptic delays reflect the time an action potential requires to travel from one neuron to another. We show that each of the co-learned features separately leads to an improvement over the baseline SNN and that the combination of both leads to state-of-the-art SNN results on all speech recognition datasets investigated with a simple 2-hidden layer feed-forward network. Our SNN outperforms the benchmark ANN on the neuromorphic datasets (Spiking Heidelberg Digits and Spiking Speech Commands), even with fewer trainable parameters. On the 35-class Google Speech Commands dataset, our SNN also outperforms a GRU of similar size. Our study presents brain-inspired improvements in SNN that enable them to excel over an equivalent ANN of similar size on tasks with rich temporal dynamics.

Spiking neural networks (SNNs) have shown great potential as a solution for realizing ultralow-power consumption on neuromorphic hardware, but obtaining deep SNNs is still a challenging problem. Existing network conversion methods can effectively obtain SNNs from the trained convolutional neural networks (CNNs) with little performance loss, however, high-precision weights in the converted SNNs would take up high-storage space nonamicable to limited memory resources. To tackle this problem, we analyze the relationship between weights and thresholds of spiking neurons and propose an efficient weights-threshold balance conversion method to obtain SNNs with binary weights, resulting in a significant memory storage reduction. The experimental results evaluated with various network structures on benchmark data sets show that the binary SNN not only needs much less memory resources compared to its high-precision counterpart but also achieves the high-recognition accuracy comparable to other state-of-the-art SNNs.

A forecast-based STDP rule suitable for neuromorphic implementation

Bio-inspired Active Learning method in spiking neural network

Efficiently solving combinatorial optimization problems (COPs) such as Max-Cut is challenging because the resources required increase exponentially with the problem size. This study proposes a hardware-friendly method for solving the Max-Cut problem by implementing a spiking neural network (SNN)-based Boltzmann machine (BM) in neuromorphic hardware systems. To implement the hardware-oriented version of the spiking Boltzmann machine (sBM), the stochastic dynamics of leaky integrate-and-fire (LIF) neurons with random walk noise are analyzed, and an innovative algorithm based on overlapping time windows is proposed. The simulation results demonstrate the effective convergence and high accuracy of the proposed method for large-scale Max-Cut problems. The proposed method is validated through successful hardware implementation on a 6-transistor/2-resistor (6T2R) neuromorphic chip with phase change memory (PCM) synapses. In addition, as an expansion of the algorithm, several annealing techniques and bias split methods are proposed to improve convergence, along with circuit design ideas for efficient evaluation of sampling convergence using cell arrays and spiking systems. Overall, the results of the proposed methods demonstrate the potential of energy-efficient and hardware-implementable approaches using SNNs to solve COPs. To the best of the author's knowledge, this is the first study to solve the Max-Cut problem using an SNN neuromorphic hardware chip.

Neuro-evolutionary algorithms optimize the synaptic connectivity of sets of candidate neural networks based on a task-dependent fitness function. Compared to the commonly used methods from machine learning, many of them not only support the adaptation of connection weights but also of the network topology. However, the evaluation of the current fitness requires running every candidate network in every generation. This becomes a major impediment especially when using biologically inspired spiking neural networks which require considerable amounts of simulation time even on powerful computers. In this paper, we address this issue by offloading the network simulation to SpiNNaker, a state-of-the art neuromorphic hardware architecture which is capable of simulating large spiking neural networks in biological real-time. We were able to apply SpiNNaker's simulation power to the popular NEAT algorithm by running all candidate networks in parallel and successfully evolved spiking neural networks for solving the XOR problem and for playing the Pac-Man arcade game

Monolithic 3D integration of neuron and synapse devices is considered a promising solution for energy-efficient and compact neuromorphic hardware. However, achieving optimal performance in both training and inference remains challenging as these processes require different synapse devices with reliable endurance and long retention. Here, we introduce a decoupling strategy to separate training and inference using monolithically integrated neuromorphic hardware with layer-by-layer fabrication. This 3D neuromorphic hardware includes neurons consisting of a single transistor (1T-neuron) in the first layer, long-term operational synapses composed of a single thin-film transistor with a SONOS structure (1TFT-synapses) in the second layer for inference, and durable synapses composed of a memristor (1M-synapses) in the third layer for training. A 1TFT-synapse, utilizing a charge-trap layer, exhibits long retention properties favorable for inference tasks. In contrast, a 1M-synapse, leveraging anion movement at the interface, demonstrates robust endurance for repetitive weight updates during training. With the proposed hybrid synapse architecture, frequent training can be performed using the 1M-synapses with robust endurance, while intermittent inference can be managed using the 1TFT-synapses with long-term retention. This decoupling of synaptic functions is advantageous for achieving a reliable spiking neural network (SNN) in neuromorphic computing.