An Asynchronous Analog-Computing Spiking Neural Network With Improved Tolerance to Nonidealities for Always-On Near-Sensor AI

In recent years, DNN-based systems are extremely popular for performing automatic speech recognition (ASR) task. They have shown better performance than other methods. To perform ASR task efficiently, segmentation of the input data also has to be accurate. There are different kinds of methods of voice activity detection (VAD) including power based statistical methods. But to incorporate with the ASR system, it is better to perform the VAD task with the help of DNN-based method. In this paper, we investigate the superiority of DNN-based VAD over power based statistical methods. Not only that but also the effect of data augmentation on the performance of VAD in various conditions is also investigated. Performance of the VAD is evaluated using CENSREC-1-C data especially developed for evaluation of VAD task. The VAD trained with multi-condition data shows better performance than the baseline and other popular power based statistical VAD tools. We have also evaluated the performance for eval1 test dataset of CSJ corpus as well as its telephone variants.

Voice activity detection (VAD) determines whether the incoming signal segments are speech or noiseand is an important technique in almost all of speech-related applications. In order to improve VAD performance in various noise environments, characterizing the speech feature has been the most crucial issue up to date. Among several proposed speech features, the context information of speech through time and vowel sound characteristics are known to current state-of-the-art speech features. Therefore, in order to reflect both on these merits, we propose vowel based VAD by Long short term memory recurrent neural network (LSTM-RNN). LSTM-RNN is known to the powerful model to capture dynamical context information through time. Moreover, with teaching the LSTM-RNN to only vowel sounds rather than whole speech, LSTM-RNN can learn more effectively because of the reduced manifold of speech. According to our experiments, proposed method shows better accuracy not only in the VAD task compared to LSTM-RNN based VAD but alsoa vowel detection task.

In this paper, a robust multiple observation likelihood ratio test (MOLRT) based voice activity detection (VAD) method is proposed. At the beginning of this paper, we introduce the Wiener filter to the observed signal in time domain which can help mitigating some noise. The reason why we use the Wiener filter for VAD is that the performance of the VAD is always better in high signal to noise ratio (SNR) range than in low SNR range. Then, some ideas are proposed to improve the performance of the MOLRT based VAD method. As we all know, a conventional MOLRT based method including three modules named likelihood ratio (LR) estimation, threshold setting and hangover technique. To improve the estimation accuracy of LR, we adopt the unbiased minimum mean-square error (MMSE) algorithm for noise power spectrum estimation in every frame, which is very effective for LR estimation in MOLRT-based VAD method. That is because the LR is a function of a prior and a posterior SNR and unbiased MMSE algorithm is very useful for noise estimation. In addition, to make our VAD method more robust, a dynamic threshold setting technique is proposed in our method, which is related to the minimum noise power spectrum. That is because minimum noise power spectrum can help us updating the value of threshold to a suitable level according to the denoised signal. Last but most important, a novel hangover algorithm is introduced in this paper comparing to the conventional HMM based hangover algorithm. In the novel hangover algorithm, the current frame is determined by the statistical result of the following speech/non-speech detections based on the likelihood ratio test. And the evaluation results reveal that proposed method significantly outperform the baseline result of LRT as regards VAD accuracy in both noise variations and low SNR conditions.

This paper presents a Spiking Neural Network (SNN) processor specifically designed to overcome the limitations of existing parallel architectures in maintaining high energy efficiency and model adaptability in a compact area footprint for Artificial Intelligence of Things (AIoT). This is achieved through two key design features: a Temporal-Spatial Post-Neuron Processing (PoNP) scheme that efficiently reuses membrane potential, maximizes parallelism, and reduces memory bank requirements; and a Model-Adaptive Crossbar design with pre-configured parameters and a dynamic switching mechanism enables processing of various SNN models through operation orchestration without efficiency degradation. Using an 8-way parallel pipeline design, the processor achieves a throughput of 128 Synaptic Operations (SOPs) per cycle, resulting in a 2.8$\boldsymbol{\times}$ enhancement in energy efficiency. Fabricated in a 40-nm CMOS process, the chip occupies a compact core area of 0.66 mm${}^{2}$. It achieves a power consumption of 6.26 mW, an energy efficiency of 0.49 pJ/SOP, and a throughput of 12.8 GSOPS at 0.75 V, 100 MHz. The chip is evaluated using typical spatial, temporal, and temporal-spatial datasets, including MIT-BIH, MNIST, N-MNIST, NavGesture, and SHD. These results demonstrate that our chip achieves best-in-class in terms of energy efficiency and latency compared to state-of-the-art architectures.

ReRAM is a promising non-volatile memory for neuromor-phic accelerators, yet it faces challenges such as high sensing power and accuracy degradation. This work proposes BiNeuroRAM, a novel spiking neural network (SNN) accelerator leveraging ReRAM-based processing-in-memory (PIM), with three key contributions: (1) It is the first to support higher-accuracy spike-tracing bipolar-integrate-and-fire (ST-BIF) neurons, achieving 80.9% accuracy on ImageNet, 8.4% higher than the previous state-of-the-art; (2) It introduces a low-power voltage sense amplifier (LPVSA) that reduces ReRAM read power by 14.7~58.2×, enhancing energy efficiency; (3) It employs an asynchronous micro-architecture that fully exploits the event-driven nature of SNNs. Experimental results show that BiNeuroRAM improves throughput density and energy efficiency by 2.08× and 2.09× on ImageNet with ResNet-18, compared to traditional integrate-and-fire (IF) neuron-based SNN accelerators.

This paper proposes a novel fully neural network based voice activity detection (VAD) method that estimates whether each speech segment is speech or non-speech even in very low signal-to-noise ratio (SNR) environments. Our innovation is to improve context-awareness of speech variability by introducing multiple auxiliary networks into the neural VAD framework. While previous studies reported that phonetic-aware auxiliary features extracted from a phoneme recognition network can improve VAD performance, none examined other effective auxiliary features for enhancing noise robustness. Thus, this paper present a neural VAD that uses auxiliary features extracted from not only the phoneme recognition network but also a speech enhancement network and an acoustic scene classification network. The last two networks are expected to improve context-awareness even in extremely low SNR environments since they can extract de-noised speech awareness and noisy environment awareness. In addition, we expect that combining these multiple auxiliary features yield synergistic improvements in VAD performance. Experiments verify the superiority of the proposed method in very low SNR environments.

In order to develop a novel voice sensor to detect human voices, the use of features which are more robust to noise is an important issue. Voice sensor is also called voice activity detection (VAD). Due to that the inherent nature of the formant structure only occurred on the speech spectrogram (well-known as voiceprint), Wu et al. were the first to use band-spectral entropy (BSE) to describe the characteristics of voiceprints. However, the performance of VAD based on BSE feature was degraded in colored noise (or voiceprint-like noise) environments. In order to solve this problem, we propose the two-dimensional part-band energy entropy (TD-PBEE) parameter based on two variables: part-band partition number upon frequency index and long-term window size upon time index to further improve the BSE-based VAD algorithm. The two variables can efficiently represent the characteristics of voiceprints on each critical frequency band and use long-term information for noisy speech spectrograms, respectively. The TD-PBEE parameter can be regarded as a PBEE parameter over time. First, the strength of voiceprints can be partly enhanced by using four entropies applied to four part-bands. We can use the four part-band energy entropies for describing the voiceprints in detail. Due to the characteristics of non-stationary for speech and various noises, we will then use long-term information processing to refine the PBEE, so the voice-like noise can be distinguished from noisy speech through the concept of PBEE with long-term information. Our experiments show that the proposed feature extraction with the TD-PBEE parameter is quite insensitive to background noise. The proposed TD-PBEE-based VAD algorithm is evaluated for four types of noises and five signal-to-noise ratio (SNR) levels. We find that the accuracy of the proposed TD-PBEE-based VAD algorithm averaged over all noises and all SNR levels is better than that of other considered VAD algorithms.

S[formula omitted]NN: Time step reduction of spiking surrogate gradients for training energy efficient single-step spiking neural networks

Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Compared with counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8×/7.4×. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNN research, with a great balance between effectiveness and energy efficiency.

This paper proposes two algorithms for Voice Activity Detection (VAD) based on sparse representation in spectro-temporal domain. The first algorithm was made using two-dimensional STRF (Spectro-Temporal Response Field) space based on sparse representation. Dictionaries with different atomic sizes and two dictionary learning methods were investigated in this approach. This algorithm revealed good results at high SNRs (signal-to-noise ratio). The second algorithm, whose approach is more complicated, suggests a speech detector using the sparse representation in four-dimensional STRF space. Due to the large volume of STRF's four-dimensional space, this space was divided into cubes, with dictionaries made for each cube separately by NMF (non-negative matrix factorization) learning algorithm. Simulation results were presented to illustrate the effectiveness of our new VAD algorithms. The results revealed that the achieved performance was 90.11% and 91.75% under -5 dB SNR in white and car noise respectively, outperforming most of the state-of-the-art VAD algorithms.

A new method was proposed to identify speech-segment endpoints based on the empirical mode decomposition (EMD) and a new wavelet entropy ratio with improving the accuracy of voice activity detection. With the EMD, the noise signals can be decomposed into several intrinsic mode functions (IMFs). Then the proposed wavelet energy entropy ratio can be used to extract the desired feature for each IMFs component. In view of the question that the method of voice endpoint detection based on the original wavelet entropy ratio cannot adapt to the low signal-to-noise ratio (SNR) condition, an appropriate positive constant was introduced to the basic wavelet energy entropy ratio with effectively improved discriminability between the speech and noise. After comparing the traditional wavelet energy entropy ratio with the proposed wavelet energy entropy ratio, the experiment results show that the proposed method is simple and fast. The speech endpoints can be accurately detected in low SNR environments.

Spiking neural networks (SNN) are computational models inspired by the brain’s ability to naturally encode and process information in the time domain. The added temporal dimension is believed to render them more computationally efficient than the conventional artificial neural networks, though their full computational capabilities are yet to be explored. Recently, in-memory computing architectures based on non-volatile memory crossbar arrays have shown great promise to implement parallel computations in artificial and spiking neural networks. In this work, we evaluate the feasibility to realize high-performance event-driven in-situ supervised learning systems using nanoscale and stochastic analog memory synapses. For the first time, the potential of analog memory synapses to generate precisely timed spikes in SNNs is experimentally demonstrated. The experiment targets applications which directly integrates spike encoded signals generated from bio-mimetic sensors with in-memory computing based learning systems to generate precisely timed control signal spikes for neuromorphic actuators. More than 170,000 phase-change memory (PCM) based synapses from our prototype chip were trained based on an event-driven learning rule, to generate spike patterns with more than 85% of the spikes within a 25 ms tolerance interval in a 1250 ms long spike pattern. We observe that the accuracy is mainly limited by the imprecision related to device programming and temporal drift of conductance values. We show that an array level scaling scheme can significantly improve the retention of the trained SNN states in the presence of conductance drift in the PCM. Combining the computational potential of supervised SNNs with the parallel compute power of in-memory computing, this work paves the way for next-generation of efficient brain-inspired systems.

In this paper, a novel approach to apply GM(1,1) model in voice activity detection (VAD) is presented. The approach is termed as grey VAD (GVAD). In GVAD, GM(1,1) model is used to estimate noise in noisy speech and therefore signal where the additive signal model is assumed. By estimated noise and signal, the signal-to-noise ratio (SNR) is calculated. Based on an adaptive threshold, speech and non-speech segments are determined. The proposed GVAD is performed in the time-domain and thus has low computational complexity. In the simulation, GVAD is verified by cases with non-stationary additive white Gaussian noise and is compared with VAD in G 729 and GSM AMR. The results indicate that the proposed GVAD is able to detect voice activity appropriately. In the given examples, the performance of GVAD is better than VAD in G 729 and GSM AMR.

Improved likelihood ratio test based voice activity detector applied to speech recognition

Speech trigger and speech command recognition technologies have become pivotal human-machine interfaces for AIoT applications. Typically, a speech-related system consists of a mel-filter-based handcrafted feature extractor and an artificial neuron-based classifier. Recent advancements [1–3] have demonstrated promising results by training the feature extractor jointly with the neural network classifier. Moreover, the emergence of spiking neuron network training algorithms and hardware designs has showcased their competitive edge over traditional artificial neural networks. Consequently, the design and training of learnable audio feature extractors and spiking neural network classifiers are the key focuses of this thesis. The first focus of this thesis is the design of an ultra-low-power, learnable audio feature extractor to support end-to-end training. Conventional low-power approaches, such as extreme pruning and quantization in the audio feature extractor (AuFEx) design, often result in decreased accuracy. To address this, the thesis proposes a hardware-aware, end-to-end training framework to co-optimize the AuFEx design, achieving high performance with minimal hardware cost. Two types of AuFEx designs are explored: a time-domain (TD) filter-based learnable audio feature extractor (AuFEx-TD) and a frequency-domain (FD) filter-based learnable audio feature extractor (AuFEx-FD). Using this framework, the AuFEx-TD achieves high accuracy (97.8%) for one wake word detection, with low power consumption (496 nW) and low latency (375 μs). The optimized AuFEx-FD demonstrates noise-robust 10-keyword spotting accuracy (87.5%@5dB – 92.2%@20dB) with low power consumption (1.26 μW). Both AuFEx-TD and AuFEx-FD designs are implemented using UMC 40nm CMOS technology to validate the framework's effectiveness. For the speech trigger or wake word detection (WWD) task, power and latency are critical design considerations. This thesis presents a sub-μW spiking neural network accelerator with an 8 ms latency. The core WWD engine employs a spiking convolutional neural network (SCNN) model, leveraging sparse activation and addition-only operations within spiking neurons. The proposed SCNN model enhances the existing framewise incremental computation flow [4] by incorporating a spike processing unit (SPU), reducing system power and latency by 16.5% and 43.2%, respectively. Extensive quantization further reduces weight and activation precisions to 4-bit and 1-bit. Additionally, a power-gating mechanism, driven by an energy-based voice activity detection (VAD) module, further minimizes power consumption in random and sparse event (RSE) scenarios. Full chip simulation results show that the chip consumes only 110 nW, with a 2.15% false alarm rate and a 3.00% false reject rate in a 10% voice event stream test. It achieves state-of-the-art recognition accuracy of 99% and 96% for one and two wake word detection tasks, respectively. For speech command recognition or keyword spotting (KWS) tasks, robustness against noise and speaker accents is essential for user experience. This thesis explores both end-to-end training and on-device transfer learning to enhance KWS chip robustness with minimal power overhead. The proposed chip comprises three main blocks: the Learnable Audio Feature Extractor (AuFEx), the Spiking-DSCNN Core (SC), and the Transfer Learning Core (TLC). The AuFEx is an optimized AuFEx-FD, with the on-chip memory for storing FD-filter parameters replaced by an FD-filter parameter generator, reducing power consumption to 1.225 μW. The SC implements a spiking neural network (SNN) model using a Depthwise Separable Convolutional Neural Network (DSCNN) topology. Multi-level encoding in spiking neurons reduces simulation time, while the inherent activation sparsity of these neurons decreases power consumption and latency. The TLC supports both forward and backward propagation for a fully connected layer and softmax layer, enabling the KWS chip to adapt to environmental and user variations using minimal labeled data and weight updates. Fabricated in UMC 40nm CMOS technology, the chip achieves 10-keyword spotting accuracy ranging from 87.5% to 92.2% under background noise levels of 5dB to 20dB. Through on-device transfer learning, the chip can improve KWS accuracy on personalized datasets by 11.73%, using only a few examples per keyword and limited training epochs. The simulated power consumption of the chip in streaming inference mode is 10.26 μW at a 250 kHz clock frequency and 0.85V supply voltage. In conclusion, this thesis presents an end-to-end spiking neural network-based speech system for speech trigger (WWD) and speech command recognition (KWS) tasks. By integrating the AuFEx-TD and the SCNN accelerator, as described in the first and second parts of the thesis, a low-latency, sub-μW wake word detection system has been developed. In the final part, a noise-robust, personalized, few-μW-level speech command recognition ASIC is introduced, demonstrating the feasibility of high-performance, low-power AIoT speech interfaces.