Synaptic dynamics: Linear model and adaptation algorithm

Identification of acoustic signals in noisy environments remains one of the most difficult of signal processing problems, and is a major obstacle to the high degree of accuracy and speed needed to identify suspicious sounds in high-security, high-safety environments. We have previously developed an acoustic recognition capability using a novel, biologically based Dynamic Synapse Neural Network (DSNN) technology. The DSNN-based technology has been demonstrated to classify target sounds with a high degree of accuracy, even in high noise conditions. In this paper we focus on extending the acoustic recognition capability of DSNNs to the problem of gunshot recognition. In order to recognize and localize the event, an array of four microphones is used for sound input. For localization purpose, time-delay estimation algorithm (TDE) is employed for triangulation. We have developed stand-alone, portable, and cost-efficient hardware with which both recognition and localization can be performed. In field-testing, the system classifies and localizes over 90% of the trained-for sounds. Performance for firecracker, starter pistol, 9-mm, and 44-caliber, explosion/firing sounds was also tested.

We have developed a speaker identification system that performs well under two situations that current identification system encounter difficulties with: when the speaker's voice is combined with interfering noise, and when multiple speakers have similar voices. Our system consists of several serial processors. First, four Chebyshev type 1 filters separate the sound signal into four overlapping spectral ranges. Each of the four signals is passed through a wavelet filter that utilizes a 3 level, Daubechies 4th order discrete wavelet transformation, generating one approximation and three detailed coefficient arrays. The resulting 16 signals are input for two gender-specific dynamic synapse neural networks that have three outputs corresponding to each of the 3 male or 3 female speakers in our study. The energy in each of the pulse train outputs of the DSNN is compared, the output with the highest energy is identified, and the corresponding speaker is chosen. The choice of which DSNN to use is determined by a gender identifier that has been developed in our laboratory. The DSNNs were trained using a genetic algorithm training method.

Spiking Neural Networks (SNNs), widely known as the third generation of neural networks, encode input information temporally using sparse spiking events, which can be harnessed to achieve higher computational efficiency for cognitive tasks. However, considering the rapid strides in accuracy enabled by state-of-the-art Analog Neural Networks (ANNs), SNN training algorithms are much less mature, leading to accuracy gap between SNNs and ANNs. In this paper, we propose different SNN training methodologies, varying in degrees of biofidelity, and evaluate their efficacy on complex image recognition datasets. First, we present biologically plausible Spike Timing Dependent Plasticity (STDP) based deterministic and stochastic algorithms for unsupervised representation learning in SNNs. Our analysis on the CIFAR-10 dataset indicates that STDP-based learning rules enable the convolutional layers to self-learn low-level input features using fewer training examples. However, STDP-based learning is limited in applicability to shallow SNNs (≤4 layers) while yielding considerably lower than state-of-the-art accuracy. In order to scale the SNNs deeper and improve the accuracy further, we propose conversion methodology to map off-the-shelf trained ANN to SNN for energy-efficient inference. We demonstrate 69.96% accuracy for VGG16-SNN on ImageNet. However, ANN-to-SNN conversion leads to high inference latency for achieving the best accuracy. In order to minimize the inference latency, we propose spike-based error backpropagation algorithm using differentiable approximation for the spiking neuron. Our preliminary experiments on CIFAR-10 show that spike-based error backpropagation effectively captures temporal statistics to reduce the inference latency by up to 8× compared to converted SNNs while yielding comparable accuracy

The KBANN (Knowledge-Based Artificial Neural Networks) approach uses neural networks to refine knowledge that can be written in the form of simple propositional rules. This idea is extended by presenting the MANNIDENT (Multivariable Artificial Neural Network Identification) algorithm by which the mathematical equations of linear dynamic process models determine the topology and initial weights of a network, which is further trained using backpropagation. This method is applied to the task of modeling a nonisothermal chemical reactor in which a first-order exothermic reaction is occurring. This method produces statistically significant gains in accuracy over both a standard neural network approach and a linear model. Furthermore, using the approximate linear model to initialize the weights of the network produces statistically less variation in model fidelity. By structuring the neural network according to the approximate linear model, the model can be readily interpreted.

We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).

Spiking neural networks (SNN) are more realistic and powerful than the preceding generations of the neural networks (e.g. multi-layer perceptron networks). The SNN can be applied for simulating the brain and its functions, as well as it is able to be employed for different applications such as pattern classification. Different methods have been proposed for supervised training of SNN, however, most of them were validated based on using the classical XOR problem, and they consume long training time if other problems are considered. This paper proposes a new supervised training method for SNN by adapting the Enhanced-Mussels Wandering Optimization algorithm. In addition, a SNN model for pattern classification is proposed. The proposed work is used for pattern classification of real-world problems. The obtained results indicate that the proposed method is competitive alternative in terms of classification accuracy and training time.

The prediction accuracy and generalization ability of neural/neurofuzzy models for chaotic time series prediction highly depends on employed network model as well as learning algorithm. In this study, several neural and neurofuzzy models with different learning algorithms are examined for prediction of several benchmark chaotic systems and time series. The prediction performance of locally linear neurofuzzy models with recently developed Locally Linear Model Tree (LoLiMoT) learning algorithm is compared with that of Radial Basis Function (RBF) neural network with Orthogonal Least Squares (OLS) learning algorithm, MultiLayer Perceptron neural network with error back-propagation learning algorithm, and Adaptive Network based Fuzzy Inference System. Particularly, cross validation techniques based on the evaluation of error indices on multiple validation sets is utilized to optimize the number of neurons and to prevent over fitting in the incremental learning algorithms. To make a fair comparison between neural and neurofuzzy models, they are compared at their best structure based on their prediction accuracy, generalization, and computational complexity. The experiments are basically designed to analyze the generalization capability and accuracy of the learning techniques when dealing with limited number of training samples from deterministic chaotic time series, but the effect of noise on the performance of the techniques is also considered. Various chaotic systems and time series including Lorenz system, Mackey-Glass chaotic equation, Henon map, AE geomagnetic activity index, and sunspot numbers are examined as case studies. The obtained results indicate the superior performance of incremental learning algorithms and their respective networks, such as, OLS for RBF network and LoLiMoT for locally linear neurofuzzy model.

The capability of artificial neural networks to use and combine simple learning rules to master different complex tasks is very motivating and has sufficed on lots of challenging pattern recognition problems. Thus, in this paper, we selected a pattern recognition task that can be tough even for humans. The aim is to investigate the capability of neural networks to learn and recognize patterns describing some segmented chest organs in binary segmented images. The geometric descriptions in addition to the entropy of those images have been leveraged on in this work for classification purposes. Two learning algorithms are used for training both feedforward and competitive neural networks which have been considered for the pattern recognition task. Overall, we show that a competitive neural network that falls into unsupervised learning category outperformed the feedforward network which is considered a supervised learning network. This outperformance is in terms of time, error rate, cost, and accuracy.KeywordsArtificial neural networksPattern recognitionSegmented chest organsCompetitive neural networksUnsupervised learningSupervised learning

Listening to music often leads to physiological responses. Do these physiological responses contain sufficient information to infer emotion induced in the listener? The current study explores this question by attempting to predict judgments of “felt” emotion from physiological responses alone using linear and neural network models. We measured five channels of peripheral physiology from 20 participants—heart rate (HR), respiration, galvanic skin response, and activity in corrugator supercilii and zygomaticus major facial muscles. Using valence and arousal (VA) dimensions, participants rated their felt emotion after listening to each of 12 classical music excerpts. After extracting features from the five channels, we examined their correlation with VA ratings, and then performed multiple linear regression to see if a linear relationship between the physiological responses could account for the ratings. Although linear models predicted a significant amount of variance in arousal ratings, they were unable to do so with valence ratings. We then used a neural network to provide a non-linear account of the ratings. The network was trained on the mean ratings of eight of the 12 excerpts and tested on the remainder. Performance of the neural network confirms that physiological responses alone can be used to predict musically induced emotion. The non-linear model derived from the neural network was more accurate than linear models derived from multiple linear regression, particularly along the valence dimension. A secondary analysis allowed us to quantify the relative contributions of inputs to the non-linear model. The study represents a novel approach to understanding the complex relationship between physiological responses and musically induced emotion.

To improve the robustness and to reduce the energy consumption of facial expression recognition, this study proposed a facial expression recognition method based on improved deep residual networks (ResNets). Residual learning has solved the degradation problem of deep Convolutional Neural Networks (CNNs); therefore, in theory, a ResNet can consist of infinite number of neural layers. On the one hand, ResNets benefit from better performance on artificial intelligence (AI) tasks, thanks to its deeper network structure; meanwhile, on the other hand, it faces a severe problem of energy consumption, especially on mobile devices. Hence, this study employs a novel activation function, the Noisy Softplus (NSP), to replace rectified linear units (ReLU) to get improved ResNets. NSP is a biologically plausible activation function, which was first proposed in training Spiking Neural Networks (SNNs); thus, NSP-trained models can be directly implemented on ultra-low-power neuromorphic hardware. We built an 18-layered ResNet using NSP to perform facial expression recognition across datasets Cohn-Kanade (CK+), Karolinska Directed Emotional Faces (KDEF) and GENKI-4K. The results achieved better anti-noise ability than ResNet using the activation function ReLU and showed low energy consumption running on neuromorphic hardware. This study not only contributes a solution for robust facial expression recognition, but also consolidates the low energy cost of their implementation on neuromorphic devices, which could pave the way for high-performance, noise-robust and energy-efficient vision applications on mobile hardware.

To present an application of optimal controller design to a block-ice production process using dynamic programming. First, we develop mathematical models for such a process by employing system identification techniques. In particular, we build two parametric models, namely, linear model and neural network model. Linear models are built with Auto-Regressive model with exogenous inputs (linear ARX model). Feedforward neural networks (NNARX model) are constructed using the information provided by the linear ARX models. In addition, the Optimal Brain Surgeon (OBS) method is employed to prune the neural network models. Comparing the experimental results obtained from these models, it indicates that linear ARX models yield reasonably good performance in terms of the model fit. Then, using the obtained models, we investigate the energy consumption and operation cost of an ice factory when using time of use (TOU) tariff and time of day (TOD) tariff. Specifically, we develop an optimal control design for block-ice production process. The optimal strategy aims to minimize the electricity cost over a finite-time horizon and is subjected to constraints involving the control input of compressors, the brine temperature, and the number of block-ices ready for sales. Moreover, various factors in the design criteria have been analyzed with respect to the energy consumption and the operation cost. We compare the performance of the optimal controllers and the conventional control strategy. The simulation reveals that the proposed optimal control design has better performance than that of the conventional control. Furthermore, the optimal control using TOU tariff can reduce the operation cost when comparing to the one using TOD tariff.

Deep convolutional neural networks (ConvNets) have achieved the state-of-the-art performance on many real-world applications. However, significant computation and storage demands are required by ConvNets. Spiking neural networks (SNNs), with sparsely activated neurons and event-driven computations, show great potential to take advantage of the ultra- low power spike-based hardware architectures. Yet, training SNN with similar accuracy as ConvNets is difficult. Recent researchers have demonstrated the work of converting ConvNets to SNNs (CNN-SNN conversion) with similar accuracy. However, the energy-efficiency of the converted SNNs is impaired by the increased classification latency. In this paper, we focus on optimizing the classification latency of the converted SNNs. First, we propose a restricted output training method to normalize the converted weights dynamically in the CNN-SNN training phase. Second, false spikes are identified and the false spike inhibition theory is derived to speedup the convergence of the classification process. Third, we propose a temporal max pooling method to approximate the max pooling operation in ConvNets without accuracy loss. The evaluation shows that the converted SNNs converge in about 30 time-steps and achieve the best classification accuracy of 94% on CIFAR -10 dataset.

Spiking neural networks (SNNs) have been shown to be accurate, fast, and efficient in classical machine vision tasks, such as object recognition or detection. It is typical to convert a pretrained deep neural network into an SNN since training SNN is not easy. The max-pooling (MP) function is widely adopted in most state-of-the-art deep neural networks. To maintain the accuracy of the SNN obtained through conversion, this function is an important element to be implemented. However, it is difficult due to the dynamic characteristics of spikes. As far as we know, existing solutions adopt additional technologies except the spiking neuron model to implement MP or approximate MP, which introduce overhead of memory storage and computation. In this letter, we propose a novel method that utilizes only the spiking neuron model to approximate MP. Our method does not incur any overhead. We validate our method with three datasets and six networks including three oxford visual geometry group-like networks. And the experimental results show that the performance (accuracy and convergence rate) of our method is as good as or even better than that of the existing method.

Joint A-SNN: Joint training of artificial and spiking neural networks via self-Distillation and weight factorization

The Sixth International Symposium on Neural Networks (ISNN 2009)