Temperature-Resilient Analog Neuromorphic Chip in Single-Polysilicon CMOS Technology

In this work we present key elements of DeepChip, a framework that bridges recent trends in machine learning with applicable forward propagation on resource-constrained devices. Main objective of this work is to reduce compute and memory requirements by removing redundancy from neural networks. DeepChip features a flexible quantizer to reduce the bit width of activations to 8-bit fixed-point and weights to an asymmetric ternary representation. In combination with novel algorithms and data compression we leverage reduced precision and sparsity for efficient forward propagation on a wide range of processor architectures. We validate our approach on a set of different convolutional neural networks and datasets: ConvNet on SVHN, ResNet-44 on CIFAR10 and AlexNet on ImageNet. Compared to single-precision floating point, memory requirements can be compressed by a factor of 43, 22 and 10 and computations accelerated by a factor of 5.2, 2.8 and 2.0 on a mobile processor without a loss in classification accuracy. DeepChip allows trading accuracy for efficiency, and for instance tolerating about 2% loss in classification accuracy further reduces memory requirements by a factor of 88, 29 and 13, and speeds up computations by a factor of 6.0, 4.3 and 5.0. Code related to this paper is available at: https://github.com/UniHD-CEG/ECML2018.

Deep Convolutional Neural Networks (CNNs) perform billions of operations for classifying a single input. To reduce these computations, this paper offers a solution that leverages a combination of runtime information and the algorithmic structure of CNNs. Specifically, in numerous modern CNNs, the outputs of compute-heavy convolution operations are fed to activation units that output zero if their input is negative. By exploiting this unique algorithmic property, we propose a predictive early activation technique, dubbed SnaPEA. This technique cuts the computation of convolution operations short if it determines that the output will be negative. SnaPEA can operate in two distinct modes, exact and predictive. In the exact mode, with no loss in classification accuracy, SnaPEA statically re-orders the weights based on their signs and periodically performs a single-bit sign check on the partial sum. Once the partial sum drops below zero, the rest of computations can simply be ignored, since the output value will be zero in any case. In the predictive mode, which trades the classification accuracy for larger savings, SnaPEA speculatively cuts the computation short even earlier than the exact mode. To control the accuracy, we develop a multi-variable optimization algorithm that thresholds the degree of speculation. As such, the proposed algorithm exposes a knob to gracefully navigate the trade-offs between the classification accuracy and computation reduction. Compared to a state-of-the-art CNN accelerator, SnaPEA in the exact mode, yields, on average, 28% speedup and 16% energy reduction in various modern CNNs without affecting their classification accuracy. With 3% loss in classification accuracy, on average, 67.8% of the convolutional layers can operate in the predictive mode. The average speedup and energy saving of these layers are 2.02x and 1.89x, respectively. The benefits grow to a maximum of 3.59x speedup and 3.14x energy reduction. Compared to static pruning approaches, which are complimentary to the dynamic approach of SnaPEA, our proposed technique offers up to 63% speedup and 49% energy reduction across the convolution layers with no loss in classification accuracy.

This paper presents a sensitivity analysis of neural network (NN) parameters to improve the performance of electricity price forecasting. The presented work is an extended version of previous works done by authors to integrate NN and similar days (SD) method for predicting electricity prices. Focus here is on sensitivity analysis of NN parameters while keeping the parameters same for SD to forecast day-ahead electricity prices in the PJM market. Sensitivity analysis of NN parameters include back-propagation learning set (BP-set), learning rate (η), momentum (α) and NN learning days (dNN). The SD parameters, i.e. time framework of SD (d=45 days) and number of selected similar price days (N=5) are kept constant for all the simulated cases. Forecasting performance is carried out by choosing two different days from each season of the year 2006 and for which, the NN parameters for the base case are considered as BP-set=500, η=0.8, α=0.1 and dNN=45 days. Sensitivity analysis has been carried out by changing the value of BP-set (500, 1000, 1500); η (0.6, 0.8, 1.0, 1.2), α (0.1, 0.2, 0.3) and dNN (15, 30, 45 and 60 days). The most favorable value of BP-set is first found out from the sensitivity analysis followed by that of η and α, and based on which the best value of dNN is determined. Sensitivity analysis results demonstrate that the best value of mean absolute percentage error (MAPE) is obtained when BP-set=500, η=0.8, α=0.1 and dNN=60 days for winter season. For spring, summer and autumn, these values are 500, 0.6, 0.1 and 45 days, respectively. MAPE, forecast mean square error and mean absolute error of reasonably small value are obtained for the PJM data, which has correlation coefficient of determination (R2) of 0.7758 between load and electricity price. Numerical results show that forecasts generated by developed NN model based on the most favorable case are accurate and efficient. Copyright © 2008 John Wiley & Sons, Ltd.

A neural network controller which is used for controlling unknown discrete-time DARMA systems is described. In a first stage, a two-layered neural network is used to estimate the unknown plant dynamics. The Widrow-Hoff delta rule is used as the learning algorithm for this network so as to minimize the difference between the plant actual response and that predicted by the neural network. In a second stage, the control law is generated online using a second two-layered neural network so that the plant output is brought to a desired reference signal. Simulation examples are presented to evaluate the design.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

In this article, the method of text classification using a convolutional neural network is presented. The problem of text classification is formulated, the architecture and the parameters of a convolutional neural network for solving the problem are described, the steps of the solution and the results of classification are given. The convolutional network which was used was trained to classify the texts of the news messages of Internet information portals. The semantic preprocessing of the text and the translation of words into attribute vectors are generated using the open word2vec model. The analysis of the dependence of the classification quality on the parameters of the neural network is presented. The using of the network allowed obtaining a classification accuracy of about 84%. In the estimation of the accuracy of the classification, the texts were checked to belong to the group of semantically similar classes. This approach allowed analyzing news messages in cases where the text themes and the number of classification classes in the training and control samples do not equal.

The momentum method has become the workhorse in the deep learning community. To theoretical understand its success, researchers put efforts in demystifying its convergence properties when optimizing neural networks. For the convex problem, it is well-known that the triple momentum (TM) method owns the fastest theoretical convergence rate among all first-order methods. However, there exists no theoretical convergence results about the TM method in solving the non-convex neural networks training problem, let alone its acceleration guarantee. In this paper, we focus on the training process of the TM method for a two-layer ReLU neural network. Inspired by the accurate characterization of the high-resolution dynamical system, we consider the high-resolution ordinary differential equation (ODE) of the TM method. Under the over-parameterized assumption, we derive that the original non-convex optimization problem can be transformed to a strongly convex task. By applying an appropriate Lyapunov function, we prove that the TM method can linearly converge to a global minimum. Compared to the heavy ball method and Nesterov’s accelerated gradient method, our result provides the first guarantee for the acceleration of the TM method in training neural networks. Through empirical experiments, the accelerated convergence of the TM method and the effect of over-parameterization are validated.

Recently, the sonar signal processing community has seen increased interest in neural networks for nonlinear classification and regression. While innovative neural network architectures have achieved remarkable results in other fields such as computer vision and natural language processing, their application to sonar data presents challenges. The multiple nonlinearities in these networks make it difficult to interpret results. Recent works such as Grad-CAM and Layer-wise Relevance Propagation attempt to empirically visualize what features are being selected by the network, but there has been little theoretical work to explain why neural networks perform so well. However, recent work by Pilanci et al. has shown that two-layer neural networks with a ReLU activation function can be reformulated as a convex optimization that can be solved optimally in polynomial time. Their results rely on the observation that ReLU neural networks are piece-wise linear models that lift the original data into a higher dimensional feature space. Using a valid activation function, we show that this Convex Neural Network framework can be extended to applications where the input data is complex-valued. We evaluate our algorithm on simulated sonar data. [This talk will present research funded by DoD Navy (NEEC) Grant number N001742010016 and the ONR grant numbers N000142112420 and N000142312503.]

Development of controlled environment in greenhouse is of prim importance for out of season production, increasing yield and enhancing the quality of produce.Due to high cost and impossibility of continuous human attendance in greenhouse, it is desirable to control the greenhouse environment by employing automatic control devices.In This study, the greenhouse conditions were controlled by using artificial neural network (ANN).First, an experimental greenhouse was built and equipped with control instruments.Then by using electronic sensors, some climatic parameter data (temperature, humidity, carbon dioxide and light index) were measured and saved during five minute periods.In the next stage, three types of ANN including feed forward neural networks with multiple delays in the input, two-layer neural network with a feedback from hidden layer and input delay and three-layer neural network with two feedbacks from hidden layers and input delay were trained by 66% of the recorded data, and were evaluated by using the remaining data.The three-layer neural network with two feedbacks from hidden layers and input delay was able to better predict humidity and light index of the greenhouse with MSE,s of 0.025 and 0.032, respectively.Temperature and infrared index were better predicted by using the feed forward neural networks with multiple delays in the input with MSE,s of 0.016 and 0.017, respectively.In all cases, training time was less than 14 minutes and simulation time being always less than 0.2 second, makes using neural network feasible for automatic control of greenhouse.

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher–student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.

We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) and establish different representation formulae. In two cases, we describe the space explicitly up to isomorphism. Using a convenient representation, we study the pointwise properties of two-layer networks and show that functions whose singular set is fractal or curved (for example distance functions from smooth submanifolds) cannot be represented by infinitely wide two-layer networks with finite path-norm. We use this structure theorem to show that the only $$C^1$$ -diffeomorphisms which preserve Barron space are affine. Furthermore, we show that every Barron function can be decomposed as the sum of a bounded and a positively one-homogeneous function and that there exist Barron functions which decay rapidly at infinity and are globally Lebesgue-integrable. This result suggests that two-layer neural networks may be able to approximate a greater variety of functions than commonly believed.

A person with one eye missing, through various reasons, may suffer psychologically as well as physically. The loss of an eye can be solved cosmetically by an ocular implant. This artificial eye appears natural, but it is static. To let the artificial eye have the same natural movement as the real eye, an ocular system is developed. The artificial eye is mounted onto a tiny servomotor. The whole system shall be able to sense the real eye movement and control the motor to drive the artificial eye to the desired position. A tiny infrared sensor array is used for this study. This paper describes an approach of using the artificial neural network to do the sensor fusion to detect the eye movement. Two types of neural networks are used for the sensor fusion and sensor fault detection and recovery respectively. Usually the sensor fusion relies on the model of the system, however, sometimes it is not possible to get an accurate model of the system, or one or several of the parameters of the system may be unknown or partially known. In addition, there may be measurement inaccuracies associated with the sensors. In this case, the conventional method may not have a good performance. An artificial neural network can learn the characteristic of a non-linear, non-modeled system through training samples. Then during the real application, the sensor signal can be used to feed the network and obtain the desired output. Using the micro sensor array to detect the eye movement we carried out an experimental study. The sensor data is amplified and digitized then sent to the computer. Two-layer neural networks are trained by the data samples. First, the trained network is used for sensor fusion, and then two neural networks are used to detect the sensor failure and recover the faulty data respectively. Experimental studies with soft sensor failure and hard sensor failure are included. The main part of this paper deals with the network training method and further considerations.

This research focuses on optimizing a neural network model for stock price prediction using Particle Swarm Optimization (PSO), considering the inherent risks and potential high returns associated with stock investment. Given the challenges posed by stock price volatility, this study combines Moving Average (MA) a fundamental statistical technique in stock market analysis with advanced data mining approaches, specifically neural networks and PSO, to enhance prediction accuracy. The primary objective is to improve the efficiency of neural networks by minimizing error rates and equipping investors with more reliable tools for financial decision-making. The proposed methodology involves converting historical stock price data into a Simple Moving Average (SMA) over a 5-day period, followed by optimizing a neural network model using PSO. This optimization process fine-tunes key parameters, particularly the weight distributions of various stock market indicators, including Open SMA, High SMA, Low SMA, and Close SMA. Model performance is evaluated using Root Mean Square Error (RMSE) as a validation metric. The findings indicate a significant enhancement in the predictive accuracy of the neural network model after PSO optimization. The optimal configuration is identified in a two-layer neural network with a specific node arrangement. This optimized model not only improves stock price forecasting precision but also has practical implications for investors and financial analysts in risk management and profit maximization.

Memristive devices1 have become a promising candidate for unconventional computing2. In this talk, I will present some of our recent work on unconventional computing experimentally implemented by using memristive devices or crossbar arrays. Using traditional non-volatile memristors with 64 stable analog resistance levels, we have built a dot-product engine based on a 128 x 64 1T1R crossbar array3. Accurate image compression and filtering have been demonstrated with such analog computing accelerator3. In addition, we have demonstrated efficient and self-adaptive in-situ learning in a two-layer neural networks using such memristive arrays4, which is expected to significantly improve the speed and energy efficiency of deep neural networks. Using our newly developed diffusive memristors5 with diffusion dynamics that is critical for neuromorphic functions, we have developed artificial synapses6 and neurons7 to more faithfully emulate their bio-counterparts and more efficiently perform spiking neural network functions. We have further integrated these artificial synapses and neurons into a small neural network, with which pattern classification and unsupervised learning have been demonstrated7. Moreover, the diffusive memristors can be used as true random number generators8 for cybersecurity applications and artificial nociceptors for robotics applications9. 1. Yang, J. J. et al. Memristive switching mechanism for metal/oxide/metal nanodevices. Nature Nanotechnology 3, 429 (2008). 2. Yang, J. J., Strukov, D. B. & Stewart, D. R. Memristive devices for computing. Nature Nanotechnology 8, 13 (2013). 3. Li, C. et al. Analogue signal and image processing with large memristor crossbars. Nature Electronics 1, 52 (2018). 4. Li, C. et al. Efficient and self-adaptive in-situ learning in multilayer memristive neural networks. Nature communications, under revision (2018). 5. Midya, R. et al. Anatomy of Ag/Hafnia-Based Selectors with 1E10 Nonlinearity. Advanced Materials 29, 1604457 (2017). 6. Wang, Z. et al. Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. Nature Materials 16, 101-108 (2017). 7. Wang, Z. et al. Fully memristive neural networks for pattern classification with unsupervised learning. Nature Electronics 1, 137 (2018). 8. Jiang, H. et al. A Novel True Random Number Generator Based on a Stochastic Diffusive Memristor. Nature communications 8, 882 (2017). 9. Yoon, J. H. et al. An artificial nociceptor based on a diffusive memristor. Nature Communications 9, 417 (2018).

A method is proposed for neural network prediction of congestion on road sections outside the control zones of transport detectors in the systems of “flexible” transport flow control at intersections with the traffic light regulation. The paper describes the operating principle of the original control system based on fuzzy logic. To develop a universal neural network solution that can be used to predict traffic on most road sections with common characteristics without the need for training for each case separately, it is proposed to identify 9 main types of sections and, accordingly, 9 neural networks. Each typical neural network differs in the amount of the input data and parameters. All neural networks are trained on samples obtained from analyzing traffic flow on road sections of each type. Typical characteristics of road sections, architecture, and parameters of neural networks are determined. Neural networks are trained and tested on the model data. Using computer simulation methods, the author assesses the control quality of the original intelligent traffic light control system, which has complete information about the transport flow (however, it is difficult to implement in practice), as well as a control system that uses a set of data obtained using transport detectors and supplemented with a neural network forecast. According to the experiment results, a system with a neural network forecast of congestion on road sections is inferior in quality of control by no more than 1.92% compared to an idealized system, which gives grounds to consider this method applicable in intelligent transport flow control systems at intersections with traffic light regulation.

In this paper, hybrid models of Kalman filter and neural network for state estimation are reviewed of their corresponding academic achievements, the creation of which is a noteworthy development in state estimation. This paper aims to provide a summary of research progress on such hybrid models and emphasize their functions and advantages. First of all, the concept and feature are paid attention to about Kalman filter, and its transmutative modes are taken into consideration. Then several popular neural network algorithms are introduced in brief. Subsequently, research results on hybrid models are analysed and discussed comprehensively. Not only Kalman filter and neural network can be adopted in succession, but also can be mixed in structure. The mixed models can also be divided into two types, the equations or parameters of state–space model are trained by neural network for Kalman filter and the parameters of neural network are updated by Kalman filter. It is proved that the hybrid models outperform than single model of Kalman filter or neural network in accuracy and generalization. Last but not least, the effectiveness of state–space equations of Kalman filter can be established by neural network in nonlinear systems is verified.