Reservoir Computing for Solving Ordinary Differential Equations

In this paper, we derive an upper bound on the generalization error of reservoir computing (RC), a special category of recurrent neural networks (RNNs). The particular RC implementation considered in this paper is the echo state network (ESN), and the generalization bound is derived via the empirical Rademacher complexity (ERC) approach. While recent work in deriving risk bounds for RC frameworks makes use of a non-standard empirical Rademacher Complexity (ERC) measure and a direct application of its definition, our work uses the standard ERC measure and tools which allow easier extension to deep RC structures and a fair comparison with other RNN structures including the vanilla RNNs. Finally, we train an ESN for the task of symbol detection, a key component in the receive processing of 5G systems and show with an example how the derived generalization bound can guide the underlying system design. To be specific, we utilize the derived generalization error together with the characterized training loss to analytically identify the optimum number of neurons in the reservoir for the symbol detection task. Simulation results using 3GPP specified channels corroborate our theoretical findings, illustrating the significance and practical relevance of our work.

In this paper, we derive a theoretical upper bound on the generalization error of reservoir computing (RC), a special category of recurrent neural networks (RNNs). The specific RC implementation considered in this paper is the echo state network (ESN), and an upper bound on its generalization error is derived via the empirical Rademacher complexity (ERC) approach. While recent work in deriving risk bounds for RC frameworks makes use of a non-standard ERC measure and a direct application of its definition, our work uses the standard ERC measure and tools allowing fair comparison with conventional RNNs. The derived result shows that the generalization error bound obtained for ESNs is tighter than the existing bound for vanilla RNNs, suggesting easier generalization for ESNs. With the ESN applied to symbol detection in MIMO-OFDM (Multiple Input Multiple Output-Orthogonal Frequency Division Multiplexing) systems, we show how the derived generalization error bound can guide underlying system design. Specifically, the derived bound together with the empirically characterized training loss is utilized to identify the optimum reservoir size in neurons for the ESN-based symbol detector. Finally, we corroborate our theoretical findings with results from simulations that employ 3GPP standards-compliant wireless channels, signifying the practical relevance of our work.

Image segmentation refers to the process of dividing an image into multiple regions which represent meaningful areas. Image segmentation is an essential step for most image analysis tasks such as object recognition and tracking, pattern recognition, content-based image retrieval, etc. In recent years, a large number of image segmentation algorithms have been developed, but achieving accurate segmentation still remains a challenging task. Recently, reservoir computing (RC) has drawn much attention in machine learning as a new model of recurrent neural networks (RNN). Echo State Network (ESN) represents one efficient realization of RC, which is initially designed to facilitate learning in Recurrent Neural Networks. In this paper we investigate the viability of ESN as feature extractor for pixel classification based colour image segmentation. Extensive experiments are conducted on real world colour image datasets and the global ESN reservoir parameters are varied to identify their operating ranges that allow the use of the reservoir nodes internal activations as new pixel features for the colour image segmentation task. A simple feed forward neural network is used to realize the ESN readout function and classify these new features. The experimental results show that the proposed method achieves high performance image segmentation comparing with state-of-the-art techniques. In addition, a set of empirically derived guidelines for setting the reservoir global parameters are proposed.

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

From logical reasoning to mental simulation, biological and artificial neural systems possess an incredible capacity for computation. Such neural computers offer a fundamentally novel computing paradigm by representing data continuously and processing information in a natively parallel and distributed manner. To harness this computation, prior work has developed extensive training techniques to understand existing neural networks. However, the lack of a concrete and low-level machine code for neural networks precludes us from taking full advantage of a neural computing framework. Here we provide such a machine code along with a programming framework by using a recurrent neural network—a reservoir computer—to decompile, code and compile analogue computations. By decompiling the reservoir’s internal representation and dynamics into an analytic basis of its inputs, we define a low-level neural machine code that we use to program the reservoir to solve complex equations and store chaotic dynamical systems as random-access memory. We further provide a fully distributed neural implementation of software virtualization and logical circuits, and even program a playable game of pong inside of a reservoir computer. Importantly, all of these functions are programmed without requiring any example data or sampling of state space. Finally, we demonstrate that we can accurately decompile the analytic, internal representations of a full-rank reservoir computer that has been conventionally trained using data. Taken together, we define an implementation of neural computation that can both decompile computations from existing neural connectivity and compile distributed programs as new connections.

In this paper, we investigate online learning-based MIMO-OFDM symbol detection strategies focusing on a special recurrent neural network (RNN) - reservoir computing (RC). We first introduce the Time-Frequency RC to take advantage of the structural information inherent in OFDM signals. Using the time domain RC and the time-frequency RC as building blocks, we provide two extensions of the shallow RC to RCNet: 1) Stacking multiple time domain RCs; 2) Stacking multiple time-frequency RCs into a deep structure. The combination of RNN dynamics, the time-frequency structure of MIMO-OFDM signals, and the deep network enables RCNet to handle the interference and nonlinear distortion of MIMO-OFDM signals to outperform existing methods. Unlike most existing NN-based detection strategies, RCNet is also shown to provide a good generalization performance even with a limited online training set (i.e, similar amount of reference signals/training as standard model-based approaches). Numerical experiments demonstrate that the introduced RCNet can offer a faster learning convergence and as much as 20% gain in bit error rate over a shallow RC structure by compensating for the nonlinear distortion of the MIMO-OFDM signal, such as due to power amplifier compression in the transmitter or due to finite quantization resolution in the receiver.

Reservoir Computing (RC) is a paradigm for efficient training of Recurrent Neural Networks (RNNs). The Echo State Network (ESN), a type of RC paradigm, has been widely used for time series forecasting. Whereas, few works exist on classification with ESN. In this paper, we shed light on the use of ESN for pattern recognition problem, i.e. emotion recognition from Electroencephalogram (EEG). We show that the reservoir with its recurrence is able to perform the feature extraction step directly from the EEG raw. Such kind of recurrence rich of nonlinearities allows the projection of the input data into a high dimensional state space. It is well known that the ESN fails due to the poor choices of its initialization. Nevertheless, we show that pretraining the ESN with the Intrinsic Plasticity (IP) rule remedies the shortcoming of randomly initialization. To validate our approach, we tested our system on the benchmark DEAP containing EEG signals of 32 subjects and the results were promising.

With the growing use of artificial neural networks (ANNs) in temporal data processing tasks, the cost of training for complex ANNs is an escalating concern. Physical reservoir computing (RC), a variation of recurrent neural networks (RNNs), obviate the need for most data intensive matrix vector multiplication in the recurrent layer by evolving the RC’s internal states with the inherent nonlinear dynamics and short-term memory. In this study, we show that magnetic skyrmion confined in a fixed geometry forming the soft layer of a magnetic tunnel junction (MTJ) can work as a RC and perform autonomous long-term prediction of temporal data. Our proposed skyrmion reservoir allows for manipulation of spin dynamics with ultra energy efficient voltage controlled magnetic anisotropy modulation (VCMA) method. Furthermore, the boundary effect on the skyrmion from the geometric edges provides necessary consistency property of the reservoir. We employ our proposed reservoir for the modeling and prediction of the chaotic time series such as Mackey-Glass and dynamic time-series data, such as household building energy loads. For autonomous run, the predicted output is fed to the input of the reservoir. By comparing our spintronic physical RC approach with energy load forecasting algorithms, such as LSTMs and RNNs, we conclude that the proposed framework presents good performance in achieving high predictions accuracy, while also requiring low memory and energy both of which are at a premium in hardware resource and power constrained edge applications. Further, the proposed approach is shown to require very small training datasets and at the same time being at least 16× energy efficient compared to the sequence to sequence LSTM for accurate household load predictions. Higher endurance, fast processing and well-established technology (i.e., MTJ) to integrate with CMOS technology makes such spintronic reservoir attractive over other emergent memory device-based reservoirs.

This thesis deals with recurrent neural networks, a particular class of artificial neural networks which can learn a generative model of input sequences. The input is mapped, through a feedback loop and a non-linear activation function, into a hidden state, which is then projected into the output space, obtaining either a probability distribution or the new input for the next time-step. This work consists mainly of two parts: a theoretical study for helping the understanding of recurrent neural networks framework, which is not yet deeply investigated, and their application to non-linear prediction problems, since recurrent neural networks are really powerful models suitable for solving several practical tasks in different fields. For what concerns the theoretical part, we analyse the weaknesses of state-of-the-art models and tackle them in order to improve the performance of a recurrent neural network. Firstly, we contribute in the understanding of the dynamical properties of a recurrent neural network, highlighting the close relation between the definition of stable limit cycles and the echo state property of an echo state network. We provide sufficient conditions for the convergence of the hidden state to a trajectory, which is uniquely determined by the input signal, independently of the initial states. This may help extend the memory of the network and increase the design options for the network. Moreover, we develop a novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem. Our new method allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps. Exploiting the singular value decomposition of the vanishing factors in the gradient and random matrices theory, we find that the singular values have to be confined in a narrow interval and derive conditions about their root mean square value. Then, we also improve the efficiency of the training of a recurrent neural network, defining a new method for speeding up this process. Thanks to a least square regularization, we can initialize the parameters of the network, in order to set them closer to the minimum and running fewer epochs of classical training algorithms. Moreover, it is also possible to completely train the network with our initialization method, running more iterations of it without losing in performance with respect to classical training algorithms. Finally, it is also possible to use it as a real-time learning algorithm, adjusting the parameters to the new data through one iteration of our initialization. In the last part of this thesis, we apply recurrent neural networks to non-linear prediction problems. We consider prediction of numerical sequences, estimating the following input choosing it from a probability distribution. We study an automatic text generation problem, where we need to predict the following character in order to compose words and sentences, and a path prediction of walking mobile users in the central area of a city, as a sequence of crossroads. Then, we analyse the prediction of video frames, discovering a wide range of applications related to the prediction of movements. We study the collision problem of bouncing balls, taking into account only the sequence of video frames without any knowledge about the physical characteristics of the problem, and the distribution over days of mobile user in a city and in a whole region. Finally, we address the state-of-the-art problem of missing data imputation, analysing the incomplete spectrogram of audio signals. We restore audio signals with missing time-frequency data, demonstrating via numerical experiments that a performance improvement can be achieved involving recurrent neural networks.

Real-time processing of space-and-time-variant signals is imperative for perception and real-world problem-solving. In the brain, spatio-temporal stimuli are converted into spike trains by sensory neurons and projected to the neurons in subcortical and cortical layers for further processing. Reservoir Computing (RC) is a neural computation paradigm that is inspired by cortical Neural Networks (NN). It is promising for real-time, on-line computation of spatio-temporal signals. An RC system incorporates a Recurrent Neural Network (RNN) called reservoir, the state of which is changed by a trajectory of perturbations caused by a spatio-temporal input sequence. A trained, non- recurrent, linear readout-layer interprets the dynamics of the reservoir over time. Echo-State Network (ESN) [1] and Liquid-State Machine (LSM) [2] are two popular and canonical types of RC system. The former uses non-spiking analog sigmoidal neurons – and, more recently, Leaky Integrator (LI) neurons – and a normalized random connectivity matrix in the reservoir. Whereas, the reservoir in the latter is composed of Leaky Integrate-and-Fire (LIF) neurons, distributed in a 3-D space, which are connected with dynamic synapses through a probability function. The major difference between analog neurons and spiking neurons is in their neuron model dynamics and their inter-neuron communication mechanism. However, RC systems share a mysterious common property: they exhibit the best performance when reservoir dynamics undergo a criticality [1–6] – governed by the reservoirs’ connectivity parameters, |λmax| ≈ 1 in ESN, λ ≈ 2 and w in LSM – which is referred to as the edge of chaos in [3–5]. In this study, we are interested in exploring the possible reasons for this commonality, despite the differences imposed by different neuron types in the reservoir dynamics. We address this concern from the perspective of the information representation in both spiking and non-spiking reservoirs. We measure the Mutual Information (MI) between the state of the reservoir and a spatio-temporal spike-trains input, as well as that, between the reservoir and a linearly inseparable function of the input, temporal parity. In addition, we derive Mean Cumulative Mutual Information (MCMI) quantity from MI to measure the amount of stable memory in the reservoir and its correlation with the temporal parity task performance. We complement our investigation by conducting isolated spoken-digit recognition and spoken-digit sequence-recognition tasks. We hypothesize that a performance analysis of these two tasks will agree with our MI and MCMI results with regard to the impact of stable memory in task performance. It turns

This paper provides a comparison between two methods for time series forecasting. The first method is based on traditional recurrent neural networks (RNNs) while the second method is based in Reservoir Computing (RC). Reservoir Computing is a new paradigm that offers an intuitive methodology for using the temporal processing power of RNNs without the inconvenience of training them. So we decided to compare the advantages / disadvantages of using Reservoir Computing and RNNs in the problem of time series forecasting. The first method uses a Nonlinear Autoregressive Network with exogenous inputs (NARX). Optimization was carried out on the NARX architecture through an optimization procedure focused on the best mean squared error (MSE) metrics in the training set. The second method, called RCDESIGN, combines an evolutionary algorithm with Reservoir Computing and simultaneously looks for the best values of parameters, topology and weight matrices without rescaling the reservoir by the spectral radius. Nevertheless RCDESIGN has yielded fast tracking and excellent performance in some benchmark problems including the Narma and Mackey-Glass time-series.

The adoption of wind energy has grown significantly in recent years. New, cost-effective technologies have been developed, led by customer awareness of green technologies and a legal framework proposed at the European Union level. The stochastic nature of wind speed is transferred to wind turbine output, making wind energy difficult to predict. The main scope of predicting wind energy production is to be proactive in balancing and reserving energy to meet demand. When the prediction identifies a potential gap between supply and demand, additional energy from other sources must be generated and supplied. Creating a synergy of physical devices through advanced sensing capabilities, software, storage and analytics capabilities, the Industrial Internet of Things is enabling the effective transition to wind energy through automation by removing many of the disadvantages in a way that has recently become accessible. This research focuses on the data analytics, proposing a fast univariate network-based approach for wind energy prediction, using Feed Forward Neural Networks, Recurrent Neural Networks, Long-Short Term Memory, Gated Recurrent Unit, and Convolutional Neural Networks. Moreover, by introducing the theoretical fundamentals, the implementation method and the hyperparameters of the final models, this article becomes unique in the context of wind energy. At the time of this study, no prior research studies have presented a direct comparison between feedforward, recurrent, and convolutional neural networks ? these being the most important in the field of supervised learning.

Neuroendocrine system mainly consists of hypothalamus, anterior pituitary, and target organ. In this paper, a three-state-variable delayed Goodwin model with two Hill functions is considered, where the Hill functions with delays denote the hormonal feedback suppressions from target organ to hypothalamus and to anterior in the reproductive hormonal axis. The existence of Hopf bifurcation shows the circadian rhythms of neuroendocrine system. The direction and stability of Hopf bifurcation are also analyzed using the normal form theory and the center manifold theorem for functional differential equations. Furthermore, based on the sparse identification algorithm, it is verified that the transient time series generated from the delayed Goodwin model cannot be equivalently presented by ordinary differential equations from the viewpoint of data when considering that a library of candidates are at most cubic terms. The reason is because the solution space of delayed differential equations is of infinite dimensions. Finally, we report that reservoir computing can predict the periodic behaviors of the delayed Goodwin model accurately if the size of reservoir and the length of data used for training are large enough. The predicting performances are evaluated by the mean squared errors between the trajectories generated from the numerical simulations and the reservoir computing.

As a novel recurrent neural network (RNN), an echo state network (ESN) that utilizes a reservoir with many randomly connected internal units and only trains the readout, avoids increased complexity of training procedures faced by traditional RNN. The ESN can cope with complex nonlinear systems because of its dynamical properties and has been applied in hydrological forecasting and load forecasting. Due to the linear regression algorithm usually adopted by generic ESN to train the output weights, an ill-conditioned solution might occur, degrading the generalization ability of the ESN. In this study, the ESN with Bayesian regularization (BESN) is proposed for short-term power production forecasting of small hydropower (SHP) plants. According to the Bayesian theory, the weights distribution in space is considered and the optimal output weights are obtained by maximizing the posterior probabilistic distribution. The evidence procedure is employed to gain optimal hyperparameters for the BESN model. The recorded data obtained from the SHP plants in two different counties, located in Yunnan Province, China, are utilized to validate the proposed model. For comparison, the feed-forward neural networks with Levenberg-Marquardt algorithm (LM-FNN) and the generic ESN are also employed. The results indicate that BESN outperforms both LM-FNN and ESN.

Reservoir computing (RC) is considered as a suitable alternative for descending gradient methods in recursive neural networks (RNNs) training. The echo state network (ESN) is a platform for RC and nonlinear system simulation in the cloud environment with many external users. In the past researches, the highest eigenvalue of reservoir connection weight (spectral radius) was used to predict reservoir dynamics. Some researchers have illustrated; the characteristics of scale-free and small-world can improve the approximation capability in echo state networks; however, recent studies have shown importance of the infrastructures such as clusters and the stability criteria of these reservoirs as altered. In this research, we suggest a high clustered ESN called HCESN that its internal neurons are interconnected in form of clusters. Each of the clusters contains one backbone and a number of local nodes. We implemented a classical clustering algorithm, called K-means, and three optimization algorithms including genetic algorithm (GA), differential evolution (DE), and particle swarm optimization (PSO) to improve the clustering efficiency of the new reservoir and compared them with each other. For investigating the spectral radius and predictive power of the resulting reservoirs, we also applied them to the laser time series and the Mackey-Glass dynamical system. It is demonstrated that new clustered reservoirs have some specifications of biologic neural systems and complex networks like average short path length, high clustering coefficient, and power-law distribution. The empirical results illustrated that the ESN based on PSO could strikingly enhance echo state property (ESP) and obtains less chaotic time series prediction error compared with other works and the original version of ESN. Therefore, it can approximate nonlinear dynamical systems and predict the chaotic time series.