Algorithm and Implementation of Distributed ESN Using Spark Framework and Parallel PSO

As a novel recurrent neural network (RNN), an echo state network (ESN) that utilizes a reservoir with many randomly connected internal units and only trains the readout, avoids increased complexity of training procedures faced by traditional RNN. The ESN can cope with complex nonlinear systems because of its dynamical properties and has been applied in hydrological forecasting and load forecasting. Due to the linear regression algorithm usually adopted by generic ESN to train the output weights, an ill-conditioned solution might occur, degrading the generalization ability of the ESN. In this study, the ESN with Bayesian regularization (BESN) is proposed for short-term power production forecasting of small hydropower (SHP) plants. According to the Bayesian theory, the weights distribution in space is considered and the optimal output weights are obtained by maximizing the posterior probabilistic distribution. The evidence procedure is employed to gain optimal hyperparameters for the BESN model. The recorded data obtained from the SHP plants in two different counties, located in Yunnan Province, China, are utilized to validate the proposed model. For comparison, the feed-forward neural networks with Levenberg-Marquardt algorithm (LM-FNN) and the generic ESN are also employed. The results indicate that BESN outperforms both LM-FNN and ESN.

This thesis deals with recurrent neural networks, a particular class of artificial neural networks which can learn a generative model of input sequences. The input is mapped, through a feedback loop and a non-linear activation function, into a hidden state, which is then projected into the output space, obtaining either a probability distribution or the new input for the next time-step. This work consists mainly of two parts: a theoretical study for helping the understanding of recurrent neural networks framework, which is not yet deeply investigated, and their application to non-linear prediction problems, since recurrent neural networks are really powerful models suitable for solving several practical tasks in different fields. For what concerns the theoretical part, we analyse the weaknesses of state-of-the-art models and tackle them in order to improve the performance of a recurrent neural network. Firstly, we contribute in the understanding of the dynamical properties of a recurrent neural network, highlighting the close relation between the definition of stable limit cycles and the echo state property of an echo state network. We provide sufficient conditions for the convergence of the hidden state to a trajectory, which is uniquely determined by the input signal, independently of the initial states. This may help extend the memory of the network and increase the design options for the network. Moreover, we develop a novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem. Our new method allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps. Exploiting the singular value decomposition of the vanishing factors in the gradient and random matrices theory, we find that the singular values have to be confined in a narrow interval and derive conditions about their root mean square value. Then, we also improve the efficiency of the training of a recurrent neural network, defining a new method for speeding up this process. Thanks to a least square regularization, we can initialize the parameters of the network, in order to set them closer to the minimum and running fewer epochs of classical training algorithms. Moreover, it is also possible to completely train the network with our initialization method, running more iterations of it without losing in performance with respect to classical training algorithms. Finally, it is also possible to use it as a real-time learning algorithm, adjusting the parameters to the new data through one iteration of our initialization. In the last part of this thesis, we apply recurrent neural networks to non-linear prediction problems. We consider prediction of numerical sequences, estimating the following input choosing it from a probability distribution. We study an automatic text generation problem, where we need to predict the following character in order to compose words and sentences, and a path prediction of walking mobile users in the central area of a city, as a sequence of crossroads. Then, we analyse the prediction of video frames, discovering a wide range of applications related to the prediction of movements. We study the collision problem of bouncing balls, taking into account only the sequence of video frames without any knowledge about the physical characteristics of the problem, and the distribution over days of mobile user in a city and in a whole region. Finally, we address the state-of-the-art problem of missing data imputation, analysing the incomplete spectrogram of audio signals. We restore audio signals with missing time-frequency data, demonstrating via numerical experiments that a performance improvement can be achieved involving recurrent neural networks.

The echo state network (ESN) is simpler and costs less training time than traditional recurrent neural networks. Due to linear regression algorithm usually adopted by standard ESN to calibrate model parameters, the over-fitting phenomenon easily occurs. To overcome this shortcoming, a Bayesian echo state network (BESN) model is proposed for daily rainfall-runoff forecasting. The BESN model combined Bayesian theory and ESN obtains the optimal output weights via maximizing posterior probabilistic density and improves its generalization ability. Two Case studies on daily inflow forecasting for Ansha Reservoir and Xinfengjiang Reservoir show that the BESN model is effective and feasible and can provide better forecast accuracy than the traditional BP neural network and ESN models.

Reservoir Computing (RC) is a paradigm for efficient training of Recurrent Neural Networks (RNNs). The Echo State Network (ESN), a type of RC paradigm, has been widely used for time series forecasting. Whereas, few works exist on classification with ESN. In this paper, we shed light on the use of ESN for pattern recognition problem, i.e. emotion recognition from Electroencephalogram (EEG). We show that the reservoir with its recurrence is able to perform the feature extraction step directly from the EEG raw. Such kind of recurrence rich of nonlinearities allows the projection of the input data into a high dimensional state space. It is well known that the ESN fails due to the poor choices of its initialization. Nevertheless, we show that pretraining the ESN with the Intrinsic Plasticity (IP) rule remedies the shortcoming of randomly initialization. To validate our approach, we tested our system on the benchmark DEAP containing EEG signals of 32 subjects and the results were promising.

Echo State Networks (ESNs) are recurrent neural networks that map an input signal to a high-dimensional dynamical system, called reservoir, and possess adaptive output weights. The output weights are trained such that the ESN’s output signal fits the desired target signal. Classical reservoirs are sparse and randomly connected networks. In this article, we investigate the effect of different network topologies on the performance of ESNs. Specifically, we use two types of networks to construct clustered reservoirs of ESN: the clustered Erdös–Rényi and the clustered Barabási-Albert network model. Moreover, we compare the performance of these clustered ESNs (CESNs) and classical ESNs with the random reservoir by employing them to two different tasks: frequency filtering and the reconstruction of chaotic signals. By using a clustered topology, one can achieve a significant increase in the ESN’s performance.

Echo state network (ESN), which efficiently models nonlinear dynamic systems, has been proposed as a special form of recurrent neural network. However, most of the proposed ESNs consist of complex reservoir structures, leading to excessive computational cost. Recently, minimum complexity ESNs were proposed and proved to exhibit high performance and low computational cost. In this paper, we propose a simple deterministic ESN with a loop reservoir, i.e., an ESN with an adjacent-feedback loop reservoir. The novel reservoir is constructed by introducing regular adjacent feedback based on the simplest loop reservoir. Only a single free parameter is tuned, which considerably simplifies the ESN construction. The combination of a simplified reservoir and fewer free parameters provides superior prediction performance. In the benchmark datasets and real-world tasks, our scheme obtains higher prediction accuracy with relatively low complexity, compared to the classic ESN and the minimum complexity ESN. Furthermore, we prove that all the linear ESNs with the simplest loop reservoir possess the same memory capacity, arbitrarily converging to the optimal value.

Artificial neural networks have been shown to perform well in automatic speech recognition (ASR) tasks, although their complexity and excessive computational costs have limited their use. Recently, a recurrent neural network with simplified training, the echo state network (ESN), was introduced by Jaeger and shown to outperform conventional methods in time series prediction experiments. We created the predictive ESN classifier by combining the ESN with a state machine framework. In small-vocabulary ASR experiments, we compared the noise-robust performance of the predictive ESN classifier with a hidden Markov model (HMM) as a function of model size and signal-to-noise ratio (SNR). The predictive ESN classifier outperformed an HMM by 8-dB SNR, and both models achieved maximum noise-robust accuracy for architectures with more states and fewer kernels per state. Using ten trials of random sets of training/validation/test speakers, accuracy for the predictive ESN classifier, averaged between 0 and 20 dB SNR, was 81plusmn3%, compared to 61plusmn2% for an HMM. The closed-form regression training for the ESN significantly reduced the computational cost of the network, and the reservoir of the ESN created a high-dimensional representation of the input with memory which led to increased noise-robust classification.

The echo state network (ESN) has recently been proposed for modeling complex dynamic systems. The ESN is a sparsely connected recurrent neural network with most of its weights fixed a priori to randomly chosen values. The only trainable weights are those on links connected to the outputs. The ESN can demonstrate remarkable performance after seemingly effortless training. This brief paper discusses ESN in a broader context of applications of recurrent neural networks (RNN) and highlights challenges on the road to practical applications.

A small-world topology has been found in the cortical neural connectivity. However, the role of the topology in neural information processing has yet not been well understood. In this article, we investigate the performance of an echo state network (ESN) within a small-world topology in an economical or cost-effective environment, i.e., reduced number of input/output reservoir nodes. The ESN, a type of recurrent neural network, has a reservoir network where nodes are connected to each other with fixed weights. We introduce the small-world topology into the reservoir network. The ESN learns about the connected weights from the reservoir nodes to an output layer. In order to leverage the potential of the small-world topology, we limit the number of the reservoir nodes that receive external input (i.e., input nodes) or omit their signals to the output layer (i.e., output nodes). In addition, we segregate the input nodes from the output nodes, thereby necessitating the propagation of the input signals to the output nodes through the small-world reservoir. In our experiment, the ESNs learned to predict the next input of chaotic time-series. The small-world ESN exhibited high performance even when the number of input and output nodes was reduced, whereas the performance of the standard random or fully connected ESNs declined with reduced number of nodes.

Funding Acknowledgements Type of funding sources: Public grant(s) – National budget only. Main funding source(s): This research project is funded by Tuscany Region Background/Introduction Electrocardiograms (ECGs) are rapidly moving from analog to digital versions. Consequently, a series of automatic analyses of standard 12-lead ECGs are attracting interest for their ability to support clinicians in the automatic recognition of specific features associated with different cardiac diseases [2]. Artificial Intelligence applications and Machine Learning (ML) algorithms have gained much attention in the last years for their ability to figure out patterns from data independently, without being explicitly taught rules. Peculiar features define the ECGs of patients with Brugada Syndrome (BrS); however, ambiguities still exist for the correct diagnosis of BrS and discrimination with respect to other pathologies. Purpose The BrAID (Brugada syndrome and Artificial Intelligence applications to Diagnosis) project aims to develop an innovative system for diagnosing Type 1 BrS based on ECG pattern recognition through the application of ML algorithms. In this work, an application of Echo State Networks (ESN), a type of Recurrent Neural Network (RNN), for the diagnosis of BrS from ECG is presented. Methods After approval from the Local Ethical Committees, 12-lead ECGs were obtained in patients enrolled in 5 Centers diagnosed with typical spontaneous Type 1 pattern (coved) (group A, 81 patients). Baseline ECG was also collected in patients undergoing the ajmaline test, classified as positive (group B, 37 patients) or negative (group C, 14 patients) according to test results. 174 patients with no clinical and familial history of arrhythmias were considered controls (group D). Data were collected from 4 beats extracted from the ECGs as input to the ESN. The datasets obtained in the different groups were used for the ESN model’s training and assessment (testing) through a double cross-validation approach. Results As shown in Table 1, the performances using three leads (V1, V2, V3) or V2 only were compared. The algorithm performance was assessed in all the datasets (group A+B+C+D) and in spontaneous BrS (group A) and controls (group D). A good accuracy (79.21%) was seen when the three leads were considered for groups A and D only; the best test set accuracy (80.20%) was obtained in the case in which V2 only was used as input in all the datasets. Conclusion(s) In this work, a novel system for diagnosing Type 1 BrS using an ESN approach was developed. Our preliminary results show that this ML model is able to detect ECG patterns associated with Type 1 BrS with good and comparable accuracy both when three leads (79.21% ) or V2 only (80.20%) were analyzed. The future availability of larger datasets could improve the model performance, increasing the ESN potentialities as a clinical support system tool to be used in everyday clinical practice. Table 1. The accuracy, specificity, and sensitivity reported for each dataset group are obtained through double cross-validation.

Although many real-world time series are complex, developing methods that can learn from their behavior effectively enough to enable reliable forecasting remains challenging. Recently, several machine-learning approaches have shown promise in addressing this problem. In particular, the echo state network (ESN) architecture, a type of recurrent neural network where neurons are randomly connected and only the read-out layer is trained, has been proposed as suitable for many-step-ahead forecasting tasks. ESNs have been shown to be effective in predicting complex time series while reducing training costs, thereby improving efficiency. Deep echo state networks, which extend the architecture of ESNs by stacking multiple reservoir layers, have also been proposed as an option to learn different features of the dynamics across the layers, but to date they have received less study. In this work, we analyzed the performance of deep ESNs for several variations in network structure, including a hybrid approach that integrated a knowledge-based model. We found that for specific network structures, deep ESNs could improve prediction accuracy over baseline ESNs with the same number of neurons by up to 65% for a chaotic data set derived from the Mackey-Glass model and 14% for an experimental data set of complex zebrafish cardiac voltage recordings. Similarly, deep hybrid ESNs could reduce error compared to same-size flat hybrid ESNs by up to 59% for the Mackey-Glass data set and 11% for the experimental data set. Our results also showed that the hybrid approach was especially beneficial for the experimental data set and that deep network structures improved prediction robustness.

Analysis of prediction performance in wavelet minimum complexity echo state network

In this chapter we review two additional types of Recurrent Neural Network, which present important differences with respect to the architectures described so far. More specifically, we introduce the nonlinear auto-regressive with eXogenous inputs (NARX) neural network and the Echo State Network. Both these networks have been largely employed in Short Term Load Forecast applications and they have been shown to be more effective than other methods based on statistical models. The main differences of NARX networks and Echo State Networks with respect to the other previously described models, are both in terms of their architecture and, in particular, in their training procedure. Indeed, both these architectures are designed in such a way that Back Propagation Through Time is not necessary. Specifically, in NARX the network output is replaced by the expected ground truth and this allows to train the network like a feedforward architecture. On the other hand, in a Echo State Network only the outermost linear layer is trained, usually by means of ridge regression. Due to these fundamental differences, some of the properties and training approaches discussed in the previous sections do not hold for the NARX and Echo State Network models and we reserved a separate chapter to review these models.

Echo state networks (ESNs), as efficient and powerful computational models for approximating nonlinear dynamical systems, have been successfully applied in financial time series forecasting. Reservoir constructions in standard ESNs rely on trials and errors in real applications due to a series of randomized model building stages. A novel form of ESN with deterministically constructed reservoir is competitive with standard ESN by minimal complexity and possibility of optimizations for ESN specifications. In this paper, forecasting performances of deterministic ESNs are investigated in stock price prediction applications. The experiment results on two benchmark datasets (Shanghai Composite Index and S&P500) demonstrate that deterministic ESNs outperform standard ESN in both accuracy and efficiency, which indicate the prospect of deterministic ESNs for financial prediction.

A lot of attention is now being focused on connectionist models known under the name “reservoir computing”. The most prominent example of these approaches is a recurrent neural network architecture called an echo state network (ESN). ESNs were successfully applied in more real-valued time series modeling tasks and performed exceptionally well. Also using ESNs for processing symbolic sequences seems to be attractive. In this work we experimentally support the claim that the state space of ESN is organized according to the Markovian architectural bias principles when processing symbolic sequences. We compare performance of ESNs with connectionist models explicitly using Markovian architectural bias property, with variable length Markov models and with recurrent neural networks trained by advanced training algorithms. Moreover we show that the number of reservoir units plays a similar role as the number of contexts in variable length Markov models.KeywordsRecurrent Neural NetworkHide UnitSymbolic SequenceRecurrent NetworkEcho State NetworkThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.