Modeling deterministic echo state network with loop reservoir

Echo state networks (ESNs), as efficient and powerful computational models for approximating nonlinear dynamical systems, have been successfully applied in financial time series forecasting. Reservoir constructions in standard ESNs rely on trials and errors in real applications due to a series of randomized model building stages. A novel form of ESN with deterministically constructed reservoir is competitive with standard ESN by minimal complexity and possibility of optimizations for ESN specifications. In this paper, forecasting performances of deterministic ESNs are investigated in stock price prediction applications. The experiment results on two benchmark datasets (Shanghai Composite Index and S&P500) demonstrate that deterministic ESNs outperform standard ESN in both accuracy and efficiency, which indicate the prospect of deterministic ESNs for financial prediction.

Optimizing simple deterministically constructed cycle reservoir network with a Redundant Unit Pruning Auto-Encoder algorithm

Echo state networks (ESNs) are a novel form of recurrent neural networks (RNNs) that provide an efficient and powerful computational model approximating nonlinear dynamical systems. A unique feature of an ESN is that a large number of neurons (the "reservoir") are used, whose synaptic connections are generated randomly, with only the connections from the reservoir to the output modified by learning. Why a large randomly generated fixed RNN gives such excellent performance in approximating nonlinear systems is still not well understood. In this brief, we apply random matrix theory to examine the properties of random reservoirs in ESNs under different topologies (sparse or fully connected) and connection weights (Bernoulli or Gaussian). We quantify the asymptotic gap between the scaling factor bounds for the necessary and sufficient conditions previously proposed for the echo state property. We then show that the state transition mapping is contractive with high probability when only the necessary condition is satisfied, which corroborates and thus analytically explains the observation that in practice one obtains echo states when the spectral radius of the reservoir weight matrix is smaller than 1.

Echo state network (ESN) has recently attracted increasing interests because of its superior capability in modeling nonlinear dynamic systems. In the conventional echo state network model, its dynamic reservoir (DR) has a random and sparse topology, which is far from the real biological neural networks from both structural and functional perspectives. We hereby propose three novel types of echo state networks with new dynamic reservoir topologies based on complex network theory, i.e., with a small-world topology, a scale-free topology, and a mixture of small-world and scale-free topologies, respectively. We then analyze the relationship between the dynamic reservoir structure and its prediction capability. We utilize two commonly used time series to evaluate the prediction performance of the three proposed echo state networks and compare them to the conventional model. We also use independent and identically distributed time series to analyze the short-term memory and prediction precision of these echo state networks. Furthermore, we study the ratio of scale-free topology and the small-world topology in the mixed-topology network, and examine its influence on the performance of the echo state networks. Our simulation results show that the proposed echo state network models have better prediction capabilities, a wider spectral radius, but retain almost the same short-term memory capacity as compared to the conventional echo state network model. We also find that the smaller the ratio of the scale-free topology over the small-world topology, the better the memory capacities.

Deterministic echo state network (ESN) models integrated with particle swarm optimization (PSO) are proposed to improve the accuracy and efficiency of stock price forecasting. ESNs have been successfully applied to financial time series forecasting because of their efficient and powerful computational ability in approximating nonlinear dynamical systems. However, reservoir construction in standard ESNs is primarily driven by a series of randomized model-building stages, because of which both researchers and practitioners have to rely on a series of trials and errors to determine parameters. An ESN with a deterministically constructed reservoir is comparable in performance to a standard ESN and has minimal complexity as well as potential for optimizations with regard to ESN parameters. In this paper, forecasting performances of the proposed PSO-DESN models are compared with those of standard ESNs for stock price prediction on the benchmark dataset of S&P 500. The comparison results demonstrate that the proposed PSO-DESNs exhibit better performance in stock price forecasting in terms of both accuracy and efficiency, thereby verifying the potential of PSO-DESNs for financial predictions.

Load forecasting impacts directly financial returns and information in electrical systems planning. A promising approach to load forecasting is the Echo State Network (ESN), a recurrent neural network for the processing of temporal dependencies. The low computational cost and powerful performance of ESN make it widely used in a range of applications including forecasting tasks and nonlinear modeling. This paper presents a Bayesian optimization algorithm (BOA) of ESN hyperparameters in load forecasting with its main contributions including helping the selection of optimization algorithms for tuning ESN to solve real-world forecasting problems, as well as the evaluation of the performance of Bayesian optimization with different acquisition function settings. For this purpose, the ESN hyperparameters were set as variables to be optimized. Then, the adopted BOA employs a probabilist model using Gaussian process to find the best set of ESN hyperparameters using three different options of acquisition function and a surrogate utility function. Finally, the optimized hyperparameters are used by the ESN for predictions. Two datasets have been used to test the effectiveness of the proposed forecasting ESN model using BOA approaches, one from Poland and another from Brazil. The results of optimization statistics, convergence curves, execution time profile, and the hyperparameters’ best solution frequencies indicate that each problem requires a different setting for the BOA. Simulation results are promising in terms of short-term load forecasting quality and low error predictions may be achieved, given the correct options settings are used. Furthermore, since there is not an optimal global optimization solution known for real-world problems, correlations among certain values of hyperparameters are useful to guide the selection of such a solution.

Free-space optical (FSO) communication has established its reputation for delivering ultra-high data-rate services. Over the time-varying turbulence channels, the channel state information (CSI) tends to be outdated due to the feedback delay. This may lead to considerable performance deterioration in FSO systems. This paper studies a prediction scheme for accurately updating CSI over FSO turbulence-induced fading channels. We employ the echo state network (ESN) model, a form of recurrent neural network (RNN) with low computational complexity yet high efficiency, for turbulence channel prediction in FSO systems. We verify the prediction model using measured FSO channel data. Simulation results indicate that the ESN model has excellent prediction performance in FSO systems. Furthermore, we also highlight the effectiveness of the ESN model by comparing it with conventional CSI prediction models in terms of complexity and accuracy performance.

This paper introduces an Electrical Submersible Pump (ESP) model based on the Echo State Network (ESN) neural network. ESPs are commonly used in oil extraction, and their performance is influenced by a variety of factors, making accurate modeling crucial for optimizing operations and maintenance. ESN is a type of recurrent neural network with unique dynamic memory capabilities, making it suitable for handling time-series data and modeling nonlinear dynamic systems. The article details the ESN network structure and its application in the ESP model, including how to use ESN to predict performance parameters and diagnose faults in ESPs. Compared with traditional models, the ESN model shows significant advantages in prediction accuracy and computational efficiency. The results indicate that the ESN-based ESP model can effectively simulate the dynamic behavior of ESPs, providing a powerful tool for real-time monitoring and fault prevention of ESPs.

The performance of echo state networks (ESNs) in temporal pattern learning tasks depends both on their memory capacity (MC) and their non-linear processing. It has been shown that linear memory capacity is maximized when ESN neurons have linear activation, and that a trade-off between non-linearity and linear memory capacity is required for temporal pattern learning tasks. The more recent distance-based delay networks (DDNs) have shown improved memory capacity over ESNs in several benchmark temporal pattern learning tasks. However, it has not thus far been studied whether this increased memory capacity comes at the cost of reduced non-linear processing. In this paper, we advance the hypothesis that DDNs in fact achieve a better trade-off between linear MC and non-linearity than ESNs, by showing that DDNs can have strong non-linearity with large memory spans. We tested this hypothesis using the NARMA-30 task and the bitwise delayed XOR task, two commonly used reservoir benchmark tasks that require a high degree of both non-linearity and memory.

Artificial neural networks have been shown to perform well in automatic speech recognition (ASR) tasks, although their complexity and excessive computational costs have limited their use. Recently, a recurrent neural network with simplified training, the echo state network (ESN), was introduced by Jaeger and shown to outperform conventional methods in time series prediction experiments. We created the predictive ESN classifier by combining the ESN with a state machine framework. In small-vocabulary ASR experiments, we compared the noise-robust performance of the predictive ESN classifier with a hidden Markov model (HMM) as a function of model size and signal-to-noise ratio (SNR). The predictive ESN classifier outperformed an HMM by 8-dB SNR, and both models achieved maximum noise-robust accuracy for architectures with more states and fewer kernels per state. Using ten trials of random sets of training/validation/test speakers, accuracy for the predictive ESN classifier, averaged between 0 and 20 dB SNR, was 81plusmn3%, compared to 61plusmn2% for an HMM. The closed-form regression training for the ESN significantly reduced the computational cost of the network, and the reservoir of the ESN created a high-dimensional representation of the input with memory which led to increased noise-robust classification.

Echo state networks (ESN), a type of reservoir computing (RC) architecture, are efficient and accurate artificial neural systems for time series processing and learning. An ESN consists of a core of recurrent neural networks, called a reservoir, with a small number of tunable parameters to generate a high-dimensional representation of an input, and a readout layer which is easily trained using regression to produce a desired output from the reservoir states. Certain computational tasks involve real-time calculation of high-order time correlations, which requires nonlinear transformation either in the reservoir or the readout layer. Traditional ESN employs a reservoir with sigmoid or tanh function neurons. In contrast, some types of biological neurons obey response curves that can be described as a product unit rather than a sum and threshold. Inspired by this class of neurons, we introduce a RC architecture with a reservoir of product nodes for time series computation. We find that the product RC shows many properties of standard ESN such as short-term memory and nonlinear capacity. On standard benchmarks for chaotic prediction tasks, the product RC maintains the performance of a standard nonlinear ESN while being more amenable to mathematical analysis. Our study provides evidence that such networks are powerful in highly nonlinear tasks owing to high-order statistics generated by the recurrent product node reservoir.

In this paper, the problem of proactive caching is studied for cloud radio access networks (CRANs). In the studied model, the baseband units (BBUs) can predict the content request distribution and mobility pattern of each user and determine which content to cache at remote radio heads and the BBUs. This problem is formulated as an optimization problem, which jointly incorporates backhaul and fronthaul loads and content caching. To solve this problem, an algorithm that combines the machine learning framework of echo state networks (ESNs) with sublinear algorithms is proposed. Using ESNs, the BBUs can predict each user's content request distribution and mobility pattern while having only limited information on the network's and user's state. In order to predict each user's periodic mobility pattern with minimal complexity, the memory capacity of the corresponding ESN is derived for a periodic input. This memory capacity is shown to capture the maximum amount of user information needed for the proposed ESN model. Then, a sublinear algorithm is proposed to determine which content to cache while using limited content request distribution samples. Simulation results using real data from Youku and the Beijing University of Posts and Telecommunications show that the proposed approach yields significant gains, in terms of sum effective capacity, that reach up to 27.8% and 30.7%, respectively, compared with two baseline algorithms: random caching with clustering and random caching without clustering.

The paper expounds the question of echo state network‐based fixed‐time adaptive tracking control for a type of multi‐input multi‐output nonlinear strict‐feedback system with input delayed saturation. Based on the feature that echo state network can obtain better estimation performance at lower computational cost, it is utilized to approximate the unknown nonlinear function in the controller design course. By constructing an auxiliary system, a time‐delay system with input saturation is eliminated. Moreover, command filtering techniques can be applied to avoid the “complexity explosion” problem in the backstepping method. Then, the closed‐loop system is attested to be semi‐global practical fixed‐time stable. In the end, a simulation shows the valid scheme.

Echo State Networks (ESNs) have attracted wide attention for their superior performance in time series prediction. However, it is difficult to design an ESN to match with the given application. In this paper, an approach is proposed to design growing echo state networks. The basic idea of the proposed method is to design a growing reservoir with multiple sub-reservoirs by adding hidden units to the network group by group. First, several subservoirs are synchronously constructed by using the singular value decomposition. Then, every subservoir is evaluated and the best one is selected to be added to the network. Finally, two time series are used to validate the proposed approach.

Real-time Error Compensation Transfer Learning with Echo State Networks for Enhanced Wind Power Prediction