Time series prediction using deep echo state networks

Although many real-world time series are complex, developing methods that can learn from their behavior effectively enough to enable reliable forecasting remains challenging. Recently, several machine-learning approaches have shown promise in addressing this problem. In particular, the echo state network (ESN) architecture, a type of recurrent neural network where neurons are randomly connected and only the read-out layer is trained, has been proposed as suitable for many-step-ahead forecasting tasks. ESNs have been shown to be effective in predicting complex time series while reducing training costs, thereby improving efficiency. Deep echo state networks, which extend the architecture of ESNs by stacking multiple reservoir layers, have also been proposed as an option to learn different features of the dynamics across the layers, but to date they have received less study. In this work, we analyzed the performance of deep ESNs for several variations in network structure, including a hybrid approach that integrated a knowledge-based model. We found that for specific network structures, deep ESNs could improve prediction accuracy over baseline ESNs with the same number of neurons by up to 65% for a chaotic data set derived from the Mackey-Glass model and 14% for an experimental data set of complex zebrafish cardiac voltage recordings. Similarly, deep hybrid ESNs could reduce error compared to same-size flat hybrid ESNs by up to 59% for the Mackey-Glass data set and 11% for the experimental data set. Our results also showed that the hybrid approach was especially beneficial for the experimental data set and that deep network structures improved prediction robustness.

This study presents a novel, streamlined and scalable deep Echo State Network (ESN) architecture for demand forecasting in electricity markets. In contemporary electricity markets, where each new day it becomes more difficult to achieve smooth technical operations and not too much volatile electricity prices, the market stakeholders face the problem of forecasting energy demand with increasingly high accuracy and computational efficiency, because knowing demand in advance helps manage coming problems. Deep ESNs demonstrated themselves to be very fitting for the forecasting endeavor. We hence propose a deep ESN architecture for demand forecasting that has higher efficiency than currently used deep ESNs, while maintaining similar forecasting accuracy. Our design strategy is based on disentangling the ESN readout matrix from the individual reservoirs that make the ESN deep, just maintaining the readout weights from the output layer reservoir only. Furthermore, we use Particle Swarm optimization (PSO) to optimize the inter-reservoir connecting weights. We will prove, by numerical testing, that this allows our architecture to achieve an accuracy close to that of current deep ESNs, while being more scalable and being based on fewer parameters than the standard deep ESNs. Specifically, we evaluate this architecture by forecasting a daily average electricity demand time series from the Spanish electricity market. Our architecture, once optimized by PSO, is shown to improve over some common benchmarks and state-of-the-art methods.

The deep echo state network demonstrates strong predictive performance in time series tasks, but inter-layer weight optimization and reservoir structure design are still challenging tasks. To address this, we propose the cooperative co-evolutionary deep echo state network (CCEDESN). Firstly, inspired by the hierarchical structure of the brain, we introduce a modular hierarchical design. Each layer of the reservoir contains multiple sub-reservoirs, with central neurons in each sub-reservoir interacting with those in other sub-reservoirs to enhance information processing. Secondly, an advanced cooperative co-evolutionary algorithm is proposed to simultaneously optimize the layer connection weights and hyperparameters of the deep echo state network. We evaluate the CCEDESN model on the Mackey-Glass system, sunspot data, and Apple stock opening prices. The experimental results show that the CCEDESN model achieves superior prediction accuracy compared to other models.

Nowadays , nonlinear time series and artificial neural networks (ANN) models are used for forecasting in the field of business, agriculture and soon. Recent studies have shown, ANN have been successfully used for forecasting of financial and agriculture data series The classical methods used for time series prediction like Box-Jenkins or ARIMA assumes that there is a linear relationship between inputs and outputs. ANN have more advantages that can approximate to model both linear and nonlinear structures in time series, they are not able to handling both structures equally well. The autoregressive integrated moving average (ARIMA) model and two ANN models namely, Radial basis function neural networks (RBFNN), and Elman recurrent neural networks (ERNN) methods were applied to Hyderabad airport traffic data. The data obtained for 15 years from 2002–2003 to 2016–2017 about domestic and international passenger of International Airport of Hyderabad, India. In this research paper, we compared the performances of ARIMA, RBFNN and ERNN were based on three measures: mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square error (RMSE). The results showed that RBFNN obtained the smallest MAE, MAPE and RMSE in both the modeling and forecasting processes. The performances of the three models ranked in ascending order were: ARIMA, ERNN and the RBFNN model. Keywords: T ime series, forecasting, artificial neural networks, ARIMA models, radial basis function neural networks, and Elman recurrent neural networks Cite this Article R. Ramakrishna, Berhe Aregay, Tewodros Gebregergs. The Comparison in Time Series Forecasting of Air Traffic Data by Autoregressive Integrated Moving Average Model, Radial Basis Function and Elman Recurrent Neural Networks. Research & Reviews: Journal of Statistics . 2018; 7(3): 75–90p.

A novel echo state network for multivariate and nonlinear time series prediction

Echo State Networks (ESNs) have attracted wide attention for their superior performance in time series prediction. However, it is difficult to design an ESN to match with the given application. In this paper, an approach is proposed to design growing echo state networks. The basic idea of the proposed method is to design a growing reservoir with multiple sub-reservoirs by adding hidden units to the network group by group. First, several subservoirs are synchronously constructed by using the singular value decomposition. Then, every subservoir is evaluated and the best one is selected to be added to the network. Finally, two time series are used to validate the proposed approach.

Modeling data-driven sensor with a novel deep echo state network

In this paper, a novel model, named double-reservoir echo state networks (DR-ESN), is proposed. DR-ESN is constructed by two reservoirs which are connected in series, thus the performance of abstracting the characteristics from the prediction task is improved. A sufficient condition is provided to ensure the stability of DR-ESN. The batch gradient method and ridge regression method are utilized to optimize the six parameters of DR-ESN and train the readouts, respectively. DR-ESN is verified by two different experiments, chaotic time series prediction and real-valued function time series prediction. The simulation results demonstrates that DR-ESN has a more precise result than leaky-ESN in predicting the time series.

As a novel recursion neural network, Echo State Networks (ESN) are characterized by strong nonlinear prediction capability and effective and straightforward training algorithms. However, conventional ESN predictions require a large volume of training samples. Meanwhile, the time sequence data are complicated and unstable, resulting in insufficient learning of this network and difficult training. As a result, the accuracies of conventional ESN predictions are limited. Aimed at this issue, a time series prediction model of Grey Wolf optimized ESN has been proposed. Wout of ESN was optimized using the Grey Wolf algorithm and predictions of time series data were achieved using simplified training. The results indicated that the optimized time series prediction method exhibits superior prediction accuracy at a small sample size, compared with conventional prediction methods.

Time series analysis and prediction is an important means of dynamic system modeling. A new method of time series prediction based on support vector regression (SVR) is introduced to resolve the problem of non-linear system modeling. For the purpose of reducing calculation complexity, smooth method is presented to improve standard SVR arithmetic, and is utilized to build the combustion state model of flame in the furnace of utility boilers according to the feature parameters of flame image, in order to predict the combustion state of flame. The flame images are gained from the flame image gathering system on-line. The feature parameters of the flame image are extracted, and are used to determined combustion indices which can denote different combustion states of flame. The time series of combustion indices are used for constructing the smooth support vector regression (SSVR) model and predicting the combustion state of flame. The results of experimentation indicate that SSVR has excellent performance on time series prediction. Compared with traditional time series prediction method such as artificial neural network, SSVR has faster convergence speed and higher fitting precision, which effectively extends the application of SVR.

The forthcoming fifth-generation networks require improvements in cognitive radio intelligence, going towards more smart and aware radio systems. In the emerging radio intelligence approach, the empowerment of cognitive capabilities is performed through the adoption of machine learning techniques. This paper investigates the combined application of the convolutional and recurrent neural networks for the channel state information forecasting, providing a multivariate scalar time series prediction by taking into account the multiple factors dependence of the channel state conditions. Finally, the system performance has been analyzed in terms of prediction accuracy expressed as absolute deviation error and mean percentage error, in comparison with an alternative machine learning method recently proposed in the literature with the aim at solving the same prediction problem.

The power transformer is an important equipment in the power system. Its running state is directly related to the safe and stable operation of the power grid. The volume fraction of the dissolved gas in the oil of the transformer body and its variation law are closely related to the fault mode of the transformer, therefore, the dissolved gas analysis (DGA) technology in transformer oil is widely used. Actually, the relationship between the volume fraction of the dissolved gas in transformer oil and the time is a multi-dimensional time series. The sequences are arranged in the same time interval or in different time intervals, which contains the external environment of the power transformer, the operation conditions and the inherent relationship between the gas content in the transformer oil. Therefore, the operation status of the power transformer can be revealed by the prediction of the time series. Based on this, the fault type of the transformer can be determined. At present, many literatures have been applied to a single time series forecasting method, such as BP neural network and radial basis function neural network. The single prediction method can obtained certain accuracy by adjusting the weights and thresholds of the network. To further improve the prediction accuracy, a combined forecasting model should be proposed. The research result of this paper is to predict the DGA data of transformer, and has a certain guiding significance to determine the transformer fault types. The proposed method for time series prediction can also be used in other time series prediction of the electric power system.

Time series shapelets are small subsequences that maximally differentiate classes of time series. Since the inception of shapelets, researchers have used shapelets for various data domains including anthropology and health care, and in the process suggested many efficient techniques for shapelet discovery. However, multi-dimensional time series data poses unique challenges to shapelet discovery that are yet to be solved. We show that an ensemble of shapelet-based decision trees on individual dimensions works better than shapelets defined over multiple dimensions. Generating a shapelet ensemble for multidimensional time series is computationally expensive. Most of the existing techniques prune shapelet candidates for speed. In this paper, we propose a novel technique for shapelet discovery that evaluates remaining candidates efficiently. Our algorithm uses a multi-length approximate index for time series data to efficiently find the nearest neighbors of the candidate shapelets. We employ a simple skipping technique for additional candidate pruning and a voting based technique to improve accuracy while retaining interpretability. Not only do we find a significant speed increase, our techniques enable us to efficiently discover shapelets on datasets with multi-dimensional and long time series such as hours of brain activity recordings. We demonstrate our approach on a biomedical dataset and find significant differences between patients with schizophrenia and healthy controls.

There exist two difficulties in how to recognize a system with complex structure. One is that the system itself is complicated. The other is that in general we gather the time series of some state of the system only by some “observer”. Hence, we need a technique that to a great degree can be used to “recover” the global behavior of systems. It includes many important research links: the determination of the chaotic characteristics of time series and the dimension of the dynamical system the time series lies in, the establishment of the basic coordinate frame work of dynamical system; the separation of noise dynamics in this frame work and studies on prediction and macroscopic control on this base. This chapter will present some of the recent studies of the authors and the related results achieved in realizing the key technique.

Optimizing simple deterministically constructed cycle reservoir network with a Redundant Unit Pruning Auto-Encoder algorithm