Data-Centric Benchmarking of Neural Network Architectures for the Univariate Time Series Forecasting Task

Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs) have gained significant popularity in time series forecasting across diverse domains including healthcare, astronomy, and engineering. However, the inherent variability in model performance due to random weight initialization raises questions about the reliability and consistency of these architectures for time series analysis. This study addresses this concern by conducting a comprehensive benchmark evaluation of nine neural network architectures: vanilla RNN, LSTM, GRU, and six hybrid configurations (RNN-LSTM, RNN-GRU, LSTM-RNN, GRU-RNN, LSTM-GRU, and GRU-LSTM). Performance evaluation was conducted using Monte Carlo simulation with 100 iterations across three real-world datasets: sunspot activity, Indonesian COVID-19 cases, and dissolved oxygen concentration measurements. Statistical analysis employed the Friedman test to assess performance differences across architectures. Results showed no statistically significant differences among the nine architectures. Despite the lack of statistical significance, consistent performance patterns emerged favoring LSTM-based hybrid architectures. The LSTM-GRU and LSTM-RNN configurations demonstrated superior performance across multiple evaluation metrics, with LSTM-RNN excelling in sunspot and dissolved oxygen forecasting, while standalone LSTM showed optimal performance for COVID-19 prediction. These findings provide evidence-based guidance for architecture selection in time series forecasting applications, suggesting that while statistical equivalence exists among architectures, LSTM-based hybrids offer practical advantages in terms of consistency and robustness across diverse temporal patterns.

The prediction of wind energy generation is important to enhance the performance and dependability of renewable energy systems due to the rising demand for wind-generated electricity and advancements in wind energy technology competitiveness. This study leverages advanced machine learning (ML) and some other statistical and deep learning based time series forecasting models to enhance the accuracy of wind energy predictions. This comprehensive analysis includes nine ML models—Linear Regression, Random Forests (RF), Gradient Boosting Machines (GBM), Support Vector Machines (SVM), K-Nearest Neighbors (KNN), AdaBoost, XGBoost, Support Vector Regression (SVR), and Neural Networks—as well as Four time-series forecasting models—ARIMA, Temporal Convolutional Networks (TCNs), Long Short-Term Memory (LSTM) networks and GRU. Each ML model underwent rigorous cross-validation to ensure optimal performance. The assessment criteria utilized here comprised the Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and the R² Score. It was found that among the nine ML models, Random Forests, GBM and KNN consistently provided superior accuracy and robustness, making them the top choices for wind energy prediction whereas the performance of linear regression, SVM and SVR were very poor for the considered dataset. From the experiment, Random Forest, GBM, and KNN showed the best performance with low MSE values of 0.77, 1.95, and 1.51 respectively, while other models had MSEs above 7.5, with AdaBoost reaching 30. Their RMSEs (0.88, 1.40, 1.23) and MAEs (0.093, 0.73, 0.10) also indicate strong predictive accuracy compared to the rest.In this paper, time series forecasting, TCNs, LSTM and GRU networks showed strong capabilities in capturing temporal dependencies and trends within the wind energy data. Visualization techniques were employed to compare model performances comprehensively, providing clear insights into their predictive power. Therefore, this present study offers a robust framework for researchers and practitioners aiming to leverage machine learning and time series forecasting in the realm of renewable energy prediction.

Poly-linear regression with augmented long short term memory neural network: Predicting time series data

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

ABSTRACTIn this paper, the convolutional recurrent neural network (ConvLSTM) architecture is proposed to predict individual stock prices. The characteristics of stock data are automatically extracted through convolutional neural network (CNN). Afterward, the extracted features are inputted into a long short-term memory (LSTM) model with memory characteristics for prediction. Since CNN has been a representation learning model, it is quite appropriate for automatic feature extraction. Besides, the LSTM architecture of the recurrent neural network could effectively surmount the issues of gradient disappearance and expansion of the time series data. Ten stock historical data had been collected in the experimental data set. Furthermore, many commonly used technical indicators are calculated in advance for the expansion of the dimension of the training samples. The experimental results obtain a 3.449 RMSE (root-mean-square error) in average.

Online leakage current classification using convolutional neural network long short-term memory for high voltage insulators on web-based service

Analyzing the dynamics between crude oil spot prices and futures prices by maturity terms: Deep learning approaches to futures-based forecasting

Data science highlights fields of study and research such as time series, which, although widely explored in the past, gain new perspectives in the context of this discipline. This chapter presents two approaches to time series forecasting, long short-term memory (LSTM), a special kind of recurrent neural network (RNN), and Prophet, an open-source library developed by Facebook for time series forecasting. With a focus on developing forecasting processes by data mining or machine learning experts, LSTM uses gating mechanisms to deal with long-term dependencies, reducing the short-term memory effect inherent to the traditional RNN. On the other hand, Prophet encapsulates statistical and computational complexity to allow broad use of time series forecasting, prioritizing the expert's business knowledge through exploration and experimentation. Both approaches were applied to a retail time series. This case study comprises daily and half-hourly forecasts, and the performance of both methods was measured using the standard metrics.

Forecasting stock prices has always been a complex and challenging task due to its high volatility, noise in data, and abrupt price fluctuations caused by different factors. Traditional approaches, such as quantitative, fundamental, and technical analysis, often struggle to accurately predict stock price movements due to their difficulty in processing non-linear data. The evolution of machine learning, particularly neural network models, opened new opportunities for improving prediction accuracy by handling non-linear and multidimensional data with high effectiveness. This study focuses on evaluating and comparing three neural network architectures, in which the last is a combination of the first two - Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and a hybrid CNN-LSTM model, for forecasting stock prices. Using ten years of historical stock data, taken for Apple Inc. (AAPL) stocks as an example, ranging from 2015 to 2025, the models were trained and tested on key performance metrics such as Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and R². Results show that while both CNN and LSTM models show good performance individually, the hybrid CNN-LSTM model can significantly outperform them, achieving the highest accuracy with the lowest error rates. By combining the strengths of CNN for feature extraction and LSTM for temporal sequence processing, the hybrid model effectively captures complex patterns in stock price movements. By iterating over the types of layers to use, we came to the conclusion, that by combining both models into one hybrid model, some layers, like one CNN layer, and one LSTM layer should be removed. This research demonstrates the potential of hybrid neural network models in financial time-series forecasting and comprehensively analyzes their advantages over non-hybrid models.

Improving the performance of Deep Learning (DL) algorithms is a challenging problem. However, DL is applied to different types of Deep Neural Networks, and Long Short-Term Memory (LSTM) is one of them that deals with time series or sequential data. This paper attempts to overcome this problem by optimizing LSTM parameters using the Bees Algorithm (BA), which is a nature-inspired algorithm that mimics the foraging behavior of honey bees. In particular, it was used to optimize the adjustment factors of the learning rate in the forget, input, and output gates, in addition to cell candidate, in both forward and backward sides. Furthermore, the BA was used to optimize the learning rate factor in the fully connected layer. In this study, artificial porosity images were used for testing the algorithms; since the input data were images, a Convolutional Neural Network (CNN) was added in order to extract the features in the images to feed into the LSTM for predicting the percentage of porosity in the sequential layers of artificial porosity images that mimic real CT scan images of products manufactured by the Selective Laser Melting (SLM) process. Applying a Convolutional Neural Network Long Short-Term Memory (CNN-LSTM) yielded a porosity prediction accuracy of 93.17%. Although using Bayesian Optimization (BO) to optimize the LSTM parameters mentioned previously did not improve the performance of the LSTM, as the prediction accuracy was 93%, adding the BA to optimize the same LSTM parameters did improve its performance in predicting the porosity, with an accuracy of 95.17% where a hybrid Bees Algorithm Convolutional Neural Network Long Short-Term Memory (BA-CNN-LSTM) was used. Furthermore, the hybrid BA-CNN-LSTM algorithm was capable of dealing with classification problems as well. This was shown by applying it to Electrocardiogram (ECG) benchmark images, which improved the test set classification accuracy, which was 92.50% for the CNN-LSTM algorithm and 95% for both the BO-CNN-LSTM and BA-CNN-LSTM algorithms. In addition, the turbofan engine degradation simulation numerical dataset was used to predict the Remaining Useful Life (RUL) of the engines using the LSTM network. A CNN was not needed in this case, as there was no feature extraction for the images. However, adding the BA to optimize the LSTM parameters improved the prediction accuracy in the testing set for the LSTM and BO-LSTM, which increased from 74% to 77% for the hybrid BA-LSTM algorithm.

Neural networks (NNs) are well established and widely used in time series forecasting due to their frequent dominance over other linear and nonlinear models. Thus, this paper does not question their appropriateness in forecasting cryptocurrency prices; rather, it compares the most commonly used NNs, i.e. feedforward neural networks (FFNNs), long short-term memory (LSTM) and convolutional neural networks (CNNs). This paper contributes to the existing literature by defining the appropriate NN structure comparable across different NN architectures, which yields the optimal NN model for Bitcoin return forecasting. Moreover, by incorporating turbulent events such as COVID and war, this paper emerges as a stress test for NNs. Finally, inputs are carefully selected, mostly covering macroeconomic and market variables, as well as different attractiveness measures, the importance of which in cryptocurrency forecasting is tested. The main results indicate that all NNs perform the best in an environment of bullish market, where CNNs stand out as the optimal models for continuous dataset, and LSTMs emerge as optimal in direction forecasting. In the downturn periods, CNNs stand out as the best models. Additionally, Tweets, as an attractiveness measure, enabled the models to attain superior performance.

El Niño is an important quasi-cyclical climate phenomenon that can have a significant impact on ecosystems and societies. Due to the chaotic nature of the atmosphere and ocean systems, traditional methods (such as statistical methods) are difficult to provide accurate El Niño index predictions. The latest research shows that Ensemble Empirical Mode Decomposition (EEMD) is suitable for analyzing non-linear and non-stationary signal sequences, Convolutional Neural Network (CNN) is good at local feature extraction, and Recurrent Neural Network (RNN) can capture the overall information of the sequence. As a special RNN, Long Short-Term Memory (LSTM) has significant advantages in processing and predicting long, complex time series. In this paper, to predict the El Niño index more accurately, we propose a new hybrid neural network model, EEMD-CNN-LSTM, which combines EEMD, CNN, and LSTM. In this hybrid model, the original El Niño index sequence is first decomposed into several Intrinsic Mode Functions (IMFs) using the EEMD method. Next, we filter the IMFs by setting a threshold, and we use the filtered IMFs to reconstruct the new El Niño data. The reconstructed time series then serves as input data for CNN and LSTM. The above data preprocessing method, which first decomposes the time series and then reconstructs the time series, uses the idea of symmetry. With this symmetric operation, we extract valid information about the time series and then make predictions based on the reconstructed time series. To evaluate the performance of the EEMD-CNN-LSTM model, the proposed model is compared with four methods including the traditional statistical model, machine learning model, and other deep neural network models. The experimental results show that the prediction results of EEMD-CNN-LSTM are not only more accurate but also more stable and reliable than the general neural network model.

The main neural network architectures applied for product demand forecasting are examined. Particular attention is given to Recurrent Neural Networks (RNN), which demonstrate high efficiency when working with time series and detecting dependencies within data. A detailed analysis is provided for the Long Short-Term Memory (LSTM) architecture, an advanced variant of RNN that effectively addresses the vanishing gradient problem, a common issue in traditional RNN. Due to its capability to retain information over extended periods, LSTM is ideal for tasks requiring the analysis of long-term dependencies, such as seasonal fluctuations in product demand. The application of Convolutional Neural Networks (CNN) is analyzed, which show strong performance when processing structured data, such as images or matrices. With their ability to effectively detect spatial dependencies, CNN can be useful for demand forecasting when complex relationships between various data features need to be considered, particularly for demand evaluations dependent on multiple factors. The architectures Feedforward Neural Networks (FNN), Gated Recurrent Units (GRU), Attention-based models (ABM) and Autoencoders (AE) are considered, which can also be applied to solve demand forecasting problems. A comparative analysis of the architectures is made in terms of forecast accuracy and ability to work with time series. By evaluating the training speed, data processing time, and reliability of these architectures in terms of the stability of results and forecast reliability, it has been determined that LSTM architectures are the most effective for product demand forecasting. These networks provide high accuracy and reliability of results and perform well with time series, which is crucial for forecasting demand under dynamic conditions.

In recent years Neural Networks and especially Deep Neural Networks (DNN) have seen a rise in popularity. DNNs have increased the overall performance of algorithms in applications such as image recognition and classification, 2D and 3D pose detection, natural language processing, and time series forecasting. Especially in image classification and recognition, so-called Convolutional Neural Network (CNN) have gained high interest as they can reach a high accuracy, which makes them a viable solution in cancer detection or autonomous driving. As CNNs are widely used in image tasks, different visualizations techniques have bee developed to show how their internals are working. Apart from image tasks, CNNs are applicable to other problems, e.g., time series classification, time series forecasting, or natural language processing. CNNs in those contexts behave similarly, allowing for the same visualization techniques to make them more interpretable. In this chapter, we adapt image visualization algorithms to time series problems, allowing us to build granular, intuitively interpretable feature hierarchies to make a time series forecast as understandable as an image recognition task. We do so by using our previous work on power time series forecasting using CNN Auto Encoders (AEs) and applying typical CNN visualization techniques to it. Thus, we guid computer scientists to provide better interpretable figures for a time series forecasting task to application domain experts.KeywordsVisualizationConvolutional neural networksTime series data

This study assesses the effectiveness of deep learning models, specifically Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), and Long Short-Term Memory (LSTM), in assessing synthetic grip strength data for stress ball and pencil grips. The aim is to determine the optimal model for grip strength classification and tackle issues such as temporal data and class imbalance. Synthetic data was created, preprocessed, and utilized to train the models on Google Colab employing TensorFlow and Keras. Hyperparameters, such as CNN kernel dimensions and RNN/LSTM sequence lengths, were refined. Performance was evaluated by accuracy, precision, F1 scores, and loss metrics. The RNN model surpassed both CNN and LSTM, with an accuracy of 94.99% and a precision of 0.56, particularly excelling in the classification of “Normal” grip strength instances. CNN and LSTM had difficulties with time dependencies and class imbalance, leading to reduced accuracy and increased loss. Although RNN has proven to be the most effective approach, subsequent research should concentrate on mitigating class imbalance and investigating hybrid models to improve performance.