Recurrent Neural Networks for Multimodal Time Series Big Data Analytics

In smart mariculture, traditional methods are not only difficult to adapt to the complex, dynamic and changeable environment in open waters, but also have many problems, such as poor accuracy, high time complexity and poor long-term prediction. To solve these deficiencies, a new water quality prediction method based on TCN (temporal convolutional network) is proposed to predict dissolved oxygen, water temperature, and pH. The TCN prediction network can extract time series features and in-depth data features by introducing dilated causal convolution, and has a good effect of long-term prediction. At the same time, it is predicted that the network can process time series data in parallel, which greatly improves the time throughput of the model. Firstly, we arrange the 23,000 sets of water quality data collected in the cages according to time. Secondly, we use the Pearson correlation coefficient method to analyze the correlation information between water quality parameters. Finally, a long-term prediction model of water quality parameters based on a time domain convolutional network is constructed by using prior information and pre-processed water quality data. Experimental results show that long-term prediction method based on TCN has higher accuracy and less time complexity, compared with RNN (recurrent neural network), SRU (simple recurrent unit), BI-SRU (bi-directional simple recurrent unit), GRU (gated recurrent unit) and LSTM (long short-term memory). The prediction accuracy can reach up to 91.91%. The time costs of training model and prediction are reduced by an average of 64.92% and 7.24%, respectively.

Prediction of dissolved oxygen in a fishery pond based on gated recurrent unit (GRU)

Robust recurrent neural networks for time series forecasting

Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mechanism concept can help RNNs remember and forget previous information. Representing the hidden layers of an RNN with more expressive operations (i.e., tensor products) helps it learn a more complex relationship between the current input and the previous hidden layer information. These ideas can generally improve RNN performances. In this paper, we proposed a novel RNN architecture that combine the concepts of gating mechanism and the tensor product into a single model. By combining these two concepts into a single RNN, our proposed models learn long-term dependencies by modeling with gating units and obtain more expressive and direct interaction between input and hidden layers using a tensor product on 3-dimensional array (tensor) weight parameters. We use Long Short Term Memory (LSTM) RNN and Gated Recurrent Unit (GRU) RNN and combine them with a tensor product inside their formulations. Our proposed RNNs, which are called a Long-Short Term Memory Recurrent Neural Tensor Network (LSTMRNTN) and Gated Recurrent Unit Recurrent Neural Tensor Network (GRURNTN), are made by combining the LSTM and GRU RNN models with the tensor product. We conducted experiments with our proposed models on word-level and character-level language modeling tasks and revealed that our proposed models significantly improved their performance compared to our baseline models.

Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs) have gained significant popularity in time series forecasting across diverse domains including healthcare, astronomy, and engineering. However, the inherent variability in model performance due to random weight initialization raises questions about the reliability and consistency of these architectures for time series analysis. This study addresses this concern by conducting a comprehensive benchmark evaluation of nine neural network architectures: vanilla RNN, LSTM, GRU, and six hybrid configurations (RNN-LSTM, RNN-GRU, LSTM-RNN, GRU-RNN, LSTM-GRU, and GRU-LSTM). Performance evaluation was conducted using Monte Carlo simulation with 100 iterations across three real-world datasets: sunspot activity, Indonesian COVID-19 cases, and dissolved oxygen concentration measurements. Statistical analysis employed the Friedman test to assess performance differences across architectures. Results showed no statistically significant differences among the nine architectures. Despite the lack of statistical significance, consistent performance patterns emerged favoring LSTM-based hybrid architectures. The LSTM-GRU and LSTM-RNN configurations demonstrated superior performance across multiple evaluation metrics, with LSTM-RNN excelling in sunspot and dissolved oxygen forecasting, while standalone LSTM showed optimal performance for COVID-19 prediction. These findings provide evidence-based guidance for architecture selection in time series forecasting applications, suggesting that while statistical equivalence exists among architectures, LSTM-based hybrids offer practical advantages in terms of consistency and robustness across diverse temporal patterns.

To quantitatively evaluate the performance of two types of recurrent neural networks (RNNs), long short-term memory (LSTM) and gated recurrent units (GRU), using Monte Carlo dropout (MCD) to predict pharmacokinetic (PK) parameters from dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) data. DCE-MRI data for simulation studies were synthesized using the extended Tofts model and a population-averaged arterial input function (AIF). The ranges of PK parameters for training the RNNs were determined from data of patients with brain tumors. The effects of the number of training samples, number of hidden units, dropout rate (DR), and bolus arrival time delay and dispersion in AIF on the accuracy of the PK parameters were investigated, and the uncertainties for different DRs and peak signal-to-noise ratios (PSNRs) were quantified. For comparison, PK parameters were estimated using the nonlinear least-squares method. In the clinical studies, the PK parameter and uncertainty images were generated by applying the trained RNNs to DCE-MRI data. Compared with GRU, the computational cost for training the LSTM was significantly higher. The prediction accuracy of GRU decreased with decreasing numbers of training samples and hidden units, whereas the performance of LSTM remained stable. Despite an increased computational cost, MCD reduced the prediction error at low PSNR and improved the quality of PK parameter images. The simulation results recommended using a DR of 0.25-0.5 at low PSNR and ≤ 0.25 for other PSNRs. The clinical studies recommended using a DR of 0.25 and 0.5 for LSTM and GRU, respectively. MCD is effective in quantifying uncertainty in PK parameter prediction from DCE-MRI data and improves their performance, particularly at low PSNR; however, at the expense of increased computational cost. This study helps deepen our understanding of RNNs with MCD and select suitable hyperparameters for creating an RNN architecture for DCE-MRI studies.

MOOCs (Massive Open Online Courses) are definitely one of the best approach to support the international agenda about inclusive and equitable education and lifelong learning opportunities for all (SDG4) [1]. A great deal universities and institutions offer valuable free courses to their numerous students and to people around the word through MOOC platforms. However, because of huge number of learners and data generated, learner’s behaviour in those platforms remain a kind of black box for learners themselves and for courses instructors who are supposed to tutor or monitor learners in the learning process. Therefore, learner do not receive sufficient support from instructors and from their peers, during the learning process [2]. This is one the main reasons that lead to high dropout, low completion and success rate observed in the MOOCs. Many research work have suggested descriptive, predictive and prescriptive models to address this issue, but most of these models focus on predicting dropout, completion and/or success, and do not generally pay enough attention to one of the key step (learner behaviour), that comes before, and can explain dropping out and failure. Our research aims to develop a deep learning model to predict learner behaviour (learner interactions) in the learning process, in order to equip learners and course instructors with insight understanding of the learner behaviour in the learning process. This specific paper will focus on analysing relevant algorithms to develop such model. For this analysis, we used data from UNESCO-IICBA (International Institute for Capacity Building in Africa) MOOC platform, designed for teacher training in Africa, and then we examine many types of features: geographical, social behavioural and learning behavioural features. Learner’s behaviour being a time series Big data, we built the predictive model using Deep Learning algorithms for better performance and accuracy (Thanks to the power of deep learning) compared to baseline Machine learning algorithms. Time series data is best handled by recurrent neural networks (RNN) [3], so, we choose RNN and implemented/tested the three main architectures of RNN: Simple RNNs, GRU (Gated Recurrent Unit) RNNs and LSTM (Long short-term memory) RNNs. The models were trained using L2 Regularization, based on the predictions results, we unexpectedly found model with simple RNNs produced the best performance and accuracy on the dataset used than the other RNN architectures. We had couple of observations, example: we saw a correlation between video viewing and quiz behaviour and the participation of the learner to the learning process. This observation could allow teachers to provide meaningful support and guidance to at risk or less active students. We also observed that, the shorter the video or the quiz, the more the viewer. We conclude that we could use learner video or quiz viewing behaviour to predict his behaviour concerning other MOOC contents. These results suggest the need of deeper research on educational video and educational quiz designing for MOOCs.

Recurrent neural networks (RNN) have been very successful in handling sequence data. However, understanding RNN and finding the best practices for RNN learning is a difficult task, partly because there are many competing and complex hidden units, such as the long short-term memory (LSTM) and the gated recurrent unit (GRU). We propose a gated unit for RNN, named as minimal gated unit (MGU), since it only contains one gate, which is a minimal design among all gated hidden units. The design of MGU benefits from evaluation results on LSTM and GRU in the literature. Experiments on various sequence data show that MGU has comparable accuracy with GRU, but has a simpler structure, fewer parameters, and faster training. Hence, MGU is suitable in RNN's applications. Its simple architecture also means that it is easier to evaluate and tune, and in principle it is easier to study MGU's properties theoretically and empirically.

The main neural network architectures applied for product demand forecasting are examined. Particular attention is given to Recurrent Neural Networks (RNN), which demonstrate high efficiency when working with time series and detecting dependencies within data. A detailed analysis is provided for the Long Short-Term Memory (LSTM) architecture, an advanced variant of RNN that effectively addresses the vanishing gradient problem, a common issue in traditional RNN. Due to its capability to retain information over extended periods, LSTM is ideal for tasks requiring the analysis of long-term dependencies, such as seasonal fluctuations in product demand. The application of Convolutional Neural Networks (CNN) is analyzed, which show strong performance when processing structured data, such as images or matrices. With their ability to effectively detect spatial dependencies, CNN can be useful for demand forecasting when complex relationships between various data features need to be considered, particularly for demand evaluations dependent on multiple factors. The architectures Feedforward Neural Networks (FNN), Gated Recurrent Units (GRU), Attention-based models (ABM) and Autoencoders (AE) are considered, which can also be applied to solve demand forecasting problems. A comparative analysis of the architectures is made in terms of forecast accuracy and ability to work with time series. By evaluating the training speed, data processing time, and reliability of these architectures in terms of the stability of results and forecast reliability, it has been determined that LSTM architectures are the most effective for product demand forecasting. These networks provide high accuracy and reliability of results and perform well with time series, which is crucial for forecasting demand under dynamic conditions.

Recently different Deep Learning (DL) models have been emerging to predict the disease amelioration. Generally deep learning is the state of art for learning the multiple representation of neuron. Discriminant DL model includes different architecture and initial architecture is Recurrent Neural Network (RNN) which learn the data labeled in the form of sequence and it has long term dependency and vanishing gradient problem. Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) is an improved version of RNN which deal with problems in RNN. Deep care LSTM is one of the model in LSTM developed by [Trang Pham et.al] for predicting diabetes disease and compared the results with markovian and support vector machine model. In deep LSTM model some problems are reported like short term trajectories, less accuracy. To overcome the problem Deep Care GRU has been proposed to identify the diabetes disease amelioration.

Energy efficiency, scalability, and reliability are increasingly important for sustainable data centers. In this paper, we focus on forecasting real-world resource usage using neural network time series models, specifically utilizing convolutional recurrent long short-term Memory (LSTM) and gated recurrent unit (GRU) architectures. In our analysis, we compare LSTM and GRU in terms of forecasting accuracy and computational complexity during model training. We demonstrate that recurrent neural networks are more accurate and robust compared to the traditional autoregressive integrated moving average (ARIMA) time series model in this complex forecasting problem. GRU achieved a 9% reduction and LSTM a 5% reduction in forecasting mean squared error (MSE) compared to ARIMA. Furthermore, the GRU architecture with a 1D convolution layer outperforms LSTM architecture in both forecast accuracy and training time. The proposed model can be effectively applied to load forecasting as part of a data center computing cluster. In this application, the proposed GRU architecture has 25% fewer trainable parameters in the recurrent layer than the commonly used LSTM.

In view of the current problems of complex models and insufficient data processing in ultra-short-term prediction of photovoltaic power generation, this paper proposes a photovoltaic power ultra-short-term prediction model named HPO-KNN-SRU, based on a Simple Recurrent Unit (SRU), K-Nearest Neighbors (KNN), and Hunter–Prey Optimization (HPO). Firstly, the sliding time window is determined by using the autocorrelation function (ACF), partial correlation function (PACF), and model training. The Pearson correlation coefficient method is used to filter the principal meteorological factors that affect photovoltaic power. Then, the K-Nearest Neighbors (KNN) algorithm is utilized for effective outlier detection and processing to ensure the quality of input data for the prediction model, and the Hunter–Prey Optimization (HPO) algorithm is applied to optimize the parameters of the KNN algorithm. Finally, the efficient Simple Recurrent Unit (SRU) model is used for training and prediction, with the Hunter–Prey Optimization (HPO) algorithm applied to optimize the parameters of the SRU model. Simulation experiments and extensive ablation studies using photovoltaic data from the Desert Knowledge Australia Solar Centre (DKASC) in Alice Springs, Australia, validate the effectiveness of the integrated model, the KNN outlier handling, and the HPO algorithm. Compared to the Support Vector Regression (SVR), Long Short-Term Memory (LSTM), Temporal Convolutional Network (TCN), and Simple Recurrent Unit (SRU) models, this model exhibits an average reduction of 19.63% in Mean Square Error (RMSE), 27.54% in Mean Absolute Error (MAE), and an average increase of 1.96% in coefficient of determination (R2) values.

Online evolutionary neural architecture search for multivariate non-stationary time series forecasting

As a new type of currency introduced in the new millennium, cryptocurrency has established its ecosystems and attracts many people to use and invest in it. However, cryptocurrencies are highly dynamic and volatile, making it challenging to predict their future values. In this research, we use a multivariate prediction approach and three different recurrent neural networks (RNNs), namely the long short-term memory (LSTM), the bidirectional LSTM (Bi-LSTM), and the gated recurrent unit (GRU). We also propose simple three layers deep networks architecture for the regression task in this study. From the experimental results on five major cryptocurrencies, i.e., Bitcoin (BTC), Ethereum (ETH), Cardano (ADA), Tether (USDT), and Binance Coin (BNB), we find that both Bi-LSTM and GRU have similar performance results in terms of accuracy. However, in terms of the execution time, both LSTM and GRU have similar results, where GRU is slightly better and has lower variation results on average.

This study investigates the application of advanced deep learning models to forecast fossil energy prices, a critical factor influencing global economic stability. Unlike previous research, this study conducts a comparative analysis of Gated Recurrent Unit (GRU), Recurrent Neural Network (RNN), Bidirectional Long Short-Term Memory (Bi-LSTM), Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Deep Neural Network (DNN) models. The evaluation metrics employed include Root Mean Squared Error (RMSE) and Mean Absolute Percentage Error (MAPE). The results reveal that recurrent architectures, particularly GRU, LSTM, and Bi-LSTM, consistently outperform feedforward and convolutional models, demonstrating superior ability to capture temporal dependencies and nonlinear dynamics in energy markets. In contrast, the RNN and DNN show relatively weaker generalization capabilities. Additionally, visualizations of actual versus predicted prices for each model further emphasize superior forecasting accuracy of recurrent models. The results highlight the potential of deep learning in enhancing investment and policy decisions. Additionally, the results provide significant implications for policymakers and investors by emphasizing the value of accurate energy price forecasting in mitigating market volatility, improving portfolio management, and supporting evidence-based energy policies.