Spatio–Temporal Attention-Based Deep Learning Framework for Mesoscale Eddy Trajectory Prediction

With the advances in technology and smart devices, more and more attention has been paid to model spatial correlations, temporal dynamics, and friendship influence over point-of-interest (POI) checkins. Besides directly capturing general user's checkin behavior, existing works mostly highlight the intrinsic feature of POIs, i.e., spatial and temporal dependency. Among them, the family of methods based on Markov chain can capture the instance-level interaction between a pair of POI checkins, while recurrent neural network (RNN) based approaches (state-of-the-art) can deal with flexible length of checkin sequence. However, the former is not good at capturing high-order POI transition dependency, and the latter cannot distinguish the exact contribution of each POI in a historical checkin sequence. Moreover, in recurrent neural networks, local and global information is propagated along the sequence through one bottleneck i.e., hidden states only.In this work, we design a novel model to enforce contextual constraints on sequential data by designing a spatial and temporal attention mechanisms over recurrent neural network that leverages the importance of POIs visited by users in given time interval and geographical distance in successive checkins. Attention mechanism helps us to learn which POIs bounded by time difference and spatial distance in user checkin history are important for the prediction of next POI. Moreover, we also consider periodicity and friendship influence in our model design. Experimental results on two real location based social networks Gowalla, and BrightKite show that our proposed method outperforms the existing state-of-the-art deep neural network methods for next POI prediction and understanding user transition behavior. We also analyze the sensitivity of parameters including context window for capturing sequential effect, temporal context window for estimating temporal attention and spatial context window for estimating spatial attention respectively.

This study examines the predictive performance of Feature Attention (FA), Temporal Attention (TA), and Feature and Temporal Attention (FATA) within Gated Recurrent Unit (GRU), Long Short‐Term Memory (LSTM), and Transformer architectures using price data from four Chinese carbon markets (CEA, BEA, GDEA, and HBEA). Drawing on multiple forecasting accuracy measures and significance testing, the results show that attention mechanisms can enhance forecasting accuracy in certain market‐model combinations, but their effectiveness critically depends on the alignment among market conditions, model architectures, and attention mechanisms. In markets with high average prices and volatility, FA achieves the best performance with GRU and LSTM; in lower price, moderately volatile markets, TA combined with Transformer is more effective; and in the high‐price, high‐volatility CEA market, FATA shows promise when paired with Transformer, but lacks robustness across markets. These findings highlight a pronounced compatibility pattern among market conditions, model architectures, and attention mechanisms, suggesting that the deployment of attention mechanisms in carbon price forecasting should be tailored to specific market conditions and model structures rather than applied universally.

Prediction of dissolved oxygen in a fishery pond based on gated recurrent unit (GRU)

In this study, different kinds of hybrid Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) algorithms with forecasting models including Random Forest (RF), Gated Recurrent Unit (GRU), and Long Short-Term Memory (LSTM) neural networks, are developed to estimate the mean daily wind speed at the height of 2 m in Ağrı city (WS st12 ), Turkey. In these hybrid models, different layer networks of single and integrated LSTM and GRU models include general single LSTM, general single GRU, simple coupled LSTM-GRU, and novel coupled LSTM with GRU through Addition layer (i.e., LSTM + GRU model) structures are applied. The most effective parameters on the WS st12 , from a list of on-site potential meteorological parameters and wind speed values in its adjacent cities of Ağrı province from Jan 2015–Dec 2019 through the Pearson correlation coefficient method, are determined. In the hybrid CEEMDAN and DNNs-based models, State activation functions (SAF), numbers of hidden neurons (NHN), dropout rates (P-rate), and network structural architect (NSA) as the meta-parameters are tuned for lessening the impact of overfitting/underfitting dilemmas and improving modeling performance. According to the comparison plots, performance evaluation measures, and total learnable parameter (TLP), the novel developed hybrid CEEMDAN-RF-(LSTM + GRU) model is confirmed as the best approach with an R 2 of 0.86 while, in the optimal scenario using the RF model, R 2 was 0.47. Graphical Abstract Based on the graphical snapshot, this study focuses on estimating daily mean wind speed at a 2-meter height in Ağrı, Turkey, using hybrid data-driven models. The research integrates the Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) algorithm with advanced forecasting techniques, including Random Forest (RF), Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU) neural networks. The modeling framework explores various configurations, such as standalone LSTM and GRU, coupled LSTM-GRU structures, and a novel LSTM + GRU model using an Addition layer to enhance predictive accuracy.

BackgroundThe prediction of coronavirus disease in 2019 (COVID-19) in broader regions has been widely researched, but for specific areas such as urban areas the predictive models were rarely studied. It may be inaccurate to apply predictive models from a broad region directly to a small area. This paper builds a prediction approach for small size COVID-19 time series in a city.MethodsNumbers of COVID-19 daily confirmed cases were collected from November 1, 2022 to November 16, 2023 in Xuzhou city of China. Classical deep learning models including recurrent neural network (RNN), long and short-term memory (LSTM), gated recurrent unit (GRU) and temporal convolutional network (TCN) are initially trained, then RNN, LSTM and GRU are integrated with a new attention mechanism and transfer learning to improve the performance. Ten times ablation experiments are conducted to show the robustness of the performance in prediction. The performances among the models are compared by the mean absolute error, root mean square error and coefficient of determination.ResultsLSTM outperforms than others, and TCN has the worst generalization ability. Thus, LSTM is integrated with the new attention mechanism to construct an LSTMATT model, which improves the performance. LSTMATT is trained on the smoothed time series curve through frequency domain convolution augmentation, then transfer learning is adopted to transfer the learned features back to the original time series resulting in a TLLA model that further improves the performance. RNN and GRU are also integrated with the attention mechanism and transfer learning and their performances are also improved, but TLLA still performs best.ConclusionsThe TLLA model has the best prediction performance for the time series of COVID-19 daily confirmed cases, and the new attention mechanism and transfer learning contribute to improve the prediction performance in the flatten part and the jagged part, respectively.

Drones are increasingly used in several industries, including rescue, firefighting, and agriculture. If the motor connected to a drone’s propeller is damaged, there is a risk of a drone crash. Therefore, to prevent such incidents, an accurate and quick prediction tool of the motor vibrations in drones is required. In this study, normal and abnormal vibration data were collected from the motor connected to the propeller of a drone. The period and amplitude of the vibrations are consistent in normal vibrations, whereas they are irregular in abnormal vibrations. The collected vibration data were used to train six recurrent neural network (RNN) techniques: long short-term memory (LSTM), attention-LSTM (Attn.-LSTM), bidirectional-LSTM (Bi-LSTM), gated recurrent unit (GRU), attention-GRU (Attn.-GRU), and bidirectional GRU (Bi-GRU). Then, the simulation runtime it took for each RNN technique to predict the vibrations and the accuracy of the predicted vibrations were analyzed to compare the performances of the RNN model. Based on the simulation results, the Attn.-LSTM and Attn.-GRU techniques, incorporating the attention mechanism, had the best efficiency compared to the conventional LSTM and GRU techniques, respectively. The attention mechanism calculates the similarity between the input value and the to-be-predicted value in advance and reflects the similarity in the prediction.

As the fundamental part of other Intelligent Transportation Systems (ITS) applications, short-term traffic volume prediction plays an important role in various intelligent transportation tasks, such as traffic management, traffic signal control and route planning. Although Neural-network-based traffic prediction methods can produce good results, most of the models can’t be explained in an intuitive way. In this paper, we not only proposed a model that increase the short-term prediction accuracy of the traffic volume, but also improved the interpretability of the model by analyzing the internal attention score learnt by the model. we propose a spatiotemporal attention mechanism-based multistep traffic volume prediction model (SAMM). Inside the model, an LSTM-based Encoder-Decoder network with a hybrid attention mechanism is introduced, which consists of spatial attention and temporal attention. In the first level, the local and global spatial attention mechanisms considering the micro traffic evolution and macro pattern similarity, respectively, are applied to capture and amplify the features from the highly correlated entrance stations. In the second level, a temporal attention mechanism is employed to amplify the features from the time steps captured as contributing more to the future exit volume. Considering the time-dependent characteristics and the continuity of the recent evolutionary traffic volume trend, the timestamp features and historical exit volume series of target stations are included as the external inputs. An experiment is conducted using data from the highway toll collection system of Guangdong Province, China. By extracting and analyzing the weights of the spatial and temporal attention layers, the contributions of the intermediate parameters are revealed and explained with knowledge acquired by historical statistics. The results show that the proposed model outperforms the state-of-the-art model by 29.51% in terms of MSE, 13.93% in terms of MAE, and 5.69% in terms of MAPE. The effectiveness of the Encoder-Decoder framework and the attention mechanism are also verified.

To quantitatively evaluate the performance of two types of recurrent neural networks (RNNs), long short-term memory (LSTM) and gated recurrent units (GRU), using Monte Carlo dropout (MCD) to predict pharmacokinetic (PK) parameters from dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) data. DCE-MRI data for simulation studies were synthesized using the extended Tofts model and a population-averaged arterial input function (AIF). The ranges of PK parameters for training the RNNs were determined from data of patients with brain tumors. The effects of the number of training samples, number of hidden units, dropout rate (DR), and bolus arrival time delay and dispersion in AIF on the accuracy of the PK parameters were investigated, and the uncertainties for different DRs and peak signal-to-noise ratios (PSNRs) were quantified. For comparison, PK parameters were estimated using the nonlinear least-squares method. In the clinical studies, the PK parameter and uncertainty images were generated by applying the trained RNNs to DCE-MRI data. Compared with GRU, the computational cost for training the LSTM was significantly higher. The prediction accuracy of GRU decreased with decreasing numbers of training samples and hidden units, whereas the performance of LSTM remained stable. Despite an increased computational cost, MCD reduced the prediction error at low PSNR and improved the quality of PK parameter images. The simulation results recommended using a DR of 0.25-0.5 at low PSNR and ≤ 0.25 for other PSNRs. The clinical studies recommended using a DR of 0.25 and 0.5 for LSTM and GRU, respectively. MCD is effective in quantifying uncertainty in PK parameter prediction from DCE-MRI data and improves their performance, particularly at low PSNR; however, at the expense of increased computational cost. This study helps deepen our understanding of RNNs with MCD and select suitable hyperparameters for creating an RNN architecture for DCE-MRI studies.

Given the substantial volatility and non-stationarity of cryptocurrency prices, forecasting them has become a complex task within the realm of financial time series analysis. This study introduces an innovative hybrid prediction model, VMD-AGRU-RESVMD-LSTM, which amalgamates the disintegration–integration framework with deep learning techniques for accurate cryptocurrency price prediction. The process begins by decomposing the cryptocurrency price series into a finite number of subseries, each characterized by relatively simple volatility patterns, using the variational mode decomposition (VMD) method. Next, the gated recurrent unit (GRU) neural network, in combination with an attention mechanism, predicts each modal component’s sequence separately. Additionally, the residual sequence, obtained after decomposition, undergoes further decomposition. The resultant residual sequence components serve as input to an attentive GRU (AGRU) network, which predicts the residual sequence’s future values. Ultimately, the long short-term memory (LSTM) neural network integrates the predictions of modal components and residuals to yield the final forecasted price. Empirical results obtained for daily Bitcoin and Ethereum data exhibit promising performance metrics. The root mean square error (RMSE) is reported as 50.651 and 2.873, the mean absolute error (MAE) stands at 42.298 and 2.410, and the mean absolute percentage error (MAPE) is recorded at 0.394% and 0.757%, respectively. Notably, the predictive outcomes of the VMD-AGRU-RESVMD-LSTM model surpass those of standalone LSTM and GRU models, as well as other hybrid models, confirming its superior performance in cryptocurrency price forecasting.

Despite a lot of research efforts devoted in recent years, how to efficiently learn long-term dependencies from sequences still remains a pretty challenging task. As one of the key models for sequence learning, recurrent neural network (RNN) and its variants such as long short term memory (LSTM) and gated recurrent unit (GRU) are still not powerful enough in practice. One possible reason is that they have only feedforward connections, which is different from the biological neural system that is typically composed of both feedforward and feedback connections. To address this problem, this paper proposes a biologically-inspired deep network, called shuttleNet\footnote{Our code is available at \url{https://github.com/shiyemin/shuttlenet}}. Technologically, the shuttleNet consists of several processors, each of which is a GRU while associated with multiple groups of cells and states. Unlike traditional RNNs, all processors inside shuttleNet are loop connected to mimic the brain's feedforward and feedback connections, in which they are shared across multiple pathways in the loop connection. Attention mechanism is then employed to select the best information flow pathway. Extensive experiments conducted on two benchmark datasets (i.e UCF101 and HMDB51) show that we can beat state-of-the-art methods by simply embedding shuttleNet into a CNN-RNN framework.

Energy efficiency, scalability, and reliability are increasingly important for sustainable data centers. In this paper, we focus on forecasting real-world resource usage using neural network time series models, specifically utilizing convolutional recurrent long short-term Memory (LSTM) and gated recurrent unit (GRU) architectures. In our analysis, we compare LSTM and GRU in terms of forecasting accuracy and computational complexity during model training. We demonstrate that recurrent neural networks are more accurate and robust compared to the traditional autoregressive integrated moving average (ARIMA) time series model in this complex forecasting problem. GRU achieved a 9% reduction and LSTM a 5% reduction in forecasting mean squared error (MSE) compared to ARIMA. Furthermore, the GRU architecture with a 1D convolution layer outperforms LSTM architecture in both forecast accuracy and training time. The proposed model can be effectively applied to load forecasting as part of a data center computing cluster. In this application, the proposed GRU architecture has 25% fewer trainable parameters in the recurrent layer than the commonly used LSTM.

Production prediction plays an important role in decision making, development planning, and economic evaluation during the exploration and development period. However, applying traditional methods for production forecasting of newly developed wells in the conglomerate reservoir is restricted by limited historical data, complex fracture propagation, and frequent operational changes. This study proposed a Gated Recurrent Unit (GRU) neural network-based model to achieve batch production forecasting in M conglomerate reservoir of China, which tackles the limitations of traditional decline curve analysis and conventional time-series prediction methods. The model is trained by four features of production rate, tubing pressure (TP), choke size (CS), and shut-in period (SI) from 70 multistage hydraulic fractured horizontal wells. Firstly, a comprehensive data preprocessing is implemented, including excluding unfit wells, data screening, feature selection, partitioning data set, z-score normalization, and format conversion. Then, the four-feature model is compared with the model considering production only, and it is found that with frequent oilfield operations changes, the four-feature model could accurately capture the complex variance pattern of production rate. Further, Random Forest (RF) is employed to optimize the prediction results of GRU. For a fair evaluation, the performance of the proposed model is compared with that of simple Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) neural network. The results show that the proposed approach outperforms the others in prediction accuracy and generalization ability. It is worth mentioning that under the guidance of continuous learning, the GRU model can be updated as soon as more wells become available.

In recent years, machine learning and Deep Learning have increased and gathered epic success in traditional application domains and new areas of Artificial Intelligence. The performance using Deep Learning has dominated experimental results compared to conventional machine learning algorithms. This paper presents an overview of the progress that has occurred in Deep Learning (DL) concerning some application domains like Autonomous Driving, Healthcare, Voice Recognition, Image Recognition, Advertising, Predicting Natural Calamities, National Stock Exchange and many more. Additionally, deeper insights into several Deep Learning techniques, their working principles, and experimental results are scrutinized. The survey covers Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), including Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), Auto-Encoder (AE), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Deep Reinforcement Learning (DRL).

Credit risk refers to the possibility that a borrower fails to meet contractual repayment obligations, posing potential losses to lenders. This study aims to enhance post-loan default prediction in credit risk management by constructing a time-series modeling framework based on repayment behavior data, enabling the capture of repayment risks that emerge after loan issuance. To achieve this objective, a Residual Enhanced Encoder Bidirectional Long Short-Term Memory (ResE-BiLSTM) model is proposed, in which the attention mechanism is responsible for discovering long-range correlations, while the residual connections ensure the preservation of distant information. This design mitigates the tendency of conventional recurrent architectures to overemphasize recent inputs while underrepresenting distant temporal information in long-term dependency modeling. Using the real-world large-scale Freddie Mac Single-Family Loan-Level Dataset, the model is evaluated on 44 independent cohorts and compared with five baseline models, including Long Short-Term Memory (LSTM), Bidirectional LSTM (BiLSTM), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) across multiple evaluation metrics. The experimental results demonstrate that ResE-BiLSTM achieves superior performance on key indicators such as F1 and AUC, with average values of 0.92 and 0.97, respectively, and demonstrates robust performance across different feature window lengths and resampling settings. Ablation experiments and SHapley Additive exPlanations (SHAP)-based interpretability analyses further reveal that the model captures non-monotonic temporal importance patterns across key financial features. This study advances time-series–based anomaly detection for credit risk prediction by integrating global and local temporal learning. The findings offer practical value for financial institutions and risk management practitioners, while also providing methodological insights and a transferable modeling paradigm for future research on credit risk assessment.

Automatic sentence generation is an important problem in natural language processing that has many applications, including language translation, summarization, and chatbots. Deep learning techniques, such as recurrent neural networks (RNNs) and transformer models, have been shown to be effective in generating coherent and diverse sentences. Recurrent neural networks (RNNs) have been widely used in natural language processing tasks, including automatic text generation. However, the traditional RNN suffers from the vanishing gradient problem, which hinders the learning of long-term dependencies. To address this issue, long short-term memory (LSTM) and gated recurrent unit (GRU) models have been introduced that can selectively forget or update certain information in the hidden state. These models have been shown to improve the quality of automatically generated text by better capturing the long-term dependencies in the input data.