Modeling and simulating of reservoir operation using the artificial neural network, support vector regression, deep learning algorithm

Abstract. The information available on the internet nowadays is diverse and moves very quickly. Information is becoming easier to obtain by the general public with the numerous online media outlets, including news portals that provide up-to-date information insights. Various news portals earn revenue from advertising using pay-per-click methods that encourage article writers to use clickbait techniques to attract visitors. However, the negative effects of clickbait include a decrease in journalism quality and the spread of hoaxes. This problem can be prevented by using text classification to classify clickbait in news titles. One method that can be used for text classification is a neural network. Artificial neural networks use algorithms that can independently adjust input coefficient weights. This makes this algorithm highly effective for modeling non-linear statistical data. The artificial neural network algorithm, especially the Long Short-Term Memory (LSTM), has been widely used in various natural language processing fields with satisfying results, including text classification. To improve the performance of the neural network model, adjustments can be made to the model's hyperparameters. Hyperparameters are parameters that cannot be obtained through data and must be defined before the training process. In this research, the Long Short-Term Memory (LSTM) model was used in clickbait classification in news titles. Sixteen neural network models were trained with different hyperparameter configurations for each model. Hyperparameter tuning was carried out using the random search algorithm. The dataset used was the CLICK-ID dataset published by William & Sari, 2020[1], with a total of 15,000 annotated data. The research results show that the developed LSTM model has a validation accuracy of 0.8030, higher than William & Sari's research, and a validation loss of 0.4876. Using this model, researchers were able to classify clickbait in news titles with fairly good accuracy. Purpose: The study was to develop and evaluate a LSTM model with hyperparameter tuning for clickbait classification on news headlines. The thesis also aims to compare the performance of simple LSTM and bidirectional LSTM for this task. Methods: This study uses CLICK-ID dataset and applies different text preprocessing techniques. The dataset later was used to build and train 16 LSTM models with different hyperparameters and evaluates them using validation accuracy and loss. This study uses random search for hyperparameter tuning. Result: The results of the study show that the best model for clickbait classification on news headlines is a bidirectional LSTM model with one layer, 64 units, 0.2 dropout rate, and 0.001 learning rate. This model achieves a validation accuracy of 0.8030 and a validation loss of 0.4876. The results also show that hyperparameter tuning using random search can improve the performance of the LSTM models by avoiding zero probabilities and finding the optimal values for the hyperparameters. Novelty: This study compares and analyzes the different preprocessing methods on text and the different configurations of the models to find the best model for clickbait classification on news headlines. The study also uses hyperparameter tuning to tune the model into the best model and finding the optimal values for the hyperparameters.

Many countries around the globe, including Tanzania, are investing in renewable energy, partly because of the need to increase electric power generation and meet the ever-increasing electricity demands, and partly because of the need to combat global climate change caused by carbon emissions from fossil fuel-based electricity production. Kishapu Solar Project, located in Kishapu District, Shinyanga Region, Tanzania, is one of the renewable energy projects with a planned generation of 150 MW of electric power once completed, using solar photovoltaic technology. Accurate forecasting of solar radiation is important in managing and maintaining solar energy plants. Inaccurate forecasting of solar radiation can lead to poor harvest of solar energy and poor management of solar energy plants, which directly impacts electricity contribution to the national electricity grid. Long Short-Term Memory (LSTM) and its variant, PeepHole LSTM, are examples of Deep Learning models that can be used to predict solar radiation. The purpose of PeepHole LSTM is to introduce “peephole” connections in the LSTM model, which allow LSTM gates to directly inspect memory cell state and hence have richer information to process. However, there is still a research gap on the impact of “peephole” connections on the prediction performance of the LSTM model, especially in the context of forecasting solar radiation in Kishapu District, where the Kishapu Solar Project is located. The purpose of this study is to fill this research gap by developing both LSTM and PeepHole LSTM models and comparing their performances in forecasting solar radiation in Kishapu District. The 14-year Kishapu District daily solar radiation (Downward Shortwave Radiation) dataset was first pre-processed by scaling (normalizing) it using Standard Scaler and then split into training set (the first 70%), validation set (the next 15%) and test set (the last 15%) before being utilized to train and test the developed LSTM and PeepHole LSTM models. Both LSTM and PeepHole LSTM models used the previous 5 days' solar radiation values as inputs to predict the output (next day (6th day) solar radiation value). The trained and better-performing model between the two was finally saved in .h5 format and integrated with a Gradio based Web Application to provide a user interface for officials in Kishapu District and at Kishapu Solar Project to predict daily solar radiation. The findings from performance evaluation revealed that the LSTM model outperformed the PeepHole LSTM model in predicting solar radiation by achieving test Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) scores of 26.22 W/m2 and 36.65 W/m2, respectively, compared to MAE and RMSE scores of 30.84 W/m2 and 41.88 W/m2, respectively, achieved by the PeepHole LSTM model, concluding that “peephole” connections reduce the performance of the LSTM model rather than increasing it when used to forecast solar radiation in Kishapu District. This study’s recommendation to Artificial Intelligence (AI) software developers and researchers is to use the LSTM rather than the PeepHole LSTM Deep Learning model when forecasting solar radiation in Kishapu District and in locations with similar climatic conditions

This study evaluated the forecasting accuracy of various models over 5-day and 10-day trading horizons to predict the prices of orange juice futures (OJ = F). The analysis included traditional models like Autoregressive Integrated Moving Average (ARIMA) and advanced neural network models such as Long Short-Term Memory (LSTM), Recurrent Neural Network (RNN), Backpropagation Neural Network (BPNN), Support Vector Regression (SVR), and Convolutional Long Short-Term Memory (ConvLSTM), incorporating factors like the Commodities Index and the S&P500 Index. We employed loss function metrics and various tests to assess model performance. The results indicated that for the 5-day horizon, the LSTM and ConvLSTM consistently outperformed the other models. LSTM achieved the lowest error rates and demonstrated superior capability in capturing temporal dependencies, especially in single-factor and S&P500 Index predictions. ConvLSTM also performed strongly, effectively modeling spatial and temporal data patterns. In the 10-day horizon, similar trends were observed. LSTM and ConvLSTM models had significantly lower errors and better alignment with actual values. The BPNN model performed well when all factors were included, and the SVR model maintained consistent accuracy, particularly for single-factor predictions. The Diebold–Mariano (DM) test indicated significant differences in forecasting accuracy, favoring advanced neural network models. In addition, incorporating multiple influencing factors further improved predictive performance, enhancing investment outcomes and reducing risk.

Wearable electroencephalography (EEG) devices for seizure detection accuracy and reliability are deep learning (DL) applications in the field of epilepsy diagnosis. In this study, we sought to increase the accuracy of seizure detection using advanced DL algorithms on the Children’s Hospital Boston - Massachusetts Institute of Technology (CHB-MIT) EEG database. First, a fully convolutional network (FCN) was trained and assessed using accuracy and recall/precision metrics, and the early stopping technique was used to avoid overfitting. To assess the performance, the FCN was evaluated in terms of various metrics, including accuracy, precision, recall, F1-score, and receiver operating characteristic (ROC)-area under the curve (AUC). In addition, two-dimensional (2D) convolutional neural networks (CNNs) and long short-term memory (LSTM) models were used to model the database, and their performance was thoroughly measured using different metrics, graphs, and confusion matrices. Using LSTM variants, such as standard LSTM, bidirectional LSTM, stacked LSTM, and LSTM attention mechanisms, hybrid convolutional LSTM (ConvLSTM) models were trained and compared. The comparison was conducted based on the training and validation accuracy and loss, as well as the graphs resulting from the precision–recall curves. Apart from DL approaches, EEG signal analysis using time–frequency techniques, such as wavelet transform and short-time Fourier transform, has also been investigated. These methods assisted in the analysis of the time–frequency features of EEG signals in combination with DL models. This study demonstrates that the performance of wearable EEG devices can be augmented using a combination of DL and seizure signal processing techniques. The FCN achieved an accuracy of 92%, with a recall for seizures of 33%, an F1-score of 0.03, and strong ROC-AUC results. The 2D CNN achieved 96% accuracy, a seizure recall of 70%, an F1-score of 0.12, and an ROC-AUC score of 78%. The baseline LSTM struggled with effectiveness at 53% accuracy with a seizure recall of 18%. In contrast, the LSTM model, which incorporated synthetic minority oversampling technique (SMOTE) balancing, was able to reach improvements of up to 89% accuracy, with a precision of 91%, a recall of 86%, an F1-score of 0.89, and a strong ROC curve performance. Among the models, the LSTM with SMOTE was the best performer, with 89% accuracy, 91% precision, 86% recall, and an F1-score of 0.89. These results provide evidence that applying techniques for data balancing in combination with certain DL network architectures significantly improves the detection of seizures using wearable EEG devices worn on the body. We believe that real-time monitoring and high-performance systems are feasible using optimized DL frameworks. The analysis of the performance of different models allows for the understanding of the possibilities of optimizing the architectures of DL algorithms for the modern diagnosis of epilepsy in real time. The source code used to carry out the experiments is publicly available at CHB-MIT EEG Dataset Python Scripts (https://www.kaggle.com/code/adnankust/adnaneeg1).

An accurate electrical Short-term Load Forecasting (STLF) is an eminent factor in the power generation, electrical load dispatching and energy planning for the power supply companies, specifically in developing countries. This paper proposes a novel temporal feature selection-based Long Short-term Memory (LSTM) model developed by the combination of standard Artificial Neural Network (ANN) layer and LSTM for electrical short term load forecasting. The LSTM model has excellent capability of predicting the stochastic nature of an hour ahead electrical loads. The standard ANN layer consisting 11 neurons is used as an input to LSTM cells. Such a combination of ANN layer with LSTM was never proposed before. The proposed model accommodates variations in weather as well as temporal inputs like humidity, holidays, and date-time features in the hourly load data of the power supply company situated in Johor, Malaysia. This paper gives the insights of hyper parameter tuning to capture the more generalized electrical load patterns in the dataset without compromising the time complexity of the proposed model. The proposed approach was compared with five existing approaches, namely: ANN, LSTM model 1, LSTM model 2, LSTM model 3 and Convolutional Neural Network-LSTM (CNN-LSTM) using hourly load dataset of Johor. The experimental results demonstrate that the proposed approach outperformed the existing approaches in terms of root mean square error, mean absolute percentage error and Diebold-Mariano statistical inference test within 95% confidence interval.

The effective integration of photovoltaic (PV) systems into the grid which facilitates better planning and resource allocation depends on accurate day-ahead solar power forecasts. This paper proposes a Long Short-Term Memory (LSTM) model for forecasting solar power generation one day in advance with the algorithm inspired by Ant Colony Optimization (ACO). A year's worth of real-time data for a 4.5 kW PV system is used to train and test the model with input characteristics including temperature, PV power production and solar irradiation. Over the course of three seasons—summer, winter, and monsoon—the performance of the ACO-optimized LSTM model is contrasted with that of a simple LSTM model and an LSTM model enhanced by particle swarm optimization (PSO). This encompasses metrics like R-squared (R²) values, Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). The PSO-optimized LSTM model (MAE of 0.31 kW, RMSE of 0.38 kW and R² of 0.88) and the baseline LSTM model (MAE of 0.35 kW, RMSE of 0.42 kW and R² of 0.85) couldn't match the ACO-optimized LSTM model's summertime MAE of 0.27 kW, RMSE of 0.33 kW and R² of 0.92. Similar trends also surfaced throughout the monsoon and winter seasons, with the ACO-optimized model consistently outperforming the others. In the winter, for instance, it obtained an R² of 0.93, an RMSE of 0.26 kW and an MAE of 0.22 kW. With regard to hyper parameter optimization and overall performance, these findings demonstrate the ACO-optimized LSTM model's superiority over the baseline and PSO-optimized LSTM models highlighting its improved prediction precision for all seasons. The effectiveness of ACO in bolstering solar power forecast models is confirmed by this study and provides a solid basis for deep learning model improvement in practical PV applications.

An effective compression algorithm for real-time transmission data using predictive coding with mixed models of LSTM and XGBoost

We investigate the potential of using Long Short-Term Memory (LSTM) neural networks for estimating streamflow in (sub)tropical catchments under contrasting hydroclimatic regimes (semi-arid and humid). We have used 176 Brazilian catchments with at least 30 years of streamflow data and LSTM models with 16 static catchment attributes as input features. We tested different LSTM model configurations to assess their sensitivity to varying input sequence lengths (lookbacks). The primary objective was to explore the hydrological insights offered by LSTM-based streamflow models and compare their performance with the traditional GR4J hydrological model. With this design, we aim to address two research questions: (i) Does the performance of LSTM models depend on catchments' hydroclimatic characteristics? (ii) How effective are LSTM-based models for streamflow simulation in tropical and subtropical catchments under semi-arid and humid conditions? We adopt two modeling approaches: (1) regionalized models trained on catchments within the same hydroclimatic regime and (2) a composite model trained on a heterogeneous sample combining both arid and humid catchments. The findings reveal distinct sensitivities of LSTM models to hydroclimatic conditions. LSTM models exhibit higher sensitivity to the length of input sequences (lookbacks) in humid catchments, with longer sequences yielding better performance. This is attributed to the dominant hydrological processes in humid regions, which are influenced by long-term memory effects such as soil moisture and groundwater storage. Conversely, this sensitivity is not observed in semi-arid catchments, where streamflow dynamics are primarily driven by short-term precipitation events and exhibit less dependence on long-term hydrological processes. Furthermore, the composite model, which combines semi-arid and humid catchments, demonstrates a decrease in performance for semi-arid catchments. This suggests that adding catchments with contrasting hydroclimatic characteristics introduces heterogeneity in the dataset, potentially reducing the model's ability to capture the specific dynamics of semi-arid catchments. Overall, the regionalized LSTM models outperformed the GR4J model in both semi-arid and humid regimes, particularly in humid catchments. Approximately 87% of humid catchments and 50% of semi-arid catchments achieved Kling-Gupta Efficiency (KGE) values above 0.60 during the testing phase of the regionalized LSTM models. These results highlight the potential of LSTM networks for streamflow regionalization, especially in humid regions where long-term hydrological memory plays a critical role. The study underscores the strengths and limitations of LSTM models in tropical and subtropical catchments with contrasting hydroclimatic regimes. The findings suggest that LSTM models could serve as valuable tools for regional hydrological applications, aiding local and regional decision-making processes. Additionally, the results emphasize the importance of tailoring LSTM model configurations to the specific hydrological characteristics of catchments, particularly the choice of input sequence length, to maximize model performance.

Wave runup prediction is necessary for offshore structure designs and early warnings. Data-driven methods based on machine learning have inspired reduced-order solutions for wave–structure interaction problems. This study provides the quantification of deep learning algorithms’ potential for wave runup prediction. Two prominent deep learning models were utilized to predict the wave runups along the fore column of semisubmersible under head seas. The long short-term memory (LSTM) and the temporal convolutional networks (TCNs) were comprehensively compared based on the datasets from a model test carried out in the deep ocean basin. The LSTM and TCN model structures were optimized to compare prediction accuracy and computational complexity reasonably. The results reveal that (1) both developed TCN and LSTM models had a satisfied prediction accuracy of over 90%. Their predictions were extended to 10 s into the future with accuracies over 80% and 45%, respectively. (2) With the noise-extended datasets, the LSTM model was robust to noises, while the TCN model showed better prediction performance on the extreme wave runup events. (3) The incident wave and dominant rotation provided the major information for wave runup prediction. TCN and LSTM models’ prediction accuracies were 91.5% and 89.3% based on the simplified input tensors composed of incident wave and pitch. The comparison showed the great potential of the TCN model to predict the nonlinear wave runup with less time and memory costs. The input tensors’ design and optimization based on physical understanding also play a significant role in the prediction performance.

The long short-term memory (LSTM) model is widely used in multiple areas, mainly for speech recognition, natural language processing and activity recognition. In the last few years, we started to see many variants of LSTM for recurrent neural networks since its inception in 1997. However, there weren’t many studies that have addressed the LSTM’s gating mechanism. In this paper, we propose a novel LSTM framework where we modify the architecture of the LSTM unit by adding a new layer that we call the “outlier gate”. The latter controls the flow of information that goes into the LSTM cell. This added signal allows us to avoid both the carry-over effect that the outliers have on the forecasted point and a bias in the estimates of our LSTM model – caused by unusual or non-repetitive events. The proposed architecture led us to an end-to-end trainable model that we applied in this paper to a financial time-series forecasting problem. Our results demonstrate that the new proposed LSTM architecture achieves better performance than the state-of-the-art original LSTM model.

Traffic safety has been thought of as a basic feature of transportation, recent developments in civil aviation have emphasized the need for risk identification and safety prediction. This study aims to increase en-route flight safety through the development of prediction models for flight conflicts. Firstly, flight conflicts time series and traffic parameters are extracted from historical ADS-B data. In the second step, a Long Short-Term Memory (LSTM) model is trained to make a one-step-ahead prediction on the flight conflict time series. The results show that the LSTM model has the greatest prediction effect (MAE 0.3901) with comparison to other models. Based on that, we add traffic parameters (volume, density, velocity) into the LSTM model as new input variables and issue a comprehensive analysis of the relative predictive power of traffic parameters. The accuracy of prediction model is validated with a mean error of less than 3%. Based on the improvements of model performance brought by traffic parameters, LSTM models with a single traffic parameter are proposed for further discussion. The results illustrate that volume is the most important factor in promoting prediction accuracy and density has an advantage of improvement in the aspect of model stability.

The ongoing Russia–Ukraine conflict has exacerbated the global crisis of natural gas supply, particularly in Europe. During the winter season, major importers of liquefied natural gas (LNG), such as South Korea and Japan, were directly affected by fluctuating spot LNG prices. This study aimed to use machine learning (ML) to predict the Japan Korea Marker (JKM), a spot LNG price index, to reduce price fluctuation risks for LNG importers such as the Korean Gas Corporation (KOGAS). Hence, price prediction models were developed based on long short-term memory (LSTM), artificial neural network (ANN), and support vector machine (SVM) algorithms, which were used for time series data prediction. Eighty-seven variables were collected for JKM prediction, of which eight were selected for modeling. Four scenarios (scenarios A, B, C, and D) were devised and tested to analyze the effect of each variable on the performance of the models. Among the eight variables, JKM, national balancing point (NBP), and Brent price indexes demonstrated the largest effects on the performance of the ML models. In contrast, the variable of LNG import volume in China had the least effect. The LSTM model showed a mean absolute error (MAE) of 0.195, making it the best-performing algorithm. However, the LSTM model demonstrated a decreased in performance of at least 57% during the COVID-19 period, which raises concerns regarding the reliability of the test results obtained during that time. The study compared the ML models’ prediction performances with those of the traditional statistical model, autoregressive integrated moving averages (ARIMA), to verify their effectiveness. The comparison results showed that the LSTM model’s performance deviated by an MAE of 15–22%, which can be attributed to the constraints of the small dataset size and conceptual structural differences between the ML and ARIMA models. However, if a sufficiently large dataset can be secured for training, the ML model is expected to perform better than the ARIMA. Additionally, separate tests were conducted to predict the trends of JKM fluctuations and comprehensively validate the practicality of the ML models. Based on the test results, LSTM model, identified as the optimal ML algorithm, achieved a performance of 53% during the regular period and 57% d during the abnormal period (i.e., COVID-19). Subject matter experts agreed that the performance of the ML models could be improved through additional studies, ultimately reducing the risk of price fluctuations when purchasing spot LNG.

Short-term traffic speed prediction is a promising research topic in intelligent transportation systems (ITSs), which also plays an important role in the real-time decision-making of traffic control and guidance systems. However, the urban traffic speed has strong temporal, spatial correlation and the characteristic of complex nonlinearity and randomness, which makes it challenging to accurately and efficiently forecast short-term traffic speeds. We investigate the relevant literature and found that although most methods can achieve good prediction performance with the complete sample data, when there is a certain missing rate in the database, it is difficult to maintain accuracy with these methods. Recent studies have shown that deep learning methods, especially long short-term memory (LSTM) models, have good results in short-term traffic flow prediction. Furthermore, the attention mechanism can properly assign weights to distinguish the importance of traffic time sequences, thereby further improving the computational efficiency of the prediction model. Therefore, we propose a framework for short-term traffic speed prediction, including data preprocessing module and short-term traffic prediction module. In the data preprocessing module, the missing traffic data are repaired to provide a complete dataset for subsequent prediction. In the prediction module, a combined deep learning method that is an attention-based LSTM (ATT-LSTM) model for predicting short-term traffic speed on urban roads is proposed. The proposed framework was applied to the urban road network in Nanshan District, Shenzhen, Guangdong Province, China, with a 30-day traffic speed dataset (floating car data) used as the experimental sample. Results show that the proposed method outperforms other deep learning algorithms (such as recurrent neural network (RNN) and convolutional neural network (CNN)) in terms of both calculating efficiency and prediction accuracy. The attention mechanism can significantly reduce the error of the LSTM model (up to 12.4%) and improves the prediction performance.

Compressive strength is an important mechanical property of high-strength concrete (HSC), but testing methods are usually uneconomical, time-consuming, and labor-intensive. To this end, in this paper, a long short-term memory (LSTM) model was proposed to predict the HSC compressive strength using 324 data sets with five input independent variables, namely water, cement, fine aggregate, coarse aggregate, and superplasticizer. The prediction results were compared with those of the conventional support vector regression (SVR) model using four metrics, root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), and correlation coefficient (R2). The results showed that the prediction accuracy and reliability of LSTM were higher with R2 = 0.997, RMSE = 0.508, MAE = 0.08, and MAPE = 0.653 compared to the evaluation metrics R2 = 0.973, RMSE = 1.595, MAE = 0.312, MAPE = 2.469 of the SVR model. The LSTM model is recommended for the pre-estimation of HSC compressive strength under a given mix ratio before the laboratory compression test. Additionally, the Shapley additive explanations (SHAP)-based approach was performed to analyze the relative importance and contribution of the input variables to the output compressive strength.

Purpose: This study aims to predict the closing price of the EUR/JPY currency pair in the forex market using recurrent neural network (RNN) architectures, namely Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), with the incorporation of Bidirectional layers. Methods: The dataset comprises hourly price data obtained from Yahoo Finance and pre-processed accordingly. The data is divided into training and testing sets, and time series sequences are constructed for input into the models. The RNN, LSTM, and GRU models are trained using the Adam optimization algorithm with the mean squared error (MSE) loss metric. Results: Results indicate that the LSTM model, particularly when coupled with Bidirectional layers, exhibits superior predictive performance compared to the other models, as evidenced by lower MSE values. Conclusions: Therefore, the LSTM model with Bidirectional layers is the most effective in predicting the EUR/JPY currency pair's closing price in the forex market. These findings offer valuable insights for practitioners and researchers involved in financial market prediction and neural network modelling