Interpretable deep learning model for building energy consumption prediction based on attention mechanism

An essential step in surface wave exploration is the inversion of dispersion curves. By inverting dispersion curves, we can effectively establish the shear-wave velocity model and obtain reliable subsurface stratigraphic information. The inversion of dispersion curves is an inversion problem with multiple parameters and multiple poles, and obtaining a high precision solution is difficult. Among the methods of inversion of dispersion curves, local search methods are prone to fall into local extremes, and global search methods such as particle swarm optimization (PSO) and genetic algorithm (GA) present the disadvantages of slow convergence speed and low precision. Deep learning models with strong nonlinear mapping capability can effectively solve nonlinear problems. Therefore, we propose a method called PSO-optimized long short-term memory (LSTM) network (PSO-LSTM) to invert the dispersion curves in order to improve the effect of inversion of dispersion curves. The method is based on the LSTM network, and PSO is used to optimize the LSTM network structure and other parameters that need to be given manually to improve the prediction of the network. Two theoretical geological models are used in the paper: Model A and Model B to test the PSO-LSTM. The tests include the noisy data test and noise-free data test. Model A was tested without noise, and Model B was tested with noise. In addition, PSO and LSTM were tested on model A to compare the performance of PSO-LSTM. In Model A, the maximum relative errors of PSO and LSTM are 20.76% and 5.85%, respectively, and the maximum standard deviations of PSO and LSTM are 57.37 and 1.97, respectively. For PSO-LSTM, the maximum relative errors of Model A and Model B in the inverse results are 2.05% and 2.09%, and the maximum standard deviations of Model A and Model B in the inverse results are 1.23 and 3.87, respectively. The test results of Model A show that the inversion performance of PSO-LSTM is better than those of LSTM and PSO, and the performance of the network can be improved after PSO is used to optimize the network parameters. The inverse results from Model B show that the PSO-LSTM is robust and can invert the dispersion curves well even after adding noise to the model. Finally, the PSO-LSTM is used to invert the actual data from Wyoming, USA, which demonstrates that the PSO-LSTM can be used for the quantitative interpretation of Rayleigh wave dispersion curves.

This study evaluates the performance of traditional statistical methods, specifically the Seasonal Autoregressive Integrated Moving Average (SARIMA), in comparison to advanced deep learning models—Long Short-Term Memory (LSTM), Bidirectional LSTM (BiLSTM), and Convolutional Neural Network (CNN)—for wind speed forecasting across various regions in Saudi Arabia: Al-Jouf, Abha, Al-Ahsa, and Al-Dawadami. The historical wind speed dataset covering the year 2018 was utilized, with data preprocessing conducted using the Exploratory Data Analysis (EDA) method to ensure consistency and quality. The models were assessed based on three primary error metrics: Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Square Error (RMSE). Results show that CNN consistently outperforms SARIMA and the other deep learning models, particularly when forecasting stable wind speed patterns. LSTM demonstrates an ability to handle fluctuating wind speeds effectively, while BiLSTM offers advantages in capturing complex bidirectional temporal dependencies. On the other hand, SARIMA generally exhibits lower performance compared to deep learning approaches. The superior performance of CNN is likely attributed to its strength in local feature extraction and handling spatial patterns effectively, which are beneficial for short-term forecasting. These findings provide valuable insights into model selection for wind energy forecasting and can contribute to optimizing renewable energy integration and planning strategies. Future work may explore hybrid model approaches to further enhance forecasting accuracy.

Aiming at the problem that the exiting human skeleton-based action recognition methods cannot fully extract the relevant information before and after the action, resulting in low utilization efficiency of skeleton points, we propose a two-layer LSTM (long short term memory) network with attention mechanism. The network has two layers, the first LSTM network is used for skeleton coding and initialization of system storage units and the second LSTM network integrates attention mechanism to further process the data of the first layer network. An algorithm is designed to assign different weights to skeleton points according to the importance of human body, which greatly increases the recognition accuracy. Action classification is accomplished by multiple support vector machines. Through training and testing, the average recognition rate of 98.5% is achieved on KTH dataset. The experimental result shows that the proposed method is effective in human behavior recognition.

Deep learning for predicting respiratory rate from biosignals

Rock moisture, which is considered as a hidden component of the terrestrial hydrological cycle, has received little attention. In this study, the frequency-domain reflectometry (FDR) is used to obtain fluctuating rock water content in a sandstone cave of the Yungang Grottoes, China. We identified two major cycles of rock moisture addition and depletion, one in the summer and the other in the winter. By using the LSTM (Long Short-Term Memory) network and the SHAP (SHapley Additive exPlanations) method, relative humidity, air temperature and wall temperature are found to have contributions to rock moisture in the summer. By using vapor concentration and the difference between dew point temperature and wall temperature as two input variables of the LSTM network, the predicted rock water content has a very good agreement with the measured rock water content, with the Nash–Sutcliffe efficiency coefficient (NSE) being as high as 0.978. Because the two new input variables are factors directly controlling vapor condensation, they provide informative priors to the deep learning model and improved prediction performance. After identifying the causal factors of rock water content fluctuations, we also identified the mechanism controlling the multi diurnal vapor condensation. The increased vapor concentration accompanying a precipitation event leads to transport of water vapor into rock pores, which is subsequently adsorbed onto the surface of rock pores and then condensed into liquid water. With the aid of the deep learning model, this study increases understanding of sources of water in caves, which would contribute to future strategies of alleviating weathering in caves.

Scene text recognition (STR) technology has a rapid development with the rise of deep learning. Recently, the encoder-decoder framework based on attention mechanism is widely used in STR for better recognition. However, the commonly used Long Short Term Memory (LSTM) network in the framework tends to ignore certain position or visual information. To address this problem, we propose a Position Information Enhanced Encoder-Decoder (PIEED) framework for scene text recognition, in which an addition position information enhancement (PIE) module is proposed to compensate the shortage of the LSTM network. Our module tends to retain more position information in the feature sequence, as well as the context information extracted by the LSTM network, which is helpful to improve the recognition accuracy of the text without context. Besides that, our fusion decoder can make full use of the output of the proposed module and the LSTM network, so as to independently learn and preserve useful features, which is helpful to improve the recognition accuracy while not increase the number of arguments. Our overall framework can be trained end-to-end only using images and ground truth. Extensive experiments on several benchmark datasets demonstrate that our proposed framework surpass state-of-the-art ones on both regular and irregular text recognition.

Time-history responses of the bridge induced by the moving vehicle provide crucial information for bridge design, operation, maintenance, etc. As inspired by this, this work attempts to provide a new paradigm for vehicle–bridge interaction (VBI) by highlighting the comparison of different deep learning algorithms applied to the prediction of time-history responses of the bridge under vehicular loads. Particularly, three deep learning architectures with few and measurable input features developed by using fully-connected feedforward neural network, long short-term memory (LSTM) network, and convolutional neural network (CNN) are proposed on the basis of the governing equation of bridge vibrations. Three VBI systems with various vehicle models are developed and further validated to produce reliable training data. To examine the accuracy of the predictive models, two advanced metrics are exploited for time-history estimate. Moreover, the proposed deep learning models are comprehensively investigated through a parametric study on the influential factors associated with the VBI system and network architecture. The results show that deep feedforward neural network (DFNN), LSTM network, and CNN can be applied in VBI analysis to estimate the bridge time-history response. The three neural networks have comparable prediction accuracies. When considering the irregularity excitation, CNN is found to be the most efficient predictive model, while DFNN needs the least training time under perfect bridge surface condition.

<p>Flooding is one of the most financially devastating natural hazards in the world. Studying storage-discharge relations can have the potential to improve existing flood forecasting systems, which are based on rainfall-runoff models. This presentation will assess the non-linear relation between daily water storage (ΔS) and discharge (Q) simulated by physical-based hydrological models at the Rum River Watershed, a HUC8 watershed in Minnesota, between 1995-2015, by training Long Short-Term Memory (LSTM) networks and other machine learning (ML) algorithms. Currently, linear regression models do not adequately represent the relationship between the simulated total ΔS and total Q at the HUC-8 watershed (R<sup>2</sup> = 0.3667). Since ML algorithms have been used for predicting the outputs that represent arbitrary non-linear functions between predictors and predictands, they will be used for improving the accuracy of the non-linear relation of the storage-discharge dynamics. This research will mainly use LSTM networks, the time-series deep learning neural network that has already been used for predicting rainfall-runoff relations. The LSTM network will be trained to evaluate the storage-discharge relationship by comparing two sets of non-linear hydrological variables simulated by the semi-distributed Hydrological Simulated Program-Fortran (HSPF): the relationship between the simulated discharges and input hydrological variables at selected HUC-8 watersheds, including air temperatures, cloud covers, dew points, potential evapotranspiration, precipitations, solar radiations, wind speeds, and total water storage, and the dynamics between simulated discharge and input variables that do not include the total water storage. The result of this research will lay the foundation for assessing the accuracy of downscaled storage-discharge dynamics by applying similar methods to evaluate the storage-discharge dynamics at small-scaled, HUC-12 watersheds. Furthermore, its results have the potentials for us to evaluate whether downscaling of storage-discharge dynamics at the HUC-12 watershed can improve the accuracy of predicting discharge by comparing the result from the HUC-8 and the HUC-12 watersheds.</p>

This study aimed to improve apple slices’ color and drying kinetics by optimizing the hot-air drying process, utilizing machine and deep learning models. Different steam blanching times (30, 60, 90, and 120 s), drying temperatures (50, 55, 60, 65, and 70 °C), and humidity control methods (full humidity removal or temperature–humidity control) were examined. These factors significantly affected the quality of apple slices. 60 s blanching, 60 °C temperature, and full dehumidification represented the optimal drying conditions for apple slices’ dehydration, achieving better drying kinetics and the best color quality. However, the fastest drying process (40 min) was obtained at a 60 °C drying temperature combined with complete dehumidification after 90 s blanching. Furthermore, machine and deep learning models, including backpropagation (BP), convolutional neural network–long short-term memory (CNN-LSTM), temporal convolutional network (TCN), and long short-term memory (LSTM) networks, effectively predicted the moisture content and color variation in apple slices. Among these, LSTM networks demonstrated exceptional predictive performance with an R2 value exceeding 0.98, indicating superior accuracy. This study provides a scientific foundation for optimizing the drying process of apple slices and illustrates the potential application of deep learning in the agricultural processing and engineering fields.

Utilization of the Long Short-Term Memory network for predicting streamflow in ungauged basins in Korea

Abstract. Contamination of surface waters with microbiological pollutants is a major concern to public health. Although long-term and high-frequency Escherichia coli (E. coli) monitoring can help prevent diseases from fecal pathogenic microorganisms, such monitoring is time-consuming and expensive. Process-driven models are an alternative means for estimating concentrations of fecal pathogens. However, process-based modeling still has limitations in improving the model accuracy because of the complexity of relationships among hydrological and environmental variables. With the rise of data availability and computation power, the use of data-driven models is increasing. In this study, we simulated fate and transport of E. coli in a 0.6 km2 tropical headwater catchment located in the Lao People's Democratic Republic (Lao PDR) using a deep-learning model and a process-based model. The deep learning model was built using the long short-term memory (LSTM) methodology, whereas the process-based model was constructed using the Hydrological Simulation Program–FORTRAN (HSPF). First, we calibrated both models for surface as well as for subsurface flow. Then, we simulated the E. coli transport with 6 min time steps with both the HSPF and LSTM models. The LSTM provided accurate results for surface and subsurface flow with 0.51 and 0.64 of the Nash–Sutcliffe efficiency (NSE) values, respectively. In contrast, the NSE values yielded by the HSPF were −0.7 and 0.59 for surface and subsurface flow. The simulated E. coli concentrations from LSTM provided the NSE of 0.35, whereas the HSPF gave an unacceptable performance with an NSE value of −3.01 due to the limitations of HSPF in capturing the dynamics of E. coli with land-use change. The simulated E. coli concentration showed the rise and drop patterns corresponding to annual changes in land use. This study showcases the application of deep-learning-based models as an efficient alternative to process-based models for E. coli fate and transport simulation at the catchment scale.

Prediction of heating energy consumption with operation pattern variables for non-residential buildings using LSTM networks

Adaptive Cruise Control (ACC) is an important part of automotive autonomy. To have a robust Adaptive Cruise Control system, a well-defined sensor fusion system is required. This paper focuses on feature-level sensor fusion system where three fusion techniques are explored: independent Kalman filtering, interacting multiple model (IMM) filtering, and a novel hybrid technique using a combination of both an IMM Filter and deep neural network (DNN). Kalman Filtering is a popular approach for automotive sensor fusion, but heavily relies on the given kinematic models to make state estimations. We explore the IMM filter to capture multi-model motion as well as a DNN approach to build a data-driven model of the system. Our IMM filter is built with several Kalman Filters featuring different kinematic models. The proposed DNN used is a Long Short-Term Memory (LSTM) network trained on radar and camera data to forecast a vehicle's state. To overcome the weaknesses in both IMM filtering and LSTM networks, we propose a hybrid technique of consisting of both the IMM filter and the LSTM network. The proposed hybrid system uses a single filter for each sensor. The filtered sensor data is synchronized and used as the input for a trained LSTM network. Our LSTM network was trained on over 100 simulated highway driving scenarios that might occur in an ACC application. Simulations and code were developed using MATLAB and the Autonomous Driving Toolbox. Our hybrid IMM-LSTM system outperformed the independent Kalman Filtering approach in longitudinal and lateral tracking accuracy (RMSE) by 23 percent and had promising results compared to the IMM filter where the tracking error was decreased by nearly 50 percent in certain driving scenarios.

<p dir="ltr">Recurrent Neural Networks (RNNs) are a major class of Artificial Neural Networks (ANNs). Suitable network architecture is vital for RNNs to achieve high performance. However, designing the architecture of deep RNNs (DRNNs) is a complicated and time-consuming process. Each layer of a DRNN contains numerous hyper-parameters. It is challenging to optimize the hyper-parameters of all layers jointly to achieve the best possible learning performance. Trial-and-error methods for designing DRNNs have been proven to be laborious and highly costly in practice. Therefore, an efficient automatic architecture search technique is needed to design DRNNs.</p><p dir="ltr">Genetic Algorithm (GA) is a popular evolutionary computation based approach. Existing research works already explored the use of GAs in designing ANNs. But, most of the existing approaches focus on the design of fixed-depth ANNs. However, it is not easy to fix the depth of DRNNs in advance, since different problems need DRNNs of different depths. Hence, the potential of GA has to be further explored to evolve the architecture of DRNNs of varying depths.</p><p dir="ltr">The primary goal of this thesis is to develop advanced GA approaches for designing Long Short Term Memory (LSTM) based DRNN architectures of varying depths effectively and efficiently.</p><p dir="ltr">Firstly, this thesis proposes a GA-based algorithm called Two-Stage Surrogate-Assisted GA (TS-SA-GA) with a progressive incremental strategy to design the architecture of LSTM networks of varying depths. The progressive approach can effectively extend well-designed shallow networks to high performing deep networks. Moreover, the new GA-based algorithm adopts newly designed knowledge-driven crossover and mutation operators to identify and repair LSTM network designs affected by inappropriate use of activation functions, thereby significantly reducing the chances for GA to evolve hard-to-train LSTM network architectures. Furthermore, this thesis proposes a two-stage surrogate method to predict the trainability and fitness of LSTM networks. This improves the efficiency and effectiveness of the GA-based algorithm to evolve LSTM networks.</p><p dir="ltr">Secondly, this thesis proposes a new algorithm called Evolutionary Design of LSTM Ensembles (ED-LSTM-Ensemble) based on multi-objective optimization techniques to evolve LSTM networks and ensembles simultaneously to directly utilize the ensemble performance to drive the evolution of base LSTM networks. Additionally, this thesis proposes a connection weight inheritance strategy to evolve LSTM networks and ensembles efficiently and effectively.</p><p dir="ltr">Thirdly, this thesis proposes a new algorithm called Skip Connections through Evolutionary and Differential Architecture search (SCEDA) to design LSTM networks together with appropriate skip connections automatically. Designing LSTM networks together with skip connections is highly challenging. This thesis hybridize gradient-based DAS and evolutionary architecture search to optimize the architecture of LSTM networks with suitable skip connections in a single evolutionary process to improve the efficiency and effectiveness of the search process.</p><p dir="ltr">Finally, this thesis conducts an in-depth empirical analysis of the impact of activation functions on the trainability of LSTM networks based on the concept of Edge Of Chaos. The analysis reveals the strong interrelation between the activation functions across multiple layers of an LSTM network and the trainability of the network. The analysis also highlights the importance of controlling activation functions to avoid generating untrainable LSTM networks. On the basis of the analysis, this thesis proposes a machine learning model to guide the use of activation functions in LSTM networks.</p>

Modern decision-making in fixed income asset management benefits from intelligent systems, which involve the use of state-of-the-art machine learning models and appropriate methodologies. We conduct the first study of bond yield forecasting using long short-term memory (LSTM) networks, validating its potential and identifying its memory advantage. Specifically, we model the 10-year bond yield using univariate LSTMs with three input sequences and five forecasting horizons. We compare those with multilayer perceptrons (MLP), univariate and with the most relevant features. To demystify the notion of black box associated with LSTMs, we conduct the first internal study of the model. To this end, we calculate the LSTM signals through time, at selected locations in the memory cell, using sequence-to-sequence architectures, uni and multivariate. We then proceed to explain the states’ signals using exogenous information, for what we develop the LSTM-LagLasso methodology. The results show that the univariate LSTM model with additional memory is capable of achieving similar results as the multivariate MLP using macroeconomic and market information. Furthermore, shorter forecasting horizons require smaller input sequences and vice-versa. The most remarkable property found consistently in the LSTM signals, is the activation/deactivation of units through time, and the specialisation of units by yield range or feature. Those signals are complex but can be explained by exogenous variables. Additionally, some of the relevant features identified via LSTM-LagLasso are not commonly used in forecasting models. In conclusion, our work validates the potential of LSTMs and methodologies for bonds, providing additional tools for financial practitioners.