Adaptive state estimation using dynamic recurrent neural networks

The adoption of wind energy has grown significantly in recent years. New, cost-effective technologies have been developed, led by customer awareness of green technologies and a legal framework proposed at the European Union level. The stochastic nature of wind speed is transferred to wind turbine output, making wind energy difficult to predict. The main scope of predicting wind energy production is to be proactive in balancing and reserving energy to meet demand. When the prediction identifies a potential gap between supply and demand, additional energy from other sources must be generated and supplied. Creating a synergy of physical devices through advanced sensing capabilities, software, storage and analytics capabilities, the Industrial Internet of Things is enabling the effective transition to wind energy through automation by removing many of the disadvantages in a way that has recently become accessible. This research focuses on the data analytics, proposing a fast univariate network-based approach for wind energy prediction, using Feed Forward Neural Networks, Recurrent Neural Networks, Long-Short Term Memory, Gated Recurrent Unit, and Convolutional Neural Networks. Moreover, by introducing the theoretical fundamentals, the implementation method and the hyperparameters of the final models, this article becomes unique in the context of wind energy. At the time of this study, no prior research studies have presented a direct comparison between feedforward, recurrent, and convolutional neural networks ? these being the most important in the field of supervised learning.

Research in the area of artificial neural networks has advanced at a rapid pace in recent times. The artificial neural network possesses a good ability to learn adaptively. The decision process in an artificial neural network is based on certain nonlinear operations. Such nonlinearities are useful: i) in improving the convergence speed (of the algorithm); ii) to provide more general nonlinear mapping between input-output signals; and iii) to reduce the effect of outliers in the measurements. One of the most successful artificial neural networks is the so-called feed forward neural network. In this chapter first the description of the feed forward neural network and its training algorithms is given. Next, parameter estimation using this approach is discussed. The presentation of training algorithms is such that it facilitates MATLAB implementation. Subsequently, recurrent neural networks are described. Several schemes based on recurrent neural networks are presented for parameter estimation of dynamical systems. Subsequently, the genetic algorithm is described and its application to parameter estimation considered.

Data science highlights fields of study and research such as time series, which, although widely explored in the past, gain new perspectives in the context of this discipline. This chapter presents two approaches to time series forecasting, long short-term memory (LSTM), a special kind of recurrent neural network (RNN), and Prophet, an open-source library developed by Facebook for time series forecasting. With a focus on developing forecasting processes by data mining or machine learning experts, LSTM uses gating mechanisms to deal with long-term dependencies, reducing the short-term memory effect inherent to the traditional RNN. On the other hand, Prophet encapsulates statistical and computational complexity to allow broad use of time series forecasting, prioritizing the expert's business knowledge through exploration and experimentation. Both approaches were applied to a retail time series. This case study comprises daily and half-hourly forecasts, and the performance of both methods was measured using the standard metrics.

The aim of this paper is to discuss the methodology of Fault Detection and Diagnosis (FDD) in dynamic systems. Fault diagnosis is characterized as a control system that controls the ability to adapt system component faults automatically. Fault detection is an implementation of using the error signals, where when error signal is zero or approximately zero. A coupled water tank system was used as a study case model for implementing and testing the proposed methodology. The developed system should generate a set of signals to notify the process operator about the faults that are occurring, enabling changes in control strategy or control parameters. Due to the damage risks involved with sensors, actuators and structural faults of the real plant, the data set of the faults are computationally generated and the results will be collected from numerical simulations of the process model. A Recurrent Neural Networks (RNN) is employed in this paper for modeling the used system. This paper shows how to determine the structure and how to estimate the results using the gradient-based algorithm which it is allowed to obtain a Neural Network with relatively small modeling uncertainly. It describes how to increase the stability of dynamic systems using FDD based on RNN. Faults in the study case are faults in the sensors, faults in the actuators and the structural faults. The faults which are presented in the controlled system are reduced to 98% from the default value.

In this paper, we investigated interactive learning between human subjects and robot experimentally, and its essential characteristics are examined using the dynamical systems approach. Our research concentrated on the navigation system of a specially developed humanoid robot called Robovie and seven human subjects whose eyes were covered, making them dependent on the robot for directions. We compared the usual feed-forward neural network (FFNN) without recursive connections and the recurrent neural network (RNN). Although the performances obtained with both the RNN and the FFNN improved in the early stages of learning, as the subject changed the operation by learning on its own, all performances gradually became unstable and failed. Results of a questionnaire given to the subjects confirmed that the FFNN gives better mental impressions, especially from the aspect of operability. When the robot used a consolidation-learning algorithm using the rehearsal outputs of the RNN, the performance improved even when interactive learning continued for a long time. The questionnaire results then also confirmed that the subject's mental impressions of the RNN improved significantly. The dynamical systems analysis of RNNs supports these differences.

Besides the feedforward neural networks, there are the recurrent networks, where the impulses can be transmitted in both directions due to some reaction connections in these networks. Recurrent Neural Networks (RNNs) are linear or nonlinear dynamic systems. The dynamic behavior presented by the recurrent neural networks can be described both in continuous time, by differential equations and at discrete times by the recurrence relations (difference equations). The distinction between recurrent (or dynamic) neural networks and static neural networks is due to recurrent connections both between the layers of neurons of these networks and within the same layer, too. The aim of this chapter is to describe a Recurrent Fuzzy Neural Network (RFNN) model, whose learning algorithm is based on the Improved Particle Swarm Optimization (IPSO) method.

A computer assisted fault diagnosis system (CAFD) is considered which allows the early detection and localization of process faults during normal operation or on request. It is based on an on-line engineering expert system and consists of an analytical problem solution, a process knowledge base, a knowledge acquisition component and an inference mechanism. The analytic problem solution uses a process parameter estimation, and the detection of process coefficient changes, which are symptoms of process faults. The process knowledge base is comprised of analytical knowledge in the form of process models and heuristic knowledge in the form of fault trees and fault statistics. In the phase of knowledge acquisition the process specific knowledge like theoretical process models, the normal behavior and fault trees is compiled. The inference mechanism performs the fault diagnosis, based on the observed symptoms, the fault trees, fault probabilities and the process history. This is described in Part I. In Part II, case study experiments with a d.c. motor, centrifugal pump, a heat exchanger and an industrial robot show practical results of the model based fault diagnosis.

The estimation of 3D orientation based on nine-axis inertial measurement unit (IMU) is an essential technology in various applications, from unmanned aerial vehicle to human motion tracking. Various sensor fusion filter algorithms such as Kalman filter (KF) or complementary filter (CF) have been proposed for accurate 3D orientation estimation. However, the degradation of estimation performance due to disturbance components such as magnetic distortion and external acceleration is still a critical issue. An alternative approach for the orientation estimation task is to train a neural network end-to-end with a variety of massive experimental datasets consisting of the raw IMU signals and the ground truth orientations. This paper proposes a recurrent neural network (RNN) for robust IMU-based 3D orientation estimation. Overall, this paper is an extension of the previous work by Weber et al., where the RNN model was used to estimate the quaternion, but only the attitude estimation performance was investigated without considering the heading estimation. The proposed RNN receives a nine-dimensional sensor signal as an input and outputs a unit quaternion representing 3D orientation, and it can robustly estimate the orientation across various motion characteristics and magnetic environments without needing an additional filter algorithm. Verification results showed that the proposed method outperformed the conventional CF and KF algorithms. In particular, in magnetically disturbed conditions, the averaged root mean square error of the proposed RNN approach was reduced by 41.4% and 60.3%, respectively, when compared to the CF and the KF algorithm.

Effects of data smoothing and recurrent neural network (RNN) algorithms for real-time forecasting of tunnel boring machine (TBM) performance

Marker-based Optical Motion Capture (OMC) systems and associated musculoskeletal (MSK) modelling predictions offer non-invasively obtainable insights into muscle and joint loading at an in vivo level, aiding clinical decision-making. However, an OMC system is lab-based, expensive, and requires a line of sight. Inertial Motion Capture (IMC) techniques are widely-used alternatives, which are portable, user-friendly, and relatively low-cost, although with lesser accuracy. Irrespective of the choice of motion capture technique, one typically uses an MSK model to obtain the kinematic and kinetic outputs, which is a computationally expensive tool increasingly well approximated by machine learning (ML) methods. Here, an ML approach is presented that maps experimentally recorded IMC input data to the human upper-extremity MSK model outputs computed from ('gold standard') OMC input data. Essentially, this proof-of-concept study aims to predict higher-quality MSK outputs from the much easier-to-obtain IMC data. We use OMC and IMC data simultaneously collected for the same subjects to train different ML architectures that predict OMC-driven MSK outputs from IMC measurements. In particular, we employed various neural network (NN) architectures, such as Feed-Forward Neural Networks (FFNNs) and Recurrent Neural Networks (RNNs) (vanilla, Long Short-Term Memory, and Gated Recurrent Unit) and a comprehensive search for the best-fit model in the hyperparameters space in both subject-exposed (SE) as well as subject-naive (SN) settings. We observed a comparable performance for both FFNN and RNN models, which have a high degree of agreement (ravg,SE,FFNN=0.90±0.19, ravg,SE,RNN=0.89±0.17, ravg,SN,FFNN=0.84±0.23, and ravg,SN,RNN=0.78±0.23) with the desired OMC-driven MSK estimates for held-out test data. The findings demonstrate that mapping IMC inputs to OMC-driven MSK outputs using ML models could be instrumental in transitioning MSK modelling from 'lab to field'.

Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs) have gained significant popularity in time series forecasting across diverse domains including healthcare, astronomy, and engineering. However, the inherent variability in model performance due to random weight initialization raises questions about the reliability and consistency of these architectures for time series analysis. This study addresses this concern by conducting a comprehensive benchmark evaluation of nine neural network architectures: vanilla RNN, LSTM, GRU, and six hybrid configurations (RNN-LSTM, RNN-GRU, LSTM-RNN, GRU-RNN, LSTM-GRU, and GRU-LSTM). Performance evaluation was conducted using Monte Carlo simulation with 100 iterations across three real-world datasets: sunspot activity, Indonesian COVID-19 cases, and dissolved oxygen concentration measurements. Statistical analysis employed the Friedman test to assess performance differences across architectures. Results showed no statistically significant differences among the nine architectures. Despite the lack of statistical significance, consistent performance patterns emerged favoring LSTM-based hybrid architectures. The LSTM-GRU and LSTM-RNN configurations demonstrated superior performance across multiple evaluation metrics, with LSTM-RNN excelling in sunspot and dissolved oxygen forecasting, while standalone LSTM showed optimal performance for COVID-19 prediction. These findings provide evidence-based guidance for architecture selection in time series forecasting applications, suggesting that while statistical equivalence exists among architectures, LSTM-based hybrids offer practical advantages in terms of consistency and robustness across diverse temporal patterns.

Although the potential of the powerful mapping and representational capabilities of recurrent network architectures is generally recognized by the neural network research community, recurrent neural networks have not been widely used for the control of nonlinear dynamical systems, possibly due to the relative ineffectiveness of simple gradient descent training algorithms. Developments in the use of parameter-based extended Kalman filter algorithms for training recurrent networks may provide a mechanism by which these architectures will prove to be of practical value. This paper presents a decoupled extended Kalman filter (DEKF) algorithm for training of recurrent networks with special emphasis on application to control problems. We demonstrate in simulation the application of the DEKF algorithm to a series of example control problems ranging from the well-known cart-pole and bioreactor benchmark problems to an automotive subsystem, engine idle speed control. These simulations suggest that recurrent controller networks trained by Kalman filter methods can combine the traditional features of state-space controllers and observers in a homogeneous architecture for nonlinear dynamical systems, while simultaneously exhibiting less sensitivity than do purely feedforward controller networks to changes in plant parameters and measurement noise.

Space-Division Multiple Access-OFDM based wireless communication has the potential to significantly increase the spectral efficiency, system performance and number of users. Research in the development of efficient multiuser signal detection algorithms for such systems has generated much interest in recent years. This research proposes MUD schemes utilizing three neural network (NN) models like Feed Forward NN (FNN) without any hidden layer, FNN with a single hidden layer and Recurrent Neural Network (RNN) as possible alternatives to existing Genetic Algorithm (GA) based Minimum Bit Error Rate (MBER) MUD. Further, these techniques offer low complexity. Extensive simulation based performance study is carried out to prove the efficiency of the proposed techniques. Better performance of FNN with hidden layer and RNN compared to FNN without hidden layer is clearly observed in rank deficient MIMO scenario, where number of users exceeds the number of receiving antennas. The Bit Error Rate (BER) and complexity analysis of the proposed neural MUD schemes are close to optimal ML and show improvement over the previous implemented technique.

The problem of high sensitivity in modeling is well known. Small perturbations in the model parameters may result in large, undesired changes in the model behavior. A number of authors have considered the issue of sensitivity in feedforward neural networks from a probabilistic perspective. Less attention has been given to such issues in recurrent neural networks. In this article, we present a new recurrent neural network architecture, that is capable of significantly improved parameter sensitivity properties compared to existing recurrent neural networks. The new recurrent neural network generalizes previous architectures by employing alternative discrete-time operators in place of the shift operator normally used. An analysis of the model demonstrates the existence of parameter sensitivity in recurrent neural networks and supports the proposed architecture. The new architecture performs significantly better than previous recurrent neural networks, as shown by a series of simple numerical experiments.

Deep learning has been used to train neural networks to estimate the Remaining Useful Life (RUL) of a machine given sensor signals from that machine. This has resulted in some accurate state-of-the-art RUL estimation models. Here we present a novel way to construct a generative model to estimate the RUL and quantify its uncertainty. It takes the form of a Linear Gaussian State Space Model (LGSSM) and is trained using the Kalman Filter; hence, uncertainty is quantified using this LGSSM instead of by other popular methods like Monte Carlo Dropout or Deep Ensembles. This means we can train by directly using the marginal log-likelihood loss and don’t require multiple samples to represent the uncertainty. However, this method is limited to using Gaussian distributions to quantify uncertainty. We avoid needing to use non-linear variants of the filter by processing the sensors using a neural network to represent noncausal sensor sequences as a “control variable” in the LGSSM. This noncausal representation is shown to be important for achieving state-of-the-art performance. The model is tested on a turbofan engine and dust filter dataset. The code can be found on GitHub. 1 1 https://github.com/StarMarco/Kalman_DVAE • Deep Neural Networks encode time window of sensor signals. • These are used as control/forcing variables of a Linear Gaussian State Space Model (LGSSM). • The “observables" of the LGSSM are the Remaining Useful Life (RUL) values of the machine. • The Kalman filtering algorithm is used to estimate the marginal log-likelihood of these RUL estimates. • After training the LGSSM can be evaluated over time to estimate the RUL trajectories.