Ensemble data assimilation methods for improving river water quality forecasting accuracy

<p><span>The Madden-Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability with wide reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason are suboptimal state estimates obtained with standard data assimilation (DA) approaches. Those are typically based on filtering methods with Gaussian approximations and do not consider physical properties that are specifically important for the MJO.</span></p><p><span>In our recent paper (Gleiter et al. 2022), a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO with the Skeleton model. The utilized quadratic programming ensemble (QPEns) algorithm extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows to recover physically more consistent states and to respect possible associated non-Gaussian statistics. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial condition. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that even in cases where the statistical error in some waves remains comparable to the stochastic EnKF during the DA stage, their prediction is remarkably improved when using the initial state resulting from the QPEns.</span></p><p><strong>Gleiter, T., T. Janjic, N. Chen, </strong>2022, <em>Ensemble Kalman Filter based Data Assimilation for Tropical Waves in the MJO Skeleton Model, </em>QJR Meteorol Soc., https://doi.org/10.1002/qj.4245</p>

The Madden–Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability, with wide‐reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason is the suboptimal state estimates obtained with standard data assimilation (DA) approaches. These are typically based on filtering methods with Gaussian approximations and do not take into account physical properties that are important specifically for the MJO. In this article, a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO. The quadratic programming ensemble (QPEns) algorithm utilized extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows us to recover physically more consistent states and to respect possible associated non‐Gaussian statistics. The study is based on identical twin experiments with an adopted nonlinear model for tropical intraseasonal variability. This so‐called skeleton model succeeds in reproducing the main large‐scale features of the MJO and closely related tropical waves, while keeping adequate simplicity for fast experiments on intraseasonal time‐scales. Conservation laws and other crucial physical properties from the model are examined as constraints in the QPEns. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial conditions. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that, even in cases where the statistical error in some waves remains comparable with the stochastic EnKF during the DA stage, their prediction is improved remarkably when using the initial state resulting from the QPEns.

The ensemble Kalman filter (EnKF) is a data assimilation scheme based on the traditional Kalman filter update equation. An ensemble of forecasts are used to estimate the background-error covariances needed to compute the Kalman gain. It is known that if the same observations and the same gain are used to update each member of the ensemble, the ensemble will systematically underestimate analysis-error covariances. This will cause a degradation of subsequent analyses and may lead to filter divergence. For large ensembles, it is known that this problem can be alleviated by treating the observations as random variables, adding random perturbations to them with the correct statistics. Two important consequences of sampling error in the estimate of analysis-error covariances in the EnKF are discussed here. The first results from the analysis-error covariance being a nonlinear function of the backgrounderror covariance in the Kalman filter. Due to this nonlinearity, analysis-error covariance estimates may be negatively biased, even if the ensemble background-error covariance estimates are unbiased. This problem must be dealt with in any Kalman filter‐based ensemble data assimilation scheme. A second consequence of sampling error is particular to schemes like the EnKF that use perturbed observations. While this procedure gives asymptotically correct analysis-error covariance estimates for large ensembles, the addition of perturbed observations adds an additional source of sampling error related to the estimation of the observation-error covariances. In addition to reducing the accuracy of the analysis-error covariance estimate, this extra source of sampling error increases the probability that the analysis-error covariance will be underestimated. Because of this, ensemble data assimilation methods that use perturbed observations are expected to be less accurate than those which do not. Several ensemble filter formulations have recently been proposed that do not require perturbed observations. This study examines a particularly simple implementation called the ensemble square root filter, or EnSRF. The EnSRF uses the traditional Kalman gain for updating the ensemble mean but uses a ‘‘reduced’’ Kalman gain to update deviations from the ensemble mean. There is no additional computational cost incurred by the EnSRF relative to the EnKF when the observations have independent errors and are processed one at a time. Using a hierarchy of perfect model assimilation experiments, it is demonstrated that the elimination of the sampling error associated with the perturbed observations makes the EnSRF more accurate than the EnKF for the same ensemble size.

Parametric uncertainty in convection parameterization is one major source of model errors that cause model climate drift. Convection parameter tuning has been widely studied in atmospheric models to help mitigate the problem. However, in a fully coupled general circulation model (CGCM), convection parameters which impact the ocean as well as the climate simulation may have different optimal values. This study explores the possibility of estimating convection parameters with an ensemble coupled data assimilation method in a CGCM. Impacts of the convection parameter estimation on climate analysis and forecast are analyzed. In a twin experiment framework, five convection parameters in the GFDL coupled model CM2.1 are estimated individually and simultaneously under both perfect and imperfect model regimes. Results show that the ensemble data assimilation method can help reduce the bias in convection parameters. With estimated convection parameters, the analyses and forecasts for both the atmosphere and the ocean are generally improved. It is also found that information in low latitudes is relatively more important for estimating convection parameters. This study further suggests that when important parameters in appropriate physical parameterizations are identified, incorporating their estimation into traditional ensemble data assimilation procedure could improve the final analysis and climate prediction.

Challenges for data assimilation – from convective‐scale to climate

Ensuring a reliable water supply in the face of changing conditions and growing demand is a critical global challenge. Water distribution networks (WDNs) are essential infrastructure, but traditional modeling methods based on historical data often struggle to adapt in real-time and integrate new information. In order to lower model errors in WDN models, this study explores the use of a Data Assimilation (DA) method that makes use of the Ensemble Kalman Filter. The study explores the effectiveness of the DA method, and a novel Greedy Algorithm (GA) for optimizing the location of sensors. The study shows that the DA method improves the WDN model and the GA is able to successfully determine optimal sensor locations. It was observed that increasing the number of sensors in the WDN increased the effectiveness of the DA method. This study highlights the potential of data assimilation to improve WDN modeling. Water utilities can gain from more precise predictions, greater system performance, and efficient decision-making for the management and maintenance of water distribution networks by allowing models to adapt and improve dynamically

Since the publication of the first edition of this highly regarded textbook, the value of data assimilation has become widely recognized across the Earth sciences and beyond. Data assimilation methods are now being applied to many areas of prediction and forecasting, including extreme weather events, wildfires, infectious disease epidemics, and economic modeling. This second edition provides a broad introduction to applications across the Earth systems and coupled Earth–human systems, with an expanded range of topics covering the latest developments of variational, ensemble, and hybrid data assimilation methods. New toy models and intermediate-complexity atmospheric general circulation models provide hands-on engagement with key concepts in numerical weather prediction, data assimilation, and predictability. The inclusion of computational projects, exercises, lecture notes, teaching slides, and sample exams makes this textbook an indispensable and practical resource for advanced undergraduate and graduate students, researchers, and practitioners who work in weather forecasting and climate prediction.

A localized ensemble Kalman filter (EnKF) method is developed to assimilate transient flow data to calibrate a heterogeneous conductivity field. To update conductivity value at a point in a study domain, instead of assimilating all the measurements in the study domain, only limited measurement data in an area around the point are used for the conductivity updating in the localized EnKF method. The localized EnKF is proposed to solve the problems of the filter divergence usually existing in a data assimilation method without localization. The developed method is applied, in a synthetical two dimensional case, to calibrate a heterogeneous conductivity field by assimilating transient hydraulic head data. The simulations by the data assimilation with and without localized EnKF are compared. The study results indicate that the hydraulic conductivity field can be updated efficiently by the localized EnKF, while it cannot be by the EnKF. The covariance inflation and localization are found to solve the problem of the filter divergence efficiently. In comparison with the EnKF method without localization, the localized EnKF method needs smaller ensemble size to achieve stabilized results. The simulation results by the localized EnKF method are much more sensitive to conductivity correlation length than to the localization radius. The developed localized EnKF method provides an approach to improve EnKF method in conductivity calibration.

Quantifying continental-scale river discharge is essential to understanding the terrestrial water cycle but is susceptible to errors caused by a lack of observations and the limitations of hydrodynamic modeling. Data assimilation (DA) methods are increasingly used to estimate river discharge in combination with emerging river-related remote sensing products (e.g., water surface elevation [WSE], water surface slope, river width, and flood extent). However, directly comparing simulated WSE to satellite altimetry data remains challenging (e.g., because of large biases between simulations and observations or uncertainties in parameters), and large errors can be introduced when satellite observations are assimilated into hydrodynamic models. In this study we performed direct, anomaly, and normalized value assimilation experiments to investigate the capacity of DA to improve river discharge within the current limitations of hydrodynamic modeling. We performed hydrological DA using a physically-based empirical localization method applied to the Amazon Basin. We used satellite altimetry data from ENVISAT, Jason 1, and Jason 2. Direct DA was the baseline assimilation method and was subject to errors due to biases in the simulated WSE. To overcome these errors, we used anomaly DA as an alternative to direct DA. We found that the modeled and observed WSE distributions differed considerably (e.g., differences in amplitude, seasonal flow variation, and a skewed distribution due to limitations of the hydrodynamic models). Therefore, normalized value DA was performed to improve discharge estimation. River discharge estimates were improved at 24 %, 38 %, and 62 % of stream gauges in the direct, anomaly, and normalized value assimilations relative to simulations without DA. Normalized value assimilation performed best for estimating river discharge given the current limitations of hydrodynamic models. Most gauges within the river reaches covered by satellite observations accurately estimated river discharge, with Nash-Sutcliffe efficiency (NSE) > 0.6. The amplitudes of WSE variation were improved in the normalized DA experiment. Furthermore, in the Amazon Basin, normalized assimilation (median NSE = 0.50) improved river discharge estimation compared to open-loop simulation with the global hydrodynamic model (median NSE = 0.42). River discharge estimation using direct DA methods was improved by 7 % with calibration of river bathymetry based on NSE. The direct DA approach outperformed the other DA approaches when runoff was considerably biased, but anomaly DA performed best when the river bathymetry was erroneous. The uncertainties in hydrodynamic modeling (e.g., river bottom elevation, river width, simplified floodplain dynamics, and the rectangular cross-section assumption) should be improved to fully realize the advantages of river discharge DA through the assimilation of satellite altimetry. This study contributes to the development of a global river discharge reanalysis product that is consistent spatially and temporally.

We apply information theory within an ensemble‐based data assimilation approach and define information matrix in ensemble subspace. The information matrix in ensemble subspace employs a flow‐dependent forecast error covariance and it is of relatively small dimensions (equal to the ensemble size). The information matrix in ensemble subspace can be directly linked to the information matrix typically used in non‐ensemble‐based data assimilation methods, such as the Kalman Filter (KF) and the three‐dimensional variational (3D‐Var) methods, which provides a framework for consistent comparisons of information measures between different data assimilation methods.We evaluate information measures, such as degrees of freedom for signal, within the Maximum Likelihood Ensemble Filter (MLEF) data assimilation approach and compare them with those obtained using the KF approach and the 3D‐Var approach. We assimilate model‐simulated observations and use the Goddard Earth Observing System Single Column Model (GEOS‐5 SCM) as a dynamical forecast model.The experimental results demonstrate that the proposed framework is useful for comparing information measures obtained in different data assimilation approaches. These comparisons indicate that using a flow‐dependent forecast error covariance matrix (e.g. as in the KF and the MLEF experiments) is fundamentally important for adequately describing prior knowledge about the true model state when calculating information measures of assimilated observations. We also demonstrate that data assimilation results obtained using the KF and the MLEF approach (when ensemble size is larger than 10 ensemble members) are superior to the results of the 3D‐Var approach. Copyright © 2007 Royal Meteorological Society

We present a data assimilation system to estimate surface fluxes of CO 2 and other trace gases from observations of their atmospheric abundances. The system is based on ensemble data assimilation methods under development for Numerical Weather Prediction (NWP) and is the first of its kind to be used for CO 2 flux estimation. The system was developed to overcome computational limitations encountered when a large number of observations are used to estimate a large number of unknown surface fluxes. The ensemble data assimilation approach is attractive because it returns an approximation of the covariance, does not need an adjoint model or other linearization of the observation operator, and offers the possibility to optimize fluxes of chemically active trace gases (e.g., CH 4 , CO) in the same framework. We assess the performance of this new system in a pseudodata experiment that resembles the real problem we will apply this system to. The sensitivity of the method to the choice of several parameters such as the assimilation window size and the number of ensemble members is investigated. We conclude that the system is able to provide satisfactory flux estimates for the relatively large scales resolved by our current observing network and that the loss of information in the approximated covariances is an acceptable price to pay for the efficient computation of a large number of surface fluxes. The full potential of this data assimilation system will be used for near–real time operational estimates of North American CO 2 fluxes. This will take advantage of the large amounts of atmospheric data that will be collected by NOAA‐CMDL in conjunction with the implementation of the North American Carbon Program (NACP).

Data Assimilation for Marine Ecosystem Models

The forecasting of earthquake occurrence remains a significant challenge in seismology, primarily due to uncertainties in understanding the current state of stress, strength, and the parameters controlling the slip behaviour of faults. Among these parameters, friction parameters are crucial as they control the earthquake recurrence interval and their nucleation size. There is a critical gap in effectively integrating observational data with physics-based models, particularly in the face of parameter bias. This study addresses how ensemble data assimilation methods can be optimized to address these challenges and reduce uncertainties in fault-slip estimates.Our objective is to enhance the accuracy of earthquake forecasting by incorporating model error into ensemble data assimilation methods, thus improving the estimation of critical state variables such as shear stress and slip velocity. We employed the Ensemble Kalman (EnKF), Adaptive Gaussian Mixture (AGMF), and Particle Flow (PFF) filters, which are integrated with earthquake sequence models. These methods were applied in two stages: 1) Perfect model experiments using 1-D Burridge-Knopoff models to assess the benefits of including model error in estimating non-periodic and chaotic behaviours in low-dimensional systems. 2) Application in a meter-scale direct-shear laboratory setup, assimilating measurements from shear-strain gauges near the fault, to examine the effects of varying normal stress profiles on the fault on the estimates and the impact of including model error.The perfect model experiments demonstrated improved estimation of shear stress, slip velocity, and the state variable (θ), particularly in estimating non-periodic sequences and chaotic behaviour using stable periodic solutions when confronted with small parameter biases. In the laboratory setup, variations in normal stress profiles significantly influenced the information content. Sensor placement relative to the fault and seismic phase was found to critically impact the observations' informational value, with sensor proximity to the fault being a critical aspect and the information content being higher around the coseismic phase.This research provides valuable insights into the intricate process of earthquake forecasting, underscoring the role of data-assimilation techniques in enhancing our understanding and forecasting abilities in this field.

Data assimilation is the process of estimating the state of dynamic systems (linear or nonlinear, Gaussian or non-Gaussian) as accurately as possible from noisy observational data. Although the Three Dimensional Variational (3D-VAR) methods, Four Dimensional Variational (4D-VAR) methods and Ensemble Kalman filter (EnKF) methods are widely used and effective for linear and Gaussian dynamics, new methods of data assimilation are required for the general situation, that is, nonlinear non-Gaussian dynamics. General Bayesian recursive estimation theory is reviewed in this thesis. The Bayesian estimation approach provides a rather general and powerful framework for handling nonlinear, non-Gaussian, as well as linear, Gaussian estimation problems. Despite a general solution to the nonlinear estimation problem, there is no closed-form solution in the general case. Therefore, approximate techniques have to be employed. In this thesis, the sequential Monte Carlo (SMC) methods, commonly referred to as the particle filter, is presented to tackle non-linear, non-Gaussian estimation problems. In this thesis, we use the SMC methods only for the nonlinear state estimation problem, however, it can also be used for the nonlinear parameter estimation problem. In order to demonstrate the new methods in the general nonlinear non-Gaussian case, we compare Sequential Monte Carlo (SMC) methods with the Ensemble Kalman Filter (EnKF) by performing data assimilation in nonlinear and non-Gaussian dynamic systems. The models used in this study are referred to as state-space models. The Lorenz 1963 and 1966 models serve as test beds for examining the properties of these assimilation methods when used in highly nonlinear dynamics. The application of Sequential Monte Carlo methods to different fixed parameters in dynamic models is considered. Four different scenarios in the Lorenz 1063 [sic] model and three different scenarios in the Lorenz 1996 model are designed in this study for both the SMC methods and EnKF method with different filter siz

Data assimilation using support vector machines and ensemble Kalman filter for multi-layer soil moisture prediction