Applications of information theory in ensemble data assimilation

Comparative evaluation of maximum likelihood ensemble filter and ensemble Kalman filter for real-time assimilation of streamflow data into operational hydrologic models

Quantifying continental-scale river discharge is essential to understanding the terrestrial water cycle but is susceptible to errors caused by a lack of observations and the limitations of hydrodynamic modeling. Data assimilation (DA) methods are increasingly used to estimate river discharge in combination with emerging river-related remote sensing products (e.g., water surface elevation [WSE], water surface slope, river width, and flood extent). However, directly comparing simulated WSE to satellite altimetry data remains challenging (e.g., because of large biases between simulations and observations or uncertainties in parameters), and large errors can be introduced when satellite observations are assimilated into hydrodynamic models. In this study we performed direct, anomaly, and normalized value assimilation experiments to investigate the capacity of DA to improve river discharge within the current limitations of hydrodynamic modeling. We performed hydrological DA using a physically-based empirical localization method applied to the Amazon Basin. We used satellite altimetry data from ENVISAT, Jason 1, and Jason 2. Direct DA was the baseline assimilation method and was subject to errors due to biases in the simulated WSE. To overcome these errors, we used anomaly DA as an alternative to direct DA. We found that the modeled and observed WSE distributions differed considerably (e.g., differences in amplitude, seasonal flow variation, and a skewed distribution due to limitations of the hydrodynamic models). Therefore, normalized value DA was performed to improve discharge estimation. River discharge estimates were improved at 24 %, 38 %, and 62 % of stream gauges in the direct, anomaly, and normalized value assimilations relative to simulations without DA. Normalized value assimilation performed best for estimating river discharge given the current limitations of hydrodynamic models. Most gauges within the river reaches covered by satellite observations accurately estimated river discharge, with Nash-Sutcliffe efficiency (NSE) > 0.6. The amplitudes of WSE variation were improved in the normalized DA experiment. Furthermore, in the Amazon Basin, normalized assimilation (median NSE = 0.50) improved river discharge estimation compared to open-loop simulation with the global hydrodynamic model (median NSE = 0.42). River discharge estimation using direct DA methods was improved by 7 % with calibration of river bathymetry based on NSE. The direct DA approach outperformed the other DA approaches when runoff was considerably biased, but anomaly DA performed best when the river bathymetry was erroneous. The uncertainties in hydrodynamic modeling (e.g., river bottom elevation, river width, simplified floodplain dynamics, and the rectangular cross-section assumption) should be improved to fully realize the advantages of river discharge DA through the assimilation of satellite altimetry. This study contributes to the development of a global river discharge reanalysis product that is consistent spatially and temporally.

The weather research and forecasting (WRF) model and the maximum likelihood ensemble filter (MLEF) data assimilation approach are used to examine the potential impact of observations from the future Geostationary Operational Environmental Satellite, generation R (GOES-R) on improving our knowledge about clouds. Synthetic radiances are assimilated from the 10.35 μm channel of the GOES-R advanced baseline imager (ABI) employing a ‘non-identical twins’ experimental setup. The experimental results are examined for an extratropical cyclone named Kyrill that produced unusually strong winds, widespread damage and fatalities in Western Europe in January 2007. The data assimilation problem is especially challenging for this case, as there is a large error in the model-simulated radiances resulting from incorrect cloud location. Although this problem is difficult to eliminate, data assimilation results indicate the potential of GOES-R data to significantly reduce these errors.

Ensemble data assimilation methods for improving river water quality forecasting accuracy

In this study, we present a probabilistic urban inundation modeling framework that combines high-resolution process-based modeling with observed information via ensemble data assimilation (DA). We investigate the impact of multivariate flood observations on improving urban inundation prediction accuracy through synthetic experiments. The framework leverages diverse flood observations from both the urban surface and sewer system, integrating them into the modeling process using non-Gaussian sequential DA methods, such as particle filtering. The modeling framework employs the H12 model for integrated 1D sewer network and 2D surface inundation analyses, with synthetic experiments conducted in an urban catchment in Osaka, Japan. Prior to implementing DA, a sensitivity analysis is performed to assess the effects of uncertainties in inundation modeling. Major uncertainty components, such as input forcings and storm drain box efficiency, are perturbed to evaluate their influence. The synthetic DA experiments analyze the influence of various types of flood observations, including urban surface inundation depths and sewer water levels, on the posterior distributions of the model ensemble. Additionally, the impact of observation location and density on DA performance is evaluated. This study demonstrates the potential of utilizing diverse flood observations in data assimilation to enhance the accuracy and reliability of urban flood predictions.

We use an ensemble‐based data assimilation method, known as the maximum likelihood ensemble filter (MLEF), which has been coupled with a global atmospheric transport model to estimate slowly varying biases of carbon surface fluxes. Carbon fluxes for this test consist of hourly gross primary production and ecosystem, respiration over land, and air‐sea gas exchange. Persistent multiplicative biases intended to represent incorrectly simulated biogeochemical or land‐management processes such as stand age, soil fertility, or coarse woody debris were estimated for 1 year at 10° longitude by 6° latitude spatial resolution and with an 8‐week time window. We tested the model using a pseudodata experiment with an existing observation network that includes flasks, aircraft profiles, and continuous measurements. Because of the underconstrained nature of the problem, strong covariance smoothing was applied in the first data assimilation cycle, and localization schemes have been introduced. Error covariance was propagated in subsequent cycles. The coupled model satisfactorily recovered the land biases in densely observed areas. Ocean biases, however, were poorly constrained by the atmospheric observations. Unlike in batch mode inversions, the MLEF has a capability of assimilating large observation vectors and hence is suitable for assimilating hourly continuous observations and satellite observations in the future. Uncertainty was reduced further in our pseudodata experiment than by previous batch methods because of the ability to assimilate a large observation vector. Propagation of spatial covariance and dynamic localization avoid the need for prescribed spatial patterns of error covariance centered at observation sites as in previous grid‐scale methods.

Challenges for data assimilation – from convective‐scale to climate

<p><span>The Madden-Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability with wide reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason are suboptimal state estimates obtained with standard data assimilation (DA) approaches. Those are typically based on filtering methods with Gaussian approximations and do not consider physical properties that are specifically important for the MJO.</span></p><p><span>In our recent paper (Gleiter et al. 2022), a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO with the Skeleton model. The utilized quadratic programming ensemble (QPEns) algorithm extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows to recover physically more consistent states and to respect possible associated non-Gaussian statistics. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial condition. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that even in cases where the statistical error in some waves remains comparable to the stochastic EnKF during the DA stage, their prediction is remarkably improved when using the initial state resulting from the QPEns.</span></p><p><strong>Gleiter, T., T. Janjic, N. Chen, </strong>2022, <em>Ensemble Kalman Filter based Data Assimilation for Tropical Waves in the MJO Skeleton Model, </em>QJR Meteorol Soc., https://doi.org/10.1002/qj.4245</p>

The Madden–Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability, with wide‐reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason is the suboptimal state estimates obtained with standard data assimilation (DA) approaches. These are typically based on filtering methods with Gaussian approximations and do not take into account physical properties that are important specifically for the MJO. In this article, a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO. The quadratic programming ensemble (QPEns) algorithm utilized extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows us to recover physically more consistent states and to respect possible associated non‐Gaussian statistics. The study is based on identical twin experiments with an adopted nonlinear model for tropical intraseasonal variability. This so‐called skeleton model succeeds in reproducing the main large‐scale features of the MJO and closely related tropical waves, while keeping adequate simplicity for fast experiments on intraseasonal time‐scales. Conservation laws and other crucial physical properties from the model are examined as constraints in the QPEns. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial conditions. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that, even in cases where the statistical error in some waves remains comparable with the stochastic EnKF during the DA stage, their prediction is improved remarkably when using the initial state resulting from the QPEns.

ABSTRACTAn original data assimilation (DA) scheme with a general dynamics model is considered. It is shown that this scheme can be approximated by the stochastic diffusion process. The sufficient conditions to provide this approximation are formulated. Based on this algorithm a new DA method is developed. The method combines variational and statistical approaches commonly used in DA theory and minimizes the variance of the trajectory of a diffusion process in conjunction with a dynamics numerical model. In this sense the method is optimal in contrast to other DA approaches. The proposed scheme takes the model dynamics into account and in this way it differs from the well-known Kalman filter. Furthermore, the derived DA method can be applied to a very wide field of dynamical systems, for example, gas dynamics, fluid dynamics and other disciplines. However, the current study deals with oceanography and DA in oceanography specifically. Then the method is applied to the HYbrid Coordinate Ocean Model and assimilates satellite sea level anomaly data from the Archiving, Validating and Interpolating Satellite Oceanography Data over the Atlantic Ocean to correct the model state. Several numerical experiments have been performed. The experiments show that the method substantially changes the synoptic and mesoscale structure of ocean dynamics. Also, the distribution of the obtained result is estimated through the solution of the Fokker–Planck–Kolmogorov equation.

The ensemble Kalman filter (EnKF) is a data assimilation scheme based on the traditional Kalman filter update equation. An ensemble of forecasts are used to estimate the background-error covariances needed to compute the Kalman gain. It is known that if the same observations and the same gain are used to update each member of the ensemble, the ensemble will systematically underestimate analysis-error covariances. This will cause a degradation of subsequent analyses and may lead to filter divergence. For large ensembles, it is known that this problem can be alleviated by treating the observations as random variables, adding random perturbations to them with the correct statistics. Two important consequences of sampling error in the estimate of analysis-error covariances in the EnKF are discussed here. The first results from the analysis-error covariance being a nonlinear function of the backgrounderror covariance in the Kalman filter. Due to this nonlinearity, analysis-error covariance estimates may be negatively biased, even if the ensemble background-error covariance estimates are unbiased. This problem must be dealt with in any Kalman filter‐based ensemble data assimilation scheme. A second consequence of sampling error is particular to schemes like the EnKF that use perturbed observations. While this procedure gives asymptotically correct analysis-error covariance estimates for large ensembles, the addition of perturbed observations adds an additional source of sampling error related to the estimation of the observation-error covariances. In addition to reducing the accuracy of the analysis-error covariance estimate, this extra source of sampling error increases the probability that the analysis-error covariance will be underestimated. Because of this, ensemble data assimilation methods that use perturbed observations are expected to be less accurate than those which do not. Several ensemble filter formulations have recently been proposed that do not require perturbed observations. This study examines a particularly simple implementation called the ensemble square root filter, or EnSRF. The EnSRF uses the traditional Kalman gain for updating the ensemble mean but uses a ‘‘reduced’’ Kalman gain to update deviations from the ensemble mean. There is no additional computational cost incurred by the EnSRF relative to the EnKF when the observations have independent errors and are processed one at a time. Using a hierarchy of perfect model assimilation experiments, it is demonstrated that the elimination of the sampling error associated with the perturbed observations makes the EnSRF more accurate than the EnKF for the same ensemble size.

Data assimilation is a method that combines observations (ie, real world data) of a state of a system with model output for that system in order to improve the estimate of the state of the system and thereby the model output. The model is usually represented by a discretized partial differential equation. The data assimilation problem can be formulated as a large scale Bayesian inverse problem. Based on this interpretation we will derive the most important variational and sequential data assimilation approaches, in particular three‐dimensional and four‐dimensional variational data assimilation (3D‐Var and 4D‐Var) and the Kalman filter. We will then consider more advanced methods which are extensions of the Kalman filter and variational data assimilation and pay particular attention to their advantages and disadvantages. The data assimilation problem usually results in a very large optimization problem and/or a very large linear system to solve (due to inclusion of time and space dimensions). Therefore, the second part of this article aims to review advances and challenges, in particular from the numerical linear algebra perspective, within the various data assimilation approaches.

A methodology for model error estimation is proposed and examined in this study. It provides estimates of the dynamical model state, the bias, and the empirical parameters by combining three approaches: 1) ensemble data assimilation, 2) state augmentation, and 3) parameter and model bias estimation. Uncertainties of these estimates are also determined, in terms of the analysis and forecast error covariances, employing the same methodology. The model error estimation approach is evaluated in application to Korteweg–de Vries–Burgers (KdVB) numerical model within the framework of maximum likelihood ensemble filter (MLEF). Experimental results indicate improved filter performance due to model error estimation. The innovation statistics also indicate that the estimated uncertainties are reliable. On the other hand, neglecting model errors—either in the form of an incorrect model parameter, or a model bias—has detrimental effects on data assimilation, in some cases resulting in filter divergence. Although the method is examined in a simplified model framework, the results are encouraging. It remains to be seen how the methodology performs in applications to more complex models.

The 2010 Eyjafjallajökull volcano eruption had serious consequences to civil aviation. This has initiated a lot of research on volcanic ash forecasting in recent years. For forecasting the volcanic ash transport after eruption onset, a volcanic ash transport and diffusion model (VATDM) needs to be run with Eruption Source Parameters (ESPs) such as plume height and mass eruption rate as input, and with data assimilation techniques to continuously improve the forecast. Reliable and accurate ash measurements are crucial for providing successful ash clouds advices. In the first<br/>phase of this research work, simulated aircraft-based volcanic ash measurements, will be assimilated into a transport model to identify the potential benefit of this kind of observations in an assimilation system. The results show that assimilating aircraft-based measurements can improve the state of ash clouds, and can provide an improved forecast. We also show that for an advice on the aeroplane flying level, aircraft-based measurements should preferably be taken at this level. Furthermore<br/>it is shown that in order to make an acceptable advice for aviation decision makers, accurate knowledge about uncertainties of ESPs and measurements is of great importance.<br/><br/>The forecast accuracy of distal volcanic ash clouds is important for providing valid aviation advice during volcanic ash eruptions. However, because the distal part of a volcanic ash plume is far from the volcano, the influence of eruption information on this part becomes rather indirect and uncertain, resulting in inaccurate volcanic ash forecasts in these distal areas. In this thesis, we use real-life aircraft in situ observations, measured in the North-West part of Germany during the 2010 Eyjafjallajökull eruption, in an ensemble-based data assimilation system to investigate the potential improvement on the forecast accuracy with regard to the distal volcanic ash plume. We show that the error of the analyzed volcanic ash state can be significantly reduced by assimilating real-life in situ measurements. After assimilation, it is shown that the model-based aviation advice for Germany, the Netherlands and Luxembourg can be improved. We suggest that with suitable aircrafts measuring once per day across the distal volcanic ash plume, the description and prediction of volcanic ash clouds in these areas can be improved significantly.<br/><br/>Among the data assimilation approaches, the ensemble Kalman filter (EnKF) is a well-known and popular method. A proper covariance localization strategy in the analysis step of EnKF is essential for reducing spurious covariances caused by the finite ensemble size, as shown for this application for assimilation of aircraft in situ measurements. After analyzing the characteristics of the physical forecast error covariances, we present a two-way tracking approach to define the localization matrix<br/>for covariance localization. The result shows that the Two-way-tracking Localized EnKF (TL-EnKF) effectively maintains the correctly specified physical covariances and largely reduces the spurious ones. The computational cost of TL-EnKF is also evaluated and is shown to be advantageous for both serial and parallel implementations. Compared to the commonly used distance-based covariance localization, the two-way tracking approach is shown to be more suitable. In addition, the covariance inflation approach is verified as an additional improvement to TL-EnKF to achieve more accurate results.<br/><br/>A timely prediction requires that the computations of the data assimilation system can be performed quickly (at least than the Wall-clock). We therefore investigate strategies for accelerating the data assimilation algorithm. Based on evaluations of the computational time, the analysis step of the assimilation turns out to be the most expensive part. After a study on the characteristics of the ensemble ash state, we propose a mask-state algorithm which records the sparsity information of the full ensemble state matrix and transforms the full matrix into a relatively small one. This will reduce the computational cost in the analysis step. Experimental results show the mask-state algorithm significantly speeds up the analysis step. Subsequently, the total amount of computing time for volcanic ash data assimilation is reduced to an acceptable level. The mask-state algorithm is generic and thus can be embedded in any ensemble-based data assimilation framework. Moreover, ensemble-based data assimilation with the mask-state algorithm is promising and flexible, because it implements exactly the standard data assimilation without any approximation and it realizes the satisfying performance without any change of the full model.<br/><br/>Infrared satellite measurements of volcanic ash mass loadings are often used as<br/>input observations for the assimilation scheme. However, these satellite-retrieved<br/>data are often two-dimensional (2D), and cannot easily be combined with a three-dimensional (3D) volcanic ash model to improve the volcanic ash state. By integrating available data including ash mass loadings, cloud top heights and thickness information, we propose a satellite observational operator (SOO) that translates satellite-retrieved 2D volcanic ash mass loadings to 3D concentrations at the top layer of the ash cloud. Ensemble-based data assimilation is used to assimilate the extracted measurements of ash concentrations. The results show that satellite data assimilation can force the volcanic ash state to match the satellite observations, and that it improves the forecast of the ash state. Comparison with highly accurate aircraft in situ measurements shows that the effective duration of the improved volcanic ash forecasts is about a half day.

Ensuring a reliable water supply in the face of changing conditions and growing demand is a critical global challenge. Water distribution networks (WDNs) are essential infrastructure, but traditional modeling methods based on historical data often struggle to adapt in real-time and integrate new information. In order to lower model errors in WDN models, this study explores the use of a Data Assimilation (DA) method that makes use of the Ensemble Kalman Filter. The study explores the effectiveness of the DA method, and a novel Greedy Algorithm (GA) for optimizing the location of sensors. The study shows that the DA method improves the WDN model and the GA is able to successfully determine optimal sensor locations. It was observed that increasing the number of sensors in the WDN increased the effectiveness of the DA method. This study highlights the potential of data assimilation to improve WDN modeling. Water utilities can gain from more precise predictions, greater system performance, and efficient decision-making for the management and maintenance of water distribution networks by allowing models to adapt and improve dynamically