Efficient Formulation and Implementation of Data Assimilation Methods

SummaryEstimating river discharge from in situ and/or remote sensing data is a key issue for evaluation of water balance at local and global scales and for water management. Variational data assimilation (DA) is a powerful approach used in operational weather and ocean forecasting, which can also be used in this context. A distinctive feature of the river discharge estimation problem is the likely presence of significant uncertainty in principal parameters of a hydraulic model, such as bathymetry and friction, which have to be included into the control vector alongside the discharge. However, the conventional variational DA method being used for solving such extended problems often fails. This happens because the control vector iterates (i.e., approximations arising in the course of minimization) result into hydraulic states not supported by the model. In this paper, we suggest a novel version of the variational DA method specially designed for solving estimation‐under‐uncertainty problems, which is based on the ideas of iterative regularization.The method is implemented with SIC2, which is a full Saint‐Venant based 1D‐network model. The SIC2 software is widely used by research, consultant and industrial communities for modeling river, irrigation canal, and drainage network behavior. The adjoint model required for variational DA is obtained by means of automatic differentiation. This is likely to be the first stable consistent adjoint of the 1D‐network model of a commercial status in existence.The DA problems considered in this paper are offtake/tributary estimation under uncertainty in the cross‐device parameters and inflow discharge estimation under uncertainty in the bathymetry defining parameters and the friction coefficient. Numerical tests have been designed to understand identifiability of discharge given uncertainty in bathymetry and friction. The developed methodology, and software seems useful in the context of the future Surface Water and Ocean Topography satellite mission. Copyright © 2016 John Wiley & Sons, Ltd.

The Kuramoto–Sivashinsky equation plays an important role as a low‐dimensional prototype for complicated fluid dynamics systems having been studied due to its chaotic pattern forming behavior. Up to now, efforts to carry out data assimilation with this 1‐D model were restricted to variational adjoint methods domain and only Chorin and Krause (Proc. Natl. Acad. Sci. 2004; 101(42):15013–15017) tested it using a sequential Bayesian filter approach. In this work we compare three sequential data assimilation methods namely the Kalman filter approach, the sequential Monte Carlo particle filter approach and the maximum likelihood ensemble filter methods. This comparison is to the best of our knowledge novel. We compare in detail their relative performance for both linear and nonlinear observation operators. The results of these sequential data assimilation tests are discussed and conclusions are drawn as to the suitability of these data assimilation methods in the presence of linear and nonlinear observation operators. Copyright © 2009 John Wiley & Sons, Ltd.

Challenges for data assimilation – from convective‐scale to climate

Surface observations can provide crucial information for NWP models. If not assimilated carefully, however, they can degrade forecast accuracy, especially in complex terrains like the Alps. The horizontal and vertical covariances of climatological background error covariances used in the three-dimensional variational (3DVar) data assimilation (DA) method can produce unrealistic increments over sloped terrain. For instance, an observation from a valley station can still generate increments at the mountaintop, even though the valley observation may not accurately represent the mountaintop's weather conditions. We used a hybrid three-dimensional ensemble variational (Hybrid-3DEnVar) DA method to address this issue, incorporating a 50-member convection-permitting ensemble. This method was recently tested in Geosphere Austria's convective scale limited-area NWP model AROME at a 2.5 km horizontal resolution. We assimilated 2-meter temperature, 2-meter relative humidity, geopotential, and 10-meter wind components from 680 surface stations, including the Austrian TAWES network and SYNOP observations from neighbouring countries. 400 stations were actively assimilated from the observation dataset, and the rest were used to verify the analysis.  Our results present the effectiveness of this newly tested Hybrid-3DEnVar against GeoSphere Austria's operational 3DVar in assimilating surface observations over complex Alpine terrain.

Reconstructing spatiotemporal systems from sparse observations remains a long-standing challenge in several domains, including geoscience, air pollution and fluid dynamics. While various data assimilation (DA) and machine learning (ML) methods have shown potential, they still face significant challenges (see [1]): The computational burden of conventional DA algorithms (including error covariance specification), particularly for multivariate, high-dimensional systems. Sparse and movable sensor placement, which makes conventional ML models (typically requiring fixed and regularly distributed input data) cumbersome. The ill-defined nature of the sparse reconstruction problem, which poses significant risks of overfitting. We will present our recent works aimed at addressing these challenges. More specifically, we developed latent DA algorithms [2] to reduce the computational burden of variational DA methods. These algorithms demonstrate great potential in efficiently assimilating sparse observations within a reduced-order latent space constructed by neural networks, thanks to the TorchDA library [3]. The latter enables GPU implementation of mainstream data assimilation methods and supports non-explicit state-observation transformation functions, provided they can be learned by a neural network.We have also employed advanced deep learning techniques, including Voronoi-tessellation CNNs [4] and Vision Transformer-based autoencoders [5], to learn mappings from sparse observations to the complete physical space. These approaches effectively address challenges such as movable sensor placements and varying sensor numbers. Their integration with DA algorithms has also been evaluated.Finally, our recent work explores [6] the utility of generative AI techniques, particularly denoising diffusion models, for field reconstruction from sparse observations. Generative AI methods offer two main advantages: first, they produce a sample from a probability distribution rather than predicting the mean as a fixed output, which can help mitigate overfitting caused by the illy-defined problem. Second, they inherently function as ensemble predictors by generating several samples, facilitating uncertainty quantification, which is essential in data assimilation. The numerical results tested on cases ranging from fluid dynamics benchmarks to semi-operational air pollution simulations will also be discussed.[1] Cheng, S., Quilodrán-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., Xiao, D., Janjic, T., Ding, W., Guo, Y., Carrassi, A., Bocquet, M. and Arcucci, R, 2023. Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review. IEEE/CAA Journal of Automatica Sinica[2] Cheng, S., Chen, J., Anastasiou, C., Angeli, P., Matar, O.K., Guo, Y.K., Pain, C.C. and Arcucci, R., 2023. Generalised latent assimilation in heterogeneous reduced spaces with machine learning surrogate models. Journal of Scientific Computing[3] Cheng, S., Min, J., Liu, C. and Arcucci, R., 2025. TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions. Computer Physics Communications[4] Cheng, S., Liu, C., Guo, Y. and Arcucci, R., 2024. Efficient deep data assimilation with sparse observations and time-varying sensors. Journal of Computational Physics[5] Fan, H., Cheng, S., de Nazelle, A.J. and Arcucci, R., 2024. ViTAE-SL: a vision transformer-based autoencoder and spatial interpolation learner for field reconstruction. Computer Physics Communications [6] Zhuang, Y., Cheng, S. and Duraisamy, K., 2025. Spatially-aware diffusion models with cross-attention for global field reconstruction with sparse observations. Computer Methods in Applied Mechanics and Engineering

In this study, we present a probabilistic urban inundation modeling framework that combines high-resolution process-based modeling with observed information via ensemble data assimilation (DA). We investigate the impact of multivariate flood observations on improving urban inundation prediction accuracy through synthetic experiments. The framework leverages diverse flood observations from both the urban surface and sewer system, integrating them into the modeling process using non-Gaussian sequential DA methods, such as particle filtering. The modeling framework employs the H12 model for integrated 1D sewer network and 2D surface inundation analyses, with synthetic experiments conducted in an urban catchment in Osaka, Japan. Prior to implementing DA, a sensitivity analysis is performed to assess the effects of uncertainties in inundation modeling. Major uncertainty components, such as input forcings and storm drain box efficiency, are perturbed to evaluate their influence. The synthetic DA experiments analyze the influence of various types of flood observations, including urban surface inundation depths and sewer water levels, on the posterior distributions of the model ensemble. Additionally, the impact of observation location and density on DA performance is evaluated. This study demonstrates the potential of utilizing diverse flood observations in data assimilation to enhance the accuracy and reliability of urban flood predictions.

In numerical weather prediction, parameterisations are used to simulate missing physics in the model. These can be due to a lack of scientific understanding or a lack of computing power available to address all the known physical processes. Parameterisations are sources of large uncertainty in a model as parameter values used in these parameterisations cannot be measured directly and hence are often not well known; and the parameterisations themselves are also approximations of the processes present in the true atmosphere. Whilst there are many efficient and effective methods for combined state/parameter estimation in data assimilation (DA), such as state augmentation, these are not effective at estimating the structure of parameterisations. A new method of parameterisation estimation is proposed that uses sequential DA methods to estimate errors in the numerical models at each space-time point for each model equation. These errors are then fitted to pre-determined functional forms of missing physics or parameterisations that are based upon prior information. We applied the method to a one-dimensional advection model with additive model error, and it is shown that the method can accurately estimate parameterisations, with consistent error estimates. Furthermore, it is shown how the method depends on the quality of the DA results. The results indicate that this new method is a powerful tool in systematic model improvement.

In the past few years sequential data assimilation (SDA) methods have emerged as the best possible method at hand to properly treat all sources of error in hydrological modeling. However, very few studies have actually implemented SDA methods using realistic input error models for precipitation. In this study we use particle filtering as a SDA method to propagate input errors through a conceptual hydrologic model and quantify the state, parameter and streamflow uncertainties. Recent progress in satellite‐based precipitation observation techniques offers an attractive option for considering spatiotemporal variation of precipitation. Therefore, we use the PERSIANN‐CCS precipitation product to propagate input errors through our hydrologic model. Some uncertainty scenarios are set up to incorporate and investigate the impact of the individual uncertainty sources from precipitation, parameters and also combined error sources on the hydrologic response. Also probabilistic measures are used to quantify the quality of ensemble prediction.

This paper proposes a mini‐batch stochastic optimization‐based adaptive localization scheme for computing the “optimal” localization radius in data assimilation (DA) applications. After constructing a cost function of the localization radius by estimating forecast and observation error statistics, a mini‐batch stochastic gradient descent method with a novel sampling strategy is proposed to minimize the cost function. The proposed stochastic optimization algorithm is further incorporated into the DA method NLS‐i4DVar (the nonlinear least squares integral correcting four‐dimensional variational DA method), which was developed by the authors in Tian et al. (2021, https://doi.org/10.1029/2021EA001767). It is utilized to compute the “optimal” covariance localization radii adaptively and flow‐dependently inside NLS‐i4DVar. The computational cost of NLS‐i4DVar with the proposed adaptive localization scheme only increases slightly due to the use of the mini‐batch stochastic optimization algorithm. Numerical experimental results using the shallow‐water equations demonstrate that NLS‐i4DVar with the proposed adaptive localization scheme shows substantial performance improvement over the standard NLS‐i4DVar method.

<p>        While complete atmospheric chemical transport models have been developed to understanding the complex interactions of atmospheric chemistry and physics, there are large uncertainties in numerical approaches. Data assimilation is an efficient method to improve model forecast of aerosols with optimized initial conditions. We have developed a new framework for assimilating surface fine particulate matter (PM<sub>2.5</sub>) observations in coupled Weather Research and Forecasting (WRF) model and Community Multiscale Air Quality (CMAQ) model, based on nonlinear least squares four-dimensional variational (NLS-4DVar) data assimilation method. The NLS-4DVar approach, which does not require the tangent and adjoint models, has been extensive used in meteorological and environmental areas due to the low computational complexity. Two parallel experiments were designed in the observing system simulation experiments (OSSEs) to evaluate the effectiveness of this system. Hourly PM2.5 observations over China be assimilated in WRF-CMAQ model with 6-h assimilation window, while the background state without data assimilation is conducted as control experiment. The results show that the assimilation significantly reduced the uncertainties of initial conditions (ICs) for WRF-CMAQ model and leads to better forecast. The newly developed PM<sub>2.5</sub> data assimilation system can improve PM<sub>2.5</sub> prediction effectively and easily. In the future, we expect emission to be optimized together with concentrations, and integrate meteorological assimilation into aerosol assimilation system.</p>

Since the publication of the first edition of this highly regarded textbook, the value of data assimilation has become widely recognized across the Earth sciences and beyond. Data assimilation methods are now being applied to many areas of prediction and forecasting, including extreme weather events, wildfires, infectious disease epidemics, and economic modeling. This second edition provides a broad introduction to applications across the Earth systems and coupled Earth–human systems, with an expanded range of topics covering the latest developments of variational, ensemble, and hybrid data assimilation methods. New toy models and intermediate-complexity atmospheric general circulation models provide hands-on engagement with key concepts in numerical weather prediction, data assimilation, and predictability. The inclusion of computational projects, exercises, lecture notes, teaching slides, and sample exams makes this textbook an indispensable and practical resource for advanced undergraduate and graduate students, researchers, and practitioners who work in weather forecasting and climate prediction.

Using radar observations, the performances of the ensemble square root filter (EnSRF) and an indirect three-dimensional variational (3DVar) data assimilation method were compared for a mesoscale convective system (MCS) that occurred in the Front Range of the Rocky Mountains, Colorado (USA). The results showed that the root mean square innovations (RMSIs) of EnSRF were lower than 3DVar for radar reflectivity and radial velocity and that the spread of EnSRF was generally consistent with its RMSIs. EnSRF substantially improved the analysis of the MCS compared with an experiment without radar data assimilation, and it produced a slight but noticeable improvement over 3DVar in terms of both coverage and intensity. Forecast results initiated from the final analysis revealed that EnSRF generally produced the best prediction of the MCS, with improved quantitative reflectivity and precipitation forecast skills. EnSRF also demonstrated better performance than 3DVar in the prediction of neighborhood probability for reflectivity at thresholds of 20 and 35 dBZ, which better matched the observed radar reflectivity in terms of both shape and extension. Additionally, the humidity, temperature, and wind fields were also improved by EnSRF; the largest error reduction was found in the water vapor field near the surface and at upper levels.

Coupled climate models integrate atmospheric, oceanic, and land submodels, while the uncertainty of model parameters from different parameterization schemes or empirically derived parameters inevitably introduces systematic biases. Coupled parameter optimization (CPO) can reduce these biases to improve weather forecast and climate prediction, but must address strong nonlinearities inherent in coupled models. The analytical four-dimensional ensemble variational (A-4DEnVar) data assimilation method retains the nonlinear processing capability of the four-dimensional variational (4D-Var) data assimilation method but gets rid of the dependence on the adjoint model. In this study, a novel dynamic independent point (DIP) scheme is introduced to the improved A-4DEnVar, which reduces computational dimensionality and further explores a broader parameter space of dimensionality reduction through the outer loop. Based on the improved A-4DEnVar, a series of geographic-dependent CPO experiments with an idealized 2D coupled model are carried out. Results show that A-4DEnVar accurately captures the geographical characteristics of parameters and effectively optimizes cross-component parameters despite strong nonlinearity. Additionally, the DIP scheme presents significant advantages compared to the static independent point scheme, especially with fewer independent points. This work is offering a new perspective for parameter optimization in coupled general circulation models used for climate estimation and prediction.

Ensuring a reliable water supply in the face of changing conditions and growing demand is a critical global challenge. Water distribution networks (WDNs) are essential infrastructure, but traditional modeling methods based on historical data often struggle to adapt in real-time and integrate new information. In order to lower model errors in WDN models, this study explores the use of a Data Assimilation (DA) method that makes use of the Ensemble Kalman Filter. The study explores the effectiveness of the DA method, and a novel Greedy Algorithm (GA) for optimizing the location of sensors. The study shows that the DA method improves the WDN model and the GA is able to successfully determine optimal sensor locations. It was observed that increasing the number of sensors in the WDN increased the effectiveness of the DA method. This study highlights the potential of data assimilation to improve WDN modeling. Water utilities can gain from more precise predictions, greater system performance, and efficient decision-making for the management and maintenance of water distribution networks by allowing models to adapt and improve dynamically

Quantifying continental-scale river discharge is essential to understanding the terrestrial water cycle but is susceptible to errors caused by a lack of observations and the limitations of hydrodynamic modeling. Data assimilation (DA) methods are increasingly used to estimate river discharge in combination with emerging river-related remote sensing products (e.g., water surface elevation [WSE], water surface slope, river width, and flood extent). However, directly comparing simulated WSE to satellite altimetry data remains challenging (e.g., because of large biases between simulations and observations or uncertainties in parameters), and large errors can be introduced when satellite observations are assimilated into hydrodynamic models. In this study we performed direct, anomaly, and normalized value assimilation experiments to investigate the capacity of DA to improve river discharge within the current limitations of hydrodynamic modeling. We performed hydrological DA using a physically-based empirical localization method applied to the Amazon Basin. We used satellite altimetry data from ENVISAT, Jason 1, and Jason 2. Direct DA was the baseline assimilation method and was subject to errors due to biases in the simulated WSE. To overcome these errors, we used anomaly DA as an alternative to direct DA. We found that the modeled and observed WSE distributions differed considerably (e.g., differences in amplitude, seasonal flow variation, and a skewed distribution due to limitations of the hydrodynamic models). Therefore, normalized value DA was performed to improve discharge estimation. River discharge estimates were improved at 24 %, 38 %, and 62 % of stream gauges in the direct, anomaly, and normalized value assimilations relative to simulations without DA. Normalized value assimilation performed best for estimating river discharge given the current limitations of hydrodynamic models. Most gauges within the river reaches covered by satellite observations accurately estimated river discharge, with Nash-Sutcliffe efficiency (NSE) > 0.6. The amplitudes of WSE variation were improved in the normalized DA experiment. Furthermore, in the Amazon Basin, normalized assimilation (median NSE = 0.50) improved river discharge estimation compared to open-loop simulation with the global hydrodynamic model (median NSE = 0.42). River discharge estimation using direct DA methods was improved by 7 % with calibration of river bathymetry based on NSE. The direct DA approach outperformed the other DA approaches when runoff was considerably biased, but anomaly DA performed best when the river bathymetry was erroneous. The uncertainties in hydrodynamic modeling (e.g., river bottom elevation, river width, simplified floodplain dynamics, and the rectangular cross-section assumption) should be improved to fully realize the advantages of river discharge DA through the assimilation of satellite altimetry. This study contributes to the development of a global river discharge reanalysis product that is consistent spatially and temporally.