Discharge estimation under uncertainty using variational methods with application to the full Saint‐Venant hydraulic network model

Reconstructing spatiotemporal systems from sparse observations remains a long-standing challenge in several domains, including geoscience, air pollution and fluid dynamics. While various data assimilation (DA) and machine learning (ML) methods have shown potential, they still face significant challenges (see [1]): The computational burden of conventional DA algorithms (including error covariance specification), particularly for multivariate, high-dimensional systems. Sparse and movable sensor placement, which makes conventional ML models (typically requiring fixed and regularly distributed input data) cumbersome. The ill-defined nature of the sparse reconstruction problem, which poses significant risks of overfitting. We will present our recent works aimed at addressing these challenges. More specifically, we developed latent DA algorithms [2] to reduce the computational burden of variational DA methods. These algorithms demonstrate great potential in efficiently assimilating sparse observations within a reduced-order latent space constructed by neural networks, thanks to the TorchDA library [3]. The latter enables GPU implementation of mainstream data assimilation methods and supports non-explicit state-observation transformation functions, provided they can be learned by a neural network.We have also employed advanced deep learning techniques, including Voronoi-tessellation CNNs [4] and Vision Transformer-based autoencoders [5], to learn mappings from sparse observations to the complete physical space. These approaches effectively address challenges such as movable sensor placements and varying sensor numbers. Their integration with DA algorithms has also been evaluated.Finally, our recent work explores [6] the utility of generative AI techniques, particularly denoising diffusion models, for field reconstruction from sparse observations. Generative AI methods offer two main advantages: first, they produce a sample from a probability distribution rather than predicting the mean as a fixed output, which can help mitigate overfitting caused by the illy-defined problem. Second, they inherently function as ensemble predictors by generating several samples, facilitating uncertainty quantification, which is essential in data assimilation. The numerical results tested on cases ranging from fluid dynamics benchmarks to semi-operational air pollution simulations will also be discussed.[1] Cheng, S., Quilodrán-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., Xiao, D., Janjic, T., Ding, W., Guo, Y., Carrassi, A., Bocquet, M. and Arcucci, R, 2023. Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review. IEEE/CAA Journal of Automatica Sinica[2] Cheng, S., Chen, J., Anastasiou, C., Angeli, P., Matar, O.K., Guo, Y.K., Pain, C.C. and Arcucci, R., 2023. Generalised latent assimilation in heterogeneous reduced spaces with machine learning surrogate models. Journal of Scientific Computing[3] Cheng, S., Min, J., Liu, C. and Arcucci, R., 2025. TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions. Computer Physics Communications[4] Cheng, S., Liu, C., Guo, Y. and Arcucci, R., 2024. Efficient deep data assimilation with sparse observations and time-varying sensors. Journal of Computational Physics[5] Fan, H., Cheng, S., de Nazelle, A.J. and Arcucci, R., 2024. ViTAE-SL: a vision transformer-based autoencoder and spatial interpolation learner for field reconstruction. Computer Physics Communications [6] Zhuang, Y., Cheng, S. and Duraisamy, K., 2025. Spatially-aware diffusion models with cross-attention for global field reconstruction with sparse observations. Computer Methods in Applied Mechanics and Engineering

Quantifying continental-scale river discharge is essential to understanding the terrestrial water cycle but is susceptible to errors caused by a lack of observations and the limitations of hydrodynamic modeling. Data assimilation (DA) methods are increasingly used to estimate river discharge in combination with emerging river-related remote sensing products (e.g., water surface elevation [WSE], water surface slope, river width, and flood extent). However, directly comparing simulated WSE to satellite altimetry data remains challenging (e.g., because of large biases between simulations and observations or uncertainties in parameters), and large errors can be introduced when satellite observations are assimilated into hydrodynamic models. In this study we performed direct, anomaly, and normalized value assimilation experiments to investigate the capacity of DA to improve river discharge within the current limitations of hydrodynamic modeling. We performed hydrological DA using a physically-based empirical localization method applied to the Amazon Basin. We used satellite altimetry data from ENVISAT, Jason 1, and Jason 2. Direct DA was the baseline assimilation method and was subject to errors due to biases in the simulated WSE. To overcome these errors, we used anomaly DA as an alternative to direct DA. We found that the modeled and observed WSE distributions differed considerably (e.g., differences in amplitude, seasonal flow variation, and a skewed distribution due to limitations of the hydrodynamic models). Therefore, normalized value DA was performed to improve discharge estimation. River discharge estimates were improved at 24 %, 38 %, and 62 % of stream gauges in the direct, anomaly, and normalized value assimilations relative to simulations without DA. Normalized value assimilation performed best for estimating river discharge given the current limitations of hydrodynamic models. Most gauges within the river reaches covered by satellite observations accurately estimated river discharge, with Nash-Sutcliffe efficiency (NSE) > 0.6. The amplitudes of WSE variation were improved in the normalized DA experiment. Furthermore, in the Amazon Basin, normalized assimilation (median NSE = 0.50) improved river discharge estimation compared to open-loop simulation with the global hydrodynamic model (median NSE = 0.42). River discharge estimation using direct DA methods was improved by 7 % with calibration of river bathymetry based on NSE. The direct DA approach outperformed the other DA approaches when runoff was considerably biased, but anomaly DA performed best when the river bathymetry was erroneous. The uncertainties in hydrodynamic modeling (e.g., river bottom elevation, river width, simplified floodplain dynamics, and the rectangular cross-section assumption) should be improved to fully realize the advantages of river discharge DA through the assimilation of satellite altimetry. This study contributes to the development of a global river discharge reanalysis product that is consistent spatially and temporally.

Surface observations can provide crucial information for NWP models. If not assimilated carefully, however, they can degrade forecast accuracy, especially in complex terrains like the Alps. The horizontal and vertical covariances of climatological background error covariances used in the three-dimensional variational (3DVar) data assimilation (DA) method can produce unrealistic increments over sloped terrain. For instance, an observation from a valley station can still generate increments at the mountaintop, even though the valley observation may not accurately represent the mountaintop's weather conditions. We used a hybrid three-dimensional ensemble variational (Hybrid-3DEnVar) DA method to address this issue, incorporating a 50-member convection-permitting ensemble. This method was recently tested in Geosphere Austria's convective scale limited-area NWP model AROME at a 2.5 km horizontal resolution. We assimilated 2-meter temperature, 2-meter relative humidity, geopotential, and 10-meter wind components from 680 surface stations, including the Austrian TAWES network and SYNOP observations from neighbouring countries. 400 stations were actively assimilated from the observation dataset, and the rest were used to verify the analysis.  Our results present the effectiveness of this newly tested Hybrid-3DEnVar against GeoSphere Austria's operational 3DVar in assimilating surface observations over complex Alpine terrain.

Challenges for data assimilation – from convective‐scale to climate

This paper proposes a mini‐batch stochastic optimization‐based adaptive localization scheme for computing the “optimal” localization radius in data assimilation (DA) applications. After constructing a cost function of the localization radius by estimating forecast and observation error statistics, a mini‐batch stochastic gradient descent method with a novel sampling strategy is proposed to minimize the cost function. The proposed stochastic optimization algorithm is further incorporated into the DA method NLS‐i4DVar (the nonlinear least squares integral correcting four‐dimensional variational DA method), which was developed by the authors in Tian et al. (2021, https://doi.org/10.1029/2021EA001767). It is utilized to compute the “optimal” covariance localization radii adaptively and flow‐dependently inside NLS‐i4DVar. The computational cost of NLS‐i4DVar with the proposed adaptive localization scheme only increases slightly due to the use of the mini‐batch stochastic optimization algorithm. Numerical experimental results using the shallow‐water equations demonstrate that NLS‐i4DVar with the proposed adaptive localization scheme shows substantial performance improvement over the standard NLS‐i4DVar method.

In this study, the four‐dimensional variational data assimilation (4D‐Var) method in the Weather Research and Forecasting model is applied to directly assimilate hourly precipitation data to predict an extreme rainstorm process in Henan Province, China. Three simplified microphysics schemes available in 4D‐Var are assessed first, revealing that the new regularized WSM6 scheme performed relatively better in precipitation prediction. Meanwhile, precipitation data assimilation (DA) utilizing the China Meteorological Administration Land Data Assimilation System (CLDAS) V2.0 precipitation reanalysis product is evaluated against the experiments with conventional observations in DA and no assimilation. Results demonstrates that it seems that DA with precipitations is able to enhance the accuracy of precipitation forecasts. In addition, it is well known that one of the challenges in convective‐scale DA is to extract small‐scale information from the observations while maintaining the large‐scale balance and mitigating the growth and propagation of large‐scale errors. Therefore, the large‐scale analysis constraint (LSAC) is further introduced to improve precipitation forecasting. Results indicate that LSAC could effectively adjust large‐scale information, including temperature, humidity, and dynamic conditions, thereby improving the precipitation forecasting skills to some extent.

<p>        While complete atmospheric chemical transport models have been developed to understanding the complex interactions of atmospheric chemistry and physics, there are large uncertainties in numerical approaches. Data assimilation is an efficient method to improve model forecast of aerosols with optimized initial conditions. We have developed a new framework for assimilating surface fine particulate matter (PM<sub>2.5</sub>) observations in coupled Weather Research and Forecasting (WRF) model and Community Multiscale Air Quality (CMAQ) model, based on nonlinear least squares four-dimensional variational (NLS-4DVar) data assimilation method. The NLS-4DVar approach, which does not require the tangent and adjoint models, has been extensive used in meteorological and environmental areas due to the low computational complexity. Two parallel experiments were designed in the observing system simulation experiments (OSSEs) to evaluate the effectiveness of this system. Hourly PM2.5 observations over China be assimilated in WRF-CMAQ model with 6-h assimilation window, while the background state without data assimilation is conducted as control experiment. The results show that the assimilation significantly reduced the uncertainties of initial conditions (ICs) for WRF-CMAQ model and leads to better forecast. The newly developed PM<sub>2.5</sub> data assimilation system can improve PM<sub>2.5</sub> prediction effectively and easily. In the future, we expect emission to be optimized together with concentrations, and integrate meteorological assimilation into aerosol assimilation system.</p>

Since the publication of the first edition of this highly regarded textbook, the value of data assimilation has become widely recognized across the Earth sciences and beyond. Data assimilation methods are now being applied to many areas of prediction and forecasting, including extreme weather events, wildfires, infectious disease epidemics, and economic modeling. This second edition provides a broad introduction to applications across the Earth systems and coupled Earth–human systems, with an expanded range of topics covering the latest developments of variational, ensemble, and hybrid data assimilation methods. New toy models and intermediate-complexity atmospheric general circulation models provide hands-on engagement with key concepts in numerical weather prediction, data assimilation, and predictability. The inclusion of computational projects, exercises, lecture notes, teaching slides, and sample exams makes this textbook an indispensable and practical resource for advanced undergraduate and graduate students, researchers, and practitioners who work in weather forecasting and climate prediction.

Using radar observations, the performances of the ensemble square root filter (EnSRF) and an indirect three-dimensional variational (3DVar) data assimilation method were compared for a mesoscale convective system (MCS) that occurred in the Front Range of the Rocky Mountains, Colorado (USA). The results showed that the root mean square innovations (RMSIs) of EnSRF were lower than 3DVar for radar reflectivity and radial velocity and that the spread of EnSRF was generally consistent with its RMSIs. EnSRF substantially improved the analysis of the MCS compared with an experiment without radar data assimilation, and it produced a slight but noticeable improvement over 3DVar in terms of both coverage and intensity. Forecast results initiated from the final analysis revealed that EnSRF generally produced the best prediction of the MCS, with improved quantitative reflectivity and precipitation forecast skills. EnSRF also demonstrated better performance than 3DVar in the prediction of neighborhood probability for reflectivity at thresholds of 20 and 35 dBZ, which better matched the observed radar reflectivity in terms of both shape and extension. Additionally, the humidity, temperature, and wind fields were also improved by EnSRF; the largest error reduction was found in the water vapor field near the surface and at upper levels.

Ensuring a reliable water supply in the face of changing conditions and growing demand is a critical global challenge. Water distribution networks (WDNs) are essential infrastructure, but traditional modeling methods based on historical data often struggle to adapt in real-time and integrate new information. In order to lower model errors in WDN models, this study explores the use of a Data Assimilation (DA) method that makes use of the Ensemble Kalman Filter. The study explores the effectiveness of the DA method, and a novel Greedy Algorithm (GA) for optimizing the location of sensors. The study shows that the DA method improves the WDN model and the GA is able to successfully determine optimal sensor locations. It was observed that increasing the number of sensors in the WDN increased the effectiveness of the DA method. This study highlights the potential of data assimilation to improve WDN modeling. Water utilities can gain from more precise predictions, greater system performance, and efficient decision-making for the management and maintenance of water distribution networks by allowing models to adapt and improve dynamically

This Special Issue presents efficient formulations and implementations of sequential and variational data assimilation methods. The methods address three important issues in the context of operational data assimilation: efficient implementation of localization methods, sampling methods for approaching posterior ensembles under non-linear model errors, and adjoint-free formulations of four dimensional variational methods.

Coupled climate models integrate atmospheric, oceanic, and land submodels, while the uncertainty of model parameters from different parameterization schemes or empirically derived parameters inevitably introduces systematic biases. Coupled parameter optimization (CPO) can reduce these biases to improve weather forecast and climate prediction, but must address strong nonlinearities inherent in coupled models. The analytical four-dimensional ensemble variational (A-4DEnVar) data assimilation method retains the nonlinear processing capability of the four-dimensional variational (4D-Var) data assimilation method but gets rid of the dependence on the adjoint model. In this study, a novel dynamic independent point (DIP) scheme is introduced to the improved A-4DEnVar, which reduces computational dimensionality and further explores a broader parameter space of dimensionality reduction through the outer loop. Based on the improved A-4DEnVar, a series of geographic-dependent CPO experiments with an idealized 2D coupled model are carried out. Results show that A-4DEnVar accurately captures the geographical characteristics of parameters and effectively optimizes cross-component parameters despite strong nonlinearity. Additionally, the DIP scheme presents significant advantages compared to the static independent point scheme, especially with fewer independent points. This work is offering a new perspective for parameter optimization in coupled general circulation models used for climate estimation and prediction.

This study investigates the impacts of assimilating the Microwave Humidity Sounder II (MWHS-2) radiance data carried on the FY-3D and FY-3E satellites on the analyses and forecasts of Typhoon Muifa in 2022 under clear-sky conditions. Data assimilation experiments are conducted using the Weather Research and Forecasting (WRF) model coupled with the Three-Dimensional Variational (3D-Var) Data Assimilation method to compare the different behaviors of FY-3D and FY-3E radiances. Additionally, the data assimilation strategies are assessed in terms of the sequence of applying the conventional and MWHS-2 radiance data. The results show that assimilating MWHS-2 data is able to enhance the dynamic and thermal structures of the typhoon system. The experiment with FY-3E MWHS-2 assimilated demonstrated superior performance in terms of simulating the typhoon’s structure and providing a prediction of the typhoon’s intensity and track than the experiment with FY-3D MWHS-2 did. The two-step assimilation strategy that assimilates conventional observations before the radiance data has improved the track and intensity forecasts at certain times, particularly with the FY-3E MWHS-2 radiance. It appears that large-scale atmospheric conditions are more refined by initially assimilating the Global Telecommunication System (GTS) data, with subsequent satellite data assimilation further adjusting the model state. This strategy has also confirmed improvements in precipitation prediction as it enhances the dynamic and thermal structures of the typhoon system.

The mainstream data assimilation system in operation will still employ four-dimensional variational data assimilation(4DVAR) method in a long time of future. We develop a new 4DVAR system, i.e., YH4DVAR, using global spectral model as a constraint to impose a dynamic balance on the assimilation. The cost function of YH4DVAR consists of four terms: background, observations, digital filter, and bias correction, respectively. YH4DVAR employs the wavelet background error convariance, the multi-resolution incremental formulation, the tangent linear and adjoint models for dynamics core and physical processes, and ATOVS radiance data assimilation. Simultaneous spatial and spectral variations of horizontal and vertical covariance are achieved by dividing the control vector into several parts, each of which corresponds to a band of total spherical wavenumbers. Using NWP consisting of YH4DVAR and global spectral model, the anomaly correlation with the verifying analysis for geopotential height 8-day forecast on the 500 hPa isobaric surface at 12-month mean is above 60%. The formulation and implementation of YH4VAR are described in detail in this paper.

In different studies, land surface temperature (LST) observations have been assimilated into the variational data assimilation (VDA) approaches to estimate turbulent heat fluxes. The VDA methods yield accurate turbulent heat fluxes, but they need an adjoint model, which is difficult to derive and code. They also cannot directly calculate the uncertainty of their estimates. To overcome the abovementioned drawbacks, this study assimilates LST data from Geostationary Operational Environmental Satellite into the ensemble Kalman smoother (EnKS) data assimilation system to estimate turbulent heat fluxes. EnKS does not need to derive the adjoint term and directly generates statistical information on the accuracy of its predictions. It uses the heat diffusion equation to simulate LST. EnKS with the state augmentation approach finds the optimal values for the unknown parameters (i.e., evaporative fraction and neutral bulk heat transfer coefficient, CHN) by minimizing the misfit between LST observations from Geostationary Operational Environmental Satellite and LST estimations from the heat diffusion equation. The augmented EnKS scheme is tested over six Ameriflux sites with a wide range of hydrological and vegetative conditions. The results show that EnKS can predict not only the model parameters and turbulent heat fluxes but also their uncertainties over a variety of land surface conditions. Compared to the variational method, EnKS yields suboptimal turbulent heat fluxes. However, suboptimality of EnKS is small, and its results are comparable to those of the VDA method. Overall, EnKS is a feasible and reliable method for estimation of turbulent heat fluxes.