Reply on RC3

We apply information theory within an ensemble‐based data assimilation approach and define information matrix in ensemble subspace. The information matrix in ensemble subspace employs a flow‐dependent forecast error covariance and it is of relatively small dimensions (equal to the ensemble size). The information matrix in ensemble subspace can be directly linked to the information matrix typically used in non‐ensemble‐based data assimilation methods, such as the Kalman Filter (KF) and the three‐dimensional variational (3D‐Var) methods, which provides a framework for consistent comparisons of information measures between different data assimilation methods.We evaluate information measures, such as degrees of freedom for signal, within the Maximum Likelihood Ensemble Filter (MLEF) data assimilation approach and compare them with those obtained using the KF approach and the 3D‐Var approach. We assimilate model‐simulated observations and use the Goddard Earth Observing System Single Column Model (GEOS‐5 SCM) as a dynamical forecast model.The experimental results demonstrate that the proposed framework is useful for comparing information measures obtained in different data assimilation approaches. These comparisons indicate that using a flow‐dependent forecast error covariance matrix (e.g. as in the KF and the MLEF experiments) is fundamentally important for adequately describing prior knowledge about the true model state when calculating information measures of assimilated observations. We also demonstrate that data assimilation results obtained using the KF and the MLEF approach (when ensemble size is larger than 10 ensemble members) are superior to the results of the 3D‐Var approach. Copyright © 2007 Royal Meteorological Society

SummaryEstimating river discharge from in situ and/or remote sensing data is a key issue for evaluation of water balance at local and global scales and for water management. Variational data assimilation (DA) is a powerful approach used in operational weather and ocean forecasting, which can also be used in this context. A distinctive feature of the river discharge estimation problem is the likely presence of significant uncertainty in principal parameters of a hydraulic model, such as bathymetry and friction, which have to be included into the control vector alongside the discharge. However, the conventional variational DA method being used for solving such extended problems often fails. This happens because the control vector iterates (i.e., approximations arising in the course of minimization) result into hydraulic states not supported by the model. In this paper, we suggest a novel version of the variational DA method specially designed for solving estimation‐under‐uncertainty problems, which is based on the ideas of iterative regularization.The method is implemented with SIC2, which is a full Saint‐Venant based 1D‐network model. The SIC2 software is widely used by research, consultant and industrial communities for modeling river, irrigation canal, and drainage network behavior. The adjoint model required for variational DA is obtained by means of automatic differentiation. This is likely to be the first stable consistent adjoint of the 1D‐network model of a commercial status in existence.The DA problems considered in this paper are offtake/tributary estimation under uncertainty in the cross‐device parameters and inflow discharge estimation under uncertainty in the bathymetry defining parameters and the friction coefficient. Numerical tests have been designed to understand identifiability of discharge given uncertainty in bathymetry and friction. The developed methodology, and software seems useful in the context of the future Surface Water and Ocean Topography satellite mission. Copyright © 2016 John Wiley & Sons, Ltd.

ABSTRACTAn original data assimilation (DA) scheme with a general dynamics model is considered. It is shown that this scheme can be approximated by the stochastic diffusion process. The sufficient conditions to provide this approximation are formulated. Based on this algorithm a new DA method is developed. The method combines variational and statistical approaches commonly used in DA theory and minimizes the variance of the trajectory of a diffusion process in conjunction with a dynamics numerical model. In this sense the method is optimal in contrast to other DA approaches. The proposed scheme takes the model dynamics into account and in this way it differs from the well-known Kalman filter. Furthermore, the derived DA method can be applied to a very wide field of dynamical systems, for example, gas dynamics, fluid dynamics and other disciplines. However, the current study deals with oceanography and DA in oceanography specifically. Then the method is applied to the HYbrid Coordinate Ocean Model and assimilates satellite sea level anomaly data from the Archiving, Validating and Interpolating Satellite Oceanography Data over the Atlantic Ocean to correct the model state. Several numerical experiments have been performed. The experiments show that the method substantially changes the synoptic and mesoscale structure of ocean dynamics. Also, the distribution of the obtained result is estimated through the solution of the Fokker–Planck–Kolmogorov equation.

Ensuring a reliable water supply in the face of changing conditions and growing demand is a critical global challenge. Water distribution networks (WDNs) are essential infrastructure, but traditional modeling methods based on historical data often struggle to adapt in real-time and integrate new information. In order to lower model errors in WDN models, this study explores the use of a Data Assimilation (DA) method that makes use of the Ensemble Kalman Filter. The study explores the effectiveness of the DA method, and a novel Greedy Algorithm (GA) for optimizing the location of sensors. The study shows that the DA method improves the WDN model and the GA is able to successfully determine optimal sensor locations. It was observed that increasing the number of sensors in the WDN increased the effectiveness of the DA method. This study highlights the potential of data assimilation to improve WDN modeling. Water utilities can gain from more precise predictions, greater system performance, and efficient decision-making for the management and maintenance of water distribution networks by allowing models to adapt and improve dynamically

Challenges for data assimilation – from convective‐scale to climate

In this study, we present a probabilistic urban inundation modeling framework that combines high-resolution process-based modeling with observed information via ensemble data assimilation (DA). We investigate the impact of multivariate flood observations on improving urban inundation prediction accuracy through synthetic experiments. The framework leverages diverse flood observations from both the urban surface and sewer system, integrating them into the modeling process using non-Gaussian sequential DA methods, such as particle filtering. The modeling framework employs the H12 model for integrated 1D sewer network and 2D surface inundation analyses, with synthetic experiments conducted in an urban catchment in Osaka, Japan. Prior to implementing DA, a sensitivity analysis is performed to assess the effects of uncertainties in inundation modeling. Major uncertainty components, such as input forcings and storm drain box efficiency, are perturbed to evaluate their influence. The synthetic DA experiments analyze the influence of various types of flood observations, including urban surface inundation depths and sewer water levels, on the posterior distributions of the model ensemble. Additionally, the impact of observation location and density on DA performance is evaluated. This study demonstrates the potential of utilizing diverse flood observations in data assimilation to enhance the accuracy and reliability of urban flood predictions.

Reconstructing spatiotemporal systems from sparse observations remains a long-standing challenge in several domains, including geoscience, air pollution and fluid dynamics. While various data assimilation (DA) and machine learning (ML) methods have shown potential, they still face significant challenges (see [1]): The computational burden of conventional DA algorithms (including error covariance specification), particularly for multivariate, high-dimensional systems. Sparse and movable sensor placement, which makes conventional ML models (typically requiring fixed and regularly distributed input data) cumbersome. The ill-defined nature of the sparse reconstruction problem, which poses significant risks of overfitting. We will present our recent works aimed at addressing these challenges. More specifically, we developed latent DA algorithms [2] to reduce the computational burden of variational DA methods. These algorithms demonstrate great potential in efficiently assimilating sparse observations within a reduced-order latent space constructed by neural networks, thanks to the TorchDA library [3]. The latter enables GPU implementation of mainstream data assimilation methods and supports non-explicit state-observation transformation functions, provided they can be learned by a neural network.We have also employed advanced deep learning techniques, including Voronoi-tessellation CNNs [4] and Vision Transformer-based autoencoders [5], to learn mappings from sparse observations to the complete physical space. These approaches effectively address challenges such as movable sensor placements and varying sensor numbers. Their integration with DA algorithms has also been evaluated.Finally, our recent work explores [6] the utility of generative AI techniques, particularly denoising diffusion models, for field reconstruction from sparse observations. Generative AI methods offer two main advantages: first, they produce a sample from a probability distribution rather than predicting the mean as a fixed output, which can help mitigate overfitting caused by the illy-defined problem. Second, they inherently function as ensemble predictors by generating several samples, facilitating uncertainty quantification, which is essential in data assimilation. The numerical results tested on cases ranging from fluid dynamics benchmarks to semi-operational air pollution simulations will also be discussed.[1] Cheng, S., Quilodrán-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., Xiao, D., Janjic, T., Ding, W., Guo, Y., Carrassi, A., Bocquet, M. and Arcucci, R, 2023. Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review. IEEE/CAA Journal of Automatica Sinica[2] Cheng, S., Chen, J., Anastasiou, C., Angeli, P., Matar, O.K., Guo, Y.K., Pain, C.C. and Arcucci, R., 2023. Generalised latent assimilation in heterogeneous reduced spaces with machine learning surrogate models. Journal of Scientific Computing[3] Cheng, S., Min, J., Liu, C. and Arcucci, R., 2025. TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions. Computer Physics Communications[4] Cheng, S., Liu, C., Guo, Y. and Arcucci, R., 2024. Efficient deep data assimilation with sparse observations and time-varying sensors. Journal of Computational Physics[5] Fan, H., Cheng, S., de Nazelle, A.J. and Arcucci, R., 2024. ViTAE-SL: a vision transformer-based autoencoder and spatial interpolation learner for field reconstruction. Computer Physics Communications [6] Zhuang, Y., Cheng, S. and Duraisamy, K., 2025. Spatially-aware diffusion models with cross-attention for global field reconstruction with sparse observations. Computer Methods in Applied Mechanics and Engineering

Ensemble simulations have been performed with a climate model constrained to follow temperature histories obtained from a recent compilation of 56 well‐calibrated surface temperature proxy records, using a new data assimilation technique. First, we demonstrate that the data assimilation technique provides a faithful representation in the Northern Hemisphere of the signal recorded by the climate proxies at both the regional and gridbox scales. Second, by varying the external forcing, the parameters of the data assimilation method, and the parameters controlling the equilibrium climate sensitivity of the climate model, we demonstrate that the uncertainty in model results is much lower in simulations using data assimilation than in those without it. This observation implies that the data assimilation, using a set of 56 proxies, is providing an efficient and robust constraint on the simulated climate variability over the past centuries. At the hemispheric and continental scales, the model reconstructions using data assimilation are in good agreement with both the instrumental record of the past 150 years and reconstructions of climate in past centuries derived from the application of traditional statistical approaches to networks of proxy data. This increases the confidence in both the data assimilation and traditional statistical approaches. Our data assimilation method, however, is unable to provide a reliable reconstruction over the North Atlantic Ocean, which we attribute to the paucity of proxy data in that region.

The application of a novel modeling and data assimilation approach is presented, demonstrating the impact of quantitatively modeled and optimized cyclic steam candidate and steam volume selection in a mature heavy oil field. Results are reviewed for a cyclic steam operation in the San Joaquin Basin in California, including steam savings, production increases and SOR reduction in excess of 20 percent. The new approach is based on a novel modeling and data assimilation method. This paper focuses on the data assimilation aspect of the workflow, which is fast, quantifies uncertainty and enables simultaneous assimilation of multiple data sources. The approach is a combination of a modified Ensemble Kalman Filter (EnKF) with quadratic programming that enables assimilation of data from thousands of wells and different data sources with a relatively small ensemble. Additionally, since the EnKF is known to underestimate uncertainty, statistical techniques are used to correct the uncertainty estimates to conform with empirical estimates. The predictive capacity of the calibrated models is demonstrated through a statistical back-test process, wherein, the wells are fitted to historical data up to before the last steam job, and the response of the last steam job is predicted and compared to actual production. Once a calibrated model is validated, the models can then be used to predict the performance of future jobs, and thereby identify the best wells to steam, and also optimize steam volumes that minimize SOR and maximize incremental oil production due to the steam job. The above described modeling and optimization workflow has been applied to multiple fields in the San Joaquin valley. For the field chosen for this case study, production comes from the unconsolidated sands of the Pliocene Chanac and Kern River formations with porosity averaging 30%, permeability averaging 1,500-2,000 mD and net thicknesses typically between 50 and 300 feet. Dip is generally monoclinal across the field at approximately 5 – 15 degrees. The reservoir is shallow, with depths ranging from 750 – 2,250 feet. Oil gravity averages 14 – 16° API. Reservoir pressure is well below bubble point and averages 25 – 100 PSI. Data sources assimilated included production and injection rates, wellhead pressures, steam and producer temperatures, temperature profiles, steam ID logs and various petro-physical logs. Results to date indicate that application of this workflow has increased incremental oil production from the steam jobs by over 44% compared to a prior control period. And most interestingly, such an increase in production is achieved while cutting overall steam injection, as the steam volume optimization identifies inefficient use of steam. The most significant new finding is that cyclic steam injection can be quantitatively modeled and optimized rapidly to maximize profit and minimize SOR. This is important, particularly for mature heavy oil fields, in order to maximize recovery amidst varying commodity prices. The novelty of the new model is its combination of speed of data integration (less than a week) and runtime (minutes) with long-term predictive accuracy (years or decades). This is due to the unique modeling and data assimilation methodology.

Against the background of a worldwide decrease in the number of gauging stations, the estimation of river discharge using spaceborne data is crucial for hydrological research, river monitoring, and water resource management. Based on the at-many-stations hydraulic geometry (AMHG) concept, a novel approach is introduced for estimating river discharge using Sentinel-1 time series within an automated workflow. By using a novel decile thresholding method, no a priori knowledge of the AMHG function or proxy is used, as proposed in previous literature. With a relative root mean square error (RRMSE) of 19.5% for the whole period and a RRMSE of 15.8% considering only dry seasons, our method is a significant improvement relative to the optimized AMHG method, achieving 38.5% and 34.5%, respectively. As the novel approach is embedded into an automated workflow, it enables a global application for river discharge estimation using solely remote sensing data. Starting with the mapping of river reaches, which have large differences in river width over the year, continuous river width time series are created using high-resolution and weather-independent SAR imaging. It is applied on a 28 km long section of the Mekong River near Vientiane, Laos, for the period from 2015 to 2018.

In two recent studies, the authors presented a new data assimilation (DA) method for precipitation observations that does not require Gaussianity or linearity assumptions. The method, called localized ensemble mosaic assimilation (LEMA), initializes the new ensemble forecast by relaxing the background ensemble (prior) toward a single analysis composed of different column states taken from the ensemble members with the lowest error in the precipitation forecast. However, a limitation of the LEMA method is that relaxing the background ensemble toward that analysis severely reduces the spread of the ensemble, thus, limiting its usefulness for cycled DA applications. This study presents a new version of LEMA, called localized ensemble mosaic assimilation sequence (LEMAS), suitable for cycled DA operations. LEMAS constructs an ensemble of analysis mosaics using a small group of members closer to the observations instead of only the closest one. The new ensemble forecast is then initialized by recentering the prior ensemble around the mean of the analysis ensemble while scaling the original background perturbations to match the spread of the analysis mosaics. A series of ideal and real DA experiments are used to evaluate the potential of LEMAS for the assimilation of hourly accumulation observations. A comparison of LEMAS with the local ensemble transform Kalman filter (LETKF) using idealized experiments shows that LEMAS produces similar or slightly better forecast quality than the LETKF in temperature, water vapor, winds, and precipitation. Extending this comparison to real DA experiments assimilating Stage-IV precipitation observations shows that both methods produce precipitation forecasts of comparable quality.

<p><span>The Madden-Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability with wide reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason are suboptimal state estimates obtained with standard data assimilation (DA) approaches. Those are typically based on filtering methods with Gaussian approximations and do not consider physical properties that are specifically important for the MJO.</span></p><p><span>In our recent paper (Gleiter et al. 2022), a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO with the Skeleton model. The utilized quadratic programming ensemble (QPEns) algorithm extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows to recover physically more consistent states and to respect possible associated non-Gaussian statistics. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial condition. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that even in cases where the statistical error in some waves remains comparable to the stochastic EnKF during the DA stage, their prediction is remarkably improved when using the initial state resulting from the QPEns.</span></p><p><strong>Gleiter, T., T. Janjic, N. Chen, </strong>2022, <em>Ensemble Kalman Filter based Data Assimilation for Tropical Waves in the MJO Skeleton Model, </em>QJR Meteorol Soc., https://doi.org/10.1002/qj.4245</p>

The Madden–Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability, with wide‐reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason is the suboptimal state estimates obtained with standard data assimilation (DA) approaches. These are typically based on filtering methods with Gaussian approximations and do not take into account physical properties that are important specifically for the MJO. In this article, a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO. The quadratic programming ensemble (QPEns) algorithm utilized extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows us to recover physically more consistent states and to respect possible associated non‐Gaussian statistics. The study is based on identical twin experiments with an adopted nonlinear model for tropical intraseasonal variability. This so‐called skeleton model succeeds in reproducing the main large‐scale features of the MJO and closely related tropical waves, while keeping adequate simplicity for fast experiments on intraseasonal time‐scales. Conservation laws and other crucial physical properties from the model are examined as constraints in the QPEns. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial conditions. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that, even in cases where the statistical error in some waves remains comparable with the stochastic EnKF during the DA stage, their prediction is improved remarkably when using the initial state resulting from the QPEns.

Ensemble data assimilation methods for improving river water quality forecasting accuracy

Surface observations can provide crucial information for NWP models. If not assimilated carefully, however, they can degrade forecast accuracy, especially in complex terrains like the Alps. The horizontal and vertical covariances of climatological background error covariances used in the three-dimensional variational (3DVar) data assimilation (DA) method can produce unrealistic increments over sloped terrain. For instance, an observation from a valley station can still generate increments at the mountaintop, even though the valley observation may not accurately represent the mountaintop's weather conditions. We used a hybrid three-dimensional ensemble variational (Hybrid-3DEnVar) DA method to address this issue, incorporating a 50-member convection-permitting ensemble. This method was recently tested in Geosphere Austria's convective scale limited-area NWP model AROME at a 2.5 km horizontal resolution. We assimilated 2-meter temperature, 2-meter relative humidity, geopotential, and 10-meter wind components from 680 surface stations, including the Austrian TAWES network and SYNOP observations from neighbouring countries. 400 stations were actively assimilated from the observation dataset, and the rest were used to verify the analysis.  Our results present the effectiveness of this newly tested Hybrid-3DEnVar against GeoSphere Austria's operational 3DVar in assimilating surface observations over complex Alpine terrain.