Using data assimilation to study extratropical Northern Hemisphere climate over the last millennium

Quantifying continental-scale river discharge is essential to understanding the terrestrial water cycle but is susceptible to errors caused by a lack of observations and the limitations of hydrodynamic modeling. Data assimilation (DA) methods are increasingly used to estimate river discharge in combination with emerging river-related remote sensing products (e.g., water surface elevation [WSE], water surface slope, river width, and flood extent). However, directly comparing simulated WSE to satellite altimetry data remains challenging (e.g., because of large biases between simulations and observations or uncertainties in parameters), and large errors can be introduced when satellite observations are assimilated into hydrodynamic models. In this study we performed direct, anomaly, and normalized value assimilation experiments to investigate the capacity of DA to improve river discharge within the current limitations of hydrodynamic modeling. We performed hydrological DA using a physically-based empirical localization method applied to the Amazon Basin. We used satellite altimetry data from ENVISAT, Jason 1, and Jason 2. Direct DA was the baseline assimilation method and was subject to errors due to biases in the simulated WSE. To overcome these errors, we used anomaly DA as an alternative to direct DA. We found that the modeled and observed WSE distributions differed considerably (e.g., differences in amplitude, seasonal flow variation, and a skewed distribution due to limitations of the hydrodynamic models). Therefore, normalized value DA was performed to improve discharge estimation. River discharge estimates were improved at 24 %, 38 %, and 62 % of stream gauges in the direct, anomaly, and normalized value assimilations relative to simulations without DA. Normalized value assimilation performed best for estimating river discharge given the current limitations of hydrodynamic models. Most gauges within the river reaches covered by satellite observations accurately estimated river discharge, with Nash-Sutcliffe efficiency (NSE) > 0.6. The amplitudes of WSE variation were improved in the normalized DA experiment. Furthermore, in the Amazon Basin, normalized assimilation (median NSE = 0.50) improved river discharge estimation compared to open-loop simulation with the global hydrodynamic model (median NSE = 0.42). River discharge estimation using direct DA methods was improved by 7 % with calibration of river bathymetry based on NSE. The direct DA approach outperformed the other DA approaches when runoff was considerably biased, but anomaly DA performed best when the river bathymetry was erroneous. The uncertainties in hydrodynamic modeling (e.g., river bottom elevation, river width, simplified floodplain dynamics, and the rectangular cross-section assumption) should be improved to fully realize the advantages of river discharge DA through the assimilation of satellite altimetry. This study contributes to the development of a global river discharge reanalysis product that is consistent spatially and temporally.

Ensemble simulations have been performed with a climate model constrained to follow temperature histories obtained from a recent compilation of 56 well‐calibrated surface temperature proxy records, using a new data assimilation technique. First, we demonstrate that the data assimilation technique provides a faithful representation in the Northern Hemisphere of the signal recorded by the climate proxies at both the regional and gridbox scales. Second, by varying the external forcing, the parameters of the data assimilation method, and the parameters controlling the equilibrium climate sensitivity of the climate model, we demonstrate that the uncertainty in model results is much lower in simulations using data assimilation than in those without it. This observation implies that the data assimilation, using a set of 56 proxies, is providing an efficient and robust constraint on the simulated climate variability over the past centuries. At the hemispheric and continental scales, the model reconstructions using data assimilation are in good agreement with both the instrumental record of the past 150 years and reconstructions of climate in past centuries derived from the application of traditional statistical approaches to networks of proxy data. This increases the confidence in both the data assimilation and traditional statistical approaches. Our data assimilation method, however, is unable to provide a reliable reconstruction over the North Atlantic Ocean, which we attribute to the paucity of proxy data in that region.

Ensuring a reliable water supply in the face of changing conditions and growing demand is a critical global challenge. Water distribution networks (WDNs) are essential infrastructure, but traditional modeling methods based on historical data often struggle to adapt in real-time and integrate new information. In order to lower model errors in WDN models, this study explores the use of a Data Assimilation (DA) method that makes use of the Ensemble Kalman Filter. The study explores the effectiveness of the DA method, and a novel Greedy Algorithm (GA) for optimizing the location of sensors. The study shows that the DA method improves the WDN model and the GA is able to successfully determine optimal sensor locations. It was observed that increasing the number of sensors in the WDN increased the effectiveness of the DA method. This study highlights the potential of data assimilation to improve WDN modeling. Water utilities can gain from more precise predictions, greater system performance, and efficient decision-making for the management and maintenance of water distribution networks by allowing models to adapt and improve dynamically

Challenges for data assimilation – from convective‐scale to climate

A crucial branch of science called climate modeling uses mathematical equations and computer simulations to study and forecast the Earth's climate sys- tem. The main elements of climate modeling, such as general circulation models (GCMs), data assimilation methods, and numerical formulations, are outlined in this paper. GCMs, which include grid point and spectral models, are effective instruments for examining the behavior of the climate. Four-Dimensional Data Assimilation (4D-Var) is one example of a data assimilation technique that in- corporates observational data into models to improve their correctness. Numeri cal methods, ocean dynamics, heat transport, radiative transfer, and atmospheric dynamics are all included in numerical formulations. The simulation of different climate processes is possible because to these mathematical representations. Fur thermore, the detection of precipitation patterns within climate modeling is using machine learning techniques like Random Forest more frequently. This paper highlights the importance of high-performance computing (HPC) in climate modeling, boosting efficiency and simulations, in the context of research technique. Advanced data assimilation and validation techniques are also examined, as well as the influence of high-resolution modeling on small-scale climatic processes. On HPC platforms, accessibility to climate modeling is addressed. It is shown how climate modeling crosses physics, mathematics, computer science, and engineering to be interdisciplinary. A comprehensive understanding of the Earth's intricate climate system gains from the integration of all its parts, from atmospheric dynamics to data assimilation. We explore the consequences of these research approaches, their contribution to enhancing climate prediction models, and the influence of various factors on climatic variables in the debate. Climate modeling becomes an essential tool for studying precipitation patterns and climate change, ultimately improving our comprehension of the complex cli- mate system on Earth.

Since the publication of the first edition of this highly regarded textbook, the value of data assimilation has become widely recognized across the Earth sciences and beyond. Data assimilation methods are now being applied to many areas of prediction and forecasting, including extreme weather events, wildfires, infectious disease epidemics, and economic modeling. This second edition provides a broad introduction to applications across the Earth systems and coupled Earth–human systems, with an expanded range of topics covering the latest developments of variational, ensemble, and hybrid data assimilation methods. New toy models and intermediate-complexity atmospheric general circulation models provide hands-on engagement with key concepts in numerical weather prediction, data assimilation, and predictability. The inclusion of computational projects, exercises, lecture notes, teaching slides, and sample exams makes this textbook an indispensable and practical resource for advanced undergraduate and graduate students, researchers, and practitioners who work in weather forecasting and climate prediction.

SummaryEstimating river discharge from in situ and/or remote sensing data is a key issue for evaluation of water balance at local and global scales and for water management. Variational data assimilation (DA) is a powerful approach used in operational weather and ocean forecasting, which can also be used in this context. A distinctive feature of the river discharge estimation problem is the likely presence of significant uncertainty in principal parameters of a hydraulic model, such as bathymetry and friction, which have to be included into the control vector alongside the discharge. However, the conventional variational DA method being used for solving such extended problems often fails. This happens because the control vector iterates (i.e., approximations arising in the course of minimization) result into hydraulic states not supported by the model. In this paper, we suggest a novel version of the variational DA method specially designed for solving estimation‐under‐uncertainty problems, which is based on the ideas of iterative regularization.The method is implemented with SIC2, which is a full Saint‐Venant based 1D‐network model. The SIC2 software is widely used by research, consultant and industrial communities for modeling river, irrigation canal, and drainage network behavior. The adjoint model required for variational DA is obtained by means of automatic differentiation. This is likely to be the first stable consistent adjoint of the 1D‐network model of a commercial status in existence.The DA problems considered in this paper are offtake/tributary estimation under uncertainty in the cross‐device parameters and inflow discharge estimation under uncertainty in the bathymetry defining parameters and the friction coefficient. Numerical tests have been designed to understand identifiability of discharge given uncertainty in bathymetry and friction. The developed methodology, and software seems useful in the context of the future Surface Water and Ocean Topography satellite mission. Copyright © 2016 John Wiley & Sons, Ltd.

Reconstructing spatiotemporal systems from sparse observations remains a long-standing challenge in several domains, including geoscience, air pollution and fluid dynamics. While various data assimilation (DA) and machine learning (ML) methods have shown potential, they still face significant challenges (see [1]): The computational burden of conventional DA algorithms (including error covariance specification), particularly for multivariate, high-dimensional systems. Sparse and movable sensor placement, which makes conventional ML models (typically requiring fixed and regularly distributed input data) cumbersome. The ill-defined nature of the sparse reconstruction problem, which poses significant risks of overfitting. We will present our recent works aimed at addressing these challenges. More specifically, we developed latent DA algorithms [2] to reduce the computational burden of variational DA methods. These algorithms demonstrate great potential in efficiently assimilating sparse observations within a reduced-order latent space constructed by neural networks, thanks to the TorchDA library [3]. The latter enables GPU implementation of mainstream data assimilation methods and supports non-explicit state-observation transformation functions, provided they can be learned by a neural network.We have also employed advanced deep learning techniques, including Voronoi-tessellation CNNs [4] and Vision Transformer-based autoencoders [5], to learn mappings from sparse observations to the complete physical space. These approaches effectively address challenges such as movable sensor placements and varying sensor numbers. Their integration with DA algorithms has also been evaluated.Finally, our recent work explores [6] the utility of generative AI techniques, particularly denoising diffusion models, for field reconstruction from sparse observations. Generative AI methods offer two main advantages: first, they produce a sample from a probability distribution rather than predicting the mean as a fixed output, which can help mitigate overfitting caused by the illy-defined problem. Second, they inherently function as ensemble predictors by generating several samples, facilitating uncertainty quantification, which is essential in data assimilation. The numerical results tested on cases ranging from fluid dynamics benchmarks to semi-operational air pollution simulations will also be discussed.[1] Cheng, S., Quilodrán-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., Xiao, D., Janjic, T., Ding, W., Guo, Y., Carrassi, A., Bocquet, M. and Arcucci, R, 2023. Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review. IEEE/CAA Journal of Automatica Sinica[2] Cheng, S., Chen, J., Anastasiou, C., Angeli, P., Matar, O.K., Guo, Y.K., Pain, C.C. and Arcucci, R., 2023. Generalised latent assimilation in heterogeneous reduced spaces with machine learning surrogate models. Journal of Scientific Computing[3] Cheng, S., Min, J., Liu, C. and Arcucci, R., 2025. TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions. Computer Physics Communications[4] Cheng, S., Liu, C., Guo, Y. and Arcucci, R., 2024. Efficient deep data assimilation with sparse observations and time-varying sensors. Journal of Computational Physics[5] Fan, H., Cheng, S., de Nazelle, A.J. and Arcucci, R., 2024. ViTAE-SL: a vision transformer-based autoencoder and spatial interpolation learner for field reconstruction. Computer Physics Communications [6] Zhuang, Y., Cheng, S. and Duraisamy, K., 2025. Spatially-aware diffusion models with cross-attention for global field reconstruction with sparse observations. Computer Methods in Applied Mechanics and Engineering

A latent space method with maximum entropy deep reinforcement learning for data assimilation

Abstract. Data assimilation (DA) is an emerging topic in palaeoclimatology and one of the key challenges in this field. Assimilating proxy-based continental mean temperature reconstructions into the MPI-ESM model showed a lack of information propagation to small spatial scales . Here, we investigate whether this lack of regional skill is due to the methodology or to errors in the assimilated reconstructions. Error separation is fundamental, as it can lead to improvements in DA methods. We address the question by performing a new set of simulations, using two different sets of target data; the proxy-based PAGES 2K reconstructions (DA-P scheme), and the HadCRUT3v instrumental observations (DA-I scheme). Again, we employ ensemble-member selection DA using the MPI-ESM model, and assimilate Northern Hemisphere (NH) continental mean temperatures; the simulated period is 1850–1949 AD. Both DA schemes follow the large-scale target and observed climate variations well, but the assimilation of instrumental data improves the performance. This improvement cannot be seen for Asia, where the limited instrumental coverage leads to errors in the target data and low skill for the DA-I scheme. No skill on small spatial scales is found for either of the two DA schemes, demonstrating that errors in the assimilated data are not the main reason for the unrealistic representation of the regional temperature variability in Europe and the NH. It can thus be concluded that assimilating continental mean temperatures is not ideal for providing skill on small spatial scales.

Data assimilation (DA) is a statistical approach used to estimate the states of physical systems by integrating prior model predictions (background states xb​) with observational data (y). This integration produces an accurate estimate, called analysis states (​xa), by sampling or maximizing the posterior likelihood p(xxb, y). In weather forecasting, background states are generated by imperfect models, and the likelihood p(xxb) is often unknown. Observations, sourced from diverse instruments, are mapped to model space using observation operators (H). Effective DA algorithms must accurately estimate p(xxb) while accommodating various observation operators, including those involving sparse, noisy or irregular data.Traditional DA methods, such as variational assimilation, assume that the background error (​x - xb) follows a Gaussian distribution independent of xb​. This allows explicit computation of p(xxb, y) and optimization via techniques like gradient descent. While robust to various observation operators, these methods depend heavily on expert knowledge to construct error correlations and are limited by their Gaussian assumptions.Generative neural networks, particularly diffusion models, have emerged as alternatives for modeling p(xxb). Notable examples include SDA and DiffDA, which use diffusion models to learn background distributions. SDA incorporates observations via diffusion posterior sampling, while DiffDA employs the repaint technique. These approaches improve on traditional methods by capturing more complex distributions but often struggle with sparse, irregular observations. For instance, DiffDA assumes grid-aligned data, while SDA relies on assumptions that can reduce accuracy in real-world scenarios.In this research, we aim to develop a neural network-based data assimilation algorithm that not only captures the non-Gaussian characteristics of the conditional background distribution for enhanced accuracy but also effectively assimilates data under real-world observations (sparse, noisy and outside of the grid). We introduce VAE-Var, a novel data assimilation algorithm in which a variational autoencoder is first employed to learn the conditional background distribution and then the decoder component is utilized to construct a variational cost function, which, when optimized, yields the analysis states.Key advantages of VAE-Var include:This algorithm inherits the framework of traditional variational assimilation by explicitly modeling the posterior probability function p(xxb, y) and maximizing it to derive the analysis states. As a result, compared to other neural network data assimilation methods such as SDA and DiffDA, VAE-Var can better handle different types of observation operators, particularly irregular observations that do not fall on the grid points of the physical field. Unlike traditional variational assimilation algorithms, VAE-Var alleviates the dependence on expert knowledge for constructing the conditional background distribution, enabling the model to effectively capture non-Gaussian structures. This makes VAE-Var perform better in sparse observational settings. Experiments with the FengWu weather forecasting system at 0.25° resolution show that VAE-Var achieves higher accuracy than DiffDA and traditional algorithms (interpolation and 3DVar) in sparse observational settings. When integrated with FengWu, VAE-Var reliably assimilates real-world GDAS prepbufr observations over a one-year period.

The beginning of the 21st century is marked by a rapid growth of land surface satellite data and model sophistication. This offers new opportunities to estimate multiple components of the water cycle via satellite-based land data assimilation (DA) across multiple scales. By resolving more processes in land surface models and by coupling the land, the atmosphere, and other Earth system compartments, the observed information can be propagated to constrain additional unobserved variables. Furthermore, access to more satellite observations enables the direct constraint of more and more components of the water cycle that are of interest to end users. However, the finer level of detail in models and data is also often accompanied by an increase in dimensions, with more state variables, parameters, or boundary conditions to estimate, and more observations to assimilate. This requires advanced DA methods and efficient solutions. One solution is to target specific observations for assimilation based on a sensitivity study or coupling strength analysis, because not all observations are equally effective in improving subsequent forecasts of hydrological variables, weather, agricultural production, or hazards through DA. This paper offers a perspective on current and future land DA development, and suggestions to optimally exploit advances in observing and modeling systems.

Data assimilation is a method that combines observations (ie, real world data) of a state of a system with model output for that system in order to improve the estimate of the state of the system and thereby the model output. The model is usually represented by a discretized partial differential equation. The data assimilation problem can be formulated as a large scale Bayesian inverse problem. Based on this interpretation we will derive the most important variational and sequential data assimilation approaches, in particular three‐dimensional and four‐dimensional variational data assimilation (3D‐Var and 4D‐Var) and the Kalman filter. We will then consider more advanced methods which are extensions of the Kalman filter and variational data assimilation and pay particular attention to their advantages and disadvantages. The data assimilation problem usually results in a very large optimization problem and/or a very large linear system to solve (due to inclusion of time and space dimensions). Therefore, the second part of this article aims to review advances and challenges, in particular from the numerical linear algebra perspective, within the various data assimilation approaches.

Six-hourly three-dimensional ensemble variational (3DEnVar) (6H-3DEnVar) data assimilation (DA) assumes constant background error covariance (BEC) during a six-hour DA window and is, therefore, unable to account for temporal evolution of the BEC. This study evaluates the one-hourly 3DEnVar (1H-3DEnVar) and six-hourly 4DEnVar (6H-4DEnVar) DA methods for the analyses and forecasts of hurricanes with rapidly evolving BEC. Both methods account for evolving BEC in a hybrid EnVar DA system. In order to compare these methods, experiments are conducted by assimilating inner core Tail Doppler Radar (TDR) wind for Hurricane Edouard (2014) and by running the Hurricane Weather Research and Forecasting (HWRF) model. In most metrics, 1H-3DEnVar and 6H-4DEnVar analyses and forecasts verify better than 6H-3DEnVar. 6H-4DEnVar produces better thermodynamic analyses than 1H-3DEnVar. Radar reflectivity shows that 1H-3DEnVar produces better structure forecasts. For the first 24–48 h of the intensity forecast, 6H-4DEnVar forecast performs better than 1H-3DEnVar verified against the best track. Degraded 1H-3DEnVar forecasts are found to be associated with background storm center location error as a result of underdispersive ensemble storm center spread. Removing location error in the background improves intensity forecasts of 1H-3DEnVar.

Measurements from the Ozone Monitoring Instrument (OMI) and Microwave Limb Sounder (MLS), both on board the Aura spacecraft, have been used to produce daily global maps of column and profile ozone since August 2004. Here we compare and evaluate three strategies to obtain daily maps of tropospheric and stratospheric ozone from OMI and MLS measurements: trajectory mapping, direct profile retrieval, and data assimilation. Evaluation is based on an assessment that includes validation using ozonesondes and comparisons with the Global Modeling Initiative (GMI) chemical transport model. We investigate applications of the three ozone data products from near‐decadal and interannual time scales to day‐to‐day case studies. Interannual changes in zonal mean tropospheric ozone from all of the products in any latitude range are of the order 1–2 Dobson units while changes (increases) over the 8 year Aura record investigated vary by 2–4 Dobson units. It is demonstrated that all of the ozone products can measure and monitor exceptional tropospheric ozone events including major forest fire and pollution transport events. Stratospheric ozone during the Aura record has several anomalous interannual events including split stratospheric warmings in the Northern Hemisphere extratropics that are well captured using the data assimilation ozone profile product. Data assimilation with continuous daily global coverage and vertical ozone profile information is the best of the three strategies at generating a global tropospheric and stratospheric ozone product for science applications.