Comment on gmd-2022-69

A particle filter (PF) is an ensemble data assimilation method that does not assume Gaussian error distributions. Recent studies proposed local PFs (LPFs), which use localization as in the ensemble Kalman filter, to apply the PF for high-dimensional dynamics efficiently. Among others, Penny and Miyoshi developed an LPF in the form of the ensemble transform matrix of the Local Ensemble Transform Kalman Filter (LETKF). The LETKF has been widely accepted for various geophysical systems including numerical weather prediction (NWP) models. Therefore, implementing consistently with an existing LETKF code is useful. This study developed a software platform for the LPF and its Gaussian mixture extension (LPFGM) by making slight modifications to the LETKF code with a simplified global climate model known as Simplified Parameterizations, Primitive Equation Dynamics (SPEEDY). A series of idealized twin experiments were accomplished under the ideal model assumption. With large inflation by the relaxation to prior spread, the LPF showed stable filter performance with dense observations but became unstable with sparse observations. The LPFGM showed more accurate and stable performances than the LPF with both dense and sparse observations. In addition to the relaxation parameter, regulating the resampling frequency and the amplitude of Gaussian kernels was important for the LPFGM. With a spatially inhomogeneous observing network, the LPFGM was superior to the LETKF in sparsely observed regions where the background ensemble spread and non-Gaussianity are larger. The SPEEDY-based LETKF, LPF, and LPFGM systems were available as open-source software on Github (https://github.com/skotsuki/speedy-lpf) and can be adapted to various models relatively easily like the LETKF.

Abstract. A local particle filter (LPF) is introduced that outperforms traditional ensemble Kalman filters in highly nonlinear/non-Gaussian scenarios, both in accuracy and computational cost. The standard sampling importance resampling (SIR) particle filter is augmented with an observation-space localization approach, for which an independent analysis is computed locally at each grid point. The deterministic resampling approach of Kitagawa is adapted for application locally and combined with interpolation of the analysis weights to smooth the transition between neighboring points. Gaussian noise is applied with magnitude equal to the local analysis spread to prevent particle degeneracy while maintaining the estimate of the growing dynamical instabilities. The approach is validated against the local ensemble transform Kalman filter (LETKF) using the 40-variable Lorenz-96 (L96) model. The results show that (1) the accuracy of LPF surpasses LETKF as the forecast length increases (thus increasing the degree of nonlinearity), (2) the cost of LPF is significantly lower than LETKF as the ensemble size increases, and (3) LPF prevents filter divergence experienced by LETKF in cases with non-Gaussian observation error distributions.

This paper develops a parallel implementation of the ensemble Kalman filter (EnKF) based on shrinkage covariancematrix estimation. The EnKF is a sequential Monte Carlo method for parameter and state estimation of highly nonlinearmodels. In the context of ensemble based methods, background error correlations are estimated via anensemble of model realizations. In practice, the model dimension (number of model components) is several timeslarger than the ensemble size owing to the computational effort involved in one single model propagation. Thisconstraint yields to spurious correlations (correlation between distant model components in space) in the backgrounderror estimations. Commonly, localization methods are utilized in order to reduce the impact of spuriouscorrelations. For instance, correlations of model components are reduced based on some function of their physicaldistances. Some EnKF implementations are formulated in such manner that, localization methods are implicitlyinvolved in the assimilation of observations. In this context, one of the best ensemble Kalman filter implementationsis the local ensemble transform Kalman filter (LETKF). In the LETKF, each model component is surrounded by alocal box of size r and then, the information contained within the local domain (i.e., observed components) is used toperform the assimilation. However, for sparse observational networks, r can become sufficiently large with respectto the ensemble size and therefore, the local analysis corrections can be impacted by spurious correlations. Wepropose an ensemble Kalman filter implementation in which the background error covariance matrix is estimated viathe Rao-Blackwell Ledoit and Wolf (RBLW) estimator. This estimator has proven to be better conditioned than thetrue (background) error covariance matrix. In addition, the explicit computation of the RBLW estimator is notrequired in our formulation. This implies enormous savings in terms of memory considering the typical order ofbackground error covariance matrices O(10e8) Furthermore, the proposed EnKF implementation can be efficientlyperformed in parallel: each model component is surrounded by a local box, a local background error covariancematrix is estimated based on the RBLW estimator and then, the ensemble Kalman filter equations are locally used inorder to compute the local analysis corrections. After the local assimilation, each model component is mapped backonto the full model domain from where the global analysis solution is obtained. In operational data assimilation, EnKF implementations are attractive owing to their potential for parallelization. Experiments are performed making use of the Atmospheric General Circulation Model SPEEDY. The proposed method is coded in FORTRAN and MPI is used for the local assimilation of observations. The predicted variables are the zonal and meridional wind components, the temperature and the specific humidity. For each model variable, 8 layers with the T-63 resolution (192 x 96 grid points) are considered which provide a total number of 589,824 model components. The ensemble size is 94. The number of processors utilized are 96, 256, 512, 768, 1024 and 1536. The results reveal that, the use of the RBLW estimator can mitigate the impact of spurious correlations under sparse observational networks and even more, for different number of processors, the computational effort of the proposed EnKF implementation is comparable to that of the LETKF where no covariance estimation is performed. In terms of accuracy, the proposed implementation performs better than the LETKF formulation in the Root Mean Square Error sense.

Abstract. Prediction of spatiotemporal chaotic systems is important in various fields, such as numerical weather prediction (NWP). While data assimilation methods have been applied in NWP, machine learning techniques, such as reservoir computing (RC), have recently been recognized as promising tools to predict spatiotemporal chaotic systems. However, the sensitivity of the skill of the machine-learning-based prediction to the imperfectness of observations is unclear. In this study, we evaluate the skill of RC with noisy and sparsely distributed observations. We intensively compare the performances of RC and local ensemble transform Kalman filter (LETKF) by applying them to the prediction of the Lorenz 96 system. In order to increase the scalability to larger systems, we applied a parallelized RC framework. Although RC can successfully predict the Lorenz 96 system if the system is perfectly observed, we find that RC is vulnerable to observation sparsity compared with LETKF. To overcome this limitation of RC, we propose to combine LETKF and RC. In our proposed method, the system is predicted by RC that learned the analysis time series estimated by LETKF. Our proposed method can successfully predict the Lorenz 96 system using noisy and sparsely distributed observations. Most importantly, our method can predict better than LETKF when the process-based model is imperfect.

Groundwater contamination assessment can be useful in taking proper actions during environmental emergency. A three-dimensional deterministic model was taken into consideration to simulate the advective-diffusive transport of non-conservative contaminant in groundwater. Multiple stochastic data assimilation techniques, such as, ensemble Kalman filter (EnKF), local ensemble transform Kalman filter (LETKF), and the global form of the LETKF, denoted as GETKF were applied to the model. The results show that data assimilation improved contaminant concentration prediction. The EnKF method reduced the root-mean-square-error (RMSE) of the contaminant prediction from 12.5 mg/L to 1.31 mg/L, whereas the LETKF reduced that to 0.46 mg/L and GETKF reduced that to 0.38 mg/L. Approximately 89.48%, 96.30% and 96.82% improvement were made by EnKF, LETKF, and GETKF, respectively. The sensitivity analysis suggest that these data assimilation techniques are very sensitive to the observation noise, process noise, and ensemble size.

Despite the continuous improvements in numerical weather prediction (NWP) models, the quantitative precipitation forecast (QPF) is still a challenge. A crucial role in the accuracy of QPF is played by data assimilation, the technique whereby initial conditions for an NWP model are generated by combining observations of the state of the atmosphere and a previous forecast of the model itself. In this work, the direct assimilation of radar reflectivity volumes, which is still at a preliminary stage in operational frameworks, is evaluated. This is carried out using a local ensemble transform Kalman filter (LETKF) scheme and employing the COSMO-2I model, the configuration of the convection-permitting model of the COnsortium for Small-scale MOdelling (COSMO) adopted at the Regional Agency for Prevention, Environment and Energy of Emilia-Romagna region (ARPAE) to provide high-resolution weather forecasts over Italy. The crucial aspects of the assimilation of this type of observation are investigated, in particular concerning the length of the assimilation window, the estimation of the observation error and the configuration of the radar operator employed to simulate reflectivity observations from the prognostic model fields. Taking advantage of the results obtained from this investigation, a set-up for the direct assimilation of reflectivity volumes suitable for an operational implementation is defined. Accuracy of QPF and of other forecast variables obtained with this set-up is compared to that obtained with the current operational set-up employed at ARPAE to generate the initial conditions of COSMO-2I, in which radar-estimated precipitation is assimilated through a latent heat nudging scheme. Results of this comparison, which is the most extended ever performed in terms of number of forecasts involved and in the number of verification scores employed, suggest that time is ripe to directly assimilate reflectivity volumes in an operational framework using an ensemble Kalman filter scheme.

Recent numerical weather prediction systems have significantly improved medium-range forecasts by implementing hybrid background error covariance, for which climatological (static) and ensemble-based (flow-dependent) error covariance are combined. While the hybrid approach has been investigated mainly in variational systems, this study aims at exploring methods for implementing the hybrid approach for the local ensemble transform Kalman filter (LETKF). Following Kretschmer et al., the present study constructed hybrid background error covariance by adding collections of climatological perturbations to the forecast ensemble. In addition, this study proposes a new localization method that attenuates the ensemble perturbation (Z-localization) instead of inflating observation error variance (R-localization). A series of experiments with a simplified global atmospheric model revealed that the hybrid LETKF resulted in smaller forecast errors than the LETKF, especially in sparsely observed regions. Due to the larger ensemble enabled by the hybrid approach, optimal localization length scales for the hybrid LETKF were larger than those for the LETKF. With the LETKF, the Z-localization resulted in similar forecast errors as the R-localization. However, Z-localization has an advantage in enabling us to apply different localization scales for flow-dependent perturbation and climatological static perturbations with the hybrid LETKF. The optimal localization for climatological perturbations was slightly larger than that for flow-dependent perturbations. This study also proposes optimal eigendecomposition (OED) ETKF formulation to reduce computational costs. The computational expense of the OED ETKF formulation became significantly smaller than that of standard ETKF formulations as the number of climatological perturbations was increased beyond a few hundred.

Gangwon-do (GWD) has complex terrain and surface characteristics due to its location to the East Sea and the Taebaek Mountain range. This coastal location and rugged terrain can amplify snowfall mechanisms, making it challenging to accurately predict the amount and location. This study compares two methods for assimilating radar data and analyzed snowfall prediction results. The two methods compared are local ensemble transform Kalman filter (LETKF) and three-dimensional variational (3DVAR) data assimilation (DA). LETKF improved the water vapor amount and temperature using the covariance of the ensemble members, but 3DVAR improved the water vapor mixing ratio and temperature through an operator that assumed the atmosphere was saturated when reflectivity was above a certain threshold. In 2018, to understand the snowfall in GWD region and support the Pyeongchang Winter Olympic and Paralympic Games, a long-term heavy snow observation campaign was conducted. The International Collaborative Experiments for the 2018 Pyeongchang Olympic Games Projects (ICE-POP 2018) data are used to study and verify the numerical experiments. From the initial field verification using ICE-POP observation data (radiosonde), wind in LETKF was more accurately simulated than 3DVAR, but it underestimated the water vapor amount and temperature in the lower troposphere due to a lack of a water vapor and temperature observation operator. Snowfall in GWD was less simulated in LETKF, whereas snowfall of 10.0 mm or more was simulated in 3DVAR, resulting in an error of 2.62 mm lower than LETKF. The results signify that water vapor assimilation is important in radar DA and significantly impacts precipitation forecasts, regardless of the DA method used. Therefore, it is necessary to apply observation operators for water vapor and temperature in radar DA.

The assimilation of radar reflectivity data is of great interest for numerical weather prediction (NWP) models at the convective scale since these observations are very dense both in space and time and they are related to microphysical prognostic variables. So far, the assimilation of reflectivity in operational frameworks has been limited to derived products, like precipitation rates and humidity profiles. This approach is also adopted at ARPAE‐SIMC, the Hydro‐Meteo‐Climate Structure of the Regional Agency for Prevention, Environment and Energy of Emilia‐Romagna region in Italy, where latent heat nudging (LHN) is employed to assimilate precipitation rates in the COSMO‐2I model, the convection‐permitting version of the regional model developed by the COnsortium for Small‐scale MOdelling (COSMO) which is run operationally to provide high‐resolution forecasts over Italy. However, to fully exploit the information contained in these observations, the whole reflectivity volume should be directly assimilated. Nevertheless, despite several promising studies, its implementation in an operational framework has not yet been achieved. In this study, a set‐up designed to assimilate reflectivities operationally in the COSMO‐2I model through a local ensemble transform Kalman filter (LETKF) has been evaluated over 37 days in 2018 and performed 303 forecasts. The comparison with the current operational set‐up based on LHN reveals an average improvement in quantitative precipitation forecasts (QPFs) up to 7 hr, even if the impact on convective cases is much stronger than that observed in conditions of stratiform precipitation. Moreover, a small but positive impact is noticed for the RMSE of upper‐air variables, while the impact on bias is mixed. Mixed results are also obtained when considering surface variables. In the light of the results of this study, a pre‐operational parallel system in which COSMO‐2I analyses are generated replacing LHN by the direct assimilation of reflectivity volumes was implemented in April 2020.

[1] In ensemble Kalman filter, space localization is used to reduce the impact of long-distance sampling errors in the ensemble estimation of the forecast error covariance. When two variables are not physically correlated, their error covariance is still estimated by the ensemble and, therefore, it is dominated by sampling errors. We introduce a “variable localization” method, zeroing out such covariances between unrelated variables to the problem of assimilating carbon dioxide concentrations into a dynamical model using the local ensemble transform Kalman filter (LETKF) in an observing system simulation experiments (OSSE) framework. A system where meteorological and carbon variables are simultaneously assimilated is used to estimate surface carbon fluxes that are not directly observed. A range of covariance structures are explored for the LETKF, with emphasis on configurations allowing nonzero error covariance between carbon variables and the wind field, which affects transport of atmospheric CO2, but not between CO2 and the other meteorological variables. Such variable localization scheme zeroes out the background error covariance among prognostic variables that are not physically related, thus reducing sampling errors. Results from the identical twin experiments show that the performance in the estimation of surface carbon fluxes obtained using variable localization is much better than that using a standard full covariance approach. The relative improvement increases when the surface fluxes change with time and model error becomes significant.

<strong class="journal-contentHeaderColor">Abstract.</strong> Atmospheric inversion of carbon dioxide (CO<span class="inline-formula">₂</span>) measurements to better understand carbon sources and sinks has made great progress over the last 2 decades. However, most of the studies, including a four-dimensional variational ensemble Kalman filter and Bayesian synthesis approaches, directly obtain only fluxes, while CO<span class="inline-formula">₂</span> concentration is derived with the forward model as part of a post-analysis. Kang et al. (2012) used the local ensemble transform Kalman filter (LETKF), which updates the CO<span class="inline-formula">₂</span>, surface carbon flux (SCF), and meteorology fields simultaneously. Following this track, a system with a short assimilation window and a long observation window was developed (Liu et al., 2019). However, this data assimilation system faces the challenge of maintaining carbon mass conservation. To overcome this shortcoming, here we apply a constrained ensemble Kalman filter (CEnKF) approach to ensure the conservation of global CO<span class="inline-formula">₂</span> mass. After a standard LETKF procedure, an additional assimilation is used to adjust CO<span class="inline-formula">₂</span> at each model grid point and to ensure the consistency between the analysis and the first guess of the global CO<span class="inline-formula">₂</span> mass. Compared to an observing system simulation experiment without mass conservation, the CEnKF significantly reduces the annual global SCF bias from <span class="inline-formula">∼</span> 0.2 to less than 0.06 Gt and slightly improves the seasonal and annual performance over tropical and southern extratropical regions. We show that this system can accurately track the spatial distribution of annual mean SCF. And the system reduces the seasonal flux root mean square error from a priori to analysis by 48 %–90 %, depending on the continental region. Moreover, the 2015–2016 El Niño impact is well captured with anomalies mainly in the tropics.

<strong class="journal-contentHeaderColor">Abstract.</strong> Atmospheric inversion of carbon dioxide (CO<span class="inline-formula">₂</span>) measurements to better understand carbon sources and sinks has made great progress over the last 2 decades. However, most of the studies, including a four-dimensional variational ensemble Kalman filter and Bayesian synthesis approaches, directly obtain only fluxes, while CO<span class="inline-formula">₂</span> concentration is derived with the forward model as part of a post-analysis. Kang et al. (2012) used the local ensemble transform Kalman filter (LETKF), which updates the CO<span class="inline-formula">₂</span>, surface carbon flux (SCF), and meteorology fields simultaneously. Following this track, a system with a short assimilation window and a long observation window was developed (Liu et al., 2019). However, this data assimilation system faces the challenge of maintaining carbon mass conservation. To overcome this shortcoming, here we apply a constrained ensemble Kalman filter (CEnKF) approach to ensure the conservation of global CO<span class="inline-formula">₂</span> mass. After a standard LETKF procedure, an additional assimilation is used to adjust CO<span class="inline-formula">₂</span> at each model grid point and to ensure the consistency between the analysis and the first guess of the global CO<span class="inline-formula">₂</span> mass. Compared to an observing system simulation experiment without mass conservation, the CEnKF significantly reduces the annual global SCF bias from <span class="inline-formula">∼</span> 0.2 to less than 0.06 Gt and slightly improves the seasonal and annual performance over tropical and southern extratropical regions. We show that this system can accurately track the spatial distribution of annual mean SCF. And the system reduces the seasonal flux root mean square error from a priori to analysis by 48 %–90 %, depending on the continental region. Moreover, the 2015–2016 El Niño impact is well captured with anomalies mainly in the tropics.

Ensemble Kalman filters perform data assimilation by forming a background covariance matrix from an ensemble forecast. Most of the literature on ensemble Kalman filters assumes that all ensemble members come from the same model. This article presents and tests a modified local ensemble transform Kalman filter (LETKF) that takes its background covariance from a combination of a high-resolution ensemble and a low-resolution ensemble. The computational time and the accuracy of this mixed-resolution LETKF are explored and compared to the standard LETKF on a high-resolution ensemble, using simulated observation experiments with the Lorenz models II and III (more complex versions of the Lorenz-96 model). In a variety of scenarios, mixed-resolution analysis can obtain higher accuracy with similar computation time (or similar accuracy with a reduced computation time) compared to single-resolution analysis.

A novel particle filter proposed recently, the particle flow filter (PFF), avoids the long‐existing weight degeneracy problem in particle filters and, therefore, has great potential to be applied in high‐dimensional systems. The PFF adopts the idea of a particle flow, which sequentially pushes the particles from the prior to the posterior distribution, without changing the weight of each particle. The essence of the PFF is that it assumes the particle flow is embedded in a reproducing kernel Hilbert space, so that a practical solution for the particle flow is obtained. The particle flow is independent of the choice of kernel in the limit of an infinite number of particles. Given a finite number of particles, we have found that a scalar kernel fails in high‐dimensional and sparsely observed settings. A new matrix‐valued kernel is proposed that prevents the collapse of the marginal distribution of observed variables in a high‐dimensional system. The performance of the PFF is tested and compared with a well‐tuned local ensemble transform Kalman filter (LETKF) using the 1,000‐dimensional Lorenz 96 model. It is shown that the PFF is comparable to the LETKF for linear observations, except that explicit covariance inflation is not necessary for the PFF. For nonlinear observations, the PFF outperforms LETKF and is able to capture the multimodal likelihood behavior, demonstrating that the PFF is a viable path to fully nonlinear geophysical data assimilation.

Particle filters (PFs) are Monte Carlo data assimilation techniques that operate with no parametric assumptions for prior and posterior errors. A data assimilation method introduced recently, called the local PF, approximates the PF solution within neighborhoods of observations, thus allowing for its use in high-dimensional systems. The current study explores the potential of the local PF for atmospheric data assimilation through cloud-permitting numerical experiments performed for an idealized squall line. Using only 100 ensemble members, experiments using the local PF to assimilate simulated radar measurements demonstrate that the method provides accurate analyses at a cost comparable to ensemble filters currently used in weather models. Comparisons between the local PF and an ensemble Kalman filter demonstrate benefits of the local PF for producing probabilistic analyses of non-Gaussian variables, such as hydrometeor mixing ratios. The local PF also provides more accurate forecasts than the ensemble Kalman filter, despite yielding higher posterior root-mean-square errors. A major advantage of the local PF comes from its ability to produce more physically consistent posterior members than the ensemble Kalman filter, which leads to fewer spurious model adjustments during forecasts. This manuscript presents the first successful application of the local PF in a weather prediction model and discusses implications for real applications where nonlinear measurement operators and nonlinear model processes limit the effectiveness of current Gaussian data assimilation techniques.