Ensemble-derived stationary and flow-dependent background-error covariances: Evaluation in a quasi-operational NWP setting

This paper develops a parallel implementation of the ensemble Kalman filter (EnKF) based on shrinkage covariancematrix estimation. The EnKF is a sequential Monte Carlo method for parameter and state estimation of highly nonlinearmodels. In the context of ensemble based methods, background error correlations are estimated via anensemble of model realizations. In practice, the model dimension (number of model components) is several timeslarger than the ensemble size owing to the computational effort involved in one single model propagation. Thisconstraint yields to spurious correlations (correlation between distant model components in space) in the backgrounderror estimations. Commonly, localization methods are utilized in order to reduce the impact of spuriouscorrelations. For instance, correlations of model components are reduced based on some function of their physicaldistances. Some EnKF implementations are formulated in such manner that, localization methods are implicitlyinvolved in the assimilation of observations. In this context, one of the best ensemble Kalman filter implementationsis the local ensemble transform Kalman filter (LETKF). In the LETKF, each model component is surrounded by alocal box of size r and then, the information contained within the local domain (i.e., observed components) is used toperform the assimilation. However, for sparse observational networks, r can become sufficiently large with respectto the ensemble size and therefore, the local analysis corrections can be impacted by spurious correlations. Wepropose an ensemble Kalman filter implementation in which the background error covariance matrix is estimated viathe Rao-Blackwell Ledoit and Wolf (RBLW) estimator. This estimator has proven to be better conditioned than thetrue (background) error covariance matrix. In addition, the explicit computation of the RBLW estimator is notrequired in our formulation. This implies enormous savings in terms of memory considering the typical order ofbackground error covariance matrices O(10e8) Furthermore, the proposed EnKF implementation can be efficientlyperformed in parallel: each model component is surrounded by a local box, a local background error covariancematrix is estimated based on the RBLW estimator and then, the ensemble Kalman filter equations are locally used inorder to compute the local analysis corrections. After the local assimilation, each model component is mapped backonto the full model domain from where the global analysis solution is obtained. In operational data assimilation, EnKF implementations are attractive owing to their potential for parallelization. Experiments are performed making use of the Atmospheric General Circulation Model SPEEDY. The proposed method is coded in FORTRAN and MPI is used for the local assimilation of observations. The predicted variables are the zonal and meridional wind components, the temperature and the specific humidity. For each model variable, 8 layers with the T-63 resolution (192 x 96 grid points) are considered which provide a total number of 589,824 model components. The ensemble size is 94. The number of processors utilized are 96, 256, 512, 768, 1024 and 1536. The results reveal that, the use of the RBLW estimator can mitigate the impact of spurious correlations under sparse observational networks and even more, for different number of processors, the computational effort of the proposed EnKF implementation is comparable to that of the LETKF where no covariance estimation is performed. In terms of accuracy, the proposed implementation performs better than the LETKF formulation in the Root Mean Square Error sense.

Previous research suggests that an ensemble Kalman filter (EnKF) data assimilation and modeling system can produce accurate atmospheric analyses and forecasts at 30–50-km grid spacing. This study examines the ability of a mesoscale EnKF system using multiscale (36/12 km) Weather Research and Forecasting (WRF) model simulations to produce high-resolution, accurate, regional surface analyses, and 6-h forecasts. This study takes place over the complex terrain of the Pacific Northwest, where the small-scale features of the near-surface flow field make the region particularly attractive for testing an EnKF and its flow-dependent background error covariances. A variety of EnKF experiments are performed over a 5-week period to test the impact of decreasing the grid spacing from 36 to 12 km and to evaluate new approaches for dealing with representativeness error, lack of surface background variance, and low-level bias. All verification in this study is performed with independent, unassimilated observations. Significant surface analysis and 6-h forecast improvements are found when EnKF grid spacing is reduced from 36 to 12 km. Forecast improvements appear to be a consequence of increased resolution during model integration, whereas analysis improvements also benefit from high-resolution ensemble covariances during data assimilation. On the 12-km domain, additional analysis improvements are found by reducing observation error variance in order to address representativeness error. Removing model surface biases prior to assimilation significantly enhances the analysis. Inflating surface wind and temperature background error variance has large impacts on analyses, but only produces small improvements in analysis RMS errors. Both surface and upper-air 6-h forecasts are nearly unchanged in the 12-km experiments. Last, 12-km WRF EnKF surface analyses and 6-h forecasts are shown to generally outperform those of the Global Forecast System (GFS), North American Model (NAM), and the Rapid Update Cycle (RUC) by about 10%–30%, although these improvements do not extend above the surface. Based on these results, future improvements in multiscale EnKF are suggested.

This study introduces an experimental regional assimilation configuration for a 4D ensemble–variational (4D-EnVar) deterministic weather prediction system. A total of 16 assimilation experiments covering July 2014 are presented to assess both experimental regional climatological background error covariances and updates in the treatment of flow-dependent error covariances. The regional climatological background error covariances are estimated using statistical correlations between variables instead of using balance operators. These error covariance estimates allow the analyses to fit more closely with the assimilated observations than when using the lower-resolution global background error covariances (due to shorter correlation scales), and the ensuing forecasts are significantly improved. The use of ensemble-based background error covariances is also improved by reducing vertical and horizontal localization length scales for the flow-dependent background error covariance component. Also, reducing the number of ensemble members employed in the deterministic analysis (from 256 to 128) reduced computational costs by half without degrading the accuracy of analyses and forecasts. The impact of the relative contributions of the climatological and flow-dependent background error covariance components is also examined. Results show that the experimental regional system benefits from giving a lower (higher) weight to climatological (flow-dependent) error covariances. When compared with the operational assimilation configuration of the continental prediction system, the proposed modifications to the background error covariances improve both surface and upper-air RMSE scores by nearly 1%. Still, the use of a higher-resolution ensemble to estimate flow-dependent background error covariances does not yet provide added value, although it is expected to allow for a better use of dense observations in the future.

Dual-resolution (DR) hybrid variational-ensemble analysis capability was implemented within the community Weather Research and Forecasting (WRF) Model data assimilation (DA) system, which is designed for limited-area applications. The DR hybrid system combines a high-resolution (HR) background, flow-dependent background error covariances (BECs) derived from a low-resolution ensemble, and observations to produce a deterministic HR analysis. As DR systems do not require HR ensembles, they are computationally cheaper than single-resolution (SR) hybrid configurations, where the background and ensemble have equal resolutions. Single-observation tests were performed to document some characteristics of limited-area DR hybrid analyses. Additionally, the DR hybrid system was evaluated within a continuously cycling framework, where new DR hybrid analyses were produced every 6 h over ~3.5 weeks. In the DR configuration presented here, the deterministic backgrounds and analyses had 15-km horizontal grid spacing, but the 32-member WRF Model–based ensembles providing flow-dependent BECs for the hybrid had 45-km horizontal grid spacing. The DR hybrid analyses initialized 72-h WRF Model forecasts that were compared to forecasts initialized by an SR hybrid system where both the ensemble and background had 15-km horizontal grid spacing. The SR and DR hybrid systems were coupled to an ensemble adjustment Kalman filter that updated ensembles each DA cycle. On average, forecasts initialized from 15-km DR and SR hybrid analyses were not statistically significantly different, although tropical cyclone track forecast errors favored the SR-initialized forecasts. Although additional studies over longer time periods and at finer grid spacing are needed to further understand sensitivity to ensemble perturbation resolution, these results suggest users should carefully consider whether SR hybrid systems are worth the extra cost.

The ensemble Kalman filter (EnKF) is a data assimilation scheme based on the traditional Kalman filter update equation. An ensemble of forecasts are used to estimate the background-error covariances needed to compute the Kalman gain. It is known that if the same observations and the same gain are used to update each member of the ensemble, the ensemble will systematically underestimate analysis-error covariances. This will cause a degradation of subsequent analyses and may lead to filter divergence. For large ensembles, it is known that this problem can be alleviated by treating the observations as random variables, adding random perturbations to them with the correct statistics. Two important consequences of sampling error in the estimate of analysis-error covariances in the EnKF are discussed here. The first results from the analysis-error covariance being a nonlinear function of the backgrounderror covariance in the Kalman filter. Due to this nonlinearity, analysis-error covariance estimates may be negatively biased, even if the ensemble background-error covariance estimates are unbiased. This problem must be dealt with in any Kalman filter‐based ensemble data assimilation scheme. A second consequence of sampling error is particular to schemes like the EnKF that use perturbed observations. While this procedure gives asymptotically correct analysis-error covariance estimates for large ensembles, the addition of perturbed observations adds an additional source of sampling error related to the estimation of the observation-error covariances. In addition to reducing the accuracy of the analysis-error covariance estimate, this extra source of sampling error increases the probability that the analysis-error covariance will be underestimated. Because of this, ensemble data assimilation methods that use perturbed observations are expected to be less accurate than those which do not. Several ensemble filter formulations have recently been proposed that do not require perturbed observations. This study examines a particularly simple implementation called the ensemble square root filter, or EnSRF. The EnSRF uses the traditional Kalman gain for updating the ensemble mean but uses a ‘‘reduced’’ Kalman gain to update deviations from the ensemble mean. There is no additional computational cost incurred by the EnSRF relative to the EnKF when the observations have independent errors and are processed one at a time. Using a hierarchy of perfect model assimilation experiments, it is demonstrated that the elimination of the sampling error associated with the perturbed observations makes the EnSRF more accurate than the EnKF for the same ensemble size.

Diurnally varying background error covariances estimated in RMAPS-ST and their impacts on operational implementations

Assimilating remote sensing and in situ observations into a coastal model of northern South China Sea using ensemble Kalman filter

Background error covariance (BEC) is crucial in data assimilation. This paper addresses the multivariate BEC associated with black carbon, organic carbon, nitrates, sulfates, and other con- stituents of aerosol species. These aerosol species are modeled and predicted using the Model for Simulating Aerosol Interactions and Chemistry scheme (MOSAIC) in the Weather Research and Forecasting/Chemistry (WRF/Chem) model at a resolution of 4 km in Southern California. The BEC is estimated from the differences between the 36-hour and 12-hour forecasts using the NMC method. The results indicated that the maximum background error standard deviation is associated with nitrate and is larger than that of black carbon, organic carbon, and sulfate. The horizontal and vertical scale of the correlation of nitrate is much smaller than that of other species. A significant cross-correlation is found between the species of black carbon and organic carbon. The cross-correlations between nitrate and other variables are relatively smaller and exhibit a relatively smaller length scale. Single observation data assimilation experiments are performed to illustrate the effect of the BEC on analysis increments.

The effect on weather forecast performance of incorporating ensemble covariances into the initial covariance model of the four-dimensional variational data assimilation (4D-Var) Naval Research Laboratory Atmospheric Variational Data Assimilation System-Accelerated Representer (NAVDAS-AR) is investigated. This NAVDAS-AR-hybrid scheme linearly combines the static NAVDAS-AR initial background error covariance with a covariance derived from an 80-member flow-dependent ensemble. The ensemble members are generated using the ensemble transform technique with a (three-dimensional variational data assimilation) 3D-Var-based estimate of analysis error variance. The ensemble covariances are localized using an efficient algorithm enabled via a separable formulation of the localization matrix. The authors describe the development and testing of this scheme, which allows for assimilation experiments using differing linear combinations of the static and flow-dependent background error covariances. The tests are performed for two months of summer and two months of winter using operational model resolution and the operational observational dataset, which is dominated by satellite observations. Results show that the hybrid mode data assimilation scheme significantly reduces the forecast error across a wide range of variables and regions. The improvements were particularly pronounced for tropical winds. The verification against radiosondes showed a greater than 0.5% reduction in vector wind RMS differences in areas of statistical significance. The verification against self-analysis showed a greater than 1% reduction from verifying against analyses between 2- and 5-day lead time at all eight vertical levels examined in areas of statistical significance. Using the Navy's summary of verification results, the Navy Operational Global Atmospheric Prediction System (NOGAPS) scorecard, the improvements resulted in a score (+1) that justifies a major system upgrade.

: The final goal of this project is to provide the US Navy with an increased capability of using Doppler radar observations in the detection and prediction of hazardous weather events that usually have a strong randomness in nature and affect the Navy operations, especially over oceans and in remote areas. By developing a high-resolution data assimilation capability that can effectively assimilate Doppler radar observations along with other conventional and remotely-sensed data, the US Navy will have the ability to analyze and forecast the battlespace atmospheric conditions with sufficient detail and accuracy for supporting the Navy mission in threat detection, weapons deployment, and weather safe operations. The objective of the study is to develop an advanced ensemble-based radar data assimilation system for the US Navy and to address some critical scientific and technique issues associated with ensemble radar data assimilation. The radar data system that will be developed will use flow-dependent background error covariance (instead of the static background error covariance) to account for the complexity and rapid change in the dynamical and microphysical structures inside and outside storms. The system will assimilate all the observed variables from different types of sensors, including Doppler radars, satellites, UASs, and conventional meteorological observations, simultaneously to allow full interactions among the assimilated variables during the data assimilation to keep the balances among the dynamics, thermodynamics and microphysics in the model initial fields. The system will be able to use the observations from many types of radars on different platforms (WSR-88D, DoD meteorological radars and tactical radars both on-land and shipboard, etc.) with an appropriate quality control. Multi-scale data assimilation capability will also be one of the major features of the new radar data assimilation system that allows observational data at different scales to be as

Since mid-February 2013, the ensemble Kalman filter (EnKF) in operation at the Canadian Meteorological Centre (CMC) has been using a 600 × 300 global horizontal grid and 74 vertical levels. This yields 5.4 × 107 model coordinates. The EnKF has 192 members and uses seven time levels, spaced 1 h apart, for the time interpolation in the 6-h assimilation window. It follows that over 7 × 1010 values are required to specify an ensemble of trial field trajectories. This paper focuses on numerical and computational aspects of the EnKF. In response to the increasing computational challenge posed by the ever more ambitious configurations, an ever larger fraction of the EnKF software system has gradually been parallelized over the past decade. In a strong scaling experiment, the way in which the execution time decreases as larger numbers of processes are used is investigated. In fact, using a substantial fraction of one of the CMC's computers, very short execution times are achieved. As it would thus appear that the CMC's computers can handle more demanding configurations, weak scaling experiments are also performed. Here, both the size of the problem and the number of processes are simultaneously increased. The parallel algorithm responds well to an increase in either the number of ensemble members or the number of model coordinates. A substantial increase (by an order of magnitude) in the number of assimilated observations would, however, be more problematic. Thus, to the extent that this depends on computational aspects, it appears that the meteorological quality of the Canadian operational EnKF can be further improved.

[1] In ensemble Kalman filter, space localization is used to reduce the impact of long-distance sampling errors in the ensemble estimation of the forecast error covariance. When two variables are not physically correlated, their error covariance is still estimated by the ensemble and, therefore, it is dominated by sampling errors. We introduce a “variable localization” method, zeroing out such covariances between unrelated variables to the problem of assimilating carbon dioxide concentrations into a dynamical model using the local ensemble transform Kalman filter (LETKF) in an observing system simulation experiments (OSSE) framework. A system where meteorological and carbon variables are simultaneously assimilated is used to estimate surface carbon fluxes that are not directly observed. A range of covariance structures are explored for the LETKF, with emphasis on configurations allowing nonzero error covariance between carbon variables and the wind field, which affects transport of atmospheric CO2, but not between CO2 and the other meteorological variables. Such variable localization scheme zeroes out the background error covariance among prognostic variables that are not physically related, thus reducing sampling errors. Results from the identical twin experiments show that the performance in the estimation of surface carbon fluxes obtained using variable localization is much better than that using a standard full covariance approach. The relative improvement increases when the surface fluxes change with time and model error becomes significant.

Abstract. The ensemble Kalman filter (EnKF) is a widely used ensemble-based assimilation method, which estimates the forecast error covariance matrix using a Monte Carlo approach that involves an ensemble of short-term forecasts. While the accuracy of the forecast error covariance matrix is crucial for achieving accurate forecasts, the estimate given by the EnKF needs to be improved using inflation techniques. Otherwise, the sampling covariance matrix of perturbed forecast states will underestimate the true forecast error covariance matrix because of the limited ensemble size and large model errors, which may eventually result in the divergence of the filter. In this study, the forecast error covariance inflation factor is estimated using a generalized cross-validation technique. The improved EnKF assimilation scheme is tested on the atmosphere-like Lorenz-96 model with spatially correlated observations, and is shown to reduce the analysis error and increase its sensitivity to the observations.

Surface currents measured by high frequency (HF) radar arrays are assimilated into a regional ocean model over Qingdao coastal waters based on Kalman filter method. A series of numerical experiments are performed to evaluate the performance of the data assimilation schemes. In order to optimize the analysis procedure in the traditional ensemble Kalman filter (ENKF), a different analysis scheme called quasiensemble Kaman filter (QENKF) is proposed. The comparisons between the ENKF and the QENKF suggest that both them can improve the simulated error and the spatial structure. The estimations of the background error covariance (BEC) are also assessed by comparing three different methods: Monte Carlo method; Canadian quick covariance (CQC) method and data uncertainty engine (DUE) method. A significant reduction of the root-mean-square (RMS) errors between model results and the observations shows that the CQC method is able to better reproduce the error statistics for this coastal ocean model and the corresponding external forcing. In addition, the sensibility of the data assimilation system to the ensemble size is also analyzed by means of different scales of the ensemble size used in the experiments. It is found that given the balance of the computational cost and the forecasting accuracy, the ensemble size of 50 will be an appropriate choice in the Qingdao coastal waters.

Anisotropic recursive filters are implemented within a three-dimensional variational data assimilation (3DVAR) framework to efficiently model the effect of flow-dependent background error covariance. The background error covariance is based on an estimated error field and on the idea of Riishøjgaard. In the anisotropic case, the background error pattern can be stretched or flattened in directions oblique to the alignment of the grid coordinates and is constructed by applying, at each point, six recursive filters along six directions corresponding, in general, to a special configuration of oblique lines of the grid. The recursive filters are much more efficient than corresponding explicit filters used in an earlier study and are therefore more suitable for real-time numerical weather prediction. A set of analysis experiments are conducted at a mesoscale resolution to examine the effectiveness of the 3DVAR system in analyzing simulated global positioning system (GPS) slant-path water vapor observations from ground-based GPS receivers and observations from collocated surface stations. It is shown that the analyses produced with recursive filters are at least as good as those with corresponding explicit filters. In some cases, the recursive filters actually perform better. The impact of flow-dependent background errors modeled using the anisotropic recursive filters is also examined. The use of anisotropic filters improves the analysis, especially in terms of finescale structures. The analysis system is found to be effective in the presence of typical observational errors. The sensitivity of isotropic and anisotropic recursive-filter analyses to the decorrelation scales is also examined systematically.