Learning enhanced ensemble filters

Abstract. A new class of ensemble filters, called the Diffuse Ensemble Filter (DEnF), is proposed in this paper. The DEnF assumes that the forecast errors orthogonal to the first guess ensemble are uncorrelated with the latter ensemble and have infinite variance. The assumption of infinite variance corresponds to the limit of "complete lack of knowledge" and differs dramatically from the implicit assumption made in most other ensemble filters, which is that the forecast errors orthogonal to the first guess ensemble have vanishing errors. The DEnF is independent of the detailed covariances assumed in the space orthogonal to the ensemble space, and reduces to conventional ensemble square root filters when the number of ensembles exceeds the model dimension. The DEnF is well defined only in data rich regimes and involves the inversion of relatively large matrices, although this barrier might be circumvented by variational methods. Two algorithms for solving the DEnF, namely the Diffuse Ensemble Kalman Filter (DEnKF) and the Diffuse Ensemble Transform Kalman Filter (DETKF), are proposed and found to give comparable results. These filters generally converge to the traditional EnKF and ETKF, respectively, when the ensemble size exceeds the model dimension. Numerical experiments demonstrate that the DEnF eliminates filter collapse, which occurs in ensemble Kalman filters for small ensemble sizes. Also, the use of the DEnF to initialize a conventional square root filter dramatically accelerates the spin-up time for convergence. However, in a perfect model scenario, the DEnF produces larger errors than ensemble square root filters that have covariance localization and inflation. For imperfect forecast models, the DEnF produces smaller errors than the ensemble square root filter with inflation. These experiments suggest that the DEnF has some advantages relative to the ensemble square root filters in the regime of small ensemble size, imperfect model, and copious observations.

The ensemble Kalman filter (EnKF) has been shown repeatedly to be an effective method for data assimilation in large-scale problems, including those in petroleum engineering. Data assimilation for multiphase flow in porous media is particularly difficult, however, because the relationships between model variables (e.g., permeability and porosity) and observations (e.g., water cut and gas–oil ratio) are highly nonlinear. Because of the linear approximation in the update step and the use of a limited number of realizations in an ensemble, the EnKF has a tendency to systematically underestimate the variance of the model variables. Various approaches have been suggested to reduce the magnitude of this problem, including the application of ensemble filter methods that do not require perturbations to the observed data. On the other hand, iterative least-squares data assimilation methods with perturbations of the observations have been shown to be fairly robust to nonlinearity in the data relationship. In this paper, we present EnKF with perturbed observations as a square root filter in an enlarged state space. By imposing second-order-exact sampling of the observation errors and independence constraints to eliminate the cross-covariance with predicted observation perturbations, we show that it is possible in linear problems to obtain results from EnKF with observation perturbations that are equivalent to ensemble square-root filter results. Results from a standard EnKF, EnKF with second-order-exact sampling of measurement errors that satisfy independence constraints (EnKF (SIC)), and an ensemble square-root filter (ETKF) are compared on various test problems with varying degrees of nonlinearity and dimensions. The first test problem is a simple one-variable quadratic model in which the nonlinearity of the observation operator is varied over a wide range by adjusting the magnitude of the coefficient of the quadratic term. The second problem has increased observation and model dimensions to test the EnKF (SIC) algorithm. The third test problem is a two-dimensional, two-phase reservoir flow problem in which permeability and porosity of every grid cell (5,000 model parameters) are unknown. The EnKF (SIC) and the mean-preserving ETKF (SRF) give similar results when applied to linear problems, and both are better than the standard EnKF. Although the ensemble methods are expected to handle the forecast step well in nonlinear problems, the estimates of the mean and the variance from the analysis step for all variants of ensemble filters are also surprisingly good, with little difference between ensemble methods when applied to nonlinear problems.

Assimilation of satellite radiances has been proven to have positive impacts on the forecast skill, especially for regions with sparse conventional observations. Localization is an essential component to effectively assimilate satellite radiances in ensemble Kalman filters with affordable ensemble sizes. However, localizing the impact of radiance observations is not straightforward, since their location and separation from grid point model variables are not well defined. A global group filter (GGF) is applied here to provide a theoretical estimate of vertical localization functions for radiance observations being assimilated for global numerical weather prediction. As an extension of the hierarchical ensemble filter, the GGF uses groups of climatological ensembles to provide an estimated localization function that reduces the erroneous increments due to ensemble correlation sampling error. Results from an idealized simulation with known background error covariances show that the GGF localization function is superior to the optimal Gaspari and Cohn (GC) localization function. When the GGF is applied to the AMSU‐A radiances, it can provide different localization functions for different channels, which indicates the complexity and large computational cost of tuning the localization scales for radiance observations. The GC, GGF, and fitted GGF (FGGF) localization functions are compared using experiments with the NCEP GFS and the NOAA operational EnKF. Verifications relative to the conventional observations, AMSU‐A radiances, and the ECMWF analyses show that the GGF and FGGF have smaller errors than GC except in the tropics, and the advantages of the GGF and FGGF persist through 120 h forecast lead time.

Particle filters (PFs) are Monte Carlo data assimilation techniques that operate with no parametric assumptions for prior and posterior errors. A data assimilation method introduced recently, called the local PF, approximates the PF solution within neighborhoods of observations, thus allowing for its use in high-dimensional systems. The current study explores the potential of the local PF for atmospheric data assimilation through cloud-permitting numerical experiments performed for an idealized squall line. Using only 100 ensemble members, experiments using the local PF to assimilate simulated radar measurements demonstrate that the method provides accurate analyses at a cost comparable to ensemble filters currently used in weather models. Comparisons between the local PF and an ensemble Kalman filter demonstrate benefits of the local PF for producing probabilistic analyses of non-Gaussian variables, such as hydrometeor mixing ratios. The local PF also provides more accurate forecasts than the ensemble Kalman filter, despite yielding higher posterior root-mean-square errors. A major advantage of the local PF comes from its ability to produce more physically consistent posterior members than the ensemble Kalman filter, which leads to fewer spurious model adjustments during forecasts. This manuscript presents the first successful application of the local PF in a weather prediction model and discusses implications for real applications where nonlinear measurement operators and nonlinear model processes limit the effectiveness of current Gaussian data assimilation techniques.

Bayesian estimation theory provides a general approach for the state estimate of linear or nonlinear and Gaussian or non-Gaussian systems. In this study, we first explore two Bayesian-based methods: ensemble adjustment Kalman filter (EAKF) and sequential importance resampling particle filter (SIR-PF), using a well-known nonlinear and non-Gaussian model (Lorenz '63 model). The EAKF, which is a deterministic scheme of the ensemble Kalman filter (EnKF), performs better than the classical (stochastic) EnKF in a general framework. Comparison between the SIR-PF and the EAKF reveals that the former outperforms the latter if ensemble size is so large that can avoid the filter degeneracy, and vice versa. The impact of the probability density functions and effective ensemble sizes on assimilation performances are also explored. On the basis of comparisons between the SIR-PF and the EAKF, a mixture filter, called ensemble adjustment Kalman particle filter (EAKPF), is proposed to combine their both merits. Similar to the ensemble Kalman particle filter, which combines the stochastic EnKF and SIR-PF analysis schemes with a tuning parameter, the new mixture filter essentially provides a continuous interpolation between the EAKF and SIR-PF. The same Lorenz '63 model is used as a testbed, showing that the EAKPF is able to overcome filter degeneracy while maintaining the non-Gaussian nature, and performs better than the EAKF given limited ensemble size.

The ensemble Kalman filter (EnKF) is a data assimilation scheme based on the traditional Kalman filter update equation. An ensemble of forecasts are used to estimate the background-error covariances needed to compute the Kalman gain. It is known that if the same observations and the same gain are used to update each member of the ensemble, the ensemble will systematically underestimate analysis-error covariances. This will cause a degradation of subsequent analyses and may lead to filter divergence. For large ensembles, it is known that this problem can be alleviated by treating the observations as random variables, adding random perturbations to them with the correct statistics. Two important consequences of sampling error in the estimate of analysis-error covariances in the EnKF are discussed here. The first results from the analysis-error covariance being a nonlinear function of the backgrounderror covariance in the Kalman filter. Due to this nonlinearity, analysis-error covariance estimates may be negatively biased, even if the ensemble background-error covariance estimates are unbiased. This problem must be dealt with in any Kalman filter‐based ensemble data assimilation scheme. A second consequence of sampling error is particular to schemes like the EnKF that use perturbed observations. While this procedure gives asymptotically correct analysis-error covariance estimates for large ensembles, the addition of perturbed observations adds an additional source of sampling error related to the estimation of the observation-error covariances. In addition to reducing the accuracy of the analysis-error covariance estimate, this extra source of sampling error increases the probability that the analysis-error covariance will be underestimated. Because of this, ensemble data assimilation methods that use perturbed observations are expected to be less accurate than those which do not. Several ensemble filter formulations have recently been proposed that do not require perturbed observations. This study examines a particularly simple implementation called the ensemble square root filter, or EnSRF. The EnSRF uses the traditional Kalman gain for updating the ensemble mean but uses a ‘‘reduced’’ Kalman gain to update deviations from the ensemble mean. There is no additional computational cost incurred by the EnSRF relative to the EnKF when the observations have independent errors and are processed one at a time. Using a hierarchy of perfect model assimilation experiments, it is demonstrated that the elimination of the sampling error associated with the perturbed observations makes the EnSRF more accurate than the EnKF for the same ensemble size.

Innovations in machine learning algorithms have enhanced the effectiveness of malware detection systems during the previous decades. However, the advancement of high throughput technologies results in high dimensional malware data, making feature selection useful and mandatory in such datasets. The feature selection technique is an information retrieval tool that aims to improve classifiers by listing important features, which also aids in reducing computational overload. However, different feature selection algorithms select representative features using various criteria, making it difficult to determine the optimal technique for distinct domain datasets. Ensemble feature selection approaches, which integrate the results of several feature selections, can be used to overcome the inadequacies of single-feature selection methods. Therefore, this paper attempts to determine whether the heterogeneous ensemble of filter and embedded feature selection approaches, namely the heterogenous ensemble of ANOVA F-test, ReliefF, L1-penalized logistic regression, LASSO regression, Extra-Tree Classifier and XGBoost feature selection techniques, namely HEFS-ARLLEX, can provide a better classification performance than the single feature selection technique and other ensemble feature selection approaches for malware classification data. The experimental results show that HEFS-ARLLEX, which combines both filters and embedded, is a better choice, providing consistently high classification accuracy, recall, precision, specificity and F-measure and a reasonable feature reduction rate for malware classification dataset.

Rainfall-runoff models play a very important role in flood forecasting. However, these models contain large uncertainties caused by errors in both the model itself and the input data. Data assimilation techniques are being used to reduce these uncertainties. The ensemble Kalman filter (EnKF) and the particle filter (PF) both have their own strengths. Research was carried out to a possible combination between both types of filters that will lead to a new type of filters that joins the strengths of both. The so called ensemble particle filter (EnPF) new combination is tested on flood forecasting problems in both the hindcast mode as well as the forecast mode. Several proposed combinations showed considerable improvement when a hindcast comparison on synthetic data was considered. Within the forecast comparison with field data, the suggested EnPF showed remarkable improvements compared to the PF and slight improvements compared to the EnKF.

Two techniques for estimating good localization functions for serial ensemble Kalman filters are compared in observing system simulation experiments (OSSEs) conducted with the dynamical core of an atmospheric general circulation model. The first technique, the global group filter (GGF), minimizes the root-mean-square (RMS) difference between the estimated regression coefficients using a hierarchical ensemble filter. The second, the empirical localization function (ELF), minimizes the RMS difference between the true values of the state variables and the posterior ensemble mean. Both techniques provide an estimate of the localization function for an observation’s impact on a state variable with few a priori assumptions about the localization function. The ELF localizations can have values larger than 1.0 at small distances, indicating that this technique addresses localization but also can correct the prior ensemble spread in the same way as a variance inflation when needed. OSSEs using ELF localizations generally have smaller root-mean-square error (RMSE) than the optimal Gaspari and Cohn (GC) localization function obtained by empirically tuning the GC width. The localization functions estimated by the GGF are broader than those from the ELF, and the OSSEs with the GGF localization generally have larger RMSE than the optimal GC localization function. The GGFs are too broad because of spurious correlation biases that occur in the OSSEs. These errors can be reduced by using a stochastic EnKF with perturbed observations instead of a deterministic EAKF.

<p>A multiscale alignment (MSA) method was proposed by Ying (2019) for ensemble data assimilation to reduce the errors caused by displacement of coherent features. The MSA method decomposes a model state into components ranging from large to small spatial scales, then applies ensemble filters to update each scale component sequentially. After a larger scale component analysis increment is derived from the observations, displacement vectors are computed from the analysis increments through an optical flow algorithm. These displacement vectors are then used to warp the model mesh, which reduces position errors in the smaller scale components before the ensemble filter is applied again.</p><p>The MSA method is now applied to a sea ice prediction problem at NERSC to assimilate satellite-derived sea ice deformation observations into the next generation Sea Ice Model (neXtSIM) simulations. Preliminary results show that the MSA can more effectively reduce the position errors of the linear kinematic features of sea ice than the traditional ensemble Kalman filter. The alignment step is shown to be a big contributor for error reduction in our test case. We will also discuss the remaining challenges of tuning parameters in the MSA method and dealing with model deficiencies.</p>

Uncertainty in cumulus convection parameterization is one of the most important causes of model climate drift through interactions between large-scale background and local convection that use empirically set parameters. Without addressing the large-scale feedback, the calibrated parameter values within a convection scheme are usually not optimal for a climate model. This study first designs a multiple-column atmospheric model that includes large-scale feedbacks for cumulus convection and then explores the role of large-scale feedbacks in cumulus convection parameter estimation using an ensemble filter. The performance of convection parameter estimation with or without the presence of large-scale feedback is examined. It is found that including large-scale feedbacks in cumulus convection parameter estimation can significantly improve the estimation quality. This is because large-scale feedbacks help transform local convection uncertainties into global climate sensitivities, and including these feedbacks enhances the statistical representation of the relationship between parameters and state variables. The results of this study provide insights for further understanding of climate drift induced from imperfect cumulus convection parameterization, which may help improve climate modeling.

Performance assessment of the maximum likelihood ensemble filter and the ensemble Kalman filters for nonlinear problems

Comparative evaluation of maximum likelihood ensemble filter and ensemble Kalman filter for real-time assimilation of streamflow data into operational hydrologic models

Coupled data assimilation uses a coupled model consisting of multiple time-scale media to extract information from observations that are available in one or more media. Because of the instantaneous exchanges of information among the coupled media, coupled data assimilation is expected to produce self-consistent and physically balanced coupled state estimates and optimal initialization for coupled model predictions. It is also expected that applying coupling error covariance between two media into observational adjustments in these media can provide direct observational impacts crossing the media and thereby improve the assimilation quality. However, because of the different time scales of variability in different media, accurately evaluating the error covariance between two variables residing in different media is usually very difficult. Using an ensemble filter together with a simple coupled model consisting of a Lorenz atmosphere and a pycnocline ocean model, which characterizes the interaction of multiple time-scale media in the climate system, the impact of the accuracy of coupling error covariance on the quality of coupled data assimilation is studied. Results show that it requires a large ensemble size to improve the assimilation quality by applying coupling error covariance in an ensemble coupled data assimilation system, and the poorly estimated coupling error covariance may otherwise degrade the assimilation quality. It is also found that a fast-varying medium has more difficulty being improved using observations in slow-varying media by applying coupling error covariance because the linear regression from the observational increment in slow-varying media has difficulty representing the high-frequency information of the fast-varying medium.

We investigate the issue of variational and sequential data assimilation with nonlinear and non‐smooth observation operators using a two‐dimensional limited‐area shallow‐water equation model and its adjoint. The performance of the four‐dimensional variational approach (4D‐Var: two dimensions plus time) compared with that of the maximum‐likelihood ensemble filter (MLEF), a hybrid ensemble/variational method, is tested in the presence of non‐smooth observation operators.Following the work of Lewis & Overton and Karmitsa, we investigate minimization of the data‐assimilation cost functional using the limited‐memory Broyden–Fletcher–Goldfarb–Shanno (L‐BFGS) quasi‐Newton algorithm originally intended for smooth optimization and the limited‐memory bundle method (LMBM) algorithm specifically designed to address large‐scale non‐smooth minimization problems.Numerical results obtained for the MLEF method show that the LMBM algorithm yields results superior to the L‐BFGS method. Results for 4D‐Var suggest that L‐BFGS performs well when the non‐smoothness is not extreme, but fails for non‐smooth functions with large Lipschitz constants. The LMBM method is found to be a suitable choice for large‐scale non‐smooth optimization, although additional work is needed to improve its numerical stability. Finally, the results and methodologies of 4D‐Var and MLEF are compared and contrasted. Copyright © 2011 Royal Meteorological Society