Ensemble filter methods with perturbed observations applied to nonlinear problems

Abstract. A new class of ensemble filters, called the Diffuse Ensemble Filter (DEnF), is proposed in this paper. The DEnF assumes that the forecast errors orthogonal to the first guess ensemble are uncorrelated with the latter ensemble and have infinite variance. The assumption of infinite variance corresponds to the limit of "complete lack of knowledge" and differs dramatically from the implicit assumption made in most other ensemble filters, which is that the forecast errors orthogonal to the first guess ensemble have vanishing errors. The DEnF is independent of the detailed covariances assumed in the space orthogonal to the ensemble space, and reduces to conventional ensemble square root filters when the number of ensembles exceeds the model dimension. The DEnF is well defined only in data rich regimes and involves the inversion of relatively large matrices, although this barrier might be circumvented by variational methods. Two algorithms for solving the DEnF, namely the Diffuse Ensemble Kalman Filter (DEnKF) and the Diffuse Ensemble Transform Kalman Filter (DETKF), are proposed and found to give comparable results. These filters generally converge to the traditional EnKF and ETKF, respectively, when the ensemble size exceeds the model dimension. Numerical experiments demonstrate that the DEnF eliminates filter collapse, which occurs in ensemble Kalman filters for small ensemble sizes. Also, the use of the DEnF to initialize a conventional square root filter dramatically accelerates the spin-up time for convergence. However, in a perfect model scenario, the DEnF produces larger errors than ensemble square root filters that have covariance localization and inflation. For imperfect forecast models, the DEnF produces smaller errors than the ensemble square root filter with inflation. These experiments suggest that the DEnF has some advantages relative to the ensemble square root filters in the regime of small ensemble size, imperfect model, and copious observations.

The ensemble Kalman filter (EnKF) is a data assimilation scheme based on the traditional Kalman filter update equation. An ensemble of forecasts are used to estimate the background-error covariances needed to compute the Kalman gain. It is known that if the same observations and the same gain are used to update each member of the ensemble, the ensemble will systematically underestimate analysis-error covariances. This will cause a degradation of subsequent analyses and may lead to filter divergence. For large ensembles, it is known that this problem can be alleviated by treating the observations as random variables, adding random perturbations to them with the correct statistics. Two important consequences of sampling error in the estimate of analysis-error covariances in the EnKF are discussed here. The first results from the analysis-error covariance being a nonlinear function of the backgrounderror covariance in the Kalman filter. Due to this nonlinearity, analysis-error covariance estimates may be negatively biased, even if the ensemble background-error covariance estimates are unbiased. This problem must be dealt with in any Kalman filter‐based ensemble data assimilation scheme. A second consequence of sampling error is particular to schemes like the EnKF that use perturbed observations. While this procedure gives asymptotically correct analysis-error covariance estimates for large ensembles, the addition of perturbed observations adds an additional source of sampling error related to the estimation of the observation-error covariances. In addition to reducing the accuracy of the analysis-error covariance estimate, this extra source of sampling error increases the probability that the analysis-error covariance will be underestimated. Because of this, ensemble data assimilation methods that use perturbed observations are expected to be less accurate than those which do not. Several ensemble filter formulations have recently been proposed that do not require perturbed observations. This study examines a particularly simple implementation called the ensemble square root filter, or EnSRF. The EnSRF uses the traditional Kalman gain for updating the ensemble mean but uses a ‘‘reduced’’ Kalman gain to update deviations from the ensemble mean. There is no additional computational cost incurred by the EnSRF relative to the EnKF when the observations have independent errors and are processed one at a time. Using a hierarchy of perfect model assimilation experiments, it is demonstrated that the elimination of the sampling error associated with the perturbed observations makes the EnSRF more accurate than the EnKF for the same ensemble size.

Bayesian estimation theory provides a general approach for the state estimate of linear or nonlinear and Gaussian or non-Gaussian systems. In this study, we first explore two Bayesian-based methods: ensemble adjustment Kalman filter (EAKF) and sequential importance resampling particle filter (SIR-PF), using a well-known nonlinear and non-Gaussian model (Lorenz '63 model). The EAKF, which is a deterministic scheme of the ensemble Kalman filter (EnKF), performs better than the classical (stochastic) EnKF in a general framework. Comparison between the SIR-PF and the EAKF reveals that the former outperforms the latter if ensemble size is so large that can avoid the filter degeneracy, and vice versa. The impact of the probability density functions and effective ensemble sizes on assimilation performances are also explored. On the basis of comparisons between the SIR-PF and the EAKF, a mixture filter, called ensemble adjustment Kalman particle filter (EAKPF), is proposed to combine their both merits. Similar to the ensemble Kalman particle filter, which combines the stochastic EnKF and SIR-PF analysis schemes with a tuning parameter, the new mixture filter essentially provides a continuous interpolation between the EAKF and SIR-PF. The same Lorenz '63 model is used as a testbed, showing that the EAKPF is able to overcome filter degeneracy while maintaining the non-Gaussian nature, and performs better than the EAKF given limited ensemble size.

Summary The ensemble Kalman filter (EnKF) is a subject of intensive investigation for use as a reservoir management tool. For strongly nonlinear problems, however, the EnKF can fail to achieve an acceptable data match at certain times in the data assimilation process. Here, we provide two iterative EnKF procedures to remedy this deficiency and explore the validity of these iterative methods compared to the standard EnKF by considering two examples. In both examples, we are able to obtain better data matches using iterative methods than with the standard EnKF. The simplest derivation of the EnKF analysis equation "linearizes" the objective function by adding the vector of predicted data to the original combined state vector of model parameters and dynamical variables. We show that there is no assurance that this trick for turning a nonlinear problem into a linear problem results in a correct sampling of the probability density function (pdf) one wishes to sample. However, we show that augmenting the state vector with the data results in a correct procedure for sampling the pdf if, at every data assimilation step, the predicted data vector is a linear function of the combined (unaugmented) state vector. Without this assumption, we know of no way to show EnKF samples correctly.

Particle filters (PFs) are Monte Carlo data assimilation techniques that operate with no parametric assumptions for prior and posterior errors. A data assimilation method introduced recently, called the local PF, approximates the PF solution within neighborhoods of observations, thus allowing for its use in high-dimensional systems. The current study explores the potential of the local PF for atmospheric data assimilation through cloud-permitting numerical experiments performed for an idealized squall line. Using only 100 ensemble members, experiments using the local PF to assimilate simulated radar measurements demonstrate that the method provides accurate analyses at a cost comparable to ensemble filters currently used in weather models. Comparisons between the local PF and an ensemble Kalman filter demonstrate benefits of the local PF for producing probabilistic analyses of non-Gaussian variables, such as hydrometeor mixing ratios. The local PF also provides more accurate forecasts than the ensemble Kalman filter, despite yielding higher posterior root-mean-square errors. A major advantage of the local PF comes from its ability to produce more physically consistent posterior members than the ensemble Kalman filter, which leads to fewer spurious model adjustments during forecasts. This manuscript presents the first successful application of the local PF in a weather prediction model and discusses implications for real applications where nonlinear measurement operators and nonlinear model processes limit the effectiveness of current Gaussian data assimilation techniques.

Comparative evaluation of maximum likelihood ensemble filter and ensemble Kalman filter for real-time assimilation of streamflow data into operational hydrologic models

Using radar observations, the performances of the ensemble square root filter (EnSRF) and an indirect three-dimensional variational (3DVar) data assimilation method were compared for a mesoscale convective system (MCS) that occurred in the Front Range of the Rocky Mountains, Colorado (USA). The results showed that the root mean square innovations (RMSIs) of EnSRF were lower than 3DVar for radar reflectivity and radial velocity and that the spread of EnSRF was generally consistent with its RMSIs. EnSRF substantially improved the analysis of the MCS compared with an experiment without radar data assimilation, and it produced a slight but noticeable improvement over 3DVar in terms of both coverage and intensity. Forecast results initiated from the final analysis revealed that EnSRF generally produced the best prediction of the MCS, with improved quantitative reflectivity and precipitation forecast skills. EnSRF also demonstrated better performance than 3DVar in the prediction of neighborhood probability for reflectivity at thresholds of 20 and 35 dBZ, which better matched the observed radar reflectivity in terms of both shape and extension. Additionally, the humidity, temperature, and wind fields were also improved by EnSRF; the largest error reduction was found in the water vapor field near the surface and at upper levels.

The stochastic ensemble Kalman filter (EnKF) updates its ensemble members with observations perturbed with noise sampled from the distribution of the observational errors. This was shown to introduce noise into the system and may become pronounced when the ensemble size is smaller than the rank of the observational error covariance, which is often the case in real oceanic and atmospheric data assimilation applications. This work introduces an efficient serial scheme to mitigate the impact of observations’ perturbations sampling in the analysis step of the EnKF, which should provide more accurate ensemble estimates of the analysis error covariance matrices. The new scheme is simple to implement within the serial EnKF algorithm, requiring only the approximation of the EnKF sample forecast error covariance matrix by a matrix with one rank less. The new EnKF scheme is implemented and tested with the Lorenz-96 model. Results from numerical experiments are conducted to compare its performance with the EnKF and two standard deterministic EnKFs. This study shows that the new scheme enhances the behavior of the EnKF and may lead to better performance than the deterministic EnKFs even when implemented with relatively small ensembles.

This paper considers implications of different forms of the ensemble transformation in the ensemble square root filters (ESRFs) for the performance of ESRF-based data assimilation systems. It highlights the importance of using mean-preserving solutions for the ensemble transform matrix (ETM). The paper shows that an arbitrary mean-preserving ETM can be represented as a product of the symmetric solution and an orthonormal mean-preserving matrix. The paper also introduces a new flavor of ESRF, referred to as ESRF with mean-preserving random rotations. To investigate the performance of different solutions for the ETM in ESRFs, experiments with two small models are conducted. In these experiments, the performances of two mean-preserving solutions, two non-mean-preserving solutions, and a traditional ensemble Kalman filter with perturbed observations are compared. The experiments show a significantly better performance of the mean-preserving solutions for the ETM in ESRFs compared to non-mean-preserving solutions. They also show that applying the mean-preserving random rotations prevents the buildup of ensemble outliers in ESRF-based data assimilation systems.

Multiphase flow in porous media is involved in various natural and industrial applications, including water infiltration into soils, carbon geosequestration, and underground hydrogen storage. Understanding the invasion morphology at the pore scale is critical for better prediction of flow properties at the continuum scale in partially saturated permeable media. The deep learning method, as a promising technique to estimate the flow transport processes in porous media, has gained significant attention. However, existing works have mainly focused on single-phase flow, whereas the capability of data-driven techniques has yet to be applied to the pore-scale modeling of fluid–fluid displacement in porous media. Here, the conditional generative adversarial network is applied for pore-scale modeling of multiphase flow in two-dimensional porous media. The network is trained based on a data set of porous media generated using a particle-deposition method, with the corresponding invasion morphologies after the displacement processes calculated using a recently developed interface tracking algorithm. The results demonstrate the capability of data-driven techniques in predicting both fluid saturation and spatial distribution. It is also shown that the method can be generalized to estimate fluid distribution under different wetting conditions and particle shapes. This work represents the first effort at the application of the deep learning method for pore-scale modeling of immiscible fluid displacement and highlights the strength of data-driven techniques for surrogate modeling of multiphase flow in porous media.

In this study, a deterministic algorithm named Ensemble Square Root Filter (EnSRF), an algorithm significantly improved from the Ensemble Kalman Filter (EnKF), was used to integrate remotely sensed information (ASD spectral data, HJ-1 A/B CCD and Landsat-5 TM data) with a wheat (Triticum aestivum L.) growth model (WheatGrow). The analyzed values of model variables, leaf area index (LAI) and leaf nitrogen accumulation (LNA), were calculated based on EnSRF without perturbed measurements. Independent datasets were used to test EnSRF and the root mean square error (RMSE) values were 0.81 and 0.82 g m-2, with relative error (RE) values of 0.15 and 0.13, for LAI and LNA, respectively. RMSE values for LAI and LNA were 1.39 and 1.70 g m-2, respectively (RE, 0.28 and 0.34) based on EnKF, 1.17 and 1.80 g m-2 (RE, 0.24 and 0.35), respectively, based on the WheatGrow model alone, and 0.97 and 1.25 g m-2 (RE, 0.21 and 0.24), respectively, based on the remote sensing models. These results indicated that the LAI and LNA values based on EnSRF matched the measured values well compared with the EnKF, WheatGrow and remote sensing models. In addition, the predicted results are consistent with the temporal and spatial distribution of winter wheat growth status and grain yields in the study area, with RE values of less than 0.2 and 0.1 for LAI and LNA, respectively. These results provide an important approach for simulating winter wheat growth status based on combining remote sensing and crop growth models.

The iterative ensemble Kalman filter (IEnKF) was recently proposed in order to improve the performance of ensemble Kalman filtering with strongly nonlinear geophysical models. The IEnKF can be used as a lag‐one smoother and extended to a fixed‐lag smoother: the iterative ensemble Kalman smoother (IEnKS). The IEnKS is an ensemble variational method. It does not require the use of the tangent linear of the evolution and observation models, nor the adjoint of these models: the required sensitivities (gradient and Hessian) are obtained from the ensemble. Looking for optimal performance, out of the many possible extensions we consider a quasi‐static algorithm. The IEnKS is explored for the Lorenz '95 model and for a two‐dimensional turbulence model. As the logical extension of the IEnKF, the IEnKS significantly outperforms standard Kalman filters and smoothers in strongly nonlinear regimes. In mildly nonlinear regimes (typically synoptic‐scale meteorology), its filtering performance is marginally but clearly better than the standard ensemble Kalman filter and it keeps improving as the length of the temporal data assimilation window is increased. For long windows, its smoothing performance outranks the standard smoothers very significantly, a result that is believed to stem from the variational but flow‐dependent nature of the algorithm. For very long windows, the use of a multiple data assimilation variant of the scheme, where observations are assimilated several times, is advocated. This paves the way for finer reanalysis, freed from the static prior assumption of 4D‐Var but also partially freed from the Gaussian assumptions that usually impede standard ensemble Kalman filtering and smoothing.

A two-stage ensemble Kalman filter based on multiscale model reduction for inverse problems in time fractional diffusion-wave equations

The multiphase fluid flow in porous media is one of the most fundamental phenomena in various physical processes, such as oil/gas flow in reservoir, subsurface contamination dispersion, chemical separation, etc. Due to its importance, the efficient and accurate solution and prediction of multiphase flow in porous media is highly required in engineering applications and mechanism studies, which has been a research hot spot with increasing interest in recent years. However, the strong nonlinearity implicated in the multiphase flow model has brought great challenges for the computation and analysis. In addition, the permeability in Darcy-type pressure equation is always represented as a high contrast coefficient, which further complicates the simulation difficulty especially for high fidelity prediction and decision-making process. In this study, we attempt to apply one popular global model reduction method, namely Proper Orthogonal Decomposition (POD), to accelerate the simulation of incompressible multiphase flow in porous media. The cornerstone of POD is processing information from a sequence of snapshots (or instantaneous solutions) to identify a set of basis functions to construct a low-dimensional space, the most energetic part of these basis functions is then selected to represent the solution of the original system. In our work, one key point is how to establish the accurate POD reduced order model (ROM) for incompressible mul-tiphase porous flow problems. According to Li et al. [1], the traditional Galerkin projection method (directly project the original governing equation into the low-dimensional space spanned by basis functions) is incapable of introducing the fluid saturation and permeability located on the cell face in POD-ROM for incompressible multiphase flow whether in homogeneous or heterogeneous porous media, thus we adopt the matrix operation method (project the discrete governing equation instead of the original governing equation into the low-dimensional space spanned by basis functions) to build the POD-ROM in this study. Based on the proposed POD-ROM framework, we use the implicit method to solve the (non-) wetting-phase pressure and saturation equations. However, in the implicit solution of the pressure and saturation equations, the standard POD-ROM alone sometimes cannot yield an expected acceleration owing to two main reasons: (1) the nonlinearity between pressure and saturation is sensitive, thus the saturation would change largely even with slight pressure change. And the small error of pressure results would lead to a large deviation of saturation, which in turn exerts negative impacts on the computation of pressure; (2) the complexity for computing a projected nonlinear term in POD-ROM still depends on the dimension of the original full-order system, it is unavoidable in conventional POD-ROM. Therefore, more POD modes are needed to be used in order to achieve the prescribed numerical accuracy, which could worsen the speed up efficiency of proposed POD-ROM. To further reduce the computational complexity of POD-ROM to speed up the implicit solution of pressure and saturation equations, another model reduction method called Discrete Empirical Interpolation Method (DEIM) [2] is applied to treat the projected nonlinear term in the proposed incompressible multiphase porous flow POD-ROM. Different from previous studies, we develop an adaptive DEIM (ADEIM) in this work to achieve a more flexible selection of the interpolation points. Combined the POD-ROM with ADEIM, a POD-ADEIM-ROM for incompressible multiphase flow in porous media is established. The computational efficiency and numerical accuracy of the proposed model is validated through several representative numerical examples. Numerical results indicate that an outstanding acceleration (2~3 orders of magnitude faster) without sacrificing numerical accuracy significantly is obtained from the proposed model when comparing to the traditional solution method that without any acceleration technique. In addition, the effects of POD mode number, the distribution of permeability field, and the mesh density on the overall performances of the proposed model are investigated in detail.

A localized ensemble Kalman filter (EnKF) method is developed to assimilate transient flow data to calibrate a heterogeneous conductivity field. To update conductivity value at a point in a study domain, instead of assimilating all the measurements in the study domain, only limited measurement data in an area around the point are used for the conductivity updating in the localized EnKF method. The localized EnKF is proposed to solve the problems of the filter divergence usually existing in a data assimilation method without localization. The developed method is applied, in a synthetical two dimensional case, to calibrate a heterogeneous conductivity field by assimilating transient hydraulic head data. The simulations by the data assimilation with and without localized EnKF are compared. The study results indicate that the hydraulic conductivity field can be updated efficiently by the localized EnKF, while it cannot be by the EnKF. The covariance inflation and localization are found to solve the problem of the filter divergence efficiently. In comparison with the EnKF method without localization, the localized EnKF method needs smaller ensemble size to achieve stabilized results. The simulation results by the localized EnKF method are much more sensitive to conductivity correlation length than to the localization radius. The developed localized EnKF method provides an approach to improve EnKF method in conductivity calibration.