- Research Article
48
- 10.1002/hyp.9679
Hydrological ensemble prediction systems
- Dec 21, 2012
- Hydrological Processes
- Hannah L Cloke + 1 more +1
[Departement_IRSTEA]Eaux [TR1_IRSTEA]ARCEAU
Numerical weather prediction models have systematic errors and biases, which can be reduced through statistical correction methods. The most intuitive approach to correcting weather forecasts is to establish a statistical relationship between the forecast and the corresponding observations. Once the relationship is established, it can be used to correct future forecasts. These approaches, also known as statistical adaptation, are used on a daily basis to improve the quality of operational forecasts. Many methods have been proposed in the literature to calibrate ensemble forecasts, and these can be divided into two main groups: parametric (that has a hypothesis on the underlying distribution) and non-parametric methods. One of the most widespread and used methods of the parametric group is the so-called Ensemble Model Output Statistics (EMOS). We show the performance of EMOS for the post-processing of probabilistic temperature forecasts at 2m. This method assumes that the underlying distribution is Gaussian (in the case of 2m T), and the parameters to be estimated are therefore the mean and the standard deviation. First results were obtained using the ECMWF operational forecast ensemble against SYNOP observations. Further studies were carried out to investigate the reliability of the estimated parameters using not only the raw ensemble data, but also the ECMWF deterministic operational forecast HRES and the control member. The results showed an improvement, especially for the shorter lead-times. The improvement was measured using weather scores for forecast verification and performance: bias, CRPS (Continuous Ranked Probability Score) and CRPSS (Continuous Ranked Probability Skill Score). These post-processed forecasts will serve as a reference for future studies.
Hydrological ensemble prediction systems
[Departement_IRSTEA]Eaux [TR1_IRSTEA]ARCEAU
Quantile regression forests for post-processing ECWMF ensemble precipitation forecasts: hyperparameter optimization and comparison to EMOS
Ensemble forecasts are important due to their ability to characterize forecast uncertainty, which is fundamental when forecasting extreme weather. Ensemble forecasts are however often biased and underdispersed and thus need to be post-processed.A common approach for this is the use of ensemble model output statistics (EMOS), where a parametric distribution is fitted with a limited number of predictors. With recent advances in computer science and increased amounts of data available, machine learning techniques, like random forests, are becoming more popular for high dimensional regression problems. In this research, we explore the use of the quantile regression forest (QRF), a random forest adapted for conditional quantile estimation, applied to medium range gridded probabilistic precipitation forecasts. QRFs are non-parametric and allow for a larger number of predictors, which means they can possibly consider more dependencies that might otherwise not be captured with a simple EMOS.A QRF takes several hyperparameters that influence the way the decision trees in the forest are constructed. We explore the minimum number of samples needed in a leaf to split it (minimum node size) and the number of predictors explored in each split (mtry). A hyperparameter space is constructed by setting ranges for both minimum node size and mtry, and the optimal hyperparameter set is determined by performing a cross validated grid search. Here, each model is assessed based on the continuous ranked probability skill score (CRPSS). For comparison, EMOS is applied with a zero-adjusted gamma (ZAGA) distribution, using a limited number of predictors that are physically correlated to precipitation. Both methods are verified on a separate testing data set and evaluated using several scores, including CRPSS and Brier skills score (BSS).We consider 4 years (November 2018 – October 2022) of archived operational ECMWF-IFS ensemble forecasts for the Netherlands. The data is split into November 2018 – October 2021 for training and cross-validation, and October 2021 – October 2022 for testing, separating data for season, initialization time and lead-time. Forecasts are post-processed up to +10 days. Ensemble statistics on 60+ forecast variables are used as predictors. Spatially and temporally aggregated, gauge-adjusted radar observations are used as predictand. The raw ensemble is considered as the benchmark.The results of this research will determine what method will be used to post-process the ensemble precipitation forecasts in the context of the early warning center (EWC) of the Royal Netherlands Meteorological Institute. The most suitable method could differ between shorter and longer lead times.
Read moreAn objective Bayesian method for including parameter uncertainty in ensemble model output statistics
Ensemble forecasts are often calibrated using regression methods known as ensemble model output statistics (EMOS). Most of the EMOS methods described in the literature make the assumption that statistical parameter uncertainty is negligible, although making this assumption has been shown to be detrimental to forecast quality. We use simulations to show that this assumption would be expected to give unreliable forecasts and underprediction of extremes. Previous research has described how parameter uncertainty can be included in EMOS models using bootstrapping. We show that bootstrapping can be replaced with an analytical approach based on right Haar prior (RHP) theory, a branch of objective Bayesian statistics. For two commonly used EMOS models, simple linear regression (SLR) and heteroscedastic linear regression (HLR, also known as non‐homogeneous Gaussian regression), we show that RHP theory leads to perfectly reliable predictions, if the distributional assumptions are met. We apply SLR and HLR methods, with and without parameter uncertainty, to 5 years of past medium‐range forecasts for eight Northern Hemisphere locations and evaluate forecast accuracy and reliability. We find that the RHP version of HLR is the most accurate in terms of both log‐score and continuous ranked probability score. It is also the most reliable, and it reduces the underestimation of extremes seen in the other methods. Our results with regard to the benefits of including parameter uncertainty agree with the previous results based on bootstrapping, although the RHP calculations are faster and more accurate. A free software library is available for applying the various methods, and execution is computationally inexpensive. We conclude that the RHP version of HLR should be used in preference to the other methods tested. Further research would be needed to understand how to use objective Bayesian methods to include parameter uncertainty in other EMOS models.
Read moreComparison of the BMA and EMOS statistical methods for probabilistic quantitative precipitation forecasting
The main approach to probabilistic weather forecasting has been the use of ensemble forecasting. In ensemble forecasting, the probability information is generally derived by using several numerical model runs, with perturbation of the initial conditions, physical schemes or dynamic core of the numerical weather prediction (NWP) models. However, ensemble forecasting usually tends to be under‐dispersive. Statistical post‐processing has, therefore, become an essential component of any ensemble prediction system aiming to improve the quality of numerical weather forecasts as they seek to generate calibrated and sharp predictive distributions of future weather quantities. Different versions of the ensemble model output statistics (EMOS) and the Bayesian model averaging (BMA) post‐processing methods are used in the present paper to calibrate 24, 48 and 72 hr forecasts of 24 hr accumulative precipitation. The ensemble employs the weather and research forecasting (WRF) model with eight different configurations which were run over Iran for six months (September 2015–February 2016). The results reveal that the BMA and EMOS‐censored, shifted gamma (CSG) techniques are substantially successful at improving the raw WRF ensemble forecasts; however, each approach improves different aspects of the forecast quality. The BMA method is more accurate, skilful and reliable than the EMOS‐CSG method, but has poorer discrimination. Moreover, it has better resolution in predicting the probability of high‐precipitation events than the EMOS‐CSG method.
Read moreSequential Aggregation of Probabilistic Forecasts—Application to Wind Speed Ensemble Forecasts
In numerical weather prediction (NWP), the uncertainty about the future state of the atmosphere is described by a set of forecasts (called an ensemble). All ensembles have deficiencies that can be corrected via statistical post-processing methods. Several ensembles, based on different NWP models, exist and may be corrected using different statistical methods. These raw or post-processed ensembles can thus be combined. The theory of prediction with expert advice allows us to build combination algorithms with theoretical guarantees on the forecast performance. We adapt this theory to the case of probabilistic forecasts issued as stepwise cumulative distribution functions, computed from raw and post-processed ensembles. The theory is applied to combine wind speed ensemble forecasts. The second goal of this study is to explore the use of two forecast performance criteria: the continuous ranked probability score (CRPS) and the Jolliffe–Primo test. The usual way to build skilful probabilistic forecasts is to minimize the CRPS. Minimizing the CRPS may not produce reliable forecasts according to the Jolliffe–Primo test. The Jolliffe–Primo test generally selects reliable forecasts, but could lead to issuing suboptimal forecasts in terms of CRPS. We propose to use both criteria to achieve reliable and skilful probabilistic forecasts.
Read moreBivariate ensemble model output statistics approach for joint forecasting of wind speed and temperature
Forecast ensembles are typically employed to account for prediction uncertainties in numerical weather prediction models. However, ensembles often exhibit biases and dispersion errors, thus they require statistical post-processing to improve their predictive performance. Two popular univariate post-processing models are the Bayesian model averaging (BMA) and the ensemble model output statistics (EMOS). In the last few years increased interest has emerged in developing multivariate post-processing models, incorporating dependencies between weather quantities, such as for example a bivariate distribution for wind vectors or even a more general setting allowing to combine any types of weather variables. In line with a recently proposed approach to model temperature and wind speed jointly by a bivariate BMA model, this paper introduces a bivariate EMOS model for these weather quantities based on a truncated normal distribution. The bivariate EMOS model is applied to temperature and wind speed forecasts of the eight-member University of Washington mesoscale ensemble and of the eleven-member ALADIN-HUNEPS ensemble of the Hungarian Meteorological Service and its predictive performance is compared to the performance of the bivariate BMA model and a multivariate Gaussian copula approach, post-processing the margins with univariate EMOS. While the predictive skills of the compared methods are similar, the bivariate EMOS model requires considerably lower computation times than the bivariate BMA method.
Read moreExtended-range forecasting of stream water temperature with deep-learning models
Abstract. Stream water temperatures influence water quality, with effects on aquatic biodiversity, drinking-water provision, electricity production, agriculture, and recreation. Therefore, stakeholders would benefit from an operational forecasting service that would support timely action. Deep-learning models are well-suited to providing probabilistic forecasts at individual stations of a monitoring network. Here, we train and evaluate several state-of-the-art models using 10 years of data from 54 stations across Switzerland. Static catchment features, time of the year, meteorological observations from the past 64 d, and their ensemble forecasts for the following 32 d are included as predictors in the models to estimate daily maximum water temperature over the next 32 d. Results show that the temporal fusion transformer (TFT) model performs best, with a continuous rank probability score (CRPS) of 0.70 °C averaged over all lead times, stations, and 90 forecasts distributed over 1 year. The TFT is followed by the recurrent neural network encoder–decoder, with a CRPS of 0.74 °C, and the neural hierarchical interpolation for time series, with a CRPS of 0.75 °C. These deep-learning models outperform other simpler models trained at each station: random forest (CRPS = 0.80 °C), multi-layer perceptron neural network (CRPS = 0.81 °C), and autoregressive linear model (CRPS = 0.96 °C). The average CRPS of the TFT degrades from 0.38 °C at lead a time of 1 d to 0.90 °C at a lead time of 32 d, largely driven by the uncertainty of the meteorological ensemble forecasts. In addition, TFT water temperature predictions at new and ungauged stations outperform those from the other models. When analyzing the importance of model inputs, we find a dominant role of observed water temperature and future air temperature, while including precipitation and time of the year further improves predictive skill. Operational probabilistic forecasts of daily maximum water temperature are generated twice per week with our TFT model and are publicly available at https://www.drought.ch/de/impakt-vorhersagen-malefix/wassertemperatur-prognosen/ (last access: 20 March 2025). Overall, this study provides insights into the extended-range predictability of stream water temperature and into the applicability of deep-learning models in hydrology.
Read moreTrends in the predictive performance of raw ensemble weather forecasts
This study applies statistical postprocessing to ensemble forecasts of near‐surface temperature, 24 h precipitation totals, and near‐surface wind speed from the global model of the European Centre for Medium‐Range Weather Forecasts (ECMWF). The main objective is to evaluate the evolution of the difference in skill between the raw ensemble and the postprocessed forecasts. Reliability and sharpness, and hence skill, of the former is expected to improve over time. Thus, the gain by postprocessing is expected to decrease. Based on ECMWF forecasts from January 2002 to March 2014 and corresponding observations from globally distributed stations, we generate postprocessed forecasts by ensemble model output statistics (EMOS) for each station and variable. Given the higher average skill of the postprocessed forecasts, we analyze the evolution of the difference in skill between raw ensemble and EMOS. This skill gap remains almost constant over time indicating that postprocessing will keep adding skill in the foreseeable future.
Read moreA comparison of two statistical postprocessing methods for heavy‐precipitation forecasts over India during the summer monsoon
Accurate ensemble forecasts of heavy precipitation in India are vital for many applications and essential for early warning of damaging flood events, especially during the monsoon season. In this study we investigate to what extent Quantile Mapping (QM) and Ensemble Model Output Statistics (EMOS) statistical postprocessing reduce errors in precipitation ensemble forecasts over India, in particular for heavy precipitation. Both methods are applied to day‐1 forecasts at 12‐km resolution from the 23‐member National Centre for Medium Range Weather Forecasting (NCMRWF) global ensemble prediction system (NEPS‐G). By construction, QM leads to distributions close to the observed ones, while EMOS optimizes the ensemble spread, and it is not a priori clear which is better suited for practical applications. The methods are therefore compared with respect to several key aspects of the forecasts: local distributions, ensemble spread, and skill for forecasting precipitation amounts and the exceedance of heavy precipitation thresholds. The evaluation includes rank histograms, Continuous Ranked Probability Skill Scores (CRPSS), Brier Skill Scores (BSS), reliability diagrams, and receiver operating characteristic. EMOS performs best not only with respect to correcting under‐ or overdispersive ensembles, but also in terms of forecast skill for precipitation amounts and heavy precipitation events, with positive CRPSS and BSS in most regions (both up to about 0.4 in some areas), while QM in many regions performs worse than the raw forecast. QM performs best with respect to the overall local precipitation distributions. Which aspects of the forecasts are most relevant depends to some extent on how the forecasts are used. If the main criteria are the correction of under‐ or overdispersion, forecast reliability, match between the forecasted distribution for individual days and observations (CRPSS), and the skill in forecasting heavy‐precipitation events (BSS), then EMOS is the better choice for postprocessing NEPS‐G forecasts for short lead times.
Read moreCalibrated Ensemble Forecasts Using Quantile Regression Forests and Ensemble Model Output Statistics
Ensembles used for probabilistic weather forecasting tend to be biased and underdispersive. This paper proposes a statistical method for postprocessing ensembles based on quantile regression forests (QRF), a generalization of random forests for quantile regression. This method does not fit a parametric probability density function (PDF) like in ensemble model output statistics (EMOS) but provides an estimation of desired quantiles. This is a nonparametric approach that eliminates any assumption on the variable subject to calibration. This method can estimate quantiles using not only members of the ensemble but any predictor available including statistics on other variables. The method is applied to the Météo-France 35-member ensemble forecast (PEARP) for surface temperature and wind speed for available lead times from 3 up to 54 h and compared to EMOS. All postprocessed ensembles are much better calibrated than the PEARP raw ensemble and experiments on real data also show that QRF performs better than EMOS, and can bring a real gain for human forecasters compared to EMOS. QRF provides sharp and reliable probabilistic forecasts. At last, classical scoring rules to verify predictive forecasts are completed by the introduction of entropy as a general measure of reliability.
Read moreUnderstanding changes of the continuous ranked probability score using a homogeneous Gaussian approximation
Improving ensemble forecasts is a complex process which involves proper scores such as the continuous ranked probability score (CRPS). A homogeneous Gaussian (hoG) model is introduced in order to better understand the characteristics of the CRPS. An analytical formula is derived for the expected CRPS of an ensemble in the hoG model. The score is a function of the variance of the error of the ensemble mean, the mean error of the ensemble mean and the ensemble variance. The hoG model also provides a score decomposition into reliability and resolution components. We examine whether the hoG model provides a useful approximation of the CRPS when applied to operational ECMWF medium‐range ensemble forecasts. The hoG approximation describes the spatial variations of the CRPS well while moderately overestimating the mean score. Seasonal averages over large domains are within 10% of the actual CRPS. Furthermore, the ability to approximate score changes is evaluated by (a) comparing raw ensemble forecasts with postprocessed ensemble forecasts, and (b) by examining score changes associated with a recent upgrade of the IFS. Overall, the hoG approximation predicts the actual CRPS changes well. One of the main anticipated applications of the hoG approximation are new diagnostics in verification software used by NWP developers routinely. The purpose of the diagnostics is to help developers explain impacts of forecast system changes on the CRPS in terms of the changes in mean error, changes in error variance and changes in ensemble variance. The diagnostics require little additional computational resources compared to the alternative of verifying postprocessed versions of the ensemble forecasts. Therefore, it will be feasible to apply the diagnostics easily to all variables that are examined as part of the model development process.
Read moreProgress in ensemble forecasting and verification methodologies at ECMWF
<p>This talk focusses on progress in ensemble forecasting methodology (Part I) and ensemble verification methodology (Part II).</p><p>Operational ECMWF ensemble forecasts are global predictions from days to months ahead. At all forecast ranges, model uncertainties are represented stochastically with the Stochastically Perturbed Parametrization Tendency scheme (SPPT). Recently, considerable progress has been made in developing the Stochastically Perturbed Parametrization scheme (SPP). The SPP scheme offers improved physical consistency by naturally preserving the local conservation properties for energy and moisture of the unperturbed version of the corresponding parametrization. In contrast, the SPPT scheme lacks such local conservation properties, mainly because the scheme does not perturb fluxes at the surface and at the top of the atmosphere consistently with the tendency perturbations in the column.</p><p>NWP research and development relies on scoring rules to judge whether or not a change to the forecast systems results in better ensemble forecasts. A new tool will be presented that can improve the understanding of score differences between sets of forecasts for a widely used proper score, the Continuous Ranked Probability Score (CRPS). An analytical expression has been derived for the CRPS when a homogeneous Gaussian (hoG) forecast-observation distribution is considered. This leads to an approximation of the CRPS when actual verification data are considered, which deviate from a homogeneous Gaussian distribution. The hoG approximation of the CRPS permits a useful decomposition of score differences. The methodology will be illustrated with verification data for medium-range weather forecasts.</p>
Read moreAssessment of the skill of seasonal probabilistic hydrological forecasts with ParFlow/CLM over central Europe
In recent years, alternating drought and extreme precipitation events have highlighted the need for subseasonal to seasonal forecasts of the terrestrial water cycle. In particular, predictions of the impacts of dry and wet extremes on subsurface water resources are crucial to provide stakeholders in agriculture, forestry, the water sector, and other fields with information supporting the sustainable use of these resources.In this context, we release an experimental Water Resources Bulletin (https://adapter-projekt.de/bulletin/index.html) four times per year, offering probabilistic forecasts of the total subsurface water storage (TSS) anomaly at a 0.6 km resolution, from the surface down to 60 m depth, for the upcoming seven months across Germany. These seasonal forecasts are generated using the integrated, physics-based hydrological model ParFlow/CLM, forced by 50 ensemble members of the SEAS5 seasonal forecast from the European Centre for Medium-Range Weather Forecasts (ECMWF).To evaluate our forecasts, we evaluated six 7-months probabilistic forecasts covering the vegetation period (March to September) for the years 2018 to 2023 with a reference long-term historical time series based on the same ParFlow/CLM setup. The forecast skill was assessed by comparing these seasonal forecasts to a climatology-based 10-member pseudo-forecast over the 2013–2023 period (using the leave-one-out method), extracted from the reference time series.The monthly Continuous Ranked Probability Skill Score (CRPSS), which evaluates the ensemble distribution based on daily TSS data, indicates that the probabilistic forecast outperforms the climatology-based pseudo-forecast in most regions, except in 2018 and, to a lesser extent, in 2020 and 2022. This can be attributed to an under-representation of extremely dry members in the ensemble, combined with the memory effect of the initial conditions at increasing soil depths. For example, while March 2018 started with a slightly above-average TSS and experienced a strong meteorological drought leading to an agricultural drought, the initial TSS anomaly in March 2019 was already negative, with a less pronounced precipitation deficit during the vegetation period. This resulted in a much higher forecast skill, because of the memory effect accurately simulated with the physics-based model. Notably, the forecast skill only slightly decreases with increasing lead time, both for precipitation and TSS.The analysis of the Relative Operating Characteristic Skill Score (ROCSS) for the lower quintile of the TSS distribution assesses whether negative TSS anomalies (i.e., droughts) are adequately represented within the probabilistic forecast ensemble. The results are consistent with those of the CRPSS, showing lower skill in 2018. Nevertheless, the ROCSS analysis overall indicates moderate to high skill for the probabilistic forecast, while the climatology-based pseudo-forecast demonstrates no skill. This confirms that the dry conditions experienced in central Europe in recent years were captured within the probabilistic forecast, underlining the added value of these forecasts and their usefulness in the experimental Water Resources Bulletin.
Read moreWeatherBench Probability: Medium-range weather forecasts with probabilistic machine learning methods.
<p>Because the atmosphere is inherently chaotic, probabilistic weather forecasts are crucial to provide reliable information. In this work, we present an extension to the WeatherBench, a benchmark dataset for medium-range, data-driven weather prediction, which was originally designed for deterministic forecasts. We add a set of commonly used probabilistic verification metrics: the spread-skill ratio, the continuous ranked probability score (CRPS) and rank histograms. Further, we compute baseline scores from the operational IFS ensemble forecast. </p><p>Then, we compare three different methods of creating probabilistic neural network forecasts: first, using Monte-Carlo dropout during inference with a range of dropout rates; second, parametric forecasts, which optimize for the CRPS; and third, categorical forecasts, in which the probability of occurrence for specific bins is predicted. We show that plain Monto-Carlo dropout does not provide enough spread. The parametric and categorical networks, on the other hand, provide reliable forecasts, with the categorical method being more versatile.</p>
Read moreThe Continuous Ranked Probability Score for Circular Variables and its Application to Mesoscale Forecast Ensemble Verification
An analogue of the linear continuous ranked probability score is introduced that applies to probabilistic forecasts of circular quantities.This scoring rule is proper and thereby discourages hedging.The circular continuous ranked probability score reduces to angular distance when the forecast is deterministic, just as the linear continuous ranked probability score generalizes the absolute error.Furthermore, the continuous ranked probability score provides a direct way of comparing deterministic forecasts, discrete forecast ensembles, and post-processed forecast ensembles that can take the form of probability density functions.The circular continuous ranked probability score is used in this study to assess predictions of 10 m wind direction for 361 cases of mesoscale, short-range ensemble forecasts over the North American Pacific Northwest.Reference probability forecasts based on the ensemble mean and its forecast error history over the period outperform probability forecasts constructed directly from the ensemble sample statistics.These results suggest that short-term forecast uncertainty is not yet well predicted at mesoscale resolutions near the surface, despite the inclusion of multi-scheme physics diversity and surface boundary parameter perturbations in the mesoscale ensemble design.
Read more