Quantile regression forests for post-processing ECWMF ensemble precipitation forecasts: hyperparameter optimization and comparison to EMOS

Ensembles used for probabilistic weather forecasting tend to be biased and underdispersive. This paper proposes a statistical method for postprocessing ensembles based on quantile regression forests (QRF), a generalization of random forests for quantile regression. This method does not fit a parametric probability density function (PDF) like in ensemble model output statistics (EMOS) but provides an estimation of desired quantiles. This is a nonparametric approach that eliminates any assumption on the variable subject to calibration. This method can estimate quantiles using not only members of the ensemble but any predictor available including statistics on other variables. The method is applied to the Météo-France 35-member ensemble forecast (PEARP) for surface temperature and wind speed for available lead times from 3 up to 54 h and compared to EMOS. All postprocessed ensembles are much better calibrated than the PEARP raw ensemble and experiments on real data also show that QRF performs better than EMOS, and can bring a real gain for human forecasters compared to EMOS. QRF provides sharp and reliable probabilistic forecasts. At last, classical scoring rules to verify predictive forecasts are completed by the introduction of entropy as a general measure of reliability.

Bias correction of ensemble precipitation forecasts in the improvement of summer streamflow prediction skill

[Departement_IRSTEA]Eaux [TR1_IRSTEA]ARCEAU

This study applies statistical postprocessing to ensemble forecasts of near‐surface temperature, 24 h precipitation totals, and near‐surface wind speed from the global model of the European Centre for Medium‐Range Weather Forecasts (ECMWF). The main objective is to evaluate the evolution of the difference in skill between the raw ensemble and the postprocessed forecasts. Reliability and sharpness, and hence skill, of the former is expected to improve over time. Thus, the gain by postprocessing is expected to decrease. Based on ECMWF forecasts from January 2002 to March 2014 and corresponding observations from globally distributed stations, we generate postprocessed forecasts by ensemble model output statistics (EMOS) for each station and variable. Given the higher average skill of the postprocessed forecasts, we analyze the evolution of the difference in skill between raw ensemble and EMOS. This skill gap remains almost constant over time indicating that postprocessing will keep adding skill in the foreseeable future.

Accurate ensemble forecasts of heavy precipitation in India are vital for many applications and essential for early warning of damaging flood events, especially during the monsoon season. In this study we investigate to what extent Quantile Mapping (QM) and Ensemble Model Output Statistics (EMOS) statistical postprocessing reduce errors in precipitation ensemble forecasts over India, in particular for heavy precipitation. Both methods are applied to day‐1 forecasts at 12‐km resolution from the 23‐member National Centre for Medium Range Weather Forecasting (NCMRWF) global ensemble prediction system (NEPS‐G). By construction, QM leads to distributions close to the observed ones, while EMOS optimizes the ensemble spread, and it is not a priori clear which is better suited for practical applications. The methods are therefore compared with respect to several key aspects of the forecasts: local distributions, ensemble spread, and skill for forecasting precipitation amounts and the exceedance of heavy precipitation thresholds. The evaluation includes rank histograms, Continuous Ranked Probability Skill Scores (CRPSS), Brier Skill Scores (BSS), reliability diagrams, and receiver operating characteristic. EMOS performs best not only with respect to correcting under‐ or overdispersive ensembles, but also in terms of forecast skill for precipitation amounts and heavy precipitation events, with positive CRPSS and BSS in most regions (both up to about 0.4 in some areas), while QM in many regions performs worse than the raw forecast. QM performs best with respect to the overall local precipitation distributions. Which aspects of the forecasts are most relevant depends to some extent on how the forecasts are used. If the main criteria are the correction of under‐ or overdispersion, forecast reliability, match between the forecasted distribution for individual days and observations (CRPSS), and the skill in forecasting heavy‐precipitation events (BSS), then EMOS is the better choice for postprocessing NEPS‐G forecasts for short lead times.

Numerical weather prediction models have systematic errors and biases, which can be reduced through statistical correction methods. The most intuitive approach to correcting weather forecasts is to establish a statistical relationship between the forecast and the corresponding observations. Once the relationship is established, it can be used to correct future forecasts. These approaches, also known as statistical adaptation, are used on a daily basis to improve the quality of operational forecasts. Many methods have been proposed in the literature to calibrate ensemble forecasts, and these can be divided into two main groups: parametric (that has a hypothesis on the underlying distribution) and non-parametric methods. One of the most widespread and used methods of the parametric group is the so-called Ensemble Model Output Statistics (EMOS). We show the performance of EMOS for the post-processing of probabilistic temperature forecasts at 2m. This method assumes that the underlying distribution is Gaussian (in the case of 2m T), and the parameters to be estimated are therefore the mean and the standard deviation. First results were obtained using the ECMWF operational forecast ensemble against SYNOP observations. Further studies were carried out to investigate the reliability of the estimated parameters using not only the raw ensemble data, but also the ECMWF deterministic operational forecast HRES and the control member. The results showed an improvement, especially for the shorter lead-times. The improvement was measured using weather scores for forecast verification and performance: bias, CRPS (Continuous Ranked Probability Score) and CRPSS (Continuous Ranked Probability Skill Score). These post-processed forecasts will serve as a reference for future studies.

The main approach to probabilistic weather forecasting has been the use of ensemble forecasting. In ensemble forecasting, the probability information is generally derived by using several numerical model runs, with perturbation of the initial conditions, physical schemes or dynamic core of the numerical weather prediction (NWP) models. However, ensemble forecasting usually tends to be under‐dispersive. Statistical post‐processing has, therefore, become an essential component of any ensemble prediction system aiming to improve the quality of numerical weather forecasts as they seek to generate calibrated and sharp predictive distributions of future weather quantities. Different versions of the ensemble model output statistics (EMOS) and the Bayesian model averaging (BMA) post‐processing methods are used in the present paper to calibrate 24, 48 and 72 hr forecasts of 24 hr accumulative precipitation. The ensemble employs the weather and research forecasting (WRF) model with eight different configurations which were run over Iran for six months (September 2015–February 2016). The results reveal that the BMA and EMOS‐censored, shifted gamma (CSG) techniques are substantially successful at improving the raw WRF ensemble forecasts; however, each approach improves different aspects of the forecast quality. The BMA method is more accurate, skilful and reliable than the EMOS‐CSG method, but has poorer discrimination. Moreover, it has better resolution in predicting the probability of high‐precipitation events than the EMOS‐CSG method.

The National Center for Environmental Predictions (NCEP) has produced an ensemble meteorological reforecast product by using a fixed version of Global Forecast System (GFS) ensemble prediction system since 1 January 1979. The 15‐member ensemble product, with a global coverage at a 2.5° × 2.5° spatial resolution and a 14‐day lead time, has been used successfully by the River Forecast Centers of the National Weather Service (NWS) to produce basin scale precipitation and temperature ensemble forecasts in the US for several years now. This study evaluates the predictive skill of post‐processed ensemble forecasts based on GFS precipitation reforecast in China's Huai river basin. The evaluation is carried out in 15 sub‐areas of the Huai river basin and covers the 1/1/1981–31/12/2003 period. The Ensemble Pre‐Processing system version 3 (EPP3), developed at NWS, is used to develop joint probability distributions between forecasted ensemble mean precipitation and corresponding observations and to generate individual ensemble members that preserve space–time correlation of the observed precipitation data. Several statistical verification measures are used to quantify the goodness of fit between post‐processed (i.e. EPP3 processed) ensemble mean and observation and to assess the ensemble spread. Results indicate that the post‐processed forecasts have meaningful predictive skill for the first few days for ensemble daily precipitation forecasts. Predictive skill of ensemble forecasts of cumulative precipitation for lead times up to 14 days are significant. The forecast skill is highly dependent on seasonality, with relatively lower skills seen for wet summer season, when convective storm patterns dominate, as compared with other seasons. The predictive skill of the post‐processed ensemble precipitation is much better than the raw forecasts and the climatological ensemble forecasts. The results from this study suggest that the NCEP's GFS reforecasts can be a valuable resource for places other than the US. Copyright © 2012 John Wiley & Sons, Ltd.

Ensemble forecasts are often calibrated using regression methods known as ensemble model output statistics (EMOS). Most of the EMOS methods described in the literature make the assumption that statistical parameter uncertainty is negligible, although making this assumption has been shown to be detrimental to forecast quality. We use simulations to show that this assumption would be expected to give unreliable forecasts and underprediction of extremes. Previous research has described how parameter uncertainty can be included in EMOS models using bootstrapping. We show that bootstrapping can be replaced with an analytical approach based on right Haar prior (RHP) theory, a branch of objective Bayesian statistics. For two commonly used EMOS models, simple linear regression (SLR) and heteroscedastic linear regression (HLR, also known as non‐homogeneous Gaussian regression), we show that RHP theory leads to perfectly reliable predictions, if the distributional assumptions are met. We apply SLR and HLR methods, with and without parameter uncertainty, to 5 years of past medium‐range forecasts for eight Northern Hemisphere locations and evaluate forecast accuracy and reliability. We find that the RHP version of HLR is the most accurate in terms of both log‐score and continuous ranked probability score. It is also the most reliable, and it reduces the underestimation of extremes seen in the other methods. Our results with regard to the benefits of including parameter uncertainty agree with the previous results based on bootstrapping, although the RHP calculations are faster and more accurate. A free software library is available for applying the various methods, and execution is computationally inexpensive. We conclude that the RHP version of HLR should be used in preference to the other methods tested. Further research would be needed to understand how to use objective Bayesian methods to include parameter uncertainty in other EMOS models.

Forecast ensembles are typically employed to account for prediction uncertainties in numerical weather prediction models. However, ensembles often exhibit biases and dispersion errors, thus they require statistical post-processing to improve their predictive performance. Two popular univariate post-processing models are the Bayesian model averaging (BMA) and the ensemble model output statistics (EMOS). In the last few years increased interest has emerged in developing multivariate post-processing models, incorporating dependencies between weather quantities, such as for example a bivariate distribution for wind vectors or even a more general setting allowing to combine any types of weather variables. In line with a recently proposed approach to model temperature and wind speed jointly by a bivariate BMA model, this paper introduces a bivariate EMOS model for these weather quantities based on a truncated normal distribution. The bivariate EMOS model is applied to temperature and wind speed forecasts of the eight-member University of Washington mesoscale ensemble and of the eleven-member ALADIN-HUNEPS ensemble of the Hungarian Meteorological Service and its predictive performance is compared to the performance of the bivariate BMA model and a multivariate Gaussian copula approach, post-processing the margins with univariate EMOS. While the predictive skills of the compared methods are similar, the bivariate EMOS model requires considerably lower computation times than the bivariate BMA method.

Although inflow forecasts are useful information that can be used for improving reservoir operation efficiency, uncertainty is still a challenge for getting sound operation results. Ensemble precipitation forecasts can take uncertainties under consideration, so they have been a research topic for improving reservoir operations. In this paper, a rainfall–runoff model, combined of a multiple linear regression model (for non-flood seasons) and the Xinanjiang model (for flood seasons), for inflow forecasts and a stochastic dynamic programming model for reservoir operations are developed in order to effectively use the ensemble precipitation forecast. To explore the best way for using the ensemble precipitation forecast, two post-processing techniques, i.e., ensemble forecast averaging (EFA) and interval value (IV), are tested. The ensemble precipitation forecast from the European Centre for Medium-Range Weather Forecasts (ECMWF) was chosen due to its high accuracy, and the Huanren reservoir, located in China, was used to test the newly developed models. The results show that, compared with the traditional rule curve, hydropower generation increases by 4.86% and 4.55%, respectively, when EFA and IV are used, which indicates that the use of ensemble forecasts facilitates considerable improvements in operating performance.

By the end of 2022, the renewable energy share of the global electricity capacity reached 40.3% and the new installations were dominated by solar energy, showing a global increase of 21.7%. Due to the high volatility of photovoltaic energy sources, their successful integration into the electrical grid requires accurate short-term power forecasts. These forecasts are obtained from the predictions of solar irradiance, where the most advanced method is the probabilistic approach based on ensemble forecasts.  However, ensemble forecasts are often underdispersive and subject to systematic bias. Hence, they require some form of statistical post-processing, where parametric models provide full predictive distributions of the weather variables at hand.We propose a general two-step machine learning-based approach to calibrating ensemble weather forecasts, where, in the first step, improved point forecasts are generated, which then together with various ensemble statistics serve as input features of the neural network estimating the parameters of the predictive distribution [1]. In a case study based on global horizontal irradiance forecasts of the operational ensemble prediction system of the Hungarian Meteorological Service, the predictive performance of this novel method is compared with the forecast skill of the raw ensemble and the state-of-the-art ensemble model output statistics approaches [2]. We show that at least up to 48h, statistical post-processing substantially improves the predictive performance of the raw ensemble for all forecast horizons considered; the maximal gain e.g. in terms of the mean continuous ranked probability score is above 20%. Furthermore, the proposed two-step machine learning-based approach outperforms in skill its competitors.

Adapting subseasonal-to-seasonal (S2S) precipitation forecast at watersheds for hydrologic ensemble streamflow forecasting with a machine learning-based post-processing approach

The quality of ensemble precipitation forecasts across the eastern United States is investigated, specifically, version 2 of the National Centers for Environmental Prediction (NCEP) Global Ensemble Forecast System Reforecast (GEFSRv2) and Short Range Ensemble Forecast (SREF) system, as well as NCEP’s Weather Prediction Center probabilistic quantitative precipitation forecast (WPC-PQPF) guidance. The forecasts are verified using multisensor precipitation estimates and various metrics conditioned upon seasonality, precipitation threshold, lead time, and spatial aggregation scale. The forecasts are verified, over the geographic domain of each of the four eastern River Forecasts Centers (RFCs) in the United States, by considering first 1) the three systems or guidance, using a common period of analysis (2012–13) for lead times from 1 to 3 days, and then 2) GEFSRv2 alone, using a longer period (2004–13) and lead times from 1 to 16 days. The verification results indicate that, across the eastern United States, precipitation forecast bias decreases and the skill and reliability improve as the spatial aggregation scale increases; however, all the forecasts exhibit some underforecasting bias. The skill of the forecasts is appreciably better in the cool season than in the warm one. The WPC-PQPFs tend to be superior, in terms of the correlation coefficient, relative mean error, reliability, and forecast skill scores, than both GEFSRv2 and SREF, but the performance varies with the RFC and lead time. Based on GEFSRv2, medium-range precipitation forecasts tend to have skill up to approximately day 7 relative to sampled climatology.

Abstract. Skilful hydrological forecasts can benefit decision-making in water resources management and other water-related sectors that require long-term planning. In Ireland, no such service exists to deliver forecasts at the catchment scale. In order to understand the potential for hydrological forecasting in Ireland, we benchmark the skill of ensemble streamflow prediction (ESP) for a diverse sample of 46 catchments using the GR4J (Génie Rural à 4 paramètres Journalier) hydrological model. Skill is evaluated within a 52-year hindcast study design over lead times of 1 d to 12 months for each of the 12 initialisation months, January to December. Our results show that ESP is skilful against a probabilistic climatology benchmark in the majority of catchments up to several months ahead. However, the level of skill was strongly dependent on lead time, initialisation month, and individual catchment location and storage properties. Mean ESP skill was found to decay rapidly as a function of lead time, with a continuous ranked probability skill score (CRPSS) of 0.8 (1 d), 0.32 (2-week), 0.18 (1-month), 0.05 (3-month), and 0.01 (12-month). Forecasts were generally more skilful when initialised in summer than other seasons. A strong correlation (ρ=0.94) was observed between forecast skill and catchment storage capacity (baseflow index), with the most skilful regions, the Midlands and the East, being those where slowly responding, high-storage catchments are located. Forecast reliability and discrimination were also assessed with respect to low- and high-flow events. In addition to our benchmarking experiment, we conditioned ESP with the winter North Atlantic Oscillation (NAO) using adjusted hindcasts from the Met Office's Global Seasonal Forecasting System version 5. We found gains in winter forecast skill (CRPSS) of 7 %–18 % were possible over lead times of 1 to 3 months and that improved reliability and discrimination make NAO-conditioned ESP particularly effective at forecasting dry winters, a critical season for water resources management. We conclude that ESP is skilful in a number of different contexts and thus should be operationalised in Ireland given its potential benefits for water managers and other stakeholders.