Machine learning-based parametric post-processing of solar irradiance ensemble forecasts

[Departement_IRSTEA]Eaux [TR1_IRSTEA]ARCEAU

Ensemble forecasts are important due to their ability to characterize forecast uncertainty, which is fundamental when forecasting extreme weather. Ensemble forecasts are however often biased and underdispersed and thus need to be post-processed.A common approach for this is the use of ensemble model output statistics (EMOS), where a parametric distribution is fitted with a limited number of predictors. With recent advances in computer science and increased amounts of data available, machine learning techniques, like random forests, are becoming more popular for high dimensional regression problems. In this research, we explore the use of the quantile regression forest (QRF), a random forest adapted for conditional quantile estimation, applied to medium range gridded probabilistic precipitation forecasts. QRFs are non-parametric and allow for a larger number of predictors, which means they can possibly consider more dependencies that might otherwise not be captured with a simple EMOS.A QRF takes several hyperparameters that influence the way the decision trees in the forest are constructed. We explore the minimum number of samples needed in a leaf to split it (minimum node size) and the number of predictors explored in each split (mtry). A hyperparameter space is constructed by setting ranges for both minimum node size and mtry, and the optimal hyperparameter set is determined by performing a cross validated grid search. Here, each model is assessed based on the continuous ranked probability skill score (CRPSS). For comparison, EMOS is applied with a zero-adjusted gamma (ZAGA) distribution, using a limited number of predictors that are physically correlated to precipitation. Both methods are verified on a separate testing data set and evaluated using several scores, including CRPSS and Brier skills score (BSS).We consider 4 years (November 2018 – October 2022) of archived operational ECMWF-IFS ensemble forecasts for the Netherlands. The data is split into November 2018 – October 2021 for training and cross-validation, and October 2021 – October 2022 for testing, separating data for season, initialization time and lead-time. Forecasts are post-processed up to +10 days. Ensemble statistics on 60+ forecast variables are used as predictors. Spatially and temporally aggregated, gauge-adjusted radar observations are used as predictand. The raw ensemble is considered as the benchmark.The results of this research will determine what method will be used to post-process the ensemble precipitation forecasts in the context of the early warning center (EWC) of the Royal Netherlands Meteorological Institute. The most suitable method could differ between shorter and longer lead times.

Forecast performance of data-driven models depends on the local weather and climate regime, which makes model selection a tedious task for forecast practitioners. Ensemble forecasting, or forecast combination, is beneficial in such cases, in that, forecasts from multiple models are combined to form a final forecast. In ensemble forecasting, additional to the final deterministic-style forecasts, predictive distributions are also available, which can be used by grid operators for better decision-making. Such empirical predictive distributions are useful to represent the uncertainty associated with the forecasts. However, raw ensemble forecasts are often not calibrated, e.g., due to the lack of diversity in the ensemble members. The lack of ensemble spread is known as underdispersion, and it can be ameliorated through post-processing.This study aims to calibrate hourly ensemble clear-sky index forecasts, generated by 20 data-driven models, using both parametric and nonparametric post-processing techniques. Four years of data collected at 7 research-grade sites are used in the empirical part of the paper. Quantitative and qualitative methods are used to evaluate the performance of post-processing techniques in terms of calibration and sharpness. Post-processed ensemble forecasts outperform raw ensemble forecasts under all verification metrics. The proposed parametric post-processing technique, namely, generalized additive models for location, scale and shape, substantially reduces the continuous ranked probability score (CRPS) of the raw ensemble forecasts from 32–59 W/m2 to 25–45 W/m2 and quantile score from 16–30 W/m2 to 13–23 W/m2. In terms of CRPS skill score, the proposed method achieved 38–58% improvements over a climatology reference.

The National Center for Environmental Predictions (NCEP) has produced an ensemble meteorological reforecast product by using a fixed version of Global Forecast System (GFS) ensemble prediction system since 1 January 1979. The 15‐member ensemble product, with a global coverage at a 2.5° × 2.5° spatial resolution and a 14‐day lead time, has been used successfully by the River Forecast Centers of the National Weather Service (NWS) to produce basin scale precipitation and temperature ensemble forecasts in the US for several years now. This study evaluates the predictive skill of post‐processed ensemble forecasts based on GFS precipitation reforecast in China's Huai river basin. The evaluation is carried out in 15 sub‐areas of the Huai river basin and covers the 1/1/1981–31/12/2003 period. The Ensemble Pre‐Processing system version 3 (EPP3), developed at NWS, is used to develop joint probability distributions between forecasted ensemble mean precipitation and corresponding observations and to generate individual ensemble members that preserve space–time correlation of the observed precipitation data. Several statistical verification measures are used to quantify the goodness of fit between post‐processed (i.e. EPP3 processed) ensemble mean and observation and to assess the ensemble spread. Results indicate that the post‐processed forecasts have meaningful predictive skill for the first few days for ensemble daily precipitation forecasts. Predictive skill of ensemble forecasts of cumulative precipitation for lead times up to 14 days are significant. The forecast skill is highly dependent on seasonality, with relatively lower skills seen for wet summer season, when convective storm patterns dominate, as compared with other seasons. The predictive skill of the post‐processed ensemble precipitation is much better than the raw forecasts and the climatological ensemble forecasts. The results from this study suggest that the NCEP's GFS reforecasts can be a valuable resource for places other than the US. Copyright © 2012 John Wiley & Sons, Ltd.

Statistical postprocessing of global ensemble weather forecasts is revisited by leveraging recent developments in machine learning. Verification of past forecasts is exploited to learn systematic deficiencies of numerical weather predictions in order to boost postprocessed forecast performance. Here, we introduce postprocessing of ensembles with transformers (PoET), a postprocessing approach based on hierarchical transformers. PoET has two major characteristics: 1) the postprocessing is applied directly to the ensemble members rather than to a predictive distribution or a functional of it, and 2) the method is ensemble-size agnostic in the sense that the number of ensemble members in training and inference mode can differ. The PoET output is a set of calibrated members that has the same size as the original ensemble but with improved reliability. Performance assessments show that PoET can bring up to 20% improvement in skill globally for 2-m temperature and 2% for precipitation forecasts and outperforms the simpler statistical member-by-member method, used here as a competitive benchmark. PoET is also applied to the ENS-10 benchmark dataset for ensemble postprocessing and provides better results when compared to other deep learning solutions that are evaluated for most parameters. Furthermore, because each ensemble member is calibrated separately, downstream applications should directly benefit from the improvement made on the ensemble forecast with postprocessing.

The main approach to probabilistic weather forecasting has been the use of ensemble forecasting. In ensemble forecasting, the probability information is generally derived by using several numerical model runs, with perturbation of the initial conditions, physical schemes or dynamic core of the numerical weather prediction (NWP) models. However, ensemble forecasting usually tends to be under‐dispersive. Statistical post‐processing has, therefore, become an essential component of any ensemble prediction system aiming to improve the quality of numerical weather forecasts as they seek to generate calibrated and sharp predictive distributions of future weather quantities. Different versions of the ensemble model output statistics (EMOS) and the Bayesian model averaging (BMA) post‐processing methods are used in the present paper to calibrate 24, 48 and 72 hr forecasts of 24 hr accumulative precipitation. The ensemble employs the weather and research forecasting (WRF) model with eight different configurations which were run over Iran for six months (September 2015–February 2016). The results reveal that the BMA and EMOS‐censored, shifted gamma (CSG) techniques are substantially successful at improving the raw WRF ensemble forecasts; however, each approach improves different aspects of the forecast quality. The BMA method is more accurate, skilful and reliable than the EMOS‐CSG method, but has poorer discrimination. Moreover, it has better resolution in predicting the probability of high‐precipitation events than the EMOS‐CSG method.

Within the framework of this investigation, convection-permitting (CP) ensemble forecasts were generated for West Africa by combining different initial and lateral boundary conditions (IBCs) with perturbations that address the uncertainty of land-surface atmosphere interactions (land-surface perturbations). For a multi-analysis setup, IBCs were taken from model analyses of different global models; for a single-model setup, they were selected from the ensemble system of the European Centre for Medium-range Weather Forecasts (ECMWF). The different ensemble setups were assessed using common probabilistic scores as well as by spatial forecast verification of precipitation generated mainly by convective systems during the West African monsoon season. Additionally, it was investigated whether the CP ensemble forecasts were superior to the ECMWF ensemble forecasts.Probabilistic scores were higher for the single-model ensemble than for the multi-analysis setup, but the latter displayed a larger dispersion and more extreme scenarios. From this, it is concluded that the different model analyses can differ strongly from each other. The land-surface perturbations were able to generate sufficient complementary spread. While the CP simulations showed a stronger negative precipitation bias in the southernmost region near the Guinean coast, the ECMWF simulations exhibited a negative bias further north in the Sahel region, where larger convective systems occur less frequently. Not in all cases did the CP ensemble versions produce better probabilistic scores than the global ensemble forecasts, but they yielded larger spread and less underdispersion. Rank histograms, though, were also influenced by the different structure of the precipitation patterns of the global and CP forecasts. Scores improved when using a later version of the CP model as well as with the skill of the global ensemble forecasts used as IBCs. Altogether, the proposed realization of CP ensemble forecasts is found to be suited for the prediction of convective precipitation in West Africa.

Various perturbation approaches have been proposed for representing model error in convection-permitting ensemble prediction. Their evaluation usually relies on time-averaged ensemble prediction statistics and on complex case studies. In this work, their detailed physical behaviour is studied in order to understand their differences, and to help their optimization. A process-level intercomparison framework is used to investigate the widely used SPPT (stochastic perturbations of physics tendencies), independent SPPT, and random-parameters model-perturbation approaches. Ensemble predictions with the single-column version of the Arome numerical-weather-prediction model are evaluated on three different boundary-layer regimes: cumulus convection, stratocumulus-topped boundary layer, and radiation fog. The independent SPPT approach is found to produce more dispersion than the SPPT approach, particularly when several physics parametrizations are in near equilibrium. It also appears to be more numerically stable near the surface. The random parameters approach perturbations are structurally very different from the other approaches, particularly regarding cloud structure. The independent SPPT and random parameters approaches have very different sensitivities to the atmospheric conditions, which suggests that intercomparisons of ensemble-model-error approaches should carefully account for situation dependency. Substantial forecast biases are produced by random parameters with respect to the unperturbed model. These results suggest that the independent SPPT approach can bring major improvements over the SPPT approach with minimal effort, that there is some complementarity between the independent SPPT and random parameters approaches, but that implementing random-parameters-type approaches in operational applications may require careful tuning to avoid creating forecast biases.

We investigate the effect of statistical post‐processing on the probabilistic skill of discomfort index (DI) and indoor wet‐bulb globe temperature (WBGTid) ensemble forecasts, both calculated from the corresponding forecasts of temperature and dew point temperature. Two different methodological approaches to calibration are compared. In the first case, we start with joint post‐processing of the temperature and dew point forecasts and then create calibrated samples of DI and WBGTid using samples from the obtained bivariate predictive distributions. This approach is compared with direct post‐processing of the heat index ensemble forecasts. For this purpose, a novel ensemble model output statistics model based on a generalized extreme value distribution is proposed. The predictive performance of both methods is tested on the operational temperature and dew point ensemble forecasts of the European Centre for Medium‐Range Weather Forecasts and the corresponding forecasts of DI and WBGTid. For short lead times (up to day 6), both approaches significantly improve the forecast skill. Among the competing post‐processing methods, direct calibration of heat indices exhibits the best predictive performance, very closely followed by the more general approach based on joint calibration of temperature and dew point temperature. Additionally, a machine learning approach is tested and shows comparable performance for the case when one is interested only in forecasting heat index warning level categories.

The Central European limited‐area ensemble forecasting system ALADIN‐LAEF (Aire Limitée Adaptation Dynamique Développement InterNational—Limited‐Area Ensemble Forecasting) has been developed within the framework of ALADIN international cooperation and the Regional Cooperation for Limited‐Area modelling in Central Europe (RC LACE). It was put into pre‐operation in March 2007. The main feature of the pre‐operational ALADIN‐LAEF was the dynamical downscaling of the global ensemble forecast from the European Centre for Medium‐range Weather Forecasts (ECMWF). In 2009, ALADIN‐LAEF was upgraded with several methods for dealing with the forecast uncertainties to improve the forecast quality. These are: (1) the blending method, which combines the large‐scale uncertainty generated by ECMWF singular vectors with the small‐scale perturbations resolved by ALADIN breeding into atmospheric initial condition perturbations; (2) the multi‐physics approach, wherein different physics schemes are used for different forecast members to account for model uncertainties; and (3) the non‐cycling surface breeding technique, which generates surface initial condition perturbations.This article illustrates the technical details of the updated ALADIN‐LAEF and investigates its performance. Detailed verification of the upgraded ALADIN‐LAEF and a comparison with its first implementation (dynamical downscaling of ECMWF ensemble forecasts) are presented for a two‐month period in summer 2007. The results show better performance and skill for the upgraded system due to the better representation of forecast uncertainties. Copyright © 2011 Royal Meteorological Society

In countries where drought is a serious threat to agriculture, water resources and increases risk of wildfires, there has been an increasing interest in using weather modification techniques to improve local precipitation.  Despite the fact that cloud seeding is used in many places around the world, there is still not a clear consensus on its effectiveness. Various evaluation methods, including aircraft and ground-based measurements, remote sensing, statistical analysis, and numerical simulation have been widely used to evaluate the effect of cloud seeding. While each of these methods has its own benefits, they also come with limitations. For example, numerical models can simulate cases with and without cloud seeding and these scenarios can be then compared to determine how much rainfall increases after the seeding. Predictions provided by such scenarios can help with decision-making before conducting a cloud seeding experiment. The downside of numerical simulations is the presence of both systematic and random errors that originate from uncertainties initial conditions and numerical approximations. Ensemble forecasts can capture some of these uncertainties and provide a range of possible outcomes. The main goal of this study is to explore the potential of an ensemble forecasting system in evaluating the efficacy of cloud seeding. We used a limited-area ensemble forecasting system which is based on Met Office Unified Model coupled with Weather Research and Forecasting Model (WRF). The initial conditions of 13 ensemble members were created by downscaling of global ensemble model that was perturbed using the Ensemble transform Kalman filter. The WRF model was used to downscale the ensemble members to a finer resolution and simulate the seeding effect using an algorithm that was added to the Morrison microphysics scheme. As study cases, we utilized several cloud seeding experiments that were conducted during the weather modification campaign of 2022 and 2023 by the National Institute of Meteorological Sciences and the Korea Meteorological Administration. Ground-based hygroscopic cloud seeding was conducted in the mountainous region of South Korea using calcium chloride flares. Simulations were conducted for scenarios with and without seeding, and the difference in rainfall we assessed using ensemble mean, ensemble spread, and probabilities of various rainfall increment thresholds. The overall ensemble performance in rain predictability of the ensemble forecasts was evaluated using commonly used techniques, including the Brier score, reliability diagrams and ROC curves. Acknowledgments: This work was funded by the Korea Meteorological Administration Research and Development Program “Research on Weather Modification and Cloud Physics” under Grant (KMA2018-00224).

Ensemble forecasts are often calibrated using regression methods known as ensemble model output statistics (EMOS). Most of the EMOS methods described in the literature make the assumption that statistical parameter uncertainty is negligible, although making this assumption has been shown to be detrimental to forecast quality. We use simulations to show that this assumption would be expected to give unreliable forecasts and underprediction of extremes. Previous research has described how parameter uncertainty can be included in EMOS models using bootstrapping. We show that bootstrapping can be replaced with an analytical approach based on right Haar prior (RHP) theory, a branch of objective Bayesian statistics. For two commonly used EMOS models, simple linear regression (SLR) and heteroscedastic linear regression (HLR, also known as non‐homogeneous Gaussian regression), we show that RHP theory leads to perfectly reliable predictions, if the distributional assumptions are met. We apply SLR and HLR methods, with and without parameter uncertainty, to 5 years of past medium‐range forecasts for eight Northern Hemisphere locations and evaluate forecast accuracy and reliability. We find that the RHP version of HLR is the most accurate in terms of both log‐score and continuous ranked probability score. It is also the most reliable, and it reduces the underestimation of extremes seen in the other methods. Our results with regard to the benefits of including parameter uncertainty agree with the previous results based on bootstrapping, although the RHP calculations are faster and more accurate. A free software library is available for applying the various methods, and execution is computationally inexpensive. We conclude that the RHP version of HLR should be used in preference to the other methods tested. Further research would be needed to understand how to use objective Bayesian methods to include parameter uncertainty in other EMOS models.

Forecast ensembles are typically employed to account for prediction uncertainties in numerical weather prediction models. However, ensembles often exhibit biases and dispersion errors, thus they require statistical post-processing to improve their predictive performance. Two popular univariate post-processing models are the Bayesian model averaging (BMA) and the ensemble model output statistics (EMOS). In the last few years increased interest has emerged in developing multivariate post-processing models, incorporating dependencies between weather quantities, such as for example a bivariate distribution for wind vectors or even a more general setting allowing to combine any types of weather variables. In line with a recently proposed approach to model temperature and wind speed jointly by a bivariate BMA model, this paper introduces a bivariate EMOS model for these weather quantities based on a truncated normal distribution. The bivariate EMOS model is applied to temperature and wind speed forecasts of the eight-member University of Washington mesoscale ensemble and of the eleven-member ALADIN-HUNEPS ensemble of the Hungarian Meteorological Service and its predictive performance is compared to the performance of the bivariate BMA model and a multivariate Gaussian copula approach, post-processing the margins with univariate EMOS. While the predictive skills of the compared methods are similar, the bivariate EMOS model requires considerably lower computation times than the bivariate BMA method.

This study examines the sensitivity of global ensemble forecasts to the use of different approaches for specifying both the initial ensemble mean and perturbations. The current operational ensemble prediction system of the Meteorological Service of Canada uses the ensemble Kalman filter (EnKF) to define both the ensemble mean and perturbations. To evaluate the impact of different approaches for obtaining the initial ensemble perturbations, the operational EnKF approach is compared with using either no initial perturbations or perturbations obtained using singular vectors (SVs). The SVs are computed using the (dry) total-energy norm with a 48-h optimization time interval. Random linear combinations of 60 SVs are computed for each of three regions. Next, the impact of replacing the initial ensemble mean, currently the EnKF ensemble mean analysis, with the higher-resolution operational four-dimensional variational data assimilation (4D-Var) analysis is evaluated. For this comparison, perturbations are provided by the EnKF. All experiments are performed over two-month periods during both the boreal summer and winter using a system very similar to the global ensemble prediction system that became operational on 10 July 2007. Relative to the operational configuration that relies on the EnKF, the use of SVs to compute initial perturbations produces small, but statistically significant differences in probabilistic forecast scores in favor of the EnKF both in the tropics and, for a limited set of forecast lead times, in the summer hemisphere extratropics, whereas the results are very similar in the winter hemisphere extratropics. Both approaches lead to significantly better ensemble forecasts than with no initial perturbations, though results are quite similar in the tropics when using SVs and no perturbations. The use of an initial-time norm that does not include information on analysis uncertainty and the lack of linearized moist processes in the calculation of the SVs are two factors that limit the quality of the resulting SV-based ensemble forecasts. Relative to the operational configuration, use of the 4D-Var analysis to specify the initial ensemble mean results in improved probabilistic forecast scores during the boreal summer period in the southern extratropics and tropics, but a near-neutral impact otherwise.

Modern weather forecasts are typically in the form of an ensemble of forecasts obtained from multiple runs of numerical weather prediction models. Ensemble forecasts are usually biased and affected by dispersion errors, and they should be statistically corrected to gain accuracy. This is often done following a two-step approach: first, we correct the univariate forecasts, and then, we reconstruct the dependence structure non-parametrically via empirical copulas. The parametric correction of the dependence structure is limited to Gaussian copula-based methods. In this work, we propose a novel approach based on a more general parametric class of copulas called Archimedean copulas. We test the new method in both a simulated scenario and a case-study setting for multi-site temperature forecasts from the ALADIN-LAEF ensemble system in Austria. Our findings show that the state-of-the-art non-parametric techniques perform well in the simulation study. However, Archimedean copulas outperform the existing techniques, especially Gaussian copula approaches, and output well-calibrated forecasts in the real-case study. Our analysis demonstrates the usefulness of including advanced parametric copula methods in the post-processing context and the need of a more realistic simulated framework to test new methodology.