The reduction continuous rank probability score for evaluating discharge forecasts from hydrological ensemble prediction systems

[Departement_IRSTEA]Eaux [TR1_IRSTEA]ARCEAU

Improving ensemble forecasts is a complex process which involves proper scores such as the continuous ranked probability score (CRPS). A homogeneous Gaussian (hoG) model is introduced in order to better understand the characteristics of the CRPS. An analytical formula is derived for the expected CRPS of an ensemble in the hoG model. The score is a function of the variance of the error of the ensemble mean, the mean error of the ensemble mean and the ensemble variance. The hoG model also provides a score decomposition into reliability and resolution components. We examine whether the hoG model provides a useful approximation of the CRPS when applied to operational ECMWF medium‐range ensemble forecasts. The hoG approximation describes the spatial variations of the CRPS well while moderately overestimating the mean score. Seasonal averages over large domains are within 10% of the actual CRPS. Furthermore, the ability to approximate score changes is evaluated by (a) comparing raw ensemble forecasts with postprocessed ensemble forecasts, and (b) by examining score changes associated with a recent upgrade of the IFS. Overall, the hoG approximation predicts the actual CRPS changes well. One of the main anticipated applications of the hoG approximation are new diagnostics in verification software used by NWP developers routinely. The purpose of the diagnostics is to help developers explain impacts of forecast system changes on the CRPS in terms of the changes in mean error, changes in error variance and changes in ensemble variance. The diagnostics require little additional computational resources compared to the alternative of verifying postprocessed versions of the ensemble forecasts. Therefore, it will be feasible to apply the diagnostics easily to all variables that are examined as part of the model development process.

Abstract. Stream water temperatures influence water quality, with effects on aquatic biodiversity, drinking-water provision, electricity production, agriculture, and recreation. Therefore, stakeholders would benefit from an operational forecasting service that would support timely action. Deep-learning models are well-suited to providing probabilistic forecasts at individual stations of a monitoring network. Here, we train and evaluate several state-of-the-art models using 10 years of data from 54 stations across Switzerland. Static catchment features, time of the year, meteorological observations from the past 64 d, and their ensemble forecasts for the following 32 d are included as predictors in the models to estimate daily maximum water temperature over the next 32 d. Results show that the temporal fusion transformer (TFT) model performs best, with a continuous rank probability score (CRPS) of 0.70 °C averaged over all lead times, stations, and 90 forecasts distributed over 1 year. The TFT is followed by the recurrent neural network encoder–decoder, with a CRPS of 0.74 °C, and the neural hierarchical interpolation for time series, with a CRPS of 0.75 °C. These deep-learning models outperform other simpler models trained at each station: random forest (CRPS = 0.80 °C), multi-layer perceptron neural network (CRPS = 0.81 °C), and autoregressive linear model (CRPS = 0.96 °C). The average CRPS of the TFT degrades from 0.38 °C at lead a time of 1 d to 0.90 °C at a lead time of 32 d, largely driven by the uncertainty of the meteorological ensemble forecasts. In addition, TFT water temperature predictions at new and ungauged stations outperform those from the other models. When analyzing the importance of model inputs, we find a dominant role of observed water temperature and future air temperature, while including precipitation and time of the year further improves predictive skill. Operational probabilistic forecasts of daily maximum water temperature are generated twice per week with our TFT model and are publicly available at https://www.drought.ch/de/impakt-vorhersagen-malefix/wassertemperatur-prognosen/ (last access: 20 March 2025). Overall, this study provides insights into the extended-range predictability of stream water temperature and into the applicability of deep-learning models in hydrology.

Providing useful inflow forecasts of the Manantali dam is critical for zonal consumption and agricultural water supply, power production, flood and drought control and management (Shin et al., Meteorol Appl 27:e1827, 2019). Probabilistic approaches through ensemble forecasting systems are often used to provide more rational and useful hydrological information. This paper aims at implementing an ensemble forecasting system at the Senegal River upper the Manantali dam. Rainfall ensemble is obtained through harmonic analysis and an ARIMA stochastic process. Cyclical errors that are within rainfall cyclical behavior from the stochastic modeling are settled and processed using multivariate statistic tools to dress a rainfall ensemble forecast. The rainfall ensemble is used as input to run the HBV-light to product streamflow ensemble forecasts. A number of 61 forecasted rainfall time series are then used to run already calibrated hydrological model to produce hydrological ensemble forecasts called raw ensemble. In addition, the affine kernel dressing method is applied to the raw ensemble to obtain another ensemble. Both ensembles are evaluated using on the one hand deterministic verifications such the linear correlation, the mean error, the mean absolute error and the root-mean-squared error, and on the other hand, probabilistic scores (Brier score, rank probability score and continuous rank probability score) and diagrams (attribute diagram and relative operating characteristics curve). Results are satisfactory as at deterministic than probabilistic scale, particularly considering reliability, resolution and skill of the systems. For both ensembles, correlation between the averages of the members and corresponding observations is about 0.871. In addition, the dressing method globally improved the performances of ensemble forecasting system. Thus, both schemes system can help decision maker of the Manantali dam in water resources management.

Uncertainty propagation within a water level ensemble prediction system

Abstract. The mountainous snow cover is highly variable at all temporal and spatial scales. Snowpack models only imperfectly represent this variability, because of uncertain meteorological inputs, physical parameterizations, and unresolved terrain features. In situ observations of the height of snow (HS), despite their limited representativeness, could help constrain intermediate and large-scale modeling errors by means of data assimilation. In this work, we assimilate HS observations from an in situ network of 295 stations covering the French Alps, Pyrenees, and Andorra, over the period 2009–2019. In view of assimilating such observations into a spatialized snow cover modeling framework, we investigate whether such observations can be used to correct neighboring snowpack simulations. We use CrocO, an ensemble data assimilation framework of snow cover modeling, based on a particle filter suited to the propagation of information from observed to unobserved areas. This ensemble system already benefits from meteorological observations, assimilated within SAFRAN analysis scheme. CrocO also proposes various localization strategies to assimilate snow observations. These approaches are evaluated in a leave-one-out setup against the operational deterministic model and its ensemble open-loop counterpart, both running without HS assimilation. Results show that an intermediate localization radius of 35–50 km yields a slightly lower root mean square error (RMSE), and a better spread–skill than the strategy of assimilating all the observations from a whole mountain range. Significant continuous ranked probability score (CRPS) improvements of about 13 % are obtained in the areas where the open-loop modeling errors are the largest, e.g., the Haute-Ariège, Andorra, and the extreme southern Alps. Over these areas, weather station observations are generally sparser, resulting in more uncertain meteorological analyses and, therefore, snow simulations. In situ HS observations thus show an interesting complementarity with meteorological observations to better constrain snow cover simulations over large areas.

<p>In recent years, the number of large-scale hydrological forecasting systems has been steadily growing. This may lead to regions having numerous models spatially overlapping each other. Some of these regions have what we will refer to as a regional, more specialized, model for the area that performs generally better than their large-scale counterpart, considering the coarser spatial resolution and sometimes lack of calibration of the latter. Our work explored the possibility of using simple methods to retrieve hydrological information from a large-scale model, namely the National Surface and River Prediction System (NSRPS) that will eventually cover the Canadian territory, in order to improve the forecasts from a local system, namely the Système de Prévision Hydrologique (SPH) that covers most of the province of Quebec. Outputs from the two forecasting systems were thus combined using methods including the simple mean, a weighted average in which the weights are optimized either using the Kling-Gupta Efficiency (KGE) or the Continuous Ranked Probability Score (CRPS) as cost functions, or weights calculated from the residual errors of the forecasts. Bayesian Model Averaging (BMA) was also explored to combine the ensemble forecasts from both systems. The results show that it is possible to improve the local hydrological forecasts by using simple weighted combinations with forecasts from the large-scale system. Performance was assessed using many well-known criteria such as the Nash-Sutcliffe Efficiency (NSE), KGE and CRPS. Results were averaged over the 61 available gauging stations and analyzed at lead times ranging from 3 to 120 hours. We observed improvements in all criteria for lead times over 60 hours as well as no loss in performance at any lead times. Finally, the methods were also used in a leave-one-out setup to simulate performance on ungauged basins. The performance gain for ungauged basins is similar to that of the gauged basins, hinting at the fact that these simple methods could also improve forecasts in more remote territories where no measurements are available.</p>

Indian summer monsoon governs agricultural productivity, water resources, and economic activity affecting 1.4 billion people, yet accurate prediction remains challenging due to complex non-linear dynamics involving atmosphere-ocean interactions, land surface processes, and stochastic forcing. Traditional numerical weather prediction models solve deterministic partial differential equations but struggle capturing uncertainty quantification and sub-grid scale processes. Machine learning approaches demonstrate skill learning patterns from historical data but lack physical constraints and interpretability. This research develops hybrid framework integrating stochastic differential equations representing atmospheric dynamics with deep learning capturing non-linear relationships, enabling probabilistic monsoon forecasting with improved accuracy and uncertainty quantification. The research formulated Indian monsoon system as coupled stochastic differential equation system: dX_t = μ(X_t, θ)dt + σ(X_t, θ)dW_t, where X_t represents state variables (rainfall, temperature, humidity, pressure), μ denotes drift coefficient encoding deterministic dynamics, σ represents diffusion coefficient capturing stochastic fluctuations, W_t denotes Wiener process, and θ parameters learned from data. Long Short-Term Memory neural networks parameterized drift and diffusion coefficients enabling flexible representation of complex non-linear relationships while preserving stochastic process structure. Training utilized 40 years of daily rainfall data (1980-2019) from 64 India Meteorological Department stations across eight monsoon regions, supplemented by atmospheric reanalysis including temperature, humidity, wind, and sea surface temperature. Numerical integration employed Euler-Maruyama scheme with adaptive time stepping ensuring stability and accuracy. Model evaluation compared hybrid stochastic-machine learning approach against traditional numerical prediction, pure stochastic models, and pure machine learning on 2017-2019 validation period. Performance metrics included root mean square error, mean absolute error, R-squared, continuous ranked probability score, and skill scores assessing probabilistic forecast quality. Results demonstrated hybrid model achieved RMSE of 2.1 mm daily rainfall compared to 3.2 mm for pure stochastic models and 5.4 mm for deterministic numerical predictions, representing 34% accuracy improvement. R-squared reached 0.967 indicating excellent predictive skill. Probabilistic forecasts provided well-calibrated uncertainty estimates with continuous ranked probability score of 1.8 mm outperforming deterministic predictions. Spatial analysis revealed consistent performance across regions with coastal areas (R²=0.967) and northeast India (R²=0.956) achieving highest accuracy, while northwest plains showed moderate performance (R²=0.912) due to continental influences. Temporal validation demonstrated robust monsoon onset prediction with 89.3% accuracy detecting rainfall commencement within ±3 days. Computational efficiency analysis revealed hybrid model required 2.3 hours training on GPU hardware compared to 48 hours for traditional numerical models, enabling operational forecasting. The research establishes mathematical framework for physics-informed machine learning combining stochastic calculus with deep learning, providing interpretable probabilistic forecasts supporting agricultural planning, water resource management, and disaster preparedness across India's monsoon-dependent economy.

DDOS-attacks detection using an efficient measurement-based statistical mechanism

From optimal supply decisions to anticipatory control systems, wind-based energy applications rely heavily upon accurate, local, short-term forecasts of future wind speed. Recent studies have shown continuous ranked probability score (CRPS) minimizing models with Gaussian assumptions to be effective for well-researched sites where those assumptions are appropriate. We consider the more general case where Gaussianity is not assumed and access to historical data may be constrained. Deriving a CRPS expression for a minimum Extreme Value distribution, we use it to propose a site-adaptive Weibull-based CRPS-minimizing model, which is tested and shown to perform better than both deterministic and probabilistic reference models on a ground-based array of weather observation sites in northern Japan.

In numerical weather prediction (NWP), the uncertainty about the future state of the atmosphere is described by a set of forecasts (called an ensemble). All ensembles have deficiencies that can be corrected via statistical post-processing methods. Several ensembles, based on different NWP models, exist and may be corrected using different statistical methods. These raw or post-processed ensembles can thus be combined. The theory of prediction with expert advice allows us to build combination algorithms with theoretical guarantees on the forecast performance. We adapt this theory to the case of probabilistic forecasts issued as stepwise cumulative distribution functions, computed from raw and post-processed ensembles. The theory is applied to combine wind speed ensemble forecasts. The second goal of this study is to explore the use of two forecast performance criteria: the continuous ranked probability score (CRPS) and the Jolliffe–Primo test. The usual way to build skilful probabilistic forecasts is to minimize the CRPS. Minimizing the CRPS may not produce reliable forecasts according to the Jolliffe–Primo test. The Jolliffe–Primo test generally selects reliable forecasts, but could lead to issuing suboptimal forecasts in terms of CRPS. We propose to use both criteria to achieve reliable and skilful probabilistic forecasts.

We present an ensemble prediction system using a Deep Learning Weather Prediction (DLWP) model that recursively predicts six key atmospheric variables with six‐hour time resolution. This computationally efficient model uses convolutional neural networks (CNNs) on a cubed sphere grid to produce global forecasts. The trained model requires just three minutes on a single GPU to produce a 320‐member set of six‐week forecasts at 1.4° resolution. Ensemble spread is primarily produced by randomizing the CNN training process to create a set of 32 DLWP models with slightly different learned weights. Although our DLWP model does not forecast precipitation, it does forecast total column water vapor and gives a reasonable 4.5‐day deterministic forecast of Hurricane Irma. In addition to simulating mid‐latitude weather systems, it spontaneously generates tropical cyclones in a one‐year free‐running simulation. Averaged globally and over a two‐year test set, the ensemble mean RMSE retains skill relative to climatology beyond two‐weeks, with anomaly correlation coefficients remaining above 0.6 through six days. Our primary application is to subseasonal‐to‐seasonal (S2S) forecasting at lead times from two to six weeks. Current forecast systems have low skill in predicting one‐ or 2‐week‐average weather patterns at S2S time scales. The continuous ranked probability score (CRPS) and the ranked probability skill score (RPSS) show that the DLWP ensemble is only modestly inferior in performance to the European Center for Medium Range Weather Forecasts (ECMWF) S2S ensemble over land at lead times of 4 and 5–6 weeks. At shorter lead times, the ECMWF ensemble performs better than DLWP.

The mountainous snow cover is highly variable at all temporal and spatial scales. Snowpack models only imperfectly represent this variability, because of uncertain meteorological inputs, physical parameterisations, and unresolved terrain features. In-situ observations of the height of snow (HS), despite their limited representativeness, could help constrain intermediate and large scale modelling errors by means of data assimilation. In this work, we assimilate HS observations from an in-situ network of 295 stations covering the French Alps, Pyrenees and Andorra, over the period 2009–2019. In view of assimilating such observations into a spatialised snow cover modelling framework, we investigate whether such observations can be used to correct neighbouring snowpack simulations. We use CrocO, an ensemble data assimilation framework of snow cover modelling, based on a Particle Filter suited to the propagation of information from observed to unobserved areas. This ensemble system already benefits from meteorological observations, assimilated within SAFRAN analysis scheme. CrocO also proposes various localisation strategies to assimilate snow observations. These approaches are evaluated in a Leave-One-Out setup against the operational deterministic model and its ensemble open-loop counterpart, both running without HS assimilation. Results show that intermediate localisation radius of 35–50 km yield a slightly lower root mean square error (RMSE), and a better Spread-Skill than the strategy assimilating all the observations from a whole mountain range. Significant continuous ranked probability score (CRPS) improvements of about 13 % are obtained in the areas where the open-loop modelling errors are the largest, e.g. the Haute-Ariège, Andorra and the Extreme Southern Alps. Over these areas, weather station observations are generally sparser, resulting in more uncertain meteorological analyses, and therefore snow simulations. In-situ HS observations thus shows an interesting complementarity with meteorological observations to better constrain snow cover simulations over large areas.

The mountainous snow cover is highly variable at all temporal and spatial scales. Snowpack models only imperfectly represent this variability, because of uncertain meteorological inputs, physical parameterisations, and unresolved terrain features. In-situ observations of the height of snow (HS), despite their limited representativeness, could help constrain intermediate and large scale modelling errors by means of data assimilation. In this work, we assimilate HS observations from an in-situ network of 295 stations covering the French Alps, Pyrenees and Andorra, over the period 2009–2019. In view of assimilating such observations into a spatialised snow cover modelling framework, we investigate whether such observations can be used to correct neighbouring snowpack simulations. We use CrocO, an ensemble data assimilation framework of snow cover modelling, based on a Particle Filter suited to the propagation of information from observed to unobserved areas. This ensemble system already benefits from meteorological observations, assimilated within SAFRAN analysis scheme. CrocO also proposes various localisation strategies to assimilate snow observations. These approaches are evaluated in a Leave-One-Out setup against the operational deterministic model and its ensemble open-loop counterpart, both running without HS assimilation. Results show that intermediate localisation radius of 35–50 km yield a slightly lower root mean square error (RMSE), and a better Spread-Skill than the strategy assimilating all the observations from a whole mountain range. Significant continuous ranked probability score (CRPS) improvements of about 13 % are obtained in the areas where the open-loop modelling errors are the largest, e.g. the Haute-Ariège, Andorra and the Extreme Southern Alps. Over these areas, weather station observations are generally sparser, resulting in more uncertain meteorological analyses, and therefore snow simulations. In-situ HS observations thus shows an interesting complementarity with meteorological observations to better constrain snow cover simulations over large areas.

The aim of this work is to present a methodology for the reconstruction of missing fracture density within highly fractured intervals, which can represent preferential fluid flow pathways. The lack of record can be very common due to the intense presence of fractures, dissolution processes, or data acquisition issues. The superposition of numerous fractures makes the definition of fracture surfaces impossible, as a consequence, modeling such zones is challenging. In order to address this issue, the usage of direct sampling multiple-point statistics to perform gap filling in well logs is demonstrated as an alternative to other techniques. It reproduces data patterns and provides several models representing uncertainty. The method was tested in intervals from a highly fractured well, by removing previously known fracture density data, and simulating different scenarios with direct sampling. Simulation results are compared to the observed data using cross-validation and continuous rank probability score. The reference scenario training data set consists in one well and two variables: fracture density and fracture occurrence. A sensitivity analysis is carried out considering additional variables, additional wells, different intervals, resampling with extremes, and other gap filling techniques. The auxiliary variable plays an important role in pattern matching, but adding wells and logs increases the complexity of the method without improving pattern retrieval. Best results are obtained applying extreme values theory for stochastic process with the enrichment of the fracture density data at the tail region, followed by resampling of the new values. The enriched data is used for the gap filling resulting in lower continuous rank probability score, and the achievement of extreme fracture density values.