Predictive Inference Based on Markov Chain Monte Carlo Output

When predicting future events, it is common to issue forecasts that are probabilistic, in the form of probability distributions over the range of possible outcomes. Such forecasts can be evaluated using proper scoring rules. Proper scoring rules condense forecast performance into a single numerical value, allowing competing forecasters to be ranked and compared. To facilitate the use of scoring rules in practical applications, the scoringRules package in R provides popular scoring rules for a wide range of forecast distributions. This paper discusses an extension to the scoringRules package that additionally permits the implementation of popular weighted scoring rules. Weighted scoring rules allow particular outcomes to be targeted during forecast evaluation, recognizing that certain outcomes are often of more interest than others when assessing forecast quality. This introduces the potential for very flexible, user-oriented evaluation of probabilistic forecasts. We discuss the theory underlying weighted scoring rules, and describe how they can readily be implemented in practice using scoringRules. Functionality is available for weighted versions of several popular scoring rules, including the logarithmic score, the continuous ranked probability score, and the energy score. Two case studies are presented to demonstrate this, whereby weighted scoring rules are applied to univariate and multivariate probabilistic forecasts in the fields of meteorology and economics.

Proper scoring rules provide a theoretically principled framework for the quantitative assessment of the predictive performance of probabilistic forecasts. While a wide selection of such scoring rules for univariate quantities exists, there are only few scoring rules for multivariate quantities, and many of them require that forecasts are given in the form of a probability density function. The energy score, a multivariate generalization of the continuous ranked probability score, is the only commonly used score that is applicable in the important case of ensemble forecasts, where the multivariate predictive distribution is represented by a finite sample. Unfortunately, its ability to detect incorrectly specified correlations between the components of the multivariate quantity is somewhat limited. In this paper the authors present an alternative class of proper scoring rules based on the geostatistical concept of variograms. The sensitivity of these variogram-based scoring rules to incorrectly predicted means, variances, and correlations is studied in a number of examples with simulated observations and forecasts; they are shown to be distinctly more discriminative with respect to the correlation structure. This conclusion is confirmed in a case study with postprocessed wind speed forecasts at five wind park locations in Colorado.

Improving ensemble forecasts is a complex process which involves proper scores such as the continuous ranked probability score (CRPS). A homogeneous Gaussian (hoG) model is introduced in order to better understand the characteristics of the CRPS. An analytical formula is derived for the expected CRPS of an ensemble in the hoG model. The score is a function of the variance of the error of the ensemble mean, the mean error of the ensemble mean and the ensemble variance. The hoG model also provides a score decomposition into reliability and resolution components. We examine whether the hoG model provides a useful approximation of the CRPS when applied to operational ECMWF medium‐range ensemble forecasts. The hoG approximation describes the spatial variations of the CRPS well while moderately overestimating the mean score. Seasonal averages over large domains are within 10% of the actual CRPS. Furthermore, the ability to approximate score changes is evaluated by (a) comparing raw ensemble forecasts with postprocessed ensemble forecasts, and (b) by examining score changes associated with a recent upgrade of the IFS. Overall, the hoG approximation predicts the actual CRPS changes well. One of the main anticipated applications of the hoG approximation are new diagnostics in verification software used by NWP developers routinely. The purpose of the diagnostics is to help developers explain impacts of forecast system changes on the CRPS in terms of the changes in mean error, changes in error variance and changes in ensemble variance. The diagnostics require little additional computational resources compared to the alternative of verifying postprocessed versions of the ensemble forecasts. Therefore, it will be feasible to apply the diagnostics easily to all variables that are examined as part of the model development process.

There is growing interest in the field of digital soil mapping (DSM) to quantify the underlying uncertainty of point predictions through different probabilistic prediction models. Uncertainty in DSM is often described in the format of a prediction interval (PI). Yet, PIs or uncertainty estimates in general are only of value for end users if they have good quality, i.e. when they are reliable. Reliability refers to the consistency between predicted conditional probabilities and observed frequencies of independent validation data. Ideally, PIs are also sharp, which refers to the concentration of probabilistic information, i.e. the narrowness of conditional probability distributions. The prediction interval coverage probability (PICP) is currently used in DSM to assess the reliability of PIs but it is ignorant to a potential one-sided bias of its bounding quantiles. Therefore, we propose to complement the current validation procedure with new metrics suggested in the broader probabilistic literature. This includes metrics that do not only evaluate uncertainty estimates in PI format but also quantiles or full conditional probability distributions. The newly proposed metrics are the quantile coverage probability (QCP), the probability integral transform (PIT) and so-called proper scoring rules for relative comparisons. Examples of scoring rules are the continuous ranked probability score (CRPS), which can be decomposed into a reliability part (RELI) and the interval score (IS). Sharpness can be evaluated through the prediction interval width (PIW). We illustrated the use of the various metrics in a case-study using soil pH data from The Land Use and Coverage Area Frame Survey (LUCAS). Thereby, uncertainty estimates of five different models were compared: Kriging with external drift (KED), quantile regression forest (QRF), quantile regression post-processing of a random forest (QRPP RF), quantile regression neural network (QRNN) and a reference null-model (NM).  KED, NM and QRPP RF showed very good reliability according to QCP, PICP, PIT and RELI. QRF was slightly pessimistic in the centre and QRNN very overoptimistic at the edges of the conditional probability distributions. Despite this, QRF performed best according to mean CRPS and mean IS because it produced fewer outliers at the edges. As expected, NM had the lowest sharpness, i.e. the largest PIW values. Sharpness of the other models was overall similar but QRNN had sharper predictions at the edges and QRF was less sharp in the centre of the conditional probability distributions. Lastly, we also generated PIW maps to indicate the spatial uncertainty of the five prediction models. The spatial variability of PIW was larger for QRF, QRNN and QRPP RF in comparison to KED. Whereas with NM, PIW was completely uniform. 

Ensemble prediction system (EPS) meteorological forecasts are increasingly being used as input to hydrological models in order to extend flood‐warning lead‐times and to improve forecasts and the knowledge of their uncertainty. Probabilistic skill scores classically used in meteorology are usually also utilized for assessing the quality of hydrological ensemble forecasts. However, the different river discharge magnitudes can make difficult the interpretation and comparisons of these scores, as it is for the continuous rank probability score (CRPS). In this letter, a novel ‘Reduction’ CRPS (RCRPS), which takes into account the different river discharge magnitudes, is proposed, and its usefulness is exhibited. Copyright © 2013 Royal Meteorological Society

Indian summer monsoon governs agricultural productivity, water resources, and economic activity affecting 1.4 billion people, yet accurate prediction remains challenging due to complex non-linear dynamics involving atmosphere-ocean interactions, land surface processes, and stochastic forcing. Traditional numerical weather prediction models solve deterministic partial differential equations but struggle capturing uncertainty quantification and sub-grid scale processes. Machine learning approaches demonstrate skill learning patterns from historical data but lack physical constraints and interpretability. This research develops hybrid framework integrating stochastic differential equations representing atmospheric dynamics with deep learning capturing non-linear relationships, enabling probabilistic monsoon forecasting with improved accuracy and uncertainty quantification. The research formulated Indian monsoon system as coupled stochastic differential equation system: dX_t = μ(X_t, θ)dt + σ(X_t, θ)dW_t, where X_t represents state variables (rainfall, temperature, humidity, pressure), μ denotes drift coefficient encoding deterministic dynamics, σ represents diffusion coefficient capturing stochastic fluctuations, W_t denotes Wiener process, and θ parameters learned from data. Long Short-Term Memory neural networks parameterized drift and diffusion coefficients enabling flexible representation of complex non-linear relationships while preserving stochastic process structure. Training utilized 40 years of daily rainfall data (1980-2019) from 64 India Meteorological Department stations across eight monsoon regions, supplemented by atmospheric reanalysis including temperature, humidity, wind, and sea surface temperature. Numerical integration employed Euler-Maruyama scheme with adaptive time stepping ensuring stability and accuracy. Model evaluation compared hybrid stochastic-machine learning approach against traditional numerical prediction, pure stochastic models, and pure machine learning on 2017-2019 validation period. Performance metrics included root mean square error, mean absolute error, R-squared, continuous ranked probability score, and skill scores assessing probabilistic forecast quality. Results demonstrated hybrid model achieved RMSE of 2.1 mm daily rainfall compared to 3.2 mm for pure stochastic models and 5.4 mm for deterministic numerical predictions, representing 34% accuracy improvement. R-squared reached 0.967 indicating excellent predictive skill. Probabilistic forecasts provided well-calibrated uncertainty estimates with continuous ranked probability score of 1.8 mm outperforming deterministic predictions. Spatial analysis revealed consistent performance across regions with coastal areas (R²=0.967) and northeast India (R²=0.956) achieving highest accuracy, while northwest plains showed moderate performance (R²=0.912) due to continental influences. Temporal validation demonstrated robust monsoon onset prediction with 89.3% accuracy detecting rainfall commencement within ±3 days. Computational efficiency analysis revealed hybrid model required 2.3 hours training on GPU hardware compared to 48 hours for traditional numerical models, enabling operational forecasting. The research establishes mathematical framework for physics-informed machine learning combining stochastic calculus with deep learning, providing interpretable probabilistic forecasts supporting agricultural planning, water resource management, and disaster preparedness across India's monsoon-dependent economy.

Forecast performance of data-driven models depends on the local weather and climate regime, which makes model selection a tedious task for forecast practitioners. Ensemble forecasting, or forecast combination, is beneficial in such cases, in that, forecasts from multiple models are combined to form a final forecast. In ensemble forecasting, additional to the final deterministic-style forecasts, predictive distributions are also available, which can be used by grid operators for better decision-making. Such empirical predictive distributions are useful to represent the uncertainty associated with the forecasts. However, raw ensemble forecasts are often not calibrated, e.g., due to the lack of diversity in the ensemble members. The lack of ensemble spread is known as underdispersion, and it can be ameliorated through post-processing.This study aims to calibrate hourly ensemble clear-sky index forecasts, generated by 20 data-driven models, using both parametric and nonparametric post-processing techniques. Four years of data collected at 7 research-grade sites are used in the empirical part of the paper. Quantitative and qualitative methods are used to evaluate the performance of post-processing techniques in terms of calibration and sharpness. Post-processed ensemble forecasts outperform raw ensemble forecasts under all verification metrics. The proposed parametric post-processing technique, namely, generalized additive models for location, scale and shape, substantially reduces the continuous ranked probability score (CRPS) of the raw ensemble forecasts from 32–59 W/m2 to 25–45 W/m2 and quantile score from 16–30 W/m2 to 13–23 W/m2. In terms of CRPS skill score, the proposed method achieved 38–58% improvements over a climatology reference.

Many proper scoring rules such as the Brier and log scoring rules implicitly reward a probability forecaster relative to a uniform baseline distribution. Recent work has motivated weighted proper scoring rules, which have an additional baseline parameter. To date two families of weighted proper scoring rules have been introduced, the weighted power and pseudospherical scoring families. These families are compatible with the log scoring rule: when the baseline maximizes the log scoring rule over some set of distributions, the baseline also maximizes the weighted power and pseudospherical scoring rules over the same set. We characterize all weighted proper scoring families and prove a general property: every proper scoring rule is compatible with some weighted scoring family, and every weighted scoring family is compatible with some proper scoring rule.

The detection performance of a conventional control chart is usually degraded by a large sample size as in Wang and Tsung. This paper proposes a new control chart under data‐rich environment. The proposed chart is based on the continuous ranked probability score and aims to simultaneously monitor the location and the scale parameters of any continuous process. We simulate different monitoring schemes with various shift patterns to examine the chart performance. Both in‐control and out‐of‐control performances are studied through simulation studies in terms of the mean, the standard deviation, the median, and some percentiles of the average run length distribution. Simulation results show that the proposed chart keeps a high sensitivity to shifts in location and/or scale without any distributional assumptions, and the outperformance improves, as the sample size becomes larger. Examples are given for illustration.

Abstract. Stream water temperatures influence water quality, with effects on aquatic biodiversity, drinking-water provision, electricity production, agriculture, and recreation. Therefore, stakeholders would benefit from an operational forecasting service that would support timely action. Deep-learning models are well-suited to providing probabilistic forecasts at individual stations of a monitoring network. Here, we train and evaluate several state-of-the-art models using 10 years of data from 54 stations across Switzerland. Static catchment features, time of the year, meteorological observations from the past 64 d, and their ensemble forecasts for the following 32 d are included as predictors in the models to estimate daily maximum water temperature over the next 32 d. Results show that the temporal fusion transformer (TFT) model performs best, with a continuous rank probability score (CRPS) of 0.70 °C averaged over all lead times, stations, and 90 forecasts distributed over 1 year. The TFT is followed by the recurrent neural network encoder–decoder, with a CRPS of 0.74 °C, and the neural hierarchical interpolation for time series, with a CRPS of 0.75 °C. These deep-learning models outperform other simpler models trained at each station: random forest (CRPS = 0.80 °C), multi-layer perceptron neural network (CRPS = 0.81 °C), and autoregressive linear model (CRPS = 0.96 °C). The average CRPS of the TFT degrades from 0.38 °C at lead a time of 1 d to 0.90 °C at a lead time of 32 d, largely driven by the uncertainty of the meteorological ensemble forecasts. In addition, TFT water temperature predictions at new and ungauged stations outperform those from the other models. When analyzing the importance of model inputs, we find a dominant role of observed water temperature and future air temperature, while including precipitation and time of the year further improves predictive skill. Operational probabilistic forecasts of daily maximum water temperature are generated twice per week with our TFT model and are publicly available at https://www.drought.ch/de/impakt-vorhersagen-malefix/wassertemperatur-prognosen/ (last access: 20 March 2025). Overall, this study provides insights into the extended-range predictability of stream water temperature and into the applicability of deep-learning models in hydrology.

DDOS-attacks detection using an efficient measurement-based statistical mechanism

From optimal supply decisions to anticipatory control systems, wind-based energy applications rely heavily upon accurate, local, short-term forecasts of future wind speed. Recent studies have shown continuous ranked probability score (CRPS) minimizing models with Gaussian assumptions to be effective for well-researched sites where those assumptions are appropriate. We consider the more general case where Gaussianity is not assumed and access to historical data may be constrained. Deriving a CRPS expression for a minimum Extreme Value distribution, we use it to propose a site-adaptive Weibull-based CRPS-minimizing model, which is tested and shown to perform better than both deterministic and probabilistic reference models on a ground-based array of weather observation sites in northern Japan.

In numerical weather prediction (NWP), the uncertainty about the future state of the atmosphere is described by a set of forecasts (called an ensemble). All ensembles have deficiencies that can be corrected via statistical post-processing methods. Several ensembles, based on different NWP models, exist and may be corrected using different statistical methods. These raw or post-processed ensembles can thus be combined. The theory of prediction with expert advice allows us to build combination algorithms with theoretical guarantees on the forecast performance. We adapt this theory to the case of probabilistic forecasts issued as stepwise cumulative distribution functions, computed from raw and post-processed ensembles. The theory is applied to combine wind speed ensemble forecasts. The second goal of this study is to explore the use of two forecast performance criteria: the continuous ranked probability score (CRPS) and the Jolliffe–Primo test. The usual way to build skilful probabilistic forecasts is to minimize the CRPS. Minimizing the CRPS may not produce reliable forecasts according to the Jolliffe–Primo test. The Jolliffe–Primo test generally selects reliable forecasts, but could lead to issuing suboptimal forecasts in terms of CRPS. We propose to use both criteria to achieve reliable and skilful probabilistic forecasts.

Scoring rules measure the accuracy or epistemic utility of a credence assignment. A significant literature uses plausible conditions on scoring rules on finite sample spaces to argue for both probabilism—the doctrine that credences ought to satisfy the axioms of probabilism—and for the optimality of Bayesian update as a response to evidence. I prove a number of formal results regarding scoring rules on infinite sample spaces that impact the extension of these arguments to infinite sample spaces. A common condition in the arguments for probabilism and Bayesian update is strict propriety: that according to each probabilistic credence, the expected accuracy of any other credence is worse. Much of the discussion needs to divide depending on whether we require finite or countable additivity of our probabilities. I show that in a number of natural infinite finitely additive cases, there simply do not exist strictly proper scoring rules, and the prospects for arguments for probabilism and Bayesian update are limited. In many natural infinite countably additive cases, on the other hand, there do exist strictly proper scoring rules that are continuous on the probabilities, and which support arguments for Bayesian update, but which do not support arguments for probabilism. There may be more hope for accuracy-based arguments if we drop the assumption that scores are extended-real-valued. I sketch a framework for scoring rules whose values are nets of extended reals, and show the existence of a strictly proper net-valued scoring rules in all infinite cases, both for f.a. and c.a. probabilities. These can be used in an argument for Bayesian update, but it is not at present known what is to be said about probabilism in this case.

We present an ensemble prediction system using a Deep Learning Weather Prediction (DLWP) model that recursively predicts six key atmospheric variables with six‐hour time resolution. This computationally efficient model uses convolutional neural networks (CNNs) on a cubed sphere grid to produce global forecasts. The trained model requires just three minutes on a single GPU to produce a 320‐member set of six‐week forecasts at 1.4° resolution. Ensemble spread is primarily produced by randomizing the CNN training process to create a set of 32 DLWP models with slightly different learned weights. Although our DLWP model does not forecast precipitation, it does forecast total column water vapor and gives a reasonable 4.5‐day deterministic forecast of Hurricane Irma. In addition to simulating mid‐latitude weather systems, it spontaneously generates tropical cyclones in a one‐year free‐running simulation. Averaged globally and over a two‐year test set, the ensemble mean RMSE retains skill relative to climatology beyond two‐weeks, with anomaly correlation coefficients remaining above 0.6 through six days. Our primary application is to subseasonal‐to‐seasonal (S2S) forecasting at lead times from two to six weeks. Current forecast systems have low skill in predicting one‐ or 2‐week‐average weather patterns at S2S time scales. The continuous ranked probability score (CRPS) and the ranked probability skill score (RPSS) show that the DLWP ensemble is only modestly inferior in performance to the European Center for Medium Range Weather Forecasts (ECMWF) S2S ensemble over land at lead times of 4 and 5–6 weeks. At shorter lead times, the ECMWF ensemble performs better than DLWP.