Recent Trends in Information Elicitation

Many proper scoring rules such as the Brier and log scoring rules implicitly reward a probability forecaster relative to a uniform baseline distribution. Recent work has motivated weighted proper scoring rules, which have an additional baseline parameter. To date two families of weighted proper scoring rules have been introduced, the weighted power and pseudospherical scoring families. These families are compatible with the log scoring rule: when the baseline maximizes the log scoring rule over some set of distributions, the baseline also maximizes the weighted power and pseudospherical scoring rules over the same set. We characterize all weighted proper scoring families and prove a general property: every proper scoring rule is compatible with some weighted scoring family, and every weighted scoring family is compatible with some proper scoring rule.

Proper scoring rules serve to measure subjective degrees of belief. Traditional proper scoring rules are based on the assumption of expected value maximization. There are, however, many deviations from expected value, primarily due to risk aversion. Correcting techniques have been proposed in the literature for deviations due to nonlinear utility. These techniques still assumed expected utility maximization. More recently, corrections for deviations from expected utility have been proposed. The latter concerned, however, only the quadratic scoring rule, and could handle only half of the domain of subjective beliefs. Further, beliefs close to 0.5 could not be discriminated. This paper generalizes the correcting techniques to all bounded binary proper scoring rules, covers the whole domain of beliefs and, in particular, can discriminate between all degrees of belief. Thus, we fully extend the properness requirement (in the sense of identifying all degrees of subjective beliefs) to virtually all models that deviate from expected value. Copyright © 2011 John Wiley & Sons, Ltd.

Proper scoring rules provide convenient and highly efficient tools for incentive-compatible elicitations of subjective beliefs. As traditionally used, however, they are valid only under expected value maximization. This paper shows how they can be generalized to modern (non-expected utility) theories of risk and ambiguity, yielding mutual benefits: users of scoring rules can benefit from the empirical realism of non-expected utility, and analysts of ambiguity attitudes can benefit from efficient measurements using proper scoring rules. An experiment demonstrates the feasibility of our generalization.

Scoring rules measure the accuracy or epistemic utility of a credence assignment. A significant literature uses plausible conditions on scoring rules on finite sample spaces to argue for both probabilism—the doctrine that credences ought to satisfy the axioms of probabilism—and for the optimality of Bayesian update as a response to evidence. I prove a number of formal results regarding scoring rules on infinite sample spaces that impact the extension of these arguments to infinite sample spaces. A common condition in the arguments for probabilism and Bayesian update is strict propriety: that according to each probabilistic credence, the expected accuracy of any other credence is worse. Much of the discussion needs to divide depending on whether we require finite or countable additivity of our probabilities. I show that in a number of natural infinite finitely additive cases, there simply do not exist strictly proper scoring rules, and the prospects for arguments for probabilism and Bayesian update are limited. In many natural infinite countably additive cases, on the other hand, there do exist strictly proper scoring rules that are continuous on the probabilities, and which support arguments for Bayesian update, but which do not support arguments for probabilism. There may be more hope for accuracy-based arguments if we drop the assumption that scores are extended-real-valued. I sketch a framework for scoring rules whose values are nets of extended reals, and show the existence of a strictly proper net-valued scoring rules in all infinite cases, both for f.a. and c.a. probabilities. These can be used in an argument for Bayesian update, but it is not at present known what is to be said about probabilism in this case.

We present a study on the evolution of publications about applications of proper scoring rules. Specifically, we consider articles reporting the use of proper scoring rules when either measuring the accuracy of forecasts or for inducing honest reporting of private information within a certain context. Our analysis of a data set containing 201 articles published between 1950 and 2015 suggests that there has been a tremendous increase in the number of published articles about proper scoring rules over the years. Moreover, the weather/climate, prediction markets, psychology, and energy domains are the four most popular application areas. After providing some insights on how proper scoring rules are applied in different domains, we analyze the publication outlets where the articles in our data set were published. In this regard, we find that an increasing number of articles are now being published in conference proceedings related to artificial intelligence, as opposed to traditional academic journals. We conclude this review by suggesting that the wisdom-of-crowds phenomenon might be a driving force behind the recent popularity of proper scoring rules.

Probabilistic sensitivity measures as information value

This paper provides self-contained proof of a theorem relating probabilistic coherence of forecasts to their non-domination by rival forecasts with respect to any proper scoring rule. The theorem recapitulates insights achieved by other investigators, and clarifies the connection of coherence and proper scoring rules to Bregman divergence.

Evaluation of Probabilistic Forecasts: Proper Scoring Rules and Moments

When predicting future events, it is common to issue forecasts that are probabilistic, in the form of probability distributions over the range of possible outcomes. Such forecasts can be evaluated using proper scoring rules. Proper scoring rules condense forecast performance into a single numerical value, allowing competing forecasters to be ranked and compared. To facilitate the use of scoring rules in practical applications, the scoringRules package in R provides popular scoring rules for a wide range of forecast distributions. This paper discusses an extension to the scoringRules package that additionally permits the implementation of popular weighted scoring rules. Weighted scoring rules allow particular outcomes to be targeted during forecast evaluation, recognizing that certain outcomes are often of more interest than others when assessing forecast quality. This introduces the potential for very flexible, user-oriented evaluation of probabilistic forecasts. We discuss the theory underlying weighted scoring rules, and describe how they can readily be implemented in practice using scoringRules. Functionality is available for weighted versions of several popular scoring rules, including the logarithmic score, the continuous ranked probability score, and the energy score. Two case studies are presented to demonstrate this, whereby weighted scoring rules are applied to univariate and multivariate probabilistic forecasts in the fields of meteorology and economics.

Proper scoring rules provide a theoretically principled framework for the quantitative assessment of the predictive performance of probabilistic forecasts. While a wide selection of such scoring rules for univariate quantities exists, there are only few scoring rules for multivariate quantities, and many of them require that forecasts are given in the form of a probability density function. The energy score, a multivariate generalization of the continuous ranked probability score, is the only commonly used score that is applicable in the important case of ensemble forecasts, where the multivariate predictive distribution is represented by a finite sample. Unfortunately, its ability to detect incorrectly specified correlations between the components of the multivariate quantity is somewhat limited. In this paper the authors present an alternative class of proper scoring rules based on the geostatistical concept of variograms. The sensitivity of these variogram-based scoring rules to incorrectly predicted means, variances, and correlations is studied in a number of examples with simulated observations and forecasts; they are shown to be distinctly more discriminative with respect to the correlation structure. This conclusion is confirmed in a case study with postprocessed wind speed forecasts at five wind park locations in Colorado.

Proper scoring rules for general decision models

SummaryIn Bayesian inference, predictive distributions are typically in the form of samples generated via Markov chain Monte Carlo or related algorithms. In this paper, we conduct a systematic analysis of how to make and evaluate probabilistic forecasts from such simulation output. Based on proper scoring rules, we develop a notion of consistency that allows to assess the adequacy of methods for estimating the stationary distribution underlying the simulation output. We then provide asymptotic results that account for the salient features of Bayesian posterior simulators and derive conditions under which choices from the literature satisfy our notion of consistency. Importantly, these conditions depend on the scoring rule being used, such that the choices of approximation method and scoring rule are intertwined. While the logarithmic rule requires fairly stringent conditions, the continuous ranked probability score yields consistent approximations under minimal assumptions. These results are illustrated in a simulation study and an economic data example. Overall, mixture‐of‐parameters approximations that exploit the parametric structure of Bayesian models perform particularly well. Under the continuous ranked probability score, the empirical distribution function is a simple and appealing alternative option.

Proper scoring rules, particularly when used as the basis for a prediction market, are powerful tools for eliciting and aggregating beliefs about events such as the likely outcome of an election or sporting event. Such scoring rules incentivize a single agent to reveal her true beliefs about the event. Othman and Sandholm [16] introduced the idea of a decision rule to examine these problems in contexts where the information being elicited is conditional on some decision alternatives. For example, "What is the probability having ten million viewers if we choose to air new television show X? What if we choose Y?" Since only one show can actually air in a slot, only the results under the chosen alternative can ever be observed. Othman and Sandholm developed proper scoring rules (and thus decision markets) for a single, deterministic decision rule: always select the the action with the greatest probability of success. In this work we significantly generalize their results, developing scoring rules for other deterministic decision rules, randomized decision rules, and situations where there may be more than two outcomes (e.g. less than a million viewers, more than one but less than ten, or more than ten million).

Proper scoring rules, i.e., devices of a certain class for eliciting a person's probabilities and other expectations, are studied, mainly theoretically but with some speculations about application. The relation of proper scoring rules to other economic devices and to the foundations of the personalistic theory of probability is brought out. The implications of various restrictions, especially symmetry restrictions, on scoring rules is explored, usually with a minimum of regularity hypothesis.

Mechanisms for information elicitation