Continuous Distributions and Measures of Statistical Accuracy for Structured Expert Judgment

ObjectivesAssessing the cost-effectiveness of complex pharmaceutical care interventions and medication error outcomes is hindered by lack of available data on actual outcomes consequent to errors that were intercepted for patient...

Self-compacting concrete (SCC) has been developed as a type of concrete capable of filling narrow gaps in highly reinforced areas of a mold without internal or external vibration. Bleeding and segregation in SCC can be prevented by the addition of superplasticizers. Due to these favorable properties, SCC has been adopted worldwide. The workability of SCC is closely related to its yield stress and plastic viscosity levels. Therefore, the accurate prediction of yield stress and plastic viscosity of SCC has certain advantages. Predictions of the shear stress and plastic viscosity of SCC is presented in the current study using four different ensemble machine learning techniques: Light Gradient Boosting Machine (LightGBM), Extreme Gradient Boosting (XGBoost), random forest, and Categorical Gradient Boosting (CatBoost). A new database containing the results of slump flow, V-funnel, and L-Box tests with the corresponding shear stress and plastic viscosity values was curated from the literature to develop these ensemble learning models. The performances of these algorithms were compared using state-of-the-art statistical measures of accuracy. Afterward, the output of these ensemble learning algorithms was interpreted with the help of SHapley Additive exPlanations (SHAP) analysis and individual conditional expectation (ICE) plots. Each input variable’s effect on the predictions of the model and their interdependencies have been illustrated. Highly accurate predictions could be achieved with a coefficient of determination greater than 0.96 for both shear stress and plastic viscosity.

The theoretical atomization energies of some 45 CxHyOz molecules present in the Active Thermochemical Tables compilation and of particular interest to the combustion chemistry community have been computed using five composite model chemistries as titled. The species contain between 1-8 "heavy" atoms, and a few are conformationally diverse with up to nine conformers. The enthalpies of formation at 0 and 298.15 K are then derived via the atomization method and compared against the recommended values. In general, there is very good agreement between our averaged computed values and those in the ATcT; those for 1,3-cyclopentadiene exceptionally differ considerably, and we show from isodesmic reactions that the true value for 1,3-cyclopentadiene is closer to 134 kJ mol(-1) than the reported 101 kJ mol(-1). If one is restricted to using a single method, statistical measures indicate that the best methods are in the rank order G3 ≈ G4 > W1BD > CBS-APNO > CBS-QB3. The CBS-x methods do on average predict ΔfH(⊖)(298.15 K) within ≈5 kJ mol(-1) but are prone to occasional lapses. There are statistical advantages to be gained from using a number of methods in tandem, and all possible combinations have been tested. We find that the average formation enthalpy coming from using CBS-APNO/G4, CBS-APNO/G3, and G3/G4 show lower mean signed and mean unsigned errors, and lower standard and root-mean-squared deviations, than any of these methods in isolation. Combining these methods also leads to the added benefit of providing an uncertainty rooted in the chemical species under investigation. In general, CBS-APNO and W1BD tend to underestimate the formation enthalpies of target species, whereas CBS-QB3, G3, and G4 have a tendency to overestimate the same. Thus, combining CBS-APNO with a G3/G4 combination leads to an improvement in all statistical measures of accuracy and precision, predicting the ATcT values to within 0.14 ± 4.21 kJ mol(-1), thus rivalling "chemical accuracy" (±4.184 kJ mol(-1)) without the excessive cost associated with higher-level methods such as W1BD.

There is growing interest in the field of digital soil mapping (DSM) to quantify the underlying uncertainty of point predictions through different probabilistic prediction models. Uncertainty in DSM is often described in the format of a prediction interval (PI). Yet, PIs or uncertainty estimates in general are only of value for end users if they have good quality, i.e. when they are reliable. Reliability refers to the consistency between predicted conditional probabilities and observed frequencies of independent validation data. Ideally, PIs are also sharp, which refers to the concentration of probabilistic information, i.e. the narrowness of conditional probability distributions. The prediction interval coverage probability (PICP) is currently used in DSM to assess the reliability of PIs but it is ignorant to a potential one-sided bias of its bounding quantiles. Therefore, we propose to complement the current validation procedure with new metrics suggested in the broader probabilistic literature. This includes metrics that do not only evaluate uncertainty estimates in PI format but also quantiles or full conditional probability distributions. The newly proposed metrics are the quantile coverage probability (QCP), the probability integral transform (PIT) and so-called proper scoring rules for relative comparisons. Examples of scoring rules are the continuous ranked probability score (CRPS), which can be decomposed into a reliability part (RELI) and the interval score (IS). Sharpness can be evaluated through the prediction interval width (PIW). We illustrated the use of the various metrics in a case-study using soil pH data from The Land Use and Coverage Area Frame Survey (LUCAS). Thereby, uncertainty estimates of five different models were compared: Kriging with external drift (KED), quantile regression forest (QRF), quantile regression post-processing of a random forest (QRPP RF), quantile regression neural network (QRNN) and a reference null-model (NM).  KED, NM and QRPP RF showed very good reliability according to QCP, PICP, PIT and RELI. QRF was slightly pessimistic in the centre and QRNN very overoptimistic at the edges of the conditional probability distributions. Despite this, QRF performed best according to mean CRPS and mean IS because it produced fewer outliers at the edges. As expected, NM had the lowest sharpness, i.e. the largest PIW values. Sharpness of the other models was overall similar but QRNN had sharper predictions at the edges and QRF was less sharp in the centre of the conditional probability distributions. Lastly, we also generated PIW maps to indicate the spatial uncertainty of the five prediction models. The spatial variability of PIW was larger for QRF, QRNN and QRPP RF in comparison to KED. Whereas with NM, PIW was completely uniform. 

Many proper scoring rules such as the Brier and log scoring rules implicitly reward a probability forecaster relative to a uniform baseline distribution. Recent work has motivated weighted proper scoring rules, which have an additional baseline parameter. To date two families of weighted proper scoring rules have been introduced, the weighted power and pseudospherical scoring families. These families are compatible with the log scoring rule: when the baseline maximizes the log scoring rule over some set of distributions, the baseline also maximizes the weighted power and pseudospherical scoring rules over the same set. We characterize all weighted proper scoring families and prove a general property: every proper scoring rule is compatible with some weighted scoring family, and every weighted scoring family is compatible with some proper scoring rule.

When physical and data sciences are not sufficient to support models and/or decisions, expert judgment is a recognized approach to quantify the uncertainties around specific issues. Among different expert judgment methods, Structured Expert Judgment (SEJ) employs a formalized, documented procedure for obtaining probabilistic belief statements from a group of experts about unknown quantities or parameters. This provides an attractive approach for performing assessments at volcanoes characterized by large knowledge gaps by integrating diverse kinds of information. Uncertainties are likely to be large in these cases, and SEJ can quantify these uncertainties to provide scientists and decision-makers with indications of the reliability of the assessments. The final goal of this approach is to obtain the group’s synthesized uncertainty distribution (representing a new “virtual expert” often called Decision Maker - DM) around specific items (or “target questions”), that result from combining elicited judgments of all the experts.More specifically, performance-based expert elicitation relies on validating expert probability assessments through an impartial empirical trial. Thus, in a performance-based elicitation, experts are tasked with providing their estimations of probability distributions for a set of known quantities, often referred to as “seed items,” which, jointly, serve as calibration benchmarks for expert performance. Experts’ responses provide the basis for performance scoring using the Classical Model algorithm (Cooke, 1991), determined empirically on the individual expert’s attainment, jointly in terms of two separate metrics: “statistical accuracy” and “informativeness”, which are evaluated on the set of seed items overall.Traditionally, the management of a performance-based expert elicitation is time consuming and involves the collection of tens of questionnaires, often manually copied from hard-copies of the responses or email attachments. The performance-calibration algorithms and the production of standard outputs, including statistical samples of the DM responses, are necessary steps every time an elicitation is conducted, but relied on different scripts and independent pieces of software.In this study we present the latest version of the recently released software ELICIPY (de’Michieli Vitturi et al. 2024), which allows organizing and managing performance-based expert elicitation sessions in a partially automated way, resulting in a significantly enhanced efficiency. This new version includes, among other improvements, an “agreement index” to quantify the level of agreement among experts, a continuous version of the Classical Model to compute expert weights, the possibility to import weights from an external file, new plot options, an interactive dashboard for results exploration.ELICIPY enables the smooth conduction of elicitations with a relatively large number of questions and/or experts. Moreover, the management of multiple elicitation sessions, with questionnaire modifications and response updates are made easier. Some of these improvements are demonstrated on the case study of a hazard and risk assessment for the active Kolumbo submarine Volcano (Aegean Sea, Greece), which was made of 20 Seed Items and 64 Target Items in total, the latter structured into 17 subject matter groups (Bevilacqua et al., 2025).

Sustainable tourism, unlike conventional tourism, explicitly strives to maintain economic, environmental, and social sustainability in tourist destinations. Saipan, the capital and the largest island of the chain of Pacific islands known as the Commonwealth of the Northern Mariana Islands (CNMI), is a tropical destination that heavily relies on the tourist industry as a source of income. When analyzing the trends of tourist arrivals from the top three countries that account for the majority of tourist arrivals in Saipan-China, Japan, and Korea-the necessity to implement sustainable tourism in Saipan becomes clear. This article forecasts tourism demand, which is one of the main factors in determining sustainable tourism on a small island. Analyzing both long-term and cyclical trends in tourist arrivals assists budget-constrained policymakers and related organizations in implementing sustainable tourism. Therefore, forecasting accuracy is essential in tourism policy and planning. The tourist arrival data obtained from Marianas Visitors Authority’s official Japanese website are made into a time series for Chinese, Japanese, and Korean visitor data from 2006 to 2016. Given the various time-series forecasting models, we employ simple but powerful forecasting techniques, such as ARIMA and ETS to control for the possible seasonality and trend in our tourism data. Using these forecast methods, the tourist arrival data is used to predict the future trends of tourist arrivals in Saipan for the next twelve months. The performance of our forecasting model in this study is evaluated based on the five measures of accuracy, as measured by comparison with actual tourism flows: mean error, mean absolute scaled error, mean percentage error, mean absolute error, and mean absolute percentage error. A comparison of the two popular forecast techniques in this study shows that the ETS model statistically outperforms the ARIMA model in terms of various measures of accuracy. A practical implication of this finding is that the ETS approach should be used when forecasting tourism demand for small islands. We predict the visitor arrivals from China, Japan, and Korea to Saipan by using the better model. The estimated 80% and 95% intervals for the number of tourist arrivals by the ETS model are presented in this study. The upper (lower) bound is the maximum (minimum) value of the estimated tourist arrivals. We find strong evidence that the tourist arrival trend for Korean visitors to Saipan will increase in the short run; although, the Chinese and Japanese tourist arrival trends seems to be stable, except for Japanese tourist arrivals in August. Nonetheless, with such an intense demand for Saipan’s natural resources, sustainable island tourism becomes necessary. Our estimated result is useful to practitioners in the sense that they are able to prepare for carrying capacities and the means of destination management on Saipan. Proper management for maintaining the island environmentally, economically, and socially requires the preservation of indigenous culture, local quality of life, natural resources, and economic stability while accommodating the demands of arriving tourists. It is crucial to preserve the assets of Saipan in order to maintain and improve its economy. Thus, our time series modeling for forecasting the number of visitors becomes necessary to develop overall economic, environmental, and social sustainability in the island. An interesting addition to implementing sustainable tourism in Saipan is the idea of maintaining a “cap” on the number of incoming tourists visiting the island. Perhaps, with more research on Saipan’s environment, specifically an estimate of how much of the island’s resources tourists exhaust and how much the island can withstand the aforementioned, this “cap” could be practical to prevent perpetual damage to Saipan’s biodiversity, resources, culture, and economy. Along with this idea, knowing the minimum number of tourists needed to sustain Saipan’s necessities is valuable.

Streamflow forecasting is a vital task for hydrology and water resources engineering, and the different artificial intelligence (AI) approaches have been employed for this purposes until now. Additionally, the forecasting accuracy and uncertainty estimation are the meaningful assignments that need to be recognized. The addressed research investigates the potential of novel ensemble approach, Bayesian model averaging (BMA), in streamflow forecasting using daily time series data from two stations (i.e., Hongcheon and Jucheon), South Korea. Six categories (i.e., M1–M6) of input combination using different antecedent times were employed for streamflow forecasting. The outcomes of BMA model were compared with those of multivariate adaptive regression spline (MARS), M5 model tree (M5Tree), and Kernel extreme learning machines (KELM) models considering four assessment indexes, root mean square error (RMSE), Nash-Sutcliffe efficiency (NSE), correlation coefficient (R), and mean absolute error (MAE). The results revealed the superior accuracy of BMA model over three machine learning models in daily streamflow forecasting. Considering RMSE values among the best models during testing phase, the best BMA model (i.e., BMA2) enhanced the forecasting accuracy of MARS1, M5Tree4, and KELM3 models by 5.2%, 5.8%, and 3.4% in Hongcheon station. Additionally, the best BMA model (i.e., BMA1) improved the forecasting accuracy of MARS1, M5Tree1, and KELM1 models by 6.7%, 9.5%, and 3.7% in Jucheon station. In addition, the best BMA models in both stations allowed the uncertainty estimation, and produced higher uncertainty of peak flows compared to that of low flows. As one of the most robust and effective tools, therefore, the BMA model can be successfully employed for streamflow forecasting with different antecedent times.

Bovine respiratory disease (BRD) is an important health problem in dairy heifers. BRD causes considerable losses, particularly on farms that experience high levels of the disease. However, an exact quantification of the economic losses due to BRD was not available yet. Despite this lack of economic insight, dairy farmers have to make decisions with regard to prevention of the disease. To make these decisions as economically sound as possible, more accurate insight is necessary into the economic consequences of BRD on the individual dairy farm.The main objective of the research project described in this thesis was to obtain insight into the on-farm economic consequences of BRD in dairy heifers by means of a PC-based simulation model. The second objective was to collect information on the epidemiological consequences of the disease indispensable for model input.The research started with a literature review aimed at obtaining the necessary qualitative and quantitative information on both the effects of BRD on the productivity of dairy heifers and risk factors of the disease. Because relevant literature turned out to be scarce, a formal expert judgement study was held to obtain additional data on the(se) variables of interest.As a next step, a simulation model was developed that calculates the economic losses due to BRD in dairy heifers for individual dairy farm conditions in the Netherlands. Following the results of the expert judgement study, the model distinguishes between two BRD types, being calf pneumonia and a seasonal BRD outbreak.Model calculations showed that for most dairy farms in the Netherlands the economic losses due to BRD will be relatively small: around 1 % of the farm's net return to labour and management for average situations, increasing up to 3-4 % at worst. For individual farms that experience high levels of BRD, the associated losses can be as high as 10-15 % of the farm's net return to labour and management, up to 25 % for large farms. Besides for calculation of the economic losses due to BRD, the model showed also to be useful for evaluation of the on-farm cost-effectiveness of prevention of the disease. Moreover, the model is flexible and user-friendly, hence, can be used as a tool to support decision-making in dairy practice.

Expert judgement studies are a relatively new way of quantifying technical uncertainty in engineering design. Unfortunately they can be time-consuming and costly because they require physical meetings between a group of possibly widely dispersed experts. This paper investigates how web-hosted collaboration or ‘virtual teaming’ can be used to carry out such studies. It shows in particular how the functionality of commercially available web-hosted project spaces can be enhanced to meet the flexibility, democracy and transparency required for such studies to work.

Subject. The article addresses the logical and statistical modeling of financial stability indicators of enterprises. Objectives. The study aims at logical and statistical modeling of integral indicator of financial stability, and comparison of the results with expert estimates. Methods. The study employs analysis and synthesis, dialectical cognition and abstraction, expert assessments, and logical and statistical modeling. Results. Based on the assessment of financial stability coefficients (absolute liquidity, financial independence, financial stability, equity maneuverability, profitability of sales), we calculated maximum values of corresponding integral indicator for enterprises of the Donetsk People's Republic. The consistent analysis of crisis and normal values of the studied indicators enabled to substantiate the range of changes in the integral indicator of financial stability. The results of calculated variance of reproducibility helped underpin the coefficients of regression equation of the corresponding integral indicator of financial stability. Conclusions. To analyze the level of financial stability of the studied heavy and light industry enterprises operating in the Donetsk People's Republic, it is preferable to implement a fractional factorial model, and for trade and service enterprises – an expert model.

BackgroundAccording to the World Health Organization, 600 million cases of foodborne disease occurred in 2010. To inform risk management strategies aimed at reducing this burden, attribution to specific foods is necessary.ObjectiveWe present attribution estimates for foodborne pathogens (Campylobacter spp., enterotoxigenic Escherichia coli (ETEC), Shiga-toxin producing E. coli, nontyphoidal Salmonella enterica, Cryptosporidium spp., Brucella spp., and Mycobacterium bovis) in three African countries (Burkina Faso, Ethiopia, Rwanda) to support risk assessment and cost-benefit analysis in three projects aimed at increasing safety of beef, dairy, poultry meat and vegetables in these countries.MethodsWe used the same methodology as the World Health Organization, i.e., Structured Expert Judgment according to Cooke’s Classical Model, using three different panels for the three countries. Experts were interviewed remotely and completed calibration questions during the interview without access to any resources. They then completed target questions after the interview, using resources as considered necessary. Expert data were validated using two objective measures, calibration score or statistical accuracy, and information score. Performance-based weights were derived from the two measures to aggregate experts’ distributions into a so-called decision maker. The analysis was made using Excalibur software, and resulting distributions were normalized using Monte Carlo simulation.ResultsIndividual experts’ uncertainty assessments resulted in modest statistical accuracy and high information scores, suggesting overconfident assessments. Nevertheless, the optimized item-weighted decision maker was statistically accurate and informative. While there is no evidence that animal pathogenic ETEC strains are infectious to humans, a sizeable proportion of ETEC illness was attributed to animal source foods as experts considered contamination of food products by infected food handlers can occur at any step in the food chain. For all pathogens, a major share of the burden was attributed to food groups of interest. Within food groups, the highest attribution was to products consumed raw, but processed products were also considered important sources of infection.ConclusionsCooke’s Classical Model with performance-based weighting provided robust uncertainty estimates of the attribution of foodborne disease in three African countries. Attribution estimates will be combined with country-level estimates of the burden of foodborne disease to inform decision making by national authorities.

SEXTAMT: A systematic map to navigate the wide seas of factors affecting expert judgment software estimates

Some properties of progressive censored order statistics from arbitrary and uniform distributions with applications to inference and simulation

In a paper written some 25 years ago, I distinguished three contexts in which one might wish to combine expert judgements of uncertainty: the expert problem, the group decision problem and the textbook problem. Over the intervening years much has been written on the first two, which have the focus of a single decision context, but little on the third, though the closely related field of meta-analysis has developed considerably. With many developments in internet technology, particularly in relation to interactivity and communication, the textbook problem is gaining in importance since data and expert judgements can be made available over the web to be used by many different individuals to shape their own beliefs in many different contexts. Moreover, applications such as web-based decision support, e-participation and e-democracy are making algorithmic ‘solutions’ to the group decision problem attractive, despite many results showing we know that such solutions are, at best, rare and, at worst, illusory. In this paper I survey developments since my earlier paper and note some unresolved issues. Then I turn to how expert judgement might be used within web-based group decision support, as well as in e-participation and e-democracy contexts. The latter points to a growing importance of the textbook problem and suggests that Cooke’s principles for scientific reporting of expert judgement studies may need enhancing for such studies to be used by a wider audience.