Bayesian Aggregation of Experts' Forecasts

In response to Coronavirus disease 2019 (COVID-19) global pandemic, various COVID-19 vaccines were rapidly administered under emergency use authorization. Rare outcomes associated with COVID-19 vaccines might be less likely to be captured in clinical trials, leading to a knowledge gap in real-world vaccine safety. In contrast with high-income countries, many low-to-middle income countries have limited capacity to conduct active surveillance, owing to the absence of large and fully-integrated health information databases. This paper describes the study protocol, which aims to investigate risk of prespecified adverse events of special interests following COVID-19 vaccination in a partially integrated health information system with non-shareable electronic health records. The SAFECOVAC study is a longitudinal, observational retrospective study of active safety surveillance using case-based monitoring approach. This involves linkage of several administrative databases and hospitalization data monitoring to identify adverse events of special interests following administration of COVID-19 vaccines in Malaysia. The source population comprises of all individuals who received at least one dose of COVID-19 vaccine. Self-controlled design and vaccinated case-coverage design will be employed to assess risk of adverse events of special interests and determine the association with vaccine exposure. Data on vaccination records will be obtained from the national COVID-19 vaccination register to identify the vaccination platforms, doses and the timing of vaccinations. The outcome of this study is hospitalization for the adverse events of special interests between March 2021 and June 2022. The outcomes will be obtained through linkage with hospital admission database and national pharmacovigilance database. Findings will provide analysis of real-world data which can inform deliberations by government and public health decision makers relative to the refinement of COVID-19 vaccination recommendations.

Many statistical methods such as (generalized) linear regression, time series approaches or geostatistical techniques are concerned with forecasting and interpolating uncertain, unknown, or partially known quantities. To address and quantify prediction uncertainty, the forecaster ought to provide a full predictive probability distribution over the quantity or event of interest, rather than just a point forecast. In this paper we provide a brief review on diagnostic and quantitative methods for the evaluation of predictive distributions that have developed in the context of probabilistic forecasting. We also discuss the issue of deriving optimal point forecasts from a predictive distribution and illustrate the proposed evaluation methods with a data example on one day ahead predictions of ozone concentrations.

Human decision makers think categorically when facing a great variety of objects due to their limited information processing capacities. The categorical thinking by Bayesian decision makers is modeled as classifying objects into a small number of categories with respect to their prior probabilities. The classification follows a quantization rule for the prior probabilities. The categorical thinking enables decision makers to handle infinitely many objects but simultaneously causes them to lose precision of prior probabilities and, consequently, of decisions. This chapter considers group decision making by imperfect agents that only knowquantized prior probabilities for use in Bayesian likelihood ratio tests. Global decisions are made by information fusion of local decisions, but information sharing among agents before local decision making is forbidden. The quantization rule of the agents is investigated so as to achieve the minimum mean Bayes risk; optimal quantizers are designed by a novel extension to the Lloyd-Max algorithm. It is proven that agents using identical quantizers are not optimal. Thus diversity in the individual agents’ quantizers leads to optimal performance. In addition, for comparison, it is shown how much their performance gets better when information sharing and collaboration among agents before local decision making is allowed.KeywordsDecision RulePrior ProbabilityRepresentation PointFusion CenterFusion RuleThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The online social networks have experienced an unprecedented proliferation. Various platforms change the way people learning about information, particularly on ongoing events, which can only be known from mainstream media in the past. The social media platforms have many pleasing properties compared with traditional media: convenient, detailed, fast and interactive. This provides us an opportunity to learn immediate and detailed information about event of interest, which is highly preferred by decision makers and the public especially when emergencies and significant events happen.

There are two main schools of thought about statistical inference: frequentist and Bayesian. The frequentist approach relies solely on available data for predictions, while the Bayesian approach incorporates both data and prior knowledge about the event of interest. Bayesian methods were developed hundreds of years ago; however, they were rarely used due to computational challenges and conflicts between the two schools of thought. Recent advances in computational capabilities and a shift toward leveraging prior knowledge for inferences have led to increased use of Bayesian methods. Many biostatisticians with expertise in frequentist approaches lack the skills to apply Bayesian techniques. To address this gap, four faculty experts in Bayesian modeling at the University of Michigan developed a practical, customized workshop series. The training, tailored to accommodate the schedules of full-time staff, focused on immersive, project-based learning rather than traditional lecture-based methods. Surveys were conducted to assess the impact of the program. All 20 participants completed the program and when surveyed reported an increased understanding of Bayesian theory and greater confidence in using these techniques. Capstone projects demonstrated participants' ability to apply Bayesian methodology. The workshop not only enhanced the participants' skills but also positioned them to readily apply Bayesian techniques in their work. Accommodating the schedules of full-time biostatistical staff enabled full participation. The immersive project-based learning approach resulted in building skills and increasing confidence among staff statisticians who were unfamiliar with Bayesian methods and their practical applications.

This paper examines how a Bayesian decision maker might update her distributions for continuous variablesXi, i=1, 2, …, upon hearing experts' forecasts expressed as quantiles. To utilize the relationship between the decision maker and experts, and to avoid problems associated with different scales and ranges of the variables, we assume that the decision maker transforms the experts' quantiles in terms of her own prior distribution for eachXi.A model using such a transformation is presented and its properties are examined.

Bounded confidence opinion dynamics model the propagation of information in social networks. However, in the existing literature, opinions are only viewed as abstract quantities without semantics rather than as part of a decision-making system. In this work, opinion dynamics are examined when agents are Bayesian decision makers that perform hypothesis testing or signal detection, and the dynamics are applied to prior probabilities of hypotheses. Bounded confidence is defined on prior probabilities through Bayes risk error divergence, the appropriate measure between priors in hypothesis testing. This definition contrasts with the measure used between opinions in standard models: absolute error. It is shown that the rapid convergence of prior probabilities to a small number of limiting values is similar to that seen in the standard Krause-Hegselmann model. The most interesting finding in this work is that the number of these limiting values and the time to convergence changes with the signal-to-noise ratio in the detection task. The number of final values or clusters is maximal at intermediate signal-to-noise ratios, suggesting that the most contentious issues lead to the largest number of factions. It is at these same intermediate signal-to-noise ratios at which the degradation in detection performance of the aggregate vote of the decision makers is greatest in comparison to the Bayes optimal detection performance. Real-world data from the United States Senate is examined in connection with the proposed model.

A Bayesian network model for spatial event surveillance

Many engineering tasks, such as optimization, analysis, model development, model calibration, and others, can potentially exploit information from many sources. These sources include numerical models, expert opinion, and experimental data. Information fusion over these sources of information has the potential to provide a more complete quantitative picture of the current state of knowledge of a given ground truth quantity of interest. This state of knowledge can be updated as new information from any given source is acquired. In this work, we compare two information fusion approaches that both seek to combine all available information to form a surrogate model of the ground truth. These are model reification and cokriging. The comparison considers several test functions as well as a real world NACA 0012 analysis. A quantity of interest is considered for each test case, as well as the derivative of the quantity of interest in some cases. Each fusion approach performs well generally, with each being superior to the other under certain conditions.

In FY2020, the Savannah River National Laboratory (SRNL) and the Sanghani Center for Artificial Intelligence and Data Analytics at Virginia Polytechnic Institute and State University entered a collaboration funded by Department of Energy’s (DOE) Office of Defense Nuclear Nonproliferation Research and Development. The project’s mission was to take the first steps toward developing a demonstration prototype system that uses multiple machine learning and data analytics methods on largescale open data sources to identify new, developing, and/or undeclared nuclear programs. Given the SRNL team’s on-site perspective of events culminating in the DOE’s decision to pursue the Savannah River Plutonium Processing Facility (SRPPF), the team targeted the identification of events and indicators in retrospective datasets that pointed to the activity of “fissile core fabrication at the Savannah River Site” prior to the official announcement in May of 2018. A preliminary modeling pipeline was developed in FY20 that showed the datasets contained adequate signal for continuation of efforts. In FY21, a modular demonstration prototype modeling pipeline has continued in development for two text-based data sources: a broad internet archive (Webhose Ltd.) and a decahose Twitter database (i.e., a global sampling of one in every ten Tweets). The techniques that have been developed rely on graph theory and anomaly detection to identify contextual shifts in key words and phrases at various points in time such that indicators of events of interest could be identified and subsequently, events could be extracted from the corpuses. The foundational concept behind the approaches is that contextual shifts in key words and phrases can act as indicators of events of interest. Both datasets have proven successful in extracting events of interest related to pit production at the Savannah River Site prior to the official announcement. In addition, the pipelines have generated a wide range of events broadly summarized as: the awarding of DOE contracts at major sites, DOE investments in various programs, accidents at DOE national laboratories, speculations about the fate of pit production in the DOE complex, domestic and international shipments and receipts of nuclear materials at DOE sites, termination of non-proliferation agreements with Russia, termination of MOX, new weapons development approvals/testing, nuclear posture reviews, major DOE cleanup/production milestones, political opinions, and nuclear watch groups’ opinions, among many others.

Effective human–robot teaming (HRT) increasingly requires humans to work with intelligent, autonomous machines. However, novel features of intelligent autonomous systems such as social agency and incomprehensibility may influence the human’s trust in the machine. The human operator’s mental model for machine functioning is critical for trust. People may consider an intelligent machine partner as either an advanced tool or as a human-like teammate. This article reports a study that explored the role of individual differences in the mental model in a simulated environment. Multiple dispositional factors that may influence the dominant mental model were assessed. These included the Robot Threat Assessment (RoTA), which measures the person’s propensity to apply tool and teammate models in security contexts. Participants (N = 118) were paired with an intelligent robot tasked with making threat assessments in an urban setting. A transparency manipulation was used to influence the dominant mental model. For half of the participants, threat assessment was described as physics-based (e.g., weapons sensed by sensors); the remainder received transparency information that described psychological cues (e.g., facial expression). We expected that the physics-based transparency messages would guide the participant toward treating the robot as an advanced machine (advanced tool mental model activation), while psychological messaging would encourage perceptions of the robot as acting like a human partner (teammate mental model). We also manipulated situational danger cues present in the simulated environment. Participants rated their trust in the robot’s decision as well as threat and anxiety, for each of 24 urban scenes. They also completed the RoTA and additional individual-difference measures. Findings showed that trust assessments reflected the degree of congruence between the robot’s decision and situational danger cues, consistent with participants acting as Bayesian decision makers. Several scales, including the RoTA, were more predictive of trust when the robot was making psychology-based decisions, implying that trust reflected individual differences in the mental model of the robot as a teammate. These findings suggest scope for designing training that uncovers and mitigates the individual’s biases toward intelligent machines.

We study a problem where a new, unfamiliar group of agents has to decide how a joint reward should be shared among them. We focus on settings where the share that each agent receives depends on the evaluations of its peers concerning that agent's contribution to the group. We introduce a mechanism to elicit and aggregate evaluations as well as for determining agents' shares. The intuition behind the proposed mechanism is that each agent has its expected share maximized to the extent that it is well evaluated by its peers and that it is truthfully reporting its evaluations. For promoting truthfulness, the proposed mechanism uses a peer-prediction method built on strictly proper scoring rules. Under the assumption that agents are Bayesian decision makers, we show that our mechanism is incentive compatible and budget balanced. We also provide sufficient conditions under which the proposed mechanism is individually rational, resistant to some kinds of collusion, and fair.

The influence of cognitive science and psychology on decision theory is bringing about changes to assumptions about decision making, and, as a consequence, the way that decisions should be modeled and supported. The three articles published in this Special Issue on intelligent decision support and modeling reflect an emerging literature that incorporates psychology into decision modeling and decision support. This literature represents only a starting point, and much remains to be done in terms of acknowledging the influence of psychology on engineering decisions. Decision support tools will probably never completely make up for engineers’ lack of the cognitive capacity needed to make the multitude of decisions associated with engineering design in a fully informed, unbiased way. Incorporating rational choice theory and psychology into decision support tools seems to be a fruitful path toward promoting optimal decisions in engineering. The last two to three decades have brought about important changes to the field of decision theory, particularly in response to the generally accepted principle that the act of cognitive representation and framing of decisions should follow the axioms of expected utility theory. The standard theory for decision making is based on subjective expected utility theory (e.g., Savage, 1954; Schmeidler, 1989), which involves the enumeration of possibilities, an analysis of the possible outcomes, and the selection of the utility-maximizing decision (Gilboa & Schmeidler, 2001). In engineering design, decision theory is generally applied as a systematic procedure for selecting design variables when there is uncertainty over the preferences associated with the objectives (Thurston, 1991, 2001). In schools of engineering, operations research, computer science, and business around the world, students continue to be taught a set of methodologies consonant with rational choice theory (Wood, 2004), which is founded upon the analysis of information as the basis of decision. In short, decision support and modeling in engineering design generally follows a framework of selecting design variables that optimize the expected utility of the design, and, in so doing, casts engineering design within a rational process (Hazelrigg, 1998). The perceived increased accuracy and rigor of the decisions and the decision support systems built upon the premise of utility maximization can mask the realities of eliciting preferences from engineers, which may be subject to psychological biases. Kahneman and Tversky (Tversky & Kahneman, 1974; Kahneman & Tversky, 1979) drew attention to the realities of human decision making in describing the heuristics that human beings employ in decision making under uncertainty, which are subject to psychological biases that can lead to systematic and predictable errors. In recent years, Kahneman and colleagues have published a series of articles dealing with strategies to correct for these psychological biases (e.g., Kahneman & Lovallo, 1993; Kahneman et al., 2011) and entire fields of behavioral finance, behavioral economics, and behavioral strategy have grown up around the application of cognitive science and psychology to the theory and practice of decision making under uncertainty in specific contexts. This influence is now being felt in the engineering design domain in the modeling and support of engineering decisions, which is built upon a rich heritage of studies on the cognitive and behavioral strategies of engineers. The three articles published in this Special Issue reflect a shift away from normative subjective utility maximization as the only model for decision making and decision support toward greater consideration to the exigencies of engineering decision making in practice. All three articles contribute to decision support and modeling and treat these as integrated issues rather than as compartmentalized problems. The article “Bayesian Project Diagnosis for the Construction Design Process” by Matthews and Philip is perhaps the most “traditional” of the three articles. The article deals with the problem of forecasting potential problems in construction processes. We use traditional in the sense that Bayesian modeling is an accepted method for estimating the probability of an event or outcome of interest (Marshall & Oliver, 1995). They model the construction process as a Markov chain, with transition probabilities associated with progressing through various Reprint requests to: Andy Dong, Faculty of Engineering and Information Technologies, University of Sydney, Engineering Building (J05), Sydney 2006, Australia. E-mail: andy.dong@sydney.edu.au Artificial Intelligence for Engineering Design, Analysis and Manufacturing (2012), 26, 371–373. # Cambridge University Press 2012 0890-0604/12 $25.00 doi:10.1017/S0890060412000248

Diagnostic logs represent the main source of in-formation about the system runtime. However, the presence of faults typically leads to multiple errors propagating within system components, which requires analysts to dig into cascading messages for root cause analysis. This is exacerbated in complex systems, such as railway systems, composed by several devices generating high amount of logs. Filtering allows dealing with large data volumes, leading practitioners to focus on interesting events, i.e., events that should be further investigated by analysts. This paper proposes an unsupervised approach to discover filtering rules from diagnostic logs. The approach automatically infers potential events correlations, representing them as fault-trees enriched with scores. Trees define filtering rules highlighting the interesting events, while scores allow prioritizing their anal-ysis. The approach has been applied in a preliminary railway case study, which encompasses more than 710k events generated by on-board train equipment during operation.

In any text Events represent a rich source of information, such as proper names that specify the people, organizations, locations, etc., that participate in the event, dates that specify the time of the event, noun phrases that describe the event, and keywords that indicate that an interesting event has occurred. In this paper our focus is on those elements in terms of how we use natural language processing techniques to capture, analyze and identify the role each element plays in the event and understand how they are linked to form events as well as how they form the relationships between related events.