Input uncertainty and indifference-zone ranking & selection

Joint analysis of input and parametric uncertainties in watershed water quality modeling: A formal Bayesian approach

Integrated Bayesian Multi-model approach to quantify input, parameter and conceptual model structure uncertainty in groundwater modeling

Simulation-based Optimization (SbO) assumes that the simulation model is valid, and that the probability distributions used therein are accurate. However, in practice, the input probability distributions (input models) are estimated by sampling data from the real system. The errors in such estimates can have a profound impact on the optimal solution obtained by SbO. The existing two-stage framework for SbO under computational budget constraint considers only the stochastic uncertainty. In our variant, we consider the input model parameter uncertainty as well. Our algorithmic procedure is based on the stochastic kriging metamodel-assisted bootstrapping with an efficient global optimization technique which sequentially searches the optimum and incorporates Optimal Computational Budget Allocation (OCBA). This framework is also used for determining tighter worst-case bounds of the SbO with input uncertainty. The proposed framework is illustrated with the M/M/1 queuing model.

An input model is a collection of distributions together with any associated parameters that are used as primitive inputs in a simulation model. Input model uncertainty arises when one is not completely certain what distributions and/or parameters to use. This tutorial attempts to provide a sense of why one should consider input uncertainty and what methods can be used to deal with it.

Input refers the effect of driving a simulation with input distributions that are based on real-world data. At WSC 2012, Ankenman and Nelson presented a quick-and-easy experiment to assess the overall effect of input uncertainty on simulation output. When their method reveals that input uncertainty is substantial, then the natural follow-up questions are which input distributions contribute the most to input uncertainty, and from which input processes would it be most beneficial to collect more data? To answer these questions Ankenman and Nelson proposed a sequence of additional experiments that are in no sense quick. In this paper we provide a follow-up analysis that requires no additional simulation experiments beyond the overall assessment, and yet provides more information than Ankenman and Nelson. Numerical illustrations are provided.

Regression and Kriging metamodels with their experimental designs in simulation: A review

Multi-criteria evaluation of wastewater treatment plant control strategies under uncertainty

10525 Background: To evaluate the incremental short- and long-term benefits and harms of primary prevention with risk reducing medication in high-risk women receiving screening mammography. Methods: We adapted an established, validated Cancer Intervention and Surveillance Modeling Network (CISNET) breast cancer discrete event microsimulation model developed to synthesize data the impact of using risk-reducing medication and annual mammography among women with a 3% or higher five-year risk of developing breast cancer. We also examined the effects of supplemental MRI. The model follows a simulated cohort of millions of US women from birth to death. We used large observational and clinical trial data to derive input parameters for cohort-specific birth rates, breast cancer risk, incidence and stage, screening performance, survival by age, stage, and subtype, treatment efficacy, and other cause mortality. Breast cancer risk was modeled based on family history, breast density, age and history of past breast biopsy. We compared two strategies, annual 3D mammography alone vs. annual 3D mammography and a 5-year course of risk reducing medication at various starting ages, and adding MRI to each approach. Outcomes included the benefits of risk-reducing drugs (avoiding breast cancer) and screening (stage, breast cancer death). Harms included drug side effects and screening false positives and overdiagnosis. Sensitivity analysis tested the impact of uncertainty in model inputs and assumptions on results. Results: Compared to mammography alone, adding risk reducing medication could decrease invasive breast cancer incidence by 30%, and breast cancer deaths by 30% (Table). However, due to reduction in breast cancer incidence, risk reducing medication could result in a 3% increase in false positive results; adding MRI increases benefits but also increases false-positive results. Benefits and harms of risk reducing medication and breast cancer screening strategies for women at high-risk of developing breast cancer. Conclusions: Risk-reducing mediation reduces the risk of hormone-receptor positive breast cancer, and combining this with mammography (and/or MRI) improves earlier detection. Additional work is ongoing to incorporate adverse effects of therapy. Simulation modeling can be used to provide individualized data to facilitate discussions about breast cancer prevention and early detection among high-risk women seen in clinical practice.[Table: see text]

Specifying a proper input distribution is often a challenging task in simulation modeling. In practice, there may be multiple plausible distributions that can fit the input data reasonably well, especially when the data volume is not large. In this paper, we consider the problem of selecting the best from a finite set of simulated alternatives, in the presence of such input uncertainty. We model such uncertainty by an ambiguity set consisting of a finite number of plausible input distributions and aim to select the alternative with the best worst-case mean performance over the ambiguity set. We refer to this problem as robust selection of the best (RSB). To solve the RSB problem, we develop a two-stage selection procedure and a sequential selection procedure; we then prove that both procedures can achieve at least a user-specified probability of correct selection under mild conditions. Extensive numerical experiments are conducted to investigate the computational efficiency of the two procedures. Finally, we apply the RSB approach to study a queueing system’s staffing problem using synthetic data and an appointment-scheduling problem using real data from a large hospital in China. We find that the RSB approach can generate decisions significantly better than other widely used approaches. This paper was accepted by Noah Gans, stochastic models and simulation.

In this study, we consider ranking and selection problems where the simulation model is subject to input uncertainty. Under the input uncertainty, we compare system designs based on their worst-case performance, and seek to maximize the probability of selecting the design with the best performance under the worst-case scenario. By approximating the probability of correct selection (PCS), we develop an asymptotically (as the simulation budget goes to infinity) optimal solution of the resulting problem. An efficient selection procedure is designed within the optimal computing budget allocation (OCBA) framework. Numerical tests show the high efficiency of the proposed method.

Simulation is often used to study stochastic systems. A key step of this approach is to specify a distribution for the random input. This is called input modeling, which is important and even critical for simulation study. However, specifying a distribution precisely is usually difficult and even impossible in practice. This issue is called input uncertainty in simulation study. In this paper we study input uncertainty when using simulation to estimate important performance measures: expectation, probability, and value-at-risk. We propose a robust simulation (RS) approach, which assumes the real distribution is contained in a certain ambiguity set constructed using statistical divergences, and simulates the maximum and the minimum of the performance measures when the distribution varies in the ambiguity set. We show that the RS approach is computationally tractable and the corresponding results can disclose important information about the systems, which may help decision makers better understand the systems.

Sensitivity and Uncertainty

In many practical problems, simulation models are used to support complex decision-making processes comparing hundreds or thousands of solutions. These problems typically have a key objective but the final decision may be dependent on other factors, which cannot be incorporated into the simulation model. In such cases, decision-makers may request a short list of ‘good’ solutions, which work well for the main objective and satisfy one or more chance constraints. While fully sequential ranking and selection procedures can be effective at solving these problems, surveys of experimentation practice suggest that they are under-utilized, potentially due to difficulties automating commercial software. We develop an approach with just two stages of replications. The approach, which has been designed to cope with the use of common random numbers, draws on ideas from indifference zones and makes use of bootstrapping to find a subset of high quality solutions. A Python implementation is freely available.

When we use simulation to estimate the performance of a stochastic system, lack of fidelity in the random input models can lead to poor system performance estimates. Since the components of many complex systems could be dependent, we want to build input models that faithfully capture such key properties. In this paper, we use the flexible NORmal To Anything (NORTA) representation for dependent inputs. However, to use the NORTA representation we need to estimate the marginal distribution parameters and a correlation matrix from real-world data, introducing input uncertainty. To quantify this uncertainty, we employ the bootstrap to capture the parameter estimation error and an equation-based stochastic kriging metamodel to propagate the input uncertainty to the output mean. Asymptotic analysis provides theoretical support for our approach, while an empirical study demonstrates that it has good finite-sample performance.

A review of uncertainty quantification techniques is provided for a variety of situations involving uncertainties in model inputs (independent variables). The situations of interest are divided into three categories: (i) when model prediction uncertainties are quantified based on uncertainties in uncertain inputs, (ii) when parameter estimate uncertainties are calculated by propagation of uncertainties from measured inputs and outputs, and (iii) when model prediction uncertainties are quantified based on corresponding uncertainties in measured inputs and uncertain parameter estimates. For all three situations, linearization‐based and Monte Carlo‐based techniques are reviewed and details for their corresponding algorithms are presented. Recommendations are provided on which uncertainty quantification techniques are best for different types of chemical engineering models based on the amount of input uncertainty and nonlinearity over the range of plausible input and parameter values.