Ranking and selection under input uncertainty: A budget allocation formulation

Uncertainties in crop, soil and weather inputs used in growth models: Implications for simulated outputs and their applications

Input refers the effect of driving a simulation with input distributions that are based on real-world data. At WSC 2012, Ankenman and Nelson presented a quick-and-easy experiment to assess the overall effect of input uncertainty on simulation output. When their method reveals that input uncertainty is substantial, then the natural follow-up questions are which input distributions contribute the most to input uncertainty, and from which input processes would it be most beneficial to collect more data? To answer these questions Ankenman and Nelson proposed a sequence of additional experiments that are in no sense quick. In this paper we provide a follow-up analysis that requires no additional simulation experiments beyond the overall assessment, and yet provides more information than Ankenman and Nelson. Numerical illustrations are provided.

Existing research in ranking and selection has focused on the problem of selecting the best design, subset selection and selecting the set of Pareto designs. Few works have addressed the problem of complete ranking. In this research, we consider the problem of ranking all alternatives completely with consideration of input uncertainty. Given a fixed simulation budget, we aim to maximize the probability of correct ranking among all designs based on their worst-case performances. The problem is formulated as an optimal computing budget allocation model. To make this optimization problem computationally tractable, we develop an approximated probability of correct ranking and derive the asymptotic optimality condition based on it. A sequential ranking procedure is then suggested to implement the proposed simulation budget allocation rule. The high efficiency of the proposed simulation procedure is demonstrated via a set of numerical experiments. In addition, useful insights and analysis on characterizing the optimality condition and implementing the efficient budget allocation rule are provided.

In this study, we consider ranking and selection problems where the simulation model is subject to input uncertainty. Under the input uncertainty, we compare system designs based on their worst-case performance, and seek to maximize the probability of selecting the design with the best performance under the worst-case scenario. By approximating the probability of correct selection (PCS), we develop an asymptotically (as the simulation budget goes to infinity) optimal solution of the resulting problem. An efficient selection procedure is designed within the optimal computing budget allocation (OCBA) framework. Numerical tests show the high efficiency of the proposed method.

Joint analysis of input and parametric uncertainties in watershed water quality modeling: A formal Bayesian approach

Specifying a proper input distribution is often a challenging task in simulation modeling. In practice, there may be multiple plausible distributions that can fit the input data reasonably well, especially when the data volume is not large. In this paper, we consider the problem of selecting the best from a finite set of simulated alternatives, in the presence of such input uncertainty. We model such uncertainty by an ambiguity set consisting of a finite number of plausible input distributions and aim to select the alternative with the best worst-case mean performance over the ambiguity set. We refer to this problem as robust selection of the best (RSB). To solve the RSB problem, we develop a two-stage selection procedure and a sequential selection procedure; we then prove that both procedures can achieve at least a user-specified probability of correct selection under mild conditions. Extensive numerical experiments are conducted to investigate the computational efficiency of the two procedures. Finally, we apply the RSB approach to study a queueing system’s staffing problem using synthetic data and an appointment-scheduling problem using real data from a large hospital in China. We find that the RSB approach can generate decisions significantly better than other widely used approaches. This paper was accepted by Noah Gans, stochastic models and simulation.

Simulation output clearly depends on the form of the input distributions used to drive the model. Often these input distributions are fitted using finite samples of real-world data. The finiteness of the samples introduces errors in the input distributions, affecting the output. Yet this propagation of input model uncertainty to output uncertainty is rarely considered in simulation output analysis. This tutorial presents a discussion of input uncertainty issues and recently developed methodological approaches, set in the context of input uncertainty methods proposed over the past twenty years.

Efficient subset selection for the expected opportunity cost

Simulation optimisation offers great opportunities in the design and optimisation of complex systems. In the presence of multiple objectives, there is usually no single solution that performs best on all objectives. Instead, there are several Pareto-optimal (efficient) solutions with different trade-offs which cannot be improved in any objective without sacrificing performance in another objective. For the case where alternatives are evaluated on multiple stochastic criteria, and the performance of an alternative can only be estimated via simulation, we consider the problem of efficiently identifying the Pareto-optimal designs out of a (small) given set of alternatives. We present a simple myopic budget allocation algorithm for multi-objective problems and propose several variants for different settings. In particular, this myopic method only allocates one simulation sample to one alternative in each iteration. This paper shows how the algorithm works in bi-objective problems under different settings. Empirical tests show that our algorithm can significantly reduce the necessary simulation budget.

We consider the problem of ranking the topmdesigns out ofkalternatives. Using the optimal computing budget allocation framework, we formulate this problem as that of maximizing the probability of correctly ranking the topmdesigns subject to the constraint of a fixed limited simulation budget. We derive the convergence rate of the false ranking probability based on the large deviation theory. The asymptotically optimal allocation rule is obtained by maximizing this convergence rate function. To implement the simulation budget allocation rule, we suggest a heuristic sequential algorithm. Numerical experiments are conducted to compare the effectiveness of the proposed simulation budget allocation rule. The numerical results indicate that the proposed asymptotically optimal allocation rule performs the best comparing with other allocation rules.

ABSTRACTCommercial plant breeders improve economically important traits by selectively mating individuals from a given breeding population. Potential pairings are evaluated before the growing season using Monte Carlo simulation, and a mating design is created to allocate a fixed breeding budget across the parent pairs to achieve desired population outcomes. We introduce a novel objective function for this mating design problem that accurately models the goals of a certain class of breeding experiments. The resulting mating design problem is a computationally burdensome simulation optimization problem on a combinatorially large set of feasible points. We propose a two-step solution to this problem: (i) simulate to estimate the performance of each parent pair and (ii) solve an estimated version of the mating design problem, which is an integer program, using the simulation output. To reduce the computational burden when implementing steps (i) and (ii), we analytically identify a Pareto set of parent pairs that will receive the entire breeding budget at optimality. Since we wish to estimate the Pareto set in step (i) as input to step (ii), we derive an asymptotically optimal simulation budget allocation to estimate the Pareto set that, in our numerical experiments, out-performs Multi-objective Optimal Computing Budget Allocation in reducing misclassifications. Given the estimated Pareto set, we provide a branch-and-bound algorithm to solve the estimated mating design problem. Our approach dramatically reduces the computational effort required to solve the mating design problem when compared with naïve methods.

Sensitivity and Uncertainty

<strong class="journal-contentHeaderColor">Abstract.</strong> Long-range weather forecasts provide predictions of atmospheric, ocean and land surface conditions that can potentially be used in land surface and hydrological models to predict the water and energy status of the land surface or in crop growth models to predict yield for water resources or agricultural planning. However, the coarse spatial and temporal resolutions of available forecast products have hindered their widespread use in such modelling applications that usually require high resolution input data. In this study, we applied sub-seasonal (up to 4 months) and seasonal (7 months) weather forecasts from the latest European Centre for Medium-Range Weather Forecasts (ECMWF) seasonal forecasting system (SEAS5) in a land surface modelling approach using the Community Land Model version 5.0 (CLM5). Simulations were conducted for 2017&ndash;2020 forced with sub-seasonal and seasonal weather forecasts over two different domains with contrasting climate and cropping conditions: the German state of North Rhine-Westphalia and the Australian state of Victoria. We found that, after pre-processing of the forecast products (temporal downscaling of precipitation and incoming shortwave radiation), the simulations forced with seasonal and sub-seasonal forecasts were able to generate a model system response very close to reference simulation results forced by reanalysis data. Differences between seasonal and sub-seasonal experiments were insignificant. The forecast experiments were able to satisfactorily capture recorded inter-annual variations of crop yield. In addition, they also reproduced the generally higher inter-annual variability in crop yield across the Australian domain (approximately 50 % inter-annual variability in recorded yields and up to 17 % in simulated yields) compared to the German domain (approximately 15 % inter-annual variability in recorded yields and up to 5 % in simulated yields). The high and low yield seasons (2020 and 2018) among the four simulated years were clearly reproduced in forecast simulation results. Furthermore, sub-seasonal and seasonal simulations reflected the early harvest in the drought year of 2018 in the German domain. However, the simulated inter-annual yield variability was lower in all simulations compared to the official statistics. While general soil moisture trends, such as the European drought in 2018, were captured by the seasonal experiments, we found systematic over- and underestimations in both the forecast and the reference simulations compared to the Soil Moisture Active Passive Level-3 soil moisture product (SMAP L3) and the Soil Moisture Climate Change Initiative Combined dataset from the European Space Agency's (ESA CCI). These observed biases of soil moisture as well as the low inter-annual variability of simulated crop yield indicate the need to improve the representation of these variables in CLM5 to increase the model sensitivity to drought stress and other crop stressors.

In this paper, we introduce the semi-stochastic model for dealing with input uncertainty in optimization problems. This model is a hybrid between the overly pessimistic online model and the highly optimistic stochastic (or Bayesian) model. In this model, the algorithm can obtain only limited stochastic information about the future (i.e. about the input distribution)---as the amount of stochastic information we make available to the algorithm grows from no information to full information, we interpolate between the online and stochastic models. The central question in this framework is the trade-off between the performance of an algorithm, and the stochastic information that it can access. As a first step towards understanding this trade-off, we consider the ski-rental problem in the semi-stochastic setting. More precisely, given a desired competitive ratio, we give upper and lower bounds on the amount of stochastic information required by a deterministic algorithm for the ski-rental problem to achieve that competitive ratio.

Several uncertainty quantification and sensitivity analysis methods are used to determine the most sensitive geometric and material input parameters of a cantilevered pipeline conveying fluid when uncertainty is introduced to the system at the onset of instability. The full nonlinear equations of motion are modeled using the extended Hamilton’s principle and then discretized using Galerkin’s method. A parametric study is first performed, and the Morris elementary effects are calculated to obtain a preliminary understanding of how the onset speed changes when each parameter is introduced to a ± 5% uncertainty. Then, four different input uncertainty distributions, mainly, uniform and Gaussian distribution, are chosen to investigate how input distributions affect uncertainty in the output. A convergence analysis is used to determine the number of samples needed to maintain simulation accuracy while saving the most computational time. Then, Monte Carlo simulations are run, and the output distributions for each input distribution at ± 1%, ± 3% and ± 5% input uncertainty range are found and discussed. Additionally, the Pearson correlation coefficients are evaluated for different uncertainty ranges. A final Monte Carlo study is performed in which single parameters are held constant while all others still have uncertainty. Overall, the flow speed at the onset of instability is the most sensitive to changes in the outer diameter of the pipe.