Optimal computing budget allocation via sampling based unbiased approximation

In this study, we consider ranking and selection problems where the simulation model is subject to input uncertainty. Under the input uncertainty, we compare system designs based on their worst-case performance, and seek to maximize the probability of selecting the design with the best performance under the worst-case scenario. By approximating the probability of correct selection (PCS), we develop an asymptotically (as the simulation budget goes to infinity) optimal solution of the resulting problem. An efficient selection procedure is designed within the optimal computing budget allocation (OCBA) framework. Numerical tests show the high efficiency of the proposed method.

Convergence rate analysis for optimal computing budget allocation algorithms

Subset selection of best simulated systems

Optimal Computing Budget Allocation for regression with gradient information

In recent years, with the rapid development of computing technology, developing parallel procedures to solve large-scale ranking and selection (R&S) problems has attracted a lot of research attention. In this paper, we take fixed-budget R&S procedure as an example to investigate potential issues of developing parallel procedures. We argue that to measure the performance of a fixed-budget R&S procedure in solving large-scale problems, it is important to quantify the minimal growth rate of the total sampling budget such that as the number of alternatives increases, the probability of correct selection (PCS) would not decrease to zero. We call such a growth rate of the total sampling budget the rate for maintaining correct selection (RMCS). We show that a tight lower bound for the RMCS of a broad class of existing fixed-budget procedures is in the order of [Formula: see text], where k is the number of alternatives. Then, we propose a new type of fixed-budget procedure, namely the fixed-budget knockout-tournament ([Formula: see text]) procedure. We prove that, in terms of the RMCS, our procedure outperforms existing fixed-budget procedures and achieves the optimal order, that is, the order of k. Moreover, we demonstrate that our procedure can be easily implemented in parallel computing environments with almost no nonparallelizable calculations. Last, a comprehensive numerical study shows that our procedure is indeed suitable for solving large-scale problems in parallel computing environments. History: Accepted by Bruno Tuffin, Area Editor for Simulation. Funding: Y. Zhong was supported by the National Natural Science Foundation of China [Grant 72101047]. L. J. Hong was supported by the National Natural Science Foundation of China [Grants 72091211 and 72161160340]. G. Jiang was supported by the National Natural Science Foundation of China [Grants 72121001 and 72171060]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijoc.2022.1221 .

Selecting a subset of the best solutions among large-scale problems is an important area of research. When the alternative solutions are stochastic in nature, then it puts more burden on the problem. The objective of this paper is to select a set that is likely to contain the actual best solutions with high probability. If the selected set contains all the best solutions, then the selection is denoted as correct selection. We are interested in maximizing the probability of this selection; P(CS). In many cases, the available computation budget for simulating the solution set in order to maximize P(CS) is limited. Therefore, instead of distributing these computational efforts equally likely among the alternatives, the optimal computing budget allocation (OCBA) procedure came to put more effort on the solutions that have more impact on the selected set. In this paper, we derive formulas of how to distribute the available budget asymptotically to find the approximation of P(CS). We then present a procedure that uses OCBA with the ordinal optimization (OO) in order to select the set of best solutions. The properties and performance of the proposed procedure are illustrated through a numerical example. Overall results indicate that the procedure is able to select a subset of the best systems with high probability of correct selection using small number of simulation samples under different parameter settings.

We consider a ranking and selection problem with independent normal observations, and we analyze the asymptotic sampling rates of expected improvement (EI) methods in this setting. Such methods often perform well in practice, but a tractable analysis of their convergence rates is difficult because of the nonlinearity and nonconvexity of the EI calculations. We present new results indicating that, for known sampling noise, variants of EI produce asymptotic simulation allocations that are essentially identical to those chosen by the optimal computing budget allocation (OCBA) methodology, which is known to yield near-optimal asymptotic performance in ranking and selection. This is the first general equivalence result between EI and OCBA, and it provides insight into the good practical performance of EI. We also derive the limiting allocation for EI under unknown sampling variance.

The dynamics of many systems nowadays follow not only physical laws but also man-made rules. These systems are known as discrete event dynamic systems and their performances can be accurately evaluated only through simulations. Existing studies on simulation-based optimization (SBO) usually assume deterministic simulation time for each replication. However, in many applications such as evacuation, smoke detection, and territory exploration, the simulation time is stochastic due to the randomness in the system behavior. We consider the computing budget allocation for SBO's with stochastic simulation time in this paper, which has not been addressed in existing literatures to the author's best knowledge. We make the following major contribution. The relationship between simulation time and performance estimation accuracy is quantified. It is shown that when the asymptotic performance is of interest only the mean value of individual simulation time matters. Then based on the existing optimal computing budget allocation (OCBA) method for deterministic simulation time we develop OCBA for stochastic simulation time (OCBAS), and show that OCBAS is asymptotically optimal. Numerical experiments are used to discuss the impact of the variance of simulation time, the impact of correlated simulation time and performance estimation, and to demonstrate the performance of OCBAS on a smoke detection problem in wireless sensor network. The numerical results also show that OCBA for deterministic simulation time is robust even when the simulation time is stochastic.

This chapter reviews the problem of selecting the best of a finite set of alternatives, where best is defined with respect to the highest mean performance, and where the performance is uncertain but may be estimated with simulation. This problem has been explored from several perspectives, including statistical ranking and selection, multiple comparisons, and stochastic optimization. Approaches taken in the literature include frequentist statistics, Bayesian statistics, related heuristics, and asymptotic convergence in probability. This chapter presents algorithms that are derived from Bayesian and related conceptual frameworks to provide empirically effective performance for the ranking and selection problem. In particular, we motivate the optimal computing budget allocation (OCBA) algorithm and expected value of information (EVI) approaches, give example algorithms, and provide pointers to the literature for detailed derivations and extensions of these approaches.KeywordsAllocation RuleCorrect SelectionMultiple AlternativeLinear LossEqual AllocationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we introduce a sequential procedure for the Multi-Objective Ranking and Selection (MOR&S) problems that identifies the Pareto front with a guaranteed probability of correct selection (PCS). In particular, the proposed procedure is fully sequential using the test statistics built upon the generalized sequential probability ratio test (GSPRT). The main features of the new proposed procedure are: 1) a unified framework, the new procedure treats the multi-objective problems in the same way as the single-objective problems; 2) an indifference-zone-free formulation, the new procedure eliminates the necessity of indifference-zone parameter; 3) asymptotically optimality, the GSPRT achieves asymptotically the shortest expected sample size among all sequential tests; 4) general distribution, the procedure uses the empirical likelihood for generally distributed observation. A numerical evaluation demonstrates the efficiency of the new procedure.

Ordinal optimization is an efficient technique to choose and rank various engineering designs that require time-consuming discrete-event simulations. Optimal computing budget allocation (OCBA) has been an important tool to enhance its efficiency such that the best design is selected in a timely fashion. It, however, fails to address the issue of selecting the best and worst designs efficiently. The need to select both rapidly given a fixed computing budget has arisen from many applications. This work develops a new OCBA-based approach for selecting both best and worst designs at the same time. Its theoretical foundation is laid. Our numerical results show that it can well outperform all the existing methods in terms of probability of correct selection and computational efficiency.

Existing research in ranking and selection has focused on the problem of selecting the best design, subset selection and selecting the set of Pareto designs. Few works have addressed the problem of complete ranking. In this research, we consider the problem of ranking all alternatives completely with consideration of input uncertainty. Given a fixed simulation budget, we aim to maximize the probability of correct ranking among all designs based on their worst-case performances. The problem is formulated as an optimal computing budget allocation model. To make this optimization problem computationally tractable, we develop an approximated probability of correct ranking and derive the asymptotic optimality condition based on it. A sequential ranking procedure is then suggested to implement the proposed simulation budget allocation rule. The high efficiency of the proposed simulation procedure is demonstrated via a set of numerical experiments. In addition, useful insights and analysis on characterizing the optimality condition and implementing the efficient budget allocation rule are provided.

We propose to improve the efficiency of simulation optimization by integrating the notion of optimal computing budget allocation into the Cross-Entropy (CE) method, which is a global optimization search approach that iteratively updates a parameterized distribution from which candidate solutions are generated. This article focuses on continuous optimization problems. In the stochastic simulation setting where replications are expensive but noise in the objective function estimate could mislead the search process, the allocation of simulation replications can make a significant difference in the performance of such global optimization search algorithms. A new allocation scheme is developed based on the notion of optimal computing budget allocation. The proposed approach improves the updating of the sampling distribution by carrying out this computing budget allocation in an efficient manner, by minimizing the expected mean-squared error of the CE weight function. Numerical experiments indicate that the computational efficiency of the CE method can be substantially improved if the ideas of computing budget allocation are applied.

Particle Swarm Optimization (PSO) is a population-based algorithm designed to find good solutions to optimization problems. Its characteristics have encouraged its adoption to tackle a variety of problems in different fields. However, when such problems are subject to noise, the performance of PSO suffers an immediate deterioration which demands the incorporation of noise handling mechanisms. One such mechanism comprises resampling methods, which re-evaluate the solutions multiple times in order to estimate their true objective values. The state-of-the-art integration with which the best results have been obtained utilizes the resampling method named Optimal Computing Budget Allocation (OCBA). This resampling method starts by estimating the objective values of all the solutions via Equal Resampling (ER), and then sequentially allocating further re-evaluations to the estimated best solutions. However, after having a first estimate via ER, we question the importance of the additional efforts to correctly select the true best solutions when a good-enough and accurate one can be selected. In this paper, we propose a new PSO algorithm based on ER in which the additional evaluations are allocated at once to the estimated best solutions, thus skipping the complexity of using OCBA. Experiments on 20 large-scale benchmark functions subject to different levels of noise show that the proposed algorithm produces similar results to PSO with OCBA in most cases.

The limitations of traditional indifference-zone (IZ) ranking and selection procedures stem from their reliance on an IZ parameter, which is often unknown in practice and can lead to inefficiencies. Additionally, since exact evaluation of the probability of correct selection (PCS) is generally unattainable, these approaches commonly rely on a Bonferroni-Inequality-type lower bound to ensure the worst-case PCS, which introduces further conservativeness. To overcome these challenges, we propose fully sequential IZ-free procedures that eliminate the need for an IZ parameter and reduce the conservativeness associated with Bonferroni Inequality approaches. We demonstrate that, in the context of IZ-free procedures, the screening statistic corresponds to the Euclidean distance of a multi-dimensional standard Brownian motion, which behaves as a Bessel process. This connection simplifies the derivation of statistical guarantees for the proposed procedures and facilitates the analytical expressions for key parameters. Our procedures accommodate both known equal variances and unknown unequal variances, with statistical guarantees established in the known equal variance case. Numerical experiments confirm that the proposed procedures outperform existing methods in reducing PCS conservativeness and improving sample size efficiency. These advancements enhance the robustness and practicality of IZ-free procedures for diverse applications.