Subset selection of best simulated systems

Convergence rate analysis for optimal computing budget allocation algorithms

Optimal Computing Budget Allocation for regression with gradient information

Selecting a subset of the best solutions among large-scale problems is an important area of research. When the alternative solutions are stochastic in nature, then it puts more burden on the problem. The objective of this paper is to select a set that is likely to contain the actual best solutions with high probability. If the selected set contains all the best solutions, then the selection is denoted as correct selection. We are interested in maximizing the probability of this selection; P(CS). In many cases, the available computation budget for simulating the solution set in order to maximize P(CS) is limited. Therefore, instead of distributing these computational efforts equally likely among the alternatives, the optimal computing budget allocation (OCBA) procedure came to put more effort on the solutions that have more impact on the selected set. In this paper, we derive formulas of how to distribute the available budget asymptotically to find the approximation of P(CS). We then present a procedure that uses OCBA with the ordinal optimization (OO) in order to select the set of best solutions. The properties and performance of the proposed procedure are illustrated through a numerical example. Overall results indicate that the procedure is able to select a subset of the best systems with high probability of correct selection using small number of simulation samples under different parameter settings.

In a Ranking and Selection problem, the objective of allocation is vital in deriving the rule. However, most of these objectives do not have a closed form. Due to the high cost of a direct approximation, several cheap but biased substitutes were applied to simplify the problem. These simplifications however could potentially affect the optimality of efficiency and therefore influence its finite performance. Fortunately, due to the increasing accessibility of parallel hardware (e.g. GPU), a direct approximation is becoming more tractable. Thus, we want to test the performance of an allocation rule based on an unbiased and direct approximation, expecting an acceleration on the performance. In this paper, we target on one of the famous objectives, the Probability of Correct Selection (PCS). Numerical experiments were done, showing a considerable improvement in finite performance of our algorithm comparing to a traditional one.

In this study, we consider ranking and selection problems where the simulation model is subject to input uncertainty. Under the input uncertainty, we compare system designs based on their worst-case performance, and seek to maximize the probability of selecting the design with the best performance under the worst-case scenario. By approximating the probability of correct selection (PCS), we develop an asymptotically (as the simulation budget goes to infinity) optimal solution of the resulting problem. An efficient selection procedure is designed within the optimal computing budget allocation (OCBA) framework. Numerical tests show the high efficiency of the proposed method.

Ordinal optimization is an efficient technique to choose and rank various engineering designs that require time-consuming discrete-event simulations. Optimal computing budget allocation (OCBA) has been an important tool to enhance its efficiency such that the best design is selected in a timely fashion. It, however, fails to address the issue of selecting the best and worst designs efficiently. The need to select both rapidly given a fixed computing budget has arisen from many applications. This work develops a new OCBA-based approach for selecting both best and worst designs at the same time. Its theoretical foundation is laid. Our numerical results show that it can well outperform all the existing methods in terms of probability of correct selection and computational efficiency.

Evolutionary algorithm assisted by surrogate model in the framework of ordinal optimization and optimal computing budget allocation

Particle Swarm Optimization (PSO) is a population-based algorithm designed to find good solutions to optimization problems. Its characteristics have encouraged its adoption to tackle a variety of problems in different fields. However, when such problems are subject to noise, the performance of PSO suffers an immediate deterioration which demands the incorporation of noise handling mechanisms. One such mechanism comprises resampling methods, which re-evaluate the solutions multiple times in order to estimate their true objective values. The state-of-the-art integration with which the best results have been obtained utilizes the resampling method named Optimal Computing Budget Allocation (OCBA). This resampling method starts by estimating the objective values of all the solutions via Equal Resampling (ER), and then sequentially allocating further re-evaluations to the estimated best solutions. However, after having a first estimate via ER, we question the importance of the additional efforts to correctly select the true best solutions when a good-enough and accurate one can be selected. In this paper, we propose a new PSO algorithm based on ER in which the additional evaluations are allocated at once to the estimated best solutions, thus skipping the complexity of using OCBA. Experiments on 20 large-scale benchmark functions subject to different levels of noise show that the proposed algorithm produces similar results to PSO with OCBA in most cases.

The limitations of traditional indifference-zone (IZ) ranking and selection procedures stem from their reliance on an IZ parameter, which is often unknown in practice and can lead to inefficiencies. Additionally, since exact evaluation of the probability of correct selection (PCS) is generally unattainable, these approaches commonly rely on a Bonferroni-Inequality-type lower bound to ensure the worst-case PCS, which introduces further conservativeness. To overcome these challenges, we propose fully sequential IZ-free procedures that eliminate the need for an IZ parameter and reduce the conservativeness associated with Bonferroni Inequality approaches. We demonstrate that, in the context of IZ-free procedures, the screening statistic corresponds to the Euclidean distance of a multi-dimensional standard Brownian motion, which behaves as a Bessel process. This connection simplifies the derivation of statistical guarantees for the proposed procedures and facilitates the analytical expressions for key parameters. Our procedures accommodate both known equal variances and unknown unequal variances, with statistical guarantees established in the known equal variance case. Numerical experiments confirm that the proposed procedures outperform existing methods in reducing PCS conservativeness and improving sample size efficiency. These advancements enhance the robustness and practicality of IZ-free procedures for diverse applications.

The dynamics of many systems nowadays follow not only physical laws but also man-made rules. These systems are known as discrete event dynamic systems and their performances can be accurately evaluated only through simulations. Existing studies on simulation-based optimization (SBO) usually assume deterministic simulation time for each replication. However, in many applications such as evacuation, smoke detection, and territory exploration, the simulation time is stochastic due to the randomness in the system behavior. We consider the computing budget allocation for SBO's with stochastic simulation time in this paper, which has not been addressed in existing literatures to the author's best knowledge. We make the following major contribution. The relationship between simulation time and performance estimation accuracy is quantified. It is shown that when the asymptotic performance is of interest only the mean value of individual simulation time matters. Then based on the existing optimal computing budget allocation (OCBA) method for deterministic simulation time we develop OCBA for stochastic simulation time (OCBAS), and show that OCBAS is asymptotically optimal. Numerical experiments are used to discuss the impact of the variance of simulation time, the impact of correlated simulation time and performance estimation, and to demonstrate the performance of OCBAS on a smoke detection problem in wireless sensor network. The numerical results also show that OCBA for deterministic simulation time is robust even when the simulation time is stochastic.

Although particle swarm optimization (PSO) is a powerful evolutionary algorithm for solving nonlinear optimization problems in deterministic environments, many practical problems have some stochastic noise. The optimal computing budget allocation (OCBA) has been integrated into PSO in various ways to cope with this. The OCBA can mitigate the effect of noise on PSO by selecting the best solution under a limited evaluation budget. Recently, with the increasing complexity of PSO applications, the evaluation costs are also increasing rapidly, which has sparked the need for more efficient PSO in stochastic environments. This article proposes a simple yet effective adjustment to the integration of OCBA to further improve the efficiency of PSO. The proposed adjustment allows OCBA to expand its search space to find the global best position more correctly such that the entire swarm can move on a better direction under stochastic noise. The experimental results on various benchmarks demonstrate the improved performance of PSO by the proposed adjustment under a limited budget compared with the latest studies. In addition, the results regarding fighters' evasion flight optimization emphasize the practical need for the proposed adjustment.

Selection of planting locations is an important issue in plant breeding and may significantly impact the probability of correctly identifying the desired genotype based on noisy data. This is especially true for early‐stage experimentation where a set of experimental genotypes is observed in very few locations. This study aims to identify locations that are most helpful for discriminating the main effects of two or more genotypes. A regression‐based model is proposed that characterizes the sensitivity of location to the genotype main effect, and locations with significant positive sensitivity are termed discriminative. Comparing or ranking genotypes using these locations results in exaggerated expected differences between genotype main effects, allowing for more accurate comparisons in the presence of noise. Based on the proposed model, an expression for the probability of making correct genotype selection is derived given a set of locations with known location sensitivities. The probability of correct selection is then used to characterize an optimal set of location that maximizes the probability of selecting the genotype with better main effect. These locations are independent of the genotypes being compared and only depend on the number of locations being considered and the sensitivities of these locations to genotype effects. The theoretical implications of the model are validated using data obtained from a commercial soybean breeding program. The results show that by using discriminating locations correct selection is made 76% of the time, versus 67% for all locations and only 63% of the time for non‐discriminating locations.

The problem of selecting the Bernoulli population which has the highest success probability is considered. It has been noted in several articles that the probability of a correct selection is the same, uniformly in the Bernoulli p-vector (P1,P2,….,Pk), for two or more different selection procedures. We give a general theorem which explains this phenomenon. An application of particular interest arises when curtailment of a single-stage procedure (as introduced by Bechhofer and Kulkarni (1982a) )is employed; the corresponding result for curtailment of a single-stage procedure needs no proof. The use of strong curtailment in place of weak curtailment requires no more (and usually many less) observations to achieve the same.

In recent years, with the rapid development of computing technology, developing parallel procedures to solve large-scale ranking and selection (R&S) problems has attracted a lot of research attention. In this paper, we take fixed-budget R&S procedure as an example to investigate potential issues of developing parallel procedures. We argue that to measure the performance of a fixed-budget R&S procedure in solving large-scale problems, it is important to quantify the minimal growth rate of the total sampling budget such that as the number of alternatives increases, the probability of correct selection (PCS) would not decrease to zero. We call such a growth rate of the total sampling budget the rate for maintaining correct selection (RMCS). We show that a tight lower bound for the RMCS of a broad class of existing fixed-budget procedures is in the order of [Formula: see text], where k is the number of alternatives. Then, we propose a new type of fixed-budget procedure, namely the fixed-budget knockout-tournament ([Formula: see text]) procedure. We prove that, in terms of the RMCS, our procedure outperforms existing fixed-budget procedures and achieves the optimal order, that is, the order of k. Moreover, we demonstrate that our procedure can be easily implemented in parallel computing environments with almost no nonparallelizable calculations. Last, a comprehensive numerical study shows that our procedure is indeed suitable for solving large-scale problems in parallel computing environments. History: Accepted by Bruno Tuffin, Area Editor for Simulation. Funding: Y. Zhong was supported by the National Natural Science Foundation of China [Grant 72101047]. L. J. Hong was supported by the National Natural Science Foundation of China [Grants 72091211 and 72161160340]. G. Jiang was supported by the National Natural Science Foundation of China [Grants 72121001 and 72171060]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijoc.2022.1221 .

Stochastic point location (SPL) is to search for a target point on the line in stochastic environment. An SPL solver can be described as a Learning Machine (LM) attempting to locate a target point on a line. By using the prompts from stochastic environment, possibly erroneous, the LM moves along the line yielding updated estimates to approximate the target point. This paper proposes an SPL algorithm based on Optimal Computing Budget Allocation (OCBA), named as SPL-OCBA, which employs OCBA and the historical sample information to guide to the location of a target point in stationary and stochastic environment. The proposed algorithm partitions or combines the subintervals of the target line adaptively. Then, OCBA considers such subintervals as its designs and allocates the sample budget for them based on the historical data, thereby resulting in a new method. Extensive experiments show that the newly proposed algorithm is significantly more efficient than the existing ones.