Practical considerations in selecting the best set of simulated systems

We describe a practical two-stage algorithm, BootComp, for multi-objective optimization via simulation. Our algorithm finds a subset of good designs that a decision-maker can compare to identify the one that works best when considering all aspects of the system, including those that cannot be modeled. BootComp is designed to be straightforward to implement by a practitioner with basic statistical knowledge in a simulation package that does not support sequential ranking and selection. These requirements restrict us to a two-stage procedure that works with any distributions of the outputs and allows for the use of common random numbers. Comparisons with sequential ranking and selection methods suggest that it performs well, and we also demonstrate its use analyzing a real simulation aiming to determine the optimal ward configuration for a UK hospital.

Fully sequential selection procedures have been developed in the field of stochastic simulation to find the simulated system with the best expected performance when the number of alternatives is finite. Kim and Nelson proposed the procedure to allow for unknown and unequal variances and the use of common random numbers. approximates the raw sum of differences between observations from two systems as a Brownian motion process with drift and uses a triangular continuation region to decide the stopping time of the selection process. In this paper new fully sequential selection procedures are derived that employ a more effective sum of differences, which is called a controlled sum. Two provably valid procedures and an approximate procedure are described. Empirical results and a realistic illustration are provided to compare the efficiency of these procedures with other procedures that solve the same problem. [Supplemental materials are available for this article. Go to the publisher's online edition of IIE Transactions for the following supplemental resources: Proofs and guidelines to choose appropriate parameters.]

This chapter discusses procedures for selection from one-parameter and two-parameter exponential populations, and also some related inference problems. It provides a brief introduction to selection and ranking methodology explaining the salient features. The chapter describes single-stage procedures based on complete as well as Type I, Type II, and randomly censored samples. It deals with selection from two- parameter exponential populations in terms of their guaranteed lifetimes. The chapter examines single-stage, two-stage, three-stage, and purely sequential procedures. It suggests comparison of exponential guaranteed lifetimes with a standard or a control. The chapter is concerned with estimation concerning selected populations. Selection and ranking problems have generally been studied by using either the indifference zone (IZ) formulation of R. E. Bechhofer or the subset selection (SS) formulation due mainly to S. S. Gupta. There are several variations and generalizations of the basic goal in both IZ and SS formulations.

Specifying a proper input distribution is often a challenging task in simulation modeling. In practice, there may be multiple plausible distributions that can fit the input data reasonably well, especially when the data volume is not large. In this paper, we consider the problem of selecting the best from a finite set of simulated alternatives, in the presence of such input uncertainty. We model such uncertainty by an ambiguity set consisting of a finite number of plausible input distributions and aim to select the alternative with the best worst-case mean performance over the ambiguity set. We refer to this problem as robust selection of the best (RSB). To solve the RSB problem, we develop a two-stage selection procedure and a sequential selection procedure; we then prove that both procedures can achieve at least a user-specified probability of correct selection under mild conditions. Extensive numerical experiments are conducted to investigate the computational efficiency of the two procedures. Finally, we apply the RSB approach to study a queueing system’s staffing problem using synthetic data and an appointment-scheduling problem using real data from a large hospital in China. We find that the RSB approach can generate decisions significantly better than other widely used approaches. This paper was accepted by Noah Gans, stochastic models and simulation.

The indifference-zone (IZ) formulation of ranking and selection (R&S) is the foundation of many procedures that have been useful for choosing the best among a finite number of simulated alternatives. Of course, simulation models are imperfect representations of reality, which means that a simulation-based decision, such as choosing the best alternative, is subject to model risk. In this paper we explore the impact of model risk due to input uncertainty on IZ R&S. "Input uncertainty" is the result of having estimated ("fit") the simulation input models to observed real-world data. We find that input uncertainty may force the user to revise, or even abandon, their objectives when employing a R&S procedure, or it may have very little effect on selecting the best system even when the marginal input uncertainty is substantial.

In this paper we examine methods for the selection of measurement subsets using information theory. These methods are applied to an overdetermined inverse problem typical to high‐resolution spectrometry. The optimal selection of spectral channels becomes especially important with the recent introduction of high spectral resolution satellite instruments for monitoring the atmosphere. The vast amount of data these instruments provide in a continuous fashion makes it necessary to carefully consider which channels are really needed in the retrieval of geophysical quantities. Two optimization problems in channel selection, both taking the information content of the measurements as a criterion, are defined and discussed. The concept of the information content of each individual measurement is introduced, and basic analytical relations are derived. A sequential deselecting procedure is developed, and a fast algorithm for channel selection is proposed. These provide approximations to solutions of the optimization problem. The efficiencies of these selection procedures and a sequential selection procedure [Rodgers, 1996] are compared by means of a Monte Carlo generation of the forward model in a low‐dimensional case. As a real application, the selection of the most informative spectral channels for GOMOS measurements is considered.

The goal of ranking and selection (R8S) procedures is to identify the best among a finite set of alternative systems evaluated by stochastic simulation, providing a probability guarantee on the quality of the solution. To solve large-scale R8S problems, especially in parallel computing platforms where variable numbers of cores might be used, it is helpful to be able to predict the simulation budget, which is almost always the dominant portion of the running time of a given procedure for a given problem. Non-trivial issues arise due to the need to estimate the system configuration. We propose a set of methods for predicting the simulation budget. Numerical results compare our predictions for several leading R8S procedures.

AutoStat/sup TM/ is an extension package for AutoMod/sup TM/ and AutoSched/sup TM/ models that provides complete support for simulation model experimentation and statistical analysis of outputs. Within a menu-driven, point and click environment, AutoStat provides assistance for setting up runs, automated execution of runs and consolidation of outputs across replications and scenarios. AutoStat automatically sets random number seeds to achieve statistically independent replications; it also sets factor levels (input parameters) to realize desired scenarios, without having to modify the underlying model. AutoStat has a number of advanced features such as datafile factors, responses (or performance measures) read from custom user output files, and user-defined combination responses (linear combinations of other responses). AutoStat offers several statistical methods, including confidence intervals, a ranking and selection procedure, design of experiments, and warm-up determination. Outputs can be displayed and printed in tabular and graphical formats and exported to spreadsheet and other analysis software. AutoStat saves simulation modelers considerable time and automates the effort of setting up and making runs, managing the input and output files, consolidating results and conducting analyses.

Live fish recognition in the open sea is a challenging multi-class classification task. We propose a novel method to recognize fish in an unrestricted natural environment recorded by underwater cameras. This method extracts 66 types of features, which are a combination of color, shape and texture properties from different parts of the fish and reduce the feature dimensions with forward sequential feature selection (FSFS) procedure. The selected features of the FSFS are used by an SVM. We present a Balance-Guaranteed Optimized Tree (BGOT) to control the error accumulation in hierarchical classification and, therefore, achieve better performance. A BGOT of 10 fish species is automatically constructed using the inter-class similarities and a heuristic method. The proposed BGOT-based hierarchical classification method achieves about 4% better accuracy compared to state-of-the-art techniques on a live fish image dataset.

R peak detection is crucial in electrocardiogram (ECG) signal analysis to detect and diagnose cardiovascular diseases (CVDs). Herein, the dynamic mode selected energy (DMSE) and adaptive window sizing (AWS) algorithm are proposed for detecting R peaks with better efficiency. The DMSE algorithm adaptively separates the QRS components and all non-objective components from the ECG signal. Based on local peaks in QRS components, the AWS algorithm adaptively determines the Region of Interest (ROI). The Feature Extraction process computes the statistical properties of energy, frequency, and noise from each ROI. The Sequential Forward Selection (SFS) procedure is used to find the best subsets of features. Based on these characteristics, an ensemble of decision tree algorithms detects the R peaks. Finally, the R peak position on the initial ECG signal is adjusted using the R location correction (RLC) algorithm. The proposed method has an experimental accuracy of 99.94%, a sensitivity of 99.98%, positive predictability of 99.96%, and a detection error rate of 0.06%. Given the high efficiency in detection and fast processing speed, the proposed approach is ideal for intelligent medical and wearable devices in the diagnosis of CVDs.

Sequential Forward Selection is one of the most popular variable selection methods and is widely adopted in many applications, among which the design of classifiers. The main problem related to this approach regards the stability of the selection, namely the exploitation of different training data set can lead to the selection of different variable subsets. This problem is particularly critical in applications where variable selection is used to interpret the behaviour of the process or phenomenon under consideration, i.e. to understand which among a potentially huge list of variables actually affect the classification. The paper proposes a method that improves the stability of the sequential forward selection procedure while preserving and possibly improving the classification performance.

The limitations of traditional indifference-zone (IZ) ranking and selection procedures stem from their reliance on an IZ parameter, which is often unknown in practice and can lead to inefficiencies. Additionally, since exact evaluation of the probability of correct selection (PCS) is generally unattainable, these approaches commonly rely on a Bonferroni-Inequality-type lower bound to ensure the worst-case PCS, which introduces further conservativeness. To overcome these challenges, we propose fully sequential IZ-free procedures that eliminate the need for an IZ parameter and reduce the conservativeness associated with Bonferroni Inequality approaches. We demonstrate that, in the context of IZ-free procedures, the screening statistic corresponds to the Euclidean distance of a multi-dimensional standard Brownian motion, which behaves as a Bessel process. This connection simplifies the derivation of statistical guarantees for the proposed procedures and facilitates the analytical expressions for key parameters. Our procedures accommodate both known equal variances and unknown unequal variances, with statistical guarantees established in the known equal variance case. Numerical experiments confirm that the proposed procedures outperform existing methods in reducing PCS conservativeness and improving sample size efficiency. These advancements enhance the robustness and practicality of IZ-free procedures for diverse applications.

The purpose of making efficient and flexible manufacturing systems is often related to the possibility to analyze the system considering at the same time a wide number of parameters and their interactions. Simulation models are proved to be useful to support and drive company management in improving the performances of production and logistic systems. However, to achieve the expected results, a detailed model of the production and logistic system is needed as well as a structured error analysis to guarantee results reliability. The aim of this paper is to give some practical guide lines in order to drive the error analysis for discrete event stochastic simulation model that are widely used to study production and logistic system.Keywordsdiscrete eventsimulation modelerror analysisstochastic model

PurposeThe purpose of this study is to gain some insights into the number of shortages resulting from two alternative demand allocation schemes between a contractor (machine M1) and subcontractor (machine M2), on the one hand, and from inventory accumulation, on the other hand. The shortages stem from random machine breakdowns, and each machine undergoes preventive maintenance. The motivation behind inventory accumulation is to allow demand to be fulfilled even when both machines are down.Design/methodology/approachThe number of shortages stemming from all scenarios under consideration was established via computer simulation with the Arena© language.FindingsFor demand allocation that remains unchanged for the duration of the planning horizon and constant reliability of M1, it was found that, the less reliable M2 is, the more biased in favour of M1 will be the optimal demand allocation and the greater will be the number of shortages. Moreover, both dynamic demand reallocation over the planning horizon and inventory accumulation result in a substantial reduction in shortages.Research limitations/implicationsThe results are representative of the specific data, which were assumed in the simulation models. Nevertheless, this methodology is recommended for this type of analysis, as it is highly flexible and can take into account many practical considerations, which an analytical approach cannot.Practical implicationsWithin the context of unreliable production machines, the most important practical implication of this study is that the dynamic reallocation of demand between a contractor and subcontractor, along with inventory accumulation, both have the potential to yield important reductions in the number of shortages.Originality/valueThe subject‐matter of this paper was not previously reported in the literature. Furthermore, the insights gained as a result of this study can yield substantial benefits to companies in terms of improving their service levels as measured by reduced shortages.

The primary goal of this work was to apply data-driven machine learning regression to assess if resting state functional connectivity (rs-FC) could estimate measures of behavioral domains in stroke subjects who completed brain-computer interface (BCI) intervention for motor rehabilitation. The study cohort consisted of 20 chronic-stage stroke subjects exhibiting persistent upper-extremity motor deficits who received the intervention using a closed-loop neurofeedback BCI device. Over the course of this intervention, resting state functional MRI scans were collected at four distinct time points: namely, pre-intervention, mid-intervention, post-intervention and 1-month after completion of intervention. Behavioral assessments were administered outside the scanner at each time-point to collect objective measures such as the Action Research Arm Test, Nine-Hole Peg Test, and Barthel Index as well as subjective measures including the Stroke Impact Scale. The present analysis focused on neuroplasticity and behavioral outcomes measured across pre-intervention, post-intervention and 1-month post-intervention to study immediate and carry-over effects. Rs-FC, changes in rs-FC within the motor network and the behavioral measures at preceding stages were used as input features and behavioral measures and associated changes at succeeding stages were used as outcomes for machine-learning-based support vector regression (SVR) models. Potential clinical confounding factors such as age, gender, lesion hemisphere, and stroke severity were included as additional features in each of the regression models. Sequential forward feature selection procedure narrowed the search for important correlates. Behavioral outcomes at preceding time-points outperformed rs-FC-based correlates. Rs-FC and changes associated with bilateral primary motor areas were found to be important correlates of across several behavioral outcomes and were stable upon inclusion of clinical variables as well. NIH Stroke Scale and motor impairment severity were the most influential clinical variables. Comparatively, linear SVR models aided in evaluation of contribution of individual correlates and seed regions while non-linear SVR models achieved higher performance in prediction of behavioral outcomes.