A practical guide for operational validation of discrete simulation models

The paper presents a new game theoretic approach towards the validation of discrete event air combat simulation models. In the approach, statistical techniques are applied for estimating game models based on simulation data. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in air combat. The approach enhances existing methods for the validation of discrete event simulation models by incorporating the inherent game setting of air combat into the analysis. The estimated games also provide a novel game theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game theoretic approach is illustrated by analyzing simulation data produced with an existing air combat simulation model.

This paper presents a new game-theoretic approach toward the validation of discrete-event air combat (AC) simulation models and simulation-based optimization. In this approach, statistical techniques are applied for estimating games based on data produced by a simulation model. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in AC. These games are also applied for simulation-based optimization in a two-sided setting in which the action of the opponent is taken into account. In optimization, the estimated games enable the study of effectiveness of AC tactics as well as aircraft, weapons, and avionics configurations. The game-theoretic approach enhances existing methods for the validation of discrete-event simulation models and techniques for simulation-based optimization by incorporating the inherent game setting of AC into the analysis. It also provides a novel game-theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game-theoretic approach is illustrated by analyzing simulation data obtained with an existing AC simulation model.

A new method for determining the causes of discrepancies between dynamic simulation models and measured data is described. Whilst principally aimed at the validation of building thermal simulation codes, the method may rind application in the validation of simulation models from other disciplines. The new method relies on generating a model of the discrepancies in terms of the variables driving the physical reality which produced the experimental data and the corresponding simulation. Inspection of the resulting error model allows the principal contributors to simulation error to be identified, its dynamic nature to be characterised, and the likely cause of the error to be identified. The method is tested by generating a 'quasi-truth' dataset using the building thermal simulation code SERI-RES. A series of simulations is then prepared with perturbations to selected input parameters. The new tool is found to be capable of recovering details of the perturbations. Finally, the power of the new technique is demonstrated in a series of comparisons between the predictions of the model SERI-RES and data collected in outdoor test rooms. These comparisons reveal that the principal source of error in the predictions is the treatment of the interactions between the heater and the air and fabric of the room. The new technique proves sufficiently sensitive to detect the changing structure of the prediction errors as the position of the heater within the room is changed.

This paper gives the history of verification and validation of discrete-event simulation models as seen through the eyes of its authors and their experiences. The history is divided into three time periods: the early era covering years up to 1970, the awareness era covering the years of the 1970s and 1980s, and the modern era covering the years of 1990 to the present.

Statistical method, and in particular, simulation is the most widely used operations research modelling approach now employed by industry and government. Validation of a simulation model is the procedure designed to determine whether the model is an accurate representation of the portion of reality under consideration. It is a process distinct from model building or debugging (although interrelated). In this work, strategies and practical numerical techniques for validating large-scale production simulation models (or any other large system) are detailed. An evolution of the central concepts of large-scale model validation is represented first. The distinction between model user's risk and model builder's risk is highlighted as an important development which alters the problem into a consideration of the basis a decision-maker should use in determining whether to rely on the outputs of such models. Two quantitative techniques which are applications of the Turing test and of mathematical programming are de...

Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation

In this expository paper we give a general introduction to verification and validation of simulation models, define the various validation techniques, and present a recommended model validation procedure.

This paper illustrates how to improve robotic automotive body shop throughput by feeding back actual plant data for input variables for entities to improve discrete simulation model prediction and validation. The input variables are adjusted based on actual plant data, and the simulation model was run for various scenarios to verify and validate the discrete simulation model. The general tendency in an automotive plant body shop is to buy more equipment to improve throughput. It is very interesting to see how mean time to repair/mean time between failure assumptions affect the throughput. The bottleneck issues are identified for automotive plant to improve throughput.

Network-based computer simulation models are powerful tools for analyzing and guiding policy formation related to the actual systems being modeled. However, the inherent data and computationally intensive nature of this model class gives rise to fundamental challenges when it comes to executing typical experimental designs. In particular this applies to model validation. Manual management of the complex simulation work-flows along with the associated data will often require a broad combination of skills and expertise. Examples of skills include domain expertise, mathematical modeling, programming, high-performance computing, statistical designs, data management as well as the tracking all assets and instances involved. This is a complex and error-prone process for the best of practices, and even small slips may compromise model validation and reduce human productivity in significant ways. In this paper, we present a novel framework that addresses the challenges of model validation just mentioned. The components of our framework form an ecosystem consisting of (i) model unification through a standardized model configuration format, (ii) simulation data management, (iii) support for experimental designs, and (iv) methods for uncertainty quantification, and sensitivity analysis, all ultimately supporting the process of model validation. (Note that our view of validation is much more comprehensive than simply ensuring that the computational model can reproduce instance of historical data.) This is an extensible design where domain experts from e.g. experimental design can contribute to the collection of available algorithms and methods. Additionally, our solution directly supports reproducible computational experiments and analysis, which in turn facilitates independent model verification and validation. Finally, to showcase our design concept, we provide a sensitivity analysis for examining the consequences of different intervention strategies for an influenza pandemic.

The evaluation and validation of simulation models are important stages of model development. The aim of this work was to validate the Buttercup model (BCM) with experimental data from on-farm trials. Two on-farm experiments (Experiment 1 E1 and Experiment 2 E2) were carried out to evaluate the effect of supplementing dairy cattle with different sources of protein in Small Scale Dairy Farming Systems. Three concentrates (C1= maize grain 610, soybean meal 310, urea 30, and cane molasses 50g kg−1 DM; C2= maize grain 630, soybean meal 320 and cane molasses 50g kg−1 DM; and C3= maize grain 900, urea 50 and cane molasses 50g kg−1 DM) were developed using the BCM. The effect of the concentrates on milk yield, body condition score and live weight was evaluated using a double Latin Square experimental design. In E1 there were no significant differences (P> 0.05) for the variables evaluated. In E2, significant differences (P<0.05) were observed in milk fat content, for C2 (33.8g kg−1) was higher than C1 and C3 (26.2 and 26.0g kg−1). The results suggest that it was possible to fulfil the metabolisable protein requirements of the cows in these systems with low cost sources of non-protein nitrogen like the urea used in C3; and that the synthesis of microbial protein (from non-protein nitrogen and fermentable metabolisable energy of the concentrates) was enough to fulfil the cows’ requirements of MP for the milk yield levels observed in both experiments. The results also indicate that when specific input data set from the experiments was used the model predictions are acceptable, when comparing predicted vs. observed data as a very significant correlation was observed (P<0.01).

For the verification and validation of microscopic simulation models of pedestrian flow, we have performed experiments for different kind of facilities and sites where most conflicts and congestion happens e.g. corridors, narrow passages, and crosswalks. The validity of the model should compare the experimental conditions and simulation results with video recording carried out in the same condition like in real life e.g. pedestrian flux and density distributions. The strategy in this technique is to achieve a certain amount of accuracy required in the simulation model. This method is good at detecting the critical points in the pedestrians walking areas. For the calibration of suitable models we use the results obtained from analysing the video recordings in Hajj 2009 and these results can be used to check the design sections of pedestrian facilities and exits. As practical examples, we present the simulation of pilgrim streams on the Jamarat bridge (see Figure 5). The objectives of this study are twofold: first, to show through verification and validation that simulation tools can be used to reproduce realistic scenarios, and second, gather data for accurate predictions for designers and decision makers.

The proper simulation of the hygrothermal behaviour of historical buildings is a challenging task with several implications regarding the evaluation of indoor thermal comfort and the suitability of retrofit strategies that comply with the conservation of cultural heritage. An inaccurate simulation may lead to inadequate conclusions, which could result in inappropriate and dangerous actions for the preservation of the heritage buildings.Then calibration and validation of hygrothermal simulation models are essential steps to achieve more accurate and reliable results.Now, although some agencies have developed guidelines and methodologies to carry out the validation of building performance models, all of them are based on energy consumption only. However, since in some buildings the energy consumption data are not always available especially when no operating heating, ventilation and air conditioning system is installed, which is the case of many historical buildings, the microclimatic parameters are usually adopted in the validation process. In this case, neither protocols nor specific parameters have been officially recognised to perform the model validation. The present work reviewed the main approaches used by researchers for building performance model validation with special reference to historical buildings based on microclimatic parameters, highlighting the main advantages and drawbacks of the different methods reviewed. Finally, recommendations to properly carry out the model validation based on microclimatic parameters have been provided. The collected information may be useful to different subjects (e.g. designers, energy auditors, researchers, conservators, buildings’ owners and policy makers) and can drive suitable and reliable retrofit and maintenance interventions.

In this work, a literature survey on the validation of vehicle dynamics simulation models is presented. Estimating the dynamic responses of existing or proposed vehicles has a wide array of applications in the development of vehicle technologies, e.g. active suspensions, controller design, driver assistance systems, etc. Although simulation environments, measurement tools and mathematical theories on vehicle dynamics are well established, the methodical link between the experimental test data and validity analysis of the simulation model is still lacking. This report presents different views on the definition of validation, and its usage in vehicle dynamics simulation models.

This paper investigates the general problem of comparing multidimensional simulation output with a given data set (e.g., real-world historical data). This problem frequently arises in verification, validation, and calibration of simulation models with spatial output statistics as in weather/climate, epidemic, swarm/crowd, social systems, communication networks, and many other applications where the simulation output is distributed across various locations or geographical regions. In the case of univariate simulation output, two-sample statistical hypothesis tests such as the t -test are commonly used. For simulation models with multidimensional and spatial output statistics, the Hotelling’s two-sample test is widely used as the benchmark method in the simulation literature. However, the Hotelling’s test assumes that the two samples come from multivariate Gaussian distributions with equal covariance matrices, which may not be the case in many applications. To address this gap, this paper proposes a double-bootstrap method based on the Wasserstein distance for comparing two multidimensional samples. Unlike the Hotelling’s test and other parametric approaches, the proposed method does not require restrictive distributional assumptions, enabling a wider range of applications and contributing to verification, validation, and calibration of simulation models with multidimensional output. Computational experiments are performed to assess the test power, and the results indicate that the proposed method outperforms the Hotelling’s test and various other approaches. The proposed method’s applicability is illustrated through two examples related to random walk of swarm particles on a two-dimensional space and a realistic engineering application involving simulation of unmanned aerial vehicle (UAV) communication systems.

In this paper we discuss validation and verification of simulation models. Four different approaches to deciding model validity are described; two different paradigms that relate validation and verification to the model development process are presented; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are discussed; a way to document results is given; a recommended procedure for model validation is presented; and accreditation is briefly discussed.