Game theoretic validation of air combat simulation models

This paper presents a new game-theoretic approach toward the validation of discrete-event air combat (AC) simulation models and simulation-based optimization. In this approach, statistical techniques are applied for estimating games based on data produced by a simulation model. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in AC. These games are also applied for simulation-based optimization in a two-sided setting in which the action of the opponent is taken into account. In optimization, the estimated games enable the study of effectiveness of AC tactics as well as aircraft, weapons, and avionics configurations. The game-theoretic approach enhances existing methods for the validation of discrete-event simulation models and techniques for simulation-based optimization by incorporating the inherent game setting of AC into the analysis. It also provides a novel game-theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game-theoretic approach is illustrated by analyzing simulation data obtained with an existing AC simulation model.

A new method for determining the causes of discrepancies between dynamic simulation models and measured data is described. Whilst principally aimed at the validation of building thermal simulation codes, the method may rind application in the validation of simulation models from other disciplines. The new method relies on generating a model of the discrepancies in terms of the variables driving the physical reality which produced the experimental data and the corresponding simulation. Inspection of the resulting error model allows the principal contributors to simulation error to be identified, its dynamic nature to be characterised, and the likely cause of the error to be identified. The method is tested by generating a 'quasi-truth' dataset using the building thermal simulation code SERI-RES. A series of simulations is then prepared with perturbations to selected input parameters. The new tool is found to be capable of recovering details of the perturbations. Finally, the power of the new technique is demonstrated in a series of comparisons between the predictions of the model SERI-RES and data collected in outdoor test rooms. These comparisons reveal that the principal source of error in the predictions is the treatment of the interactions between the heater and the air and fabric of the room. The new technique proves sufficiently sensitive to detect the changing structure of the prediction errors as the position of the heater within the room is changed.

This paper gives the history of verification and validation of discrete-event simulation models as seen through the eyes of its authors and their experiences. The history is divided into three time periods: the early era covering years up to 1970, the awareness era covering the years of the 1970s and 1980s, and the modern era covering the years of 1990 to the present.

Statistical method, and in particular, simulation is the most widely used operations research modelling approach now employed by industry and government. Validation of a simulation model is the procedure designed to determine whether the model is an accurate representation of the portion of reality under consideration. It is a process distinct from model building or debugging (although interrelated). In this work, strategies and practical numerical techniques for validating large-scale production simulation models (or any other large system) are detailed. An evolution of the central concepts of large-scale model validation is represented first. The distinction between model user's risk and model builder's risk is highlighted as an important development which alters the problem into a consideration of the basis a decision-maker should use in determining whether to rely on the outputs of such models. Two quantitative techniques which are applications of the Turing test and of mathematical programming are de...

Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation

As the number of simulation experiments increases, the necessity for validation and verification of these models demands special attention on the part of the simulation practitioners. By analyzing the current scientific literature, it is observed that the operational validation description presented in many papers does not agree on the importance designated to this process and about its applied techniques, subjective or objective. With the expectation of orienting professionals, researchers and students in simulation, this article aims to elaborate a practical guide through the compilation of statistical techniques in the operational validation of discrete simulation models. Finally, the guide's applicability was evaluated by using two study objects, which represent two manufacturing cells, one from the automobile industry and the other from a Brazilian tech company. For each application, the guide identified distinct steps, due to the different aspects that characterize the analyzed distributions

This paper illustrates how to improve robotic automotive body shop throughput by feeding back actual plant data for input variables for entities to improve discrete simulation model prediction and validation. The input variables are adjusted based on actual plant data, and the simulation model was run for various scenarios to verify and validate the discrete simulation model. The general tendency in an automotive plant body shop is to buy more equipment to improve throughput. It is very interesting to see how mean time to repair/mean time between failure assumptions affect the throughput. The bottleneck issues are identified for automotive plant to improve throughput.

The evaluation and validation of simulation models are important stages of model development. The aim of this work was to validate the Buttercup model (BCM) with experimental data from on-farm trials. Two on-farm experiments (Experiment 1 E1 and Experiment 2 E2) were carried out to evaluate the effect of supplementing dairy cattle with different sources of protein in Small Scale Dairy Farming Systems. Three concentrates (C1= maize grain 610, soybean meal 310, urea 30, and cane molasses 50g kg−1 DM; C2= maize grain 630, soybean meal 320 and cane molasses 50g kg−1 DM; and C3= maize grain 900, urea 50 and cane molasses 50g kg−1 DM) were developed using the BCM. The effect of the concentrates on milk yield, body condition score and live weight was evaluated using a double Latin Square experimental design. In E1 there were no significant differences (P> 0.05) for the variables evaluated. In E2, significant differences (P<0.05) were observed in milk fat content, for C2 (33.8g kg−1) was higher than C1 and C3 (26.2 and 26.0g kg−1). The results suggest that it was possible to fulfil the metabolisable protein requirements of the cows in these systems with low cost sources of non-protein nitrogen like the urea used in C3; and that the synthesis of microbial protein (from non-protein nitrogen and fermentable metabolisable energy of the concentrates) was enough to fulfil the cows’ requirements of MP for the milk yield levels observed in both experiments. The results also indicate that when specific input data set from the experiments was used the model predictions are acceptable, when comparing predicted vs. observed data as a very significant correlation was observed (P<0.01).

In this expository paper we give a general introduction to verification and validation of simulation models, define the various validation techniques, and present a recommended model validation procedure.

The present paper describes a methodological approach to perform a meaningful comparison between high-speed video recordings showing the dispersion of a fragment cloud after an impact event and the corresponding numerical simulations of this event. The data acquisition is based on the (automated) analysis of the image data from the experiment in order to detect fragments, to identify them across multiple frames (tracking), and to obtain quantitative data on the number of fragments and their size distribution. This data set is usually incomplete and subject to errors. In the same way, the simulation data are analyzed; these must be prepared in such a way that they are also available as image data with properties as similar as possible to the video recordings. This allows the same fragment analysis procedure to be applied and a fair comparison to be made. The paper describes the procedure applied to the fragmentation of cryogenic pellets (neon and deuterium) for the ITER disruption mitigation system and shows how this approach can be used to calibrate unknown material parameters and validate simulation models.

Computer programs designed for traffic microsimulation have gained considerable popularity and widespread use, both in practical application as well as in research works. The number of traffic simulation models has increased significantly and until the end of the last century there were more then 50 simulation models available already. Their advantage is the ability to predict behavior in everyday traffic, as well as in emergency situations. Unlike conventional analytical methods for traffic analysis, simulation models can also display the dynamics of traffic systems and 2D or 3D animations, as a result of their modeling, are much more understandable to incompetent population. The aim of this paper is to briefly describe the basic settings of microsimulation computer programs. The AIMSUN computer program will be described in details and the process of making the AIMSUN (Advanced Interactive Microscopic Simulator for Urban and Non-Urban Networks) microsimulation model for roundabout in Radnicka Street in Rijeka will be shown. Since the traffic simulation computer programs are commonly used to estimate characteristics of traffic flow, such as the average travel time, the capacity of roads or the average speed of vehicles, models are based on adjusting the input parameters in order to reach a reasonable matching between model results and field data. This process is called calibration, and it is necessary in proving the authenticity of the simulation model and model validation. For the calibration and validation of the planned simulation model in this case, the obtained results were compared with those obtained by another analytical capacity method (Austrian method) and the SIDRA INTERSECTION (Signalized & Unsignalized Intersection Design and Research Aid) computer program which is based on the Australian calculation method for roundabout capacities.

In this work, a literature survey on the validation of vehicle dynamics simulation models is presented. Estimating the dynamic responses of existing or proposed vehicles has a wide array of applications in the development of vehicle technologies, e.g. active suspensions, controller design, driver assistance systems, etc. Although simulation environments, measurement tools and mathematical theories on vehicle dynamics are well established, the methodical link between the experimental test data and validity analysis of the simulation model is still lacking. This report presents different views on the definition of validation, and its usage in vehicle dynamics simulation models.

This paper investigates the general problem of comparing multidimensional simulation output with a given data set (e.g., real-world historical data). This problem frequently arises in verification, validation, and calibration of simulation models with spatial output statistics as in weather/climate, epidemic, swarm/crowd, social systems, communication networks, and many other applications where the simulation output is distributed across various locations or geographical regions. In the case of univariate simulation output, two-sample statistical hypothesis tests such as the t -test are commonly used. For simulation models with multidimensional and spatial output statistics, the Hotelling’s two-sample test is widely used as the benchmark method in the simulation literature. However, the Hotelling’s test assumes that the two samples come from multivariate Gaussian distributions with equal covariance matrices, which may not be the case in many applications. To address this gap, this paper proposes a double-bootstrap method based on the Wasserstein distance for comparing two multidimensional samples. Unlike the Hotelling’s test and other parametric approaches, the proposed method does not require restrictive distributional assumptions, enabling a wider range of applications and contributing to verification, validation, and calibration of simulation models with multidimensional output. Computational experiments are performed to assess the test power, and the results indicate that the proposed method outperforms the Hotelling’s test and various other approaches. The proposed method’s applicability is illustrated through two examples related to random walk of swarm particles on a two-dimensional space and a realistic engineering application involving simulation of unmanned aerial vehicle (UAV) communication systems.

Network-based computer simulation models are powerful tools for analyzing and guiding policy formation related to the actual systems being modeled. However, the inherent data and computationally intensive nature of this model class gives rise to fundamental challenges when it comes to executing typical experimental designs. In particular this applies to model validation. Manual management of the complex simulation work-flows along with the associated data will often require a broad combination of skills and expertise. Examples of skills include domain expertise, mathematical modeling, programming, high-performance computing, statistical designs, data management as well as the tracking all assets and instances involved. This is a complex and error-prone process for the best of practices, and even small slips may compromise model validation and reduce human productivity in significant ways. In this paper, we present a novel framework that addresses the challenges of model validation just mentioned. The components of our framework form an ecosystem consisting of (i) model unification through a standardized model configuration format, (ii) simulation data management, (iii) support for experimental designs, and (iv) methods for uncertainty quantification, and sensitivity analysis, all ultimately supporting the process of model validation. (Note that our view of validation is much more comprehensive than simply ensuring that the computational model can reproduce instance of historical data.) This is an extensible design where domain experts from e.g. experimental design can contribute to the collection of available algorithms and methods. Additionally, our solution directly supports reproducible computational experiments and analysis, which in turn facilitates independent model verification and validation. Finally, to showcase our design concept, we provide a sensitivity analysis for examining the consequences of different intervention strategies for an influenza pandemic.

Microscopic traffic simulation models have been playing an important role in the evaluation of transportation engineering and planning practices for the past few decades, particularly in cases in which field implementation is difficult or expensive to conduct. To achieve high fidelity and credibility for a traffic simulation model, model calibration and validation are of utmost importance. Most calibration efforts reported in the literature have focused on the informal practice, and they have seldom proposed a systematic procedure or guideline for the calibration and validation of simulation models. This paper proposes a procedure for microscopic simulation model calibration. The validity of the proposed procedure was demonstrated by use of a case study of an actuated signalized intersection by using a widely used microscopic traffic simulation model, Verkehr in Staedten Simulation (VISSIM). The simulation results were compared with multiple days of field data to determine the performance of the calibrate...