Techniques for ascertaining the validity of large-scale production simulation models

The paper presents a new game theoretic approach towards the validation of discrete event air combat simulation models. In the approach, statistical techniques are applied for estimating game models based on simulation data. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in air combat. The approach enhances existing methods for the validation of discrete event simulation models by incorporating the inherent game setting of air combat into the analysis. The estimated games also provide a novel game theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game theoretic approach is illustrated by analyzing simulation data produced with an existing air combat simulation model.

This paper presents a new game-theoretic approach toward the validation of discrete-event air combat (AC) simulation models and simulation-based optimization. In this approach, statistical techniques are applied for estimating games based on data produced by a simulation model. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in AC. These games are also applied for simulation-based optimization in a two-sided setting in which the action of the opponent is taken into account. In optimization, the estimated games enable the study of effectiveness of AC tactics as well as aircraft, weapons, and avionics configurations. The game-theoretic approach enhances existing methods for the validation of discrete-event simulation models and techniques for simulation-based optimization by incorporating the inherent game setting of AC into the analysis. It also provides a novel game-theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game-theoretic approach is illustrated by analyzing simulation data obtained with an existing AC simulation model.

A new method for determining the causes of discrepancies between dynamic simulation models and measured data is described. Whilst principally aimed at the validation of building thermal simulation codes, the method may rind application in the validation of simulation models from other disciplines. The new method relies on generating a model of the discrepancies in terms of the variables driving the physical reality which produced the experimental data and the corresponding simulation. Inspection of the resulting error model allows the principal contributors to simulation error to be identified, its dynamic nature to be characterised, and the likely cause of the error to be identified. The method is tested by generating a 'quasi-truth' dataset using the building thermal simulation code SERI-RES. A series of simulations is then prepared with perturbations to selected input parameters. The new tool is found to be capable of recovering details of the perturbations. Finally, the power of the new technique is demonstrated in a series of comparisons between the predictions of the model SERI-RES and data collected in outdoor test rooms. These comparisons reveal that the principal source of error in the predictions is the treatment of the interactions between the heater and the air and fabric of the room. The new technique proves sufficiently sensitive to detect the changing structure of the prediction errors as the position of the heater within the room is changed.

This paper gives the history of verification and validation of discrete-event simulation models as seen through the eyes of its authors and their experiences. The history is divided into three time periods: the early era covering years up to 1970, the awareness era covering the years of the 1970s and 1980s, and the modern era covering the years of 1990 to the present.

In this tutorial we present techniques for building valid and credible simulation models. Ideas to be discussed include the importance of a definitive problem formulation, discussions with subject-matter experts, interacting with the decision-maker on a regular basis, development of a written assumptions document, structured walk-through of the assumptions document, use of sensitivity analysis to determine important model factors, and comparison of model and system output data for an existing system (if any). Each idea will be illustrated by one or more real-world examples. We will also discuss the difficulty in using formal statistical techniques (e.g., confidence intervals) to validate simulation models.

The evaluation and validation of simulation models are important stages of model development. The aim of this work was to validate the Buttercup model (BCM) with experimental data from on-farm trials. Two on-farm experiments (Experiment 1 E1 and Experiment 2 E2) were carried out to evaluate the effect of supplementing dairy cattle with different sources of protein in Small Scale Dairy Farming Systems. Three concentrates (C1= maize grain 610, soybean meal 310, urea 30, and cane molasses 50g kg−1 DM; C2= maize grain 630, soybean meal 320 and cane molasses 50g kg−1 DM; and C3= maize grain 900, urea 50 and cane molasses 50g kg−1 DM) were developed using the BCM. The effect of the concentrates on milk yield, body condition score and live weight was evaluated using a double Latin Square experimental design. In E1 there were no significant differences (P> 0.05) for the variables evaluated. In E2, significant differences (P<0.05) were observed in milk fat content, for C2 (33.8g kg−1) was higher than C1 and C3 (26.2 and 26.0g kg−1). The results suggest that it was possible to fulfil the metabolisable protein requirements of the cows in these systems with low cost sources of non-protein nitrogen like the urea used in C3; and that the synthesis of microbial protein (from non-protein nitrogen and fermentable metabolisable energy of the concentrates) was enough to fulfil the cows’ requirements of MP for the milk yield levels observed in both experiments. The results also indicate that when specific input data set from the experiments was used the model predictions are acceptable, when comparing predicted vs. observed data as a very significant correlation was observed (P<0.01).

Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation

This paper presents an evaluation on passenger vehicle's ride and handling performance when converted into an electric vehicle (EV). The evaluations were done using a validated 14 degrees of freedom ride and handling model. The mathematical modelling of vehicle's ride and handling model as well as its validations are described. Two types of experiments were performed to validate the developed simulation model; the ride test and the handling test. The validated simulation model was used to evaluate the vehicle's ride and handling performance of the vehicle when converted into an electric vehicle. The evaluation involves two weight distribution ratios which are 60:40, for normal vehicle and 40:60 for EV conversion. The validated simulation model used active suspension system in order to improve the EV conversion's ride and handling performance. It is found that modification into EV affects vehicle's handling performance quite significant but not ride performance. The EV conversion's weight, which is distributed towards the rear of the vehicle, causes the vehicle to travel off from its original travelling path. The application of active suspension system is proposed to improve EV conversion's handling performance as well as its ride comfort performance.

Limited stockpiles of vaccine and antiviral drugs and other resources pose a formidable healthcare delivery challenge for an impending human-to-human transmittable influenza pandemic. The existing preparedness plans by the Center for Disease Control and Health and Human Services strongly underscore the need for efficient mitigation strategies. Such a strategy entails decisions for early response, vaccination, prophylaxis, hospitalization and quarantine enforcement. This paper presents a large-scale simulation model that mimics stochastic propagation of an influenza pandemic controlled by mitigation strategies. The impact of a pandemic is assessed via measures including total numbers of infected, dead, denied hospital admission and denied vaccine/antiviral drugs, and also through an aggregate cost measure incorporating healthcare cost and lost wages. The model considers numerous demographic and community features, daily human activities, vaccination, prophylaxis, hospitalization, social distancing, and hourly accounting of infection spread. The simulation model can serve as the foundation for developing dynamic mitigation strategies. The simulation model is tested on a hypothetical community with over 1100 000 people. A designed experiment is conducted to examine the statistical significance of a number of model parameters. The experimental outcomes can be used in developing guidelines for strategic use of limited resources by healthcare decision makers. Finally, a Markov decision process model and its simulation-based reinforcement learning framework for developing mitigation strategies are presented. The simulation-based framework is quite comprehensive and general, and can be particularized to other types of infectious disease outbreaks.

This paper describes application of Project INTERSECT, a next generation highly scalable reservoir simulator on real large scale field models. High resolution reservoir simulation is required to better define and describe fluid flow and enable improved field development and tactical operational planning. Massively parallel computing techniques overcome limitations of problem size and space resolution. This paper demonstrates that large-scale simulation models can be performed on commodity hardware taking advantage of evolution in multicpu hardware architecture and software engineering. This allows both geologists and reservoir engineers to include more realistic geologic and engineering detail for better and more reliable production optimization. Intense computer simulation is essential for effective reservoir management. The advances in reservoir characterization techniques and the industry drive towards the ‘smart oilfield’ with rapid model updates will require more efficient model processing to achieve timely field operational decisions. Parallel reservoir simulators have the potential to solve larger, more realistic problems than previously possible. The size and application of reservoir simulation problems have been limited by the availability of computing hardware, reservoir simulation architecture and of solution methods for solving large-scale heterogeneous problems. The next generation reservoir simulator demonstrates that key modeling challenges has been overcome by a software architecture and capability to model more realistic subsurface and surface models. Applications of the new reservoir simulator illustrates how typical reservoir engineering options such as local grid refinement, local grid coarsening, multilateral wells and aquifer modeling affect the overall parallel performance and scalability using highly heterogeneous large-scale models. Application of new modeling techniques highlight increased accuracy of modeling results and more reliable field development planning and reservoir management decisions.

For many biological systems, a variety of simulation models exist. A new simulation model is rarely developed from scratch, but rather revises and extends an existing one. A key challenge, however, is to decide which model might be an appropriate starting point for a particular problem and why. To answer this question, we need to identify entities and activities that contributed to the development of a simulation model. Therefore, we exploit the provenance data model, PROV-DM, of the World Wide Web Consortium and, building on previous work, continue developing a PROV ontology for simulation studies. Based on a case study of 19 Wnt/β-catenin signaling models, we identify crucial entities and activities as well as useful metadata to both capture the provenance information from individual simulation studies and relate these forming a family of models. The approach is implemented in WebProv, a web application for inserting and querying provenance information. Our specialization of PROV-DM contains the entities Research Question, Assumption, Requirement, Qualitative Model, Simulation Model, Simulation Experiment, Simulation Data, and Wet-lab Data as well as activities referring to building, calibrating, validating, and analyzing a simulation model. We show that most Wnt simulation models are connected to other Wnt models by using (parts of) these models. However, the overlap, especially regarding the Wet-lab Data used for calibration or validation of the models is small. Making these aspects of developing a model explicit and queryable is an important step for assessing and reusing simulation models more effectively. Exposing this information helps to integrate a new simulation model within a family of existing ones and may lead to the development of more robust and valid simulation models. We hope that our approach becomes part of a standardization effort and that modelers adopt the benefits of provenance when considering or creating simulation models.

In this expository paper we give a general introduction to verification and validation of simulation models, define the various validation techniques, and present a recommended model validation procedure.

Microscopic traffic simulation models have been playing an important role in the evaluation of transportation engineering and planning practices for the past few decades, particularly in cases in which field implementation is difficult or expensive to conduct. To achieve high fidelity and credibility for a traffic simulation model, model calibration and validation are of utmost importance. Most calibration efforts reported in the literature have focused on the informal practice, and they have seldom proposed a systematic procedure or guideline for the calibration and validation of simulation models. This paper proposes a procedure for microscopic simulation model calibration. The validity of the proposed procedure was demonstrated by use of a case study of an actuated signalized intersection by using a widely used microscopic traffic simulation model, Verkehr in Staedten Simulation (VISSIM). The simulation results were compared with multiple days of field data to determine the performance of the calibrate...

O objetivo deste trabalho foi demonstrar a utilidade da simulação a eventos discretos, aplicada às operações de mina subterrânea, como ferramenta de apoio à tomada de decisões. Para isso, foi desenvolvido um modelo de simulação de uma mina subterrânea na cidade de Jacobina, utilizando o software Arena. O trabalho foi desenvolvido em 03 fases, construção do modelo, validação do modelo e estudos de mudanças estratégicas sobre o sistema analisado. Após a conclusão, comparando os resultados obtidos aos resultados do sistema real, o modelo foi validado. Com o objetivo de demonstrar que a simulação é ferramenta eficaz para o auxílio à tomada de decisão, simulou-se dois novos cenários, onde foram consideradas hipóteses de mudanças estratégicas no sistema. A partir da análise dos novos cenários, foi possível comprovar que a ferramenta simulação a eventos discretos é muito eficaz para suporte à tomada de decisão em mina subterrânea.

The proper simulation of the hygrothermal behaviour of historical buildings is a challenging task with several implications regarding the evaluation of indoor thermal comfort and the suitability of retrofit strategies that comply with the conservation of cultural heritage. An inaccurate simulation may lead to inadequate conclusions, which could result in inappropriate and dangerous actions for the preservation of the heritage buildings.Then calibration and validation of hygrothermal simulation models are essential steps to achieve more accurate and reliable results.Now, although some agencies have developed guidelines and methodologies to carry out the validation of building performance models, all of them are based on energy consumption only. However, since in some buildings the energy consumption data are not always available especially when no operating heating, ventilation and air conditioning system is installed, which is the case of many historical buildings, the microclimatic parameters are usually adopted in the validation process. In this case, neither protocols nor specific parameters have been officially recognised to perform the model validation. The present work reviewed the main approaches used by researchers for building performance model validation with special reference to historical buildings based on microclimatic parameters, highlighting the main advantages and drawbacks of the different methods reviewed. Finally, recommendations to properly carry out the model validation based on microclimatic parameters have been provided. The collected information may be useful to different subjects (e.g. designers, energy auditors, researchers, conservators, buildings’ owners and policy makers) and can drive suitable and reliable retrofit and maintenance interventions.