Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation

This paper gives the history of verification and validation of discrete-event simulation models as seen through the eyes of its authors and their experiences. The history is divided into three time periods: the early era covering years up to 1970, the awareness era covering the years of the 1970s and 1980s, and the modern era covering the years of 1990 to the present.

The paper presents a new game theoretic approach towards the validation of discrete event air combat simulation models. In the approach, statistical techniques are applied for estimating game models based on simulation data. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in air combat. The approach enhances existing methods for the validation of discrete event simulation models by incorporating the inherent game setting of air combat into the analysis. The estimated games also provide a novel game theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game theoretic approach is illustrated by analyzing simulation data produced with an existing air combat simulation model.

This paper investigates the general problem of comparing multidimensional simulation output with a given data set (e.g., real-world historical data). This problem frequently arises in verification, validation, and calibration of simulation models with spatial output statistics as in weather/climate, epidemic, swarm/crowd, social systems, communication networks, and many other applications where the simulation output is distributed across various locations or geographical regions. In the case of univariate simulation output, two-sample statistical hypothesis tests such as the t -test are commonly used. For simulation models with multidimensional and spatial output statistics, the Hotelling’s two-sample test is widely used as the benchmark method in the simulation literature. However, the Hotelling’s test assumes that the two samples come from multivariate Gaussian distributions with equal covariance matrices, which may not be the case in many applications. To address this gap, this paper proposes a double-bootstrap method based on the Wasserstein distance for comparing two multidimensional samples. Unlike the Hotelling’s test and other parametric approaches, the proposed method does not require restrictive distributional assumptions, enabling a wider range of applications and contributing to verification, validation, and calibration of simulation models with multidimensional output. Computational experiments are performed to assess the test power, and the results indicate that the proposed method outperforms the Hotelling’s test and various other approaches. The proposed method’s applicability is illustrated through two examples related to random walk of swarm particles on a two-dimensional space and a realistic engineering application involving simulation of unmanned aerial vehicle (UAV) communication systems.

For the verification and validation of microscopic simulation models of pedestrian flow, we have performed experiments for different kind of facilities and sites where most conflicts and congestion happens e.g. corridors, narrow passages, and crosswalks. The validity of the model should compare the experimental conditions and simulation results with video recording carried out in the same condition like in real life e.g. pedestrian flux and density distributions. The strategy in this technique is to achieve a certain amount of accuracy required in the simulation model. This method is good at detecting the critical points in the pedestrians walking areas. For the calibration of suitable models we use the results obtained from analysing the video recordings in Hajj 2009 and these results can be used to check the design sections of pedestrian facilities and exits. As practical examples, we present the simulation of pilgrim streams on the Jamarat bridge (see Figure 5). The objectives of this study are twofold: first, to show through verification and validation that simulation tools can be used to reproduce realistic scenarios, and second, gather data for accurate predictions for designers and decision makers.

In this paper we discuss validation and verification of simulation models. Four different approaches to deciding model validity are described; two different paradigms that relate validation and verification to the model development process are presented; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are discussed; a way to document results is given; a recommended procedure for model validation is presented; and accreditation is briefly discussed.

Deploying wireless sensor networks (WSN) along a barrier line to provide surveillance against illegal intruders is a fundamental sensor-allocation problem. To maximize the detection probability of intruders with a limited number of sensors, we propose an integer non-linear program optimization model which considers multiple types of sensors and targets, probabilistic detection functions and sensor-reliability issues. An agent-based simulation (ABS) model is used to validate the analytic results and evaluate the performance of the WSN under more realistic conditions, such as intruders moving along random paths. Our experiment shows that the results from the optimization model are consistent with the results from the ABS model. This increases our confidence in the ABS model and allows us to conduct a further experiment using moving intruders, which is more realistic, but it is challenging to find an analytic solution. This experiment shows the complementary benefits of using optimization and ABS models.

In this expository paper we give a general introduction to verification and validation of simulation models, define the various validation techniques, and present a recommended model validation procedure.

The roles of conceptual modelling in improving construction simulation studies: A comprehensive review

There is little research concerning comparisons and combination of System Dynamics Simulation (SDS) and Agent Based Simulation (ABS). ABS is a paradigm used in many levels of abstraction, including those levels covered by SDS. We believe that the establishment of frameworks for the choice between these two simulation approaches would contribute to the simulation research. Hence, our work aims for the establishment of directions for the choice between SDS and ABS approaches for immune system-related problems. Previously, we compared the use of ABS and SDS for modelling agents' behaviourin an environment with no movement or interactions between these agents. We concluded that for these types of agents it is preferable to use SDS, as it takes up less computational resources and produces the same results as those obtained by the ABS model. In order to move this research forward, our next research question is: if we introduce interactions between these agents will SDS still be the most appropriate paradigm to be used? To answer this question for immune system simulation problems, we will use, as case studies, models involving interactions between tumour cells and immune effector cells. Experiments show that there are cases where SDS and ABS can not be used interchangeably, and therefore, their comparison is not straightforward.

Performance evaluation of an agent-based occupancy simulation model

In this paper we discuss verification and validation of simulation models. The different approaches to deciding model validity are described, two different paradigms that relate verification and validation to the model development process are presented, the use of graphical data statistical references for operational validity is discussed, and a recommended procedure for model validation is given.

This paper presents a new game-theoretic approach toward the validation of discrete-event air combat (AC) simulation models and simulation-based optimization. In this approach, statistical techniques are applied for estimating games based on data produced by a simulation model. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in AC. These games are also applied for simulation-based optimization in a two-sided setting in which the action of the opponent is taken into account. In optimization, the estimated games enable the study of effectiveness of AC tactics as well as aircraft, weapons, and avionics configurations. The game-theoretic approach enhances existing methods for the validation of discrete-event simulation models and techniques for simulation-based optimization by incorporating the inherent game setting of AC into the analysis. It also provides a novel game-theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game-theoretic approach is illustrated by analyzing simulation data obtained with an existing AC simulation model.

High fidelity aircraft simulations can dramatically reduce training and proficiency-related expenses, and can also reduce overall design and development costs by revealing problems early in the design process. However, developing, updating, and verifying simulation models is currently very expensive. Tremendous savings could result from innovative software tools that provide an objective and somewhat automated process for developing and verifying simulation aerodynamic models. The mathematical properties of wavelets appear to be a good match for the properties needed in a software tool capable of automatically developing and conducting verification and validation of simulation models. The primary reason for selecting wavelets over other methods is that wavelets are capable of performing localized analysis of a signal. Wavelets capture details in a signal at different resolutions at the same time. This paper addresses the feasibility of using wavelets to quickly and consistently detect irregularities in a large set of aerodynamic data. Two examples are presented. For demonstrative purposes, the cubic B-spline wavelet was used to smooth a noisy signal while obtaining 95% compression and retaining desired properties in the signal. Additionally, the linear wavelet was used to detect trends and irregularities in wind tunnel data consisting of seventeen separate data sets.

The permanently increasing performance of information and communication technologies enables the development and effective application of increasingly complex simulation models and their applications (M&S). As this trend will be an ongoing one, the demand for controlling and demonstrating the quality of a model and its applications by introducing adequate quality measures, techniques, and tools is obvious. Consequently, the introduction of accepted or even standardized verification and validation processes is an urgent requirement in this context. This paper will present an Extended M&S Verification and Validation (EV&V) framework based on multi-phase M&S-design, -development and –application processes. The basics of EV&V, related processes, and products will be discussed. In addition, we will show how this EV&V framework can be implemented and tailored according to predefined credibility or risk criteria, and under specified cost, time and application constraints.KeywordsModelling and Simulation (M&S)Verification and Validation (V&V)V&V FrameworkExtended V&V Triangle (EV&V)

An important challenge within the field of Criminology is to investigate the spatio-temporal dynamics of crime. Typical questions in this area are how the behaviour of offenders, targets, and guardians, and the emergence and displacement of criminal hot spots can be predicted. This paper presents an agent-based simulation model that can be used as an experimental tool to address such questions. The simulation model particularly focuses on the interplay between hot spots and reputation. Using the model, a number of simulation experiments have been performed, of which results have been analysed using formal techniques. The results indicate that the presented approach is able to adequately reproduce displacement patterns as described in the literature.