Pedestrian Flow Simulation Validation and Verification Techniques

Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation

This paper gives the history of verification and validation of discrete-event simulation models as seen through the eyes of its authors and their experiences. The history is divided into three time periods: the early era covering years up to 1970, the awareness era covering the years of the 1970s and 1980s, and the modern era covering the years of 1990 to the present.

In this paper we discuss validation and verification of simulation models. Four different approaches to deciding model validity are described; two different paradigms that relate validation and verification to the model development process are presented; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are discussed; a way to document results is given; a recommended procedure for model validation is presented; and accreditation is briefly discussed.

This paper investigates the general problem of comparing multidimensional simulation output with a given data set (e.g., real-world historical data). This problem frequently arises in verification, validation, and calibration of simulation models with spatial output statistics as in weather/climate, epidemic, swarm/crowd, social systems, communication networks, and many other applications where the simulation output is distributed across various locations or geographical regions. In the case of univariate simulation output, two-sample statistical hypothesis tests such as the t -test are commonly used. For simulation models with multidimensional and spatial output statistics, the Hotelling’s two-sample test is widely used as the benchmark method in the simulation literature. However, the Hotelling’s test assumes that the two samples come from multivariate Gaussian distributions with equal covariance matrices, which may not be the case in many applications. To address this gap, this paper proposes a double-bootstrap method based on the Wasserstein distance for comparing two multidimensional samples. Unlike the Hotelling’s test and other parametric approaches, the proposed method does not require restrictive distributional assumptions, enabling a wider range of applications and contributing to verification, validation, and calibration of simulation models with multidimensional output. Computational experiments are performed to assess the test power, and the results indicate that the proposed method outperforms the Hotelling’s test and various other approaches. The proposed method’s applicability is illustrated through two examples related to random walk of swarm particles on a two-dimensional space and a realistic engineering application involving simulation of unmanned aerial vehicle (UAV) communication systems.

In this expository paper we give a general introduction to verification and validation of simulation models, define the various validation techniques, and present a recommended model validation procedure.

The paper presents a new game theoretic approach towards the validation of discrete event air combat simulation models. In the approach, statistical techniques are applied for estimating game models based on simulation data. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in air combat. The approach enhances existing methods for the validation of discrete event simulation models by incorporating the inherent game setting of air combat into the analysis. The estimated games also provide a novel game theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game theoretic approach is illustrated by analyzing simulation data produced with an existing air combat simulation model.

In this paper we discuss verification and validation of simulation models. The different approaches to deciding model validity are described, two different paradigms that relate verification and validation to the model development process are presented, the use of graphical data statistical references for operational validity is discussed, and a recommended procedure for model validation is given.

The roles of conceptual modelling in improving construction simulation studies: A comprehensive review

High fidelity aircraft simulations can dramatically reduce training and proficiency-related expenses, and can also reduce overall design and development costs by revealing problems early in the design process. However, developing, updating, and verifying simulation models is currently very expensive. Tremendous savings could result from innovative software tools that provide an objective and somewhat automated process for developing and verifying simulation aerodynamic models. The mathematical properties of wavelets appear to be a good match for the properties needed in a software tool capable of automatically developing and conducting verification and validation of simulation models. The primary reason for selecting wavelets over other methods is that wavelets are capable of performing localized analysis of a signal. Wavelets capture details in a signal at different resolutions at the same time. This paper addresses the feasibility of using wavelets to quickly and consistently detect irregularities in a large set of aerodynamic data. Two examples are presented. For demonstrative purposes, the cubic B-spline wavelet was used to smooth a noisy signal while obtaining 95% compression and retaining desired properties in the signal. Additionally, the linear wavelet was used to detect trends and irregularities in wind tunnel data consisting of seventeen separate data sets.

The permanently increasing performance of information and communication technologies enables the development and effective application of increasingly complex simulation models and their applications (M&S). As this trend will be an ongoing one, the demand for controlling and demonstrating the quality of a model and its applications by introducing adequate quality measures, techniques, and tools is obvious. Consequently, the introduction of accepted or even standardized verification and validation processes is an urgent requirement in this context. This paper will present an Extended M&S Verification and Validation (EV&V) framework based on multi-phase M&S-design, -development and –application processes. The basics of EV&V, related processes, and products will be discussed. In addition, we will show how this EV&V framework can be implemented and tailored according to predefined credibility or risk criteria, and under specified cost, time and application constraints.KeywordsModelling and Simulation (M&S)Verification and Validation (V&V)V&V FrameworkExtended V&V Triangle (EV&V)

Network-based computer simulation models are powerful tools for analyzing and guiding policy formation related to the actual systems being modeled. However, the inherent data and computationally intensive nature of this model class gives rise to fundamental challenges when it comes to executing typical experimental designs. In particular this applies to model validation. Manual management of the complex simulation work-flows along with the associated data will often require a broad combination of skills and expertise. Examples of skills include domain expertise, mathematical modeling, programming, high-performance computing, statistical designs, data management as well as the tracking all assets and instances involved. This is a complex and error-prone process for the best of practices, and even small slips may compromise model validation and reduce human productivity in significant ways. In this paper, we present a novel framework that addresses the challenges of model validation just mentioned. The components of our framework form an ecosystem consisting of (i) model unification through a standardized model configuration format, (ii) simulation data management, (iii) support for experimental designs, and (iv) methods for uncertainty quantification, and sensitivity analysis, all ultimately supporting the process of model validation. (Note that our view of validation is much more comprehensive than simply ensuring that the computational model can reproduce instance of historical data.) This is an extensible design where domain experts from e.g. experimental design can contribute to the collection of available algorithms and methods. Additionally, our solution directly supports reproducible computational experiments and analysis, which in turn facilitates independent model verification and validation. Finally, to showcase our design concept, we provide a sensitivity analysis for examining the consequences of different intervention strategies for an influenza pandemic.

A new method for determining the causes of discrepancies between dynamic simulation models and measured data is described. Whilst principally aimed at the validation of building thermal simulation codes, the method may rind application in the validation of simulation models from other disciplines. The new method relies on generating a model of the discrepancies in terms of the variables driving the physical reality which produced the experimental data and the corresponding simulation. Inspection of the resulting error model allows the principal contributors to simulation error to be identified, its dynamic nature to be characterised, and the likely cause of the error to be identified. The method is tested by generating a 'quasi-truth' dataset using the building thermal simulation code SERI-RES. A series of simulations is then prepared with perturbations to selected input parameters. The new tool is found to be capable of recovering details of the perturbations. Finally, the power of the new technique is demonstrated in a series of comparisons between the predictions of the model SERI-RES and data collected in outdoor test rooms. These comparisons reveal that the principal source of error in the predictions is the treatment of the interactions between the heater and the air and fabric of the room. The new technique proves sufficiently sensitive to detect the changing structure of the prediction errors as the position of the heater within the room is changed.

This paper presents a new game-theoretic approach toward the validation of discrete-event air combat (AC) simulation models and simulation-based optimization. In this approach, statistical techniques are applied for estimating games based on data produced by a simulation model. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in AC. These games are also applied for simulation-based optimization in a two-sided setting in which the action of the opponent is taken into account. In optimization, the estimated games enable the study of effectiveness of AC tactics as well as aircraft, weapons, and avionics configurations. The game-theoretic approach enhances existing methods for the validation of discrete-event simulation models and techniques for simulation-based optimization by incorporating the inherent game setting of AC into the analysis. It also provides a novel game-theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game-theoretic approach is illustrated by analyzing simulation data obtained with an existing AC simulation model.

Verification and validation of simulation models are critical steps in engineering. This paper aims at verifying the suitability of reduced order aerodynamic models used in an aeroservoelastic framework designed to analyze the flight dynamics of flexible aircraft, known as the Cranfield Accelerated Aircraft Loads Model. This framework is designed for rapid assessment of aircraft configurations at the conceptual design stage. Therefore, it utilizes or relies on methods that are of relatively low fidelity for high computational speeds, such as modified strip theory coupled with Leishmann–Beddoes unsteady aerodynamic model. Hence, verification against higher order methods is required. Although low fidelity models are widely used for conceptual design and loads assessments, the open literature still lacks a comparison against higher fidelity models. This work focuses on steady-trimmed flight conditions and investigates the effect of aerodynamic wing deformation under such loads on aerodynamic performance. Key limitations of the reduced order models used, namely fuselage and interference effects, are discussed. The reasons for the overall agreement between the two approaches are also outlined.

Several process modeling techniques have been used in simulation projects. However, most of these techniques provide little specific support to the programming. The main cause of this is the fact that these techniques were not developed with the same logic used in simulation models. From this issue, this paper presents an industrial application of a new conceptual modeling technique, named IDEF-SIM (Integrated Definition Methods -- Simulation) currently under development by the authors. This adapted IDEF uses logic elements present in techniques such as IDEF0 and IDEF3, but in a way that is similar to the process interpretation logic usually used in simulation projects. This way, it can be noticed an increase in the conceptual model's utility, which might facilitate the simulation model programming, verification and validation and the scenarios creation. Additionally, the paper presents the benefits of using IDEF-SIM to create the conceptual model of a Brazilian tech company manufacturing cell.