On the application of wavelet methods to simulation aerodynamic modeling

This paper gives the history of verification and validation of discrete-event simulation models as seen through the eyes of its authors and their experiences. The history is divided into three time periods: the early era covering years up to 1970, the awareness era covering the years of the 1970s and 1980s, and the modern era covering the years of 1990 to the present.

Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation

This paper investigates the general problem of comparing multidimensional simulation output with a given data set (e.g., real-world historical data). This problem frequently arises in verification, validation, and calibration of simulation models with spatial output statistics as in weather/climate, epidemic, swarm/crowd, social systems, communication networks, and many other applications where the simulation output is distributed across various locations or geographical regions. In the case of univariate simulation output, two-sample statistical hypothesis tests such as the t -test are commonly used. For simulation models with multidimensional and spatial output statistics, the Hotelling’s two-sample test is widely used as the benchmark method in the simulation literature. However, the Hotelling’s test assumes that the two samples come from multivariate Gaussian distributions with equal covariance matrices, which may not be the case in many applications. To address this gap, this paper proposes a double-bootstrap method based on the Wasserstein distance for comparing two multidimensional samples. Unlike the Hotelling’s test and other parametric approaches, the proposed method does not require restrictive distributional assumptions, enabling a wider range of applications and contributing to verification, validation, and calibration of simulation models with multidimensional output. Computational experiments are performed to assess the test power, and the results indicate that the proposed method outperforms the Hotelling’s test and various other approaches. The proposed method’s applicability is illustrated through two examples related to random walk of swarm particles on a two-dimensional space and a realistic engineering application involving simulation of unmanned aerial vehicle (UAV) communication systems.

In this expository paper we give a general introduction to verification and validation of simulation models, define the various validation techniques, and present a recommended model validation procedure.

In this paper we discuss validation and verification of simulation models. Four different approaches to deciding model validity are described; two different paradigms that relate validation and verification to the model development process are presented; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are discussed; a way to document results is given; a recommended procedure for model validation is presented; and accreditation is briefly discussed.

The paper presents a new game theoretic approach towards the validation of discrete event air combat simulation models. In the approach, statistical techniques are applied for estimating game models based on simulation data. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in air combat. The approach enhances existing methods for the validation of discrete event simulation models by incorporating the inherent game setting of air combat into the analysis. The estimated games also provide a novel game theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game theoretic approach is illustrated by analyzing simulation data produced with an existing air combat simulation model.

The roles of conceptual modelling in improving construction simulation studies: A comprehensive review

In this paper we discuss verification and validation of simulation models. The different approaches to deciding model validity are described, two different paradigms that relate verification and validation to the model development process are presented, the use of graphical data statistical references for operational validity is discussed, and a recommended procedure for model validation is given.

The permanently increasing performance of information and communication technologies enables the development and effective application of increasingly complex simulation models and their applications (M&S). As this trend will be an ongoing one, the demand for controlling and demonstrating the quality of a model and its applications by introducing adequate quality measures, techniques, and tools is obvious. Consequently, the introduction of accepted or even standardized verification and validation processes is an urgent requirement in this context. This paper will present an Extended M&S Verification and Validation (EV&V) framework based on multi-phase M&S-design, -development and –application processes. The basics of EV&V, related processes, and products will be discussed. In addition, we will show how this EV&V framework can be implemented and tailored according to predefined credibility or risk criteria, and under specified cost, time and application constraints.KeywordsModelling and Simulation (M&S)Verification and Validation (V&V)V&V FrameworkExtended V&V Triangle (EV&V)

This paper presents a new game-theoretic approach toward the validation of discrete-event air combat (AC) simulation models and simulation-based optimization. In this approach, statistical techniques are applied for estimating games based on data produced by a simulation model. The estimation procedure is presented in cases involving games with both discrete and continuous decision variables. The validity of the simulation model is assessed by comparing the properties of the estimated games to actual practices in AC. These games are also applied for simulation-based optimization in a two-sided setting in which the action of the opponent is taken into account. In optimization, the estimated games enable the study of effectiveness of AC tactics as well as aircraft, weapons, and avionics configurations. The game-theoretic approach enhances existing methods for the validation of discrete-event simulation models and techniques for simulation-based optimization by incorporating the inherent game setting of AC into the analysis. It also provides a novel game-theoretic perspective to simulation metamodeling which is used to facilitate simulation analysis. The utilization of the game-theoretic approach is illustrated by analyzing simulation data obtained with an existing AC simulation model.

For the verification and validation of microscopic simulation models of pedestrian flow, we have performed experiments for different kind of facilities and sites where most conflicts and congestion happens e.g. corridors, narrow passages, and crosswalks. The validity of the model should compare the experimental conditions and simulation results with video recording carried out in the same condition like in real life e.g. pedestrian flux and density distributions. The strategy in this technique is to achieve a certain amount of accuracy required in the simulation model. This method is good at detecting the critical points in the pedestrians walking areas. For the calibration of suitable models we use the results obtained from analysing the video recordings in Hajj 2009 and these results can be used to check the design sections of pedestrian facilities and exits. As practical examples, we present the simulation of pilgrim streams on the Jamarat bridge (see Figure 5). The objectives of this study are twofold: first, to show through verification and validation that simulation tools can be used to reproduce realistic scenarios, and second, gather data for accurate predictions for designers and decision makers.

The purpose of this Software Platform Evaluation (SPE) is to document the top-level evaluation of potential software platforms on which to construct a simulation model that satisfies the requirements for a Verifiable Fuel Cycle Simulation Model (VISION) of the Advanced Fuel Cycle (AFC). See the Software Requirements Specification for Verifiable Fuel Cycle Simulation (VISION) Model (INEEL/EXT-05-02643, Rev. 0) for a discussion of the objective and scope of the VISION model. VISION is intended to serve as a broad systems analysis and study tool applicable to work conducted as part of the AFCI (including costs estimates) and Generation IV reactor development studies. This document will serve as a guide for selecting the most appropriate software platform for VISION. This is a “living document” that will be modified over the course of the execution of this work.

Statistical method, and in particular, simulation is the most widely used operations research modelling approach now employed by industry and government. Validation of a simulation model is the procedure designed to determine whether the model is an accurate representation of the portion of reality under consideration. It is a process distinct from model building or debugging (although interrelated). In this work, strategies and practical numerical techniques for validating large-scale production simulation models (or any other large system) are detailed. An evolution of the central concepts of large-scale model validation is represented first. The distinction between model user's risk and model builder's risk is highlighted as an important development which alters the problem into a consideration of the basis a decision-maker should use in determining whether to rely on the outputs of such models. Two quantitative techniques which are applications of the Turing test and of mathematical programming are de...

A new method for determining the causes of discrepancies between dynamic simulation models and measured data is described. Whilst principally aimed at the validation of building thermal simulation codes, the method may rind application in the validation of simulation models from other disciplines. The new method relies on generating a model of the discrepancies in terms of the variables driving the physical reality which produced the experimental data and the corresponding simulation. Inspection of the resulting error model allows the principal contributors to simulation error to be identified, its dynamic nature to be characterised, and the likely cause of the error to be identified. The method is tested by generating a 'quasi-truth' dataset using the building thermal simulation code SERI-RES. A series of simulations is then prepared with perturbations to selected input parameters. The new tool is found to be capable of recovering details of the perturbations. Finally, the power of the new technique is demonstrated in a series of comparisons between the predictions of the model SERI-RES and data collected in outdoor test rooms. These comparisons reveal that the principal source of error in the predictions is the treatment of the interactions between the heater and the air and fabric of the room. The new technique proves sufficiently sensitive to detect the changing structure of the prediction errors as the position of the heater within the room is changed.

Verification and validation of simulation models are critical steps in engineering. This paper aims at verifying the suitability of reduced order aerodynamic models used in an aeroservoelastic framework designed to analyze the flight dynamics of flexible aircraft, known as the Cranfield Accelerated Aircraft Loads Model. This framework is designed for rapid assessment of aircraft configurations at the conceptual design stage. Therefore, it utilizes or relies on methods that are of relatively low fidelity for high computational speeds, such as modified strip theory coupled with Leishmann–Beddoes unsteady aerodynamic model. Hence, verification against higher order methods is required. Although low fidelity models are widely used for conceptual design and loads assessments, the open literature still lacks a comparison against higher fidelity models. This work focuses on steady-trimmed flight conditions and investigates the effect of aerodynamic wing deformation under such loads on aerodynamic performance. Key limitations of the reduced order models used, namely fuselage and interference effects, are discussed. The reasons for the overall agreement between the two approaches are also outlined.