Molecular simulation of CH4 adsorption behavior in slit nanopores: Verification of simulation methods and models

In this paper we discuss validation and verification of simulation models. Four different approaches to deciding model validity are described; two different paradigms that relate validation and verification to the model development process are presented; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are discussed; a way to document results is given; a recommended procedure for model validation is presented; and accreditation is briefly discussed.

This paper investigates the general problem of comparing multidimensional simulation output with a given data set (e.g., real-world historical data). This problem frequently arises in verification, validation, and calibration of simulation models with spatial output statistics as in weather/climate, epidemic, swarm/crowd, social systems, communication networks, and many other applications where the simulation output is distributed across various locations or geographical regions. In the case of univariate simulation output, two-sample statistical hypothesis tests such as the t -test are commonly used. For simulation models with multidimensional and spatial output statistics, the Hotelling’s two-sample test is widely used as the benchmark method in the simulation literature. However, the Hotelling’s test assumes that the two samples come from multivariate Gaussian distributions with equal covariance matrices, which may not be the case in many applications. To address this gap, this paper proposes a double-bootstrap method based on the Wasserstein distance for comparing two multidimensional samples. Unlike the Hotelling’s test and other parametric approaches, the proposed method does not require restrictive distributional assumptions, enabling a wider range of applications and contributing to verification, validation, and calibration of simulation models with multidimensional output. Computational experiments are performed to assess the test power, and the results indicate that the proposed method outperforms the Hotelling’s test and various other approaches. The proposed method’s applicability is illustrated through two examples related to random walk of swarm particles on a two-dimensional space and a realistic engineering application involving simulation of unmanned aerial vehicle (UAV) communication systems.

This paper gives the history of verification and validation of discrete-event simulation models as seen through the eyes of its authors and their experiences. The history is divided into three time periods: the early era covering years up to 1970, the awareness era covering the years of the 1970s and 1980s, and the modern era covering the years of 1990 to the present.

Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation

In this thesis, the theory and practice of an innovative coding error isolation method is enhanced and extended. Methods and tools that have been developed to address the limitations of the original approach and to extend its applicability are presented. For dynamic simulations to be credible, verification of the computer code must be an integral part of the modelling process. Simulation verification is a process that ensures simulation code (computerized model) adequately describes the conceptual (e.g. mathematical) model. It should not be confused with simulation validation, which is a process that ensures the computerized model adequately describes reality. Verification of simulation models is an important step in the model development process, but until now it has been a tedious and time-consuming, or even often overlooked, activity. Simulation coding error isolation is a very specific type of software verification. Previous research has shown that it is possible to use the mathematical properties of a model to link the erroneous behaviour of a computerized model to the coding errors involved in the implementation. A novel method developed recently, based on the concept of hack-to-hack testing, has been demonstrated to be capable of isolating coding errors in simulation models. This so-called FEAMAD (FEAture MAtrix based Debugging) method, had a number of limitations. In particular, the feature matrices, which characterise each class of error in the code, were manually generated, which is inefficient and error-prone; the time required to isolate errors increased combinatorially with the size and complexity of the model severely limiting its applicability to complex models. Furthermore, the method was subjected to limited testing in real-world applications. The research for this thesis has been undertaken to address these limitations. Feature matrices are arguably the most important element of the FEAMAD method. Feature matrices are vectors/matrices that characterise the geometric properties of coding errors. They are generated by analysing the model implementation, which determines the effect of coding errors on the model output. Previously these feature matrices were generated by visually inspecting the model implementation and manually creating the matrices. Methods to automatically generate and verify the feature matrices have been developed in this work. The methods utilize code parsing and mutation analysis tools. A prototype of the Automatic Feature MAtrix Generation (AFMAG) method has also successfully been implemented and demonstrated with wastewater treatment models. Maximising the efficiency and effectiveness of the FEAMAD method is important to enable isolation of errors in large, complex models. It has been discovered that the model equations themselves could be 'designed' to ensure that coding errors manifest themselves in a model output in such a way that isolation efficiency and effectiveness is maximised. A method has been developed that employs this idea of model 'design' to maximise isolation efficiency and effectiveness. This method significantly reduces the times required for isolating errors. The FEAMAD method was previously demonstrated with models implemented on a single platform. Replicating the demonstrated effectiveness in cross-platform situations has proven to be much more complicated. The cross-platform data transfer results in significant numerical errors, so much so that the effect of some errors may be masked. A novel solution to the problem, a double state-variable injection method, has been developed and demonstrated to be highly effective. Consequently, the FEAMAD method has been enhanced to debug models implemented in different simulation platforms. Coding error isolation, through comparing model implementations on multiple platforms, represents a major application area of the FEAMAD method. Code parsing and mutation analysis tools have also been used to support a method developed for error threshold optimisation. This method allows error threshold selection to be optimised for particular model implementations. As a whole, this research, and indeed the individual methods that have been developed, provide valuable, practical tools for those working in the modelling and simulation field.

Even after several decades of research, modeling is considered an art, with a high liability to produce incorrect abstractions of real world systems. Therefore, validation and verification of simulation models is considered an indispensable method to establish the credibility of developed models. In the process of parallelizing or distributing a given credible simulation model, a bias is introduced, possibly leading to serious errors in simulation results. Depending on the mechanisms used for parallelization or distribution, a separate validation of the parallel or distributed model is required. A necessary first step for such a validation is an understanding of the sources of bias that might occur through parallelization or distribution of a simulation model. The intention of this paper is to give an overview of the various types of bias and to give a formal definition of the bias and its quantification.

In this expository paper we give a general introduction to verification and validation of simulation models, define the various validation techniques, and present a recommended model validation procedure.

The roles of conceptual modelling in improving construction simulation studies: A comprehensive review

This paper presents the development of a simulation model for high-voltage gallium nitride (GaN) high-electron-mobility transistors (HEMT) in a cascode structure. A method is proposed to accurately extract the device package parasitic inductance, which is of vital importance to better predict the high-frequency switching performance of the device. The simulation model is verified by a double-pulse tester, and the results match well both in terms of device switching waveform and switching energy. Based on the simulation model, an investigation of the package influence on the cascode GaN HEMT is presented, and several critical parasitic inductances are identified and verified. Finally, a detailed loss breakdown is made for a buck converter, including a comparison between hard switching and soft switching. The results indicate that the switching loss is a dominant part of the total loss in hard-switching conditions in MHz high-frequency range and below 8~10A operation current, therefore soft switching is preferred to achieve high-frequency and high-efficiency operation of the high-voltage GaN HEMT.

Statistical evaluation and verification of digital simulation models

In this paper we discuss verification and validation of simulation models. The different approaches to deciding model validity are described, two different paradigms that relate verification and validation to the model development process are presented, the use of graphical data statistical references for operational validity is discussed, and a recommended procedure for model validation is given.

High fidelity aircraft simulations can dramatically reduce training and proficiency-related expenses, and can also reduce overall design and development costs by revealing problems early in the design process. However, developing, updating, and verifying simulation models is currently very expensive. Tremendous savings could result from innovative software tools that provide an objective and somewhat automated process for developing and verifying simulation aerodynamic models. The mathematical properties of wavelets appear to be a good match for the properties needed in a software tool capable of automatically developing and conducting verification and validation of simulation models. The primary reason for selecting wavelets over other methods is that wavelets are capable of performing localized analysis of a signal. Wavelets capture details in a signal at different resolutions at the same time. This paper addresses the feasibility of using wavelets to quickly and consistently detect irregularities in a large set of aerodynamic data. Two examples are presented. For demonstrative purposes, the cubic B-spline wavelet was used to smooth a noisy signal while obtaining 95% compression and retaining desired properties in the signal. Additionally, the linear wavelet was used to detect trends and irregularities in wind tunnel data consisting of seventeen separate data sets.

For the verification and validation of microscopic simulation models of pedestrian flow, we have performed experiments for different kind of facilities and sites where most conflicts and congestion happens e.g. corridors, narrow passages, and crosswalks. The validity of the model should compare the experimental conditions and simulation results with video recording carried out in the same condition like in real life e.g. pedestrian flux and density distributions. The strategy in this technique is to achieve a certain amount of accuracy required in the simulation model. This method is good at detecting the critical points in the pedestrians walking areas. For the calibration of suitable models we use the results obtained from analysing the video recordings in Hajj 2009 and these results can be used to check the design sections of pedestrian facilities and exits. As practical examples, we present the simulation of pilgrim streams on the Jamarat bridge (see Figure 5). The objectives of this study are twofold: first, to show through verification and validation that simulation tools can be used to reproduce realistic scenarios, and second, gather data for accurate predictions for designers and decision makers.

The permanently increasing performance of information and communication technologies enables the development and effective application of increasingly complex simulation models and their applications (M&S). As this trend will be an ongoing one, the demand for controlling and demonstrating the quality of a model and its applications by introducing adequate quality measures, techniques, and tools is obvious. Consequently, the introduction of accepted or even standardized verification and validation processes is an urgent requirement in this context. This paper will present an Extended M&S Verification and Validation (EV&V) framework based on multi-phase M&S-design, -development and –application processes. The basics of EV&V, related processes, and products will be discussed. In addition, we will show how this EV&V framework can be implemented and tailored according to predefined credibility or risk criteria, and under specified cost, time and application constraints.KeywordsModelling and Simulation (M&S)Verification and Validation (V&V)V&V FrameworkExtended V&V Triangle (EV&V)

SUMMARY A numerical simulation model of the roller test stand located at Munich and loaded by a bogie is discussed including its technical structure, the governing physical equations of motion and the structure of the simulation program. Both, the set up of the mathematical and numerical models time and the computation time of simulation runs have been considerably reduced (by a factor of 20) using formula manipulation programs. Simulation results concerning the starting behaviour of a bogie, stationary limit cycle behaviour of bogies with ideal and wear profile and influence of gauge changes and spring/damper modifications on limit cycle behaviour of a bogie are presented, some of which are compared with experimental results gained from test facility measurements. The simulation results are in good agreement with the experimental results and provide an experimental verification of the roller rig simulation model presented.