A Model-Driven Approach for Conducting Simulation Experiments

Goal-directed reproducible experimentation with simulation models is still a significant challenge. The underutilization of design of experiments, limited transparency in the collection and analysis of results, and ad-hoc adaptation of experiments as learning takes place continue to hamper reproducibility and hence cause a credibility gap. In this study, we propose a strategy that leverages the synergies between model-driven engineering, intelligent agent technology, and variability modeling to support the management of the lifecycle of a simulation experiment. Experiment design and workflow models are introduced for configurable experiment synthesis and execution. Feature-based variability modeling is used to design a family of experiments, which can be leveraged by ontology-driven software agents to configure, execute, and reproduce experiments. Online experiment adaptation is proposed as a strategy to facilitate dynamic experiment model updating as objectives shift from validation to variable screening, understanding, and optimization.

There is a growing trend towards migrating applications and services to the cloud. This trend has led to the emergence of different cloud service providers (CSPs), in turn leading to different cost models offered by these CSPs to lease their resources, variabilities in the granularity and specification of resources provided, and heterogeneous APIs offered by the CSPs to the users to program resource requests and deployment for their cloud-hosted services. These challenges make it hard for customers of the cloud to seamlessly transition their services to the cloud or migrate between different CSPs. To address these challenges, this paper presents a solution based on model-driven engineering (MDE). Specifically, we describe the design of the domain-specific modeling languages in our MDE framework and the associated generative mechanisms that address the challenges related to estimating performance and cost to host the services in the cloud, automated deployment and resource management.

There is an increasing awareness of the added value that efficient design of experiments can provide to simulation studies. However, many practitioners lack the proper training required to design, execute and analyze a well-designed simulation experiment. In this study, we present the initial stages of a model-driven engineering based tool for managing simulation experiments. Underlying our approach are an experiment ontology and a feature model that capture the statistical design of experiments expert knowledge which users might be lacking. In its current state, the tool provides support for the design and execution of simple experiments. Using a web-based interface, the user is guided through an experiment design wizard that produces an experiment model that can be exported as an XML file containing the experiment's description. This XML can be used to synthesize scripts that can be run by a simulator and shared across different platforms.

Valid models are central for credible simulation studies. If those models do not exist, they need to be developed. In fact, entire simulation studies are often aimed at developing valid models. Thereby, successive model refinement and execution of diverse simulation experiments are closely intertwined. Whereas software-based support for individual simulation experiments exists, the intricate interdependencies and the diversity of tasks that govern simulation studies have prevented a more comprehensive support. To achieve the required flexibility while adhering to the constraints that apply between individual tasks, we adopt a declarative, artifact-based workflow approach. Therefore, central products of these simulation studies are identified and specified as artifacts: the conceptual model (with a focus on formally defined requirements), the simulation model, and the experiment. Each artifact is characterized by stages the artifact moves through to reach certain milestones and which are guarded by conditions. Thereby, the relations and constraints between artifacts become explicit. This is instrumental to check and ensure the consistency between conceptual model and simulation model, to automatically execute simulation experiments to probe the specified requirements, and to develop plans to provide goal-directed guidance to the user. We demonstrate the approach by using it to repeat an existing simulation study.

A domain-specific language for the specification of UCON policies

Systems Biology requires increasingly complex simulation models. Effectively interpreting and building upon previous simulation results is both difficult and time consuming. Thus, simulation results often cannot be reproduced exactly; making it difficult for other modellers to validate results and take the next step in a simulation study. The Simulation Experiment Description Mark-up Language~(SED-ML), a subset of the Minimum Information About a Simulation Experiment~(MIASE) guidelines, promises to solve this problem by prescribing the form and content of the information required to reproduce simulation experiments. SED-ML is detailed enough to enable automatic rerunning of simulation experiments. Here, we present a web-based simulation-experiment repository that lets modellers develop SED-ML compliant simulation-experiment descriptions The system encourages modellers to annotate their experiments with text and images, experimental data and domain meta-information. These informal annotations aid organisation and classification of the simulations and provide rich search criteria. They complement SED-ML's formal precision to produce simulation-experiment descriptions that can be understood by both men and machines. The system combines both human-readable and formal machine-readable content, thus ensuring exact reproducibility of the simulation results of a modelling study.

Simulation experiments generate large data sets that must be converted into recommendations for decision makers. This article explores using a Natural Language Generation (NLG) approach for writing summaries of simulation experiments. The article discusses the steps required to convert simulation experiment data to text and highlights the unique aspects of data to text for simulation experiments. Automation of report generation can potentially reduce the time and cost of simulation studies and improve the reporting of results. A prototype software system was developed and applied to a simulation to illustrate the benefits of a NLG approach.

Diverse methods for complex simulation experiments can contribute to developing and gaining insights into simulation models, for example simulation-based optimization, sensitivity analysis, or statistical model-checking. An effective tool for conducting simulation experiments must be highly flexible to support such a broad range of experimental methods. Furthermore, to facilitate reproducibility and communication of simulation experiments, an effective tool for simulation experimentation must yield experiment descriptions that are easily portable, executable, and human-readable. In this tutorial we introduce SESSL, a domain-specific language for setting up simulation experiments. SESSL is flexible and extensible, and experiment descriptions are executable, often succinct, and can be executed reproducibly across machines and operating systems. Based on a few examples, we demonstrate how SESSL can be leveraged to easily conduct complex simulation experiments while reusing existing software and methods, and how SESSL’s capabilities can be extended and combined with arbitrary simulation software via bindings.

Simulation studies are intricate processes and user support for conducting more consistent, systematic, and efficient simulation studies is needed. Simulation experiments as one crucial part of a simulation study can benefit from semi-automatic method selection, parameterization, and execution. However, this largely depends on the context in which the experiment is conducted. Context information about a simulation study can be provided in form of provenance that documents which artifacts contributed in developing a simulation model. We present an approach that exploits provenance to support best practices for simulation experiments. The approach relies on 1) explicitly specified provenance information, 2) an ontology of methods, 3) best practices rules, and 4) integration with a previously developed experiment generation pipeline. We demonstrate our approach by conducting a sensitivity analysis experiment within a cell biological simulation study.

Background and Purpose: Agent-based modelling and simulation (ABS) is growing in many areas like, e.g., management, social and computer sciences. However, the similar trend does not seem to occur within the field of business process management (BPM), even though simulation approaches like discrete event simulation or system dynamics are well established and widely used. Thus, in our paper we investigate the advantages and disadvantages of agent-based modelling and simulation in the field of BPM in simulation experiments. Design/Methodology/Approach: In our research, we investigate if there is a necessity for ABS in the field of BPM with our own simulation experiments to compare traditional and ABS models. For this purpose, we use simulation framework MAREA, which is a simulation environment with integrated ERP system. Our model is a complex system of a trading company selling computer cables. For the verification of our model, we use automated process discovery techniques. Results: In our simulations, we investigated the impact of changes in resources’ behavior on the outcome of company’s order to cash process (O2C). Simulations experiments demonstrated that even small changes might have statistically significant effect on outcomes of the processes and decisions based on such outcomes. Simulation experiments also demonstrated that the impact of randomly distributed fluctuations of well-being have a diminishing tendency with the increasing number of sales representatives involved in the process. Conclusions: Our research revealed several advantages and disadvantages of using ABS in business process modelling. However, as we show, many of them were at least partially addressed in the recent years. Thus, we believe that ABS will get more attention in the field of BPM similarly to other fields like, e.g., social sciences. We suggested areas in BPM simulations, e.g., modelling of resources, be it human or technological resources, where there is a need for ABS.

Critical systems are present in our daily life and affect many aspects providing essential support to several human activities; they are employed in several application domains, providing support to economic activities, transportation, communication, health-care. Such systems must meet strict non-functional requirements, including dependability requirements, and should be able to cope with competitive market needs. Their criticality implies the necessity to meet several requirements, often dictated by international standards, whose fulfillment must be demonstrated in order to achieve necessary certifications. Quantitative evaluations are hence needed to assess and demonstrate compliance with target requirements, since design phases of the lifecycle. For this purpose modeling approaches have been investigated during the years; their aim is to have a of the overall system which allows to assess and demonstrate these properties. It is commonly known that the adoption of formal models in industry, although if strongly recommended when not mandatory, is slowed down by difficulties dictated both by the complexity of the models (which reflects the complexity of the systems) and by the need to have skilled staff in the usage of formal languages. For this reason, also if the academic research trends have focused on complex modeling approaches and techniques, formal models are not so used in industries: in particular only combinatorial models are widely adopted, thanks to their intuitive representation. The great simplifications introduced by combinatorial models lead to approximated results, hence system designers do not trust in them. In the last years the need of a model engineering is noticed: there is the necessity to define appropriate methodologies and processes to support development activities of complex models, aiming at improving the quality/cost ratio. In this way three main research trends have been introduced: formal models generation, multiformalism, and compositional approaches. This thesis aims at defining a methodology for quantitative analysis of critical system that can integrate Model-Based evaluations, conducted by using formal models, with Model-Driven Engineering (MDE) paradigms, taking advantage of the actual research trends. More in detail, according to such approach, critical systems properties can be assessed starting from high level models expressed in proper languages (usually called Domain Specific Modeling Languages - DSMLs), and defining automatic transformation chains, able to generate formal models. The methodology is implementable into cost-effective processes in order to reduce time-to-market specifically during verification stages of critical system development lifecycles. Such objective will be achieved by both improving usability and re-usability of typical Model-Driven artifacts. The adoption of this methodology for railway system dependability analysis may lead to an estimated costs reduction of approximately 30%. The usage of a high-level central language significantly decreases also the training time of new analysts, as they only need to learn the DSML usage (properly developed for the application domain). This thesis defines also two high-level modeling languages: the first, CIP_VAM, has been defined to vulnerability aspects of physical infrastructures, while the second, DAM-Rail, allows the dependability modeling in railway domain. Another important contribution of this thesis is given in the definition of new methods and formal operators: in detail a reference architecture for performability and Quality of Service evaluation is defined, as well as the formal concept of Model Template is introduced (to enable reuse). At last two techniques which deals with scalability are addressed: stub and reduced models. Even if the focus is on quantitative evaluation, the proposed methodology is useful also for the complete design (based on centralized informations), hence it is possible to extend it for automatic generation of documents and, more generally, of all those artifacts which support system development. The proposed approach has been applied with different goals to case studies coming from three different domains: railway systems, conferencing systems, and sensor network. These systems have different and complementary properties in terms of number of involved components, scalability, etc. Produced models are described, and obtained results are shown and commented.

It's very significant to correctly identify the hierarchical relation among attributes correctly when constructing a diagnosis test.As we know,there are various methods to identify the hierarchical relation among attributes in different cognitive diagnosis models(CDMs).Rule Space Model(RSM) is a kind of great influence CDM which was developed by Tatsuoka and her associates.In RSM,the task of attribute identification is performed after the test items have already been developed.And then an incidence Q matrix can be determined which reflects hierarchical relation.However,in Leighton,Gierl and Hunka's(2004) Attribute Hierarchy Method(AHM) the organization of attributes(attributes,number of attributes and attribute hierarchical relation) should be determined before developing the test items.In RSM and AHM the importance of correctly identifying the attributes and their hierarchical relation cannot be overstated,and the attributes and their hierarchical relations serve as the most important input variables to the models because they provide the basis for interpreting the results.The hierarchical relation among attributes describes the domain knowledge structures.Understanding the domain knowledge structures in highly specific detail provides a rational basis for proposing and evaluating potential improvements in the measurement of general proficiency.Otherwise,improvement remains largely a trial-and-error process.It has been proved that is not reliable to get the hierarchical relation by analyzing the test items,even the resulting hierarchical relations are not unique.Now,structure learning of Bayesian Networks would be introduced to study the hierarchical relation from the examinees' attribute mastery patterns.On the one hand,this paper conducted a simulation study in which Cui,Leighton and Zheng(2006)'s attribute hierarchical relation was employed,and the sample size of 5000 was used.In the simulated study,different sliping rates was considered for the purpose of testing the robustness of Bayesian structure learning.On the other hand,this paper conducted an empirical study to research the attributes which were commonly used in different numeration representation system converting,and the sample size was 189.Also the hierarchical relation generated by Bayesian structure learning and the hierarchical relation extracted by domain expert was compared.The result of simulation study and empirical study all showed the Bayesian Networks was a very useful tool in determining the hierarchical relation among attributes in the realm of education measurement and psychometrics.It is proved that the result for reference is valuable through structure learning of Bayesian Networks in simulated experiments and practical application.Meanwhile,the Bayesian Networks also can be used to test the hierarchical relation in RSM.

For many biological systems, a variety of simulation models exist. A new simulation model is rarely developed from scratch, but rather revises and extends an existing one. A key challenge, however, is to decide which model might be an appropriate starting point for a particular problem and why. To answer this question, we need to identify entities and activities that contributed to the development of a simulation model. Therefore, we exploit the provenance data model, PROV-DM, of the World Wide Web Consortium and, building on previous work, continue developing a PROV ontology for simulation studies. Based on a case study of 19 Wnt/β-catenin signaling models, we identify crucial entities and activities as well as useful metadata to both capture the provenance information from individual simulation studies and relate these forming a family of models. The approach is implemented in WebProv, a web application for inserting and querying provenance information. Our specialization of PROV-DM contains the entities Research Question, Assumption, Requirement, Qualitative Model, Simulation Model, Simulation Experiment, Simulation Data, and Wet-lab Data as well as activities referring to building, calibrating, validating, and analyzing a simulation model. We show that most Wnt simulation models are connected to other Wnt models by using (parts of) these models. However, the overlap, especially regarding the Wet-lab Data used for calibration or validation of the models is small. Making these aspects of developing a model explicit and queryable is an important step for assessing and reusing simulation models more effectively. Exposing this information helps to integrate a new simulation model within a family of existing ones and may lead to the development of more robust and valid simulation models. We hope that our approach becomes part of a standardization effort and that modelers adopt the benefits of provenance when considering or creating simulation models.

Recently, there is a growing interest in Systems of Systems (SoS), their architecture, security and application domains. However, their specific characteristics such as the operational independence of SoS constituent systems (CS), the absence of central authority and their emergent behavior make the modeling of their structure, behavior and security a complex task. One of the current main security challenges in the context of SoS is the cascading attack problem. The challenge is to predict the concatenation/sequence of CS's vulnerabilities that could be triggered resulting in destructive cascading failures and take corrective actions to reduce the cost, development time and effect of later changes. In this paper, we propose a domain specific modeling language (DSML) to represent SoS security rchitecture. Having SoS security models will enable the discovery, analysis and resolution of cascading attacks, in the architecture phase, preventing development time and cost wastage. Following a Model Driven Engineering (MDE) approach, we generate a graphical editor for our DSML and use it to model a Smart Campus case study.

In this paper, we consider computational super-resolution inverse diffraction phase retrieval. The optical setup is lensless, with a spatial light modulator for aperture phase coding. The paper is focused on experimental tests of the super-resolution sparse phase amplitude retrieval algorithm. We start from simulations and proceed to physical experiments. Both simulation tests and experiments demonstrate good-quality imaging for super-resolution with a factor of 4 and a serious advantage over diffraction-limited resolution as defined by Abbe’s criterion.