Efficient Verification of Distributed Protocols Using Stateful Model Checking

We propose to perform statistical probabilistic model checking by using perfect simulation in order to verify steady-state and time unbounded until formulas over Markov chains. The model checking of probabilistic models by statistical methods has received increased attention in the last years since it provides an interesting alternative to numerical model checking which is poorly scalable with the increasing model size. In previous statistical model checking works, unbounded until formulas could not be efficiently verified, and steady-state formulas had not been considered due to the burn-in time problem to detect the steady-state. Perfect simulation is an extension of Markov Chain Monte Carlo (MCMC) methods that allows us to obtain exact steady-state samples of the underlying Markov chain, and thus it avoids the burn-in time problem to detect the steady-state. Therefore we suggest to verify time unbounded until and steady-state dependability properties for large Markov chains through statistical model checking by combining perfect simulation and statistical hypothesis testing.

A parameter generation method based on model checking is proposed to tackle the parameter selection of hierarchical scheduling systems in Integrated Modular Avionics (IMA) by combining the classical symbolic model checking and the Statistical Model Checking (SMC). It builds a generic timed automata network to describe the temporal behavior of hierarchical systems. A distributed genetic algorithm is adopted to search the optimum partition parameters with respect to processor utilization while guaranteeing the schedulability of the system, which is formulated as safety properties of symbolic model checking and hypothesis testing of SMC. Comparing with the widely-used response time analysis, the formal model of this method is more expressive to cover complex features. The application of SMC alleviates the "state space explosion" of classical model checking. Finally, the parameter generation experiments show that the present method is able to find the global optimum solutions in the parameter space.

Transaction‐level modeling with SystemC has been very successful in describing the behavior of embedded systems by providing high‐level executable models, in which many of them have inherent probabilistic behaviors, eg, random data and unreliable components. It is thus crucial to have both quantitative and qualitative analysis of the probabilities of system properties.Such analysis can be conducted by constructing a formal model of the system under verification and using Probabilistic Model Checking. However, this method is infeasible for large systems, due to the state space explosion. In this article, we demonstrate the successful use of statistical model checking to conduct such analysis directly from large SystemC models and allow designers to express a wide range of useful properties.The first contribution of this work is a framework to verify properties expressed in Bounded Linear Temporal Logic for SystemC models with both timed and probabilistic characteristics.Second, the framework allows users to expose a rich set of user code primitives as atomic propositions in Bounded Linear Temporal Logic. Moreover, users can define their own fine‐grained time resolution rather than the boundary of clock cycles in the SystemC simulation.The third contribution is an implementation of a statistical model checker. It contains an automatic monitor generation for producing execution traces of the model‐under‐verification, the mechanism for automatically instrumenting the model‐under‐verification, and the interaction with statistical model checking algorithms.

Neural networks (NN) are taking over ever more decisions thus far taken by humans, even though verifiable system-level guarantees are far out of reach. Neither is the verification technology available, nor is it even understood what a formal, meaningful, extensible, and scalable testbed might look like for such a technology. The present paper is an attempt to improve on both the above aspects. We present a family of formal models that contain basic features of automated decision-making contexts and which can be extended with further orthogonal features, ultimately encompassing the scope of autonomous driving. Due to the possibility to model random noise in the decision actuation, each model instance induces a Markov decision process (MDP) as verification object. The NN in this context has the duty to actuate (near-optimal) decisions. From the verification perspective, the externally learnt NN serves as a determinizer of the MDP, the result being a Markov chain which as such is amenable to statistical model checking. The combination of an MDP and an NN encoding the action policy is central to what we call “deep statistical model checking” (DSMC). While being a straightforward extension of statistical model checking, it enables to gain deep insight into questions like “how high is the NN-induced safety risk?”, “how good is the NN compared to the optimal policy?” (obtained by model checking the MDP), or “does further training improve the NN?”. We report on an implementation of DSMC inside the ModestToolset in combination with externally learnt NNs, demonstrating the potential of DSMC on various instances of the model family, and illustrating its scalability as a function of instance size as well as other factors like the degree of NN training.

Symbolic state-space exploration meets statistical model checking

Hybrid Petri nets with general transitions (HPnGs) provide a formalism for modeling safety-critical systems and evaluating their dependability with means of model checking. HPnGs form a restricted subclass of Stochastic Hybrid Automata and allow discrete, continuous and stochastic variables. Previously, discrete-event simulation and Statistical Model Checking (SMC) have been used to overcome the restrictions of existing analytical approaches, e.g., to a limited number of random variables. Also when simulating, the evolution of continuous variables has been restricted to piecewise-linear trajectories, where derivatives do not change between two events. Here, we extend the modeling formalism, the simulation and SMC approach to variables with a non-linear continuous evolution. The core idea of this extension lies in transforming the input system into a so-called second-order quantized state system. A case study on the Kinetic Battery Model validates our approach by comparing results to those obtained by Matlab.

Probabilistic Computation Tree Logic (PCTL) is frequently used to formally specify control objectives such as probabilistic reachability and safety. In this work, we focus on model checking PCTL specifications statistically on Markov Decision Processes (MDPs) by sampling, e.g., checking whether there exists a feasible policy such that the probability of reaching certain goal states is greater than a threshold. We use reinforcement learning to search for such a feasible policy for PCTL specifications, and then develop a statistical model checking (SMC) method with provable guarantees on its error. Specifically, we first use upper-confidence-bound (UCB) based Q-learning to design an SMC algorithm for bounded-time PCTL specifications, and then extend this algorithm to unbounded-time specifications by identifying a proper truncation time by checking the PCTL specification and its negation at the same time. Finally, we evaluate the proposed method by case studies.

Statistical model checking (SMC) for the analysis of multi-agent systems has been studied in the recent past. A feature peculiar to multi-agent systems in the context of statistical model checking is that of aggregate queries–temporal logic formula that involves a large number of agents. To answer such queries through Monte Carlo sampling, the statistical approach to model checking simulates the entire agent population and evaluates the query. This makes the simulation overhead significantly higher than the query evaluation overhead. This problem becomes particularly challenging when the model checking queries involve multiple attributes of the agents. To alleviate this problem, we propose a population sampling algorithm that simulates only a subset of all the agents and scales to multiple attributes, thus making the solution generic. The population sampling approach results in increased efficiency (a gain in running time of 50%–100%) for a marginal loss in accuracy (between 1% and 5%) when compared with the exhaustive approach (which simulates the entire agent population to evaluate the query), especially for queries that involve limited time horizons. Finally, we report parallel versions of the above algorithms. We explore different strategies of core allocation, both for exhaustive simulations of all agents and the sampling approach. This yields further gains in running time, as expected. The parallel approach, when combined with the sampling idea, results in improving the efficiency (a gain in running time of 100%–150%) with a minor loss when compared with the exhaustive approach in accuracy (between 1% and 5%).

Rare event simulation for highly dependable systems with fast repairs

The paper presents the existing verification methods for control algorithms in power electronics systems, including the application of model checking techniques. In the industry, the most frequently used verification methods are simulations and experiments; however, they have to be performed manually and do not give a 100% confidence that the system will operate correctly in all situations. Here we show the recent advancements in verification and performance assessment of power electronics systems with the usage of formal methods. Symbolic model checking can be used to achieve a guarantee that the system satisfies user-defined requirements, while statistical model checking combines simulation and statistical methods to gain statistically valid results that predict the behavior with high confidence. Both methods can be applied automatically before physical realization of the power electronics systems, so that any errors, incorrect assumptions or unforeseen situations are detected as early as possible. An additional functionality of verification with the use of formal methods is to check the converter operation in terms of reliability in various system operating conditions. It is possible to verify the distribution and uniformity of occurrence in time of the number of transistor switching, transistor conduction times for various current levels, etc. The information obtained in this way can be used to optimize control algorithms in terms of reliability in power electronics. The article provides an overview of various verification methods with an emphasis on statistical model checking. The basic functionalities of the methods, their construction, and their properties are indicated.

In the railway domain, an interlocking is the system ensuring safe train traffic inside a station by controlling its active elements such as the signals or points. Modern interlockings are configured using particular data, called application data, reflecting the track layout and defining the actions that the interlocking can take. The safety of the train traffic relies thereby on application data correctness, errors inside them can cause safety issues such as derailments or collisions. Given the high level of safety required by such a system, its verification is a critical concern. In addition to the safety, an interlocking must also ensure that availability properties, stating that no train would be stopped forever in a station, are satisfied. Most of the research dealing with this verification relies on model checking. However, due to the state space explosion problem, this approach does not scale for large stations. More recently, a discrete event simulation approach limiting the verification to a set of likely scenarios, was proposed. The simulation enables the verification of larger stations, but with no proof that all the interesting scenarios are covered by the simulation. In this paper, we apply an intermediate statistical model checking approach, offering both the advantages of model checking and simulation. Even if exhaustiveness is not obtained, statistical model checking evaluates with a parametrizable confidence the reliability and the availability of the entire system.

We consider the problem of verifying stochastic models of biochemical networks against behavioral properties expressed in temporal logic terms. Exact probabilistic verification approaches such as, for example, CSL/PCTL model checking, are undermined by a huge computational demand which rule them out for most real case studies. Less demanding approaches, such as statistical model check-ing, estimate the likelihood that a property is satisfied by sampling executions out of the stochastic model. We propose a methodology for efficiently estimating the likelihood that a LTL property φ holds of a stochastic model of a biochemical network. As with other statistical verification tech-niques, the methodology we propose uses a stochastic simulation algorithm for generating execution samples, however there are three key aspects that improve the efficiency: first, the sample generation is driven by on-the-fly verification of φ which results in optimal overall simulation time. Second, the confidence interval estimation for the probability of φ to hold is based on an efficient variant of the Wilson method which ensures a faster convergence. Third, the whole methodology is designed according to a parallel fashion and a prototype software tool has been implemented that performs the sampling/verification process in parallel over an HPC architecture. 1

Cyber-Physical Systems (CPSs) as advanced embedded systems integrating computation with physical process are increasingly penetrating into our life. Modeling and analysis for such systems closely involved with us are actively researched. A current challenging problem is how to take advantages of existing technologies like SysML/MARTE, Modelica and Statistical Model Checking (SMC) through effective integration. Moreover, the lack of efficient methodologies or tools for modeling and analysis of CPSs makes the gap between design and analysis models hard to bridge. To solve these problems, we present a framework named Modana to achieve an integrated process from modeling with SysML/MARTE to analysis with SMC for CPSs in terms of Non Functional Properties (NFP) such as time, energy, etc. Functional Mock-up Interface (FMI), as a connecting link between modeling and analysis, plays a major role in coordinating various tools for co-simulation to generate traces as the input of statistical model checker. To demonstrate the capability of Modana framework, we model energy-aware buildings as a case study, and discuss the analysis on energy consumption in different scenarios.

The UML behavioral models are used in understanding and communicating the problem domain concepts, during the requirement analysis phase of system development. The absence of formal semantics for UML behavioral models makes it difficult to build automated tools for their analysis, simulation and validation. These semantics should be well defined based on a formal language that yields the fundamental requirements for a rigorous validation of the specification models. In this paper, an approach is proposed for formal analysis and simulation of the UML behavioral models. Initially, we define UML activity diagram semantics. These semantics are translated into a Coloured Petri Nets (CPN). Hence, In order to allow a more concrete model behavior analysis, we coined the mapping rules from the specification models composed of objects and events. The results of analysis are in the form of a CPN-Tools report by means of state space analysis or model checking, to illustrate the proposed methodology we used a case study.

Statistical Model Checking (SMC) is a powerful and widely used approach that consists in estimating the probability for a system to satisfy a temporal property. This is done by monitoring a finite number of executions of the system, and then extrapolating the result by using statistics. The answer is correct up to some confidence that can be parameterized by the user. It is known that SMC mitigates the state-space explosion problem and allows to approximate undecidable queries. The approach has been implemented in several toolsets such as Plasma Lab, and successfully applied in a wide range of diverse areas such as systems biology, robotic, or automotive. In this paper, we add two new modest contributions to the cathedral of results on SMC. The first contribution is an algorithm that can be used to monitor changes in the probability distribution to satisfy a bounded-time property at runtime. Concretely, the algorithm constantly monitors the execution of the deployed system, and raises a flag when it observes that the probability has changed significantly. This is done by extending the applicability of the CUSUM algorithm used in signal processing into the formal validation setting. Our second contribution is to show how the programming interface of Plasma Lab can be exploited in order to make SMC technology directly available in toolsets used by designers. This integration is done by exploiting simulation facilities of design tools. Our approach thus differs from the one adopted by other SMC/formal verification toolsets which assume the existence of formal semantics for the design language, as well as a compiling chain to the rather academic one used by validation tool. The concept results in the integration of Plasma Lab as a library of the Simulink toolset. The contributions are illustrated by using Plasma Lab to verify a Simulink case study modelling a pig shed temperature controller.