ARCH-COMP17 Category Report: Bounded Model Checking of Hybrid Systems with Piecewise Constant Dynamics

This report presents results of a friendly competition for formal verification of continuous and hybrid systems with piecewise constant dynamics. The friendly competition took place as part of the workshop Applied Verification for Continuous and Hybrid Systems (ARCH) in 2018. In this second edition, five tools have been applied to solve five different benchmark problems in the category for piecewise constant dynamics: BACH, Lyse, PHAVer, PHAVer-lite, and VeriSiMPL. Compared to last year, a new tool has participated (PHAVer-lite) and a benchmark has been made more complex (Dutch Railway Network). The result is a snapshot of the current landscape of tools and the types of benchmarks they are particularly suited for. Due to the diversity of problems, we are not ranking tools, yet the presented results probably provide the most complete assessment of tools for the safety verification of continuous and hybrid systems with piecewise constant dynamics up to this date.

This report presents the results of a friendly competition for formal verification and policy synthesis of stochastic models. It also introduces new benchmarks within this category, and recommends next steps for this category towards next year’s edition of the competition. The friendly competition took place as part of the workshop Applied Verification for Continuous and Hybrid Systems (ARCH) in Spring 2019.

Model checking of hybrid systems is usually expressed in terms of the following reachability problem for hybrid automata (HA) [6]: given an HA M, a set of initial states I, and a set of unsafe states U, determine whether there exists a trajectory of M starting in an initial state and ending in an unsafe state. The time-bounded version of this problem considers trajectories that are within a given time bound T.We introduce the State Classification Problem (SCP), a generalization of the model checking problem for hybrid systems. Let B = {0,1} be the set of Boolean values. Given an HA M with state space S(M), time bound T, and set of unsafe states U ⊂ S(M), the SCP problem is to find a function F*: S(M) → B such that for all s ∈ S(M), F* (s) = 1 if M |= Reach (U ,s,T), i.e., if it is possible for M, starting in s, to reach a state in U within time T ; F*(s) = 0 otherwise. A state s ⊂ S(M) is called positive if F*(s) = 1. Otherwise, s is negative. We call such a function a state classifier.State classification is also useful in at least two other contexts. First, due to random disturbances, a hybrid system may restart in a random state outside the initial region, and we may wish to check the system's safety from that state. Secondly, a classifier can be used for online model checking [10], where in the process of monitoring a system's behavior, one would like to determine, in real-time, the fate of the system going forward from the current (non-initial) state.This paper shows how deep neural networks (DNNs) can be used for state classification, an approach we refer to as Neural State Classification (NSC). An NSC classifier is subject to false positives (FPs) and, more importantly, false negatives (FNs). An FP occurs when a state s is deemed positive when it is actually negative, and, likewise, an FN occurs when s is deemed negative when it is actually positive.A well-trained NSC classifier offers high accuracy, runs in constant time (approx. 1 ms in our experiments), and takes constant space (e.g., a DNN with l hidden layers and n neurons only requires functions of dimension l · n for its encoding). This makes NSC classifiers very appealing for applications such as online model checking, a type of analysis subject to strict time and space constraints.Our approach can also classify states of parametric HA by encoding each parameter as an additional input to the classifier. This makes NSC more versatile than state-of-the-art hybrid system reachability tools, which provide little or no support for parametric analysis [3,4].The NSC method is summarized in Figure 1. We train the state classifier using supervised learning, where the training examples are derived by sampling the state and parameter spaces according to some distribution. Reachability values for the examples are computed by invoking an oracle, i.e., an hybrid system model checker [4] or a simulator when the system is deterministic.We evaluate a trained state classifier by estimating its accuracy, false-positive rate, and false-negative rate (together with their confidence intervals) on a test dataset of fresh samples. This allows us to quantify how well the classifier extrapolates to unseen states, i.e., the probability that it correctly predicts reachability for any state.Inspired by statistical model checking [8], we also provide statistical guarantees through sequential hypothesis testing to certify (up to some confidence level) that the classifier meets prescribed accuracy levels on unseen data.We also consider two tuning methods that can reduce and virtually eliminate false negatives: a new method called falsification-guided adaptation that iteratively re-trains the classifier with false negatives found through adversarial sampling; and threshold selection, which adjusts the NN's classification threshold to favor FPs over FNs.We have applied NSC to six nonlinear hybrid system benchmarks, achieving an accuracy of 99.25% to 99.98%, and a false-negative rate of 0.0033 to 0, which we further reduced to 0.0015 to 0 after tuning the classifier. We believe that this level of accuracy is acceptable in many practical applications, and that these results demonstrate the promise of the NSC approach.In the rest of this extended abstract, we provide more details about the NSC approach and discuss experimental results.

In this paper we present HySAT, a bounded model checker for linear hybrid systems, incorporating a tight integration of a DPLL---based pseudo---Boolean SAT solver and a linear programming routine as core engine. In contrast to related tools like MathSAT, ICS, or CVC, our tool exploits the various optimizations that arise naturally in the bounded model checking context, e.g.isomorphic replication of learned conflict clauses or tailored decision strategies, and extends them to the hybrid domain. We demonstrate that those optimizations are crucial to the performance of the tool.

Verifying properties of hybrid systems can be highly complex. To reduce the effort required to produce a correct proof, the use of mechanical verification techniques is promising. Recently, we extended a mechanical verification system, originally developed to reason about deterministic real-time automata, to verify properties of hybrid systems. To evaluate our approach, we applied our extended proof system to a solution, based on the Lynch-Vaandrager timed automata model, of the Steam Boiler Controller problem, a hybrid systems benchmark. This paper reviews our mechanical verification system, which builds on SRI's Prototype Verification System (PVS), and describes the features we added to handle hybrid systems. It also discusses some errors we detected in applying our system to the benchmark problem. We conclude with a summary of insights we acquired in using our system to specify and verify hybrid systems.

A methodology for safety verification of continuous and hybrid systems using barrier certificates has been proposed recently. Conditions that must be satisfied by a barrier certificate can be formulated as a convex program, and the feasibility of the program implies system safety in the sense that there is no trajectory starting from a given set of initial states that reaches a given unsafe region. The dual of this problem, i.e., the reachability problem, concerns proving the existence of a trajectory starting from the initial set that reaches another given set. Using insights from the linear programming duality appearing in the discrete shortest path problem, we show in this paper that reachability of continuous systems can also be verified through convex programming. Several convex programs for verifying safety and reachability, as well as other temporal properties such as eventuality, avoidance, and their combinations, are formulated. Some examples are provided to illustrate the application of the proposed methods. Finally, we exploit the convexity of our methods to derive a converse theorem for safety verification using barrier certificates.

Hybrid systems combining discrete and continuous dynamics arise as mathematical models of various artificial and natural systems, and as an approximation to complex continuous systems. Reachability analysis has been the principal research question in the verification of hybrid systems, even though it is a wellknown result that most non-trivial subclasses of hybrid systems reachability and most verification problems are undecidable [1]. Nonetheless, various decidable subclasses have been identified, including polygonal hybrid systems (SPDIs) [2]. SPDIs can be used, for instance, in the analysis of approximations of non-linear differential equations in two-dimensions.KeywordsModel CheckHybrid SystemPhase PortraitHybrid AutomatonReachability AnalysisThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Hybrid systems are characterized by combining discrete and continuous behaviors. Verification of hybrid systems is, in general, a diffcult task due to the potential complexity of the continuous dynamics. Currently, there are different formalisms and tools which are able to analyze specific types of hybrid systems, model checking being one of the most used approaches. In this paper, we describe an extension of Java PathFinder in order to analyze hybrid systems. We apply a general methodology which has been successfully used to extend Spin. This methodology is non-intrusive, and uses external libraries, such as the Parma Polyhedra Library, to abstract the continuous behavior of the hybrid system.

REACHABILITY COMPUTATION FOR UNCERTAIN PLANAR AFFINE SYSTEMS USING LINEAR ABSTRACTIONS

Intelligent environments can be described as hybrid systems, which combine continuous dynamics, modelling the behaviour of physical components, and discrete dynamics, modelling the software components that control the evolution of the physical variables. The growing boom in intelligent environments makes the construction of complex discrete components necessary, which may require the use of more sophisticated concurrent languages. Hy-tccp is a concurrent language for modelling hybrid systems with high-level notations that facilitate the description of computational systems, abstracting away from the implementation details. In this paper, we present an operational semantics based on hybrid automata for Hy-tccp which is a theoretical basis for the analysis and verification of hybrid systems like intelligent environment.

A Tool for Modular Modelling and Verification of Hybrid Systems

Hybrid systems are networks of interacting digital devices and continuous plants reacting to a changing environment. Our multiple agent hybrid control architecture ([KN93b], [KN93c]) is based on the notion of hybrid system state. The latter incorporates evolution models using differential or difference equations, logic constraints, and geometric constraints. The set of hybrid states of a hybrid system can be construed in a variety of ways as a differentiable (or a C ∞) manifold which we have called the carrier manifold ([KNRG95]). We have suggested that for control problems the coordinates of points of the carrier manifolds should be selected to incorporate all information about system state, control state, and environment needed to choose new values of control parameters.This paper provides an outline of our ongoing work which expresses a wide class of control problems as relaxed, or convexified, calculus of variations problems on carrier manifolds. Weierstrass pointed out that many calculus of variations problems with time as an independent variable can be rewritten as parametric problems, that is, as problems where time is not an explicit variable, by moving to a larger manifold where time is an extra variable, x n+1=t, with the extra constraint x n+1=1. Then Finsler (1918) pointed out that each such parametric calculus of variations problem induces a metric ground form ds 2 associated with the manifold. We reduce a wide class of optimal control problems to calculating the Cartan affine connection associated with the Finsler metric ground form induced by the corresponding convexified calculus of variations problem. For any prescribed ε of deviation from the variational minimum, we then use chatterings of control flows on the manifold to approximate to a connection and use this to compute a finite state control program which enforces ε-optimal behavior. Thus we find that Finsler manifolds and their associated differential geometry and control flows form a computational basis for extraction and verification of hybrid systems and intelligent control systems in general. We remark that other connections, not arising from metrics, can be usefully employed in control too.Here we discuss only the simplest problem of the calculus of variations and optimal control: find an “optimal” path x(t) among admissible paths on the manifold M extending from initial point x(t 0 )=x0 to endpoint x(t 1 )=x1 with a given end direction. x(t 1 )=y 1. The theory of Finsler manifolds is not familiar to most electrical engineers and certainly not to computer scientists. The goal of this paper is to give some historical background and a brief description of how one interprets connections as optimal controls and extraction of finite state control programs as extraction of approximations to connections. One can also unwind and use connections similarly for non-parametric problems. A book length mathematical treatment is in preparation.KeywordsHybrid SystemTangent SpaceTangent VectorParallel TransportFinsler SpaceThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Safety verification and reachability analysis for hybrid systems

Reachability analysis has recently proved to be a useful technique for analysing the behaviour of under-specified biological models. In this paper, we propose a method exploiting the eigenstructure of a linear continuous system to efficiently estimate a bounded interval containing the time at which the system can reach a target set from an initial set. Then this estimation can be directly integrated in an existing algorithm for hybrid systems with linear continuous dynamics, to speed up reachability computations. Furthermore, it can also be used to improve time-efficiency of the hybridization technique that is based on a piecewise-linear approximation of non-linear continuous dynamics. The proposed method is illustrated on a number of examples including a biological model.

The interplay of random phenomena with discrete-continuous dynamics deserves increased attention in many systems of growing importance. Their verification needs to consider both stochastic behaviour and hybrid dynamics. In the verification of classical hybrid systems, one is often interested in deciding whether unsafe system states can be reached. In the stochastic setting, we ask instead whether the probability of reaching particular states is bounded by a given threshold. In this thesis, we consider stochastic hybrid systems and develop a general abstraction framework for deciding such problems. This gives rise to the first mechanisable technique that can, in practice, formally verify safety properties of systems which feature all the relevant aspects of nondeterminism, general continuous-time dynamics, and probabilistic behaviour. Being based on tools for classical hybrid systems, future improvements in the effectiveness of such tools directly carry over to improvements in the effectiveness of our technique. We extend the method in several directions. Firstly, we discuss how we can handle continuous probability distributions. We then consider systems which we are in partial control of. Next, we consider systems in which probabilities are parametric, to analyse entire system families at once. Afterwards, we consider systems equipped with rewards, modelling costs or bonuses. Finally, we consider all orthogonal combinations of the extensions to the core model.