Causal Temporal Reasoning for Markov Decision Processes

Over the last two decades, there has been an extensive study of logical formalisms on specifying and verifying real-time systems. Temporal logics have been an important research subject within this direction. Although numerous logics have been introduced for formal specification of real-time and complex systems, an up to date survey of these logics does not exist in the literature. In this paper we analyse various temporal formalisms introduced for specification, including propositional/first-order linear temporal logics, branching temporal logics, interval temporal logics, real-time temporal logics and probabilistic temporal logics. We give decidability, axiomatizability, expressiveness, model checking results for each logic analysed. We also provide a comparison of features of the temporal logics discussed.

Causal reasoning and logical reasoning are two important types of reasoning abilities for human intelligence. However, their relationship has not been extensively explored under machine intelligence context. In this paper, we explore how the two reasoning abilities can be jointly modeled to enhance both accuracy and explainability of machine learning models. More specifically, by integrating two important types of reasoning ability--counterfactual reasoning and (neural) logical reasoning--we propose Counterfactual Collaborative Reasoning (CCR), which conducts counterfactual logic reasoning to improve the performance. In particular, we use recommender system as an example to show how CCR alleviate data scarcity, improve accuracy and enhance transparency. Technically, we leverage counterfactual reasoning to generate "difficult" counterfactual training examples for data augmentation, which--together with the original training examples--can enhance the model performance. Since the augmented data is model irrelevant, they can be used to enhance any model, enabling the wide applicability of the technique. Besides, most of the existing data augmentation methods focus on "implicit data augmentation" over users' implicit feedback, while our framework conducts "explicit data augmentation" over users explicit feedback based on counterfactual logic reasoning. Experiments on three real-world datasets show that CCR achieves better performance than non-augmented models and implicitly augmented models, and also improves model transparency by generating counterfactual explanations.

Spoken dialogue managers have benefited from using stochastic planners such as Markov Decision Processes (MDPs). However, so far, MDPs do not handle well noisy and ambiguous speech utterances. We use a Partially Observable Markov Decision Process (POMDP)-style approach to generate dialogue strategies by inverting the notion of dialogue state; the state represents the user's intentions, rather than the system state. We demonstrate that under the same noisy conditions, a POMDP dialogue manager makes fewer mistakes than an MDP dialogue manager. Furthermore, as the quality of speech recognition degrades, the POMDP dialogue manager automatically adjusts the policy.

Decision-making based on probabilistic reasoning often involves selecting a subset of expensive observations that best predict the system state. In an earlier work, adopting the general notion of value of information (VoI) first introduced by Krause and Guestrin, Ghosh and Ramakrishnan considered the problem of determining optimal conditional observation plans in temporal graphical models, based on non-myopic (non-greedy) VoI, over a finite time horizon. They cast the problem as determining optimal policies in finite-horizon, non-discounted Markov Decision Processes (MDPs). However, there are many practical scenarios where a time horizon is undefinable. In this paper, we consider the VoI optimization problem over an infinite (or equivalently, undefined) time horizon. Adopting an approach similar to Ghosh and Ramakrishnan's, we cast this problem as determining optimal policies in infinite-horizon, finite-state, discounted MDPs. Although our MDP-based framework addresses Dynamic Bayesian Networks (DBNs) that are more restricted than those addressed by Ghosh and Ramakrishnan, we incorporate Krause and Guestrin's general idea of VoI even though it was fundamentally envisioned for finite-horizon settings. We establish the utility of our approach on two graphical models based on real-world datasets.

AITIA-PM: Discovering the true causes of events in a process mining context

In this paper we describe PRISM, a tool being developed at the University of Birmingham for the analysis of probabilistic systems. PRISM supports three probabilistic models: discrete-time Markov chains, Markov decision processes and continuous-time Markov chains. Analysis is performed through model checking such systems against specifications written in the probabilistic temporal logics PCTL and CSL. The tool features three model checking engines: one symbolic, using BDDs (binary decision diagrams) and MTBDDs (multi-terminal BDDs); one based on sparse matrices; and one which combines both symbolic and sparse matrix methods. PRISM has been successfully used to analyse probabilistic termination, performance, and quality of service properties for a range of systems, including randomized distributed algorithms, manufacturing systems and workstation clusters.KeywordsModel CheckMarkov Decision ProcessBinary Decision DiagramSymbolic Model CheckerModel Check ToolThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Background and Objectives: Numerical iterative methods are widely used to compute reachability probabilities and expected rewards in probabilistic model checking of Markov decision processes. Several approaches have been proposed to improve the performance of these iterative methods. Reducing the total number of iterations is an important goal that is followed by several techniques in this paper.Methods: In this paper, we consider MDPs with different levels of non-determinism. We show that modified policy iteration performs better than the other standard iterative methods when the degree of non-determinism increases. We propose some novel methods to improve the performance of the modified policy iteration method. Our approach is to define several criteria to avoid useless iterations and updates of the modified policy iteration method. The first criterion considers the graphical structure of the model to define the number of iterations after each policy modification. The second criterion is a dynamic one to determine when a policy should be updated. The third proposed approach in this work is to use several priority heaps to select states for value updates.Results: Our proposed methods are implemented in the PRISM model checker and applied on several standard case studies. The results of experiments show that the running times of our approaches are less than the running time of the standard and previous methods for most case studies. The main reason for these results is that the total numbers of iterations and updates are reduced by our approaches, which results in an improvement in the performance of the method. While the running times are reduced by our approaches, the precision of computations are kept in most cases.Conclusion: The proposed techniques are able to reduce the number of iterations and accelerate the convergence to the optimal policies. The also prioritize the commutations to reduce the total number of updates.

Quantitative verification techniques offer an effective meansof computing performance and reliability properties for a wide range of systems.In many cases, it is necessary to perform repeated analyses of a system,for example to identify trends in results, determine optimal system parametersor when performing online analysis for adaptive systems.We argue the need for incremental quantitative verification techniqueswhich are able to re-use results from previous verification runs in order to improve efficiency.We report on recently proposed techniques for incremental quantitative verification of Markov decision processes,based on a decomposition of the model into its strongly connected components.We give an overview of the method, describe a number of useful optimisationsand show experimental results that illustrate significant gains in run-time performanceusing the incremental approach.

We address the problem of controlling a noisy differential drive mobile robot such that the probability of satisfying a specification given as a bounded linear temporal logic formula over a set of properties at the regions in the environment is maximized. We assume that the vehicle can determine its precise initial position in a known map of the environment. However, motivated by practical limitations, we assume that the vehicle is equipped with noisy actuators and, during its motion in the environment, it can only measure the angular velocity of its wheels using limited accuracy incremental encoders. Assuming the duration of the motion is finite, we map the measurements to a Markov decision process (MDP). We use recent results in statistical model checking to obtain an MDP control policy that maximizes the probability of satisfaction. We translate this policy to a vehicle feedback control strategy and show that the probability that the vehicle satisfies the specification in the environment is bounded from below by the probability of satisfying the specification on the MDP. We illustrate our method with simulations and experimental results.

Reinforcement Learning with Temporal Logic Constraints

In this thesis, we introduce and examine four new temporal logic formalisms that can be used as specification languages for the automated verification of the reliability of hardware and software designs with respect to a desired behavior. The work is organized in two parts. In the first one, we reason about two logics for computations, GCTL* and MCTL*, which are useful to describe a correct execution of monolithic closed systems. In the second one, instead, we focus on two logics for strategies, SL and mATL*, which are useful to formalize several interesting properties about interactive plays in multi-entities systems modeled as multi-agent games. In the “Logics for Computations” part, we first study the immersion of the idea of graded quantifications into the temporal-logic framework. In first order logic, existential and universal quantifiers express the concepts of the existence of at least one individual object satisfying a formula, or that all individual objects satisfy a formula. In other logics, these quantifiers have been generalized to express that, for a given non-negative integer n, at least n or all but n individuals satisfy a particular formula. Here, we consider GCTL, a temporal logic with graded path quantifiers, which allows to describe properties like “there exist at least n different classes of computational fluxes in which a system reaches a predetermined state”, where the classes over paths are computed by means of a predetermined equivalence relation. More precisely, we uniformly extend the classic concept of graded quantifiers from states to paths, through the use of a concept of path equivalence with respect to a given path formula. About this logic, in particular, we study the expressiveness and succinctness relationships with respect to GµCalculus and the complexity of the satisfiability problem, which results to be ExpTime-Complete. This research is partially based on the works [BMM09] “Graded Computation Tree Logic” and [BMM10] “Graded Computation Tree Logic with Binary Coding” published, respectively, in the proceedings of the “IEEE Symposium on Logic in Computer Science, 2009” and “EACSL Annual Conference on Computer Science Logic, 2010”. Preliminary results can be also found in [Mog07]. Furthermore, we consider special quantifiers over substructures, which allow to select, using parametric criteria, small critical parts of a system to be successively verified. In literature, there are some attempts to define a logic that allows to modify the underlying structure under exam and then to verify on it some assigned property. However, as far as we know, none of them is able to select minimal submodels of a given property describing the criteria on which then execute the verification process. Here, we base our work on the search of a new operator that merges the concept of quantifiers on structures with that one derived by a generalization of the concept of pruning. The results of this work, is a class of three different extensions of CTL* with minimal model quantifiers, which we name MCTL*. Regarding these logics, we study several reductions among them, as well as the satisfiability problem that we prove to be highly undecidability, i.e., Σ_1^1-Hard, for two out of the three cases. This research is partially based on the work [MM09] “Branching-Time Temporal Logics with Minimal Model Quantifiers” published in the proceedings of the “International Conference on Developments in Language Theory, 2009”. In the “Logics for Strategies” part, we first study the problem of defining a new specification language through which it is possible to express several important properties of multi-entities systems that are neither expressible using classical monolithic temporal logics, such as CTL*, nor using two-agent-teams temporal logics, such as ATL*. In literature, we can found some proposal of logics that try to achieve this goal, but unfortunately, none of them succeeds completely on all the aspects. Among them, one of the most important attempts is CHP-SL, a logics in which one can use variables over strategies. However, this logic has a deep weakness, since it does not allow to describe games with more than two players and even two-players concurrent games. Here, we introduce SL, a logic with a syntax similar in some aspects to the first order logic, in which the strategies of the agent building the game are treated as first order objects on which we can quantify. This logic generalizes CHP-SL, by allowing the specification of the correct behavior of multi-agent concurrent games. In SL, for example, we are able to express very complex but useful Nash equilibria that are not expressible with CHP-SL. We enlighten that Nash equilibrium is one of the most important concepts in game theory. For the introduced logic, we solve two problems left open in the work on CHP-SL. Precisely, we show that the related model-checking problem is 2ExpTime-Complete, thus not harder of the same problem for several subsumed logics, while we prove that its satisfiability problem is highly undecidable, i.e., Σ_1^1-Hard. This research is partially based on the work [MMV10a] “Reasoning About Strategies” published in the proceedings of the “IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science, 2010”. Finally, we consider the concept of relentful strategic reasoning, i.e., a formalism that expresses the ability of a strategy to be used not only to achieve a first given goal, but also to change its final goal in dependence of the history of the play. In the context of planning, memoryful quantification, i.e., quantification over computations that does not lose information about the past along the time, is one of the principal way to express the fact that a system is able to achieve a desired result, and shift to a different goal if some event happens. However, this kind of quantification was not considered before in the context of multi-agent planning. Here, we introduce mATL*, a fusion of the classic alternating temporal logic ATL* with memoryful quantification, with the aim of covering the previous idea. About this logics, we prove that, although it is equivalent to ATL*, it is exponentially more succinct. Nevertheless, we prove that both the model-checking and the satisfiability problems remain 2ExpTime-Complete, as for ATL*. This research is partially based on the work [MMV10b] “Relentful Strategic Reasoning in Alternating-Time Temporal Logic” published in the proceedings of the “International Conference on Logic for Programming Artificial Intelligence and Reasoning, 2010”.

Probabilistic Computation Tree Logic (PCTL) is frequently used to formally specify control objectives such as probabilistic reachability and safety. In this work, we focus on model checking PCTL specifications statistically on Markov Decision Processes (MDPs) by sampling, e.g., checking whether there exists a feasible policy such that the probability of reaching certain goal states is greater than a threshold. We use reinforcement learning to search for such a feasible policy for PCTL specifications, and then develop a statistical model checking (SMC) method with provable guarantees on its error. Specifically, we first use upper-confidence-bound (UCB) based Q-learning to design an SMC algorithm for bounded-time PCTL specifications, and then extend this algorithm to unbounded-time specifications by identifying a proper truncation time by checking the PCTL specification and its negation at the same time. Finally, we evaluate the proposed method by case studies.

Hyperproperties are properties that describe the correctness of a system as a relation between multiple executions. Hyperproperties generalize trace properties and include information-flow security requirements, like noninterference, as well as requirements like symmetry, partial observation, robustness, and fault tolerance. We initiate the study of the specification and verification of hyperproperties of Markov decision processes (MDPs). We introduce the temporal logic PHL (Probabilistic Hyper Logic), which extends classic probabilistic logics with quantification over schedulers and traces. PHL can express a wide range of hyperproperties for probabilistic systems, including both classical applications, such as probabilistic noninterference, and novel applications in areas such as robotics and planning. While the model checking problem for PHL is in general undecidable, we provide methods both for proving and for refuting formulas from a fragment of the logic. The fragment includes many probabilistic hyperproperties of interest.

This work contributes to reduce the cost of the verification process of model-based systems, especially the ones designed as SysML activity diagrams, by relying on the concept of diagram calls defined in SysML standard. By exploiting the diagram calls concept, we propose a mechanism to transform a given diagram into its equivalent fractal (hierarchical) form. Further, we present an abstraction algorithm to reduce the size of the obtained SysML activity diagram by ignoring the irrelevant behaviors and by merging similar artifacts. To do verification, the abstracted diagram is transformed automatically into PRISM source code to check the system's requirements that are specified in the probabilistic temporal logic, PCTL. The soundness of the provided framework is presented for each step, and, its practical effectiveness is demonstrated on diagrams obtained by reversing the open source of OpenSAF middle-ware.

In this paper, We consider an uncertain Markov decision process (MDP) with a control cost and a linear temporal logic (LTL) control specification. We propose a reinforcement learning (RL) based method for design of an optimal control policy by which the controlled MDP satisfies the control specification with probability 1 and minimizes an expected discounted sum of the control costs. First, we construct a deterministic Rabin automaton (DRA) that accepts all and only infinite words satisfying the LTL control specification. Second, we construct a product MDP of the MDP and the DRA to represent a dynamic control policy that satisfies the control specification. Third, we modify the product MDP in order to apply RL to the design of an optimal control policy. The control action of the modified product MDP is a pair of a pattern and an action, where the pattern is a set of actions. Moreover, we introduce a reward that represents both the satisfaction of the control specification and the minimally restrictiveness of the pattern. Finally, we proposed an algorithm for design of an optimal control policy that consists of a sequential decision making of two steps. At the first decision making, we select a pattern that maximizes a discounted sum of the reward. At the second one, we select an action from the pattern selected at the first one such that it minimizes the expected discounted sum of the costs. Moreover, we consider an illustrative example to show that the proposed algorithm can obtain an optimal control policy.