Probabilistic Plan Synthesis for Coupled Multi-Agent Systems

Probabilistic Computation Tree Logic (PCTL) is frequently used to formally specify control objectives such as probabilistic reachability and safety. In this work, we focus on model checking PCTL specifications statistically on Markov Decision Processes (MDPs) by sampling, e.g., checking whether there exists a feasible policy such that the probability of reaching certain goal states is greater than a threshold. We use reinforcement learning to search for such a feasible policy for PCTL specifications, and then develop a statistical model checking (SMC) method with provable guarantees on its error. Specifically, we first use upper-confidence-bound (UCB) based Q-learning to design an SMC algorithm for bounded-time PCTL specifications, and then extend this algorithm to unbounded-time specifications by identifying a proper truncation time by checking the PCTL specification and its negation at the same time. Finally, we evaluate the proposed method by case studies.

With the advances of Artificial Intelligence (AI) and autonomous robotics, robots are increasingly involved closely with human society, such as in manufacturing system, autonomous driving and intelligent service robots. While robots are good at handling repeated routine work, human are more adaptive and flexible to changing factors that may bring uncertainties and cost non-trivial efforts for robots to overcome. Thus efficient collaboration between human and robots can bring huge economic profits, but also require high standard for robot controller during task completion because the safety of human must be guaranteed. In this paper, we build a formal design theory enabling an automated human-robot collaboration with performance guarantees. Markov Decision Processes (MDPs) are used to model robots with uncertainties and Partially Observable Markov Decision Processes (POMDPs) are used to model human with hidden intents. Within a supervisory control framework, we propose to use L* learning algorithm to learn a correct supervisor such that global collaboration task given as Probabilistic Computation Tree Logic (PCTL) can be satisfied. An example from manufacturing system is discussed to further illustrate the design method.

We address the problem of synthesizing control strategies for Ellipsoidal Markov Decision Processes (EMDP), i.e., MDPs whose transition probabilities are expressed using ellipsoidal uncertainty sets. The synthesized strategy aims to maximize the total expected reward of the EMDP, constrained to a specification expressed in Probabilistic Computation Tree Logic (PCTL). We prove that the EMDP strategy synthesis problem for the fragment of PCTL disabling operators with a finite time bound is NP-complete and propose a novel sound and complete algorithm to solve it.

In this paper, we propose a novel top-down design method for the development of collective behaviors of swarm robotics systems called property-driven design. Swarm robotics systems are usually designed and developed using a code-and-fix approach, that is, the developer devises, tests and modifies the individual robot behaviors until a desired collective behavior is obtained. The code-and-fix approach can be very time consuming and relies completely on the ingenuity and expertise of the designer. The idea of property-driven design is that a swarm robotics system can be described by specifying formally a set of desired properties. In an iterative process similar to test-driven development, the developer produces a model of the system that satisfies the desired properties. Subsequently, the system is implemented in simulation and using real robots. Property-driven design helps to minimize the risk of developing a system that does not satisfy the required properties, and to promote the reuse of hardware independent models. In this paper, we start by giving a general description of the method. We then present a possible way to apply it by using Discrete Time Markov Chains (DTMC) and Probabilistic Computation Tree Logic* (PCTL*). Finally, we conclude by presenting the application of the proposed method to the design and development of a swarm robotics system performing aggregation.

The role of human behaviour in enterprise security is one of the little studied aspects. The author proposes a reinforcement model of collaborative security employing basic concepts from game theory, socio-psychology, and probabilistic model-checking. The proposed model aims towards solving the problem of inducing positive network effect to enable user centric monitoring of security violations, in particular, against violations related to ”semantic manipulation” of context dependent logical resources. Preventing such violations using existing security enforcement mechanisms is neither feasible nor cost effective. The author defines a payoff mechanism to formalize the model by stipulating appropriate payoffs as reward, punishment, and community price according to reporting of genuine or false violations, non-reporting of the detected violations, and proactive reporting of vulnerabilities and threats by the users. Correctness properties of the model are defined in terms of probabilistic robustness property and constraints for economic feasibility of the payoffs. For estimating the payoff parameters, system and user behaviours are further modelled in terms of probabilistic finite state machines (PFSM) and likelihood of the success of the model is specified using probabilistic computation tree logic (PCTL). PRISM model checker based automated quantitative analysis elicits the process of the estimation of various parameters in the model using PFSMs and PCTL formulas.

Driver behavior has long been considered as particularly relevant for the development of automotive applications, especially that recently these applications are increasingly trying to adapt to the driver. However, drivers behave differently in the different traffic situations, hence the need of techniques to enable cars to learn from their drivers and create a model of his behavior. Actually, future generation of cars will be equipped with all sorts of sensing, computing and communication devices that will allow them to acquire all information about the state of the vehicle, the driver and the environment. And, hence make easier the driving behavior learning process. The present paper addresses the problem of modeling and learning the behavior of a driver in an intelligent car by presenting an approach for the construction and verification of a learned driving model. First, we propose a new way for modeling the driver-vehicle and environment, which consists of considering driver-vehicle as a rectangular hybrid input output automaton while representing contextual information about driving environment as conditions on the automaton variables. The construction of the model is ensured through a continuous monitoring of the driver-vehicle and environment system. The use of rectangular predicate states and environmental conditions will facilitate the verification of driving behavior. We then present a formal verification of properties of the constructed model expressed in Probabilistic Computational Tree Logic (PCTL) to assess its convenience to different traffic situations.

Energy consumption has become an essential factor in designing modern computer system architecture. Because of physical limits, the termination of Moore’s law and Dennard’s scaling has forced the computer design community to investigate new approaches to meet the requirements for computing resources. Approximate computing has emerged as a promising method for reducing energy consumption while trading a controllable quality loss. This paper asserts that an approximated cache coherence protocol preserves overall energy for computation. We can approximate the cache coherence protocol by adding approximated cache lines to a certain level without hindering the output. This paper introduces an enhanced approximated version of the MSI (Modified Shared Invalid) cache coherence protocol MSI-A (Modified Shared Invalid-Approx). We have verified MSI-A and MSI by employing LTL specifications in the NuSMV model checker. To illustrate the benefits of MSI-A, we have added DTMC (Discrete-Time Markov Chain) with PCTL (Probabilistic Computational Tree Logic). Although the PCTL proves the theory of approximation, we have also simulated the MSI-A in the TEJAS hardware simulator on PARSEC 3.0 to investigate the energy gains and cycle gains of MSI-A in varied applications. The cache lines considered to be approx are between 10 and 30 percent. Each application benefited from approximation according to its nature, and VIPS has indicated a total energy gain of 30.18 percent.

Partially Observable Markov Decision Process (POMDP) has been widely used in the robotics to model uncertainties from sensors, actuators and the environment. However, such comprehensiveness makes the planning in POMDP generally very difficult. Existing work often searches for an optimal control policy with respect to predefined reward functions, which may require a large memory and is computationally expensive. We propose to use formal methods and learn a Deterministic Finite Automaton (DFA) as a supervisor to regulate the behavior of a Partially Observable Markov Decision Process (POMDP), such that it satisfies the given specification in Probabilistic Computation Tree Logic (PCTL). For such a purpose, we modify the L* learning algorithm and define oracles for membership queries and conjectures. We further show that the termination and correctness of the design algorithm are guaranteed. A simple example is used for illustration.

The ability to reason about and respond to their own emotional states can enhance the believability of Non-Player Characters (NPCs). In this paper, we use a Partially Observable Markov Decision Process (POMDP)-based framework to model emotion over time. A two-level appraisal model, involving quick and reactive vs. slow and deliberate appraisals, is proposed for the creation of affective autonomous characters based on POMDPs, wherein the probability of goal satisfaction is used in an appraisal and reappraisal process for emotion generation. We not only extend Probabilistic Computation Tree Logic (PCTL) for reasoning about the properties of emotional states based on POMDPs but also illustrate how four reactive (primary) emotions and nine deliberate (secondary) emotions can be derived by combining PCTL with the belief-desire theory of emotion. The results of an empirical study suggest that the proposed model can be used to create characters that appear to be more believable and more intelligent.

This study proposes a hierarchical probabilistic computation tree logic, HpCTL, which is an extension of the standard probabilistic computation tree logic pCTL, as a theoretical basis for hierarchical probabilistic CTL model checking. Hierarchical probabilistic model checking is a new paradigm that can appropriately verify hierarchical randomized (or stochastic) systems. Furthermore, a probability-measure-independent translation from HpCTL into pCTL is defined, and a theorem for embedding HpCTL into pCTL is proved using this translation. Finally, the relative decidability of HpCTL with respect to pCTL is proved using this embedding theorem. These embedding and relative decidability results allow us to reuse the standard pCTL-based probabilistic model checking algorithms to verify hierarchical randomized systems that can be described using HpCTL.

Dynamic production control in parallel processing systems under process queue time constraints

Traffic control performance on networks depends on the flow response to the policy adopted, which in turn contributes to determine the optimal signal settings. This paper focuses on the relationship between local and network wide traffic control policies within the combined traffic control and assignment problem. Through a full exploration of the solution space, an in depth cross comparison is performed between the well-known local policies P0 and Equisaturation, versus the global policies Maximum Throughput and Minimum Delay, to verify how the two local policies approximate the optimal settings for signalized intersections. Realistic traffic dynamics, such as congestion, multiple controllers and spillback are considered, to empirically determine the conditions under which the local policies are able to approximate global performances. After presenting the different local and global control policies, experiments are performed on simple toy networks. The complexity of the underlying network and, therefore, of the problems' boundary conditions is then increased, allowing us to showcase how the different metrics perform in different situations. Finally, conclusions on the results are drawn.

Port supply chains play a very important role in the process of economic globalization. Lack of trust of the mechanism is the main factor in restricting the development of port supply chains. Blockchains have great potential to solve the trust problem among all participants using port supply chains, which can reduce costs and improve efficiency. As the bridge between blockchains and port supply chains, smart contracts reconstruct the business process of blockchain-empowered port supply chains. In this article, we present an architecture of a consortium blockchain-empowered port supply chain system, and propose a system verification framework for the smart contracts of port supply chains with probabilistic behaviors. The smart contracts are modeled as DTMCs (Discrete-Time Markov Chains), which are automatically transformed through the BPMN (Business Process Model and Notation) description of the smart contracts. The requirements are specified by PCTL (Probabilistic Computation Tree Logic). Moreover, we implement the customs clearance process of the Shanghai Yangshan Port based on blockchain Hyperledger Fabric, and reconstruct the clearance process with smart contracts. We use it to demonstrate the effectiveness of this framework, and identify the smart contracts that do not meet the expected needs of users.

According to our survey about the express delivery from hundreds of campuses, we find that the location of commodity storage places impacts the delivery success rate and service quality. Thus, how to select the proper delivery location is vital to logistics enterprises, which contributes to improve the work efficiency and reduce delivery costs. In this paper, probabilistic model checking is used to verify the delivery fetch system, which evaluates the solution of location distributions in a quantitative way. First, it formalizes the fetch process of express delivery system between business and customer in the form of Discrete-Time Markov Chain (DTMC) when considering the stochastic behavior. Second, Probabilistic Computation Tree Logic (PCTL) is introduced as the verification property to the temporal behavior checking. Third, formal verifications are conducted by the supporting tool PRISM concerning on the transition probabilities computing. Furthermore, verification results are proven to follow the law of Bernouli Large Numbers, which aims to illustrate that the simulation result is close to the actual express delivery situation. Fourth, the express delivery location model is extended as new structure Co-DTMC in order to calculate time consumption and cost consumption, where the punishment factor is designed for the purpose of optimization. Finally, experiments are carried out to show that our approach can effectively select an optimal place for express delivery, which provides an alternative solution to integrate time and cost into the location selection.

In this paper, we propose a novel top-down design method for the development of collective behaviors of swarm robotics systems called property-driven design. Swarm robotics systems are usually designed and developed using a code-and-fix approach, that is, the developer devises, tests and modifies the individual robot behaviors until a desired collective behavior is obtained. The code-and-fix approach can be very time consuming and relies completely on the ingenuity and expertise of the designer. The idea of property-driven design is that a swarm robotics system can be described by specifying formally a set of desired properties. In an iterative process similar to test-driven development, the developer produces a model of the system that satisfies the desired properties. Subsequently, the system is implemented in simulation and using real robots. Property-driven design helps to minimize the risk of developing a system that does not satisfy the required properties, and to promote the reuse of hardware independent models. In this paper, we start by giving a general description of the method. We then present a possible way to apply it by using Discrete Time Markov Chains (DTMC) and Probabilistic Computation Tree Logic* (PCTL*). Finally, we conclude by presenting the application of the proposed method to the design and development of a swarm robotics system performing aggregation.