Module checking of pushdown multi-agent systems

This monograph provides an overview of distributed online optimization in multi-agent systems. Online optimization approaches planning and decision problems from a robust learning perspective, where one learns through feedback from sequentially arriving costs, resembling a game between a learner (agent) and the environment. Recently, multi-agent systems have become important in diverse areas including smart power grids, communication networks, machine learning, and robotics, where agents work with decentralized data, costs, and decisions to collectively minimize a system-wide cost. In such settings, agents make distributed decisions and collaborate with neighboring agents through a communication network, leading to scalable solutions that often perform as well as centralized methods. The monograph offers a unified introduction, starting with fundamental algorithms for basic problems, and gradually covering state-of-the-art techniques for more complex settings. The interplay between individual agent learning rates, network structure, and communication complexity is highlighted in the overall system performance.

Reinforcement Learning has long been employed to solve sequential decision-making problems with minimal input data. However, the classical approach requires a long time to learn a suitable policy, especially in Multiagent Systems. The teacher-student framework proposes to mitigate this problem by integrating an advising procedure in the learning process, in which an experienced agent (human or not) can advise a student to guide her exploration. However, the teacher is assumed to be an expert in the learning task. We here propose an advising framework where multiple agents advise each other while learning in a shared environment, and the advisor is not expected to necessarily act optimally. Our experiments in a simulated Robot Soccer environment show that the learning process is improved by incorporating this kind of advice.

Reinforcement learning, a machine learning technique, has already shown advancement in solving complex sequential decision-making problems. Many tasks involve multiple agents and require sequential decision-making policies to achieve common goals, such as warehouse automation, autonomous driving, and game-playing. To derive the policies for all agents, these problems can be modelled as multi-agent systems and addressed by multi-agent reinforcement learning (MARL). However, optimizing policies in multi-agent scenarios presents significant challenges due to the intricate behaviours of multiple agents and the non-stationary nature of the environments’ complex dynamics. Firstly, optimizing policies in multi-agent scenarios presents significant challenges due to the complexity of multi-agent behaviours, especially in partially observable environments. Also, the dynamic nature of agents’ behaviours and their interactions with other agents lead to changes in the environment’s states and agents’ observations over time, which is more complicated in open environments. Furthermore, the need to balance individual and collective objectives in certain real-world multi-agent environments also complicates the decision-making process. This doctoral thesis aims to tackle the following three fundamental multi-agent research problems and propose a solution for each. Our research covers from theoretical analysis to practical application. We start by studying the problem of learning an efficient policy under partially observable environments. We notice that a group of agents cooperates to com- pete against another group of agents (named opponents), with the limitation that information about the opponents is inaccessible due to partial observability. To address this issue, we propose a novel multi-agent distributional actor-critic algorithm to achieve speculative opponent modeling with purely local information (i.e., the controlled agent’s observations, actions, and rewards). The actor maintains a speculated belief of the opponents, which we call the speculative opponent models, to predict opponents’ actions using local observations and make decisions accordingly. The distributional critic models the return distribution of the policy. It reflects the quality of the actor and thus can guide the training of the imaginary opponent model that the actor relies on. Extensive experiments confirm that our method successfully models opponents’ behaviours without their data and delivers superior performance against baseline methods with a faster convergence speed. Furthermore, in some environments, teammates’ numbers and policies change with the market, requiring workers to adapt to perform different sets of tasks across time. To solve this issue, we propose an RL-based method that allows the controlled agent to collaborate with dynamic teammates in open environments. The controlled agent maintains a dual teamwork situation inference model to capture the current teamwork state and facilitate reasonable decision-making under partial observability. Considering the dynamic types of teammates, we first leverage the Chinese Restaurant Process-based model to categorize versatile teammate policies into distinct clusters, which improves the efficiency of identifying current teamwork situations. Next, to model the heterogeneous relationships among agents and accommodate the varying number of teammates, we apply heterogeneous graph attention neural networks to learn the representation of the teamwork situation. Extensive experiments confirm that our method outperforms state-of-the-art baselines with faster convergence in various ad hoc teamwork tasks. Lastly, in certain real-world applications, such as routing problems and warehouse management, decision-makers must balance overall benefits with individual fairness among agents. Achieving both learning efficiency and fairness concurrently presents a complex, multi-objective, joint-policy optimization challenge. Moreover, existing approaches are predominantly confined to simulation environments. To address the above issues, we present a pioneering MARL approach to balance individual and collective objectives for agents collaboration. Extensive experiments on synthetic and real-world datasets demonstrate that our method not only surpasses the state-of-the-art DRL methods, but also exhibits advantages over established heuristics, notably in achieving significantly faster optimization speeds. This method underlines the essential integration of fairness into real-world applications, representing a major advancement in creating equitable and efficient logistics solutions. To conclude, this doctoral thesis investigates three fundamental multi-agent decision-making research problems that are ubiquitous and unsolved. The proposed three MARL methodology solutions achieve efficient policy training and performance for agents in multi-agent environments with uncertainties raised by the partially observable, the open environment, and the individual-collective objectives of MARL. This thesis delves into novel designs of various critical components of MARL, including MDP formulations, policy networks, training algorithms, and inference methods. These contributions significantly elevate the effectiveness and efficiency of cooperative MARL, establishing new performance benchmarks.

Cooperative boolean games (CBG) are a family of coalitional games where agents may depend on each other for the satisfaction of their personal goals. In Dunne et al. (2008, Cooperative Boolean games. In Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2008), 1015–1022), the authors define as solution concept the notion of core showing that several decision problems, such as core-emptiness, are Π2p-complete. In this work, we investigate how to improve the computation of the core. In particular, we introduce two different types of dependence networks, abstract dependence networks and refined dependence networks, that are used to define the notion of stable coalitions and Δ-reduction, respectively. Stable coalitions enable to focus on a subset of the agents and use results to determinate the core of the whole game. Δ-reduction prunes the search space by returning a set of actions that are not admissible to be executed. We present an algorithm based on stable coalitions and a Δ-reduction implemented in Prolog and experimental results that show how they effectively improve the computation of the core.

As the structure of multi-agent systems (MAS) inherently meets the requirements of autonomy and decentralization for supply chains, there are many researchers engaged in application of the MAS architecture to assist to simulating and modeling supply chain dynamics, as well as solving some local decision problems, e.g. transportation planning, production scheduling, and etc.. However, our studies are focused on developing a MAS for supply chain management in the operational level, which integrates planning, execution and supervising. In our framework, intra-organizational task allocation and interorganizational task re-allocation are critical to effective coordination and scheduling in the supply chain. And we introduce a sealed combinatorial auction mechanism to solve this problem. In order to facilitate the mechanism, the components of different agents and the negotiation protocol supporting interactions between them are presented, and some related topics are also discussed in this paper.

In this article, we study cooperative multiagent systems (MASs) with multiple tasks by using reinforcement learning (RL)-based algorithms. The target for a single-agent RL system is represented by its scalar reward signals. However, for an MAS with multiple cooperative tasks, the holistic reward signal consists of multiple parts to represent the tasks, which makes the problem complicated. Existing multiagent RL algorithms search distributed policies with holistic reward signals directly, making it difficult to obtain an optimal policy for each task. This article provides efficient learning-based algorithms such that each agent can learn a joint optimal policy to accomplish these multiple tasks cooperatively with other agents. The main idea of the algorithms is to decompose the holistic reward signal for each agent into multiple parts according to the subtasks, and then the proposed algorithms learn multiple value functions with the decomposed reward signals and update the policy with the sum of distributed value functions. In addition, this article presents a theoretical analysis of the proposed approach. Finally, the simulation results for both discrete decision-making and continuous control problems have demonstrated the effectiveness of the proposed algorithms.

We present Probabilistic Doxastic Temporal (PDT) Logic, a formalism to represent and reason about probabilistic beliefs and their temporal evolution in multi-agent systems. This formalism enables the quantification of agents’ beliefs through probability intervals and incorporates an explicit notion of time. We discuss how over time agents dynamically change their beliefs in facts, temporal rules, and other agents’ beliefs with respect to any new information they receive. We introduce an appropriate formal semantics for PDT Logic and show that it is decidable. Alternative options of specifying problems in PDT Logic are possible. For these problem specifications, we develop different satisfiability checking algorithms and provide complexity results for the respective decision problems. The use of probability intervals enables a formal representation of probabilistic knowledge without enforcing (possibly incorrect) exact probability values. By incorporating an explicit notion of time, PDT Logic provides enriched possibilities to represent and reason about temporal relations.

We address the problem of learning temporal properties from the branching-time behavior of systems. Existing research in this field has mostly focused on learning linear temporal properties specified using popular logics, such as Linear Temporal Logic (LTL) and Signal Temporal Logic (STL). Branching-time logics such as Computation Tree Logic (CTL) and Alternating-time Temporal Logic (ATL), despite being extensively used in specifying and verifying distributed and multi-agent systems, have not received adequate attention. Thus, in this paper, we investigate the problem of learning CTL and ATL formulas from examples of system behavior. As input to the learning problems, we rely on the typical representations of branching behavior as Kripke structures and concurrent game structures, respectively. Given a sample of structures, we learn concise formulas by encoding the learning problem into a satisfiability problem, most notably by symbolically encoding both the search for prospective formulas and their fixed-point based model checking algorithms. We also study the decision problem of checking the existence of prospective ATL formulas for a given sample. We implement our algorithms in a Python prototype and have evaluated them to extract several common CTL and ATL formulas used in practical applications.

Communication is one of central activities in multiagent systems. It enables the knowledge sharing among multiple agents and improves the planning quality in a long run. In this paper, we study communication decision problems in the framework of interactive dynamic influence diagrams~(I-DIDs). I-DIDs are recognized probabilistic graphical models for sequential decision making in uncertain multiagent settings. We extend the representation to explicitly model communication actions as well as their relations to other variables in the domain. The challenging work is on developing an incentive mechanism that drives level 0 agents to learn communication while they act alone in a dynamic environment. We present solutions to the new model and show meaningful communication strategies in a multiagent problem domain.

Dialogue protocols have been the subject of considerable attention with respect to their potential applications in multiagent system environments. Formalisations of such protocols define classes of dialogue locutions, concepts of a dialogue state, and rules under which a dialogue proceeds. One important consideration in implementing a protocol concerns the criteria an agent should apply in choosing which utterance will constitute its next contribution: ideally, an agent should select a locution that (by some measure) optimises the outcome. The precise interpretation of optimise may vary greatly depending on the nature and intent of a dialogue area. One option is to choose the locution that results in a minimal length debate. We present a formal setting for considering the problem of deciding if a particular utterance is optimal in this sense and show that this decision problem is both np–hard and co-np–hard.

Distributed dynamic pricing of multiple perishable products using multi-agent reinforcement learning

Optimizing the operation of cooperative multi-agent systems that can deal with large and realistic problems has become an important focal area of research in the multi-agent community. In this paper, we first present a new model, the OC-DEC-MDP (Opportunity Cost Decentralized Markov Decision Process), that allows us to represent large multi-agent decision problems with temporal and precedence constraints. Then, we propose polynomial algorithms to efficiently solve problems formalized by OC-DEC-MDPs. The problems we deal with consist of a set of agents that have to execute a set of tasks in a cooperative way. The agents cannot communicate during task execution and they must respect resource and temporal constraints. Our approach is based on Decentralized Markov Decision Processes (DEC-MDPs) and uses the concept of opportunity cost borrowed from economics to obtain approximate control policies. Experimental results show that our approach produces good quality solutions for complex problems which are out of reach of existing approaches.

Bayesian-based Distributed Sequential Decision In Rail Transit Cognitive Radio

Learning automata are shown to be an excellent tool for creating learning multi-agent systems. Most algorithms used in current automata research expect the environment to end in an explicit end-stage. In this end-stage the rewards are given to the learning automata (i.e. Monte Carlo updating). This is however unfeasible in sequential decision problems with infinite horizon where no such end-stage exists. In this paper we propose a new algorithm based on one-step returns that uses bootstrapping to find good equilibrium paths in multi-stage games

The extension of rough set model is a crucial and vast research direction in rough set theory. Meanwhile decision making can be considered as a mental process in which human beings make a choice among several alternatives. However, with the increasing complexity of real decision making problems, the decision makers frequently face the challenge of characterizing their preferences in an uncertain context. In the present paper, we initiate a multi attribute group decision making problem in the presence of multi attribute and multi decision in decision making with preferences. We further present the concept of soft preference relation and soft dominance relation corresponding to decision attribute in the multi criteria and multi decision information system. Further we put forward the idea of two types of optimistic/pessimistic multigranulation (soft dominance based optimistic/pessimistic multigranulation decision theoretic) approximations and their applications in solving a multi agent conflict analysis decision problem. The proposed method addresses the limitations of the Pawlak model and Sun’s conflict analysis model and thus improve these models. Finally, the results on labor management negotiation problems show that the proposed algorithms are more effective and efficient for feasible consensus strategy when compared with other techniques.