The Effect of Congestion Frequency and Saturation on Coordinated Traffic Routing

Coordinated multi-agent reinforcement learning (MARL) provides a promising approach to scaling learning in large cooperative multi-agent systems. Distributed constraint optimization (DCOP) techniques have been used to coordinate action selection among agents during both the learning phase and the policy execution phase (if learning is off-line) to ensure good overall system performance. However, running DCOP algorithms for each action selection through the whole system results in significant communication among agents, which is not practical for most applications with limited communication bandwidth. In this paper, we develop a learning approach that generalizes previous coordinated MARL approaches that use DCOP algorithms and enables MARL to be conducted over a spectrum from independent learning (without communication) to fully coordinated learning depending on agents' communication bandwidth. Our approach defines an interaction measure that allows agents to dynamically identify their beneficial coordination set (i.e., whom to coordinate with) in different situations and to trade off its performance and communication cost. By limiting their coordination set, agents dynamically decompose the coordination network in a distributed way, resulting in dramatically reduced communication for DCOP algorithms without significantly affecting overall learning performance. Essentially, our learning approach conducts co-adaptation of agents' policy learning and coordination set identification, which outperforms approaches that sequence them.

Complete algorithms have been proposed to solve problems modelled as distributed constraint optimization (DCOP). However, there are only few attempts to address real world scenarios using this formalism, mainly because of the complexity associated with those algorithms. In the present work we compare three complete algorithms for DCOP, aiming at studying how they perform in complex and dynamic scenarios of increasing sizes. In order to assess their performance we measure not only standard quantities such as number of cycles to arrive to a solution, size and quantity of exchanged messages, but also computing time and quality of the solution which is related to the particular domain we use. This study can shed light in the issues of how the algorithms perform when applied to problems other than those reported in the literature (graph coloring, meeting scheduling, and distributed sensor network).

The establishment of a common operating picture through shared situation awareness is critical for the management of first responder operations. Situation awareness is the knowledge of the state and physical location of people, objects and terrain within an area of interest. First responder operations can be quite complex, involving the coordination of and collaboration among teams distributed over large areas. Adding to this complexity is the use of heterogeneous networks with diverse radios, traffic flows and applications. This paper describes some initial results using distributed constraint optimization (DCOP) techniques to facilitate situation awareness for first responders. A brief background on DCOP is presented, along with how DCOP can be applied to enable situation awareness. An example demonstrating the application of DCOP algorithms to a particular situation awareness problem is presented. Finally, some critical issues that arise when utilizing DCOP are described, along with some approaches to addressing these issues. By integrating these DCOP techniques with situation awareness tools, we believe that many of the information distribution, interoperability, and networking difficulties faced in the first responder environment can be overcome.

The Distributed Pseudo-tree Optimization Procedure (DPOP) is a well-known message passing algorithm that provides optimal solutions to Distributed Constraint Optimization Problems (DCOPs) in cooperative multi-agent systems. However, the traditional DCOP formulation does not consider constraints that must be satisfied (hard constraints), rather it concentrates only on constraints that place no restriction on satisfaction (soft constraints). This is a serious shortcoming as many real-world applications involve both types of constraints. Traditional DPOP algorithms are not able to benefit from the existence of hard constraints, where an additional calculation is required to handle such constraints. This results in longer runtimes. Thus scalability remains an issue. Additionally, in the standard DPOP, the agents are arranged as a Depth First Search (DFS) pseudo-tree, but recent work has shown that the construction of pseudo-trees in this way often leads to chain-like communication structures that greatly impair the algorithm’s performance. To address these issues, we develop an algorithm that speeds up the DPOP algorithm by reducing the size of the messages exchanged and increases parallelism in the pseudo tree. For this purpose, initially, we improve the path for exchanging messages. Next, we introduce a new form of constraint propagation, which we call cross-edge consistency. Our theoretical evaluation shows that our proposed algorithm is complete and correct. In empirical evaluations, our algorithm achieves a significant reduction in the runtime, ranging from 4% to 96%, compared to the state-of-the-art.

A vessel rotation is a sequence in which a vessel visits the different terminals in a large port for loading and unloading containers. Constructing a vessel rotation involves making a lot of appointments through communication. Nowadays, the communication between vessel operator and terminal operator is done through telephone, fax, and e-mail based on the operator’s knowledge and experience. With this practice, it happens frequently that appointments cannot be met either by the vessel operator or by the terminal operator. In this paper we propose a technique to solve the vessel rotation planning problem in an automated way. For this, we propose to model the problem as a distributed constraint optimization problem (DCOP). DCOP has generated much interest in the artificial intelligence and constraint programming communities, and a number of algorithms have been developed for solving DCOPs. However, DCOP algorithms have been rarely used in practice. This paper presents a first approach based on DCOP for generating reasonable and optimized rotation plans for vessel operators. To evaluate the performance of the proposed approach, different DCOP algorithms are considered. We evaluate the solution quality and computational and communication costs of these algorithms when solving the vessel rotation planning problem.

We propose a novel incomplete cooperative algorithm for distributed constraint optimization problems (DCOPs) denoted as Cooperative Constraint Approximation (CoCoA). The key strategy of the algorithm is to use a semi-greedy approach in which knowledge is distributed amongst neighboring agents, and assigning a value only once instead of an iterative approach. Furthermore, CoCoA uses a unique-first approach to improve the solution quality. It is designed such that it can solve DCOPs as well as Asymmetric DCOPS, with only few messages being communicated between neighboring agents. Experimentally, through evaluating graph coloring problems, randomized (A)DCOPs, and a sensor network communication problem, we show that CoCoA is able to very quickly find solutions of high quality with a smaller communication overhead than state-of-the-art DCOP solvers such as DSA, MGM-2, ACLS, MCS-MGM and Max-Sum. In our asymmetric use case problem of a sensor network, we show that CoCoA not only finds the best solution, but also finds this solution faster than any other algorithm.

Distributed Constraint Optimization Problems (DCOPs) have been used to model a number of multi-agent coordination problems. In DCOPs, agents are assumed to have complete information about the utility of their possible actions. However, in many real-world applications, such utilities are stochastic due to the presence of exogenous events that are beyond the direct control of the agents. This paper addresses this issue by extending the standard DCOP model to Expected Regret DCOP (ER-DCOP) for DCOP applications with uncertainty in constraint utilities. Different from other approaches, ER-DCOPs aim at minimizing the overall expected regret of the problem. The paper proposes the ER-DPOP algorithm for solving ER-DCOPs, which is complete and requires a linear number of messages with respect to the number of agents in the problem. We further present two implementations of ER-DPOP---GPU- and ASP-based implementations---that orthogonally exploit the problem structure and present their evaluations on random networks and power network problems.

Distributed Constraint Optimization Problems (DCOPs) can be used to model a number of multi-agent coordination problems. The conventional DCOP model assumes that the subproblem that each agent is responsible for (i.e. the mapping of nodes in the constraint graph to agents) is part of the model description. While this assumption is often reasonable, there are many applications where there is some flexibility in making this assignment. In this paper, we focus on this gap and make the following contributions: (1) We formulate this problem as an optimization problem, where the goal is to find an assignment that minimizes the completion time of the DCOP algorithm (e.g. Action-GDL or Max-Sum) that operates on this mapping. (2) We propose a novel heuristic, called MNA, that can be executed in a centralized or decentralized manner. (3) Our empirical evaluation illustrates a substantial reduction in completion time, ranging from 16% to 40%, without affecting the solution quality of the algorithms, compared to the current state of the art. In addition, we observe empirically that the completion time obtained from our approach is near-optimal; it never exceeds more than 10% of what can be achieved from the optimal node-to-agent mapping.

Distributed constraint optimization problems (DCOPs) are well-suited for modeling multi-agent coordination problems where the primary interactions are between local subsets of agents. However, one limitation of DCOPs is the assumption that the constraint rewards are without uncertainty. Researchers have thus extended DCOPs to Stochastic DCOPs (SDCOPs), where rewards are sampled from known probability distribution reward functions, and introduced algorithms to find solutions with the largest expected reward. Unfortunately, such a solution might be very risky, that is, very likely to result in a poor reward. Thus, in this paper, we make three contributions: (1) we propose a stricter objective for SDCOPs, namely to find a solution with the most stochastically dominating probability distribution reward function; (2) we introduce an algorithm to find such solutions; and (3) we show that stochastically dominating solutions can indeed be less risky than expected reward maximizing solutions.

Emergency responders are faced with a number of significant challenges when managing major disasters. First, the number of rescue tasks posed is usually larger than the number of responders (or agents) and the resources available to them. Second, each task is likely to require a different level of effort in order to be completed by its deadline. Third, new tasks may continually appear or disappear from the environment, thus requiring the responders to quickly recompute their allocation of resources. Fourth, forming teams or coalitions of multiple agents from different agencies is vital since no single agency will have all the resources needed to save victims, unblock roads and extinguish the fires which might erupt in the disaster space. Given this, coalitions have to be efficiently selected and scheduled to work across the disaster space so as to maximize the number of lives and the portion of the infrastructure saved. In particular, it is important that the selection of such coalitions should be performed in a decentralized fashion in order to avoid a single point of failure in the system. Moreover, it is critical that responders communicate only locally given they are likely to have limited battery power or minimal access to long-range communication devices. Against this background, we provide a novel decentralized solution to the coalition formation process that pervades disaster management. More specifically, we model the emergency management scenario defined in the RoboCup Rescue disaster simulation platform as a coalition formation with spatial and temporal constraints (CFST) problem where agents form coalitions to complete tasks, each with different demands. To design a decentralized algorithm for CFST, we formulate it as a distributed constraint optimization problem and show how to solve it using the state-of-the-art Max-Sum algorithm that provides a completely decentralized message-passing solution. We then provide a novel algorithm (F-Max-Sum) that avoids sending redundant messages and efficiently adapts to changes in the environment. In empirical evaluations, our algorithm is shown to generate better solutions than other decentralized algorithms used for this problem.

The Distributed Stochastic Algorithm (DSA) is a distributed hill-climbing technique for solving large Distributed Constraint Optimization Problems (DCOPs) such as distributed scheduling, resource allocation, and distributed route planning. The best known version of DSA, DSA-B, works by having agents change their assignments with probability p when making that change will improve their solution (a hill-climbing move). To escape local minima, DSA-B performs a lateral escape move by switching to another equally good value with the same probability p. It is unclear why hill climbing and escape moves are chosen with the same probability. We investigate the performance effects of making these moves with different probabilities, pH and pL. Through empirical evaluation, we discover that the efficiency of DSA can not only be considerably improved, but can be more specifically tuned to a particular domain or user's needs when these two move types are considered separately. Our work also shows that DSA can outperform both DBA and DPP when it is properly tuned.

Distributed Constraint Optimization (DCOP) is rapidly emerging as a prominent technique for multiagent coordination. Unfortunately, rigorous quantitative evaluations of privacy loss in DCOP algorithms have been lacking despite the fact that agent privacy is a key motivation for applying DCOPs in many applications. Recently, Maheswaran et al. [3, 4] introduced a framework for quantitative evaluations of privacy in DCOP algorithms, showing that early DCOP algorithms lose more privacy than purely centralized approaches and questioning the motivation for applying DCOPs. Do state-of-the art DCOP algorithms suffer from a similar shortcoming? This paper answers that question by investigating the most efficient DCOP algorithms, including both DPOP and ADOPT.

Distributed Constraint Satisfaction (DisCSP) and Distributed Constraint Optimization (DCOP) are formal frameworks that can be used to model a variety of problems in which multiple decision-makers cooperate towards a common goal: from computing an equilibrium of a game, to vehicle routing problems, to combinatorial auctions. In this thesis, we independently address two important issues in such multi-agent problems: 1) how to provide strong guarantees on the protection of the privacy of the participants, and 2) how to anticipate future, uncontrollable events. On the privacy front, our contributions depart from previous work in two ways. First, we consider not only constraint privacy (the agents' private costs) and decision privacy (keeping the complete solution secret), but also two other types of privacy that have been largely overlooked in the literature: agent privacy, which has to do with protecting the identities of the participants, and topology privacy, which covers information about the agents' co-dependencies. Second, while previous work focused mainly on quantitatively measuring and reducing privacy loss, our algorithms provide stronger, qualitative guarantees on what information will remain secret. Our experiments show that it is possible to provide such privacy guarantees, while still scaling to much larger problems than the previous state of the art. When it comes to reasoning under uncertainty, we propose an extension to the DCOP framework, called DCOP under Stochastic Uncertainty (StochDCOP), which includes uncontrollable, random variables with known probability distributions that model uncertain, future events. The problem becomes one of making optimal offline decisions, before the true values of the random variables can be observed. We consider three possible concepts of optimality: minimizing the expected cost, minimizing the worst-case cost, or maximizing the probability of a-posteriori optimality. We propose a new family of StochDCOP algorithms, exploring the tradeoffs between solution quality, computational and message complexity, and privacy. In particular, we show how discovering and reasoning about co-dependencies on common random variables can yield higher-quality solutions.

Distributed Constraint Optimization Problems (DCOPs) are a powerful tool to model multi-agent coordination problems that are distributed by nature. The formulation is suitable for problems where variables are discrete and constraint utilities are represented in tabular form. However, many real-world applications have variables that are continuous and tabular forms thus cannot accurately represent constraint utilities. To overcome this limitation, researchers have proposed the Continuous DCOP (C-DCOP) model, which are DCOPs with continuous variables. But existing approaches usually come with some restrictions on the form of constraint utilities and are without quality guarantees. Therefore, in this paper, we (i) propose an exact algorithm to solve a specific subclass of C-DCOPs; (ii) propose an approximation method with quality guarantees to solve general C-DCOPs; (iii) propose additional C-DCOP algorithms that are more scalable; and (v) empirically show that our algorithms outperform existing state-of-the-art C-DCOP algorithms when given the same communication limitations.

The Distributed Constraint Optimization Problem (DCOP) has been studied as a fundamental optimization problem that represents various problems on multiagent systems. We focus on the asymmetric DCOPs where each objective function is differently defined as an evaluation of an agent. This class of problems is studied as a multi-objective problem for the preferences of individual agents. In this work, we investigate the possibility of a solution framework based on relaxation methods as a scalable and inexact solution approach for this class of problems. We address a bottleneck problem that minimizes the worst-case cost value. As the first study, we apply a penalty method to the minimization problems of the maximum cost values.