Decentralized Bayesian reinforcement learning for online agent collaboration

Coordinated multi-agent reinforcement learning (MARL) provides a promising approach to scaling learning in large cooperative multi-agent systems. Distributed constraint optimization (DCOP) techniques have been used to coordinate action selection among agents during both the learning phase and the policy execution phase (if learning is off-line) to ensure good overall system performance. However, running DCOP algorithms for each action selection through the whole system results in significant communication among agents, which is not practical for most applications with limited communication bandwidth. In this paper, we develop a learning approach that generalizes previous coordinated MARL approaches that use DCOP algorithms and enables MARL to be conducted over a spectrum from independent learning (without communication) to fully coordinated learning depending on agents' communication bandwidth. Our approach defines an interaction measure that allows agents to dynamically identify their beneficial coordination set (i.e., whom to coordinate with) in different situations and to trade off its performance and communication cost. By limiting their coordination set, agents dynamically decompose the coordination network in a distributed way, resulting in dramatically reduced communication for DCOP algorithms without significantly affecting overall learning performance. Essentially, our learning approach conducts co-adaptation of agents' policy learning and coordination set identification, which outperforms approaches that sequence them.

In this paper, we investigate multi-agent learning (MAL) in a multi-agent resource selection problem (MARS) in which a large group of agents are competing for common resources. Since agents in such a setting are self-interested, MAL in MARS domains typically focuses on the convergence to a set of non-cooperative equilibria. As seen in the example of prisoner's dilemma, however, selfish equilibria are not necessarily optimal with respect to the natural objective function of a target problem, e.g., resource utilization in the case of MARS. Conversely, a centrally administered optimization of physically distributed agents is infeasible in many real-life applications such as transportation traffic problems. In order to explore the possibility for a middle ground solution, we analyze two types of costs for evaluating MAL algorithms in this context. The quality loss of a selfish algorithm can be quantitatively measured by the price of anarchy, i.e., the ratio of the objective function value of a selfish solution to that of an optimal solution. Analogously, we introduce the price of monarchy of a learning algorithm to quantify the practical cost of coordination in terms of communication cost. We then introduce a multi-agent social learning approach named A Few Good Agents (AFGA) that motivates self-interested agents to cooperate with one another to reduce the price of anarchy, while bounding the price of monarchy at the same time. A preliminary set of experiments on the El Farol bar problem, a simple example of MARS, show promising results.

This paper investigates a relatively new direction in Multiagent Reinforcement Learning. Most multiagent learning techniques focus on Nash equilibria as elements of both the learning algorithm and its evaluation criteria. In contrast, we propose a multiagent learning algorithm that is optimal in the sense of finding a best-response policy, rather than in reaching an equilibrium. We present the first learning algorithm that is provably optimal against restricted classes of non-stationary opponents. The algorithm infers an accurate model of the opponentýs non-stationary strategy, and simultaneously creates a best-response policy against that strategy. Our learning algorithm works within the very general framework of n-player, general-sum stochastic games, and learns both the game structure and its associated optimal policy.

In this paper, we deal with the sequential decision making problem of agents operating in computational economies, where there is uncertainty regarding the trustworthiness of service providers populating the environment. Specifically, we propose a generic Bayesian trust model, and formulate the optimal Bayesian solution to the exploration-exploitation problem facing the agents when repeatedly interacting with others in such environments. We then present a computationally tractable Bayesian reinforcement learning algorithm to approximate that solution by taking into account the expected value of perfect information of an agent's actions. Our algorithm is shown to dramatically outperform all previous finalists of the international Agent Reputation and Trust (ART) competition, including the winner from both years the competition has been run.

This article shows that seemingly diverse implementations of multi-agent reinforcement learning share the same basic building block in their learning dynamics: a mathematical term that is closely related to the gradient of the expected reward. Specifically, two independent branches of multi-agent learning research can be distinguished based on their respective assumptions and premises. The first branch assumes that the value function of the game is known to all players, which is then used to update the learning policy based on Gradient Ascent. Notable algorithms in this branch include Infinitesimal Gradient Ascent (IGA) [7], the variation Win or Learn Fast IGA (WoLF) [3] and the Weighted Policy Learner [1]. The second branch of multi-agent learning is concerned with learning in unknown environments, using interaction-based Reinforcement Learning, and contains those algorithms which have been shown to be formally connected to the replicator equations of Evolutionary Game Theory. In this case, the learning agent updates its policy based on a sequence of pairs that indicate the quality of the actions taken. Notable algorithms include Cross Learning (CL) [4], Regret Minimization (RM) [6], and Frequency Adjusted Q-learning (FAQ) [5]. This article demonstrates inherent similarities between these diverse families of algorithms by comparing their underlying learning dynamics, derived as the continuous time limit of their policy updates. These dynamics have already been investigated for algorithms from each family separately [1, 2, 3, 5, 6, 7], however, they have not yet been discussed in context of the relation to each other, and the origin of their similarity has not been discussed satisfactorily. In addition to the formal derivation, directional field plots of the learning dynamics in representative classes of two-player two-action games illustrate the similarities and strengthen the theoretical findings.

In this paper we develop a Bayesian policy search approach for Multi-Agent RL (MARL), which is model-free and allows for priors on policy parameters. We present a novel optimization algorithm based on hybrid MCMC, which leverages both the prior and gradient information estimated from trajectories. Our experiments demonstrate the automatic discovery of roles through reinforcement learning in a real-time strategy game.

Integrating Business Intelligence (BI) processes in an information system requires a form of strategic scanning system for which the information is the main source of efficiency and decision support. A process of strategic scanning system network is primarily a cooperative approach to sharing knowledge that actors are producers of information. The dynamics of the actor’s interactions allow gradual building of shared knowledge. This paper proposes Multi Agent System (MAS) architecture which facilitates the integration of a process of strategic scanning system network in the information system, to emerge relevant information from simple information while ensuring the quality and safety information . In particular, this approach is geared towards supporting system properties specially focused on cooperative multi-agent system. It gives finally an overview of implementation of a prototype of the proposed solution limited for the moment to the integration of processes most used in the majority of information systems.

State-of-the-art multi-agent reinforcement learning has achieved remarkable success in recent years. The success has been mainly based on the assumption that all teammates perfectly cooperate to optimize a global objective in order to achieve a common goal. While this may be true in the ideal case, these approaches could fail in practice, since in multi-agent systems (MAS), all agents may be a potential source of failure. In this presentation, we focus on resilience in cooperative MAS and propose an Antagonist-Ratio Training Scheme (ARTS) by reformulating the original target MAS as a mixed cooperative-competitive game between a group of protagonists which represent agents of the target MAS and a group of antagonists which represent failures in the MAS. While the protagonists can learn robust policies to ensure resilience against failures, the antagonists can learn malicious behavior to provide an adequate test suite for other MAS. We empirically evaluate ARTS in a cyber physical production domain and show the effectiveness of ARTS w.r.t. resilience and testing capabilities.

Learning in multi-agent settings has recently garnered much interest, the result of which has been the development of somewhat effective multi-agent learning (MAL) algorithms for repeated normal-form games. However, general-purpose MAL algorithms for richer environments, such as general-sum repeated stochastic (Markov) games (RSGs), are less advanced. Indeed, previously created MAL algorithms for RSGs are typically successful only when the behavior of associates meets specific game theoretic assumptions and when the game is of a particular class (such as zero-sum games). In this paper, we present a new algorithm, called Pepper, that can be used to extend MAL algorithms designed for repeated normal-form games to RSGs. We demonstrate that Pepper creates a family of new algorithms, each of whose asymptotic performance in RSGs is reminiscent of its asymptotic performance in related repeated normal-form games. We also show that some algorithms formed with Pepper outperform existing algorithms in an interesting RSG.

Multi-agent learning is a crucial method to control or find solutions for systems, in which more than one entity needs to be adaptive. In today's interconnected world, such systems are ubiquitous in many domains, including auctions in economics, swarm robotics in computer science, and politics in social sciences. Multi-agent learning is inherently more complex than single-agent learning and has a relatively thin theoretical framework supporting it. Recently, multi-agent learning dynamics have been linked to evolutionary game theory, allowing the interpretation of learning as an evolution of competing policies in the mind of the learning agents. The dynamical system from evolutionary game theory that has been linked to Q-learning predicts the expected behavior of the learning agents. Closer analysis however allows for two interesting observations: the predicted behavior is not always the same as the actual behavior, and in case of deviation, the predicted behavior is more desirable. This discrepancy is elucidated in this article, and based on these new insights Frequency Adjusted Q- (FAQ-) learning is proposed. This variation of Q-learning perfectly adheres to the predictions of the evolutionary model for an arbitrarily large part of the policy space. In addition to the theoretical discussion, experiments in the three classes of two-agent two-action games illustrate the superiority of FAQ-learning.

Cooperative robotics has gained considerable traction in the industrial sector with the expeditious use of multi-agent systems. Containment control is an emergent topic in cooperative multi-agent systems, with applications in areas such as warehouse robotics, inventory management, and transportation. Containment control is an amalgamation of formation and consensus control, which predominantly uses groups of multiple agents assigned as leaders and followers. This survey paper provides a comprehensive overview of key topics in containment control of multi-agent systems ranging from containment concepts to controllers. The paper discusses the role of communication and network topologies, the importance of accurate sensor measurements and agent dynamics, and various controller design and stability analysis techniques. The paper concludes by highlighting the promising research directions and challenges in the field along with real-world applications. Overall, this survey paper provides a valuable resource for researchers, practitioners, and students interested in the development of advanced multi-agent systems with containment control.

This paper gives aspects related to a cooperation scenario in the framework of multiagent systems. The presentation is focused on a multiagent system that consists of two agents, the Master and the Apprentice. The theoretical basis of the cooperation scenario is the definition of the most probable process, and two algorithms are used with this regard. The formulation of the cooperation scenario is exemplified for a case study that builds an architecture of successively placed bricks in the workspace.

Designing mechanisms that leverage cooperation between agents has been a long-lasting goal in Multiagent Systems. The task is especially challenging when agents are selfish, lack common goals and face social dilemmas, i.e., situations in which individual interest conflicts with social welfare. Past works explored mechanisms that explain cooperation in biological and social systems, providing important clues for the aim of designing cooperative artificial societies. In particular, several works show that cooperation is able to emerge when specific network structures underlie agents' interactions. Notwithstanding, social dilemmas in which defection is highly tempting still pose challenges concerning the effective sustainability of cooperation. Here we propose a new redistribution mechanism that can be applied in structured populations of agents. Importantly, we show that, when implemented locally (\textiti.e., agents share a fraction of their wealth surplus with their nearest neighbors), redistribution excels in promoting cooperation under regimes where, before, defection prevailed.

The formation control problem of intelligent driving vehicles originates from the research on task planning and cooperation of multi-agent system, which is mainly aimed at how multi-intelligent driving vehicles cooperate to complete multi-task team driving behaviors such as formation preparation, formation maintenance, formation change and obstacle avoidance in traffic environment. From the control point of view, The vehicle queue is controlled by a plurality of single vehicle nodes, and individual vehicles are controlled through information interaction among the nodes, and then coupled with each other to form a dynamic system, so that the vehicle queue forms a multi-agent system, which is modeled and analyzed to realize an adaptive formation control model based on MPC and particle swarm optimization algorithm. Finally, Collaborative control of intelligent vehicle formation is simulated to achieve coordinated control of vehicle queue based on multi-agent system.

In this paper we will examine two main aspects of trust dynamics: a) How direct experiences involving trust, with their successes or failures, influence the future trust of an agent about similar facts. We challenge the trivial idea that always success increases trust while failure decreases it. Of course, this primitive view cannot be avoided till Trust is modeled just as a simple index, a dimension, a number; for example reduced to mere subjective probability. We claim that a cognitive attribution process is needed in order to update trust on the basis of an 'interpretation' of the outcome of A's reliance on B and of B's performance (failure or success). b) How the fact that A trusts B and relies on it in situation Ω can actually (objectively) influence B's trustworthiness in the /Spl Omega/ situation. Either trust is a selffulfilling prophecy that modifies the subjective probability of the predicted event; or it is a self-defeating strategy by negatively influencing the events. These phenomena are very crucial in human societies (states, market, groups), but also in computer mediated Organizations, Interactions (EC), Cooperation (CSCW) and even in Multi-Agent Systems with autonomous agents. We present a formal model of these dynamic nontrivial aspects.