Frequency adjusted multi-agent Q-learning

This article shows that seemingly diverse implementations of multi-agent reinforcement learning share the same basic building block in their learning dynamics: a mathematical term that is closely related to the gradient of the expected reward. Specifically, two independent branches of multi-agent learning research can be distinguished based on their respective assumptions and premises. The first branch assumes that the value function of the game is known to all players, which is then used to update the learning policy based on Gradient Ascent. Notable algorithms in this branch include Infinitesimal Gradient Ascent (IGA) [7], the variation Win or Learn Fast IGA (WoLF) [3] and the Weighted Policy Learner [1]. The second branch of multi-agent learning is concerned with learning in unknown environments, using interaction-based Reinforcement Learning, and contains those algorithms which have been shown to be formally connected to the replicator equations of Evolutionary Game Theory. In this case, the learning agent updates its policy based on a sequence of pairs that indicate the quality of the actions taken. Notable algorithms include Cross Learning (CL) [4], Regret Minimization (RM) [6], and Frequency Adjusted Q-learning (FAQ) [5]. This article demonstrates inherent similarities between these diverse families of algorithms by comparing their underlying learning dynamics, derived as the continuous time limit of their policy updates. These dynamics have already been investigated for algorithms from each family separately [1, 2, 3, 5, 6, 7], however, they have not yet been discussed in context of the relation to each other, and the origin of their similarity has not been discussed satisfactorily. In addition to the formal derivation, directional field plots of the learning dynamics in representative classes of two-player two-action games illustrate the similarities and strengthen the theoretical findings.

In this paper, we investigate multi-agent learning (MAL) in a multi-agent resource selection problem (MARS) in which a large group of agents are competing for common resources. Since agents in such a setting are self-interested, MAL in MARS domains typically focuses on the convergence to a set of non-cooperative equilibria. As seen in the example of prisoner's dilemma, however, selfish equilibria are not necessarily optimal with respect to the natural objective function of a target problem, e.g., resource utilization in the case of MARS. Conversely, a centrally administered optimization of physically distributed agents is infeasible in many real-life applications such as transportation traffic problems. In order to explore the possibility for a middle ground solution, we analyze two types of costs for evaluating MAL algorithms in this context. The quality loss of a selfish algorithm can be quantitatively measured by the price of anarchy, i.e., the ratio of the objective function value of a selfish solution to that of an optimal solution. Analogously, we introduce the price of monarchy of a learning algorithm to quantify the practical cost of coordination in terms of communication cost. We then introduce a multi-agent social learning approach named A Few Good Agents (AFGA) that motivates self-interested agents to cooperate with one another to reduce the price of anarchy, while bounding the price of monarchy at the same time. A preliminary set of experiments on the El Farol bar problem, a simple example of MARS, show promising results.

This paper investigates a relatively new direction in Multiagent Reinforcement Learning. Most multiagent learning techniques focus on Nash equilibria as elements of both the learning algorithm and its evaluation criteria. In contrast, we propose a multiagent learning algorithm that is optimal in the sense of finding a best-response policy, rather than in reaching an equilibrium. We present the first learning algorithm that is provably optimal against restricted classes of non-stationary opponents. The algorithm infers an accurate model of the opponentýs non-stationary strategy, and simultaneously creates a best-response policy against that strategy. Our learning algorithm works within the very general framework of n-player, general-sum stochastic games, and learns both the game structure and its associated optimal policy.

An agent must learn and adapt quickly when playing against other agents. This process is challenging in particular when playing in stochastic environments against other learning agents. In this paper, we introduce a fast and adaptive learning algorithm for repeated stochastic games (FAL-SG). FAL-SG utilizes lossy game abstraction to reduce the state space of the game and facilitate learning and adapting rapidly. We analyze FAL-SG's performance by proving bounds on the abstraction loss and prediction mistakes and show that FAL-SG satisfies three criteria prescribed for multiagent learning algorithms. We successfully establish the robustness and scalability of FAL-SG with extensive theoretical and experimental results.

Learning in multi-agent settings has recently garnered much interest, the result of which has been the development of somewhat effective multi-agent learning (MAL) algorithms for repeated normal-form games. However, general-purpose MAL algorithms for richer environments, such as general-sum repeated stochastic (Markov) games (RSGs), are less advanced. Indeed, previously created MAL algorithms for RSGs are typically successful only when the behavior of associates meets specific game theoretic assumptions and when the game is of a particular class (such as zero-sum games). In this paper, we present a new algorithm, called Pepper, that can be used to extend MAL algorithms designed for repeated normal-form games to RSGs. We demonstrate that Pepper creates a family of new algorithms, each of whose asymptotic performance in RSGs is reminiscent of its asymptotic performance in related repeated normal-form games. We also show that some algorithms formed with Pepper outperform existing algorithms in an interesting RSG.

The key technology concerning the dispatch and control of the cyber-physical-social systems (CPSS) integration and group machine learning (ML) based Web-of-Cells (WoC) is systematically investigated, aiming to develop an intelligent dispatching system that has a high penetration of distributed generations (DG). Based on practical engineering demands and the weakly-centralized WoC, which is characterized by self-organized co-evolution, high independence, high-efficiency synergy, and autonomous learning, a variety of advanced theoretical tools such as complex network theory, group ML, evolutionary game theory, and CPSS-based parallel system theory have been adopted to address the following key issue: How can we achieve overall optimal dispatching and control decision-making in a class of complex systems relying on a large number of group cells with characteristics of limited information, weak controllability, small capacity, and wide distribution? Starting from this, four basic scientific issues are discussed: 1) a modeling method for a self-organized coupled network of the CPSS integration-based WoC; 2) a stability analysis and stability control system of the self-organized evolution of the WoC; 3) a highly autonomous group intelligent decision (GID) method of an independent cell; 4) multi-cell synergetic evolutionary game and GID theory. Hence, an innovative breakthrough on the intersection of complex network game theory and group ML is expected to be obtained, contributing to the emergence of group knowledge in complex circumstances and a significant improvement in the level of GID. Lastly, based on the theoretical investigation in this paper, explorations on the development of the CPSS platform for the WoC as well as its engineering practice in application are conducted. Furthermore, the in-depth development of the WoC, as well as its anticipated technical challenges, are prospected and analyzed with the hope of applying it on practical smart distribution grid demonstration projects in the future.

What evolutionary game theory tells us about multiagent learning

Solving complex but structured problems in a decentralized manner via multiagent collaboration has received much attention in recent years. This is natural, as on one hand, multiagent systems usually possess a structure that determines the allowable interactions among the agents; and on the other hand, the single most pressing need in a cooperative multiagent system is to coordinate the local policies of autonomous agents with restricted capabilities to serve a system-wide goal. The presence of uncertainty makes this even more challenging, as the agents face the additional need to learn the unknown environment parameters while forming (and following) local policies in an online fashion. In this paper, we provide the first Bayesian reinforcement learning (BRL) approach for distributed coordination and learning in a cooperative multiagent system by devising two solutions to this type of problem. More specifically, we show how the Value of Perfect Information (VPI) can be used to perform efficient decentralised exploration in both model-based and model-free BRL, and in the latter case, provide a closed form solution for VPI, correcting a decade old result by Dearden, Friedman and Russell. To evaluate these solutions, we present experimental results comparing their relative merits, and demonstrate empirically that both solutions outperform an existing multiagent learning method, representative of the state-of-the-art.

Coordinated multi-agent reinforcement learning (MARL) provides a promising approach to scaling learning in large cooperative multi-agent systems. Distributed constraint optimization (DCOP) techniques have been used to coordinate action selection among agents during both the learning phase and the policy execution phase (if learning is off-line) to ensure good overall system performance. However, running DCOP algorithms for each action selection through the whole system results in significant communication among agents, which is not practical for most applications with limited communication bandwidth. In this paper, we develop a learning approach that generalizes previous coordinated MARL approaches that use DCOP algorithms and enables MARL to be conducted over a spectrum from independent learning (without communication) to fully coordinated learning depending on agents' communication bandwidth. Our approach defines an interaction measure that allows agents to dynamically identify their beneficial coordination set (i.e., whom to coordinate with) in different situations and to trade off its performance and communication cost. By limiting their coordination set, agents dynamically decompose the coordination network in a distributed way, resulting in dramatically reduced communication for DCOP algorithms without significantly affecting overall learning performance. Essentially, our learning approach conducts co-adaptation of agents' policy learning and coordination set identification, which outperforms approaches that sequence them.

Reinforcement learning (RL) is an efficient learning method for Markov decision processes (MDPs); ant colony system (ACS) is an efficient method for solving combinatorial optimization problems. Based on the update policy of reinforcement values in RL and the cooperating method of the indirect media communication in ACS, this paper proposes the Q-ACS multi-agent cooperating learning method for the learning agents to share episodes beneficial to the exploitation of the accumulated knowledge and to utilize the learned reinforcement values efficiently. Further, taking the visited times into account, this paper proposes the T-ACS multi-agent learning method that the learning agents share better policies beneficial to the exploration during agent's learning processes. Meanwhile, in the light of the indirect media communication among heterogeneous multi-agents, this paper presents a heterogeneous multi-agent RL method, the D-ACS. The agents in our methods are given a simply cooperating way exchanging information in the form of reinforcement values updated in the common model of all agents. Owning the advantages of exploring the unknown environment actively and exploiting learned knowledge effectively, the proposed methods are able to solve both MDPs and combinatorial optimization problems effectively. To results of simulations on the hunter game and the traveling salesman problem, this paper discusses the role of the indirect media communication on the multi-agent cooperation learning system and analyzes its efficiency. The results of experiments also demonstrate that our methods perform competitively with representative methods on each domain respectively.

Two multi-agent policy iteration learning algorithms are proposed in this work. The two proposed algorithms use the exponential moving average approach along with the Q-learning algorithm as a basis to update the policy for the learning agent so that the agent's policy converges to a Nash equilibrium policy. The first proposed algorithm uses a constant learning rate when updating the policy of the learning agent, while the second proposed algorithm uses two different decaying learning rates. These two decaying learning rates are updated based on either the Win-or-Learn-Fast (WoLF) mechanism or the Win-or-Learn-Slow (WoLS) mechanism. The WoLS mechanism is introduced in this article to make the algorithm learn fast when it is winning and learn slowly when it is losing. The second proposed algorithm uses the rewards received by the learning agent to decide which mechanism (WoLF mechanism or WoLS mechanism) to use for the game being learned. The proposed algorithms have been theoretically analyzed and a mathematical proof of convergence to pure Nash equilibrium is provided for each algorithm. In the case of games with mixed Nash equilibrium, our mathematical analysis shows that the second proposed algorithm converges to an equilibrium. Although our mathematical analysis does not explicitly show that the second proposed algorithm converges to a Nash equilibrium, our simulation results indicate that the second proposed algorithm does converge to Nash equilibrium. The proposed algorithms are examined on a variety of matrix and stochastic games. Simulation results show that the second proposed algorithm converges in a wider variety of situations than state-of-the-art multi-agent reinforcement learning algorithms.

Extract Large and complex datasets collecting digital breadcrumbs of human behaviour have become non-negligible sources for social science researchers, students, and practitioners. ‘Big data’ has made a place for itself as a commodity in academic, commercial, and non-commercial settings. There have been numerous new descriptions for defining social scientists with quantitative expertise: empirical social scientist, computational social scientist, quantitative social scientists, social data scientist, etc. Nonetheless, a social scientist with solid big data skills is not yet a common profile in academia and research institutions. Neither is a computer scientist with broad interdisciplinary skills in social sciences. As the amount of published material increases exponentially in many disciplines, specialisation seems to be a reasonable response to keep up with the state of the art, and to produce technically and conceptually solid research. Interdisciplinary teams are vital in providing the critical synthesis from these separate specialisations, and this book is put together to help bridge the language gap between the different specialisations needed for migration analysis from big data sources.

Quantitative finance has had a long tradition of a bottom-up approach to complex systems inference via multi-agent systems (MAS). These statistical tools are based on modelling agents trading via a centralised order book, in order to emulate complex and diverse market phenomena. These past financial models have all relied on so-called zero-intelligence agents, so that the crucial issues of agent information and learning, central to price formation and hence to all market activity, could not be properly assessed. In order to address this, we designed a next-generation MAS stock market simulator, in which each agent learns to trade autonomously via reinforcement learning. We calibrate the model to real market data from the London Stock Exchange over the years 2007 to 2018, and show that it can faithfully reproduce key market microstructure metrics, such as various price autocorrelation scalars over multiple time intervals. Agent learning thus enables accurate emulation of the market microstructure as an emergent property of the MAS.

Selection as a unifying process in the biological and social sciences

Normative systems are a widely used framework to coordinate interdependent activities in multi-agent systems. Most research in this area has focused on how to compute normative systems that effectively accomplish a coordination task, as well as additional criteria such as synthesising norms that do not over-regulate a system, and the emergence of norms that remain stable over time. We introduce a framework for the synthesis of stable normative systems that are sufficient and necessary for coordination. Our approach is based on ideas from evolutionary game theory. We simulate multi-agent systems in which useful norms are more likely to prosper than useless norms. We empirically show the effectiveness of our approach in a simulated traffic domain.