Double-oracle algorithm for computing an exact nash equilibrium in zero-sum extensive-form games

We study the computation of equilibria of two-strategy anonymous games, via algorithms that may proceed via a sequence of adaptive queries to the game's payoff function, assumed to be unknown initially. The general topic we consider is query complexity, that is, how many queries are necessary or sufficient to compute an exact or approximate Nash equilibrium. We show that exact equilibria cannot be found via query-efficient algorithms. We also give an example of a 2-strategy, 3-player anonymous game that does not have any exact Nash equilibrium in rational numbers. Our main result is a new randomized query-efficient algorithm that finds a $$On^{-1/4}$$-approximate Nash equilibrium querying $$\tilde{O}n^{3/2}$$ payoffs and runs in time $$\tilde{O}n^{3/2}$$. This improves on the running time of pre-existing algorithms for approximate equilibria of anonymous games, and is the first one to obtain an inverse polynomial approximation in poly-time. We also show how this can be used to get an efficient PTAS. Furthermore, we prove that $$\varOmega n \log {n}$$ payoffs must be queried in order to find any $$\epsilon $$-well-supported Nash equilibrium, even by randomized algorithms.

We prove the existence of e-Nash equilibrium strategies with support logarithmic in the number of pure strategies. We also show that the payoffs to all players in any (exact) Nash equilibrium can be e-approximated by the payoffs to the players in some such logarithmic support e-Nash equilibrium. These strategies are also uniform on a multiset of logarithmic size and therefore this leads to a quasi-polynomial algorithm for computing an e-Nash equilibrium. To our knowledge this is the first subexponential algorithm for finding an e-Nash equilibrium. Our results hold for any multiple-player game as long as the number of players is a constant (i.e., it is independent of the number of pure strategies). A similar argument also proves that for a fixed number of players m, the payoffs to all players in any m-tuple of mixed strategies can be e-approximated by the payoffs in some m-tuple of constant support strategies.We also prove that if the payoff matrices of a two person game have low rank then the game has an exact Nash equilibrium with small support. This implies that if the payoff matrices can be well approximated by low rank matrices, the game has an e-equilibrium with small support. It also implies that if the payoff matrices have constant rank we can compute an exact Nash equilibrium in polynomial time.

The most profound reason why the Nash equilibrium concept has to be refined lies in the fact that a Nash equilibrium of an extensive form game may prescribe irrational behavior at unreached parts of the game tree. Until now, however, we have Mainly studied games in normal form. This comprehensive study yielded a deeper insight into the relations between various refinements of the Nash equilibrium concept. Our goal in this chapter is to investigate to what extend this study has yielded results which are also relevant for extensive form games. It will turn out that the insights we obtained are valuable, but that many results we proved for normal form games cannot be generalized to extensive form games.

Extensive-form games are a powerful framework for modeling sequential multi-agent interactions. In extensive-form games with imperfect information, Nash equilibria are generally used as a solution concept, but computing a Nash equilibrium can be intractable in large games. Instead, a variety of techniques are used to find strategies that approximate Nash equilibria. Traditionally, an approximate Nash equilibrium strategy is evaluated by measuring the strategy's worst-case performance, or exploitability. However, because exploitability fails to capture how likely the worst-case is to be realized, it provides only a limited picture of strategy strength, and there is extensive empirical evidence showing that exploitability can correlate poorly with one-on-one performance against a variety of opponents. In this thesis, we introduce a class of adaptive opponents called pretty-good responses that exploit a strategy but only have limited exploitative power. By playing a strategy against a variety of counter-strategies created with pretty-good responses, we get a more complete picture of strategy strength than that offered by exploitability alone. In addition, we show how standard no-regret algorithms can me modified to learn strategies that are strong against adaptive opponents. We prove that this technique can produce optimal strategies for playing against pretty-good responses. We empirically demonstrate the effectiveness of the technique by finding static strategies that are strong against Monte Carlo opponents who learn by sampling our strategy, including the UCT Monte Carlo tree search algorithm.

Multi-person stopping games with players’ priorities are considered. Players observe sequentially offers Y1,Y2,… at jump times T1,T2,… of a Poisson process. Y1,Y2,… are independent identically distributed random variables. Each accepted offer Yn results in a reward Gn=Ynr(Tn), where r is a non-increasing discount function. If more than one player wants to accept an offer, then the player with the highest priority (the lowest ordering) gets the reward. We construct Nash equilibrium in the multi-person stopping game using the solution of a multiple optimal stopping time problem with structure of rewards {Gn}. We compare rewards and stopping times of the players in Nash equilibrium in the game with the optimal rewards and optimal stopping times in the multiple stopping time problem. It is also proved that presented Nash equilibrium is a Pareto optimum of the game. The game is a generalization of the Elfving stopping time problem to multi-person stopping games with priorities.

Learning in Games by Random Sampling

I consider games with ambiguous payoffs played by non-Expected Utility decision makers. Three equilibrium solutions are studied. Nash equilibrium in which equilibrium mixed strategies must be best responses, Crawford equilibrium in beliefs and pure equilibrium in beliefs in which equilibrium strategies are mixtures of best responses, with the latter restricting best responses to pure actions. I study the interactions between ambiguity preferences on one side and equilibrium properties on the other. I show how the equilibrium concepts differ, computing necessary and sufficient conditions for existence and equivalence. I also show how these solution concepts fare against two fundamental principles of Nash equilibrium in standard games: the principle of indifference and the reduction principle. Given both are central to the computation of Nash equilibrium in games with Expected Utility players, their failure indicates how relaxing the Expected Utility hypothesis may disrupt standard game theoretic results such as the characterization of equilibria in two-player games.

About a symmetric three-players zero-sum game we will show the following results. 1. A modified version of Sion's minimax theorem with the coincidence of the maximin strategy and the minimax strategy are proved by the existence of a symmetric Nash equilibrium. 2. The existence of a symmetric Nash equilibrium is proved by the modified version of Sion's minimax theorem with the coincidence of the maximin strategy and the minimax strategy. Thus, they are equivalent. If a zero-sum game is asymmetric, maximin strategies and minimax strategies of players do not correspond to Nash equilibrium strategies. If it is symmetric, the maximin strategies and the minimax strategies constitute a Nash equilibrium. However, without the coincidence of the maximin strategy and the minimax strategy there may exist an asymmetric equilibrium in a symmetric three-players zero-sum game.

The economic dispatch problem for a Microgrid with independent prosumers, presents a market scenario where the prosumers act strategically maximizing their own benefits, and restricting strategic information such as energy production costs. In this context, the power allocation needs to follow a bidding process instead of a conventional economic dispatch process. Based on a game theory approach, a new average bidding algorithm is proposed, which achieves that the prosumer bids converge to an exact and efficient Nash equilibrium (ENE). At the ENE the power allocation minimizes the total energy cost in the Microgrid, and the set of bids maximizes the individual profit of each prosumer. Convergence conditions of the algorithm are presented, and it is shown that unlike of previous approaches the accuracy of the ENE does not depend on the step-size of the iterations. The performance of the bidding process is verified in numerical simulations, finding advantages of the proposed strategy respect to the Nash equilibrium seeking algorithm in terms of accuracy, and applicability for online operation.

Learning Nash equilibrium (NE) in complex zero-sum games with multi-agent reinforcement learning (MARL) can be extremely computationally expensive. Curriculum learning is an effective way to accelerate learning, but an under-explored dimension for generating a curriculum is the difficulty-to-learn of the subgames –games induced by starting from a specific state. In this work, we present a novel subgame curriculum learning framework for zero-sum games. It adopts an adaptive initial state distribution by resetting agents to some previously visited states where they can quickly learn to improve performance. Building upon this framework, we derive a subgame selection metric that approximates the squared distance to NE values and further adopt a particle-based state sampler for subgame generation. Integrating these techniques leads to our new algorithm, Subgame Automatic Curriculum Learning (SACL), which is a realization of the subgame curriculum learning framework. SACL can be combined with any MARL algorithm such as MAPPO. Experiments in the particle-world environment and Google Research Football environment show SACL produces much stronger policies than baselines. In the challenging hide-and-seek quadrant environment, SACL produces all four emergent stages and uses only half the samples of MAPPO with self-play. The project website is at https://sites.google.com/view/sacl-neurips.

Among other solution concepts, the notion of the pure Nash equilibrium plays a central role in Game Theory. Pure Nash equilibria in a game characterize situations with non-cooperative deterministic players in which no player has any incentive to unilaterally deviate from the current situation in order to achieve a higher payoff. Unfortunately, it is well known that there are games that do not have pure Nash equilibria. Furhermore, even in games where the existence of equilibria is guaranteed, their computation can be a computationally hard task. Such negative results significantly question the importance of pure Nash equilibria as solution concepts that characterize the behavior of rational players. Approximate pure Nash equilibria, which characterize situations where no player can significantly improve her payoff by unilaterally deviating from her current strategy, could serve as alternative solution concepts provided that they exist and can be computed efficiently. In this letter, we discuss recent positive algorithmic results for approximate pure Nash equilibria in congestion games.

We argue in favor of a departure from the equilibrium approach in game theory towards the less ambitious goal of describing only the actual behavior of rational players. The notions of Nash equilibrium and its refinements require a specification of the players’ choices and beliefs not only along the equilibrium play but also at counterfactual histories. We discuss an alternative—counterfactual-free—approach that focuses on choices and beliefs along the actual play, while being silent on choices and beliefs at unreached histories. Such an approach was introduced in an earlier paper that considered only perfect-information games. Here we extend the analysis to general extensive-form games (allowing for imperfect information) and put forward a behavioral notion of self-confirming play, which is close in spirit to the literature on self-confirming equilibrium. We also extend, to general extensive-form games, the characterization of rational play that is compatible with pure-strategy Nash equilibrium.

Abstract: Nash equilibrium is an important concept for studying human behavior in group decision making process. Given the complexity of finding Nash equilibria, computational tools are necessary to find them. Several programs were developed for this task. However, available programs are either not comprehensive or might be of difficult installation and handling, creating a “barrier of entry” to non-specialists. The aims of this research are twofold: (i) firstly, it was to identify and to discuss about the available programs for finding Nash equilibria; and (ii) secondly, based on the theoretical proprieties of a Nash equilibrium, to develop a program capable of finding all pure Nash equilibria in games with “n” players and “m” strategies (“n” and “m” being finite numbers) as a Macro tool for Microsoft Excel®. It is expected that the program can contributed to the area of Operations Research by providing a new tool that facilitate the use of game theory concepts within group decision-making problem-solving scenarios enabling practical applications using a widespread software.

Computational game theory has many applications in the modern world in both adversarial situations and the optimization of social good. While there exist many algorithms for computing solutions in two-player interactions, finding optimal strategies in multiplayer interactions efficiently remains an open challenge. This paper focuses on computing the multiplayer Team-Maxmin Equilibrium with Coordination device (TMECor) in zero-sum extensive-form games. TMECor models scenarios when a team of players coordinates ex ante against an adversary. Such situations can be found in card games (e.g., in Bridge and Poker), when a team works together to beat a target player but communication is prohibited; and also in real world, e.g., in forest-protection operations, when coordinated groups have limited contact during interdicting illegal loggers. The existing algorithms struggle to find a TMECor efficiently because of their high computational costs. To compute a TMECor in larger games, we make the following key contributions: (1) we propose a hybrid-form strategy representation for the team, which preserves the set of equilibria; (2) we introduce a column-generation algorithm with a guaranteed finite-time convergence in the infinite strategy space based on a novel best-response oracle; (3) we develop an associated-representation technique for the exact representation of the multilinear terms in the best-response oracle; and (4) we experimentally show that our algorithm is several orders of magnitude faster than prior state-of-the-art algorithms in large games.

Subgame perfect equilibria are specific Nash equilibria in perfect information games in extensive form. They are important because they relate to the rationality of the players. They always exist in infinite games with continuous real-valued payoffs, but may fail to exist even in simple games with slightly discontinuous payoffs. This article considers only games whose outcome functions are measurable in the Hausdorff difference hierarchy of the open sets (i.e. \({ {\Delta }}^0_2\) when in the Baire space), and it characterizes the families of linear preferences such that every game using these preferences has a subgame perfect equilibrium: the preferences without infinite ascending chains (of course), and such that for all players a and b and outcomes x, y, z we have \(\lnot (z <_a y <_a x \,\wedge \, x <_b z <_b y)\). Moreover at each node of the game, the equilibrium constructed for the proof is Pareto-optimal among all the outcomes occurring in the subgame. Additional results for non-linear preferences are presented.