Concave Utility Reinforcement Learning: The Mean-field Game Viewpoint

We present a simulation-based approach for solution of mean field games (MFGs), using the framework of empirical game-theoretical analysis (EGTA). Our primary method employs a version of the double oracle, iteratively adding strategies based on best response to the equilibrium of the empirical MFG among strategies considered so far. We present Fictitious Play (FP) and Replicator Dynamics as two subroutines for computing the empirical game equilibrium. Each subroutine is implemented with a query-based method rather than maintaining an explicit payoff matrix as in typical EGTA methods due to a representation issue we highlight for MFGs. By introducing game model learning and regularization, we significantly improve the sample efficiency of the primary method without sacrificing the overall learning performance. Theoretically, we prove that a Nash equilibrium (NE) exists in the empirical MFG and show the convergence of iterative EGTA to NE of the full MFG with either subroutine. We test the performance of iterative EGTA in various games and show that it outperforms directly applying FP to MFGs in terms of iterations of strategy introduction.

Mean Field Game systems describe equilibrium configurations in differential games with infinitely many infinitesimal interacting agents. We introduce a learning procedure (similar to the Fictitious Play) for these games and show its convergence when the Mean Field Game is potential.

The theory of mean field games aims at studying deterministic or stochastic differential games (Nash equilibria) as the number of agents tends to infinity. Since very few mean field games have explicit or semi-explicit solutions, numerical simulations play a crucial role in obtaining quantitative information from this class of models. They may lead to systems of evolutive partial differential equations coupling a backward Bellman equation and a forward Fokker–Planck equation. In the present survey, we focus on such systems. The forward-backward structure is an important feature of this system, which makes it necessary to design unusual strategies for mathematical analysis and numerical approximation. In this survey, several aspects of a finite difference method used to approximate the previously mentioned system of PDEs are discussed, including convergence, variational aspects and algorithms for solving the resulting systems of nonlinear equations. Finally, we discuss in details two applications of mean field games to the study of crowd motion and to macroeconomics, a comparison with mean field type control, and present numerical simulations.

A traffic system can be interpreted as a multiagent system, wherein vehicles choose the most efficient driving approaches guided by interconnected goals or strategies. This paper aims to develop a family of mean field games (MFG) for generic second-order traffic flow models (GSOM), in which cars control individual velocity to optimize their objective functions. GSOMs do not generally assume that cars optimize self-interested objectives, so such a game-theoretic reinterpretation offers insights into the agents’ underlying behaviors. In general, an MFG allows one to model individuals on a microscopic level as rational utility-optimizing agents while translating rich microscopic behaviors to macroscopic models. Building on the MFG framework, we devise a new class of second-order traffic flow MFGs (i.e., GSOM-MFG), which control cars’ acceleration to ensure smooth velocity change. A fixed-point algorithm with fictitious play technique is developed to solve GSOM-MFG numerically. In numerical examples, different traffic patterns are presented under different cost functions. For real-world validation, we further use an inverse reinforcement learning approach (IRL) to uncover the underlying cost function on the next-generation simulation (NGSIM) data set. We formulate the problem of inferring cost functions as a min-max game and use an apprenticeship learning algorithm to solve for cost function coefficients. The results show that our proposed GSOM-MFG is a generic framework that can accommodate various cost functions. The Aw Rascle and Zhang (ARZ) and Light-Whitham-Richards (LWR) fundamental diagrams in traffic flow models belong to our GSOM-MFG when costs are specified. History: This paper has been accepted for the Transportation Science Special Issue on ISTTT25 Conference. Funding: X. Di is supported by the National Science Foundation [CAREER Award CMMI-1943998]. E. Iacomini is partially supported by the Italian Research Center on High-Performance Computing, Big Data and Quantum Computing (ICSC) funded by MUR Missione 4-Next Generation EU (NGEU) [Spoke 1 “FutureHPC & BigData”]. C. Segala and M. Herty thank the Deutsche Forschungsgemeinschaft (DFG) for financial support [Grants 320021702/GRK2326, 333849990/IRTG-2379, B04, B05, and B06 of 442047500/SFB1481, HE5386/18-1,19-2,22-1,23-1,25-1, ERS SFDdM035; Germany’s Excellence Strategy EXC-2023 Internet of Production 390621612; and Excellence Strategy of the Federal Government and the Länder]. Support through the EU DATAHYKING is also acknowledged. This work was also funded by the DFG [TRR 154, Mathematical Modelling, Simulation and Optimization Using the Example of Gas Networks, Projects C03 and C05, Project No. 239904186]. Moreover, E. Iacomini and C. Segala are members of the Indam GNCS (Italian National Group of Scientific Calculus).

This book describes the latest advances in the theory of mean field games, which are optimal control problems with a continuum of players, each of them interacting with the whole statistical distribution of a population. While it originated in economics, this theory now has applications in areas as diverse as mathematical finance, crowd phenomena, epidemiology, and cybersecurity. Because mean field games concern the interactions of infinitely many players in an optimal control framework, one expects them to appear as the limit for Nash equilibria of differential games with finitely many players as the number of players tends to infinity. The book rigorously establishes this convergence, which has been an open problem until now. The limit of the system associated with differential games with finitely many players is described by the so-called master equation, a nonlocal transport equation in the space of measures. After defining a suitable notion of differentiability in the space of measures, the authors provide a complete self-contained analysis of the master equation. Their analysis includes the case of common noise problems in which all the players are affected by a common Brownian motion. They then go on to explain how to use the master equation to prove the mean field limit. The book presents two important new results in mean field games that contribute to a unified theoretical framework for this exciting and fast-developing area of mathematics.

In this article, we consider mean field games between a dominating player and a group of representative agents, each of which acts similarly and also interacts with each other through a mean field term being substantially influenced by the dominating player. We first provide the general theory and discuss the necessary condition for the optimal controls and equilibrium condition by adopting adjoint equation approach. We then present a special case in the context of linear-quadratic framework, in which a necessary and sufficient condition can be asserted by stochastic maximum principle; we finally establish the sufficient condition that guarantees the unique existence of the equilibrium control. The proof of the convergence result of finite player game to mean field counterpart is provided in Appendix.

formula omitted]-Nash equilibrium in stochastic differential games with mean-field interaction and controlled jumps

We develop the fictitious play algorithm in the context of the linear programming approach for mean field games of optimal stopping and mean field games with regular control and absorption. This algorithm allows to approximate the mean field game population dynamics without computing the value function by solving linear programming problems associated with the distributions of the players still in the game and their stopping times/controls. We show the convergence of the algorithm using the topology of convergence in measure in the space of subprobability measures, which is needed to deal with the lack of continuity of the flows of measures. Numerical examples are provided to illustrate the convergence of the algorithm.

This paper studies the power allocation of an ultra-dense cellular network consisting of multiple full-duplex (FD) base stations (BSs) serving a large number of half-duplex (HD) user equipments (UEs) located in a wide geographical area. Each BS consists of a baseband unit (BBU) that is responsible for signal processing and BS control, and a radio remote unit (RRU) that corresponds to a radio transceiver remotely built closer to the UEs. We consider a wireless-powered cellular network in which the BBU can periodically charge the RRU. We model the energy efficiency and coverage optimization problem for this network as a mean field game. We consider the weighted energy efficiency of the BS as the main performance metrics and evaluate the optimal strategy that can be adopted by the FD BSs in an ultra-dense network setting. Based on the interference and network energy efficiency models of the mean field game theory, Nash equilibrium of our proposed game is derived. Utilizing solutions of Hamilton-Jacobi-Bellman (HJB) and Fokker-Planck-Kolmogorov (FPK) equations, a new power transmission strategy is developed for the BSs to optimize the energy efficiency of 5G cellular networks with full duplex transmissions. Simulation results indicate that the proposed strategy not only improves the energy efficiency but also ensures the average network coverage probability converges to a stable level.

In this paper, we study a generalization of reflected backward doubly stochastic differential equations (RBDSDEs) and present a link to a general mean field game. In our case, the RBDSDEs are associated with a lower optional not right continuous barrier. First, we establish the existence and uniqueness of a solution of such RBDSDEs. We then study a mean field game with a new type of common noise related to an electricity grid with storage allowing jumps and prove the existence of a mean field Nash equilibrium.

Mean field game theory was initiated a little more than 15 years ago with the aim of simplifying the search for Nash equilibria in games with a large number of weakly interacting players. Since then, a lot has been done. Numerous equilibrium existence results have been obtained, using different characterizations and in various contexts. The analysis of the master equation, which describes the evolution of the value of the game, has also seen significant progress, which has, for example, allowed establishing in certain cases the convergence of games with a finite number of players. However, mean field games remain of a complex nature. For instance, the typical lack of uniqueness of solutions raises selection issues that are still poorly understood. The objective of the note is to present some of the latest advances, as well as some avenues to address further challenging questions.

We propose a simple model of inter-bank borrowing and lending where the evolution of the log-monetary reserves of $N$ banks is described by a system of diffusion processes coupled through their drifts in such a way that stability of the system depends on the rate of inter-bank borrowing and lending. Systemic risk is characterized by a large number of banks reaching a default threshold by a given time horizon. Our model incorporates a game feature where each bank controls its rate of borrowing/lending to a central bank. The optimization reflects the desire of each bank to borrow from the central bank when its monetary reserve falls below a critical level or lend if it rises above this critical level which is chosen here as the average monetary reserve. Borrowing from or lending to the central bank is also subject to a quadratic cost at a rate which can be fixed by the regulator. We solve explicitly for Nash equilibria with finitely many players, and we show that in this model the central bank acts as a clearing house, adding liquidity to the system without affecting its systemic risk. We also study the corresponding Mean Field Game in the limit of large number of banks in the presence of a common noise.

We propose a new approach to mean field games with major and minor players. Our formulation involves a two player game where the optimization of the representative minor player is standard while the major player faces an optimization over conditional McKean–Vlasov stochastic differential equations. The definition of this limiting game is justified by proving that its solution provides approximate Nash equilibriums for large finite player games. This proof depends upon the generalization of standard results on the propagation of chaos to conditional dynamics. Because it is of independent interest, we prove this generalization in full detail. Using a conditional form of the Pontryagin stochastic maximum principle (proven in the Appendix), we reduce the solution of the mean field game to a forward–backward system of stochastic differential equations of the conditional McKean–Vlasov type, which we solve in the linear quadratic setting. We use this class of models to show that Nash equilibriums in our formulation can be different from those originally found in the literature.

In this paper, we consider discrete-time dynamic games of the mean-field type with a finite number $N$ of agents subject to an infinite-horizon discounted-cost optimality criterion. The state space...

An integral control formulation of mean field game based large scale coordination of loads in smart grids