In this paper, we introduce a regularized mean-field game and study learning of this game under an infinite-horizon discounted reward function. Regularization is introduced by adding a strongly concave regularization function to the one-stage reward function in the classical mean-field game model. We establish a value iteration based learning algorithm to this regularized mean-field game using fitted Q-learning. The regularization term in general makes reinforcement learning algorithm more robust to the system components. Moreover, it enables us to establish error analysis of the learning algorithm without imposing restrictive convexity assumptions on the system components, which are needed in the absence of a regularization term.

We analyze different scenarios of defensive medicine in a novel framework based on game theory and network analysis, where links in the network represent healing relationships between a physician and a patient. The physician should choose between providing the optimal treatment or an inferior one, which can amount to practicing defensive medicine. The patient should choose whether to litigate or not if an adverse event occurs. A major result of such analysis is that the steady state does not depend on the litigiousness of the initial system or the initial distribution of strategies among physicians or the distribution of patients over physicians. Moreover, reaching a virtuous steady state or an entirely defensive one appears to be independent of the fact that patients take into account the quality of treatments directly or they rely merely on popularity when choosing their physicians.

This paper addresses the problem of learning an equilibrium efficiently in general-sum Markov games through decentralized multi-agent reinforcement learning. Given the fundamental difficulty of calculating a Nash equilibrium (NE), we instead aim at finding a coarse correlated equilibrium (CCE), a solution concept that generalizes NE by allowing possible correlations among the agents’ strategies. We propose an algorithm in which each agent independently runs optimistic V-learning (a variant of Q-learning) to efficiently explore the unknown environment, while using a stabilized online mirror descent (OMD) subroutine for policy updates. We show that the agents can find an \(\epsilon \)-approximate CCE in at most \(\widetilde{O}( H^6S A /\epsilon ^2)\) episodes, where S is the number of states, A is the size of the largest individual action space, and H is the length of an episode. This appears to be the first sample complexity result for learning in generic general-sum Markov games. Our results rely on a novel investigation of an anytime high-probability regret bound for OMD with a dynamic learning rate and weighted regret, which would be of independent interest. One key feature of our algorithm is that it is decentralized, in the sense that each agent has access to only its local information, and is completely oblivious to the presence of others. This way, our algorithm can readily scale up to an arbitrary number of agents, without suffering from the exponential dependence on the number of agents.