Individual-Level Inverse Reinforcement Learning for Mean Field Games

Abstract

Translate Article

**Read paper on the following link:** https://ifaamas.org/Proceedings/aamas2022/pdfs/p253.pdf **Abstract:** The recent mean field game (MFG) formalism has enabled the application of inverse reinforcement learning (IRL) methods in large-scale multi-agent systems, with the goal of inferring reward signals that can explain demonstrated behaviours of large populations. The existing IRL methods for MFGs are built upon reducing an MFG to a Markov decision process (MDP) defined on the collective behaviours and average rewards of the population. However, this paper reveals that the reduction from MFG to MDP holds only for the fully cooperative setting. This flaw invalidates existing IRL methods on MFGs with non-cooperative environments. To measure more general behaviours in large populations, we study the use of individual behaviours to directly infer ground-truth reward functions for MFGs. We propose Mean Field IRL (MFIRL), the first IRL framework for MFGs that can handle both cooperative and non-cooperative environments. Based on this theoretically justified framework, we develop a practical algorithm effective for MFGs with continuous state-action spaces and unknown dynamics. We evaluate MFIRL on both cooperative and mixed cooperative-competitive scenarios with many agents. Results show that MFIRL excels in reward recovery, sample efficiency and robustness to changing dynamics, compared to the method based on the reduction to MDP.