Inverse two-player zero-sum dynamic games

Replicator dynamics of the Hawk-Dove game with different stochastic noises in infinite populations

In this chapter, we present simulation examples and an experimental case study illustrating the application of the inverse optimal control and noncooperative dynamic game theory that we have developed in previous chapters. In the first part of this chapter, we illustrate and compare bilevel and minimum-principle methods for solving a simulated inverse noncooperative dynamic game problem. We also present several examples, including failure cases, illustrating the importance of key steps in the minimum-principle methods for both inverse optimal control and inverse noncooperative dynamic (and differential) games. The second part of this chapter is concerned with an experimental case study involving the use of inverse methods to identify human driver behavior for advanced driver assistance technology.

A logic Petri net model for dynamic multi-agent game decision-making

A new concept of strong conflict equilibrium is proposed that supplements the well-known fundamental system of conflict equilibria and considerably increases the possibility of finding a unique strongest equilibrium (solution) in any game problem. The efficiency of this new equilibrium is illustrated by static and dynamic game problems.

Online Partially Model-Free Solution of Two-Player Zero Sum Differential Games

This chapter focuses on the computation of the saddle-point equilibrium of a zero-sum discrete time dynamic game in a state-feedback policy. It begins by considering solution methods for two-player zero sum dynamic games in discrete time, assuming a finite horizon stage-additive cost that Player 1 wants to minimize and Player 2 wants to maximize, and taking into account a state feedback information structure. The discussion then turns to discrete time dynamic programming, the use of MATLAB to solve zero-sum games with finite state spaces and finite action spaces, and discrete time linear quadratic dynamic games. The chapter concludes with a practice exercise that requires computing the cost-to-go for each state of the tic-tac-toe game, and the corresponding solution.

For motion planning and verification, as well as for model-based control, it is important to have an accurate dynamic system model. In many cases, friction is a significant component of the model. When considering the solution of the rigid-body forward dynamics problem in systems such as robots, we can often appeal to the fundamental theorem of differential equations to prove the existence and uniqueness of the solution. It is known, however, that when Coulomb friction is added to the dynamic equations, the forward dynamic solution may not exist and if it exists, it is not necessarily unique. In these situations, the inverse dynamic problem is ill-posed as well. Thus there is the need to understand the nature of the existence and uniqueness problems and to know under what conditions these problems arise. In this paper, we show that even single degree of freedom systems can exhibit existence and uniqueness problems. Next, we introduce compliance in the otherwise rigid-body model. This has two effects. First, the forward and inverse problems become well-posed. Thus we are guaranteed unique solutions. Secondly, we show that the extra dynamic solutions associated with the rigid model are unstable.

This paper presents an inverse problem in analytical dynamics. The inverse problem is to construct the Lagrangian when the integrals of a system are given. Firstly, the differential equations are obtained by using the time derivative of the integrals. Secondly, the differential equations can be written in the Lagrange equations under certain conditions and the Lagrangian can be obtained. Finally, two examples are given to illustrate the application of the result.

In this chapter, we present inverse noncooperative dynamic games as an extension and generalization of discrete-time inverse optimal control. We specifically investigate the problem of computing the parameters in the cost functions of players in noncooperative dynamic games from data. We pose inverse noncooperative dynamic game problems for both open-loop or feedback Nash equilibrium solution concepts and for situations in which the available data consists of whole or truncated state and control sequences. We present methods for solving these inverse problems that exploit either bilevel optimization or conditions for the existence of Nash equilibria derived from discrete-time minimum principles. For open-loop Nash equilibria, we show that minimum-principle methods reduce to linear or quadratic programs under certain parameterizations of the class of player cost functions (as in our treatment of discrete-time inverse optimal control). We also establish conditions under which these open-loop methods are guaranteed to compute unique player cost-function parameters. Finally, we highlight the difficulty of pursuing minimum-principle methods of inverse noncooperative dynamic games for the feedback Nash equilibrium solution concept, before providing a method and associated uniqueness results for the important special case of infinite-horizon linear-quadratic feedback noncooperative dynamic games.

The purpose of the present work is to study whether genetic algorithms can solve an inverse field problem in magnetic fluid dynamics (MFD) efficiently. We have investigated how the interface between two fluids of different conductivity in a highly simplified model of an aluminum electrolysis cell can be reconstructed by means of external magnetic field measurements. The knowledge of the interface deformation can be used to prevent undesired instabilities in aluminum reduction cells .

In this chapter, differential games are studied for continuous-time linear and nonlinear systems, including two-player zero-sum games, multi-player zero-sum games, and multi-player nonzero-sum games, via a series of adaptive dynamic programming (ADP) approaches. First, an integral policy iteration algorithm is developed to learn online the Nash equilibrium of two-player zero-sum differential games with completely unknown continuous-time linear dynamics using the state and control data. Second, multi-player zero-sum differential games for a class of continuous-time uncertain nonlinear systems are solved by using a novel iterative ADP algorithm. Via neural network modeling for the system dynamics, the ADP technique is employed to obtain the optimal control pair iteratively so that the iterative value function reaches the optimal solution of the zero-sum differential games. Finally, an online synchronous approximate optimal learning algorithm based on policy iteration is developed to solve multi-player nonzero-sum games of continuous-time nonlinear systems without the requirement of exact knowledge of system dynamics.

Off-policy evaluation (OPE) is the problem of evaluating new policies using historical data obtained from a different policy. In the recent OPE context, most studies have focused on single-player cases, and not on multi-player cases. In this study, we propose OPE estimators constructed by the doubly robust and double reinforcement learning estimators in two-player zero-sum Markov games. The proposed estimators project exploitability that is often used as a metric for determining how close a policy profile (i.e., a tuple of policies) is to a Nash equilibrium in two-player zero-sum games. We prove the exploitability estimation error bounds for the proposed estimators. We then propose the methods to find the best candidate policy profile by selecting the policy profile that minimizes the estimated exploitability from a given policy profile class. We prove the regret bounds of the policy profiles selected by our methods. Finally, we demonstrate the effectiveness and performance of the proposed estimators through experiments.

In this paper, we mainly study the simplification and improvement of Santilli's methods in Birkhoffian system, which is a more general type of basic dynamic system. The theories and methods of Birkhoffian dynamics have been used in hadron physics, quantum physics, rotational relativity theory, and fractional dynamics. As is well known, Lagrangian inverse problem, Hamiltonian inverse problem, and Birkhoffian inverse problem are the main objects of the dynamic inverse problems. The results given by Douglas (Douglas J 1941 Trans. Amer. Math. Soc. 50 71) and Havas[Havas P 1957 Nuovo Cimento Suppl. Ser. X5 363] show that only the self-adjoint Newtonian systems can be represented by Lagrange's equations, so the Lagrangian inverse problem is not universal for a holonomic constrained mechanical system. Furthermore, from the equivalence between Lagrange's equation and Hamilton's equation, Hamiltonian inverse problem is not universal. A natural question is then raised:whether there exists a self-adjoint dynamical model whose inverse problem is universal for holonomic constrained mechanical systems, in the field of analytical mechanics.An in-depth study of this issue in the 1980s by R. M. Santilli shows that a universal self-adjoint model exists for a holonomic constrained mechanic system that satisfies the basic conditions of locality, analyticity, and formality. The Birkhoff's equation is a natural extension of the Hamilton's equation, which shows the geometric properties of a nonconservative system as a general symplectic structure. This more general symplectic structure provides the geometry for the study of the non-conservative system preserving structure algorithms. Therefore, it is particularly important to study the problem of the Birkhoffian representation for the holonomic constrained system.For the inverse problem of Birkhoff's dynamics, studied mainly are the condition under which the mechanical systems can be represented by Birkhoff's equations and the construction method of Birkhoff's functions. However, due to the extensiveness and complexity of the holonomic nonconservative system, Birkhoff's dynamical functions do not have so simple construction method as Lagrange function and Hamilton function. The research results of this issue are very few. The existing construction methods are mainly for three constructions proposed by Santilli[Santilli R M 1983 Foundations of Theoretical Mechanics Ⅱ (New York:Springer-Verlag) pp25-28], and there are still many technical problems to be solved in the applications of these methods.In order to solve these problems, this article mainly focuses on the following content. First, according to the existence theorem of Cauchy-Kovalevskaya type equations, we prove that the autonomous system always has an autonomous Birkhoffian representation. Second, a more concise method is given to prove that Santilli's second method can be simplified. An equivalent relationship implied in Santilli's third method is found, an improved Santilli's third method is proposed, and the MATLAB programmatic calculation of the method is studied. Finally, the full text is summarized and the results are discussed.

This brief proposes a game-theoretic inverse reinforcement learning (GT-IRL) framework, which aims to learn the parameters in both the dynamic system and individual cost function of multistage games from demonstrated trajectories. Different from the probabilistic approaches in computer science community and residual minimization solutions in control community, our framework addresses the problem in a deterministic setting by differentiating Pontryagin's maximum principle (PMP) equations of open-loop Nash equilibrium (OLNE), which is inspired by Jin et al. (2020). The differentiated equations for a multi-player nonzero-sum multistage game are shown to be equivalent to the PMP equations for another affine-quadratic nonzero-sum multistage game and can be solved by some explicit recursions. A similar result is established for two-player zero-sum games. Simulation examples are presented to demonstrate the effectiveness of our proposed algorithms.

We consider a dynamic inverse problem for a dynamical system describing propagation of waves in a Krein string. We reduce the dynamical system to the integral equation and consider the important special case when the density of a string is given by a finite number of point masses distributed on the interval. We derive the Krein-type equation and solve the dynamic inverse problem in this particular case. We also consider the approximation of constant density by point-mass densities uniformly distributed on the interval and the effect of appearing of the finite speed of a wave propagation in the dynamical system.