Discrete-time stochastic minimax control of partially observable systems

Chapter 1 - A Very Short Tour of Game Theory

Learning algorithms for stochastic dynamic games

Chapter 3 - Moving from Static to Dynamic Games

In this paper we propose definitions of equivalence via stochastic bisimulation and of equivalence of stochastic external behavior for the class of discrete-time stochastic nonlinear control systems with possibly degenerate distributed disturbances. The notions are inspired by the analogue notions that exist for probabilistic chains and for behavioral systems, respectively. Connections between the two notions and also with stochastic reachability are discussed. For the subclass of discrete-time stochastic linear control systems with multivariate possibly degenerate normal distributed disturbances, geometric necessary and sufficient conditions for checking stochastic bisimulation equivalence and stochastic external behavior equivalence are derived. The proposed results nicely extend the theory of bisimulation equivalence and equivalence of external behavior known for non-deterministic control systems to a stochastic setting.

Robust and risk sensitive control of discrete time, finite state systems on an infinite horizon is considered, with either state or output feedback. The solution of the state feedback robust control problem is characterized in terms of the value of an average cost dynamic game. The risk sensitive stochastic optimal control problem is solved using the policy iteration algorithm, and the optimal rate is expressed in terms of the value of a stochastic dynamic game with average cost per unit time criterion. By taking a small noise limit a deterministic dynamic game is obtained, which is closely related to the robust control problem. For the problem with output feedback (partial state information) the analysis depends on introducing appropriate information states for the risk sensitive and robust control problems.

We extend earlier works on continuous potential games to the most general case: stochastic time varying environment, stochastic rewards, non-reduced form and constrained state-action sets. We provide conditions for a Markov Nash equilibrium (MNE) of the game to be equivalent to the solution of a single control problem. Then, we address the problem of learning this MNE when the reward and state transition models are unknown. We follow a reinforcement learning approach and extend previous algorithms for working with constrained state-action subsets of real vector spaces. As an application example, we simulate a network flow optimization model, in which the relays have batteries that deplete with a random factor. The results obtained with the proposed framework are close to optimal.

An examination is made of a stochastic dynamic discrete-time game, and the possibility of obtaining approximate Nash equilibria which result in Pareto outcomes is studied. The time horizon is infinite. In the context of a simple linear quadratic model strategies that may achieve the aforementioned aim are suggested, and several insights and results are discussed

Stochastic dynamic zero-sum two-person games are considered. The maximal total expected costs of the first player are the criterion where the maximum is taken on all of the strategies of the second player. Sufficient conditions of the existence of optimal resp. (μ, ∊)-optimal strategies are given by methods of dynamic programming.

Shadow Banks and Systemic Risks

We consider the robust (H???) control problem for linear time-invariant systems over unreliable communication channels that are subject to packet losses from sensors to the controller and from the controller to the actuators. Moreover, the acknowledgments of control packet transmissions are also sent over reverse unreliable communication channels, which is known as the quasi-TCP model. The robust control problem is formulated within a stochastic zero-sum dynamic game framework, from which we obtain an H??? controller that minimizes the worst-case cost function in the presence of an adversary in the dynamical system. We characterize a set of existence conditions of the H??? controller in terms of the disturbance attenuation parameter ?? and packet loss rates. For the least disturbance attenuation scenario (as ?? ??? 8), we show that the H??? control system converges to the corresponding LQG system. We also show that the H??? controller for the TCP- and UDP-cases can be arrived at as appropriate limits of the solution for the quasi-TCP problem. Numerical examples are presented to illustrate the theoretical results.

In stochastic control, information structure arguments have been crucial for stochastic analysis. Such an approach is often called static reduction in dynamic team theory (or decentralized stochastic control) and has been an effective method for establishing existence and approximation results for optimal policies. In this paper, we classify such static reductions into three categories: (i) those that are policy-independent (introduced by Witsenhausen in [17]), (ii) those that are policy-dependent (introduced by Ho and Chu [7], [8] for partially nested dynamic teams), and (iii) static measurement with control-sharing reduction (where the measurements become static although control actions are shared according to the partially nested information structure). For these reductions, while there exist bijection relationships between glob-ally optimal solutions of dynamic teams and their reductions, in general there is no bijection for person-by-person optimal policies. We also establish a similar result (but not identical) concerning stationary solutions. We present sufficient conditions under which bijection relationships hold. Under static measurement with control-sharing reduction, connections between optimality concepts can be established under relaxed conditions. An implication is a convexity characterization of dynamic teams under static measurement with control-sharing reduction. Some counterparts for stochastic games are also discussed.

Various algorithms for numerical solutions of discounted stochastic games are presented. For zero-sum two-person games, the existing algorithms are compared on randomly generated games and a hybrid algorithm is proposed. For general-sum N-person games, a new mathematical programming formulation which permits the numerical solution of a game by using a non-linear programming code Is presented.

We study stochastic dynamic games with a large number of players, where players are coupled via their cost functions. A standard solution concept for stochastic games is Markov perfect equilibrium (MPE). In MPE, each player's strategy is a function of its own state as well as the state of the other players. This makes MPE computationally prohibitive as the number of players becomes large. An approximate solution concept called oblivious equilibrium (OE) was introduced, where each player¿s decision depends only on its own state and the "long-run average" state of other players. This makes OE computationally more tractable than MPE. It was shown that, under a set of assumptions, as the number of players become large, OE closely approximates MPE. In this paper we relax those assumptions and generalize that result to cases where the cost functions are unbounded. Furthermore, we show that under these relaxed set of assumptions, the OE approximation result can be applied to large population linear quadratic Gaussian (LQG) games.

In this paper, we consider two players zero-sum dynamical games in linear stochastic systems described by input-output model with unknown parameters, and with an ergodic output-tracking quadratic index. We combine the ideas of game theory and control theory as a modeling method, and investigate the corresponding stochastic adaptive game problems. There is few existing results related to such stochastic adaptive game problems, since uncertainty raises the difficulty of making decisions for both players. We attempt to cope with this problem by the ideas and methods of adaptive control. We will firstly use the standard least squares to estimate the unknown parameters, and then construct the adaptive strategy profile for both players by using the so-called ``certainty equivalence principle. We will prove that, the adaptive strategic profile makes the system globally stable, and that the action profile is an asymptotic Nash equilibrium solution to our game problem in a certain sense.

This paper studies the discrete-time indefinite stochastic linear quadratic optimal control problem with the equality terminal state constraints,which can be transformed into a mathematical programming with equality constraints. By means of matrix Lagrange theorem, a necessary condition for the existence of optimal linear state feedback control is given. The previous results on discrete-time stochastic linear quadratic optimal control without constraints, can be viewed as corollaries of the main theorem of this paper.