Output feedback Q-learning for discrete-time linear zero-sum games with application to the H-infinity control

The bipartite consensus (BC) issue for nonlinear multiagent systems (NMASs) with unknown system dynamics information is investigated in this article. Initially, the dynamics of NMASs are represented using the Takagi-Sugeno (T-S) fuzzy model. Subsequently, to achieve distributed control, a minmax game policy is introduced, where each agent aims to minimize its performance index while its neighbors attempt to maximize it. Consequently, the BC problem for NMASs is reformulated as a zero-sum game, transforming the controller design into solving a set of game algebraic Riccati equations (GAREs). To solve such equations, a novel scaling off-policy iteration (PI) algorithm is proposed. The key features of the proposed learning algorithm can be outlined in three main aspects: 1) during the learning process, the reliance on system dynamics is relaxed; 2) compared with the PI method, the requirement for initial admissible control policies is eliminated; and 3) a more rapid convergence speed is achieved than traditional value iteration. Finally, the effectiveness and advantages of the proposed method are validated through a simulation example and a series of comparative experiments.

Closed-Loop Strategy for Stackelberg Game Problem with Incomplete Information Structures

This article addresses the online reinforcement Q-learning algorithms to design H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> tracking controller for unknown discrete-time linear systems. An augmented system composed of the original system and the command generator is constructed, and a discounted performance function is introduced to establish a discounted game algebraic Riccati equation (GARE). The existence conditions of a solution to the GARE are proposed and a lower bound is found for the discount factor to assure the stability of the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> tracking control solution. The Q-function Bellman equation is then derived, based on which the reinforcement Q-learning algorithm is developed to learn the solution to H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> tracking control problem without knowing the system dynamics. Both state-data-driven and output-data-driven reinforcement Q-learning algorithms toward finding the control policies are proposed. Unlike the value function approximation (VFA)-based approach, it is proved that the Q-learning scheme brings out no bias of solution to the Q-function Bellman equation under the probing noise satisfying the persistent excitation (PE) condition, and therefore, converges to the nominal discounted GARE solution. Moreover, the proposed output-data-driven method is more powerful than the state-data-driven method as it may not be available to completely measure the full system states in practical applications. A simulation example with a single-phase voltage-source UPS inverter is used to verify the effectiveness of the proposed Q-learning algorithms.

This paper presents an output feedback design approach based on the adaptive control scheme developed for nonlinearly parameterized systems, to achieve global output regulation for a class of nonlinear systems in output feedback form. We solve the output regulation problem without the knowledge of the sign and the value of the high frequency gain a priori. It is not necessary to have both the limiting assumptions that the exogenous signal w and the unknown parameter μ belong to a prior known compact set and the high frequency gain has a determinate lower and upper bounds. The effectiveness of the proposed algorithm is shown with the help of an example.

Adaptive control for output feedback nonlinear systems in the presence of modeling errors

This paper considers the output feedback adaptive controller design problem for a class of discrete-time nonlinear systems in output feedback form with unknown control directions and preceded by unknown hysteresis. The problem of lacking in a-priori knowledge on the control directions and unknown hysteresis are solved by using the discrete Nussbaum gain and Prandtl–Ishlinskii model, respectively. The system is transformed into the form of a nonlinear auto regressive moving average (NARMA) model to construct an output feedback control. To overcome the noncausal problem in the control design, future output prediction laws and parameter update laws with the dead-zone technique are constructed on the basis of the NARMA model. The proposed control algorithm guarantees that all the signals in the controlled system are bound and the output tracking error is made to be neighbourhood around zero, ultimately. Finally, simulations are performed on a nonlinear system to show the effectiveness of the proposed method

Optimal Tracking Control of Linear Discrete-Time Systems Under Cyber Attacks

Output feedback form of Chua’s circuit and modular adaptive control of chaos using single measurement

This paper studies the cooperative global robust output regulation problem for the nonlinear multi-agent systems in output feedback form with relative degree $r=2$ subject to arbitrarily large uncertainty. The robust control method alone cannot handle arbitrarily large uncertainty. By combining the robust and adaptive high gain technique, we develop a recursive procedure to construct a distributed output feedback control law to deal with the problem. The theoretical result of this paper is applied to the leader-following consensus problem of a family of hyper-chaotic Lorenz systems.

We apply dynamic programming principle to discuss two optimal investment problems by using zero-sum and nonzero-sum stochastic game approaches in a continuous-time Markov regime-switching environment within the frame work of behavioral finance. We represent different states of an economy and, consequently, investors’ floating levels of psychological reactions by a D-state Markov chain. The first application is a zero-sum game between an investor and the market, and the second one formulates a nonzero-sum stochastic differential portfolio game as the sensitivity of two investors’ terminal gains. We derive regime-switching Hamilton–Jacobi–Bellman–Isaacs equations and obtain explicit optimal portfolio strategies with Feynman–Kac representations of value functions. We illustrate our results in a two-state special case and observe the impact of regime switches by comparative results.

Data-based adaptive neural network optimal output feedback control for nonlinear systems with actuator saturation

SummaryThis article considers the output regulation problem of continuous‐time linear systems subject to input saturation in the scenario that the system dynamics are unknown. Based on the low‐gain technique and the output feedback technique, we construct an output feedback control law to solve the output regulation problem with input saturation. Since the system dynamics are unknown, we estimate the feedback and feedforward terms in the control law with a two‐step data‐driven method. An output feedback adaptive dynamic programming algorithm is proposed to estimate the feedback term. An online updating algorithm based on output tracking error is proposed to estimate the feedforward term directly without solving the output regulation equations. Both algorithms don't rely on the measurement of the internal states of the exosystem and the original system. We show that the input saturation caused by the iterative feedforward gain matrix in the online updating algorithm doesn't affect the convergence of the algorithm. With these two data‐driven algorithms, the control law doesn't rely on the dynamics anymore. The low‐gain control law obtained by the model‐free method prevents input saturation from occurring and hence solves the output regulation problem. Finally, a numerical example is provided to validate the theoretical results.

This paper is concerned with the output feedback \(\mathcal {H}_\infty \) control problem for a class of stochastic nonlinear systems with time-varying state delays; the system dynamics is governed by the stochastic time-delay It\(\hat{o}\)-type differential equation with state and disturbance contaminated by white noises. The design of the output feedback \(\mathcal {H}_\infty \) control is based on the stochastic dissipative theory. By establishing the stochastic dissipation of the closed-loop system, the delay-dependent and delay-independent approaches are proposed for designing the output feedback \(\mathcal {H}_\infty \) controller. It is shown that the output feedback \(\mathcal {H}_\infty \) control problem for the stochastic nonlinear time-delay systems can be solved by two delay-involved Hamilton–Jacobi inequalities. A numerical example is provided to illustrate the effectiveness of the proposed methods.

In this article, an output‐feedback Q‐learning algorithm is proposed for the discrete‐time linear system to deal with the tracking control problem. The problem is formulated as a zero‐sum game in the Stackelberg game framework with a discount factor to make the value function bounded. According to the principle of optimality, the game algebraic Riccati equation (GARE) is derived and solved by the Q‐learning algorithm to get the optimal solution of the Stackelberg game without requiring the knowledge of system dynamics and state. It is proved that the solution of the algorithm converges to the optimal control input and the worst‐case disturbance with excitation noises during training, and the Stackelberg strategy can achieve a lower disturbance attenuation level than the Nash one. Moreover, the impacts of the discount factor on the stability of the closed‐loop system and solvability of the GARE are analyzed to provide some criteria for the choice of the discount factor. Simulation examples are provided to validate the effectiveness of the algorithm.

In this paper, distributed flocking strategies have been exploited for multi-agent two-player zero-sum games. Two main challenges are addressed, i.e. (a) handling system uncertainties and disturbances, and (b) achieving optimality. Adopting the emerging Approximate Dynamic Programming (ADP) technology, a novel distributed adaptive flocking design is proposed to optimize the multi-agent two-player zero-sum games even when the system dynamics and disturbances are unknown. First, to evaluate the multi-agent flocking performance and effects from disturbances, a novel flocking cost function is developed. Next, an innovative type of online neural network (NN) based identifier is proposed to approximate the multi-agent zero-sum game system dynamics effectively. Subsequently, another novel neural network (NN) is proposed to approximate the optimal flocking cost function by using the Hamilton-Jacobi-Isaacs (HJI) equation in a forward in time manner. Moreover, a novel additional term is designed and included into the NN update law to relax the stringent requirement of initial admissible control. Eventually, the distributed adaptive optimal flocking design is obtained by using the learnt Multi-agent zero-sum games system dynamics and approximated optimal flocking cost function. Simulation results demonstrate the effectiveness of proposed scheme.