Adaptive optimal control approach to robust tracking of uncertain linear systems based on policy iteration

The innovation of optimal learning control methods is profoundly propelled due to the improvement of the learning ability. In this article, we investigate the synthesis of initialization and acceleration for optimal learning control algorithms. This approach contrasts with traditional methods that concentrate solely on either the improvement of initialization or acceleration. Specifically, we establish a novel relaxed policy iteration (PI) algorithm with self-learning horizon for stochastic optimal control. Notably, by suitably utilizing self-learning horizon, we can directly evaluate inadmissible policies to reduce the initialization burden. Meanwhile, the inadmissible policy can be rapidly optimized with few learning iterations. Then, several critical conclusions of relaxed optimal control are established by discussing algorithm convergence and system stability. Furthermore, to provide the convincing application potentials, a class of unconventional problems is effectively solved by the relaxed PI algorithm, including the dynamics with external noises and nonzero equilibrium. Finally, we present a series of nonlinear benchmarks with practical applications to comprehensively evaluate the performance of relaxed PI. The experimental results obtained from these diverse benchmarks uniformly highlight the effectiveness of self-learning horizon mechanism.

In this paper, we study the optimal control problem for Takagi–Sugeno (T-S) fuzzy systems with disturbances. Due to the presence of disturbances in the T-S fuzzy systems, a fuzzy optimal state feedback control approach is presented by employing a two-player zero-sum game. Since the analytical optimal control solutions and the worst-case disturbance policies of T-S fuzzy systems can be boiled down to solving the algebraic Riccati equations (AREs), which are difficult to be obtained directly, a model-free policy iteration (PI) learning algorithm is proposed to obtain their approximation solutions. It is proved that the developed fuzzy optimal state feedback controller can ensure the fuzzy systems to be asymptotically stable and satisfy the disturbance attenuation condition simultaneously. Also, the designed PI learning algorithm can converge to their optimal solutions. Finally, we apply the developed fuzzy optimal state feedback control method to the truck-trailer system and the simulation results demonstrated the effectiveness of the developed scheme.

This paper deals with a robust tracking problem with a discounted factor. A new auxiliary system is established in terms of norm‐bounded time‐varying uncertainties. It is shown that the robust discounted tracking problem for the auxiliary system solves the original problem. Then, the new robust discounted tracking problem is represented as a well‐known zero‐sum game problem. Moreover, the robust tracking Bellman equation and the robust tracking Algebraic Riccati equation (RTARE) are inferred. A lower bound of a discounted factor for stability is obtained to assure the stability of the closed‐loop system. Based on the auxiliary system, the system is reshaped in a new structure that is applicable to Reinforcement Learning methods. Finally, an online Q‐learning algorithm without the knowledge of system matrices is proposed to solve the algebraic Riccati equation associated with the robust discounted tracking problem for the auxiliary system. Simulation results are given to verify the effectiveness and merits of the proposed method.

Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

The paper presents a reinforcement learning approach for optimal tracking control of switched systems with application to a grid-tied hybrid generation system. To enhance interaction with the irregular environment, reference trajectory is learned via controller from states to optimal control. The main issue is to solve the optimal tracking control problem for a hybrid generation system consisting of multiple switched subsystems, and reinforcement learning can seek the globally optimal solution well without knowing accurate system dynamics. The investigated learning algorithm is used to generate an optimum map based on the learned ultimate value without knowledge of system parameters and obtains the optimal control law via deriving of algebraic Riccati equation (ARE) with unnecessary knowing of command generator dynamics. The optimal control solution can converge the online learning algorithm well based on policy iteration as verification in the simulation.

We consider the robust and perfect tracking (RPT) problem for multivariable linear systems with external disturbances. The problem is to design a proper controller such that the resulting overall closed-loop system is asymptotically stable and the controlled output almost perfectly tracks a given reference signal with an arbitrarily fast settling time in the face of external disturbances and initial conditions. We derive a set of necessary and sufficient conditions under which the RPT problem is solvable. Under these solvability conditions, we develop algorithms for constructing state and output feedback laws, explicitly parameterized in /spl epsi/, that solve the RPT problem. In our construction of feedback laws, we propose a controller structure which enables us to design a tracking controller without introducing additional integrators regardless what type the system is.

Optimal control methods have gained significant attention due to their promising performance in nonlinear systems. In general, an optimal control method is regarded as an optimization process for solving the optimal control laws. However, for uncertain nonlinear systems with complex optimization objectives, the solving of optimal reference trajectories is difficult and significant that might be ignored to obtain robust performance. For this problem, a double-closed-loop robust optimal control (DCL-ROC) is proposed to maintain the optimal control reliability of uncertain nonlinear systems. First, a double-closed-loop scheme is established to divide the optimal control process into a closed-loop optimization process that solves optimal reference trajectories and a closed-loop control process that solves optimal control laws. Then, the ability of the optimal control method can be improved to solve complex uncertain optimization problems. Second, a closed-loop robust optimization (CL-RO) algorithm is developed to express uncertain optimization objectives as data-driven forms and adjust optimal reference trajectories in a close loop. Then, the optimality of reference trajectories can be improved under uncertainties. Third, the optimal reference trajectories are tracked by an adaptive controller to derive the optimal control laws without certain system dynamics. Then, the adaptivity and reliability of optimal control laws can be improved. The experimental results demonstrate that the proposed method can achieve better performance than other optimal control methods.

In this paper the robust asymptotic tracking and disturbance rejection problem is studied for linear periodic discrete-time systems, and for general classes of disturbance functions and reference signals, consisting in free responses of periodic processes. The necessary and sufficient conditions for the existence of a solution, which involve the periodic notion of invariant zero, are found. A constructive design procedure of the periodic compensator is given; it is based on the internal model principle.

In this chapter, by using neural network (NN)-based online learning optimal control approach, a decentralized control strategy is developed to stabilize a class of continuous-time large-scale interconnected nonlinear systems . It is proven that the decentralized control strategy of the overall system can be established by adding appropriate feedback gains to the optimal control laws of the isolated subsystems . Then, an online policy iteration (PI) algorithm is developed to solve the Hamilton–Jacobi–Bellman equations related to the optimal control problem. By constructing a set of critic NNs, the cost functions can be obtained by NN approximation, followed by the control laws. Furthermore, as a generalization, an NN-based decentralized control law is developed to stabilize the large-scale interconnected nonlinear systems using an online model-free integral PI algorithm. The model-free PI approach can solve the decentralized control problem for large-scale interconnected nonlinear systems with unknown dynamics. Finally, two simulation examples are provided to illustrate the effectiveness of the present decentralized control scheme.

The problem of robust tracking and model following is considered for a class of linear dynamical systems with multiple delayed state perturbations, time–varying uncertain parameters, and disturbance. A class of continuous memoryless state feedback controllers is proposed for robust tracking of dynamical signals. The proposed robust tracking controllers can guarantee that the tracking error decreases asymptotically to zero in the presence of multiple delayed state perturbations, time–varying uncertain parameters, and disturbance. A procedure for designing such a class of zero–error tracking state feedback controllers is also introduced. Finally, a numerical example is given to demonstrate the validity of the results.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

Policy iteration, as an adaptive/approximate dynamic programming-based approach for optimal control is investigated. The context is optimal control of discrete-time nonlinear dynamics with undiscounted cost functions. Convergence of the learning iterations and uniqueness of the solution to the corresponding Bellman equation are established, leading to the optimality of the limit function, i.e., the learning results. Moreover, given the faster convergence of the learning under policy iteration compared with value iteration-based learning algorithms, some theoretical results are developed which prove that starting with a similar initial guess, policy iteration will not converge slower than value iteration. Finally, some numerical analyses are presented to demonstrate the results in practice.

In this paper, the Bellman equation is used to solve the optimal adaptive control of quantized linear discrete-time system with unknown dynamics. To mitigate the effect of the quantization errors, the dynamics of the quantization error bound and an update law for tuning the range of the dynamic quantizer are derived. Subsequently, by using adaptive dynamic programming technique, the infinite horizon optimal regulation problem of the uncertain quantized linear discrete-time system is solved in a forward-in-time manner without using value and/or policy iterations. The asymptotic stability of the closed-loop system is verified by standard Lyapunov stability approach in the presence of state and input quantizers. The effectiveness of the proposed method is verified via simulation.

This paper studies data-driven learning-based methods for the finite-horizon optimal control of linear time-varying discrete-time systems. First, a novel finite-horizon Policy Iteration (PI) method for linear time-varying discrete-time systems is presented. Its connections with existing infinite-horizon PI methods are discussed. Then, both data-driven off-policy PI and Value Iteration (VI) algorithms are derived to find approximate optimal controllers when the system dynamics is completely unknown. Under mild conditions, the proposed data-driven off-policy algorithms converge to the optimal solution. Finally, the effectiveness and feasibility of the developed methods are validated by a practical example of spacecraft attitude control.

Adaptive nearly optimal control for a class of continuous-time nonaffine nonlinear systems with inequality constraints