Model-Free Optimal Control for Affine Nonlinear Systems With Convergence Analysis

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

A data-driven energy management method for parallel PHEVs based on action dependent heuristic dynamic programming (ADHDP) model

Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem

An incremental iterative Q-learning algorithm (IIQLA) is proposed to tackle the optimal secure control problem for cyber-physical systems under false data injection attacks. Within a zero-sum game framework, the secure control problem is transformed into solving an iterative algebraic Riccati equation. To derive the optimal secure control policy from the equation, the IIQLA is developed, which does not require prior knowledge of the system dynamics. This algorithm utilizes two auxiliary variables to separate the behavior policy and the target policy, thereby improving the exploration of data. As a result, the devised control policies are more conservative, leading to solutions that are closer to the optimal policy. Moreover, by introducing an adaptive learning rate, the proposed IIQLA can accelerate the convergence speed and decrease the number of required iterations, thus alleviating the computation burden. The convergence of the proposed IIQLA with different learning rates is also analyzed. In addition, the closed-loop system is guaranteed to be asymptotically stable, and the exploration noise does not introduce bias into the optimal policies. Finally, a simulation example is provided to demonstrate the effectiveness of the proposed approach.

Learning-based optimal cooperative control using edge-based triggering strategy

We consider a finite-capacity single-server queuing system in which the problems of capacity design and service quality control are integrated. The amount of time spent on servicing a customer is used as a measure of the quality of service provided. The expected reward from servicing is assumed to be a nondecreasing function of the service duration, while a linear holding cost is assumed for customers waiting in the system. We maximize the long-run average return per unit time by optimally controlling the quality of service to be provided as a function of the workload facing the server for different waiting-room capacities. We show that the optimal control policy for a given capacity is monotone and also examine the effect of varying the system capacity on the optimal control policy. Furthermore, we analyze the design problem of selecting an optimal capacity by assuming that an optimal service control policy will be followed for each system capacity.

In this paper, optimal synchronization problem of multi-agent systems (MASs) with time-varying input delay is investigated. To remove the effect of input delay, the delayed MAS model is firstly transformed into a new discrete-time delay-free version. Then we establish an equivalence relation on the performance indexes of the two systems so as to formulate a new optimal consensus control problem of the transformed delay-free system. Based on the Bellman optimality theorem, we derive optimal consensus control policies from the coupled HJB equations. A policy iteration algorithm is introduced to learn the solutions to the HJB equations in real-time. To execute iterative ADP method, the critic neural networks (NN) are utilized to approximate the value functions and help to calculate the control policies. Finally, a numerical simulation is provided to show the effectiveness of the proposed approach.

Reduction of energy consumption has attracted much attention of the researchers and practitioners in the field of sustainable manufacturing. However, the issue of integrated optimal production and energy control for sustainable manufacturing systems has not yet been investigated sufficiently. In this paper, we study such a problem for a single machine and single product-type manufacturing system, with the objective of minimizing the sum of production cost and energy consumption cost. To solve this problem, a Markov decision process model is developed, based on which, we obtain the integrated optimal production and energy control policy by the policy iteration method. Furthermore, we reveal the structural characteristics of optimal control policy and analyze the influence of system parameters on it by comparing different cases.

Dynamic control of a closed two-stage queueing network for outfitting process in shipbuilding

Considers a failure prone production system with deterministic production cycles. The objective is to find the optimal production rate to minimize long-run average cost. The authors first show that the optimal control policy belongs to a type of switching curve policy which has a special structural property and can be characterized by a single parameter. The authors then establish a relationship between the surplus process of the system under the optimal control policy and the workload process of a D/G/1 queue, based on which the existing results in queueing theory which can be applied to obtain the steady-state probability distribution of the surplus process under the optimal control policy. >

This chapter obtains optimal solutions for encoders and controllers under quadratic cost functions for linear Gaussian systems controlled over Gaussian channels, proving also the existence of optimal solutions. Furthermore, the chapter identifies conditions under which optimal coding and control policies are linear. A large class of network settings where optimal policies are non-linear is identified.

SummaryThis paper considers optimal consensus control problem for unknown nonlinear multiagent systems (MASs) subjected to control constraints by utilizing event‐triggered adaptive dynamic programming (ETADP) technique. To deal with the control constraints, we introduce nonquadratic energy consumption functions into performance indices and formulate the Hamilton‐Jacobi‐Bellman (HJB) equations. Then, based on the Bellman's optimality principle, constrained optimal consensus control policies are designed from the HJB equations. In order to implement the ETADP algorithm, the critic networks and action networks are developed to approximate the value functions and consensus control policies respectively based on the measurable system data. Under the event‐triggered control framework, the weights of the critic networks and action networks are only updated at the triggering instants which are decided by the designed adaptive triggered conditions. The Lyapunov method is used to prove that the local neighbor consensus errors and the weight estimation errors of the critic networks and action networks are ultimately bounded. Finally, a numerical example is provided to show the effectiveness of the proposed ETADP method.

This article studies a distributed minmax strategy for multiplayer games and develops reinforcement learning (RL) algorithms to solve it. The proposed minmax strategy is distributed, in the sense that it finds each player's optimal control policy without knowing all the other players' policies. Each player obtains its distributed control policy by solving a distributed algebraic Riccati equation in a multiplayer noncooperative game. This policy is found against the worst policies of all the other players. We guarantee the existence of distributed minmax solutions and study their L2 and asymptotic stabilities. Under mild conditions, the resulting minmax control policies are shown to improve robust gain and phase margins of multiplayer systems compared to the standard linear-quadratic regulator controller. Distributed minmax solutions are found using both model-based policy iteration and data-driven off-policy RL algorithms. Simulation examples verify the proposed formulation and its computational efficiency over the nondistributed Nash solutions.

Dynamic power management (DPM) plays a significant role to save the power consumption effectively in both design and operational phases of computer-based systems. Especially, it is well known that the state-dependent control policy by monitoring the energy state in each component or the whole system is efficient for power saving in server systems whose system state such as the number of transaction requests can be completely observed. In this paper we consider an optimal power-aware design in a cluster system and formulate the DPM problem by means of a Markov decision process. We derive the dynamic programming equation for the optimal control policy which maximizes the expected reward per unit electrical power, called the power effectiveness, and give a policy iteration algorithm to determine the optimal control policy sequentially. In a numerical example, we consider a cluster system with two service nodes and describe the arrival stream of the transaction request process by a Markov modulated Poisson process. It is shown that the optimal control policy maximizing the power effectiveness is given in the form of decision table.