Adaptive near-optimal neuro controller for continuous-time nonaffine nonlinear systems with constrained input

Learning-based robust output tracking control for unknown discrete-time nonlinear systems with dynamic uncertainty

Data-based adaptive neural network optimal output feedback control for nonlinear systems with actuator saturation

An online adaptive optimal control is proposed for continuous-time nonlinear systems with completely unknown dynamics, which is achieved by developing a novel identifier-critic-based approximate dynamic programming algorithm with a dual neural network (NN) approximation structure. First, an adaptive NN identifier is designed to obviate the requirement of complete knowledge of system dynamics, and a critic NN is employed to approximate the optimal value function. Then, the optimal control law is computed based on the information from the identifier NN and the critic NN, so that the actor NN is not needed. In particular, a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier NN and critic NN simultaneously, which converge to small neighbourhoods around their ideal values. The closed-loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the Lyapunov theory. The proposed adaptation algorithm is also improved to achieve finite-time convergence of the NN weights. Finally, simulation results are provided to exemplify the efficacy of the proposed methods.

In this paper, the direct neural dynamic programming technique is utilized to solve the Hamilton Jacobi-Bellman (HJB) equation online and forward-in-time for the decentralized nearly optimal control of nonlinear interconnected discrete-time systems in affine form with unknown internal subsystem and interconnection dynamics. Only the state vector of the local subsystem is considered measurable. The decentralized optimal controller design for each subsystem consists of an action neural network (NN) that is aimed to provide a nearly optimal control signal, and a critic NN which approximates the cost function. The NN weights are tuned online for both the NNs. It is shown that all subsystems signals are uniformly ultimately bounded (UUB) and that the subsystem inputs approach their corresponding nearly optimal control inputs with bounded error.

摘要: 确定学习运用自适应控制和动力学系统的概念与方法, 研究未知动态环境下的知识获取、表达、存储和利用等问题. 针对产生周期或回归轨迹的连续 非线性动态系统, 确定学习可以对其未知系统动态进行局部准确建模, 其基本要 素包括: 1)使用径向基函数(Radial basis function, RBF)神经网络; 2)对于周期(或回归)状态轨迹 满足部分持续激励条件; 3)在周期(或回归)轨迹的邻域内实现对非线性系统动态的局部准确神经网络逼近(局部准确建模); 4)所学的知识以时不变且空间分布的方式表达、以常值神经网络权值的方式存储, 并可在动态环境下用于动态模式的快速识别或者闭环神经网络控制. 本文针对离散动态系统, 扩展了确定学习理论, 提出一个根据时态数据序列对离散动态系统进行建模与控制的框架. 首先, 运用确定学习原理和离散系统的自适应辨识方法, 实现对产生时态数据的离散非线性系统的未知动态进行局部准确的神经网络建模, 并利用此建模结果对时态数据序列进行时不变表达. 其次, 提出时态数据序列的基于动力学的相似性定义, 以及对离散动态系统产生的时态数据序列(亦可称为动态模式)进行快速识别方法. 最后, 针对离散非线性控制系统, 实现了基于时态数据序列对控制系统动态的闭环辨识(局部准确建模). 所学关于闭环动态的知识可用于基于模式的智能控制. 本文表明确定学习可以为时态数据挖掘的研究提供新的途径, 并为基于数据的建模与控制等问题提供新的研究思路. 关键词: 确定学习 / 时态数据序列 / 离散动态系统 / 基于数据的建模 / 部分持续激励条件 / 时态数据挖掘 / 动态模式识别 / 基于模式的控制

In this paper, the infinite horizon optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems is investigated using neural-network-based online solution of Hamilton-Jacobi-Bellman (HJB) equation. By establishing an appropriate bounded function and defining a modified cost function, the optimal robust guaranteed cost control problem is transformed into an optimal control problem. It can be observed that the optimal cost function of the nominal system is nothing but the optimal guaranteed cost of the original uncertain system. A critic neural network is constructed to facilitate the solution of the modified HJB equation corresponding to the nominal system. More importantly, an additional stabilizing term is introduced for helping to verify the stability, which reinforces the updating process of the weight vector and reduces the requirement of an initial stabilizing control. The uniform ultimate boundedness of the closed-loop system is analyzed by using the Lyapunov approach as well. Two simulation examples are provided to verify the effectiveness of the present control approach.

This paper studies the problem of event-triggered optimal control (ETOC) for continuous-time nonlinear systems and proposes a novel event-triggering condition that enables designing ETOC methods directly based on the solution of the Hamilton-Jacobi-Bellman (HJB) equation. We provide formal performance guarantees by proving a predetermined upper bound. Moreover, we also prove the existence of a lower bound for interexecution time. For implementation purposes, an adaptive dynamic programming (ADP) method is developed to realize the ETOC using a critic neural network (NN) to approximate the value function of the HJB equation. Subsequently, we prove that semiglobal uniform ultimate boundedness can be guaranteed for states and NN weight errors with the ADP-based ETOC. Simulation results demonstrate the effectiveness of the developed ADP-based ETOC method.

A novel neural network approximation scheme that is especially appropriate for adaptive control of nonlinear dynamical systems is proposed. In light of the new function approximation characterization, a neuro adaptive control framework for continuous- and discrete-time nonlinear uncertain dynamical systems is also presented. Specifically, the proposed neural network control framework is Lyapunov-based and, unlike standard neural network controllers guaranteeing ultimate boundedness, the framework guarantees partial asymptotic stability of the closed-loop system, that is, asymptotic stability with respect to part of the closed-loop system states associated with the system plant dynamics. The neuro adaptive controllers are constructed without requiring explicit knowledge of the system dynamics other than the assumption that the plant dynamics are continuous and piecewise continuously differentiable, and that the approximation error of uncertain system nonlinearities lie in a small gain-type norm bounded conic sector. This allows us to show that the standard neural network controllers are in fact capable of achieving partial asymptotic stability around the system equilibrium point for continuous-and discrete-time uncertain systems.

In this paper, the design of a neural network (NN) based adaptive model-based event-triggered control of an uncertain single input single output (SISO) nonlinear discrete time system in affine form is presented. The controller uses an adaptive estimator consisting of a single-layer NN not only to approximate the internal dynamics of an affine nonlinear discrete-time system but also to provide an estimate of the state vector during inter event interval. The NN weights of the adaptive NN estimator are tuned in a aperiodic manner at the event trigger instants unlike periodic updates in standard adaptive neural network (NN) control. A dead zone operator is used to reset the event trigger error to zero as long as the system states continue to remain in a bounded region due to NN reconstruction errors. Lyapunov method is used to derive the event trigger condition, prove uniform ultimate boundedness (UUB) of the NN weight estimation error and system states.

A reinforcement learning (RL) based adaptive dynamic programming (ADP) is developed to learn the approximate optimal stabilization input of the servo mechanisms, where the unknown system dynamics are approximated with a three-layer neural network (NN) identifier. First, the servo mechanism model is constructed and a three-layer NN identifier is used to approximate the unknown servo system. The NN weights of both the hidden layer and output layer are synchronously tuned with an adaptive gradient law. An RL-based critic three-layer NN is then used to learn the optimal cost function, where NN weights of the first layer are set as constants, NN weights of the second layer are updated by minimizing the squared Hamilton-Jacobi-Bellman (HJB) error. The optimal stabilization input of the servomechanism is obtained based on the three-layer NN identifier and RL-based critic NN scheme, which can stabilize the motor speed from its initial value to the given value. Moreover, the convergence analysis of the identifier and RL-based critic NN is proved, the stability of the cost function with the proposed optimal input is analyzed. Finally, a servo mechanism model and a complex system are provided to verify the correctness of the proposed methods.

In this paper, the output feedback based finite-horizon near optimal regulation of nonlinear affine discrete-time systems with unknown system dynamics is considered by using neural networks (NNs) to approximate Hamilton-Jacobi-Bellman (HJB) equation solution. First, a NN-based Luenberger observer is proposed to reconstruct both the system states and the control coefficient matrix. Next, reinforcement learning methodology with actor-critic structure is utilized to approximate the time-varying solution, referred to as the value function, of the HJB equation by using a NN. To properly satisfy the terminal constraint, a new error term is defined and incorporated in the NN update law so that the terminal constraint error is also minimized over time. The NN with constant weights and time-dependent activation function is employed to approximate the time-varying value function which is subsequently utilized to generate the finite-horizon near optimal control policy due to NN reconstruction errors. The proposed scheme functions in a forward-in-time manner without offline training phase. Lyapunov analysis is used to investigate the stability of the overall closed-loop system. Simulation results are given to show the effectiveness and feasibility of the proposed method.

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems. Unlike existing optimal state feedback control, the control input of the optimal parallel control is introduced into the feedback system. However, due to the introduction of control input into the feedback system, the optimal state feedback control methods can not be applied directly. To address this problem, an augmented system and an augmented performance index function are proposed firstly. Thus, the general nonlinear system is transformed into an affine nonlinear system. The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically. It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function. Moreover, an adaptive dynamic programming ( ADP ) technique is utilized to implement the optimal parallel tracking control using a critic neural network ( NN ) to approximate the value function online. The stability analysis of the closed-loop system is performed using the Lyapunov theory, and the tracking error and NN weights errors are uniformly ultimately bounded ( UUB ) . Also, the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals. Finally, the effectiveness of the developed optimal parallel control method is verified in two cases.

We develop a decentralized dynamic event-based control strategy for nonlinear systems subject to matched interconnections. To begin with, we introduce a dynamic event-based sampling mechanism, which relies on the system's states and the variables generated by time-based differential equations. Then, we prove that the decentralized event-based controller for the whole system is composed of all the optimal event-based control policies of nominal subsystems. To derive these optimal event-based control policies, we design a critic-only architecture to solve the related event-based Hamilton-Jacobi-Bellman equations in the reinforcement learning framework. The implementation of such an architecture uses only critic neural networks (NNs) with their weight vectors being updated through the gradient descent method together with concurrent learning. After that, we demonstrate that the asymptotic stability of closed-loop nominal subsystems and the uniformly ultimate boundedness stability of critic NNs' weight estimation errors are guaranteed by using Lyapunov's approach. Finally, we provide simulations of a matched nonlinear-interconnected plant to validate the present theoretical claims.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

In this work, a neural-networks (NNs)-based adaptive asymptotic tracking control scheme is presented for a class of uncertain nonstrict feedback nonlinear systems with time-varying full-state constraints. First, we construct a novel exponentially decaying nonlinear mapping to map the constrained system states to new system states without constraints. Instead of the traditional barrier Lyapunov function methods, the feasible conditions which require the virtual control signals satisfying the constraint requirements are removed. By employing the Nussbaum design method to eliminate the effect of unknown control gains, the general assumption about the signs of the unknown control gains is relaxed. Then, the nonstrict feedback form of the system can be pulled back to the strict feedback form through the basic properties of radial basis function NNs. Simultaneously, the intermediate control signals and the desired controller are constructed by the backstepping process and the Nussbaum design method. The designed controller can ensure that all signals in the whole closed-loop system are bounded without the violation of the constraints and hold the asymptotic tracking performance. In the end, a practical example about a brush dc motor driving a one-link robot manipulator is given to illustrate the effectiveness of the proposed design scheme.