Distributed Optimal Flocking Design for Multi-agent Two-Player Zero-Sum Games with Unknown System Dynamics and Disturbance

Learning from adaptive neural network control of an underactuated rigid spacecraft

In this paper, the nearly optimal solution for discrete-time (DT) affine nonlinear control systems in the presence of partially unknown internal system dynamics and disturbances is considered. The approach is based on successive approximate solution of the Hamilton-Jacobi-Isaacs (HJI) equation, which appears in optimal control. Successive approximation approach for updating control and disturbance inputs for DT nonlinear affine systems are proposed. Moreover, sufficient conditions for the convergence of the approximate HJI solution to the saddle point are derived, and an iterative approach to approximate the HJI equation using a neural network (NN) is presented. Then, the requirement of full knowledge of the internal dynamics of the nonlinear DT system is relaxed by using a second NN online approximator. The result is a closed-loop optimal NN controller via offline learning. A numerical example is provided illustrating the effectiveness of the approach.

In this paper, a novel adaptive optimal control algorithm based on approximate dynamic programming (ADP) approach is proposed for integrated active front steering (AFS) and direct yaw moment control (DYC). The corrective yaw moment and active steering angle are generated online without knowing system dynamics, which is realised by using a neural network (NN) identifier to identify the unknown system dynamics and a critic NN to calculate the optimal control action, respectively. Control commands are executed via active steering angle on front wheels and proper brake torque distribution on the effective wheels. Computer simulations under three different driving manoeuvres, i.e., lane change manoeuvre, step steer manoeuvre and sine with dwell manoeuvre, are carried out to evaluate the proposed control method. Simulation results show that the proposed ADP-based control method demonstrates improved tracking performance in terms of enhancing vehicle handling and stability performance when encountering the varying longitudinal velocity, the uncertain cornering stiffness and the different road/tyre friction coefficients. Model-free and self-adaptive properties of the proposed method provide a new solution to vehicle stability controller design instead of the commonly used model-based methods.

This paper proposes the work on the back stepping controller for the class of Discrete-time nonlinear system under amplitude & rate saturation, dead-zone constraints. A robust adaptive neural network (NN) control is developed for a general class of uncertain single-input-single output (SISO) discrete time nonlinear system with unknown system dynamics and input nonlinear ties. Here ( ) function is employed to saturate the amplitude and rate of system input. For input nonlinear ties like saturation and dead zone, discrete time SISO nonlinear system in combination with back stepping and Lyapunov synthesis is proposed for adaptive NN design with guaranteed stability. To reduce the effect of dead-zone a dead-zone compensator or dead-zone inverse is designed. Chebyshev Neural Networks (CNNs) are used to approximate the unknown nonlinear functions in system dynamics. New weight update laws are derived to make this scheme adaptive and show the stability of this scheme. Finally simulation results are presented in this paper to show the effectiveness of proposed scheme.

Sensory data have becoming widely available in large volume and variety due to the increasing presence and adoption of the Internet of Things. Such data can be tremendously useful if they are processed properly in a timely fashion. They could play a key role in the coordination of industrial production. It is thus desirable to explore an effective and efficient scheme to support data tracking and monitoring. This paper intends to propose a novel automatic learning scheme to improve the tracking efficiency while maintaining or improving the data tracking accuracy. A core strategy in the proposed scheme is the design of Laguerre neural network (LaNN)-based approximate dynamic programming (ADP). As a traditional optimal learning strategy, ADP is a popular approach for data processing. The action neural network (NN) and the critic NN as two important components in ADP have big impact on the performance of ADP. In this paper, a LaNN is employed as the implementation of the action NN in ADP considering Laguerre polynomials' approximation capability. In addition, this LaNN-based ADP is integrated into an online parameter-tuning framework to optimize those parameters of characteristic model that is used to trace the data in the tracking control system. Meanwhile, this article provides an associated Lyapunov convergence analysis to guarantee a uniformly ultimately boundedness property for tracking errors in the proposed approach. Furthermore, the proposed LaNN-based ADP optimal online parameter-tuning scheme is validated using a temperature dynamic tracking control task. The simulation results demonstrate that the scheme has satisfactory learning performance over time.

In this chapter, differential games are studied for continuous-time linear and nonlinear systems, including two-player zero-sum games, multi-player zero-sum games, and multi-player nonzero-sum games, via a series of adaptive dynamic programming (ADP) approaches. First, an integral policy iteration algorithm is developed to learn online the Nash equilibrium of two-player zero-sum differential games with completely unknown continuous-time linear dynamics using the state and control data. Second, multi-player zero-sum differential games for a class of continuous-time uncertain nonlinear systems are solved by using a novel iterative ADP algorithm. Via neural network modeling for the system dynamics, the ADP technique is employed to obtain the optimal control pair iteratively so that the iterative value function reaches the optimal solution of the zero-sum differential games. Finally, an online synchronous approximate optimal learning algorithm based on policy iteration is developed to solve multi-player nonzero-sum games of continuous-time nonlinear systems without the requirement of exact knowledge of system dynamics.

It is well-known that the controls of neural network require that all the inputs of their input layer are available. Under this circumstance, the architecture of the neural network is constrained. Hence, this kind of neurocontrol cannot apply to a wide class of nonlinear and unknown dynamical systems, or the control system that requires more sensors. Although the unknown system state and the unknown system dynamics can be achieved from an adaptive observer and a learning neural network, respectively, the simultaneous existence of the estimated state error and the modeling error makes the control system more likely to be unstable. In this paper, a novel neurocontroller based on the concept of sliding mode with estimated state is constructed to tackle a wide class of unknown and nonlinear dynamical systems. The focal stability of the overall system can be verified by the Lyapunov stability criteria. Finally, simulations are presented to verify the usefulness of the proposed method.

A reinforcement learning (RL) based adaptive dynamic programming (ADP) is developed to learn the approximate optimal stabilization input of the servo mechanisms, where the unknown system dynamics are approximated with a three-layer neural network (NN) identifier. First, the servo mechanism model is constructed and a three-layer NN identifier is used to approximate the unknown servo system. The NN weights of both the hidden layer and output layer are synchronously tuned with an adaptive gradient law. An RL-based critic three-layer NN is then used to learn the optimal cost function, where NN weights of the first layer are set as constants, NN weights of the second layer are updated by minimizing the squared Hamilton-Jacobi-Bellman (HJB) error. The optimal stabilization input of the servomechanism is obtained based on the three-layer NN identifier and RL-based critic NN scheme, which can stabilize the motor speed from its initial value to the given value. Moreover, the convergence analysis of the identifier and RL-based critic NN is proved, the stability of the cost function with the proposed optimal input is analyzed. Finally, a servo mechanism model and a complex system are provided to verify the correctness of the proposed methods.

A new approach for automotive engine torque and air-fuel ratio control is presented in this paper. A class of adaptive critic designs that can be classified as (model-free) action-dependent heuristic dynamic programming is used in the present project. Adaptive critic designs are defined as designs that approximate dynamic programming in the general case, i.e., approximate optimal control over time in noisy, nonlinear environment. The present work uses a system, called "critic", to approximate the cost function in dynamic programming and thus to achieve optimal control. The goals of the present learning control design for automotive engines are emissions reduction and maintenance of optimal performance under various operating conditions. Using the data obtained from a test vehicle, we first develop a neural network model of the engine. A neural network controller is then designed based on the idea of approximate dynamic programming to achieve optimal control. In the simulation studies, the initial controller is trained using the neural network engine model developed rather than the actual engine. We have developed and tested self-learning neural network controllers for both engine torque and exhaust airfuel ratio control. For both control problems, good transient performance of the neural network controller has been observed. A distinct feature of the present technique is the controller's real-time adaptation capability based on real vehicle data which allows the neural network controller to be further refined and improved in real-time vehicle operation through continuous learning.

In this paper, a novel on-line value iteration (VI) adaptive (or approximate) dynamic programming (ADP) is developed to solve the optimal control problems for nonlinear discrete-time affine systems, which updated the value function and policy function at the each step running of the system. The critical point of this algorithm is to combine local VI-ADP algorithm with gaussian distribution regarded the current states as the expected. Moreover, for implementing the on-line VI-ADP algorithm, the actor-critic structure composed of the neural networks realizes the value function and the strategy function. In addition, the model neural network enables the algorithm to replace the system dynamics in a data-driven way, that is, without of the system dynamic knowledge is feasible. Finally, simulations show that this on-line VI-ADP algorithm can solve the regulation and tracking problems effectively.

This paper addresses the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$H_{\infty }$</tex-math></inline-formula> output regulation problem of affine nonlinear systems with unmeasurable states and completely unknown dynamics. The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$H_{\infty }$</tex-math></inline-formula> output regulation problem is formulated into a two-player zero-sum differential game problem with the control input and disturbance being the two adversary players. A model-based global state- feedback optimal solution to the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$H_{\infty }$</tex-math></inline-formula> output regulation problem is presented by solving a Hamilton-Jacobi-Isaacs (HJI) equation, and a state- feedback reinforcement learning (RL) algorithm is proposed to solve the HJI equation without knowledge of system dynamics. A linear data-driven observer is designed to estimate the unmeasurable states and a novel output- feedback RL algorithm is developed based on the state- feedback RL framework by employing a critic-actor-disturbance neural network (NN) with a hidden layer. The salient found of this work is that, under the proposed output- feedback RL algorithm, the closed-loop system can reach globally stable. A numerical example is provided to demonstrate the effectiveness of the proposed solution.

Data-based adaptive neural network optimal output feedback control for nonlinear systems with actuator saturation

Output feedback Q-learning for discrete-time linear zero-sum games with application to the H-infinity control

This chapter provides some examples from the literature of how feed-forward neural networks are used in three different contexts in manufacturing operations. Operational design problems involve the determination of design parameters, such as number of kanbans, in order to optimize the performance of the system. Operational-system decision support refers to the use of neural networks as decision-support mechanisms in predicting system performance in response to certain settings of system parameters and current environmental factors. Operational-system-control problems are distinguished from decision support in that the consequences of a control decision are both an immediate return and putting the system in a new state from which another control decision needs to be taken. In operational control, new ideas are emerging using neural networks in approximate dynamic programming. Manufacturing systems can be very complex. There are many factors that may influence the performance of these systems; yet in many cases, the true relationship between these factors and the system outcomes is not fully understood. Neural networks have been given a great deal of attention in recent years with their ability to learn complex mappings even when presented with a partial, and even noisy, set of data. This has resulted in their being considered as a means to study and perhaps even optimize the performance of manufacturing operations. This chapter provides some examples from the literature of how neural networks are used in three different contexts in manufacturing systems. The categories (1) operational design, (2) operational decision-support systems, and (3) operational control are distinguished by the time context within which the models are used. Some examples make use of simulation models to produce training data, while some use actual production data. In some applications, the network is used to simply predict performance or outcomes, while in others the neural network is used in the determination of optimal parameters or to recommend good settings. Readers who wish to explore further examples of neural networks in manufacturing can examine Udo (1992), Zhang and Huang (1995), and Wang, Tang, and Roze (2001). We begin with two areas in which neural networks have found extensive use in manufacturing. Operational-system design has seen considerable use of neural networks as metamodels that can stand in place of the system, as we attempt to understand its behavior and optimize design parameters. Operational-system decision support refers to the use of neural networks as decision-support mechanisms in predicting system performance in response to certain settings of system parameters. We close with a short introduction to an area where we anticipate seeing growing numbers of applications, namely the use of approximate dynamic programming methods to develop real-time controllers for manufacturing systems.

In this paper, the output feedback based finite-horizon near optimal regulation of nonlinear affine discrete-time systems with unknown system dynamics is considered by using neural networks (NNs) to approximate Hamilton-Jacobi-Bellman (HJB) equation solution. First, a NN-based Luenberger observer is proposed to reconstruct both the system states and the control coefficient matrix. Next, reinforcement learning methodology with actor-critic structure is utilized to approximate the time-varying solution, referred to as the value function, of the HJB equation by using a NN. To properly satisfy the terminal constraint, a new error term is defined and incorporated in the NN update law so that the terminal constraint error is also minimized over time. The NN with constant weights and time-dependent activation function is employed to approximate the time-varying value function which is subsequently utilized to generate the finite-horizon near optimal control policy due to NN reconstruction errors. The proposed scheme functions in a forward-in-time manner without offline training phase. Lyapunov analysis is used to investigate the stability of the overall closed-loop system. Simulation results are given to show the effectiveness and feasibility of the proposed method.