Optimized Formation Control Using Simplified Reinforcement Learning for a Class of Multiagent Systems With Unknown Dynamics

In this paper, we described a class of nonlinear multi-agent dynamic systems in consensus problem which every agent is multi-dimensional dynamic system. The control objective is expect to find suitable consensus controller for every agent such that 1) guarantee all the signals in the dynamic systems remain bounded. 2) design the consensus controllers for every agent then the average-consensus behavior of the multi-agent systems can be obtained.

This paper deals with the optimal bipartite consensus control (OBCC) for heterogeneous multi-agent systems (MASs) with time-delay based on reinforcement learning (RL) scheme. MASs, denoted by communication network, is modeled as a signed graph, and the competitive (cooperative) relationship is represented by negative (positive) edges. Firstly, Heterogeneous MASs is depicted by cooperative and competitive (hereinafter referred to as coopetition) network, and local neighborhood bipartite consensus errors and their performance index function (PIF) are put forward to formulate OBCC problem. Secondly, a proper model reduction method is developed to tackle time-delay in coopetition network, and then MASs with time-delay can be transformed into a delay-free MASs. Thirdly, a RL algorithm is employed to solve the solution of Hamilton-Jacobi-Bellman (HJB) equations under OBCC laws, and substantially we adopt neural networks (NNs) to calculate the control policy and PIF online. Finally, the pivotal simulation validates availability of proposed algorithm.

In this article, an optimized leader-following consensus control scheme is proposed for the nonlinear strict-feedback-dynamic multi-agent system by learning from the controlling idea of optimized backstepping technique, which designs the virtual and actual controls of backstepping to be the optimized solution of corresponding subsystems so that the entire backstepping control is optimized. Since this control needs to not only ensure the optimizing system performance but also synchronize the multiple system state variables, it is an interesting and challenging topic. In order to achieve this optimized control, the neural network approximation-based reinforcement learning (RL) is performed under critic-actor architecture. In most of the existing RL-based optimal controls, since both the critic and actor RL updating laws are derived from the negative gradient of square of the Hamilton-Jacobi-Bellman (HJB) equation's approximation, which contains multiple nonlinear terms, their algorithm are inevitably intricate. However, the proposed optimized control derives the RL updating laws from the negative gradient of a simple positive function, which is correlated with the HJB equation; hence, it can be significantly simple in the algorithm. Meanwhile, it can also release two general conditions, known dynamic and persistence excitation, which are required in most of the RL-based optimal controls. Therefore, the proposed optimized scheme can be a natural selection for the high-order nonlinear multi-agent control. Finally, the effectiveness is demonstrated by both theory and simulation.

In this paper, we are concerned with the problem of containment control for a class of nonlinear multi-agent system. The local dynamics under consideration is governed by a nonlinear system with time delays and white Gaussian noise. The multi-agent system consists of both leaders and followers, and the containment control protocol proposed is an interaction between the leaders and followers characterised by time-varying delays and switching topology. Under a certain assumption on the connectivity of the whole network, by constructing a Lyapunov–Krasovskii functional, and using the average dwell time approach and some new analytical techniques, the sufficient conditions are established to ensure that each follower converges in mean square to the convex hull spanned by the leaders. A numerical example is presented to illustrate the effectiveness of the theoretical results.

Dual ML-ADHDP method for heterogeneous discrete-time nonlinear multi-agent systems with unknown dynamics and time delay

This article studies the semi-global adaptive sampled-data formation control problem for a class of nonlinear multiagent systems. The nonlinear growth rate is not limited, which expands the applicable scope of the obtained results. To remove the assumption that the system matrix of the leader system can be obtained by all follower systems, a sampled-data adaptive distributed observer is proposed to estimate not only the state but also the system matrix of the leader system, which are measured only at discrete time. All the estimation errors of the observers tend to zero as time tends to infinity. Then, by using the estimated state, a sampled-data adaptive control law is designed to achieve the formation control practically. Finally, a simulation example is shown to demonstrate the feasibility of our result.

This paper considers the consensus problem of two class of multi-agent systems with communication disturbance by using iterative learning control(ILC) method and both of them has a fixed information topology with a spanning tree. Firstly, a distributed iterative learning-based control protocol is designed for a time-varying linear multi-agent system, a sufficient condition is obtained such that the consensus problem is solved. By the same way, a sufficient condition is also given for a class of nonlinear time-varying multi-agent systems to solve consensus problem. The main contribution of the proposed results is that they can restrain any type of repetitive disturbance. The validity of them is verified by the simulation results.

SummaryIn this article, an optimized formation control algorithm is presented for unmanned surface vehicles (USVs) with collision avoidance and prescribed performance. The prescribed formation geometry is designed in the leader‐follower formation architecture, in which each vehicle tracks its intermediary leader with preserving a desired separation. A prescribed performance control design technique is introduced to guarantee the transient and steady‐state performance specifications on formation errors. Radial basis function neural networks (NNs) are employed to approximate modeling uncertainties including damping terms and unmodeled dynamics. Based on an actor‐critic learning strategy, a reinforcement learning (RL) algorithm is proposed to ensure the optimality of formation control and the specified tracking accuracy simultaneously, in which actor NNs take appropriate control behaviors by interacting with the external environment, and critic NNs evaluate the control performance and generate a reinforcement signal to actor NNs for facilitating the improvement of subsequent behaviors. Stability analysis shows that the proposed optimal formation controller achieves semi‐global uniform ultimate boundedness of closed‐loop adaptive systems with prescribed performance. Comparative simulation results illustrate the effectiveness and superiority of the presented control algorithm.

Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

A reinforcement learning (RL) based adaptive dynamic programming (ADP) is developed to learn the approximate optimal stabilization input of the servo mechanisms, where the unknown system dynamics are approximated with a three-layer neural network (NN) identifier. First, the servo mechanism model is constructed and a three-layer NN identifier is used to approximate the unknown servo system. The NN weights of both the hidden layer and output layer are synchronously tuned with an adaptive gradient law. An RL-based critic three-layer NN is then used to learn the optimal cost function, where NN weights of the first layer are set as constants, NN weights of the second layer are updated by minimizing the squared Hamilton-Jacobi-Bellman (HJB) error. The optimal stabilization input of the servomechanism is obtained based on the three-layer NN identifier and RL-based critic NN scheme, which can stabilize the motor speed from its initial value to the given value. Moreover, the convergence analysis of the identifier and RL-based critic NN is proved, the stability of the cost function with the proposed optimal input is analyzed. Finally, a servo mechanism model and a complex system are provided to verify the correctness of the proposed methods.

In this article, the issue of leader-following consensus control is addressed for nonlinear multiagent systems with unknown dynamics under switching topologies. The packaged unknown nonlinear dynamics is approximated by neural networks, and the approximate error is compensated by an adaptive robust term. A novel neuro-adaptive consensus controller is derived by incorporating the backstepping recursive design and the average dwell-time method. It is demonstrated that the boundness of the closed-loop multiagent systems is ensured under switching topologies, and the follower agents have the competence to chase the leader agent with the tracking errors asymptotically converging to zero. Simulation investigations have verified the availability of the given leader-following consensus control algorithm.

Adaptive fully distributed consensus for a class of heterogeneous nonlinear multi-agent systems

This paper investigates the consensus tracking problem with predefined performances requirements for a class of unknown nonlinear multi-agent systems with hysteresis quantizer and external disturbances under a directed graph topology. Neural network observers are designed to estimate unmeasurable states and the the consensus tracking problem with performance requirements is transformed to a stabilization problem by prescribed performance error transformation schemes. The novel consensus protocol can be applied to a more general class of nonlinear multi-agent systems since the Lipschitz condition is avoided and state information is not required. It is strictly proved that all signals in the closed-loop systems are cooperatively uniformly ultimately bounded and both the transient and steady performances of the consensus tracking satisfy prescribed performance requirements. Finally, two numerical examples are presented to validate the effectiveness of the proposed strategy.

This article addresses the event-based adaptive optimal fault-tolerant consensus control problem for a class of nonlinear multiagent systems (MASs) under a directed graph. It can be solved that both control gains and actuator failure parameters are unknown in considered MASs, which enhances the practicability of optimal consensus control. First, a reinforcement learning-based distributed optimal control law is designed by constructing the identifier-critic-actor learning networks. Furthermore, by utilizing the distributed optimal control law as an auxiliary variable, an adaptive fault-tolerant controller is proposed to effectively compensate for actuator failures. Meanwhile, a co-design scheme is proposed for the construction of an event-triggered control input with the fault-tolerant property. It is proven that the designed control input ensures the boundedness of closed-loop systems through rigorous stability analysis. Finally, the effectiveness of the developed approach can be illustrated via simulations of numerical nonlinear MASs and a group of autonomous underwater vehicles.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex