Hierarchical Multiagent Formation Control Scheme via Actor-Critic Learning.

A novel policy iteration algorithm, called the continuous-time time-varying (CTTV) policy iteration algorithm, is presented in this paper to obtain the optimal control laws for infinite horizon CTTV nonlinear systems. The adaptive dynamic programming (ADP) technique is utilized to obtain the iterative control laws for the optimization of the performance index function. The properties of the CTTV policy iteration algorithm are analyzed. Monotonicity, convergence, and optimality of the iterative value function have been analyzed, and the iterative value function can be proven to monotonically converge to the optimal solution of the Hamilton-Jacobi-Bellman (HJB) equation. Furthermore, the iterative control law is guaranteed to be admissible to stabilize the nonlinear systems. In the implementation of the presented CTTV policy algorithm, the approximate iterative control laws and iterative value function are obtained by neural networks. Finally, the numerical results are given to verify the effectiveness of the presented method.

To intercept a maneuvering target with a predetermined impact angle, a computational intelligence guidance law was proposed in this paper. Based on the theory of two-player zero-sum differential games, this problem is resolved efficiently by solving the Hamilton–Jacobi–Isaacs (HJI) equation. The Nash equilibrium solution of HJI equation can be solved with a policy iteration (PI) algorithm. Instead of using the offline PI algorithm, an online PI algorithm is introduced, in which the disturbance and control policies can be updated simultaneously. It can be proved that the online PI algorithm is a replacement for Newton’s iterative algorithm, the convergence of which is ensured by Kantorovich’s theorem. In the scenario of missiles intercepting targets, an adaptive critic structure based on a neural network (NN) is proposed to implement the online PI algorithm. Only one critic NN approximator is used in the PI algorithm to calculate a value function and the approximate Nash equilibrium solution. It is not necessary to acquire the exact internal dynamics information of nonlinear systems on the basis of online data sampling. The effectiveness of the computational intelligence guidance law is proven by simulation results.

In this paper, a new generalized policy iteration (GPI) based adaptive dynamic programming (ADP) algorithm is developed to solve optimal control problems for infinite horizon discrete-time nonlinear systems. The GPI algorithm is a general idea of interacting policy and value iteration algorithms of ADP. There are two iteration indices, which iterate for policy improvement and policy evaluation, respectively, in the GPI algorithm. The convergence properties of the GPI algorithm are developed. Finally, simulation results are presented to illustrate the performance of the developed algorithm.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

Optimal distributed time-varying formation control for second-order multiagent systems: LQR-based method

This article is concerned with the asynchronous $H_{\infty }$ control design based on model-free policy iteration (PI) algorithm for a class of discrete-time hidden Markov jump system, where a hidden Markov model is developed to characterize the asynchronous phenomenon between the controller modes and the system modes. A pair of zero-sum asynchronous control and disturbance strategies are constructed to achieve a tradeoff between value function and control performance. The presented approach shows two pivotal aspects: 1) the asynchronous PI algorithm is not dependent on strict temporal alignment between the controller and the system's dynamics, enhancing flexibility of the control scheme and 2) it relies on the collected data to solve the algebraic Reccati equation iteratively, which avoids the need for system-internal and transfer probability information, and circumvents the interference of coupled terms. Subsequently, it is verified that the designed PI algorithm monotonically converges to an optimal solution and the system based on this optimal solution is stochastically stable in the mean-square sense. Finally, the effectiveness of this approach is validated by conducting a simulation experiment on a DC motor device system.

UUV two-phase formation and priority avoidance control considering steering amplitude limitation

In this paper, the distributed zero-sum differential game problem for multi-agent nonlinear systems is investigated by using adaptive dynamic programming (ADP) technique, which brings together the distributed control, zero-sum differential graphical game and ADP. To find the cooperative Nash-equilibrium solutions of the associated coupled Hamilton-Jacobi-Isaacs (HJI) equations, the ADP technique is employed, in which a critic network is constructed to approximate the cooperative cost function online with a novel updating law. By utilizing the Lyapunov direct method, the closed-loop system and weight estimation errors are guaranteed to be uniformly ultimately bounded (UUB). Finally, simulation results demonstrate the effectiveness of the proposed method.

In this paper, a formation control problem is studied for a class of second-order nonlinear multi-agent systems (MASs) with unknown state-delays. In MAS formation control, the uncertainties of the system are approximated by neural networks (NNs). Moreover, a novel Lyapunov function is constructed to offset the unknown state delays. The effectiveness of the proposed control scheme is demonstrated by simulation results.

Formation control for nonlinear multi-agent systems by robust output regulation

This paper studies time-varying output formation control for linear heterogeneous multi-agent systems (MASs) under signed graphs, employing an adaptive dynamic event-based approach. The signed graph framework is used to model agent interactions, encompassing both cooperative and antagonistic behaviours. Firstly, a data-driven learning algorithm is given for the estimation of the unknown leader system matrix. Secondly, for a single leader MAS, a distributed adaptive dynamic event-based bipartite compensator is designed to estimate the leader's state. This triggering function utilises the internal dynamic variable to dynamically adjust the threshold, ensuring fewer triggering instances compared to traditional static thresholds. Thirdly, a controller integrating the compensator and observer is developed to solve the time-varying output formation control problem for heterogeneous MASs with signed graphs under a single leader. Subsequently, the findings are extended to multi-leader MASs, providing a solution for the formation control problem in such scenarios. Finally, several illustrative examples are provided to validate the theoretical results.

This paper introduces an online fault-tolerant control method for nonlinear time-delay systems with actuator faults using a policy iteration algorithm. A new performance index function is proposed to account for time-delay states, from which control laws are derived. The approach combines policy iteration with a fault compensator, solving the Hamilton-Jacobi-Bellman equation associated with this innovative value function via a critic neural network. Actuator faults are reconstructed for online fault compensation without the need for fault detection and isolation. The Lyapunov functional analysis demonstrates the closed-loop system's asymptotic stability in the presence of actuator faults and time-delay states. Simulation results validate the effectiveness of the proposed fault compensation strategy. The key contribution of this paper is the incorporation of time-delay states into the value function of the policy iteration algorithm, addressing the problem of fault compensation in nonlinear time-delay systems.

Formation control and obstacle avoidance for multi-agent systems have attracted more and more attention. In this paper, the problems of formation control and obstacle avoidance are investigated by means of a consensus algorithm. A novel distributed control model is proposed for the multi-agent system to form the anticipated formation as well as achieve obstacle avoidance. Based on the consensus algorithm, a distributed control function consisting of three terms (formation control term, velocity matching term, and obstacle avoidance term) is presented. By establishing a novel formation control matrix, a formation control term is constructed such that the agents can converge to consensus and reach the anticipated formation. A new obstacle avoidance function is developed by using the modified potential field approach to make sure that obstacle avoidance can be achieved whether the obstacle is in a dynamic state or a stationary state. A velocity matching term is also put forward to guarantee that the velocities of all agents converge to the same value. Furthermore, stability of the control model is proven. Simulation results are provided to demonstrate the effectiveness of the proposed control.

In this study1, formation control and collision avoidance problem of multi-agent system with switching communication topology has been researched. Due to changes in the communication topology, collision between agents may occur during the course of formation, especially when the distance between agents is very close. Therefore, we have proposed a stable and effective formation control law with collision avoidance capability for the multi-agent system. It is assumed that the topology graph is directed and switching. Besides, the agent is modeled as a single integrator. Our formation control law is the combination of a consensus-based formation control method and a collision avoidance algorithm. The collision avoidance algorithm is adopted based on the artificial potential field (APF) method. In addition, we have adapted the potential function to ensure connectivity. The convergence is guaranteed even in the simultaneous execution of the consensus-based controller and the collision avoidance algorithm. And also, all agents can converge to the desired position rapidly. Some simulations have shown the validation of our scheme.

Multi-agent formation control is an important part of distributed perception and cooperation, which is convenient to complete various complex tasks and would be a key research direction in the future. This paper reviews the corresponding problems of formation control and the existing centralized and distributed formation control strategies. In particular, we discuss four types of distributed formation control methods based on position and displacement in the global coordinate system and distance and bearing in the nonglobal coordinate system, respectively. Moreover, this paper analyzes affine formation which does not require the global coordinate system. Combined with the current practical applications of multi-agent systems, the latest research for the formation control of the unmanned aerial vehicle (UAV), unmanned ground vehicle (UGV), unmanned surface vehicle (USV) and autonomous underwater vehicle (AUV) is given. Finally, the challenges and opportunities in this burgeoning field are discussed.