Lyapunov Redesign-Based Optimal Consensus Control for Multi-Agent Systems With Uncertain Dynamics

This paper deals with the optimal bipartite consensus control (OBCC) for heterogeneous multi-agent systems (MASs) with time-delay based on reinforcement learning (RL) scheme. MASs, denoted by communication network, is modeled as a signed graph, and the competitive (cooperative) relationship is represented by negative (positive) edges. Firstly, Heterogeneous MASs is depicted by cooperative and competitive (hereinafter referred to as coopetition) network, and local neighborhood bipartite consensus errors and their performance index function (PIF) are put forward to formulate OBCC problem. Secondly, a proper model reduction method is developed to tackle time-delay in coopetition network, and then MASs with time-delay can be transformed into a delay-free MASs. Thirdly, a RL algorithm is employed to solve the solution of Hamilton-Jacobi-Bellman (HJB) equations under OBCC laws, and substantially we adopt neural networks (NNs) to calculate the control policy and PIF online. Finally, the pivotal simulation validates availability of proposed algorithm.

Purpose This paper proposes an adaptive distributed observer-based optimal control protocol designed to tackle the challenges posed by unavailable state information, without relying on global information, while ensuring optimal system performance in heterogeneous multi-agent systems. The purpose of this paper is to achieve distributed output tracking control under these constraints. Design/methodology/approach An adaptive distributed observer using single adaptive and local static gains estimates leader’s output. A feedback-feedforward control policy is designed for optimal tracking. Unlike conventional methods, this reduces communication load and broadens applications when leader’s state is unmeasurable. Findings A numerical simulation example related to autonomous ground vehicle is given to demonstrate that the proposed control algorithm can ensure output tracking in heterogeneous multi-agent systems. Research limitations/implications Although the proposed algorithm can effectively guarantee output tracking, further research is needed to design an optimal controller for model-free system dynamics in unknown environments. Practical implications The proposed method can be implemented for formation control of multi-robot systems since each robot can measure the relative output information, thereby effectively overcoming the limitation of unavailable system states. Originality/value This paper introduces a novel adaptive distributed observer-based optimal control protocol for heterogeneous multi-agent systems, uniquely relying on a single adaptive gain and local output information, eliminating the need for global network data. This reduces communication overhead and enhances applicability in scenarios where the leader’s state is unmeasurable. Unlike existing methods, it integrates optimal tracking control via the Bellman equation, considering transient performance. Its value lies in practical implementation for multi-robot formation control, though limitations include the need for further research into model-free systems in unknown environments.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.

In this paper, we present an optimal control approach using Linear Matrix Inequalities (LMIs) for trajectory tracking control of a three-wheeled omnidirectional mobile robot in the presence of external disturbances on the robot's actuators and noise in the robot's sensor measurements. First, a state-space representation of the omnidirectional robot dynamics is derived using a point-mass dynamic model. Then, we propose an LMI-based full-state feedback H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -optimal controller for the tracking problem. The robot's tracking performance with the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -optimal controller is compared to its performance with a classical full-state feedback tracking controller in simulations with circular and bowtie-shaped reference trajectories. In order to evaluate our proposed controller in practice, we also implement the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -optimal and classical controllers for these reference trajectories on a three-wheeled omnidirectional robot. The H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -optimal controller guarantees stabilization of the robot motion and attenuates the effects of frictional disturbances and measurement noise on the robot's tracking performance. Using the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -optimal controller, the robot is able to track the reference trajectories with up to a 47.8% and 45.8% decrease in the maximum pose and twist errors, respectively, over a full cycle of the trajectory compared to the classical controller. The simulation and experimental results show that our LMI-based H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> -optimal controller is robust to undesired effects of disturbances and noise on the dynamic behavior of the robot during trajectory tracking and can outperform the classical controller in attenuating their effects.

This paper concentrates on the fixed-time optimal consensus issue of multi-agent systems (MASs) under false data injection (FDI) attacks. To mitigate FDI attacks on sensors and actuators that may cause systems to deviate from the reference trajectory, a zero-sum game framework is established, where the secure control protocol aims at the better system performance, yet the attacker plays a contrary role. By solving the Hamilton–Jacobi–Isaacs (HJI) equation related to the zero-sum game, an optimal secure tracking controller based on the event-triggered mechanism (ETM) is obtained to decrease the consumption of system resources while the fixed-time consensus can be guaranteed. Moreover, a critic-only online reinforcement learning (RL) algorithm is proposed to approximate the optimal policy, in which the critic neural networks are constructed by the experience replay-based approach. The unmanned aerial vehicle (UAV) systems are adopted to verify the feasibility of the presented approach.

Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

In this paper, we study the optimal control problem for Takagi–Sugeno (T-S) fuzzy systems with disturbances. Due to the presence of disturbances in the T-S fuzzy systems, a fuzzy optimal state feedback control approach is presented by employing a two-player zero-sum game. Since the analytical optimal control solutions and the worst-case disturbance policies of T-S fuzzy systems can be boiled down to solving the algebraic Riccati equations (AREs), which are difficult to be obtained directly, a model-free policy iteration (PI) learning algorithm is proposed to obtain their approximation solutions. It is proved that the developed fuzzy optimal state feedback controller can ensure the fuzzy systems to be asymptotically stable and satisfy the disturbance attenuation condition simultaneously. Also, the designed PI learning algorithm can converge to their optimal solutions. Finally, we apply the developed fuzzy optimal state feedback control method to the truck-trailer system and the simulation results demonstrated the effectiveness of the developed scheme.

Optimal control of interconnected systems with time-correlated noises: Application to vehicle platoon

This paper studies the decentralized optimal control of discrete-time system with inputdelay, where a large number of agents with the identical decoupling dynamical equations and the coupling cost function through the mean field are considered. The decentralized and centralized optimal controllers are proposed by the optimal tracking control of LQR problem with delay. They are proved that the optimal controllers and the optimal cost function of the centralized and decentralized solutions are equivalent for the optimal control problem. An illustrative example is given to show the efficiency of the decentralized optimal controllers.

In this study, a finite‐time online optimal controller was designed for a nonlinear wheeled mobile robotic system (WMRS) with inequality constraints, based on reinforcement learning (RL) neural networks. In addition, an extended cost function, obtained by introducing a penalty function to the original long‐time cost function, was proposed to deal with the optimal control problem of the system with inequality constraints. A novel Hamilton‐Jacobi‐Bellman (HJB) equation containing the constraint conditions was defined to determine the optimal control input. Furthermore, two neural networks (NNs), a critic and an actor NN, were established to approximate the extended cost function and the optimal control input, respectively. The adaptation laws of the critic and actor NN were obtained with the gradient descent method. The semi‐global practical finite‐time stability (SGPFS) was proved using Lyapunov's stability theory. The tracking error converges to a small region near zero within the constraints in a finite period. Finally, the effectiveness of the proposed optimal controller was verified by a simulation based on a practical wheeled mobile robot model.

This paper investigates the robust optimal formation control for the position subsystem of quadrotor unmanned aerial vehicles (UAVs) subject to external disturbances and collision constraints. To prevent collision with both members of the formation and external obstacles, a collision avoidance potential function is constructed using relative position and velocity information. The basic bounded control input can ensure the stable flight and collision avoidance of a quadrotor UAV system. Based on the approximate dynamic programming (ADP) framework and two‐player zero‐sum differential game theory, the optimal controller is designed to further enhance the control performance of the system. The optimal value function is approximated by a single layer neural network, which avoids solving complex nonlinear Hamilton‐JacobiIsaac (HJI) equation. The stability of the closed loop system is proved. The effectiveness of the robust optimal formation controller based on learning is validated through MATLAB simulation in two distinct scenarios.

Approximate Optimal Bipartite Consensus Control for Multi-Agent Systems Based on Neural Networks Driven with Least-Squares Historical Difference Matrices

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

Optimal yaw moment control law for improved vehicle handling

The centralized paradigm of a single controller and a single plant upon which modern control theory is built is no longer applicable to modern cyber-physical systems of interest, such as the power-grid, software defined networks or automated highways systems, as these are all large-scale and spatially distributed. Both the scale and the distributed nature of these systems has motivated the decentralization of control schemes into local sub-controllers that measure, exchange and act on locally available subsets of the globally available system information. This decentralization of control logic leads to different decision makers acting on asymmetric information sets, introduces the need for coordination between them, and perhaps not surprisingly makes the resulting optimal control problem much harder to solve. In fact, shortly after such questions were posed, it was realized that seemingly simple decentralized optimal control problems are computationally intractable to solve, with the Wistenhausen counterexample being a famous instance of this phenomenon. Spurred on by this perhaps discouraging result, a concerted 40 year effort to identify tractable classes of distributed optimal control problems culminated in the notion of quadratic invariance, which loosely states that if sub-controllers can exchange information with each other at least as quickly as the effect of their control actions propagates through the plant, then the resulting distributed optimal control problem admits a convex formulation. The identification of quadratic invariance as an appropriate means of convexifying distributed optimal control problems led to a renewed enthusiasm in the controller synthesis community, resulting in a rich set of results over the past decade. The contributions of this thesis can be seen as being a part of this broader family of results, with a particular focus on closing the gap between theory and practice by relaxing or removing assumptions made in the traditional distributed optimal control framework. Our contributions are to the foundational theory of distributed optimal control, and fall under three broad categories, namely controller synthesis, architecture design and system identification. We begin by providing two novel controller synthesis algorithms. The first is a solution to the distributed H∞ optimal control problem subject to delay constraints, and provides the only known exact characterization of delay-constrained distributed controllers satisfying an H∞ norm bound. The second is an explicit dynamic programming solution to a two player LQR state-feedback problem with varying delays. Accommodating varying delays represents an important first step in combining distributed optimal control theory with the area of Networked Control Systems that considers lossy channels in the feedback loop. Our next set of results are concerned with controller architecture design. When designing controllers for large-scale systems, the architectural aspects of the controller such as the placement of actuators, sensors, and the communication links between them can no longer be taken as given -- indeed the task of designing this architecture is now as important as the design of the control laws themselves. To address this task, we formulate the Regularization for Design (RFD) framework, which is a unifying computationally tractable approach, based on the model matching framework and atomic norm regularization, for the simultaneous co-design of a structured optimal controller and the architecture needed to implement it. Our final result is a contribution to distributed system identification. Traditional system identification techniques such as subspace identification are not computationally scalable, and destroy rather than leverage any a priori information about the system's interconnection structure. We argue that in the context of system identification, an essential building block of any scalable algorithm is the ability to estimate local dynamics within a large interconnected system. To that end we propose a promising heuristic for identifying the dynamics of a subsystem that is still connected to a large system. We exploit the fact that the transfer function of the local dynamics is low-order, but full-rank, while the transfer function of the global dynamics is high-order, but low-rank, to formulate this separation task as a nuclear norm minimization problem. Finally, we conclude with a brief discussion of future research directions, with a particular emphasis on how to incorporate the results of this thesis, and those of optimal control theory in general, into a broader theory of dynamics, control and optimization in layered architectures.