Distributed control of leader-follower systems under adversarial inputs using reinforcement learning

This paper solves the containment problem of multi-agent systems on undirected graph with multiple active leaders using off-policy reinforcement learning (RL). The leaders are active in the sense that there exists bounded control input in the dynamics which is unknown to all followers and the followers are heterogeneous with different dynamics. Not only the steady states of agent i but also the transient trajectories are taken into account to impose optimality to the proposed containment control. Inhomogeneous algebraic Riccati equations (ARE) are derived to solve the optimal containment control protocol. To avoid the requirement of agents' dynamics to obtain containment control, an off-policy RL algorithm is developed to solve the inhomogeneous AREs online in real time and without requiring any knowledge of the agents' dynamics. Finally, a simulation example is presented to illustrate the effectiveness of the proposed algorithm.

To behave adaptively, we must learn from the consequences of our actions. Doing so is difficult when the consequences of an action follow a delay. This introduces the problem of temporal credit assignment. When feedback follows a sequence of decisions, how should the individual assign credit to the intermediate actions that comprise the sequence? Research in reinforcement learning provides 2 general solutions to this problem: model-free reinforcement learning and model-based reinforcement learning. In this review, we examine connections between stimulus-response and cognitive learning theories, habitual and goal-directed control, and model-free and model-based reinforcement learning. We then consider a range of problems related to temporal credit assignment. These include second-order conditioning and secondary reinforcers, latent learning and detour behavior, partially observable Markov decision processes, actions with distributed outcomes, and hierarchical learning. We ask whether humans and animals, when faced with these problems, behave in a manner consistent with reinforcement learning techniques. Throughout, we seek to identify neural substrates of model-free and model-based reinforcement learning. The former class of techniques is understood in terms of the neurotransmitter dopamine and its effects in the basal ganglia. The latter is understood in terms of a distributed network of regions including the prefrontal cortex, medial temporal lobes, cerebellum, and basal ganglia. Not only do reinforcement learning techniques have a natural interpretation in terms of human and animal behavior but they also provide a useful framework for understanding neural reward valuation and action selection.

Reinforcement learning (RL) is one of the popular methods for intelligent control and decision making in the field of robotics recently. The goal of RL is to learn an optimal policy of the agent by interacting with the environment via trail and error. There are two main algorithms for RL problems, including model-free and model-based methods. Model-free RL is driven by historical trajectories and empirical data of the agent to optimize the policy, which needs to take actions in the environment to collect the trajectory data and may cause the damage of the robot during training in the real environment. The main different between model-based and model-free RL is that a model of the transition probability in the interaction environment is employed. Thus the agent can search the optimal policy through internal simulation. However, the model of the transition probability is usually estimated from historical data in a single environment with statistical errors. Therefore, an issue is faced by the agent is that the optimal policy is sensitive to perturbations in the model of the environment which can lead to serious degradation in performance. Robust RL aims to learn a robust optimal policy that accounts for model uncertainty of the transition probability to systematically mitigate the sensitivity of the optimal policy in perturbed environments. In this overview, we begin with an introduction to the algorithms in RL, then focus on the model uncertainty of the transition probability in robust RL. In parallel, we highlight the current research and challenges of robust RL for robot control. To conclude, we describe some research areas in robust RL and look ahead to the future work about robot control in complex environments.

Animals integrate knowledge about how the state of the environment evolves to choose actions that maximise reward. Such goal-directed behaviour - or model-based (MB) reinforcement learning (RL) - can flexibly adapt choice to changes, being thus distinct from simpler habitual - or model-free (MF) RL - strategies. Previous inactivation and neuroimaging work implicates prefrontal cortex (PFC) and the caudate striatal region in MB-RL; however, details are scarce about its implementation at the single-neuron level. Here, we recorded from two PFC regions – the dorsal anterior cingulate cortex (ACC) and dorsolateral PFC (DLPFC), and two striatal regions, caudate and putamen – while two rhesus macaques performed a sequential decision-making (two-step) task in which MB-RL involves knowledge about the statistics of reward and state transitions. All four regions, but particularly the ACC, encoded the rewards received and tracked the probabilistic state transitions that occurred. However, ACC (and to a lesser extent caudate) encoded the key variables of the task - namely the interaction between reward, transition and choice – which underlies MB decision-making. ACC and caudate neurons also encoded MB-derived estimates of choice values. Moreover, caudate value estimates of the choice options flipped when a rare transition occurred, demonstrating value update based on structural knowledge of the task. The striatal regions were unique (relative to PFC) in encoding the current and previous rewards with opposing polarities, reminiscent of dopaminergic neurons, and indicative of a MF prediction error. Our findings provide a deeper understanding of selective and temporally dissociable neural mechanisms underlying goal-directed behaviour.

Over the last couple of decades the demand for high precision and enhanced performance of physical systems has been steadily increasing. This demand often results in miniaturization and complex design, thus increasing the need for complex nonlinear control methods. Some of the state of the art nonlinear methods are stymied by the requirement of full state information, model and parameter uncertainties, mathematical complexity, etc. For many scenarios it is nearly impossible to consider all the uncertainties during the design of a feedback controller. Additionally, while designing a modelbased nonlinear control there is no standard mechanism to incorporate performance measures. Some of the mentioned issues can be addressed by using online learning. Animals and humans have the ability to share, explore, act or respond, memorize the outcome and repeat the task to achieve a better outcome when they encounter the same or a similar scenario. This is called learning from interaction. One instance of this approach is reinforcement learning (RL). However, RL methods are hindered by the curse of dimensionality, non-interpretability and non-monotonic convergence of the learning algorithms. This can be attributed to the intrinsic characteristics of RL, as it is a modelfree approach and hence no standard mechanism exists to incorporate a priori model information. In this thesis, learning methods are proposed which explicitly use the available system knowledge. This can be seen as a new class of approaches that bridge model-based and model-free methods. These methods can address some of the hurdles mentioned earlier. For example, i) a prior system information can speed up the learning, ii) new control objectives can be achieved which otherwise would be extremely difficult to attain using only model-based methods, iii) physical meaning can be attributed to the learned controller. The developed approach is as follows: themodel of the given physical system is represented in the port-Hamiltonian (PH) form. For the system dynamics in PH form a passivity-based control (PBC) law is formulated, which often requires the solution to a set of partial differential equations (PDEs). Instead of finding an analytical solution, the PBC control law is parameterized using an unknown parameter vector. Then, by using a variation of the standard actor-critic learning algorithm, the unknown parameters can be learned online. Using the principles of stochastic approximation theory, a proof of convergence for the developed method is shown. The proposedmethods are evaluated for the stabilization and regulation ofmechanical and electro-mechanical systems. The simulation and experimental results show comparable learning curves. In the final part of the thesis a novel integral reinforcement learning approach is developed to solve for the optimal output tracking control problem for a set of linear heterogeneous multi-agent systems. Unlike existing methods, this approach does not need to solve either the output regulator equation or requires a p-copy of the leader’s dynamics in the agent’s control law. A detailed numerical evaluation has been conducted to show the feasibility of the developed method.

In this paper, the problem of formation control of heterogeneous multi-agent systems is investigated. The heterogeneous multi-agent system is composed of first-order and second-order agents, where the second-order agents cannot obtain the velocity information. Firstly, without considering the input saturation, the distributed formation control protocols are designed for multi-agent systems with static leader and active leader respectively. By using Lyapunov stability theory and LaSalle's invariance principle, the protocols are proved to be effective. Then due to the actor saturation, the formation protocols for multi-agent systems with input saturation based on distributed observers and auxiliary systems are presented. Finally, the simulation results are given to illustrate the effectiveness of the proposed protocols.

This paper mainly studied the $H_{\infty}$ formation control of multi-agent system with external disturbances and time delays. For the second-order multi-agent system, the distributed formation control protocol is designed, and used the Lyapunov stability theory to analyze the sufficient conditions for the given system to achieve asymptotic stability by the method of linear matrix inequality, so that the multi-agent system can achieve the desired formation and speed of the agent, and also can meet the $H_{\infty}$ performance index. Finally, a simulation example is used to verify the validity of the conclusion.

This article studies the event-triggered output feedback consensus problem for a class of high-order uncertain nonlinear multiagent systems with totally unknown non-identical control directions. Firstly, the reduced-order filters are constructed by utilizing the dynamic gain technique to compensate the system uncertainties and unmeasurable states of the followers. Based on the filter states and output information, the asymptotic output consensus result is guaranteed with the developed distributed adaptive output feedback control protocol, while the inherent issue "complexity explosion" in backstepping design is avoided. Then, to solve the interaction of multiple Nussbaum functions in a single inequality, the Nussbaum functions with different frequencies are adopted, with which the unknown directions of multiagent system are no longer limited in identical. In addition, we propose a dynamic triggering strategy with time-varying threshold parameters to reduce the updates of controller without Zeno phenomenon. Finally, simulation results are provided to illustrate the effectiveness of theoretical algorithm.

Purpose This paper proposes an adaptive distributed observer-based optimal control protocol designed to tackle the challenges posed by unavailable state information, without relying on global information, while ensuring optimal system performance in heterogeneous multi-agent systems. The purpose of this paper is to achieve distributed output tracking control under these constraints. Design/methodology/approach An adaptive distributed observer using single adaptive and local static gains estimates leader’s output. A feedback-feedforward control policy is designed for optimal tracking. Unlike conventional methods, this reduces communication load and broadens applications when leader’s state is unmeasurable. Findings A numerical simulation example related to autonomous ground vehicle is given to demonstrate that the proposed control algorithm can ensure output tracking in heterogeneous multi-agent systems. Research limitations/implications Although the proposed algorithm can effectively guarantee output tracking, further research is needed to design an optimal controller for model-free system dynamics in unknown environments. Practical implications The proposed method can be implemented for formation control of multi-robot systems since each robot can measure the relative output information, thereby effectively overcoming the limitation of unavailable system states. Originality/value This paper introduces a novel adaptive distributed observer-based optimal control protocol for heterogeneous multi-agent systems, uniquely relying on a single adaptive gain and local output information, eliminating the need for global network data. This reduces communication overhead and enhances applicability in scenarios where the leader’s state is unmeasurable. Unlike existing methods, it integrates optimal tracking control via the Bellman equation, considering transient performance. Its value lies in practical implementation for multi-robot formation control, though limitations include the need for further research into model-free systems in unknown environments.

In this paper, a novel robust distributed consensus control scheme based on event-triggered adaptive sliding mode control is proposed for multiagent systems with unknown disturbances in a leader-follower framework. First, an adaptive multivariate disturbance observer is utilized to compensate for the disturbance of each agent. Next, a distributed consensus control protocol is constructed via integral sliding mode control, in which a novel adaptive law is designed for the switching gain to overcome the unknown perturbations. An event-triggered strategy is designed to update the control input. Furthermore, the feasibility of the proposed scheme is rigorously analyzed by Lyapunov theory, and a lower bound expression for the inter-event time is derived to guarantee that Zeno behavior can be excluded. The proposed nonlinear consensus algorithm is remarkable in that it does not require any information about the bounds of the disturbances. Finally, compared with existing methods, the proposed algorithm is validated through detailed numerical simulations. In addition, the proposed algorithm is applied to a group of UAVs in this paper, and the results show that it has more application value.

This article investigates the leader-following consensus problem of discrete-time nonlinear multiagent systems (MASs) over fading channels. With the consideration of the transmission among followers maybe affected by the fading networks, the nonidentical fading channels model is constructed. To reduce the transmission network burden, the dynamical event-triggered mechanism (DETM) is developed. Different from most of existing event-triggered strategies, the threshold parameter in the developed dynamical event-triggering condition is dynamically adjusted according to a dynamic rule. Based on the DETM, a distributed consensus control protocol is designed under fading channels. Then, sufficient criteria are provided to ensure that MASs can achieve the leaser-following consensus, and satisfy the H∞ performance index in the presence of fading channels. The desired controller parameters can be derived in terms of solutions of matrix inequalities that are lightly solvable. In the end, simulation results show that the designed dynamical event transmission policy is capable of diminishing communication burden more promptly and effectively than some existing ones.

Observer-based consensus of multi-agent systems under odd distributed impulsive control protocol

This paper investigates finite-horizon optimal consensus control problem for unknown multiagent systems with state delays. It is well known that optimal consensus control is the solutions to the coupled Hamilton-Jacobi-Bellman (HJB) equations. An off-policy reinforcement learning (RL) algorithm is developed to learn the two-stage optimal consensus solutions to the coupled time-varying HJB equations using the measurable state data instead of the knowledge of the state-delayed system dynamics. Subsequently, for each agent, a single critic neural network (NN) is utilized to approximate the time-varying cost function and help to calculate optimal consensus control policy. Based on the method of weighted residuals, adaptive weight update laws for the critic NNs are proposed. Finally, the simulation results are provided to illustrate the effectiveness of the proposed off-policy RL method.

This paper is concerned with consensus of heterogeneous nonlinear multiagent systems via distributed control. Both the cases of leaderless and leader-following are systematically investigated. Different from some existing results, completed consensus can be reached in this paper among heterogeneous multiagent network instead of bounded-consensus. First, a novel distributed control protocol is proposed, and some general consensus criterions are derived for multiagent systems without leader. Second, a leader with unknown but bounded input for the heterogeneous multiagent network is considered; aperiodically intermittent communications among followers are considered to avoid channel blocking in this case. Finally, two simulation examples are presented to verify the effectiveness of the main results.

This paper proposes Cooperative and competitive Reinforcement And Imitation Learning (CRAIL) for selecting an appropriate policy from a set of multiple heterogeneous modules and training all of them in parallel. Each learning module has its own network architecture and improves the policy based on an off-policy reinforcement learning algorithm and behavior cloning from samples collected by a behavior policy that is constructed by a combination of all the policies. Since the mixing weights are determined by the performance of the module, a better policy is automatically selected based on the learning progress. Experimental results on a benchmark control task show that CRAIL successfully achieves fast learning by allowing modules with complicated network structures to exploit task-relevant samples for training.