MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models

This paper investigates the error accumulation problem in centralized training and decentralized execution (CTDE) policy‐based multi‐agent reinforcement learning (MARL) algorithms, which arises from local observation inaccuracies. To address this issue, we propose a novel MARL algorithm that incorporates imitation learning using local observations. Firstly, by analysing the multi‐agent proximal policy optimization algorithm and examining the problems arising when global states are replaced with local observations, it is proved that insufficient observations can lead to information loss, thereby introducing errors of advantage function, and it is demonstrated that the generalized advantage estimation method accumulates errors during the training process. Then, imitation learning is introduced and a novel training framework that combines reinforcement learning and imitation learning is proposed. During the reinforcement learning phase, an MARL agent trained with global observations acts as an expert. Subsequently, imitation learning is applied to train another agent that mimics the expert's decisions using only local observations. Finally, the effectiveness of this algorithm is verified in some commonly used multi‐agent environments, which demonstrates its superior performance compared to traditional multi‐agent reinforcement learning algorithms.

Deep reinforcement learning is the technology of artificial neural networks in the field of decision-making and control. The traditional model-free reinforcement learning algorithm requires a large amount of environment interactive data to iterate the algorithm. This model’s performance also suffers due to low utilization of training data, while the model-based reinforcement learning (MBRL) algorithm improves the efficiency of the data, MBRL locks into low prediction accuracy. Although MBRL can utilize the additional data generated by the dynamic model, a system dynamics model with low prediction accuracy will provide low-quality data and affect the algorithm’s final result. In this paper, based on the A3C (Asynchronous Advantage Actor-Critic) algorithm, an improved model-based deep reinforcement learning algorithm using a learning degree network (MBRL-LDN) is presented. By comparing the differences between the predicted states outputted by the proposed multidynamic model and the original predicted states, the learning degree of the system dynamics model is calculated. The learning degree represents the quality of the data generated by the dynamic model and is used to decide whether to continue to interact with the dynamic model during a particular episode. Thus, low-quality data will be discarded. The superiority of the proposed method is verified by conducting extensive contrast experiments.

Unmanned aerial vehicle (UAV) confrontation scenarios play a crucial role in the study of agent behavior selection and decision planning. Multi-agent reinforcement learning (MARL) algorithms serve as a universally effective method guiding agents toward appropriate action strategies. They determine subsequent actions based on the state of the agents and the environmental information that the agents receive. However, traditional MARL settings often result in one party agent consistently outperforming the other party due to superior strategies, or both agents reaching a strategic stalemate with no further improvement. To solve this issue, we propose a semi-static deep deterministic policy gradient algorithm based on MARL. This algorithm employs a centralized training and decentralized execution approach, dynamically adjusting the training intensity based on the comparative strengths and weaknesses of both agents’ strategies. Experimental results show that during the training process, the strategy of the winning team drives the losing team’s strategy to upgrade continuously, and the relationship between the winning team and the losing team keeps changing, thus achieving mutual improvement of the strategies of both teams. The semi-static reinforcement learning algorithm improves the win-loss relationship conversion by 8% and reduces the training time by 40% compared with the traditional reinforcement learning algorithm.

Aim and ApproachLarge cooling water system has great potential of energy saving due to improper operation control. The coupled hydraulic and thermodynamic characteristics among chillers, pumps, and cooling towers lead to the difficulty of cooling water system operation. This study investigates an advanced control approach of multi-agent reinforcement learning (RL) for a large cooling water system. The multiple RL agents interact with each other to deal with the complex coupled characteristics in the cooling water system. The proposed approach is implemented in a real-world cooling water system and validated with experiments.Scientific Innovation and RelevanceIn this study, a multi-agent RL approach is proposed to separately control the chillers, pumps, and cooling towers in the cooling water system. The soft actor critic algorithm is used in the RL agent, as it is an efficient and advanced RL algorithm for continuous and discrete control problems. The agents are designed to have private states, private actions, and shared rewards. The reward functions are carefully designed to balance the agents for global optima.To provide a realistic environment for RL training, a detailed physical modeling platform of a cooling water system is established and validated. This platform is capable of hydraulic and thermodynamic calculation of cooling water systems with customized numbers and characteristics of chillers, pumps, and cooling towers, which is scalable for different cooling water systems.The proposed multi-agent RL controller is implemented in the building automation system of a real cooling water system. A one-month experimental study of the multi-agent RL performance is conducted. The proposed multi-agent RL controller is compared with a single-agent RL controller and a rule-based controller through real-world experiments to demonstrate the energy performance. This is a pilot study of multi-agent RL approach to real-world cooling water system control.Preliminary Results and ConclusionsThe preliminary simulation study is conducted in the established cooling water system model. The electricity consumption simulation error is 4.0% in coefficient of variation of mean absolute error (CVMAE) compared with the measured 15-minute electricity consumption. The 6-month simulation shows that the single-agent RL can save 6.2% energy from the rule-based control, while the multi-agent RL can save 7.2% electricity consumption of the cooling water system. In the following study, the multi-agent RL implementation will be practiced with more experiments and applications for cooling water system control.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

This paper presents a GA-based multi-agent reinforcement learning bidding approach (GMARLB) for performing multi-agent reinforcement learning. GMARLB integrates reinforcement learning, bidding and genetic algorithms. The general idea of our multi-agent systems is as follows: There are a number of individual agents in a team, each agent of the team has two modules: Q module and CQ module. Each agent can select actions to be performed at each step, which are done by the Q module. While the CQ module determines at each step whether the agent should continue or relinquish control. Once an agent relinquishes its control, a new agent is selected by bidding algorithms. We applied GA-based GMARLB to the Backgammon game. The experimental results show GMARLB can achieve a superior level of performance in game-playing, outperforming PubEval, while the system uses zero built-in knowledge.

P2P power trading based on reinforcement learning for nanogrid clusters

Model-based offline reinforcement learning (RL) algorithms have emerged as a promising paradigm for offline RL. These algorithms usually learn a dynamics model from a static dataset of transitions, use the model to generate synthetic trajectories, and perform conservative policy optimization within these trajectories. However, our observations indicate that policy optimization methods used in these model-based offline RL algorithms are not effective at exploring the learned model and induce biased exploration, which ultimately impairs the performance of the algorithm. To address this issue, we propose Offline Conservative ExplorAtioN (OCEAN), a novel rollout approach to model-based offline RL. In our method, we incorporate additional exploration techniques and introduce three conservative constraints based on uncertainty estimation to mitigate the potential impact of significant dynamic errors resulting from exploratory transitions. Our work is a plug-in method and can be combined with classical model-based RL algorithms, such as MOPO, COMBO, and RAMBO. Experiment results of our method on the D4RL MuJoCo benchmark show that OCEAN significantly improves the performance of existing algorithms.

Distributed entropy-regularized multi-agent reinforcement learning with policy consensus

Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, the centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action space. The multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now, the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Most existing studies in MARL focus on designing efficient communication and coordination among traditional Q-learning agents. This paper presents, for the first time, a fully scalable and decentralized MARL algorithm for the state-of-the-art deep RL agent, advantage actor critic (A2C), within the context of ATSC. In particular, two methods are proposed to stabilize the learning procedure, by improving the observability and reducing the learning difficulty of each local agent. The proposed multi-agent A2C is compared against independent A2C and independent Q-learning algorithms, in both a large synthetic traffic grid and a large real-world traffic network of Monaco city, under simulated peak-hour traffic dynamics. The results demonstrate its optimality, robustness, and sample efficiency over the other state-of-the-art decentralized MARL algorithms.

Reinforcement learning (RL) and multi-agent reinforcement learning (MARL) are disciplines concerned with defining automatically the behaviour of an agent, or a set of interacting agents, by means of reward mechanisms coming from the environment. An important research issue in the context of RL and MARL is the definition of approaches to combine the knowledge of multiple learning agents to improve the overall performance of the multi-agent system (MAS). This paper illustrates how to improve RL and MARL algorithms by utilizing results from multi-linear algebra such as tensors and tensor factorizations. In particular, the focus is on showing how to modify existing algorithms from literature to include a tensor factorization step applied to the Q-Tables learned by the individual agents to generalize the knowledge about the actions performed in the environment. The modified algorithms are then evaluated in three RL and MARL scenarios against their unmodified version to show the benefits of the tensor factorization step.

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning

Smart factories employ intelligent transportaton systems such as autonomous mobile robots (AMRs) to support real-time adjusted production flows for agile and flexible production. While decentralized transportation task execution provides a scalable multirobot system (MRS) for a smart factory, new coordination challenges arise in implementing such a system. Transportation-MRS collaborates with production-MRS to accommodate just-in-time (JIT) production, leading to nonstationary transportation tasks that transportation-MRS must learn and adapt to. Also, decentralized operation on a shared shop floor means that one robot cannot factor in peer robots’ task execution planning, leading to competitive collisions. Meanwhile, predictive coordination with communication among multiple learning and adapting intelligent robots is still an open problem. On top of identifying the aforementioned challenges, this article first proposes a multifloor transportation graph model to discretize transportation task execution and allow real-time adjustment of transportation paths toward collision-free. We introduce a unique collaborative multi-intelligent robot system approach taking each robot as a cyber–physical agent with automated artificial intelligence (AI) workflow. First, it includes a novel multiagent reinforcement learning (MARL) algorithm, where each robot predictively plans collision-avoidant paths. Second, we introduce a token-passing mechanism to resolve inevitable competitive collisions due to nonstationary tasks. The proposed approach innovatively uses the multifloor model as a domain model for planning. By allowing competitive collision to occur and resolve, a robot only needs to learn and adapt to uncertain parts of the environment—nonstationary tasks and peer robots’ paths. Computational experiments show that our approach is both sample-efficient and computationally efficient. The transportation-MRS quickly reaches near-optimal performance levels, which are empirically shown to scale with the number of robots involved.

Due to the advantages of a centralized critic to estimate the Q-function value and decentralized actors to optimize the agents’ policies, counterfactual multi-agent (COMA) stands out in most multi-agent reinforcement learning (MARL) algorithms. The sharing of policy parameters can improve sampling efficiency and learning effectiveness, but it may lead to a lack of policy diversity. Hence, to balance parameter sharing and diversity among agents in COMA has been a persistent research topic. In this paper, an effective training method for a COMA policy network based on a differential evolution (DE) algorithm is proposed, named DE-COMA. DE-COMA introduces individuals in a population as computational units to construct the policy network with operations such as mutation, crossover, and selection. The average return of DE-COMA is set as the fitness function, and the best individual of policy network will be chosen for the next generation. By maintaining better parameter sharing to enhance parameter diversity, multi-agent strategies will become more exploratory. To validate the effectiveness of DE-COMA, experiments were conducted in the StarCraft II environment with 2s_vs_1sc, 2s3z, 3m, and 8m battle scenarios. Experimental results demonstrate that DE-COMA significantly outperforms the traditional COMA and most other multi-agent reinforcement learning algorithms in terms of win rate and convergence speed.