Decision-making with cooperative multi-agent reinforcement learning

Cooperative multiagent reinforcement learning (MARL) has attracted significant attention and has the potential for many real-world applications. Previous arts mainly focus on facilitating the coordination ability from different aspects (e.g., nonstationarity and credit assignment) in single-task or multitask scenarios, ignoring the stream of tasks that appear in a continual manner. This ignorance makes the continual coordination an unexplored territory, neither in problem formulation nor efficient algorithms designed. Toward tackling the mentioned issue, this article proposes an approach, multiagent continual coordination via progressive task contextualization (MACPro). The key point lies in obtaining a factorized policy, using shared feature extraction layers but separated independent task heads, each specializing in a specific class of tasks. The task heads can be progressively expanded based on the learned task contextualization. Moreover, to cater to the popular centralized training with decentralized execution (CTDE) paradigm in MARL, each agent learns to predict and adopt the most relevant policy head based on local information in a decentralized manner. We show in multiple multiagent benchmarks that existing continual learning methods fail, while MACPro is able to achieve close-to-optimal performance. More results also disclose the effectiveness of MACPro from multiple aspects, such as high generalization ability.

The cooperative multiagent reinforcement learning (MARL) has been widely used in many practical applications. Despite its success, a fundamental issue arises in MARL that agents face the dilemma of whether to select the best action to maximize rewards or to acquire more information collectively by exploring the novel states/actions due to partial observability. To solve this issue, existing methods merge exploration and exploitation methods. However, these methods are always suboptimal and may lead to failure in finishing tasks. In this article, we theoretically prove the existence of a latent state that can guarantee the optimal individual and global policies. Moreover, we prove that such a latent state can be approximately obtained by local observations. Based on the analysis, we propose a method named unified MARL (UMARL), which is a weighted value function factorization approach unifying exploitation and exploration in one framework. Specifically, we design the agent representation network (ARN) and individual weighting networks (IWNs) to learn agents' unified representations and weights of credit. Moreover, a latent state regularizer (LSR) is designed to encourage agents' representations to approximate the latent state. Extensive experiments show that UMARL can achieve superior performance compared with 12 state-of-the-art methods on $m$ -step matrix game, level-based foraging (LBF), StarCraft II, and Google research football (GRF). The source code is available at: https://github.com/CrazyBayes/UMARL.

_ The paper presents a novel approach of cooperative multiagent reinforcement learning (CMARL) combined with A* search to address ship multicabin equipment layout considering pipe route, aiming to minimize pipe cost while considering practical requirements. The formulation is established through equipment simplification and grid marking, and A* search is utilized to value the pipe route. By designing equipment states, the equipment layout in each cabin is solved using a CMARL approach that involves three actions. Subsequently, comparative experiments were conducted on an engine room case by CMARL against genetic algorithm and single multiagent reinforcement learning methods under the condition of coupling with A* search. The parameter values for these methods were sampled using Latin Hypercube. The findings demonstrate that CMARL has superior combination properties. Keywords ship equipment layout; multicabin layout; cooperative multiagent reinforcement learning; A* search; pipe route

The effectiveness of multi-agent reinforcement learning (MARL) hinges largely on the meticulous arrangement of objectives. Yet, conventional MARL methods might not completely harness the inherent structures present in environmental states and agent relationships for goal organization. This study is conducted within the domain of military training simulations, which are typically characterized by complexity, heterogeneity, non-stationary and doctrine-driven environments with a clear organizational hierarchy and a top-down chain of command. This research investigates the approximation and integration of the organizational hierarchy into MARL for cooperative training scenarios, with the goal of streamlining the processes of reward engineering and enhancing team coordination. In the preliminary experiments, we employed two-tiered commander-subordinate feudal hierarchical (CSFH) networks to separate the prioritized team goal and individual goals. The empirical results demonstrate that the proposed framework enhances learning efficiency. It guarantees the learning of a prioritized policy for the commander agent and encourages subordinate agents to explore areas of interest more frequently, guided by appropriate soft constraints imposed by the commander.

A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using MARL methods for LLM collaboration and highlights the associated challenges.

The majority of multi-agent reinforcement learning (MARL) implementations aim to optimize systems with respect to a single objective, despite the fact that many real-world problems are inherently multi-objective in nature. Research into multi-objective MARL is still in its infancy, and few studies to date have dealt with the issue of credit assignment. Reward shaping has been proposed as a means to address the credit assignment problem in single-objective MARL, however it has been shown to alter the intended goals of a domain if misused, leading to unintended behaviour. Two popular shaping methods are potential-based reward shaping and difference rewards, and both have been repeatedly shown to improve learning speed and the quality of joint policies learned by agents in single-objective MARL domains. This work discusses the theoretical implications of applying these shaping approaches to cooperative multi-objective MARL problems, and evaluates their efficacy using two benchmark domains. Our results constitute the first empirical evidence that agents using these shaping methodologies can sample true Pareto optimal solutions in cooperative multi-objective stochastic games.

Cooperative multiplayer games are based on rules where players must collaborate to solve certain tasks. These games bring specific challenges when using Multi-Agent Reinforcement Learning (MARL), since they present requirements related to the training of these collaborative behaviors, such as partial observations, non-stationary, and the problem of credit assignment. One of the approaches used in MARL to solve these challenges is centralized training with decentralized execution. The idea is to use the available knowledge about the full state and information of the environment in the training phase, but policy learning takes place in a decentralized way, not depending on this knowledge. In this work, we study the approach of centralized training with decentralized execution. We seek to validate whether the division of knowledge about the environment (e.g. observations, perception of objects, enemies, obstacles) by different groups (different centralized critics) improves learning performance in multi-agent environments. Our results show that specifying a centralized critic per knowledge improves the training, but it also increases the time of the training process.

E-commerce multi-agent systems have some features of autonomy and commercial transaction that improve rapidly and become a powerful tool for buying and selling products online. While multi-agent systems can break interactions in distributed and heterogeneous environments, malicious behaviors of the agents are a potential hazard in distributed e-commerce applications. Hence, calculating the reliability and reputation of agents play a critical role in determining how interactions occur and who to interact with. The way of calculating the reliability and reputation of each agent should be based on different elements, and there are no standard elements to compute reliability and reputation of each agent in multi-agent environment. This paper analyses some recent proposed methods in calculating reputation and reliability of each agent. The ultimate aim of this study is to define the standard elements which should be considered to calculate the reliability and reputation of agents in e-commerce multi-agent environment.

The paper gives novel approach Multiagent Cooperative Reinforcement Learning by Expert Agents (MCRLEA) for dynamic decision making in the retail application. Furthermore, it put up different cooperation schemes for multiagent cooperative reinforcement learning i.e. EQ learning, EGroup, EDynamic, EGoal driven and Expert agents scheme. Implementation outcome includes a demonstration of recommended cooperation schemes that are competent enough to speedup the collection of agents that achieve excellent action policies. Accordingly this approach presents three retailer stores in the retail market place. Retailers can help to each other and can obtain profit from cooperation knowledge through learning their own strategies that just stand for their aims and benefit. The vendors are the knowledgeable agents in the hypothesis to employ cooperative learning to train in the circumstances. Assuming significant hypothesis on the vendor’s stock policy, restock period, arrival process of the consumers, the approach is formed as Markov decision process model that makes it possible to design learning algorithms. The proposed algorithms noticeably learn dynamic consumer performance. Moreover, the paper illustrates results of Cooperative Reinforcement Learning Algorithms of three shop agents for the period of one year sale duration and then demonstrated the results using proposed approach for three shop agents for the period of one year sale duration. The results obtained by the proposed expert agent based cooperation approach show that such methods can put into a quick convergence of agents in the dynamic environment.

Centralized Training for Decentralized Execution, where agents are trained offline in a centralized fashion and execute online in a decentralized manner, has become a popular approach in Multi-Agent Reinforcement Learning (MARL). In particular, it has become popular to develop actor-critic methods that train decentralized actors with a centralized critic where the centralized critic is allowed access global information of the entire system, including the true system state. Such centralized critics are possible given offline information and are not used for online execution. While these methods perform well in a number of domains and have become a de facto standard in MARL, using a centralized critic in this context has yet to be sufficiently analyzed theoretically or empirically. In this paper, we therefore formally analyze centralized and decentralized critic approaches, and analyze the effect of using state-based critics in partially observable environments. We derive theories contrary to the common intuition: critic centralization is not strictly beneficial, and using state values can be harmful. We further prove that, in particular, state-based critics can introduce unexpected bias and variance compared to history-based critics. Finally, we demonstrate how the theory applies in practice by comparing different forms of critics on a wide range of common multi-agent benchmarks. The experiments show practical issues such as the difficulty of representation learning with partial observability, which highlights why the theoretical problems are often overlooked in the literature.

Previous deep multi-agent reinforcement learning (MARL) algorithms have achieved impressive results, typically in symmetric and homogeneous scenarios. However, asymmetric heterogeneous scenarios are prevalent and usually harder to solve. In this paper, the main discussion is about the cooperative heterogeneous MARL problem in asymmetric heterogeneous maps of the Starcraft Multi-Agent Challenges (SMAC) environment. Recent mainstream approaches use policy-based actor-critic algorithms to solve the heterogeneous MARL problem with various individual agent policies. However, these approaches lack formal definition and further analysis of the heterogeneity problem. Therefore, a formal definition of the Local Transition Heterogeneity (LTH) problem is first given. Then, the LTH problem in SMAC environment can be studied. To comprehensively reveal and study the LTH problem, some new asymmetric heterogeneous maps in SMAC are designed. It has been observed that baseline algorithms fail to perform well in the new maps. Then, the authors propose the Grouped Individual-Global-Max (GIGM) consistency and a novel MARL algorithm, Grouped Hybrid Q-Learning (GHQ). GHQ separates agents into several groups and keeps individual parameters for each group. To enhance cooperation between groups, GHQ maximizes the mutual information between trajectories of different groups. A novel hybrid structure for value factorization in GHQ is also proposed. Finally, experiments on the original and the new maps show the fabulous performance of GHQ compared to other state-of-the-art algorithms.

Cooperative multi-agent reinforcement learning (MARL) has emerged as a powerful paradigm for addressing complex real-world challenges, including autonomous robot control, strategic decision-making, and decentralized coordination in unmanned swarm systems. However, it still faces challenges in learning proper coordination among multiple agents. The lack of effective knowledge sharing and experience interaction mechanisms among agents has led to substantial performance decline, especially in terms of low sampling efficiency and slow convergence rates, ultimately constraining the practical applicability of MARL. To address these challenges, this paper proposes a novel framework termed Reward redistribution and Experience reutilization based Coordination Optimization (RECO). This innovative approach employs a hierarchical experience pool mechanism that enhances exploration through strategic reward redistribution and experience reutilization. The RECO framework incorporates a sophisticated evaluation mechanism that assesses the quality of historical sampling data from individual agents and optimizes reward distribution by maximizing mutual information across hierarchical experience trajectories. Extensive comparative analyses of computational efficiency and performance metrics across diverse environments reveal that the proposed method not only enhances training efficiency in multi-agent gaming scenarios but also significantly strengthens algorithmic robustness and stability in dynamic environments.

Flexible manipulators based on soft robotic technologies demonstrate compliance and dexterous maneuverability with virtually infinite degrees-of-freedom. Such systems have great potential in assistive and surgical fields where safe human-robot interaction is a prime concern. However, in order to enable practical application in these environments, intelligent control frameworks are required that can automate low-level sensorimotor skills to reach targets with high precision. We designed a novel motor learning algorithm based on cooperative Multi-Agent Reinforcement Learning that enables high-dimensional manipulators to exploit an abstracted state-space through a reward-guided mechanism to find solutions that have a guaranteed precision. We test our algorithm on a simulated planar 6-DOF with a discrete action-set and show that the all the points reached by the manipulator average an accuracy of 0.0056m (±0.002). The algorithm was found to be repeatable. We further validated our concept on the Baxter robotic arm to generate solutions up to 0.008m, exceptions being the joint angle accuracy and calibration of the robot.

Reinforcement learning has been an important technique for adaptive decision-making of multi-agent systems in uncertain environments. However, the curse of dimensionality in multi-agent reinforcement learning usually causes the slow learning convergence or even failure. In this paper, a novel Online Heuristics Extraction method, which can integrate the prior heuristic policy with a learned heuristic policy, is presented. The new method can be incorporated into a tabular or approximate cooperative multi-agent reinforcement learning algorithm so as to speed up the learning process. Simulation results on a cooperative learning task show that, with the new method, a much better learning convergence can be achieved.

Parameter sharing is a common strategy in multi-agent reinforcement learning (MARL) to make the training more efficient and scalable. However, applying parameter sharing among agents indiscriminately hinders the emergence of agents diversity and degrades the final cooperative performance. To better balance parameter sharing and agents diversity, we propose a novel Agent-Driven Modular Network (ADMN), where agents share a base network consisting of multiple specialized modules, and each agent has its own routing to connect these modules. In ADMN, modules are shared among agents to improve the training efficiency, while the combination of different modules brings rich diversity. The agent routing at different time steps is learned end-to-end to achieve a dynamic and adaptive balance. Specifically, we also propose an information-theoretical regularization between the routing of agents and their behavior to further guarantee the identifiability of different routing. We evaluated ADMN in challenging StarCraft micromanagement games and Google Research Football games, and results demonstrate the superior performance of ADMN, particularly in larger or heterogeneous cooperative tasks.