Multi-agent Path Planning for Logistics Cargo Environment Using LSTM Based Reinforcement Learning

Multi-agent path finding in dynamic environments is of great academic and practical value for multi-robot systems in the real world. To improve the effectiveness and efficiency of the learning process during path planning in dynamic environments, we introduce an algorithm called Attention and BicNet based Multi-agent path planning with effective reinforcement (AB-Mapper) under the actor-critic reinforcement learning framework. In this framework, on one hand, we design an actor-network that can utilize the BicNet with communication function to achieve the intra-team coordination. On the other hand, we propose a critic network that can selectively allocate attention weights to surrounding agents. This attention mechanism allows an individual agent to automatically learn a better evaluation of actions by considering the behaviours of its surrounding agents. Compared with the SOTA method Mapper in crowded environments with dynamic obstacles, our AB-Mapper is more effective (90.27±0.06% vs. 61.65±13.90% in terms of mean success rate) in solving the general multi-agent path finding problem.

Multi-agent path finding (MAPF) has been widely used for many real-world problems, such as warehouse automation. Reinforcement learning (RL) has been employed to develop partially observable distributed MAPF algorithms that can be scaled to any number of agents or map sizes. However, the RL-based MAPF policy may cause oscillation behavior, deadlock and detours, which can lead to the failure of MAPF. This paper introduces R<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula>, a new hybrid policy framework for MAPF, in which the RL-based policy is incorporated with the random walk policy and the reverse resumable A<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{*}$</tex-math></inline-formula>-based policy. When an agent exhibits oscillation behavior or falls into a deadlock, the random walk policy helps the agent escape the trouble. When no other agents are within the agent's field of vision, we utilize the reverse resumable A<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{*}$</tex-math></inline-formula>-based policy to assist the agent in avoiding any potential detours. This paper uses the state-of-the-art RL-based MAPF algorithm, Decision Causal Communication, to do the simulation experiments on eight benchmark grid maps. The results show that R<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula> can significantly improve the performance of the baseline Decision Causal Communication algorithm.

Deep reinforcement learning-based Multi-Agent Path Finding (MAPF) has gained significant attention due to its remarkable adaptability to environments. Existing methods primarily leverage multi-agent communication in a fully-decentralized framework to maintain scalability while enhancing information exchange among agents. However, as the number of agents and obstacles increases, the environment becomes more complex, making cooperation between agents becomes more difficult, and crowding occurs from time to time. To address these issues, we propose a decentralized planner C3PIL, which integrates a Controlled Communication mechanism for Crowd Perception and uses Imitation Learning to improve policy learning. C3PIL first introduces a crowd perception communication module that perceives environmental crowd information and incorporates it into the controlled communication. This effectively prevents and mitigates crowded situations. Furthermore, we employ generative adversarial imitation learning to learn a reward function from expert experiences. It reduces the possible misleading caused by the fixed reward function, improves the flexibility and diversity of agent behaviors, and ultimately enables agents to cooperate effectively. Finally, experimental results show that C3PIL not only outperforms previous learning-based MAPF methods, but also further enhances the cooperation of agents and significantly reduces crowding in complex environments.

The multi-agent path finding (MAPF) problem is crucial to improve the efficiency of warehouse systems. Compared with traditional centralized methods, which encounter escalating computational complexities with increasing scale, reinforcement learning-based methods has been proven to be an effective method for solving MAPF problem. Nevertheless, in the complex and large-scale scenarios, the policies learned by existing reinforcement learning-based methods are generally inadequate to address the challenges effectively. By leveraging the concepts of policy evaluation and policy evolution, this paper aims to improve performance and sample efficiency. Consequently, we introduce an MAPF method based on evolutionary reinforcement learning. In particular, we design a collaborative policy network model based on reinforcement learning. Thereafter, a novel evolutionary reinforcement learning training framework is constructed. Through the quantitative evaluation mechanism, policy evaluation is carried out, and evolutionary algorithm is used for policy evolution, so that the collaborative policy could better guide the agent to complete the path finding task. We test on high-density warehouse environment instances of various map sizes, and the experimental results show that our method has high success rate and low average steps.

The multi-agent path planning problem is usually addressed via the use of reinforcement learning methods. In this article, a new Lludo game path planning algorithm based on Q-learning was proposed, and a set of cooperation proprieties that the agents have to follow in order to reach the goal location. To accomplish that a Lludo game environment modeling is introduced according to the Markov decision process (MDP) principles. The main objective of this work is to increase agents’ cooperation rate with the aim of decreasing the execution time of the assigned tasks. To demonstrate the improvement produced by this proposal, the path planning algorithm is applied in comparison with a greedy strategy also based on Q-learning. During this comparison, the execution time as well as the agents’ reward acquisition were examined in different cases. The simulation results reflect the advantages provided by this new algorithm in comparison with related work.

With the development of Unmanned Aerial Vehicles (UAVs) in recent years, UAV-mounted Mobile Edge Computing (MEC) systems are widely used for broadband connectivity and emergency communications in areas without internet. However, the onboard energy of UAVs is limited, and how to provide long-term computing services for ground User Equipments (UEs) still faces significant challenges. In this paper, UAVs are used as mobile base stations to offer MEC services for UEs. Unlike existing solutions that mainly solve UAV path planning from convex optimization methods, we propose a Multi-Agent Path planning (MAP) scheme based on deep reinforcement learning. We aim to minimize the total energy consumption of UAVs while completing the offloading tasks of UEs. At the same time, fairness is taken into account to ensure UE offloading balance and UAV load balance. To this end, we establish the optimization problem and design the state space, action space, and reward function for modeling each UAV through Deep Neural Networks (DNNs). Extensive simulation experiments show that the proposed scheme outperforms other benchmark schemes.

Multi-agent path finding (MAPF), in multi-agent systems, is a challenging and meaningful problem, in which all agents are required to effectively reach their goals concurrently with not colliding with each other and avoiding the obstacles. Effective extraction from the agent’s observation, effective utilization of historical information, and efficient communication with neighbor agents are the challenges to completing the cooperative task. To tackle these issues, in this paper, we propose a well-designed model, which utilizes the local states of nearby agents and obstacles and outputs an optimal action for each agent to execute. Our approach has three major components: 1) observation encoder which uses CNN to extract local partial observation and GRU to make full use of historical information, 2) communication block which uses attention mechanism to combine the agent’s partial observation with its neighbors, and 3) decision block with the purpose to output the final action policy. Based on the three major components, all agents formulate their own decentralized policies to apply. Finally, we use success rate and extra time rate to measure our approach and other well-known algorithms. The results show that our method outperforms the baselines, demonstrating the efficiency and effectiveness of our approach, especially in the case of large scale in the world.KeywordsMulti-agent path finding (MAPF)Reinforcement learningDecentralized planningDeep learning

Multiagent path finding (MAPF) is employed to find collision-free paths to guide agents traveling from an initial to a target position. The advanced decentralized approach utilizes communication between agents to improve their performance in environments with high-density obstacles. However, it dramatically reduces the robustness of multiagent systems. To overcome this difficulty, we propose a novel method for solving MAPF problems. In this method, expert data are transformed into supervised signals by proposing a hot supervised contrastive loss, which is combined with reinforcement learning to teach fully-decentralized policies. Agents reactively plan paths online in a partially observable world while exhibiting implicit coordination without communication with others. We introduce the self-attention mechanism in the policy network, which improves the ability of the policy network to extract collaborative information between agents from the observation data. By designing simulation experiments, we demonstrate that the learned policy achieved good performance without communication between agents. Furthermore, real-world application experiments demonstrate the effectiveness of our method in practical applications.