HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

Sequential decision-making over long timescales and in complex task environments is an important problem in Artificial Intelligence (AI). An effective approach to tackle this problem is to autonomously decompose a long-horizon task into a sequence of simpler subtasks or subgoals. We refer to this approach as Autonomous Task Decomposition (ATD) in the thesis and study it for multi-agent coordination using model-free Hierarchical Reinforcement Learning (HRL), single-agent goal-reaching using model-free HRL, and single-agent goal-reaching using model-based planning. The objective of the thesis is to develop novel methods to address three important challenges related to ATD, which are as follows: 1. Effective multi-agent HRL under sparse global rewards and complex inter-dependencies among agents. 2. Efficient unification of autonomous subgoal discovery and single-agent HRL without slow learning. 3. Learning models for planning-based ATD that produce more rewarding and feasible plans. In this regard, the thesis introduces three novel ATD methods as follows: 1. Inter Subtask Empowerment based Multi-agent Options (ISEMO) is introduced for effective multi-agent HRL by using auxiliary rewards that capture the inter-dependencies among HRL agents and their (handcrafted) subtasks. ISEMO leads to better coordinated performance of the inter-dependent agents on a complex Search & Rescue task, compared to a standard multi-agent HRL method. 2. End-to-End Hierarchical Reinforcement Learning with Integrated Discovery of Salient Subgoals (LIDOSS) is introduced for efficient unification of subgoal discovery and HRL for single-agent goal-reaching, by using a probability-based subgoal discovery heuristic integrated with the subgoal selection policy. LIDOSS accelerates end-to-end learning and leads to higher goal-reaching success rates compared to a state-of-the-art HRL method. 3. Finally, Learning Subgoal Graph using Value-based Subgoal Discovery and Automatic Pruning (LSGVP) is introduced to learn subgoal graph-based planning models that produce more rewarding and feasible plans for single-agent goal-reaching. LSGVP uses cumulative reward-based subgoal discovery and automatic pruning of erroneous connections in the subgoal graph. It achieves higher positive cumulative rewards and higher success rates compared to other state-of-the-art subgoal graph-based planning methods, while also being more data-efficient than model-free HRL.

Hierarchical reinforcement learning methods have not been able to simultaneously abstract and reuse subtasks with discounted value functions. The contribution of this paper is to introduce two completion functions that jointly decompose the value function hierarchically to solve this problem. The significance of this result is that the benefits of hierarchical reinforcement learning can be extended to discounted value functions and to continuing (infinite horizon) reinforcement learning problems. This paper demonstrates the method with the an algorithm that discovers subtasks automatically. An example is given where the optimum policy requires a subtask never to terminate.

Humanoid robots, featuring human-like appearances and learning capabilities, can be seamlessly integrated into various human-robot interaction scenarios. This work presents a lightweight pneumatic musculoskeletal robotic arm featuring a seven-degrees-of-freedom joint arrangement mimicking that of a human arm. Its compact joint design and antagonistic muscle configuration simplify control complexity while maintaining a human-like appearance and movement capability. To enable efficient learning of manipulation tasks, Musculoskeletal Hierarchical Reinforcement Learning (MuHRL) is introduced. MuHRL is designed based on the mechanical structure of the proposed robotic arm and employs a two-level hierarchical policy that decouples top-level task planning from bottom-level joint angle control. The top level incorporates an incremental target posture to balance response speed and precise control, optimizing task performance, while the bottom level adopts a reusable multi-network angle control policy to improve angle control precision. Simulation experiments demonstrate that MuHRL exhibits superior sample efficiency and task performance than standard reinforcement learning and hierarchical reinforcement learning methods across three benchmark tasks. A force control experiment conducted on the hardware platform further validates its effectiveness on the musculoskeletal robot.

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement–</i> Significant performance milestones in reinforcement learning have been achieved in recent years, with autonomous agents demonstrating super-human performance across a wide variety of tasks. Before these algorithms can be extensively deployed in real-world defense applications, a greater level of trust must first be achieved. ADT was an important step towards developing the trust necessary to operationalize these algorithms, by demonstrating their effectiveness on a foundational yet relevant problem in a high-fidelity simulation environment. Developed for the program, our hierarchical reinforcement learning agent was designed alongside of and competed against active fighter pilots, and ultimately defeated a graduate of the United States Air Force's F-16 Weapons Instructor Course in match play.

Deep reinforcement learning (DRL) has become a popular learning paradigm for decision and control and has been widely applied in robot manipulation in recent years. However, due to its special learning pattern of “trial and error”, there are still some remaining problems with DRL. Such as exploration dilemma, sample inefficient, and slow convergence, are to be refined, especially when faced with complex long-horizon tasks. As a solution for these limits, hierarchical reinforcement learning (HRL) is proposed and developed by decomposing challenging tasks into multiple simpler subtasks, to efficiently solve the main task in a“divide and conquer” manner. At present, there are comprehensive HRL methods for robotic manipulation tasks, while a review is lacking. To facilitate researchers to form a general view of this field, we systematically summarize related HRL methods for robotic manipulation. This review carries out literature sortation in a novel taxonomy of subtask generation and divides HRL methods into two categories: handcrafted subtask generation and learning-based subtask generation. A great number of representative methods are analyzed in detail. In the end, we also present some important future directions for HRL.

Hierarchical reinforcement learning has achieved good results in solving complex learning tasks. Hierarchical reinforcement learning mainly includes on-policy and off-policy methods. On-policy cannot be applied to actual scenarios due to low data utilization. Therefore, off-policy reinforcement learning methods have become the main development direction. However, in the off-policy method, because the data in the replay buffer comes from different policy, the upper-layer policy gives the same goal, and the lower-layer policy is constantly updated and shifts to a different state, so the upper-layer policy cannot be stably trained. Aiming at the above problems, we propose a hierarchical reinforcement learning algorithm for environment representation based on the mutual bisimulation Metrics. When training the upper policy, the lower policy is used as a part of the environment, which is called virtual representation environment. The output is used for feature extraction, and the feature value is used as the state value of the upper-level policy net. Using our proposed method to compare a variety of complex tasks with the current main hierarchical reinforcement learning has achieved the stability and training effect improvement.

A method of network attack-defense game and collaborative defense decision-making based on hierarchical multi-agent reinforcement learning

Multi-agent systems often face challenges such as elevated communication demands, intricate interactions, and difficulties in transferability. To address the issues of complex information interaction and model scalability, we propose an innovative hierarchical graph attention actor-critic reinforcement learning method. This method naturally models the interactions within a multi-agent system as a graph, employing hierarchical graph attention to capture the complex cooperative and competitive relationships among agents, thereby enhancing their adaptability to dynamic environments. Specifically, graph neural networks encode agent observations as single feature-embedding vectors, maintaining a constant dimensionality irrespective of the number of agents, which improves model scalability. Through the "inter-agent" and "inter-group" attention layers, the embedding vector of each agent is updated into an information-condensed and contextualized state representation, which extracts state-dependent relationships between agents and model interactions at both individual and group levels. We conducted experiments across several multi-agent tasks to assess our proposed method's effectiveness, stability, and scalability. Furthermore, to enhance the applicability of our method in large-scale tasks, we tested and validated its performance within a curriculum learning training framework, thereby enhancing its transferability.

Vehicle routing problem with time windows (VRPTW) is a practical and complex vehicle routing problem (VRP) which is faced by thousands of companies in logistics and transportation. Usually, VRP is solved by traditional heuristic algorithms. Recently, deep learning models under the reinforcement learning (RL) framework have been proposed to solve variants of VRP. In our study, we propose to use the hierarchical RL to find an optimal policy for generating optimal routes in VRPTW. The hierarchical RL structure includes a low level which generates feasible solutions and a high level which further searches for an optimal solution. Experimental results show that the proposed hierarchical RL model outperforms the non-hierarchical RL model and the heuristic algorithms Google OR-Tools. The proposed model also shows generalization capability in three different scenarios: varied time window constraints, from small-scale to large-scale problems, and generalization across different datasets. The flexible framework of hierarchical RL can also be applied to solve other complex VRPs with multiple objectives.

42nd IEEE International Conference on Distributed Computing Systems, July 10 - July 13, 2022, Bologna, Italy

The paradigm shift toward 6G-enabled immersive services, including extended reality (XR) communication, holographic interaction, and distributed multisensor intelligence, imposes unprecedented challenges to vehicular networks. These mission-critical applications require zero-interruption continuity, deterministic ultra-low latency, and sustained high-throughput task execution in highly dynamic vehicular environments. To achieve stability, adaptability, and efficiency in 6G-vehicular edge computing (VEC), this paper proposes a new three-layer hierarchical deep reinforcement learning scheduling (H <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula> DRA) framework, which involves an interruption prediction-empowered adaptive task allocation (IPred-ATA) technique that dynamically optimizes scheduling destinations through proactive failure prediction and multi-objective utility maximization. We further design a clustering-assisted hierarchical reinforcement learning (CAHRL) algorithm that addresses the challenges of multi-hop path selection within dynamic network topologies by decomposing the large-scale, complex state-action space into manageable hierarchical structures. Simulations demonstrate that the proposed H <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula> DRA outperforms classical algorithms by 48% in stability and by 32% in task throughput.

Deep reinforcement learning (DRL) has been extensively used to address portfolio optimization problems. DRL agents acquire knowledge and make decisions through unsupervised interactions with their environment without requiring explicit knowledge of the joint dynamics of portfolio assets. Among these DRL algorithms, the combination of actor-critic algorithms and deep function approximators is the most widely used DRL algorithm. Here, we find that training the DRL agent using the actor-critic algorithm and deep function approximators may lead to scenarios where the improvement in the DRL agent's risk-adjusted profitability is insignificant. We argue that such situations primarily arise from the following two problems: sparsity in positive reward and the curse of dimensionality. These limitations prevent DRL agents from comprehensively learning asset price change patterns in the training environment. As a result, the DRL agents cannot effectively explore the dynamic portfolio optimization policy to improve the risk-adjusted profitability in the training process. To address these problems, we propose a novel multi-agent learning system based on the hierarchical deep reinforcement learning (HDRL) algorithmic framework in this research. Under this framework, the agents work together as a learning system for portfolio optimization. Specifically, by designing an auxiliary agent that works together with the executive agent for optimal policy exploration, the learning system can focus on exploring the policy with higher risk-adjusted return in the action space with positive return and low variance. The performance of the proposed learning system is evaluated using a portfolio of 29 stocks from the Dow Jones index in four different experiments. In the training process, the objective functions of the actor and critic both ultimately achieve stable convergence in the training process. The risk-adjusted profitability of our learning system in the training environment is significantly improved. Hence, we prove that the policies executed by our learning system in out-sample experiments originate from the DRL agents' comprehensive learning of asset price change patterns in the training environment. Furthermore, we find that adopting the auxiliary agent and HDRL training algorithm can efficiently overcome the issue of the curse of dimensionality and improve the training efficiency in the positive reward sparse environment. In each back-test experiment, the proposed learning system is compared to sixteen traditional strategies and ten strategies based on machine learning algorithms in the performance of profitability and risk control ability. The empirical results in the four evaluation experiments demonstrate the efficacy of our learning system, which outperforms all other strategies by at least 8.2% in terms of Sharpe ratio, Sorino ratio, and Calmar ratio. This indicates that the policies learned in the training environment can exhibit excellent generalization ability in the back-testing experiments.

The growing integration of distributed energy resources and advancements in communication and information technology have necessitated the development of smart grids with advanced demand response capabilities. This paper proposes a hierarchical reinforcement learning framework integrating intelligent home energy management and a peer-to-peer energy trading community. The low-level policy optimizes home energy management by leveraging a fuzzy actor-critic reinforcement learning algorithm with a centralized-training-decentralized-execution structure. It regulates energy consumption based on Time-of-Use tariffs while considering user dissatisfaction levels. Then, the low-level policy communicates surplus or deficit energy to the high-level policy. The high-level policy employs the fuzzy actor-critic reinforcement learning algorithm under the decentralized-training-decentralized-execution structure, with value decomposition networks to generate a privacy-preserving cooperative-competitive strategy for pricing in a dynamic continuous double auction market. While optimizing individual agent benefits, the high-level policy also fosters cooperation to enhance energy trade within the community. The simulations using the real-world data demonstrate the effectiveness of the low-level policy in managing the energy consumption and determining the required/excessive energy in the market. The comparisons with the purely cooperative and purely competitive markets represent the superiority of the proposed approach in terms of increased load transactions, buyers' cost savings, and sellers' revenue.

‎This thesis focuses on modelling deception in competitive differential games‎. ‎By definition‎, ‎differential games are games where two agents (or two groups of agents) have conflicting objectives‎. ‎In other words‎, ‎maximizing an agent's (or a group of agents') objective will minimize the other agent's (or a group of agents') pay-off‎. ‎To increase the pay-off of one group‎, ‎we implement deception in a differential game‎. ‎On the other hand‎, ‎the other group learns counter-deception to cope with the adversarials' strategy‎. ‎To address deception and counter-deception‎, ‎we implement the game of guarding territory and its derivatives‎. ‎Our primary attempt consists of modelling deception using a hierarchical structure for pursuit-evasion games‎. ‎The deception structure is made of two levels‎: ‎a lower-level policy‏ ‎(LLP)‎ that contains the optimal state-action values to reach a goal and a higher-level policy (HLP) that can change the lower-level policy's goal‎. ‎While the players try to deceive their opponents using deception‎, ‎the opponents try to detect deception and overcome the deceitful players‎. ‎Reinforcement learning is the primary learning strategy in our study‎. We address the assignment problem in pursuit-evasion games‎, ‎where multiple agents exist on each side‎. ‎The assignment problem is solved by proposing a two-level policy system‎, ‎where the pursuer's policy is calculated via a fuzzy actor-critic learning (FACL) algorithm‎. ‎Our study is the first study on pursuit-evasion games where the assignment can be changed in the middle of the simulation‎. In addition‎, ‎we proposed a hierarchical reinforcement learning scheme to provide broader control over the reward function hyperparameters. In other words, an HLP learns reward functions' hyperparameters.

Electric vehicle (EV) is emerging as an effective choice to reduce carbon emissions in the modern transportation system. However, the large access of EVs brings tremendous passive influence on the stability and performance of power grid operation. Thus, it is valuable to investigate the charging power management method to improve charging experience of EV drivers while maintaining normal bus voltage of the power system. This paper builds the charging power control problem of multiple EVs as a hierarchical Markov decision process (MDP) model and propose a hierarchical multi-objective reinforcement learning method (Hamlet) to obtain the real-time charging power control decisions. Specifically, by treating EVs with the same remaining charging time as the same agent, this paper addresses the dynamic state and action spaces issue and significantly reduce the input dimension of both state and action spaces. Besides, this paper quantifies drivers' anxiety about the battery volume and distributes the comfort reward over the entire charging interval, which overcomes the delayed comfort reward issue. In order to alleviate the influence of multiple objectives on training stability, this paper develops a multi-objective learning method to dynamically adjust the optimization direction. Simulation results on IEEE 33 bus and 69 bus test feeders prove the validity of the proposed method in mitigating voltage fluctuation, minimizing bill payment and maximizing battery comfort.