Multi-Agent Hierarchical Graph Attention Actor-Critic Reinforcement Learning.

Recently, some challenging tasks in multi-agent systems have been solved by some hierarchical reinforcement learning methods. Inspired by the intra-level and inter-level coordination in the human nervous system, we propose a novel value decomposition framework HAVEN based on hierarchical reinforcement learning for fully cooperative multi-agent problems. To address the instability arising from the concurrent optimization of policies between various levels and agents, we introduce the dual coordination mechanism of inter-level and inter-agent strategies by designing reward functions in a two-level hierarchy. HAVEN does not require domain knowledge and pre-training, and can be applied to any value decomposition variant. Our method achieves desirable results on different decentralized partially observable Markov decision process domains and outperforms other popular multi-agent hierarchical reinforcement learning algorithms.

Autonomous control in high-dimensional, continuous state spaces is a persistent and important challenge in the fields of robotics and artificial intelligence. Because of high risk and complexity, the adoption of AI for autonomous combat systems has been a long-standing difficulty. In order to address these issues, DARPA's AlphaDogfight Trials (ADT) program sought to vet the feasibility of and increase trust in AI for autonomously piloting an F-16 in simulated air-to-air combat. Our submission to ADT solves the high-dimensional, continuous control problem using a novel hierarchical deep reinforcement learning approach consisting of a high-level policy selector and a set of separately trained low-level policies specialized for excelling in specific regions of the state space. Both levels of the hierarchy are trained using off-policy, maximum entropy methods with expert knowledge integrated through reward shaping. Our approach outperformed human expert pilots and achieved a second-place rank in the ADT championship event. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement–</i> Significant performance milestones in reinforcement learning have been achieved in recent years, with autonomous agents demonstrating super-human performance across a wide variety of tasks. Before these algorithms can be extensively deployed in real-world defense applications, a greater level of trust must first be achieved. ADT was an important step towards developing the trust necessary to operationalize these algorithms, by demonstrating their effectiveness on a foundational yet relevant problem in a high-fidelity simulation environment. Developed for the program, our hierarchical reinforcement learning agent was designed alongside of and competed against active fighter pilots, and ultimately defeated a graduate of the United States Air Force's F-16 Weapons Instructor Course in match play.

Deep reinforcement learning (DRL) has been extensively used to address portfolio optimization problems. DRL agents acquire knowledge and make decisions through unsupervised interactions with their environment without requiring explicit knowledge of the joint dynamics of portfolio assets. Among these DRL algorithms, the combination of actor-critic algorithms and deep function approximators is the most widely used DRL algorithm. Here, we find that training the DRL agent using the actor-critic algorithm and deep function approximators may lead to scenarios where the improvement in the DRL agent's risk-adjusted profitability is insignificant. We argue that such situations primarily arise from the following two problems: sparsity in positive reward and the curse of dimensionality. These limitations prevent DRL agents from comprehensively learning asset price change patterns in the training environment. As a result, the DRL agents cannot effectively explore the dynamic portfolio optimization policy to improve the risk-adjusted profitability in the training process. To address these problems, we propose a novel multi-agent learning system based on the hierarchical deep reinforcement learning (HDRL) algorithmic framework in this research. Under this framework, the agents work together as a learning system for portfolio optimization. Specifically, by designing an auxiliary agent that works together with the executive agent for optimal policy exploration, the learning system can focus on exploring the policy with higher risk-adjusted return in the action space with positive return and low variance. The performance of the proposed learning system is evaluated using a portfolio of 29 stocks from the Dow Jones index in four different experiments. In the training process, the objective functions of the actor and critic both ultimately achieve stable convergence in the training process. The risk-adjusted profitability of our learning system in the training environment is significantly improved. Hence, we prove that the policies executed by our learning system in out-sample experiments originate from the DRL agents' comprehensive learning of asset price change patterns in the training environment. Furthermore, we find that adopting the auxiliary agent and HDRL training algorithm can efficiently overcome the issue of the curse of dimensionality and improve the training efficiency in the positive reward sparse environment. In each back-test experiment, the proposed learning system is compared to sixteen traditional strategies and ten strategies based on machine learning algorithms in the performance of profitability and risk control ability. The empirical results in the four evaluation experiments demonstrate the efficacy of our learning system, which outperforms all other strategies by at least 8.2% in terms of Sharpe ratio, Sorino ratio, and Calmar ratio. This indicates that the policies learned in the training environment can exhibit excellent generalization ability in the back-testing experiments.

A method of network attack-defense game and collaborative defense decision-making based on hierarchical multi-agent reinforcement learning

Sequential decision-making over long timescales and in complex task environments is an important problem in Artificial Intelligence (AI). An effective approach to tackle this problem is to autonomously decompose a long-horizon task into a sequence of simpler subtasks or subgoals. We refer to this approach as Autonomous Task Decomposition (ATD) in the thesis and study it for multi-agent coordination using model-free Hierarchical Reinforcement Learning (HRL), single-agent goal-reaching using model-free HRL, and single-agent goal-reaching using model-based planning. The objective of the thesis is to develop novel methods to address three important challenges related to ATD, which are as follows: 1. Effective multi-agent HRL under sparse global rewards and complex inter-dependencies among agents. 2. Efficient unification of autonomous subgoal discovery and single-agent HRL without slow learning. 3. Learning models for planning-based ATD that produce more rewarding and feasible plans. In this regard, the thesis introduces three novel ATD methods as follows: 1. Inter Subtask Empowerment based Multi-agent Options (ISEMO) is introduced for effective multi-agent HRL by using auxiliary rewards that capture the inter-dependencies among HRL agents and their (handcrafted) subtasks. ISEMO leads to better coordinated performance of the inter-dependent agents on a complex Search & Rescue task, compared to a standard multi-agent HRL method. 2. End-to-End Hierarchical Reinforcement Learning with Integrated Discovery of Salient Subgoals (LIDOSS) is introduced for efficient unification of subgoal discovery and HRL for single-agent goal-reaching, by using a probability-based subgoal discovery heuristic integrated with the subgoal selection policy. LIDOSS accelerates end-to-end learning and leads to higher goal-reaching success rates compared to a state-of-the-art HRL method. 3. Finally, Learning Subgoal Graph using Value-based Subgoal Discovery and Automatic Pruning (LSGVP) is introduced to learn subgoal graph-based planning models that produce more rewarding and feasible plans for single-agent goal-reaching. LSGVP uses cumulative reward-based subgoal discovery and automatic pruning of erroneous connections in the subgoal graph. It achieves higher positive cumulative rewards and higher success rates compared to other state-of-the-art subgoal graph-based planning methods, while also being more data-efficient than model-free HRL.

Hierarchical reinforcement learning has achieved good results in solving complex learning tasks. Hierarchical reinforcement learning mainly includes on-policy and off-policy methods. On-policy cannot be applied to actual scenarios due to low data utilization. Therefore, off-policy reinforcement learning methods have become the main development direction. However, in the off-policy method, because the data in the replay buffer comes from different policy, the upper-layer policy gives the same goal, and the lower-layer policy is constantly updated and shifts to a different state, so the upper-layer policy cannot be stably trained. Aiming at the above problems, we propose a hierarchical reinforcement learning algorithm for environment representation based on the mutual bisimulation Metrics. When training the upper policy, the lower policy is used as a part of the environment, which is called virtual representation environment. The output is used for feature extraction, and the feature value is used as the state value of the upper-level policy net. Using our proposed method to compare a variety of complex tasks with the current main hierarchical reinforcement learning has achieved the stability and training effect improvement.

Electric vehicle (EV) is emerging as an effective choice to reduce carbon emissions in the modern transportation system. However, the large access of EVs brings tremendous passive influence on the stability and performance of power grid operation. Thus, it is valuable to investigate the charging power management method to improve charging experience of EV drivers while maintaining normal bus voltage of the power system. This paper builds the charging power control problem of multiple EVs as a hierarchical Markov decision process (MDP) model and propose a hierarchical multi-objective reinforcement learning method (Hamlet) to obtain the real-time charging power control decisions. Specifically, by treating EVs with the same remaining charging time as the same agent, this paper addresses the dynamic state and action spaces issue and significantly reduce the input dimension of both state and action spaces. Besides, this paper quantifies drivers' anxiety about the battery volume and distributes the comfort reward over the entire charging interval, which overcomes the delayed comfort reward issue. In order to alleviate the influence of multiple objectives on training stability, this paper develops a multi-objective learning method to dynamically adjust the optimization direction. Simulation results on IEEE 33 bus and 69 bus test feeders prove the validity of the proposed method in mitigating voltage fluctuation, minimizing bill payment and maximizing battery comfort.

Dynamic spectrum access (DSA) has been regarded as a viable solution to the spectrum shortage problem. To find idle spectrum, partial spectrum sensing could be employed by selecting a suitable sensing window (SW). Since the SW selection determines how many available bands to access, the transmission performance after the access could be used to guide the SW selection. Hence, a sophisticated joint design on spectrum sensing and access is necessary, which, however, is a challenging task when considering the dynamic nature of spectrum environment, and also the mutual impact among different secondary users (SUs). In this paper, we propose a joint partial spectrum sensing and power allocation (PA) scheme to facilitate SUs to make the best decisions on SW and PA to maximize the network throughput with reduced mutual interference. Considering the environmental dynamics and spectrum uncertainty, we develop a viable solution based on hierarchical multi-agent deep reinforcement learning (HMADRL). Our solution enables mutual design with two stages: making each SU learn the best SW and PA strategies autonomously while adapting to the dynamic environment. By using both simulated spectrum data and real spectrum data measured by SAM60-BX, we have demonstrated the effectiveness of our proposed scheme.

This paper presents a hierarchical deep reinforcement learning (DRL) method for the scheduling of energy consumptions of smart home appliances and distributed energy resources (DERs) including an energy storage system (ESS) and an electric vehicle (EV). Compared to Q-learning algorithms based on a discrete action space, the novelty of the proposed approach is that the energy consumptions of home appliances and DERs are scheduled in a continuous action space using an actor–critic-based DRL method. To this end, a two-level DRL framework is proposed where home appliances are scheduled at the first level according to the consumer’s preferred appliance scheduling and comfort level, while the charging and discharging schedules of ESS and EV are calculated at the second level using the optimal solution from the first level along with the consumer environmental characteristics. A simulation study is performed in a single home with an air conditioner, a washing machine, a rooftop solar photovoltaic system, an ESS, and an EV under a time-of-use pricing. Numerical examples under different weather conditions, weekday/weekend, and driving patterns of the EV confirm the effectiveness of the proposed approach in terms of total cost of electricity, state of energy of the ESS and EV, and consumer preference.

We present a hierarchical deep reinforcement learning (DRL) framework with prominent sampling efficiency and sim-to-real transfer ability for fast and safe navigation: the low-level DRL policy enables the robot to move towards the target position and keep a safe distance to obstacles simultaneously; the high-level DRL policy is supplemented to further enhance the navigation safety. We select a waypoint located on the path from the robot to the ultimate goal as the sub-goal to reduce the state space and avoid sparse reward. Moreover, the path is generated based on either a local or a global map, which can significantly improve the sampling efficiency, safety, and generalization ability of the proposed DRL framework. Additionally, a target-directed representation for the action space can be derived based on the sub-goal to improve the motion efficiency and reduce the action space. In order to demonstrate the eminent sampling efficiency, motion performance, obstacle avoidance, and generalization ability of the proposed framework, we implement sufficient comparisons with the non-learning navigation methods and DRL-based baselines, with videos, data, code, and other supplemental material shown on our website<xref ref-type="fn" rid="fn1">¹</xref><fn id="fn1"><label>¹</label> https://github.com/zw199502/RL <inline-formula><tex-math notation="LaTeX">$\_$</tex-math></inline-formula> navigation </fn>.

Beam hopping (BH) is crucial for managing communication resources in large-scale multi-satellite, multi-beam Low Earth Orbit (LEO) constellation networks. However, achieving high-dynamic multi-satellite cooperative BH in LEO mega-constellations while satisfying multi-dimensional resource allocation demands remains a significant challenge. To address the interdependence of long-term coverage stability and short-term resource volatility, this paper proposes a distributed Beam Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework. Within this framework, a hierarchical Duplex Dueling Multi-agent Q-learning (QPLEX) architecture decouples the high-dimensional joint optimization problem into two tractable layers: satellite-level cell association and beam-level resource scheduling. The former handles coverage planning under load-balancing and interference constraints, while the latter manages dynamic resource allocation amid spatiotemporal traffic variations. Additionally, Chemical Reaction Optimization (CRO) and Cross-Layer Attention (CLA) mechanisms are introduced to address the discrete beam interference problem and the inter-layer information isolation challenge, respectively. Simulation results show that the proposed method, BH-HMARL, performs better than baseline methods. Further results confirm its scalability, with performance remaining stable despite variations in the number of satellites, beams, and cells. Moreover, BH-HMARL achieves real-time scheduling with execution time under 5ms per agent, satisfying slot duration constraints.

Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach

The growing integration of distributed energy resources and advancements in communication and information technology have necessitated the development of smart grids with advanced demand response capabilities. This paper proposes a hierarchical reinforcement learning framework integrating intelligent home energy management and a peer-to-peer energy trading community. The low-level policy optimizes home energy management by leveraging a fuzzy actor-critic reinforcement learning algorithm with a centralized-training-decentralized-execution structure. It regulates energy consumption based on Time-of-Use tariffs while considering user dissatisfaction levels. Then, the low-level policy communicates surplus or deficit energy to the high-level policy. The high-level policy employs the fuzzy actor-critic reinforcement learning algorithm under the decentralized-training-decentralized-execution structure, with value decomposition networks to generate a privacy-preserving cooperative-competitive strategy for pricing in a dynamic continuous double auction market. While optimizing individual agent benefits, the high-level policy also fosters cooperation to enhance energy trade within the community. The simulations using the real-world data demonstrate the effectiveness of the low-level policy in managing the energy consumption and determining the required/excessive energy in the market. The comparisons with the purely cooperative and purely competitive markets represent the superiority of the proposed approach in terms of increased load transactions, buyers' cost savings, and sellers' revenue.

‎This thesis focuses on modelling deception in competitive differential games‎. ‎By definition‎, ‎differential games are games where two agents (or two groups of agents) have conflicting objectives‎. ‎In other words‎, ‎maximizing an agent's (or a group of agents') objective will minimize the other agent's (or a group of agents') pay-off‎. ‎To increase the pay-off of one group‎, ‎we implement deception in a differential game‎. ‎On the other hand‎, ‎the other group learns counter-deception to cope with the adversarials' strategy‎. ‎To address deception and counter-deception‎, ‎we implement the game of guarding territory and its derivatives‎. ‎Our primary attempt consists of modelling deception using a hierarchical structure for pursuit-evasion games‎. ‎The deception structure is made of two levels‎: ‎a lower-level policy‏ ‎(LLP)‎ that contains the optimal state-action values to reach a goal and a higher-level policy (HLP) that can change the lower-level policy's goal‎. ‎While the players try to deceive their opponents using deception‎, ‎the opponents try to detect deception and overcome the deceitful players‎. ‎Reinforcement learning is the primary learning strategy in our study‎. We address the assignment problem in pursuit-evasion games‎, ‎where multiple agents exist on each side‎. ‎The assignment problem is solved by proposing a two-level policy system‎, ‎where the pursuer's policy is calculated via a fuzzy actor-critic learning (FACL) algorithm‎. ‎Our study is the first study on pursuit-evasion games where the assignment can be changed in the middle of the simulation‎. In addition‎, ‎we proposed a hierarchical reinforcement learning scheme to provide broader control over the reward function hyperparameters. In other words, an HLP learns reward functions' hyperparameters.

The paradigm shift toward 6G-enabled immersive services, including extended reality (XR) communication, holographic interaction, and distributed multisensor intelligence, imposes unprecedented challenges to vehicular networks. These mission-critical applications require zero-interruption continuity, deterministic ultra-low latency, and sustained high-throughput task execution in highly dynamic vehicular environments. To achieve stability, adaptability, and efficiency in 6G-vehicular edge computing (VEC), this paper proposes a new three-layer hierarchical deep reinforcement learning scheduling (H <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula> DRA) framework, which involves an interruption prediction-empowered adaptive task allocation (IPred-ATA) technique that dynamically optimizes scheduling destinations through proactive failure prediction and multi-objective utility maximization. We further design a clustering-assisted hierarchical reinforcement learning (CAHRL) algorithm that addresses the challenges of multi-hop path selection within dynamic network topologies by decomposing the large-scale, complex state-action space into manageable hierarchical structures. Simulations demonstrate that the proposed H <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula> DRA outperforms classical algorithms by 48% in stability and by 32% in task throughput.