Autonomous Network Defence Using Multi-Agent Reinforcement Learning and Self-Play

As cyber threats become increasingly automated and sophisticated, novel solutions must be introduced to improve defence of enterprise networks. Deep Reinforcement Learning (DRL) has demonstrated potential in mitigating these advanced threats. Single DRL Agents have proven utility toward execution of autonomous cyber defence. Despite the success of employing single DRL Agents, this approach presents significant limitations, especially regarding scalability within large enterprise networks. An attractive alternative to the single agent approach is the use of Multi-Agent Reinforcement Learning (MARL). However, developing MARL agents is costly with few options for examining MARL cyber defence techniques against adversarial agents. This paper presents a MARL network security environment, the fourth iteration of the Cyber Autonomy Gym for Experimentation (CAGE) challenges. This challenge was specifically designed to test the efficacy of MARL algorithms in an enterprise network. Our work aims to evaluate the potential of MARL as a robust and scalable solution for autonomous network defence.

As cyber threats become increasingly automated and sophisticated, novel solutions must be introduced to improve defense of enterprise networks. Deep reinforcement learning (DRL) has demonstrated potential in mitigating these advanced threats. Single DRL agents have proven utility toward execution of autonomous cyber defense. Despite the success of employing single DRL agents, this approach presents significant limitations, especially regarding scalability within large enterprise networks. An attractive alternative to the single‐agent approach is the use of multi‐agent reinforcement learning (MARL). However, developing MARL agents is costly with few options for examining MARL cyber defense techniques against adversarial agents. This paper presents a MARL network security environment, the fourth iteration of the cyber autonomy gym for experimentation (CAGE) challenges. This challenge was specifically designed to test the efficacy of MARL algorithms in an enterprise network. Our work aims to evaluate the potential of MARL as a robust and scalable solution for autonomous network defense.

Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, the centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action space. The multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now, the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Most existing studies in MARL focus on designing efficient communication and coordination among traditional Q-learning agents. This paper presents, for the first time, a fully scalable and decentralized MARL algorithm for the state-of-the-art deep RL agent, advantage actor critic (A2C), within the context of ATSC. In particular, two methods are proposed to stabilize the learning procedure, by improving the observability and reducing the learning difficulty of each local agent. The proposed multi-agent A2C is compared against independent A2C and independent Q-learning algorithms, in both a large synthetic traffic grid and a large real-world traffic network of Monaco city, under simulated peak-hour traffic dynamics. The results demonstrate its optimality, robustness, and sample efficiency over the other state-of-the-art decentralized MARL algorithms.

We initiate the study of how to perturb the reward in a zero-sum Markov game with two players to induce a desirable Nash equilibrium, namely arbitrating. Such a problem admits a bi-level optimization formulation. The lower level requires solving the Nash equilibrium under a given reward function, which makes the overall problem challenging to optimize in an end-to-end way. We propose a backpropagation scheme that differentiates through the Nash equilibrium, which provides the gradient feedback for the upper level. In particular, our method only requires a black-box solver for the (regularized) Nash equilibrium (NE). We develop the convergence analysis for the proposed framework with proper black-box NE solvers and demonstrate the empirical successes in two multi-agent reinforcement learning (MARL) environments. Supplementary for all the proofs in this paper could be found in: https://arxiv.org/abs/2302.10058.

Aim and ApproachLarge cooling water system has great potential of energy saving due to improper operation control. The coupled hydraulic and thermodynamic characteristics among chillers, pumps, and cooling towers lead to the difficulty of cooling water system operation. This study investigates an advanced control approach of multi-agent reinforcement learning (RL) for a large cooling water system. The multiple RL agents interact with each other to deal with the complex coupled characteristics in the cooling water system. The proposed approach is implemented in a real-world cooling water system and validated with experiments.Scientific Innovation and RelevanceIn this study, a multi-agent RL approach is proposed to separately control the chillers, pumps, and cooling towers in the cooling water system. The soft actor critic algorithm is used in the RL agent, as it is an efficient and advanced RL algorithm for continuous and discrete control problems. The agents are designed to have private states, private actions, and shared rewards. The reward functions are carefully designed to balance the agents for global optima.To provide a realistic environment for RL training, a detailed physical modeling platform of a cooling water system is established and validated. This platform is capable of hydraulic and thermodynamic calculation of cooling water systems with customized numbers and characteristics of chillers, pumps, and cooling towers, which is scalable for different cooling water systems.The proposed multi-agent RL controller is implemented in the building automation system of a real cooling water system. A one-month experimental study of the multi-agent RL performance is conducted. The proposed multi-agent RL controller is compared with a single-agent RL controller and a rule-based controller through real-world experiments to demonstrate the energy performance. This is a pilot study of multi-agent RL approach to real-world cooling water system control.Preliminary Results and ConclusionsThe preliminary simulation study is conducted in the established cooling water system model. The electricity consumption simulation error is 4.0% in coefficient of variation of mean absolute error (CVMAE) compared with the measured 15-minute electricity consumption. The 6-month simulation shows that the single-agent RL can save 6.2% energy from the rule-based control, while the multi-agent RL can save 7.2% electricity consumption of the cooling water system. In the following study, the multi-agent RL implementation will be practiced with more experiments and applications for cooling water system control.

Recently, the researches on multi-agent reinforcement learning (MARL) have attracted tremendous interest in many applications, especially for autonomous driving. The main problem of MARL is how to deal with the uncertainty in the environment and the interaction between the connected agents. To solve the problem, a distributed robust temporal differential deep Q-network algorithm (MARTD-DQN) was developed in this paper. MARTD-DQN consists of two parts, the decentralized MARL algorithm (DMARL) and the robust TD deep Q-network algorithm (RTD-DQN). DMARL improves the robustness of the policy estimation by fusing the states from the neighbors over communicated networks. RTD- DQN improves the robustness to outliers through on-line estimation of the uncertainty. By combining the two algorithms, the proposed algorithm can be robust not only to node failures but also to the outliers. Then the proposed algorithm is applied to ACC simulations of autonomous cars. The simulation results are given to show the efficiency of the proposed algorithm.

With the development of unmanned combat air vehicles (UCAVs) and artificial intelligence (AI), within visual range (WVR) air combat confrontations utilizing intelligent UCAVs are expected to be widely used in future air combats. As controlling highly dynamic and uncertain WVR air combats from the ground stations of the UCAV is not feasible, it is necessary to develop an algorithm that can generate highly intelligent air combat strategies in order to enable UCAV to independently complete air combat missions. In this paper, a 1-vs.-1 WVR air combat strategy generation algorithm is proposed using the multi-agent deep deterministic policy gradient (MADDPG). A 1-vs.-1 WVR air combat is modeled as a two-player zero-sum Markov game (ZSMG). A method for predicting the position of the target is introduced into the model in order to enable the UCAV to predict the target’s actions and position. Moreover, to ensure that the UCAV is not limited by the constraints of the basic fighter maneuver (BFM) library, the action space is considered to be a continuous one. At the same time, a potential-based reward shaping method is proposed in order to improve the efficiency of the air combat strategy generation algorithm. Finally, the efficiency of the air combat strategy generation algorithm and the intelligence level of the resulting strategy is verified through simulation experiments. The results show that an air combat strategy using target position prediction is superior to the one that does not use target position prediction.

Reinforcement learning (RL) is a promising technique for optimizing traffic signal controllers that dynamically respond to real-time traffic conditions. Recent efforts that applied Multi-Agent RL (MARL) to this problem have shown remarkable improvement over centralized RL, with the scalability to solve large problems by distributing the global control to local RL agents. Unfortunately, it is also easy to get stuck in local optima because each agent only has partial observability of the environment with limited communication. To tackle this, we borrow ideas from feudal RL and propose a novel MARL approach combining with the feudal hierarchy. Specifically, we split the traffic network into several regions, where each region is controlled by a manager agent and the agents who control the traffic signals are its workers. In our method, managers coordinate their high-level behaviors and set goals for their workers in the region, while each lower-level worker controls traffic signals to fulfill the managerial goals. By doing so, we are able to coordinate globally while retain scalability. We empirically evaluate our method both in a synthetic traffic grid and real-world traffic network using the SUMO simulator. Our experimental results demonstrate that our approach outperforms the state-of-the-art in almost all evaluation metrics commonly used for traffic signal control.

Reinforcement learning, a machine learning technique, has already shown advancement in solving complex sequential decision-making problems. Many tasks involve multiple agents and require sequential decision-making policies to achieve common goals, such as warehouse automation, autonomous driving, and game-playing. To derive the policies for all agents, these problems can be modelled as multi-agent systems and addressed by multi-agent reinforcement learning (MARL). However, optimizing policies in multi-agent scenarios presents significant challenges due to the intricate behaviours of multiple agents and the non-stationary nature of the environments’ complex dynamics. Firstly, optimizing policies in multi-agent scenarios presents significant challenges due to the complexity of multi-agent behaviours, especially in partially observable environments. Also, the dynamic nature of agents’ behaviours and their interactions with other agents lead to changes in the environment’s states and agents’ observations over time, which is more complicated in open environments. Furthermore, the need to balance individual and collective objectives in certain real-world multi-agent environments also complicates the decision-making process. This doctoral thesis aims to tackle the following three fundamental multi-agent research problems and propose a solution for each. Our research covers from theoretical analysis to practical application. We start by studying the problem of learning an efficient policy under partially observable environments. We notice that a group of agents cooperates to com- pete against another group of agents (named opponents), with the limitation that information about the opponents is inaccessible due to partial observability. To address this issue, we propose a novel multi-agent distributional actor-critic algorithm to achieve speculative opponent modeling with purely local information (i.e., the controlled agent’s observations, actions, and rewards). The actor maintains a speculated belief of the opponents, which we call the speculative opponent models, to predict opponents’ actions using local observations and make decisions accordingly. The distributional critic models the return distribution of the policy. It reflects the quality of the actor and thus can guide the training of the imaginary opponent model that the actor relies on. Extensive experiments confirm that our method successfully models opponents’ behaviours without their data and delivers superior performance against baseline methods with a faster convergence speed. Furthermore, in some environments, teammates’ numbers and policies change with the market, requiring workers to adapt to perform different sets of tasks across time. To solve this issue, we propose an RL-based method that allows the controlled agent to collaborate with dynamic teammates in open environments. The controlled agent maintains a dual teamwork situation inference model to capture the current teamwork state and facilitate reasonable decision-making under partial observability. Considering the dynamic types of teammates, we first leverage the Chinese Restaurant Process-based model to categorize versatile teammate policies into distinct clusters, which improves the efficiency of identifying current teamwork situations. Next, to model the heterogeneous relationships among agents and accommodate the varying number of teammates, we apply heterogeneous graph attention neural networks to learn the representation of the teamwork situation. Extensive experiments confirm that our method outperforms state-of-the-art baselines with faster convergence in various ad hoc teamwork tasks. Lastly, in certain real-world applications, such as routing problems and warehouse management, decision-makers must balance overall benefits with individual fairness among agents. Achieving both learning efficiency and fairness concurrently presents a complex, multi-objective, joint-policy optimization challenge. Moreover, existing approaches are predominantly confined to simulation environments. To address the above issues, we present a pioneering MARL approach to balance individual and collective objectives for agents collaboration. Extensive experiments on synthetic and real-world datasets demonstrate that our method not only surpasses the state-of-the-art DRL methods, but also exhibits advantages over established heuristics, notably in achieving significantly faster optimization speeds. This method underlines the essential integration of fairness into real-world applications, representing a major advancement in creating equitable and efficient logistics solutions. To conclude, this doctoral thesis investigates three fundamental multi-agent decision-making research problems that are ubiquitous and unsolved. The proposed three MARL methodology solutions achieve efficient policy training and performance for agents in multi-agent environments with uncertainties raised by the partially observable, the open environment, and the individual-collective objectives of MARL. This thesis delves into novel designs of various critical components of MARL, including MDP formulations, policy networks, training algorithms, and inference methods. These contributions significantly elevate the effectiveness and efficiency of cooperative MARL, establishing new performance benchmarks.

Automated clash resolution for reinforcement steel design in concrete frames via Q-learning and Building Information Modeling

This paper investigates the error accumulation problem in centralized training and decentralized execution (CTDE) policy‐based multi‐agent reinforcement learning (MARL) algorithms, which arises from local observation inaccuracies. To address this issue, we propose a novel MARL algorithm that incorporates imitation learning using local observations. Firstly, by analysing the multi‐agent proximal policy optimization algorithm and examining the problems arising when global states are replaced with local observations, it is proved that insufficient observations can lead to information loss, thereby introducing errors of advantage function, and it is demonstrated that the generalized advantage estimation method accumulates errors during the training process. Then, imitation learning is introduced and a novel training framework that combines reinforcement learning and imitation learning is proposed. During the reinforcement learning phase, an MARL agent trained with global observations acts as an expert. Subsequently, imitation learning is applied to train another agent that mimics the expert's decisions using only local observations. Finally, the effectiveness of this algorithm is verified in some commonly used multi‐agent environments, which demonstrates its superior performance compared to traditional multi‐agent reinforcement learning algorithms.

Reinforcement learning (RL) and multi-agent reinforcement learning (MARL) are disciplines concerned with defining automatically the behaviour of an agent, or a set of interacting agents, by means of reward mechanisms coming from the environment. An important research issue in the context of RL and MARL is the definition of approaches to combine the knowledge of multiple learning agents to improve the overall performance of the multi-agent system (MAS). This paper illustrates how to improve RL and MARL algorithms by utilizing results from multi-linear algebra such as tensors and tensor factorizations. In particular, the focus is on showing how to modify existing algorithms from literature to include a tensor factorization step applied to the Q-Tables learned by the individual agents to generalize the knowledge about the actions performed in the environment. The modified algorithms are then evaluated in three RL and MARL scenarios against their unmodified version to show the benefits of the tensor factorization step.

DeepMAG: Deep reinforcement learning with multi-agent graphs for flexible job shop scheduling

The combination of deep neural networks and reinforcement learning had received more and more attention in recent years, and the attention of reinforcement learning of single agent was slowly getting transferred to multiagent. Regret minimization was a new concept in the theory of gaming. In some game issues that Nash equilibrium was not the optimal solution, the regret minimization had better performance. Herein, we introduce the regret minimization into multiagent reinforcement learning and propose a multiagent regret minimum algorithm. This chapter first introduces the Nash Q-learning algorithm and uses the overall framework of Nash Q-learning to minimize regrets into the multiagent reinforcement learning and then verify the effectiveness of the algorithm in the experiment.

Multi-agent reinforcement learning has generally been studied under an assumption inherited from classical reinforcement learning: that the reward function is the exclusive property of the environment, and is only altered by external factors. In this work, we break free of this assumption and introduce peer rewarding, in which agents can deliberately influence each others' reward function. We formalize this more general setting and discuss its properties in depth. We also empirically study gifting, a peer rewarding mechanism which allows agents to reward other agents as part of their action space. We demonstrate that this approach can greatly improve learning progression in a resource appropriation setting and provide a preliminary analysis of the complex effects of gifting on the learning dynamics.