A survey on multi-agent reinforcement learning and its application

As one technique for autonomous driving, vehicular networks can achieve high efficiency with vehicle-and-infrastructure cooperation, bringing high safety and many value-added services. To achieve higher communication efficiency, much effort has been done to cope with the resource allocation issues for vehicular networks. Nevertheless, due to the strong nonconvexity and nonlinearity, the classical joint resource allocation problem in vehicular networks is typically NP-hard. The multiagent reinforcement learning (MARL) has emerged as a promising solution to tackle this challenge but its stability and scalability are not satisfactory when the amount of vehicles gets increased. In this article, we mainly investigate the issue of joint spectrum and power allocation in vehicular communication networks, and carefully consider the interactions between the vehicles and environment by incorporating the cooperative stochastic game theory with MARL, named complete-game MARL (CG-MARL), to achieve a better convergence and stability with the theoretical computational complexity <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(n^{N})$ </tex-math></inline-formula> with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> denoting the dimension of action space and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> denoting the number of V2X Vehicular. Furthermore, the mean-field game (MFG) theory is employed to further enhance the MARL for decreasing the horrible computing resource consumption caused by the CG-MARL to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(n^{2})$ </tex-math></inline-formula> while maintaining an approximate performance. The simulation results demonstrate that the proposed mean-field-aided MARL (MF-MARL) for vehicular network resource allocation can achieve 95% near-optimal performance with much lower complexity, which indicates its significant potentials in the scenarios with massive and dense vehicles.

Traditional multi-agent reinforcement learning (MARL) algorithms typically implement global parameter sharing across various types of heterogeneous agents without meticulously differentiating between different action semantics. This approach results in the action semantic conflict problem, which decreases the generalization ability of policy networks across heterogeneous types of agents and decreases the cooperation among agents in intricate scenarios. Conversely, completely independent agent parameters significantly escalate computational costs and training complexity. To address these challenges, we introduce an adaptive MARL algorithm named Generalized Action-Prediction Optimization (GAPO). First, we introduce the Generalized Action Space (GAS), which represents the union of all agent actions with distinct semantics. All agents first compute their unified representation in the GAS, and then generate their heterogeneous action policies with different available action masks. Second, in order to further improve cooperation between heterogeneous groups, we propose a Cross-Group Prediction (CGP) loss, which adaptively predicts the action policies of other groups by leveraging trajectory information. We integrate the GAPO into both value-based and policy-based MARL algorithms, giving rise to two practical algorithms: G-QMIX and G-MAPPO. Experimental results obtained within the SMAC, MPE, MAMuJoCo, and RPE environments demonstrate the superiority of G-QMIX and G-MAPPO over several state-of-the-art MARL methods, validating the effectiveness of our proposed adaptive generalized MARL approach.

Ensuring the security of computer networks is of utmost importance, and intrusion detection plays a vital role in safeguarding these systems. Traditional intrusion detection systems (IDSs) often suffer from drawbacks like reliance on outdated rules and centralized architectures, limiting their performance in the face of evolving threats and large-scale data networks. To address these challenges, we present an advanced anomaly detection-based IDS that utilizes a decentralized communicative multi-agent reinforcement learning (MARL). In our approach, multiple reinforcement learning agents collaborate in intrusion detection, effectively mitigating the non-stationarity problem and introducing a specialized secure communication method. We further enhance the learning process by incorporating external knowledge. Our approach is evaluated through extensive experiments conducted on the benchmark NSL Knowledge Discovery and Data Mining dataset. These experiments encompass diverse scenarios, involving varying numbers of agents to prove scalability feature. The results underscore the effectiveness of our method, which surpasses the performance of existing state-of-the-art solutions based on MARL, achieving a high accuracy rate of 97.80%.

Early threat detection is an increasing part of the cybersecurity landscape, given the growing scale and scope of cyberattacks in the recent years. Increasing exploitation of software vulnerabilities, especially in the manufacturing sector, demonstrates the ongoing need for autonomous network defence. In this work, we model the problem as a zero-sum Markov game between an attacker and defender reinforcement learning agents. Previous methods test their approach on a single topology or limit the agents to a subset of the network. However, real world networks are rarely fixed and often add or remove hosts based on demand, link failures, outages, or other factors. We do not confine our research to a fixed network in terms of size and topology, but instead are interested in larger networks and varied topologies to determine the scalability and robustness of the approach. We consider additional topologies and a robust training curriculum that incorporates network topologies to build more general, capable agents. We also use PPO which offers a good balance of computational complexity and convergence speed.

In this paper, we investigate the amalgamation of cell-free (CF) and extremely large-scale multiple-input multiple-output (XL-MIMO) technologies, referred to as a CF XL-MIMO, as a promising advancement for enabling future mobile networks. To address the computational complexity and communication power consumption associated with conventional centralized optimization, we focus on user-centric dynamic networks in which each user is served by an adaptive subset of access points (AP) rather than all of them. We begin our research by analyzing a joint resource allocation problem for energy-efficient CF XL-MIMO systems, encompassing cooperative clustering and power control design, where all clusters are adaptively adjustable. Then, we propose an innovative double-layer multi-agent reinforcement learning (MARL)-based scheme, which offers an effective strategy to tackle the challenges of high-dimensional signal processing. In the section of numerical results, we compare various algorithms with different network architectures. These comparisons reveal that the proposed MARL-based cooperative architecture can effectively strike a balance between system performance and communication overhead, thereby improving energy efficiency performance. It is important to note that increasing the number of user equipments participating in information sharing can effectively enhance SE performance, which also leads to an increase in power consumption, resulting in a non-trivial trade-off between the number of participants and EE performance.

In this paper we propose Multi-Agent Proxy Proximal Policy Optimization (MA3PO), a novel multi-agent deep reinforcement learning algorithm that tackles the challenge of cooperative continuous multi-agent control. Our method is driven by the observation that most existing multi-agent reinforcement learning algorithms mainly focus on discrete state/action spaces and are thus computationally infeasible when extended to environments with continuous state/action spaces. To address the issue of computational complexity and to better model intra-agent collaboration, we make use of the recently successful Proximal Policy Optimization algorithm that effectively explores of continuous action spaces, and incorporate the notion of intrinsic motivation via meta-gradient methods so as to stimulate the behavior of individual agents in cooperative multi-agent settings. Towards these ends, we design proxy rewards to quantify the effect of individual agent-level intrinsic motivation onto the team-level reward, and apply meta-gradient methods to leverage such an addition so that our algorithm can learn the team-level cumulative reward effectively. Experiments on various multi-agent reinforcement learning benchmark environments with continuous action spaces demonstrate that our algorithm is not only comparable with the existing state-of-the-art benchmarks, but also significantly reduces training time complexity.

An important role played by industrial Internet of Things (IIoTs) networks is supporting the operations of autonomous mobile robots (AMRs) by leveraging multiple-access edge caching servers. At the same time, judicious content caching strategies are essential for minimizing content retrieval delays and the costs associated with updating caches. In this study, a novel multiagent reinforcement learning (MARL)-based cache update strategy termed multiagent cache update (MACU) is proposed. MACU leverages the multiagent deep deterministic policy gradient (MADDPG) framework and aims to optimize cache updates to reduce Age of Information (AoI), minimize events where AoI exceeds acceptable levels, and ensure that the costs associated with performing cache updates are kept low. Furthermore, to mitigate the computational complexity of training agents with MACU, the MACU with global critic (MACU-GC) variant is introduced, which diverges from traditional MADDPG by employing a singular global critic for enhanced training efficiency. Extensive numerical evaluations showcase the proposed strategies’ superiority over conventional deep reinforcement learning-based caching methods, achieving significant improvements in AoI costs, caching costs, and AoI violation costs, while effectively reducing content retrieval latency, enhancing hit rates, and optimizing AoI and link load metrics.

Relay-aided Device-to-Device (D2D) communication combining visible light communication (VLC) with radio frequency (RF) is a promising paradigm in the internet of things (IoT). Static relay limits the flexibility and maintaining connectivity of relays in Hybrid VLC/RF IoT systems. By using a drone as a relay station, it is possible to avoid obstacles such as buildings and to communicate in a line-of-sight (LoS) environment, which naturally aligns with the requirement of VLC Systems. To further support the application of VLC in the IoT, subject to the challenges imposed by the constrained coverage, the lack of flexibility, poor reliability, and connectivity, drone relay-aided D2D communication appears on the horizon and can be cost-effectively deployed for the large-scale IoT. This paper proposes a joint resource allocation and drones relay selection scheme, aiming to maximize the D2D system sum rate while ensuring the quality of service (QoS) requirements for cellular users (CUs) and D2D users (DUs). First, we construct a two-phase coalitional game to tackle the resource allocation problem, which exploits the combination of VLC and RF, as well as incorporates a greedy strategy. After that, a distributed cooperative multi-agent reinforcement learning (MARL) algorithm, called WoLF policy hill-climbing (WoLF-PHC), is proposed to address the drones relay selection problem. Moreover, to further reduce the computational complexity, we propose a lightweight neighbor–agent-based WoLF-PHC algorithm, which only utilizes historical information of neighboring DUs. Finally, we provide an in-depth theoretical analysis of the proposed schemes in terms of complexity and signaling overhead. Simulation results illustrate that the proposed schemes can effectively improve the system performance in terms of the sum rate and outage probability with respect to other outstanding algorithms.

Polymer Reaction Engineering in China

The topic of lethal autonomous weapon systems has recently caught public attention due to extensive news coverage and apocalyptic declarations from famous scientists and technologists. Weapon systems with increasing autonomy are being developed due to fast improvements in machine learning, robotics, and automation in general. These developments raise important and complex security, legal, ethical, societal, and technological issues that are being extensively discussed by scholars, nongovernmental organizations (NGOs), militaries, governments, and the international community. Unfortunately, the robotics community has stayed out of the debate, for the most part, despite being the main provider of autonomous technologies. In this column, we review the main issues raised by the increase of autonomy in weapon systems and the state of the international discussion. We argue that the robotics community has a fundamental role to play in these discussions, for its own sake, to provide the often-missing technical expertise necessary to frame the debate and promote technological development in line with the IEEE Robotics and Automation Society (RAS) objective of advancing technology to benefit humanity

In this paper, a distributed intelligence algorithm is used to manage the optimal power flow problem in islanded microgrids. The methodology provides a suboptimal solution although the error is limited to a few percent as compared to a centralized approach. The solution algorithm is multi-agent based. According to the method, couples of agents communicate with each other only if the buses where they are located are electrically connected. The overall prizing system required for learning uses a feedback from an approximated model of the network. Based on the latter, a distributed reiforcement learning algorithm is implemented to minimize the joule losses while meeting operational constraints. Simulation studies with a small microgrids show that the method is computationally efficient and capable of providing sub-optimal solutions. Due to the limited computational complexity, the proposed method has great potential for online implementation.

This research characterized how faculty educate environmental engineering (EnvE) undergraduate and graduate students on ethical and societal issues (ESI), in comparison to civil, chemical, and mechanical engineering (CCM). In a national survey, responses from 158 instructors of EnvE students representing 114 institutions were received, and compared to 505 CCM responses. Despite the fact that 97% of the respondents taught EnvE students about ESI in their courses, only 30% felt that undergraduate students in their program received sufficient education on both the societal impacts of technology and ethical issues; only 20% felt this way about their graduate program. CCM instructors had similar opinions about the lack of sufficient ESI education. Survey respondents integrated a broad range of ESI topics into their courses, with poverty and social justice issues more common in EnvE compared to CCM. Examples of ESI teaching and assessment methods in courses ranging from sophomore or junior level engineering courses to graduate courses are provided. Co-curricular activities including engineering service groups, professional societies, design competitions, and research experiences also provide opportunities to educate EnvE students on ESI. This article is the first to present the results of a large-scale study on ESI education in EnvE as compared to related disciplines. Through these results, the article hopes to inspire others to integrate ESI-related topics into the courses and co-curricular activities that they mentor, and work with faculty in their program to provide a comprehensive ESI education that equips students with the knowledge and values to behave ethically in practice.

Support Vector Machines (SVMs) are machine learning models with many diverse applications. The performance of these models depends on a set of assigned hyper-parameters. The task of hyper-parameter tuning has been performed by metaheuristics methods and recent studies have shown that the effectiveness of these methods is statistically equivalent. In this work we compare four bio-inspired metaheuristics (Bat Algorithm, Firefly Algorithm, Particle Swarm Optimization Algorithm and Social Emotional Optimization Algorithm) to test the hypothesis that the efficiency among these differs while the effectiveness remains. Experimental results on several classification problems indicate that there exist bio-inspired algorithms with higher efficiency, in terms of the required number of SVM evaluations to find optimal hyper-parameters. Based on these results the Bat Algorithm is recommended for SVM hyper-parameter tuning.

No abstract.

Detection and classification of partial discharge (PD) signal in XLPE cable is very important to find out the root cause of insulation failure. The purpose of this work is to improve the accuracy and effectiveness in classifying various types of PD signals including corona, surface, and internal discharges in a noisy environment using machine learning (ML) and deep learning (DL) techniques. For ML models, statistical features such as skewness, kurtosis, standard deviation, and variance are extracted from the PD signals and used for training. In contrast, the DL model focuses on continuous wavelet transforms (CWT) based scalogram images which highlight the resolution of the signal’s energy and are used for feature extraction in the classification study. Instead of using traditional phase-resolved partial discharge (PRPD) plots, this work introduces a novel approach to converting PD signals to scalogram images using CWT. These scalogram images provide a visual representation of the frequency components of different PD types and how they change over time. The wavelet-based scalogram images are used as input features for the proposed convolutional neural network (CNN) and state-of-the-art (SOTA) DL models, along with shallow ML classifiers for model training, validation, and testing. Along with this k-fold cross-validation and hyperparameter tuning are also employed to enhance the overall performance of the model. The classification results demonstrate that the proposed CNN model achieves up to 97.33% recognition accuracy and less computational complexity as compared to various ML and DL models.