Multi-agent reinforcement learning for edge information sharing in vehicular networks

To guarantee heterogeneous performance requirements of diverse vehicular services, it is necessary to design a full cooperative policy for both vehicle to infrastructure (V2I) links and vehicle to vehicle (V2V) links. This paper investigates how to improve the quality of experience (QoE) of the V2I users for video services while satisfying the delay requirements of both V2I and V2V links. In specific, a QoE maximization problem is formulated with consideration of vehicular collaboration where task offloading decision, channel reuse decision and power allocation of V2V users are all included. A multi-agent reinforcement learning (MARL) framework is then designed, where a new reward function is proposed to evaluate the utility of the considered network. Thereafter, a proximal policy optimization approach is proposed to enable each V2V user to learn policy individually with the shared global network reward. The effectiveness of the proposed approach is finally validated with comparison of other baseline approaches through extensive simulation experiments.

Smart transportation networks are challenged by a critical challenge of traffic congestion and inevitably requires a complex package of management solutions. This paper introduces a Secure Multi-Agent Reinforcement Learning (MARL) framework that optimizes traffic whilst protecting from malicious threats. Traffic Prediction Dataset accessible through Kaggle is used as the foundation for training reinforcement learning agents from four junctions who cooperate to manage traffic signals. The widespread preprocessing procedures that also dealt with data gaps in addition to dimension transformation and value normalization generated genuine traffic prediction models. The Multi-Agent Reinforcement Learning framework reduced the mean waiting time by 45.7% with a 47.0% traffic passing rate higher than that achieved using standard fixed-duration signal control techniques. Homomorphic encryption, when integrated with blockchain protocols, instituted novel security protocols for agent to agent communication, and this yielded a reduction of adversarial attacks below 90% levels. The implementation framework provided a 39.3% reduction in the fuel consumption when implemented in real-time urban traffic management systems. Secure Multi-Agent Reinforcement Learning (MARL) systems show their potential aptitude to propel smart transportation networks in the while providing increased efficiency and security as well as sustainability effectiveness.

A Multi-Agent Reinforcement Learning Framework for Public Health Decision Analysis.

Financial institutions are increasingly dealing with difficulties in preserving transaction integrity due to escalating digital fraud and extensive cloud-based operations. Conventional machine learning and rule-based detection systems frequently have latency in replies, restricted adaptability, and an absence of verifiable audit trails. The proposed study presents a Hybrid Multi-Agent Reinforcement Learning (MARL) and Blockchain framework to overcome existing limitations, integrating adaptive multi-agent coordination for real-time fraud detection with a permissioned blockchain ledger to guarantee secure, tamper-proof validation of agent decisions. MARL agents adaptively acquire optimal detection strategies through incentive shaping and inter-agent communication, while blockchain consensus and smart contracts substantiate decision proofs for regulatory adherence. The experimental evaluation of the Kaggle Financial Transactions dataset demonstrates improved accuracy relative to baseline models: Accuracy 0.989, Precision 0.981, Recall 0.987, F1-Score 0.984, AUC-ROC 0.992, with negligible latency (1.8 ms/transaction). The comparative research reveals a persistent superiority over Random Forest, XGBoost, CNN- LSTM, and Transformer-based classifiers, showcasing robustness, scalability, and real-time application. The suggested study enhances the area by merging reinforcement learning adaptability with blockchain security, establishing a realistic, high-throughput, and reliable framework for cloud- based financial systems, while presenting a model for future research in agentic AI-driven fraud prevention.

As a significant issue in the development of Smart City, traffic light control is closely related to public transport. Multi-agent reinforcement learning is a feasible paradigm to control decentralized systems. In this work, we design a recurrent communication module for traffic light control systems. Aiming at lightweight decentralized learning, we decouple the interaction among entities, using an improved recurrent neural network to share messages. Regarding traffic lights as a partially observable Markov Decision Process, we develop a complete reinforcement learning model. Every light receives messages from its neighbors through the improved recurrent neural network. For the further lights, its neighbor lights can play the role of intermediation to transmit messages. By our communication framework, traffic lights can learn their control policies in a decentralized way, and distribution control can be executed through a communication channel. Several simulated experiments show that our method outperforms the fixed time method and the existing reinforcement learning-based frameworks under different traffic demands.

Within the cloud manufacturing paradigm, Cloud Manufacturing Service Composition (CMSC) is a core technology for intelligent resource orchestration in Cloud Manufacturing Platforms (CMP). However, existing research faces critical limitations in real-world CMP operations: single-task-centric optimization ignores resource sharing/competition among coexisting manufacturing tasks (MTs), causing performance degradation and resource “starvation”; traditional heuristics require full re-execution for new scenarios, failing to support real-time online decision-making; single-agent reinforcement learning (RL) lacks mechanisms to balance global efficiency and inter-task fairness, suffering from inherent fairness defects. To address these challenges, this paper proposes a fairness-aware multi-task CMSC method based on Multi-Agent Reinforcement Learning (MARL) under the Centralized Training with Decentralized Execution (CTDE) framework, targeting the symmetry-breaking issue of uneven resource allocation among MTs and aiming to achieve symmetry restoration by restoring relative balance in resource acquisition. The method constructs a multi-task CMSC model that captures real-world resource sharing/competition among concurrent MTs, and integrates a centralized global coordination agent into the MARL framework (with independent task agents per MT) to dynamically regulate resource selection probabilities, overcoming single-agent fairness defects while preserving distributed autonomy. Additionally, a two-layer attention mechanism is introduced—task-level self-attention for intra-task subtask correlations and global state self-attention for critical resource features—enabling precise synergy between local task characteristics and global resource states. Experiments verify that the proposed method significantly enhances inter-task fairness while maintaining superior global Quality of Service (QoS), demonstrating its effectiveness in balancing efficiency and fairness for dynamic multi-task CMSC.

Efficient parking distribution is crucial for urban traffic management; nevertheless, variable demand and spatial disparities raise considerable obstacles. Current research emphasizes local optimization but neglects the fundamental challenges of real-time parking allocation, resulting in inefficiencies within intricate metropolitan settings. This research delineates two key issues: (1) A dynamic imbalance between supply and demand, characterized by considerable fluctuations in parking demand over time and across different locations, rendering static allocation solutions inefficient; (2) spatial resource optimization, aimed at maximizing the efficiency of limited parking spots to improve overall system performance and user satisfaction. We present a Multi-Agent Reinforcement Learning (MARL) framework that incorporates adaptive optimization and intelligent collaboration for dynamic parking allocation to tackle these difficulties. A reinforcement learning-driven temporal decision mechanism modifies parking assignments according to real-time data, whilst a Graph Neural Network (GNN)-based spatial model elucidates inter-parking relationships to enhance allocation efficiency. Experiments utilizing actual parking data from Melbourne illustrate that Multi-Agent Reinforcement Learning (MARL) substantially surpasses conventional methods (FIFO, SIRO) in managing demand variability and optimizing resource distribution. A thorough quantitative investigation confirms the strength and flexibility of the suggested method in various urban contexts.

This paper presents a novel approach to designing beam codebooks for downlink multiuser hybrid multiple-input–multiple-output (MIMO) wireless communication systems, leveraging multi-agent reinforcement learning (MARL). The primary objective is to develop an environment-specific beam codebook composed of non-interfering beams, learned by cooperative agents within the MARL framework. Machine learning (ML)-based beam codebook design for downlink communications have been based on channel state information (CSI) feedback or only reference signal received power (RSRP), consisting of an offline training and user clustering phase. In massive MIMO, the full CSI feedback data is of large size and is resource-intensive to process, making it challenging to implement efficiently. RSRP alone for a stand-alone base station is not a good marker of the position of a receiver. Hence, in this work, uplink CSI estimated at the base station along with feedback of RSRP and binary acknowledgment of the accuracy of received data is utilized to design the beamforming codebook at the base station. Simulations using sub-array antenna and ray-tracing channel demonstrate the proposed system’s ability to learn topography-aware beam codebook for arbitrary beams serving multiple user groups simultaneously. The proposed method extends beyond mono-lobe and fixed beam architectures by dynamically adapting arbitrary shaped beams to avoid inter-beam interference, enhancing the overall system performance. This work leverages MARL’s potential in creating efficient beam codebooks for hybrid MIMO systems, paving the way for enhanced multiuser communication in future wireless networks.

Underwater monitoring is extremely complex due to the lack of a global localization system, limited communication and environmental factors such as turbidity and darkness that limit visibility, affecting control and situational awareness. Typically, monitoring relies on a single autonomous underwater vehicle (AUV) or a set of independent AUVs; techniques which are prone to failure as they rely only on onboard odometry and sensors, making missions vulnerable to malfunctions, damage, and noise. To address these challenges, we propose a Multi-Agent Reinforcement Learning (MARL) framework to enable cooperation among multiple AUVs, mitigating the limitations of the underwater environment. Our in-silico solution focuses on a group of robots learning a strategy to follow a partially hidden underwater pipe without global localization, while dealing with environmental disturbances affecting sensors and actuators. The numerosity of the agents, and most importantly their collaboration, helps overcome underwater visibility constraints. By sharing relative position information of neighboring agents with respect to the pipe, navigation is improved. By introducing quantitative measures for pipe exploration, we show that cooperation significantly enhances system performance compared to independent agents. Emerging collaboration among robots allows the swarm to complete pipe inspections faster and more efficiently than non-cooperative baseline models of non-interacting agents, even under extremely reduced visibility scenarios. Moreover, single agents also benefit from cooperation, learning effective policies more quickly and covering a longer portion of the pipe. Finally, our model guarantees explainability. We analyze learned strategies and provide a visualization method that allows the interpretation of the learned policies. • Reinforcement learning is applied to pipeline following by underwater robotic agent. • In conditions of poor visibility, a single agent is not able to complete the mission. • In contrast, multi-agent team completes this task using reinforcement learning. • Swarm collaboration enables faster and more efficient task completion. • Collaboration enables agents develop more efficient individual strategies.

Aim and ApproachLarge cooling water system has great potential of energy saving due to improper operation control. The coupled hydraulic and thermodynamic characteristics among chillers, pumps, and cooling towers lead to the difficulty of cooling water system operation. This study investigates an advanced control approach of multi-agent reinforcement learning (RL) for a large cooling water system. The multiple RL agents interact with each other to deal with the complex coupled characteristics in the cooling water system. The proposed approach is implemented in a real-world cooling water system and validated with experiments.Scientific Innovation and RelevanceIn this study, a multi-agent RL approach is proposed to separately control the chillers, pumps, and cooling towers in the cooling water system. The soft actor critic algorithm is used in the RL agent, as it is an efficient and advanced RL algorithm for continuous and discrete control problems. The agents are designed to have private states, private actions, and shared rewards. The reward functions are carefully designed to balance the agents for global optima.To provide a realistic environment for RL training, a detailed physical modeling platform of a cooling water system is established and validated. This platform is capable of hydraulic and thermodynamic calculation of cooling water systems with customized numbers and characteristics of chillers, pumps, and cooling towers, which is scalable for different cooling water systems.The proposed multi-agent RL controller is implemented in the building automation system of a real cooling water system. A one-month experimental study of the multi-agent RL performance is conducted. The proposed multi-agent RL controller is compared with a single-agent RL controller and a rule-based controller through real-world experiments to demonstrate the energy performance. This is a pilot study of multi-agent RL approach to real-world cooling water system control.Preliminary Results and ConclusionsThe preliminary simulation study is conducted in the established cooling water system model. The electricity consumption simulation error is 4.0% in coefficient of variation of mean absolute error (CVMAE) compared with the measured 15-minute electricity consumption. The 6-month simulation shows that the single-agent RL can save 6.2% energy from the rule-based control, while the multi-agent RL can save 7.2% electricity consumption of the cooling water system. In the following study, the multi-agent RL implementation will be practiced with more experiments and applications for cooling water system control.

Recently, the researches on multi-agent reinforcement learning (MARL) have attracted tremendous interest in many applications, especially for autonomous driving. The main problem of MARL is how to deal with the uncertainty in the environment and the interaction between the connected agents. To solve the problem, a distributed robust temporal differential deep Q-network algorithm (MARTD-DQN) was developed in this paper. MARTD-DQN consists of two parts, the decentralized MARL algorithm (DMARL) and the robust TD deep Q-network algorithm (RTD-DQN). DMARL improves the robustness of the policy estimation by fusing the states from the neighbors over communicated networks. RTD- DQN improves the robustness to outliers through on-line estimation of the uncertainty. By combining the two algorithms, the proposed algorithm can be robust not only to node failures but also to the outliers. Then the proposed algorithm is applied to ACC simulations of autonomous cars. The simulation results are given to show the efficiency of the proposed algorithm.

As one technique for autonomous driving, vehicular networks can achieve high efficiency with vehicle-and-infrastructure cooperation, bringing high safety and many value-added services. To achieve higher communication efficiency, much effort has been done to cope with the resource allocation issues for vehicular networks. Nevertheless, due to the strong nonconvexity and nonlinearity, the classical joint resource allocation problem in vehicular networks is typically NP-hard. The multiagent reinforcement learning (MARL) has emerged as a promising solution to tackle this challenge but its stability and scalability are not satisfactory when the amount of vehicles gets increased. In this article, we mainly investigate the issue of joint spectrum and power allocation in vehicular communication networks, and carefully consider the interactions between the vehicles and environment by incorporating the cooperative stochastic game theory with MARL, named complete-game MARL (CG-MARL), to achieve a better convergence and stability with the theoretical computational complexity <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(n^{N})$ </tex-math></inline-formula> with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> denoting the dimension of action space and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> denoting the number of V2X Vehicular. Furthermore, the mean-field game (MFG) theory is employed to further enhance the MARL for decreasing the horrible computing resource consumption caused by the CG-MARL to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(n^{2})$ </tex-math></inline-formula> while maintaining an approximate performance. The simulation results demonstrate that the proposed mean-field-aided MARL (MF-MARL) for vehicular network resource allocation can achieve 95% near-optimal performance with much lower complexity, which indicates its significant potentials in the scenarios with massive and dense vehicles.

Effective policy optimization in multiagent reinforcement learning (MARL) necessitates extensive exploration of high-dimensional state-action spaces. However, such exploration may not only trigger unsafe states but also compromise system stability, posing significant challenges for deployment in safety-critical systems. To address this challenge, this article proposes a safety-stability layer that integrates robust control barrier functions (RCBFs) and input-to-state stable control Lyapunov functions (ISS-CLFs) for multiagent systems operating in unknown environments with uncertain dynamics. Furthermore, by integrating safety-stability constraints with a MARL framework, during the training phase, we exclusively focus on goal-reaching objectives to expand the policy network's exploration space, while in the deployment phase, policy outputs are filtered through a real-time safety-stability layer. In addition, an event-triggered mechanism for action compensation calculation is designed based on safety condition assessments to conserve computational resources. Finally, the effectiveness of the proposed method is validated through simulation experiments in dynamic multiunicycle environments. The results demonstrate that our approach not only ensures strict adherence to safety constraints but also significantly enhances the task execution efficiency of multiagent systems.

Video Recognition has drawn great research interest and great progress has been made. A suitable frame sampling strategy can improve the accuracy and efficiency of recognition. However, mainstream solutions generally adopt hand-crafted frame sampling strategies for recognition. It could degrade the performance, especially in untrimmed videos, due to the variation of frame-level saliency. To this end, we concentrate on improving untrimmed video classification via developing a learning-based frame sampling strategy. We intuitively formulate the frame sampling procedure as multiple parallel Markov decision processes, each of which aims at picking out a frame/clip by gradually adjusting an initial sampling. Then we propose to solve the problems with multi-agent reinforcement learning (MARL). Our MARL framework is composed of a novel RNN-based context-aware observation network which jointly models context information among nearby agents and historical states of a specific agent, a policy network which generates the probability distribution over a predefined action space at each step and a classification network for reward calculation as well as final recognition. Extensive experimental results show that our MARL-based scheme remarkably outperforms hand-crafted strategies with various 2D and 3D baseline methods. Our single RGB model achieves a comparable performance of ActivityNet v1.3 champion submission with multi-modal multi-model fusion and new state-of-the-art results on YouTube Birds and YouTube Cars.

Excessive fluctuations in travel time between stops and demand at bus stops during bus operations can lead to operational instability in bus systems, such as bus bunching. To tackle this issue, this paper presents a dynamic bus holding control strategy leveraging multi-agent reinforcement learning to stabilize bus system operations and prevent bus bunching. First, bus motion system is constructed, and the rules for bus operation and passenger behavior are defined. Then, agent-based transit operation management is established, the elements of the multi-agent reinforcement learning framework are outlined, and a centralized training and decentralized execution method is proposed. Additionally, an event-driven simulation environment is developed for training and testing the agents. Finally, extensive numerical simulations are conducted to evaluate the proposed method against baseline approaches using various performance metrics. The results demonstrate that the proposed method effectively captures the dynamics of the bus system and accounts for the long-term impacts of current decisions, resulting in the most balanced bus trajectories, optimal passenger load distribution, and minimal total holding time.