M-VP2: Microservice-Oriented Vulnerability Patch Planning - A Cost-Aware Approach Using Multi-Agent Reinforcement Learning

In a multi-agent setting, the optimal policy of a single agent is largely dependent on the behavior of other agents. We investigate the problem of multi-agent reinforcement learning, focusing on decentralized learning in non-stationary domains for mobile robot navigation. We identify a cause for the difficulty in training non-stationary policies: mutual adaptation to sub-optimal behaviors, and we use this to motivate a curriculum-based strategy for learning interactive policies. The curriculum has two stages. First, the agent leverages policy gradient algorithms to learn a policy that is capable of achieving multiple goals. Second, the agent learns a modifier policy to learn how to interact with other agents in a multi-agent setting. We evaluated our approach on both an autonomous driving lane-change domain and a robot navigation domain.

A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using MARL methods for LLM collaboration and highlights the associated challenges.

Recently, the researches on multi-agent reinforcement learning (MARL) have attracted tremendous interest in many applications, especially for autonomous driving. The main problem of MARL is how to deal with the uncertainty in the environment and the interaction between the connected agents. To solve the problem, a distributed robust temporal differential deep Q-network algorithm (MARTD-DQN) was developed in this paper. MARTD-DQN consists of two parts, the decentralized MARL algorithm (DMARL) and the robust TD deep Q-network algorithm (RTD-DQN). DMARL improves the robustness of the policy estimation by fusing the states from the neighbors over communicated networks. RTD- DQN improves the robustness to outliers through on-line estimation of the uncertainty. By combining the two algorithms, the proposed algorithm can be robust not only to node failures but also to the outliers. Then the proposed algorithm is applied to ACC simulations of autonomous cars. The simulation results are given to show the efficiency of the proposed algorithm.

Many real-world tasks on practical control systems involve the learning and decision-making of multiple agents, under limited communications and observations. In this paper, we study the problem of networked multi-agent reinforcement learning (MARL), where multiple agents perform reinforcement learning in a common environment, and are able to exchange information via a possibly time-varying communication network. In particular, we focus on a collaborative MARL setting where each agent has individual reward functions, and the objective of all the agents is to maximize the network-wide averaged long-term return. To this end, we propose a fully decentralized actor-critic algorithm that only relies on neighbor-to-neighbor communications among agents. To promote the use of the algorithm on practical control systems, we focus on the setting with continuous state and action spaces, and adopt the newly proposed expected policy gradient to reduce the variance of the gradient estimate. We provide convergence guarantees for the algorithm when linear function approximation is employed, and corroborate our theoretical results via simulations.

Prescriptive maintenance extends predictive maintenance by not only forecasting potential failures, but also recommending concrete maintenance actions, explicitly accounting for operational, organizational, and resource-related constraints. Existing approaches in this area are often fragmented, as they typically focus either on technical diagnostics or on isolated optimization criteria, without integrating production schedules, resource availability, and organizational decision constraints into a unified decision- making framework. This creates a research gap in the integration of diagnostic and organizational data, thereby limiting real-time decision-making. The aim of this study was to develop and validate a hybrid prescriptive maintenance algorithm, combining fuzzy logic, neural network classification, and Analytic Hierarchy Process (AHP)-based scenario evaluation, aligned with the principles of Industry 4.0. The algorithm was implemented in MATLAB. A central design feature was the integration of sensor-based measurement data with organizational inputs to dynamically generate decision scenarios, accounting for time, cost, and resource constraints. This integration enabled the algorithm to consistently generate recommendations that respected budgetary and temporal constraints, demonstrating its effectiveness in translating combined technical and organizational data into actionable maintenance decisions. The algorithm was validated using real-world production data, achieving decision accuracy in the range of 94–99%, defined as consistency with expert-validated reference decisions, with a relative prediction deviation below 0.05%. It was further observed that when organizational and contextual data were not considered (such as production schedules, resource availability, and personnel qualifications), over 85% of the recommended maintenance scenarios deviated from the reference decisions determined based on realized production outcomes, with deviation assessed in terms of completion time and cost. The findings demonstrate that the developed prescriptive maintenance algorithm significantly improves decision accuracy, optimizes resource utilization, and reduces both downtime and operational costs, highlighting its practical impact and value for industrial maintenance management.

Densely deployed femtocells in heterogeneous networks (HetNets) improve the quality-of-service (QoS) and quality-of-experience (QoE) for both next-generation networks and end users, respectively. However, users near the femtocell edge, i.e., femto edge users (FEUs), which are far away from the Femto base station (FBS) may receive a low signal-to-interference-plus-noise ratio (SINR). Moreover, due to the distributed and dynamic nature of HetNets, femtocells may not resist various eavesdropping attacks from different attackers. Thus, it is essential to ensure the wireless channel's secrecy through which FEUs communicate with FBS since they may face eavesdropping attacks and may have low SINR and achievable data rates. Aiming to enhance the secrecy rate of femtocells, in this article, we formulate a joint optimization problem of beamforming vectors, artificial noise (AN), and power control for terahertz (THz)-enabled femtocells. Addressing this non-convex optimization problem is challenging due to the system's complexity and dynamic nature of FEUs. Thus, we translate the optimization problem into a multi-agent reinforcement learning (MARL) problem using the Markov decision process (MDP). Then, to solve the MDP with continuous action space, two deep reinforcement learning (DRL) based schemes, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Deep Deterministic Policy Gradient-based Secrecy Maximization (DDPG-SM)</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Asynchronous-Advantage-Actor-Critic based Secrecy Maximization (A3C-SM)</i> are proposed to maximize the secrecy rate of femtocells. The secrecy rate performance of the proposed schemes is compared and analyzed with Advantage-Actor-Critic (A2C) and state-of-the-art Perfect Secrecy Rate Maximization (PSRM) schemes. Simulation results show that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> schemes achieve 37.73% and 22.64% better average secrecy rate in comparison to the PSRM scheme. Also, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> scheme outperforms all other <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> , A2C, and PSRM schemes in terms of average secrecy rate, received SINR, and energy-efficiency of femtocells.

Decentralized actor-critic (AC) is one of the most dominant algorithms for dealing with multiagent reinforcement learning (MARL) problems. However, exploration-efficient, sample-efficient, and communication-efficient are difficult to achieve simultaneously by existing decentralized AC methods. For this reason, this article develops a decentralized multiagent AC algorithm by incorporating entropy regularization to improve exploration with theoretical guarantees, referred to as multi-agent AC algorithm with entropy regularization (MACE). Moreover, we rigorously prove that MACE can achieve sample complexity $\mathcal {O}(\epsilon ^{-2}\ln \epsilon ^{-1})$ and communication complexity of $\mathcal {O}(\epsilon ^{-1}\ln \epsilon ^{-1})$ , which match the best complexities at present. Finally, the performance of MACE is also evaluated on reinforcement learning (RL) tasks. The experimental results show that the proposed algorithm achieves better exploration efficiency than state-of-the-art decentralized AC-type algorithms.

This paper investigates the spectrum sharing problem in vehicular networks, where multiple vehicle-to-vehicle (V2V) links reuse the frequency spectrum preoccupied by vehicle-to-infrastructure (V2I) links. We model the resource sharing as a multi-agent reinforcement learning (RL) problem, which is then solved using a fingerprint-based deep Q-network method. The V2V links, each acting as an agent, collectively interact with the vehicular environment, receive distinctive observations yet a common reward, and then improve policy design through updating their Q-networks with gained experiences. Simulation results demonstrate desirable performance of the proposed resource allocation scheme based on multi-agent RL in terms of both V2I capacity and V2V payload delivery probability.

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL) has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with human-driven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision-making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic (MA2C) method is proposed with a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is designed to incorporate fuel efficiency, driving comfort, and the safety of autonomous driving. A comprehensive experimental study is made that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency, safety, and driver comfort.

Dynamic resource provisioning is a challenging technique to meet the service level agreement (SLA) requirements of multi-tier applications in virtualization-based cloud computing. Prior efforts have addressed this challenge based on either a cost-oblivious approach or a cost-aware approach. However, both approaches may suffer frequent SLA violations under flash crowd conditions. Because they ignore the benefit gained that a multi-tier application continuously guarantees the SLA in the new configuration. In this paper, we propose a benefit-aware approach with feedback control theory to solve this problem. Experimental results based on live workload traces show that our approach can reduce resource provisioning cost by as much as 30% compared with a costoblivious approach, and can effectively reduce SLA violations compared with a cost-aware approach.

Recent advancements in the domain of Network Function Virtualization (NFV), and rollout of next-generation networks have necessitated the requirement for the upkeep of latency-critical application architectures in future networks and communications. While Cloud service providers recognize the evolving mission-critical requirements in latency sensitive verticals such as autonomous driving, multimedia, gaming, telecommunications, and virtual reality, there is a wide gap to bridge the Quality of Service (QoS) constraints for the end-user experience. Most latency-critical services are over-provisioned on all fronts to offer reliability, which is inefficient towards scalability in the long run. To address this, we propose a strategy to model frequent violations on the application level as a multi-output target to enable more complex decision-making in the management of virtualised communication networks. In this work, we utilize data from a real-world deployment to configure and draft a realistic set of Service Level Objectives (SLOs) for a voice based NFV application, and develop a deep neural network based multi-label classification methodology to identify and predict multiple categories of SLO breaches associated with an application state. With this, we aim to gain granular SLA and SLO violation insights, enabling us to study and mitigate their impact and inform precision in drafting proactive scaling policies. We further compare the performance against a set of multi-label compatible machine learning classifiers, and address class imbalance in a multi-label setup. We perform a comprehensive evaluation to assess the performance on example-based, label-based and ranking-based measures, and demonstrate the suitability of deep learning in such a use-case.

Cloud computing has reached the core of the digital ecosystems and provides high-availability and scale-able access to the services of computational services. Nevertheless, the active resource requirements, enforcement of service-level agreement (SLA), energy usage restrictions, and a variation in the cost-performance represent significant challenges to the cloud resource management. The conventional heuristic-based schedulers in clouds orchestrators like Kube and OpenStack fail to adjust to real-time workload variability, fractional resources, violation of SLA and inflated operational costs frequently occur. The adaptive, reward-driven decision-making properties of reinforcement learning (RL) have been discovered to be an interesting alternative to a heuristic policy. However, centralized RL has scalability and state observability scale in distributed clouds. As a way of overcoming these limitations, this study hypothesizes a Multi-Agent Reinforcement Learning (MARL) model of autonomous resource scheduling and optimization in cloud infrastructures. The offered structure suggests the use of decentralized intelligent agents executing a coordinated management of computational, storage, and network allocations. The agents are optimized to adapt local behaviors under environment feedback with the aid of cooperative communication to enhance global optimization. The system uses hybrid in reinforcement system consisting of Deep Q-Learning (DQN) and Proximal Policy Optimization (PPO) to trade between exploration, stability, and on-policy improvement. The goal of an adaptive reward model is to maximize important performance indicators (KPIs): SLA adherence, task completion latency, energy efficiency and balance resource utilization. The Cloud sim Plus experimental measurements were performed in a simulated multi-cluster environment and supported by extra features of container virtualization. The proposed version of the MARL scheduler showed a better throughput (18.7%), SLA violations reduction (27.4%) and energy consumption (14.3) than the baseline strategies like Round-Robin, Min-Min, and centralized DQN models. Convergence time was reduced and decision correlation under the workload bursts was improved due to the incorporation of the inter-agent communication. The research paper has a tremendous impact in the field by proposing a foresight, smart cloud orchestration framework that has the capability to adapt itself in the distributed large scale systems. Additionally, we offer an improvement in algorithms, performance analysis, system workflow optimization, and theoretical support of a cooperative reinforcement learning in clouds. The implication of these findings in the future is that the applicability of the research to edge and fog computing settings can be applied to Industry 5.0 automation

Multi-agent reinforcement learning (MARL) has made significant strides in enabling coordinated behaviors among autonomous agents. However, most existing approaches assume that communication is instantaneous, reliable, and has unlimited bandwidth; these conditions are rarely met in real-world deployments. This survey systematically reviews recent advances in robust and efficient communication strategies for MARL under realistic constraints, including message perturbations, transmission delays, and limited bandwidth. Furthermore, because the challenges of low-latency reliability, bandwidth-intensive data sharing, and communication-privacy trade-offs are central to practical MARL systems, we focus on three applications involving cooperative autonomous driving, distributed simultaneous localization and mapping, and federated learning. Finally, we identify key open challenges and future research directions, advocating a unified approach that co-designs communication, learning, and robustness to bridge the gap between theoretical MARL models and practical implementations.

A Service Level Agreement (SLA) is a legal contract between parties to ensure the Quality of Service (QoS). It specifies one or more service level objectives (SLO), to ensure that the QoS delivered has met customer expectations. However, It becomes hard to guarantee QoS levels and detect SLA violations. Therefore, we propose to use MDE (Model Driven Engineering) to express the SLA contract requirements. This latter, created with a specific modeling language (DSML), will be used harmonically with a Multi-agent systems (MASs) in order to monitor SLA violations in real-time. Indeed, MASs are suitable tools for self-detection of failures and self-monitoring of cloud operations and services, QoS negotiation and SLA management. They are designed to operate in a dynamically changing environment. Our main motivation is firstly to use MDE technology for the creation of the SLA contract and then to integrate MASs in order to control the quality of service contract and guarantee transparency and symmetry with respect to the SLA contract between prospective signatories.

Critical networked services are usually regulated by Service Level Agreements (SLAs). In order to ensure SLAs are being met, it is necessary to monitor Service Level Objectives (SLOs). Active measurement mechanisms are usually chosen to perform this monitoring task, which requires measurement probes to be activated in network devices. However, these probes are expensive in terms of computational resources consumption, thus, active measurement mechanisms usually can cover only a fraction of what could be measured, which can lead to SLA violations being missed. Besides that, highly dynamic networking patterns require the ongoing selection of the candidate network destinations for probing and their respective prioritization, a practice that is not well suited for human administrators because configuring the probes is labor-intensive and error-prone. A possibility to improve the detection of SLA violations is the employment of Peer-to-Peer (P2P) technology in order to steer tasks related to a distributed decision making process for probe activation. In this context, a P2P management overlay can be used to coordinate the probe activation and to share measurement results among the network devices. For a node to rely on measurement data from a peer to determine which probes to configure, it needs to know which of the peers are best correlated with itself, i.e., which nodes have the most significance in terms of being indicative of service level violations that might be observed by the node itself. We propose an autonomic P2P solution to coordinate the placement of active measurement probes in large-scale networks. The edge nodes of the network cooperate via a P2P management overlay to determine what destinations should be monitored. Each edge node determines autonomously what destinations to probe considering local measurements and measurement data from other edge nodes. The measurements considered are traffic information from passive measurement results and past service level measurement results from active measurement results. The proposed solution is evaluated using simulation and the results show its feasibility and interesting features.