Decentralized Reinforcement Learning for the Online Optimization of Distributed Systems

In this paper we show the application of multi-agent modeling and simulation with distributed reinforcement learning to one of the major problems in power system operations, i.e. voltage control. In this research some agents in the power network work together to provide a desirable voltage profile, using a combination of multi-agent system (MAS) technology and some of the reinforcement learning approaches. In this schema, individual agents who are assigned to voltage controller devices in the power system learn from their experiences to control the system voltage, and also cooperate and communicate with each other to satisfy the whole team goals. A detailed evaluation of methods for controlling voltage in power systems, including multi-agent coordination and distributed reinforcement learning (DRL), demonstrates that this framework yields effective plans, good agent coordination, and successful implementation. In the proposed approach, agent development and communication simulation have been carried out in the Java Agent Development (JADE) framework.

Despite recent advances in reinforcement learning (RL), its application in safety critical domains like autonomous vehicles is still challenging. Although punishing RL agents for risky situations can help to learn safe policies, it may also lead to highly conservative behavior. In this paper, we propose a distributional RL framework in order to learn adaptive policies that can tune their level of conservativity at run-time based on the desired comfort and utility. Using a proactive safety verification approach, the proposed framework can guarantee that actions generated from RL are fail-safe according to the worst-case assumptions. Concurrently, the policy is encouraged to minimize safety interference and generate more comfortable behavior. We trained and evaluated the proposed approach and baseline policies using a high level simulator with a variety of randomized scenarios including several corner cases which rarely happen in reality but are very crucial. In light of our experiments, the behavior of policies learned using distributional RL can be adaptive at run-time and robust to the environment uncertainty. Quantitatively, the learned distributional RL agent drives in average 8 seconds faster than the normal DQN policy and requires 83\% less safety interference compared to the rule-based policy with slightly increasing the average crossing time. We also study sensitivity of the learned policy in environments with higher perception noise and show that our algorithm learns policies that can still drive reliable when the perception noise is two times higher than the training configuration for automated merging and crossing at occluded intersections.

In recent years, investment strategies on financial markets using deep learning have attracted a significant amount of research attention. The objective of these studies is to obtain investment action that has a low risk and increases profit. On the other hand, Distributional Reinforcement Learning (DRL) expands the action-value function to a discrete distribution in reinforcement learning, which can control risk. However, DRL has not yet been used to learn investment action. In this study, we construct a low-risk investment trading model using DRL. This model is backtested on Nikkei 225 dataset and compared with Deep Q Network (DQN). We evaluate performance in terms of final asset amounts, their standard deviation, and the Sharpe ratio. The experimental results show that the proposed DRL-based method can learn low-risk actions with increasing profit, outperforming the compared method DQN.

As the demand for Network Function Virtualization accelerates, service providers are expected to advance the way they manage and orchestrate their network services to offer lower latency services to their future users. Modern services require complex data flows between Virtual Network Functions, placed in separate network domains, risking an increase in latency that compromises the offered latency constraints. This shift requires high levels of automation to deal with the scale and load of future networks. In this paper, we formulate the Service Function Chaining (SFC) placement problem and then we tackle it by introducing SCHEMA, a Distributed Reinforcement Learning (RL) algorithm that performs complex SFC orchestration for low latency services. We combine multiple RL agents with a Bidding Mechanism to enable scalability on multi-domain networks. Finally, we use a simulation model to evaluate SCHEMA, and we demonstrate its ability to obtain a 60.54% reduction of average service latency when compared to a centralised RL solution.

Distributional reinforcement learning (DRL) enhances the understanding of the effects of the randomness in the environment by letting agents learn the distribution of a random return, rather than its expected value as in standard reinforcement learning. Meanwhile, a challenge in DRL is that the policy evaluation typically relies on the representation of the return distribution, which needs to be carefully designed. In this paper, we address this challenge for the special class of DRL problems that rely on a discounted linear quadratic regulator (LQR), which we call <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">distributional LQR</i>. Specifically, we provide a closed-form expression for the distribution of the random return, which is applicable for all types of exogenous disturbance as long as it is independent and identically distributed (i.i.d.). We show that the variance of the random return is bounded if the fourth moment of the exogenous disturbance is bounded. Furthermore, we investigate the sensitivity of the return distribution to model perturbations. While the proposed exact return distribution consists of infinitely many random variables, we show that this distribution can be well approximated by a finite number of random variables. The associated approximation error can be analytically bounded under mild assumptions. When the model is unknown, we propose a model-free approach for estimating the return distribution, supported by sample complexity guarantees. Finally, we extend our approach to partially observable linear systems. Numerical experiments are provided to illustrate the theoretical results.

A multi-agent system (MAS) approach on wireless sensor networks (WSNs) comprising sensor-actuator nodes is very promising as it has the potential to tackle the resource constraints inherent in these networks by efficiently coordinating the activities among the nodes. In this paper, we consider the coordinated sensing coverage problem and study the behavior and performance of four distributed reinforcement learning (DRL) algorithms: (i) fully distributed Q-learning, (ii) distributed value function (DVF), (iii) optimistic DRL, and (iv) frequency maximum Q-Iearning (FMQ). We present results from simulation studies and actual implementation of these DRL algorithms on Crossbow Mica2 motes, and compare their performance in terms of incurred communication and computational costs, energy consumption and the achieved level of sensing coverage. Issues such as convergence to local or global optima, as well as speed of convergence are also considered. These implementation results show that the DVF agents outperform other agents in terms of both convergence and energy consumption

In the field of the artificial intelligence, more and more attention has been paid to the reinforcement learning algorithm with the advantage of its self-learning and self-adaptability. With the development of the multiagent theory in distributed artificial intelligence, the distributed reinforcement learning is becoming the focus of this research. A model of the multirobots' team formation is used as the study model to illuminate the high-level behavior control of the robots with the usage of the reinforcement learning. Now, few people apply this way to solve such problem. In the reinforcement learning algorithm explained here, the inside reinforcement signals and outside reinforcement signals are applied to show the interests of the robot and its whole group. The control system of the robot is composed of the high-level behavior control and the low-level action control. With this multilayer control, the task of every part is clear. In the low-level action control, the fuzzy control is used here for the mechanical character of the robot. After using the multilayer architecture and fuzzy control algorithm, the speed of learning and convergence of the reinforcement is faster.

Self-play has been shown to be effective to provide a proper training curriculum for a reinforcement learning agent in competitive multi-agent environments without direct supervision. However, its performance is still unstable for problems with sparse rewards, e.g., the scoring task with goalkeeper for robots in RoboCup soccer. It is challenging to solve these tasks in reinforcement learning, especially for those that require combining high-level actions with flexible control. To address these challenges, we introduce a distributed self-play training framework for an extended proximal policy optimization (PPO) algorithm that learns to act in parameterized action space and plays against a group of opponents, i.e., a league. Experiments on the domain of simulated RoboCup soccer show that, the approach is effective and learns more robust policies against various opponents compared to existing reinforcement learning methods. A demonstration video is available online at https://youtu.be/BuLli1vND4.

Homogeneous Decision Networks for Multi-Agent Formation Control via Distributed Reinforcement Learning

The increasing number of electric vehicles (EVs) necessitates the installation of more charging stations. The challenge of managing these grid-connected charging stations leads to a multi-objective optimal control problem where station profitability, user preferences, grid requirements and stability should be optimized. However, it is challenging to determine the optimal charging/discharging EV schedule, since the controller should exploit fluctuations in the electricity prices, available renewable resources and available stored energy of other vehicles and cope with the uncertainty of EV arrival/departure scheduling. In addition, the growing number of connected vehicles results in a complex state and action vectors, making it difficult for centralized and single-agent controllers to handle the problem. In this paper, we propose a novel Multi-Agent and distributed Reinforcement Learning (MARL) framework that tackles the challenges mentioned above, producing controllers that achieve high performance levels under diverse conditions. In the proposed distributed framework, each charging spot makes its own charging/discharging decisions toward a cumulative cost reduction without sharing any type of private information, such as the arrival/departure time of a vehicle and its state of charge, addressing the problem of cost minimization and user satisfaction. The framework significantly improves the scalability and sample efficiency of the underlying Deep Deterministic Policy Gradient (DDPG) algorithm. Extensive numerical studies and simulations demonstrate the efficacy of the proposed approach compared with Rule-Based Controllers (RBCs) and well-established, state-of-the-art centralized RL (Reinforcement Learning) algorithms, offering performance improvements of up to 25% and 20% in reducing the energy cost and increasing user satisfaction, respectively.

Categorical Distributional Reinforcement Learning (CDRL) has demonstrated superior sample efficiency in learning complex tasks compared to conventional Reinforcement Learning (RL) approaches. However, the practical application of CDRL is encumbered by challenging projection steps, detailed parameter tuning, and domain knowledge. This paper addresses these challenges by introducing a pioneering Continuous Distributional Model-Free RL algorithm tailored for continuous action spaces. The proposed algorithm simplifies the implementation of distributional RL, adopting an actor-critic architecture wherein the critic outputs a continuous probability distribution. Additionally, we propose an ensemble of multiple critics fused through a Kalman fusion mechanism to mitigate overestimation bias. Through a series of experiments, we validate that our proposed method provides a sample-efficient solution for executing complex continuous-control tasks.

This paper presents the implementation of a coordinated decision-making agent for emergency response scenarios. The agent’s implementation uses reinforcement learning (RL). RL is a machine learning technique that enables an agent to learn from experimenting. The agent’s learning is based on rewards, and feedback signals proportional to how good its actions are. The simulation platform used was infrastructure interdependencies simulator, in which, we have tested suitability of the approach in previous studies. In this paper, we have added new features to our previous solution, for enabling faster convergence and distributed processing. These additions include an enhanced reward scheme and a scheduler for orchestrating the distributed training. We include two test cases. The first case is a compact model with four critical infrastructures. In this model, the agent’s training required only 10% of the attempts needed in our previous version. Improvements in convergence come from adding a shaping reward scheme. We trained the agent across 24 simultaneous configurations of our model. The training process elapsed 4 min. The extended case included more infrastructures and a higher level of detail. The dimensionality of the problem grew by a factor of 4000, but the training converged in less episodes. We tested the extended model over 96 parallel instances (potential scenarios) with completion in 2.87 min. The results show a fast and stable convergence. This agent can help during multiple stages of emergency response including real-time situations.

Real-time bidding (RTB) has become the dominant mechanism for programmatic display advertising, and the industry has migrated from second-price to first-price auctions. First-price auctions simplify settlement but fundamentally change the cost dynamics: the bidder pays its own bid, which amplifies overbidding losses, increases spend volatility, and makes smooth budget delivery (pacing) more difficult. Most academic auto-bidding literature assumes second-price payment, and many budget-constrained reinforcement learning (RL) methods optimize only expected performance, without explicit downside-risk control. This paper proposes RA-BCB, a risk-aware budget-constrained auto-bidding framework for first-price RTB. RA-BCB combines (i) a value model (pCTR/pCVR) trained from logged impression–click–conversion chains, (ii) an offline replay auction simulator that re-prices wins using first-price payment, and (iii) a distributional constrained RL agent that optimizes a Conditional-Value-at-Risk (CVaR) objective under a daily budget constraint. The agent acts at a time-aggregated granularity (24 hourly slots), selecting a continuous bid multiplier that scales a base bid derived from predicted value. A dual (Lagrangian) update enforces the expected budget constraint, and an explicit pacing deviation penalty reduces intra-day spend variance. Offline replay experiments on the iPinYou RTB benchmark (nine campaigns with bid, impression, click, and conversion logs) and a semi-synthetic first-price evaluation built from the Criteo Display Ads (Kaggle 2014) click logs show that RA-BCB improves ROI and cost-efficiency while maintaining high budget utilization. Compared with linear bidding b_i = λ·ŷ_i with λ selected on training logs to fully utilize the budget, RA-BCB increases weighted value by 43.1% at 50% budget, reduces eCPC by 30.6%, and improves the 10%-tail ROI (CVaR0.1) by 2.42×, while producing near-linear pacing curves.

In this paper, we consider cyber-physical system (CPS) with multiple remote state estimation under denial-of-service (DoS) attack in infinite time-horizon. The sensors monitor the system and send their local state estimate to remote estimators by choosing the local channels in “State 0” or “State 1”. The aim of sensors is to find policies for choosing local channel in a specific state to transmit message to minimize the total estimation error covariance on account of energy-saving in an infinite time-horizon. The DoS attacker aims to achieve the opposite goal by choosing channels to attack or not. The games between sensors and DoS attacker under two different structures of public information are investigated, that is the open-loop case (where sensors and attacker cannot observe others' behaviors) and the closed-loop case (where sensors and attacker can observe the others' behaviors causally). For the open-loop case with assumption that the DoS attacker can get the information from the remote estimators to the sensors, the distributed reinforcement learning algorithms for sensors and attacker based on local information are proposed to find their Nash equilibrium policies, respectively. Further, we consider in closed loop case that the DoS attacker cannot get the information from the remote estimators to the sensors which leads to asymmetric information between the sensors and attacker. To derive Nash equilibrium policies for sensors and attacker, we convert the original game into a belief-based continuous-state stochastic game. The convergence of distributed reinforcement learning method is proved. Some simulations are presented to demonstrate its effectiveness.

With the breakthrough of AlphaGo, deep reinforcement learning has become a recognized technique for solving sequential decision-making problems. Despite its reputation, data inefficiency caused by its trial and error learning mechanism makes deep reinforcement learning difficult to apply in a wide range of areas. Many methods have been developed for sample efficient deep reinforcement learning, such as environment modelling, experience transfer, and distributed modifications, among which distributed deep reinforcement learning has shown its potential in various applications, such as human-computer gaming and intelligent transportation. In this paper, we conclude the state of this exciting field, by comparing the classical distributed deep reinforcement learning methods and studying important components to achieve efficient distributed learning, covering single player single agent distributed deep reinforcement learning to the most complex multiple players multiple agents distributed deep reinforcement learning. Furthermore, we review recently released toolboxes that help to realize distributed deep reinforcement learning without many modifications of their non-distributed versions. By analysing their strengths and weaknesses, a multi-player multi-agent distributed deep reinforcement learning toolbox is developed and released, which is further validated on Wargame, a complex environment, showing the usability of the proposed toolbox for multiple players and multiple agents distributed deep reinforcement learning under complex games. Finally, we try to point out challenges and future trends, hoping that this brief review can provide a guide or a spark for researchers who are interested in distributed deep reinforcement learning.