Natural Actor-Critic

The parameter space of a statistical learning machine has a Riemannian metric structure in terms of its objective function. [1] Amari proposed the concept of that takes the Riemannian metric of the parameter space into account. Kakade [2] applied it to policy gradient reinforcement learning, called a natural policy gradient (NPG). Although NPGs evidently depend on the underlying Riemannian metrics, careful attention was not paid to the alternative choice of the metric in previous studies. In this paper, we propose a Riemannian metric for the joint distribution of the state-action, which is directly linked with the average reward, and derive a new NPG named Natural State-action Gradient(NSG). Then, we prove that NSG can be computed by fitting a certain linear model into the immediate reward function. In numerical experiments, we verify that the NSG learning can handle MDPs with a large number of states, for which the performances of the existing (N)PG methods degrade.

Theoretical Findings Validate Historical Data Reuse for Improved Policy Optimization A new study, “Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate” by Yifan Lin, Yuhao Wang, and Enlu Zhou, explores an advanced approach to reinforcement learning. The research focuses on improving policy optimization by reusing historical trajectories through importance sampling in natural policy gradient methods. The authors rigorously analyze the convergence properties of this approach and demonstrate that reusing past data enhances convergence rates while maintaining theoretical guarantees. Their findings have practical implications for applications where data collection is costly or limited, such as robotics and autonomous systems. By integrating these insights into policy optimization frameworks, the study provides a valuable contribution to the field of reinforcement learning.

This paper presents a model-free approximation for the Hessian of the performance of deterministic policies to use in the context of Reinforcement Learning based on Quasi-Newton steps in the policy parameters. We show that the approximate Hessian converges to the exact Hessian at the optimal policy, and allows for a superlinear convergence in the learning, provided that the policy parametrization is rich. The natural policy gradient method can be interpreted as a particular case of the proposed method. We analytically verify the formulation in a simple linear case and compare the convergence of the proposed method with the natural policy gradient in a nonlinear example.

Policy gradient methods have been frequently applied to problems in control and reinforcement learning with great success, yet existing convergence analysis still relies on non-intuitive, impractical and often opaque conditions. In particular, existing rates are achieved in limited settings, under strict regularity conditions. In this work, we establish explicit convergence rates of policy gradient methods, extending the convergence regime to weakly smooth policy classes with L2 integrable gradient. We provide intuitive examples to illustrate the insight behind these new conditions. Notably, our analysis also shows that convergence rates are achievable for both the standard policy gradient and the natural policy gradient algorithms under these assumptions. Lastly we provide performance guarantees for the converged policies.

A large class of sequential decision-making problems under uncertainty, with broad applications from preventive maintenance to event-triggered control can be modeled in the framework of semi-Markov decision processes (SMDPs). Unlike Markov decision processes (MDPs), SMDPs are underexplored in the online and reinforcement learning (RL) settings. In this paper, we extend the well-known deterministic policy gradient (DPG) theorem in MDPs to SMDPs under average-reward criterion. The existing stochastic policy gradient methods not only require, in general, a large number of samples for training, but they also suffer from high variance in the gradient estimation when applied to problems with deterministic optimal policy. Our DPG method can potentially remedy these issues. On the basis of this method and depending on the choice of a critic, different actor–critic algorithms can easily be developed in the RL setup. We present two example actor–critic algorithms. Both algorithms employ our developed policy gradient theorem for their actors, but use two different critics; one uses a simple SARSA update while the other one uses the same on-policy update but with compatible function approximators. We demonstrate the efficacy of our method both mathematically and via simulations.

In 5G, eMBB services are defined to support high data rate, while URLLC services focus on low latency and high reliability. Multiplexing these two services on the same wireless radio frequency leads to a challenging radio resource allocation problem due to their heterogeneous requirements. In this paper, we formulate this problem as two non-linear programming non-convex optimization subproblems, aiming to maximize the average data rate of all eMBB services while satisfying the delay constraint of each URLLC service. We propose an event-driven deep reinforcement learning (DRL) based resource allocation mechanism (EDRL-RAM), which includes two schedulers: eMBB scheduler and URLLC scheduler to achieve long-term optimization of eMBB and URLLC performance. The eMBB scheduler will intelligently allocate resource for each incoming eMBB event, and the URLLC scheduler will intelligently distribute the incoming URLLC event during the ongoing transmissions of eMBB services. The proposed EDRL-RAM makes full use of four different DRL techniques to deal with stochastic event arrivals and network conditions, namely Policy Gradient (PG), Deep Q-learning Network (DQN), Advantage Actor Critic (A2C), and Deep Deterministic Policy Gradient (DDPG), in both eMBB and URLLC schedulers. The simulation results show that in our proposed EDRL-RAM, the order of data rate and delay performance is DDPG, A2C, DQN, and PG. The data rate and delay performance of the proposed EDRL-RAM utilizing any of the four DRL techniques are better than SAFE-TS, which is the best available related work.

Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. In this paper, we classify RL into direct and indirect RL according to how they seek the optimal policy of the Markov decision process problem. The former solves the optimal policy by directly maximizing an objective function using gradient descent methods, in which the objective function is usually the expectation of accumulative future rewards. The latter indirectly finds the optimal policy by solving the Bellman equation, which is the sufficient and necessary condition from Bellman's principle of optimality. We study policy gradient forms of direct and indirect RL and show that both of them can derive the actor-critic architecture and can be unified into a policy gradient with the approximate value function and the stationary state distribution, revealing the equivalence of direct and indirect RL. We employ a Gridworld task to verify the influence of different forms of policy gradient, suggesting their differences and relationships experimentally. Finally, we classify current mainstream RL algorithms using the direct and indirect taxonomy, together with other ones including value-based and policy-based, model-based and model-free.

Reinforcement learning offers various gradient-based algorithms to tackle model-free optimization problems. However, a major drawback of such techniques is premature convergence due to a lack of exploration. This paper reviews several Evolution Strategies and policy gradient algorithms using the natural gradient in order to reach better convergence. By incorporating knowledge of second-order derivatives, the natural gradient prevents premature convergence. We will present benchmarks on the Trust Region Policy Optimization (TRPO) as a natural policy gradient algorithm and on Separable Natural Evolution Strategies (sNES) and compare results and implementations of these two algorithms. Our experience has shown that sNES performs better in most benchmarks while being easier to handle due to having less hyperparameters.

While the recent advanced military operational concept requires an intelligent support of command and control, Reinforcement Learning (RL) has not been actively studied in the military domain. This study points out the limitations of RL for military applications from literature review and aims at improving the understanding of RL for military decision support under the limitations. Most of all, the black box characteristic of Deep RL makes the internal process difficult to understand in addition to complex simulation tools. A scalable weapon selection RL framework is built which can be solved either by a tabular form or a neural network form. The transition of the Deep Q-Network (DQN) solution to the tabular form makes it easier to compare the result to the Q-learning solution. Furthermore, rather than using one or two RL models selectively as before, RL models are divided as an actor and a critic, and systematically compared. A random agent, Q-learning and DQN agents as a critic, a Policy Gradient (PG) agent as an actor, Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) agents as an actor-critic approach are designed, trained, and tested. The performance results show that the trained DQN and PPO agents are the best decision supporter candidates for the weapon selection RL framework.

SummaryOne desired aspect of microservice architecture is the ability to self‐adapt its own architecture and behavior in response to changes in the operational environment. To achieve the desired high levels of self‐adaptability, this research implements distributed microservice architecture model running a swarm cluster, as informed by the Monitor, Analyze, Plan, and Execute over a shared Knowledge (MAPE‐K) model. The proposed architecture employs multiadaptation agents supported by a centralized controller, which can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q‐learning network (DRQN). It is argued that such integration between DRQN and Markov decision process (MDP) agents in a MAPE‐K model offers distributed microservice architecture with self‐adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resource overprovisioning and thrashing. The performance of DRQN is evaluated against deep Q‐learning and policy gradient algorithms, including (1) a deep Q‐learning network (DQN), (2) a dueling DQN (DDQN), (3) a policy gradient neural network, and (4) deep deterministic policy gradient. The DRQN implementation in this paper manages to outperform the aforementioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training time. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services‐oriented architecture and offers better performance than other dynamic decision‐making algorithms.

Ant Colony Optimization (ACO) has served as a widely-utilized metaheuristic algorithm for decades for solving combinatorial optimization problems. Since its initial construction, ACO has seen a wide variety of modifications and connections to Reinforcement Learning (RL). Substantial parallels can be seen as early as 1995 with Ant-Q's relationship with Q-learning, through 2022 with ADACO's connection with Policy Gradient. In this work, we describe ACO, more specifically the Stochastic Gradient Descent ACO algorithm (ACOSGD), explicitly as an off-policy Policy Gradient (PG) method. We also incorporate experience replay into several ACO algorithm variants, including AS, MaxMin-ACO, ACOSGD, ADACO, and our two policy gradient-based versions: PGACO and PPOACO, drawing the connection to elitist ACO strategies. We show that our implementation of PG in ACO with experience replay and a baselined reward update strategy applied to eight TSP problems of varying sizes performs competitively with both fundamental ACO and SGD-based ACO versions. We also show that the replay buffer seems to unilaterally improve the performance of ACO algorithms through an ablation study.

In numerous reinforcement learning (RL) problems involving safety-critical systems, a key challenge lies in balancing multiple objectives while simultaneously meeting all stringent safety constraints. To tackle this issue, we propose a primal-based framework that orchestrates policy optimization between multi-objective learning and constraint adherence. Our method employs a novel natural policy gradient manipulation method to optimize multiple RL objectives and overcome conflicting gradients between different objectives, since the simple weighted average gradient direction may not be beneficial for specific objectives due to misaligned gradients of different objectives. When there is a violation of a hard constraint, our algorithm steps in to rectify the policy to minimize this violation. Particularly, We establish theoretical convergence and constraint violation guarantees, and our proposed method also outperforms prior state-of-the-art methods on challenging safe multi-objective RL tasks.

Artificial intelligence, fueled by machine learning and deep learning techniques, is revolutionizing various domains. Reinforcement learning (RL) stands out as a potent method for training agents to navigate complex environments and make informed decisions. Our focus is on applying RL techniques, specifically Convolutional Neural Networks (CNNs) combined with policy gradient methods, to enhance the gameplay experience of Khmer chess. Our goal is to surpass the performance of traditional chess engines. The system employs deep neural networks to train AI agents, enabling self-play iterations for strategy refinement. Specifically, we utilize RL technology to iteratively enhance game strategies based on self-matching results, ultimately improving the system's chess proficiency. Our approach entails developing a CNN-based RL system tailored for Khmer chess, encompassing strategies, value evaluation mechanisms, and rule adaptations specific to the game. We utilize deep neural networks to facilitate agent training through self-play iterations, leveraging RL techniques for continual strategy refinement. To enhance training efficiency, we introduce a segmentation method for Khmer chess stages, optimizing the neural network's learning process by mapping game situations to optimal actions based on cumulative rewards. Furthermore, we integrate RL principles to guide action selection towards maximizing reward values, employing Deep Q-Learning with policy gradient for optimal decision-making. With the experimental validation demonstrates the efficacy of our CNN-based RL system in enhancing Khmer chess gameplay. The system exhibits self-improvement, adaptability, and human-like gameplay characteristics, enriching player experience and entertainment value. Moreover, the proposed approach showcases improved training efficiency compared to conventional RL-based chess systems, highlighting its efficacy and scalability for AI-driven game enhancements.

We study the global convergence of a Fisher–Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action spaces. The flow is a continuous-time analogue of a policy mirror descent method. We establish the global well-posedness of the gradient flow and demonstrate its exponential convergence to the optimal policy. Moreover, we prove the flow is stable with respect to gradient evaluation, offering insights into the performance of a natural policy gradient flow with log-linear policy parameterisation. To overcome challenges stemming from the lack of the convexity of the objective function and the discontinuity arising from the entropy regulariser, we leverage the performance difference lemma and the duality relationship between the gradient and mirror descent flows. Our analysis provides a theoretical foundation for developing various discrete policy gradient algorithms.

We introduce a novel framework to account for sensitivity to rewards uncertainty in sequential decision-making problems. While risk-sensitive formulations for Markov decision processes studied so far focus on the distribution of the cumulative reward as a whole, we aim at learning policies sensitive to the uncertain/stochastic nature of the rewards, which has the advantage of being conceptually more meaningful in some cases. To this end, we present a new decomposition of the randomness contained in the cumulative reward based on the Doob decomposition of a stochastic process, and introduce a new conceptual tool - the \textit{chaotic variation} - which can rigorously be interpreted as the risk measure of the martingale component associated to the cumulative reward process. We innovate on the reinforcement learning side by incorporating this new risk-sensitive approach into model-free algorithms, both policy gradient and value function based, and illustrate its relevance on grid world and portfolio optimization problems.