Out-of-Distribution Detection for Reinforcement Learning Agents with Probabilistic Dynamics Models

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

In computer science, reinforcement learning is a powerful framework with which artificial agents can learn to maximize their performance for any given Markov decision process (MDP). Advances over the last decade, in combination with deep neural networks, have enjoyed performance advantages over humans in many difficult task settings. However, such frameworks perform far less favorably when evaluated in their ability to generalize or transfer representations across different tasks. Existing algorithms that facilitate transfer typically are limited to cases in which the transition function or the optimal policy is portable to new contexts, but achieving “deep transfer” characteristic of human behavior has been elusive. Such transfer typically requires discovery of abstractions that permit analogical reuse of previously learned representations to superficially distinct tasks. Here, we demonstrate that abstractions that minimize error in predictions of reward outcomes generalize across tasks with different transition and reward functions. Such reward-predictive representations compress the state space of a task into a lower dimensional representation by combining states that are equivalent in terms of both the transition and reward functions. Because only state equivalences are considered, the resulting state representation is not tied to the transition and reward functions themselves and thus generalizes across tasks with different reward and transition functions. These results contrast with those using abstractions that myopically maximize reward in any given MDP and motivate further experiments in humans and animals to investigate if neural and cognitive systems involved in state representation perform abstractions that facilitate such equivalence relations.

For the task under Markov Decision Process, this paper presents a novel multiagent Reinforcement Learning (RL) with perception and conversion action mechanism that learning agents observe adversary agent and convert adversarial action to learning agents' corresponding action as observing state variation incurred by the adversary agent in the task environment during learning processes. Meanwhile, this paper surveys inexpensive communication ways among learning agents utilizing both the direct communication and the indirect media communication to realize agents' cooperation. The direct communication is realized by sharing sensation; the indirect media communication is realized by updating reinforcement values on the common environment observation. Then, a multiagent RL algorithm, Q-ac multiagent RL method, is proposed. By perception and conversion action, the learning agents extend learning episodes and derive more observation by less action. The direct communication enhances agents' observation ability to the environment, and the indirect media communication improves agents' ability deriving the optimal action policy. The simulation results on hunter game demonstrate the efficiency of the proposed method.

Explainable reinforcement learning (XRL) is an emerging subfield of explainable machine learning that has attracted considerable attention in recent years. The goal of XRL is to elucidate the decision-making process of reinforcement learning (RL) agents in sequential decision-making settings. Equipped with this information, practitioners can better understand important questions about RL agents (especially those deployed in the real world), such as what the agents will do and why. Despite increased interest, there exists a gap in the literature for organizing the plethora of papers—especially in a way that centers the sequential decision-making nature of the problem. In this survey, we propose a novel taxonomy for organizing the XRL literature that prioritizes the RL setting. We propose three high-level categories: feature importance, learning process and Markov decision process, and policy-level. We overview techniques according to this taxonomy, highlighting challenges and opportunities for future work. We conclude by using these gaps to motivate and outline a roadmap for future work.

Reinforcement learning (RL) is an efficient learning method for Markov decision processes (MDPs); ant colony system (ACS) is an efficient method for solving combinatorial optimization problems. Based on the update policy of reinforcement values in RL and the cooperating method of the indirect media communication in ACS, this paper proposes the Q-ACS multi-agent cooperating learning method for the learning agents to share episodes beneficial to the exploitation of the accumulated knowledge and to utilize the learned reinforcement values efficiently. Further, taking the visited times into account, this paper proposes the T-ACS multi-agent learning method that the learning agents share better policies beneficial to the exploration during agent's learning processes. Meanwhile, in the light of the indirect media communication among heterogeneous multi-agents, this paper presents a heterogeneous multi-agent RL method, the D-ACS. The agents in our methods are given a simply cooperating way exchanging information in the form of reinforcement values updated in the common model of all agents. Owning the advantages of exploring the unknown environment actively and exploiting learned knowledge effectively, the proposed methods are able to solve both MDPs and combinatorial optimization problems effectively. To results of simulations on the hunter game and the traveling salesman problem, this paper discusses the role of the indirect media communication on the multi-agent cooperation learning system and analyzes its efficiency. The results of experiments also demonstrate that our methods perform competitively with representative methods on each domain respectively.

Financial assets are a type of source of income for the retired. There are problems regarding decisions in managing financial assets. One way to manage financial portfolios is by using reinforcement learning. Reinforcement learning is a machine learning method that uses rewards and punishments as feedback for various learning agents to maximize cumulative rewards. This method requires a Markov Decision Process (MDP) environment to describe the problem to the learning agent. We will use Actor Critic using Kronecker-Factored Trust Region (ACKTR), which reduces the amount of sampling needed for learning. ACKTR excels in sample reduction by using Kronecker-Factored Approximated Curvature (K-FAC) and trust region in approximating the natural gradient for an agent's learning. In this research, we use an MDP that describes personal retirement portfolio with securities, before implementing ACKTR and analyzing it. MDP used in this research is constructed as model-free MDP and total reward (cumulative reward) is influenced by discount rate and action. Results of this implementation shows that lower discount rates are superior to higher discount rates in average of cumulative rewards, and higher discount rates lead to riskier agents.

Transfer learning is the one of methods to improve the efficiency of training for reinforcement learning (RL) agents. That is the method of transferring knowledge which is acquired in the past tasks or given by experts to similar tasks. However, if the RL agents don't retain knowledge that is effective for improving learning speed or quality, there is a problem in that it causes negative transfer, which reduces learning efficiency. This study proposes a method for deleting knowledge which are less the number of state observation through trial and error by RL agents. Our proposed method was applied to path planning for a two-wheeled mobile robot to verify its effectiveness, and it was confirmed that the proposed method reduced the amount of knowledge by 98% and the learning time by 32%.

Safety is still one of the major research challenges in reinforcement learning (RL). In this paper, we address the problem of how to avoid safety violations of RL agents during exploration in probabilistic and partially unknown environments. Our approach combines automata learning for Markov Decision Processes (MDPs) and shield synthesis in an iterative approach. Initially, the MDP representing the environment is unknown. The agent starts exploring the environment and collects traces. From the collected traces, we passively learn MDPs that abstractly represent the safety-relevant aspects of the environment. Given a learned MDP and a safety specification, we construct a shield. For each state-action pair within a learned MDP, the shield computes exact probabilities on how likely it is that executing the action results in violating the specification from the current state within the next k steps. After the shield is constructed, the shield is used during runtime and blocks any actions that induce a too large risk from the agent. The shielded agent continues to explore the environment and collects new data on the environment. Iteratively, we use the collected data to learn new MDPs with higher accuracy, resulting in turn in shields able to prevent more safety violations. We implemented our approach and present a detailed case study of a Q-learning agent exploring slippery Gridworlds. In our experiments, we show that as the agent explores more and more of the environment during training, the improved learned models lead to shields that are able to prevent many safety violations.KeywordsAutomata learningShieldingMarkov Decision Processes

Reinforcement Learning (RL) Markov Decision Processes is studied with an emphasis on the well-studied exploration problem. We first formulate and discuss a definition of “efficient” algorithms that is termed Probably Approximately Correct (PAC) in RL. Next we provide general sufficient conditions for such an algorithm that applies to several different modeling assumptions. The conditions can be used to demonstrate that efficient learning is possible in finite MDPs, with either a model-based or model-free approach, in factored MDPs, and in continuous MDPs with linear dynamics. In the reinforcement-learning (RL) problem (Sutton & Barto 1998), an agent acts in an unknown or incompletely known environment with the goal of maximizing an external reward signal. In the most standard mathematical formulation of the problem, the environment is modeled as a Markov Decision Process (MDP) and the goal of the agent is to obtain near-optimal discounted return. Over the years, many algorithms have been proposed for this problem, but analyses of their performances have been relatively scarce. In fact, until recently, most theoretical guarantees have been that certain algorithms will discover an optimal policy in the limit, after an infinite amount of experience. In contrast, several attempts have been made to study “Probably Approximately Correct” or PAC-MDP algorithms, which exhibit near-optimal behavior in polynomial time and experience. This paper discusses several extensions of those results. We present a theorem that provides sufficient conditions for an algorithm to be PAC-MDP. We examine these conditions and show how they can be applied to prove that efficient learning is possible in three interesting scenerios: finite MDPs (i.e. the “Tabular case”),

Reinforcement Learning (RL) is a machine learning methodology that develops the capability to make sequential decisions in intricate issues using trial-and-error techniques. RL has become increasingly prevalent for decision-making and control tasks in diverse fields such as industrial processes, biochemical systems and energy management. This review paper presents a comprehensive examination of the development, models, algorithms and practical uses of RL, with a specific emphasis on its application in process control. The study examines the fundamental theories, methodology and applications of RL, classifying them into two categories: classical RL such as such as Markov decision processes (MDP) and deep RL viz., actor critic methods. RL is a topic of discussion in multiple process industries, such as industrial chemical process control, biochemical process control, energy systems, wastewater treatment and the oil and gas sector. Nevertheless, the paper also highlights challenges that hinder its larger acceptance, including the requirement for substantial computational resources, the complexity of simulating real-world settings and the challenge of guaranteeing the stability and resilience of RL algorithms in dynamic and unpredictable environments. RL has demonstrated significant promise, but more research is needed to fully integrate it into industrial and environmental systems in order to solve the current challenges. Abbreviations: AC: Actor critic; AI: Artificial intelligence; ANN: Artificial neural networks; A3C: Asynchronous advantage actor critic; CRL : Classical Reinforcement learning; CV : Controlled variable; DDPG : Deep deterministic policy gradient; DQN: Deep Q network; DRL: Deep reinforcement learning; DP: Dynamic programming; FOMDP: Fully observable Markov decision process; GRU: Gated recurrent unit; LQR: Linear quadratic regulator; LSTM: Long short-term memory; ML: Machine learning; MV : Manipulated variable; MC: Monte Carlo; MDP: Markov decision process; MPC: Model predictive controller; MIMO: Multi input multi output; PG: Policy gradient; PID: Proportional integral derivative; PPO: Proximal policy optimisation; RL: Reinforcement learning; PPO: Proximal policy optimisation; SAC: Soft actor critic; SISO: Single input single output; TD: Temporal difference; TRPO: Trust region policy optimisation; TD3: Twin delayed deep deterministic policy gradient.

Trust is a critical factor in human-robot interaction. This study investigates how using human data to train reinforcement learning (RL) agents affects perceived performance and trust. We conducted two experiments: an online study using the video game Seaquest, and a physical robot study involving a wire loop task. In both, RL agents were trained using demonstrations from human users with varying skill levels, as well as without human data. Participants then rated the agents’ performance and how much they trusted them. Our results for the online study indicate that RL agents trained with expert human data received significantly higher trust ratings than other agents, and that trust was higher in replay and long training conditions compared to short and no training. Videos showing beginner players were also trusted, despite lower performance, although these differences did not always reach statistical significance. This highlights that trust is influenced by factors beyond mere performance, suggesting a role for cues such as perceived human involvement. In the physical robot study, our findings indicate that trust was significantly influenced by the type of training data, with data from an experienced human having a positive effect under certain training conditions. Performance ratings were influenced primarily by the training approach, with replay receiving higher ratings than short training. Although trust was notably higher under certain conditions, performance ratings did not always align with trust, indicating a partial dissociation between the two measures. Together, these results suggest that training with high-quality human data can enhance the trustworthiness of RL agents under appropriate training conditions. However, trust depends not only on performance, but also on how human-like an agent's behavior appears. Designing agents for trust-sensitive applications may therefore benefit from aligning behavior with user expectations, rather than optimizing performance alone.

In this article, a policy-iteration-based active disturbance rejection control (ADRC) is proposed for uncertain nonlinear systems to achieve real-time output tracking performance, regardless of the specific relative degree of the system. The approach integrates a partial control input generator with a policy-iteration-based reinforcement learning (RL) agent for degree weight adjustment. The partial control input generator includes each ith order partial control input, which is constructed following the ADRC design framework for an ith order system. The RL agent adjusts the degree weights (its actions) to enhance the dominance of the partial control input corresponding to the unknown relative degree through iterative policy refinement. The RL agent is designed to minimize the quadratic reward as the performance index function while enhancing the influence of the partial control input associated with the correct relative degree via the policy iteration procedure. All signals in the closed-loop system (including the time-varying degree weights) ensure semi-global uniformly ultimately boundness using the Lyapunov stability theorem and the affinely quadratically stable property. Consequently, the degree weight adjustments by the RL agent do not affect the closed-loop stability. The proposed method does not require system dynamics, specific relative degree, external disturbances, and other state variable sensing beyond output sensing. The performance of the proposed method was validated via simulations for two different-order uncertain nonlinear systems and experiments using a permanent magnet synchronous motor testbed.

Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning

RL-VAEGAN: Adversarial defense for reinforcement learning agents via style transfer

The Learning is an indispensable phase in the cognition cycle of cognitive radio network. It corresponds between the executed actions and the estimated rewards. Based on this phase, the agent learns from past experiences to improve his actions in the next interventions. In the literature, there are several methods that treat the artificial learning. Among them, we cite the reinforcement learning that look for the optimal policy, for ensuring the maximum reward.The present work exposes an approach, based on a model of reinforcement learning, namely Markov decision process, to maximize the sum of transfer rates of all secondary users. Such conception defines all notions relative to an environment with finite set of states, including: the agent, all states, the allowed actions with a given state, the obtained reward after the execution of an action and the optimal policy. After the implementation, we remark a correlation between the started policy and the optimal policy, and we improve the performances by referring to a previous work.