Model-Free IRL Using Maximum Likelihood Estimation

Reinforcement learning (RL) provides a computational model to animal’s autonomous acquisition of behaviors even in an uncertain environment. Inverse reinforcement learning (IRL) is its opposite; given a history of behaviors of an agent, IRL attempts to determine the unknown characteristics, like a reward function, of the agent. Conventional IRL methods usually assume the agent has taken a stationary policy that is optimal in the environment. However, real RL agents do not necessarily take stationary policy, because they are often on the way of adapting to their own environments. Especially when facing an uncertain environment, an intelligent agent should take a mixed (or switching) strategy consisting of an exploitation that is best at the current situation and an exploration to resolve the environmental uncertainty. In this study, we propose a new IRL method that can identify both of a non-stationary policy and a fixed but unknown reward function, based on the behavioral history of a learning agent; in particular, we estimate a change point of the behavior policy from an exploratory one in the agent’s early stage of the learning and an exploitative one in its later learning stage. When applied to a computer simulation during a simple maze task of an agent, our method could identify the change point of the behavior policy and the fixed reward function, only from the agent’s history of behaviors.

Robots had a great impact on the manufacturing industry ever since the early seventies when companies such as KUKA and ABB started deploying their first industrial robots. These robots merely performed very specific tasks in specific ways within well-defined environments. Still, they proved to be very useful as they could exceed human performance at these tasks. However, in order to enable robots to enter our daily life, they need to become more versatile and need to operate in much less structured environments. This thesis is partly devoted to stretching these limitations by means of learning, namely imitation learning (IL) and inverse reinforcement learning (IRL). Reinforcement learning (RL) is a powerful approach to enable robots to solve a task in an unknown environment. The practitioner describes a desired behavior by specifying a reward function and the robot autonomously interacts with the environment in order to find a control policy that generates high accumulated reward. However, RL is not suitable for teaching new tasks by non-experts because specifying appropriate reward functions can be difficult. Demonstrating the desired behavior is often easier for non-experts. Imitation learning can be used in order to enable the robot to reproduce the demonstrations. However, without explicitly inferring and modeling the intentions of the demonstrations, it can become difficult to solve the task for unseen situations. Inverse reinforcement learning (IRL) therefore aims to infer a reward function from the demonstrations, such that optimizing this reward function yields the desired behavior even for different situations. This thesis introduces a unifying approach to solve the inverse reinforcement learning problem in the same way as the reinforcement learning problem. This is achieved by framing both problems as information projection problems, i.e., we strive to minimize the relative entropy between a probabilistic model of the robot behavior and a given desired distribution. Furthermore, a trust region on the robot behavior is used to stabilize the optimization. For inverse reinforcement learning, the desired distribution is implicitly given by the expert demonstrations. The resulting optimization can be efficiently solved using state-of-the-art reinforcement learning methods. For reinforcement learning, the log-likelihood of the desired distribution is given by the reward function. The resulting optimization problem corresponds to a standard reinforcement learning formulation, except for an additional objective of maximizing the entropy of the robot behavior. This entropy objective adds little overhead to the optimization, but can lead to better exploration and more diversified policies. Trust-region I-projections are not only useful for training robots, but can also be applied to other machine learning problems. I-projections are typically used for variational inference, in order to approximate an intractable distribution by a simpler model. However, the resulting optimization problems are usually optimized based on stochastic gradient descent which often suffers from high variance in the gradient estimates. As trust-region I-projections where shown to be effective for reinforcement learning and inverse reinforcement learning, this thesis also explores their use for variational inference. More specifically, trust-region I-projections are investigated for the problem of approximating an intractable distribution by a Gaussian mixture model (GMM) with an adaptive number of components. GMMs are highly desirable for variational inference because they can yield arbitrary accurate approximations while inference from GMMs is still relatively cheap. In order to make learning the GMM feasible, we derive a lower bound that enables us to decompose the objective function. The optimization can then be performed by iteratively updating individual components using a technique from reinforcement learning. The resulting method is capable of learning approximations of significantly higher quality than existing variational inference methods. Due to the similarity of the underlying optimization problems, the insights gained from our variational inference method are also useful for IL and IRL. Namely, a similar lower bound can be applied also for the I-projection formulation of imitation learning. However, whereas for variational inference the lower bound serves to decompose the objective function, for imitation learning it allows us to provide a reward signal to the robot that does not depend on its behavior. Compared to reward functions that are relative to the current behavior of the robot---which are typical for popular adversarial methods---behavior-independent reward functions have the advantages that we can show convergence even for greedy optimization. Furthermore, behavior-independent reward functions solve the inverse reinforcement learning problem, thereby closing the gap between imitation learning and IRL. However, algorithms derived from our non-adversarial formulation are actually very similar to existing AIL methods, and we can even show that adversarial inverse reinforcement learning (AIRL) is indeed an instance of our formulation. AIRL was derived from an adversarial formulation, and we point out several problems of that derivation. In contrast, we show that AIRL can be straightforwardly derived from out non-adversarial formulation. Furthermore, we demonstrate that the non-adversarial formulation can be also used to derive novel algorithms by presenting a non-adversarial method for offline imitation learning.

Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning

We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risksensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology and economics. We propose a gradient-based inverse reinforcement learning algorithm that minimizes a loss function defined on the observed behavior. We demonstrate the performance of the proposed technique on two examples, the first of which is the canonical Grid World example and the second of which is an MDP modeling passengers' decisions regarding ride-sharing. In the latter, we use pricing and travel time data from a ride-sharing company to construct the transition probabilities and rewards of the MDP.

This article proposes an online data-based inverse reinforcement learning (IRL) scheme to solve optimal control problem for nonlinear systems with control constraints, in which the unknown reward functions are recovered based on the systems’ demonstrated state and input data. To deal with control constraint, we introduce a saturation function to formulate the original constrained optimal control problem into a new unconstrained optimal control problem. Then a data-based identifier using neural network (NN) approximation technique is designed to estimate the system’s dynamics. Subsequently, we develop a data-based IRL approach to learn the unknown reward function and establish the weight tuning law of the value function and reward function using demonstrated state and input data. The proof of the uniform boundedness of the weight estimation error is presented. A simulation example is provided to verify the effectiveness of the proposed approach.

Inverse reinforcement learning (IRL) addresses the problem of recovering the unknown reward function for a given Markov decision problem (MDP) given the corresponding optimal policy or a perturbed version thereof. This paper studies the space of possible solutions to the general IRL problem, when the agent is provided with incomplete/imperfect information regarding the optimal policy for the MDP whose reward must be estimated. We focus on scenarios with finite state-action spaces and discuss the constraints imposed on the set of possible solutions when the agent is provided with (i) perturbed policies; (ii) optimal policies; and (iii) incomplete policies. We discuss previous works on IRL in light of our analysis and show that, with our characterization of the solution space, it is possible to determine non-trivial closed-form solutions for the IRL problem. We also discuss several other interesting aspects of the IRL problem that stem from our analysis.

Learning from demonstration, or imitation learning, is the process of learning to act in an environment from examples provided by a teacher. Inverse reinforcement learning (IRL) is a specific form of learning from demonstration that attempts to estimate the reward function of a Markov decision process from examples provided by the teacher. The reward function is often considered the most succinct description of a task. In simple applications, the reward function may be known or easily derived from properties of the system and hard coded into the learning process. However, in complex applications, this may not be possible, and it may be easier to learn the reward function by observing the actions of the teacher. This paper provides a comprehensive survey of the literature on IRL. This survey outlines the differences between IRL and two similar methods - apprenticeship learning and inverse optimal control. Further, this survey organizes the IRL literature based on the principal method, describes applications of IRL algorithms, and provides areas of future research.

Reinforcement learning (RL)-based brain machine interfaces (BMIs) provide a promising solution for paralyzed people. Enhancing the decoding performance of RL-based BMIs relies on the design of effective reward signals. Inverse reinforcement learning (IRL) offers an approach to infer subjects' own evaluation from the observed behavior. However, applying IRL to extract reward information in complex BMI tasks requires consideration of the dynamics of subjects' goal during the control process. This dynamic nature of subjects' evaluation requires the IRL method to be able to estimate a time varying reward function. Previous IRL methods applied in BMI systems only estimated a static reward function. Existing IRL algorithms for dynamic reward estimation employ optimization methods to approximate the reward map for each state at each time, which demands substantial amounts of data to achieve convergence. In this paper, we propose a dynamic IRL method to estimate the feedback-driven reward of subjects during BMI tasks. We utilize a state-observation model to continuously infer the reward value for each state, with sensory feedback serving as the external input to model the transition process of the reward. We evaluate our proposed method on a simulated BMI fetch task, which is a multistep task with a time varying reward function. Our method demonstrates improved reward estimation close to the ground truth value, and it significantly outperforms the existing dynamic IRL method when the map size exceeds 25(p<0.01). These preliminary results suggests that the dynamic IRL method for feedback-driven reward estimation holds potential for improving the design of RL-based BMIs.

Rewards are critical hyperparameters in reinforcement learning (RL), since in most cases different reward values will lead to greatly different performance. Due to their commercial value, RL rewards become the target of reverse engineering by the inverse reinforcement learning (IRL) algorithm family. Existing efforts typically utilize two metrics to measure the IRL performance: the expected value difference and the mean reward loss, which we call them EVD and MRL respectively. Unfortunately, in some cases, EVD and MRL can give completely opposite results, due to MRL focusing on whole state-space rewards while EVD only considering partly sampled rewards. Such situation naturally rises to one fundamental question: whether current metrics and assessment are sufficient and accurate for more general use. Thus, in this paper, based on the metric called normalized mutual information of reward clusters (C-NMI) we propose a novel IRL assessment; we aim to fill this research gap by considering a middle-granularity state space between the entire state space and the specific sampling space. We utilize the agglomerative nesting algorithm (AGNES) to control dynamical C-NMI computing via a 4-order tensor model with injected manipulated trajectories. With such a model, we can uniformly capture different-dimension values of MRL, EVD, and C-NMI, and perform more comprehensive and accurate assessment and analyses. Extensive experiments on several mainstream IRLs are experimented in Object World, hence revealing that the assessing accuracy of our method increases 110.13&#x0025; and 116.59&#x0025; respectively when compared with the EVD and MRL. Meanwhile, C-NMI is more robust than EVD and MRL under different demonstrations.

Reinforcement Learning (RL) algorithms depend on well-defined reward functions for policy optimization. Designing such functions is a complex task, even for domain experts. However, valid task demonstrations can still be collected from moderately skilled individuals. This motivates the use of methods such as Imitation Learning (IL) and Inverse Reinforcement Learning (IRL), where an expert provides demonstrations, allowing the algorithm to infer a policy or reward function that aligns with observed behavior. A common assumption in IRL is that demonstrations come from highly skilled experts. While some studies have explored the impact of suboptimal demonstrators, the influence of intentionally malicious demonstrations remains underexplored. This study introduces an adversarial demonstrator framework that systematically perturbs a subset of demonstrations to manipulate the reward function inferred by an IRL algorithm. Additionally, it quantifies the impact of such adversarial manipulations on the learned policy. Our results show that simply altering 10% of the demonstrations can lead the IRL algorithm to learn a faulty reward function, ultimately degrading the performance of the trained policy by up to three times the effect of adding 10% random trajectories to the result.

Energy-efficient and damage-recovery slithering gait design for a snake-like robot based on reinforcement learning and inverse reinforcement learning

Inverse reinforcement learning (IRL) is the problem of learning the preferences of an agent from observing its behavior on a task. While this problem is witnessing sustained attention, the related problem of online IRL - where the observations are incrementally accrued, yet the real-time demands of the application often prohibit a full rerun of an IRL method - has received much less attention. We introduce a formal framework for online IRL, called incremental IRL (I2RL), and a new method that advances maximum entropy IRL with hidden variables, to this setting. Our formal analysis shows that the new method has a monotonically improving performance with more demonstration data, as well as probabilistically bounded error, both under full and partial observability. Experiments in a simulated robotic application, which involves learning under occlusion, show the significantly improved performance of online IRL as compared to both batch IRL and an online imitation learning method.

ABSTRACTIn this article, the initial excitation‐based inverse reinforcement learning methods are presented for continuous‐time linear non‐zero‐sum games. The policy iteration and value iteration algorithms are presented for the inverse reinforcement learning problems, and an online‐verifiable initial excitation condition is given to guarantee the convergence of the presented algorithms. Comparing with the traditional inverse reinforcement learning algorithms for linear non‐zero‐sum games, the presented algorithms relax the requirement on data‐storage mechanism. Furthermore, the requirement on the initial stabilizing state feedback matrices is relaxed in the presented initial excitation‐based value iteration algorithm. The properties of the presented initial excitation‐based policy iteration and value iteration algorithms are analyzed. Simulation results show the efficiency of the presented algorithms.

This chapter provides an overview of the most popular methods of inverse reinforcement learning (IRL) and imitation learning (IL). These methods solve the problem of optimal control in a data-driven way, similarly to reinforcement learning, however with the critical difference that now rewards are not observed. The problem is rather to learn the reward function from the observed behavior of an agent. As behavioral data without rewards is widely available, the problem of learning from such data is certainly very interesting. This chapter provides a moderate-level technical description of the most promising IRL methods, equips the reader with sufficient knowledge to understand and follow the current literature on IRL, and presents examples that use simple simulated environments to evaluate how these methods perform when the “ground-truth” rewards are known. We then present use cases for IRL in quantitative finance which include applications in trading strategy identification, sentiment-based trading, option pricing, inference of portfolio investors, and market modeling.

Model-free episodic reinforcement learning problems define the environment reward with functions that often provide only sparse information throughout the task. Consequently, agents are not given enough feedback about the fitness of their actions until the task ends with success or failure. Previous work addresses this problem with reward shaping. In this paper we introduce a novel approach to improve model-free reinforcement learning agents' performance with a three step approach. Specifically, we collect demonstration data, use the data to recover a linear function using inverse reinforcement learning and we use the recovered function for potential-based reward shaping. Our approach is model-free and scalable to high dimensional domains. To show the scalability of our approach we present two sets of experiments in a two dimensional Maze domain, and the 27 dimensional Mario AI domain. We compare the performance of our algorithm to previously introduced reinforcement learning from demonstration algorithms. Our experiments show that our approach outperforms the state-of-the-art in cumulative reward, learning rate and asymptotic performance.