Online Inverse Reinforcement Learning Under Occlusion

Robots had a great impact on the manufacturing industry ever since the early seventies when companies such as KUKA and ABB started deploying their first industrial robots. These robots merely performed very specific tasks in specific ways within well-defined environments. Still, they proved to be very useful as they could exceed human performance at these tasks. However, in order to enable robots to enter our daily life, they need to become more versatile and need to operate in much less structured environments. This thesis is partly devoted to stretching these limitations by means of learning, namely imitation learning (IL) and inverse reinforcement learning (IRL). Reinforcement learning (RL) is a powerful approach to enable robots to solve a task in an unknown environment. The practitioner describes a desired behavior by specifying a reward function and the robot autonomously interacts with the environment in order to find a control policy that generates high accumulated reward. However, RL is not suitable for teaching new tasks by non-experts because specifying appropriate reward functions can be difficult. Demonstrating the desired behavior is often easier for non-experts. Imitation learning can be used in order to enable the robot to reproduce the demonstrations. However, without explicitly inferring and modeling the intentions of the demonstrations, it can become difficult to solve the task for unseen situations. Inverse reinforcement learning (IRL) therefore aims to infer a reward function from the demonstrations, such that optimizing this reward function yields the desired behavior even for different situations. This thesis introduces a unifying approach to solve the inverse reinforcement learning problem in the same way as the reinforcement learning problem. This is achieved by framing both problems as information projection problems, i.e., we strive to minimize the relative entropy between a probabilistic model of the robot behavior and a given desired distribution. Furthermore, a trust region on the robot behavior is used to stabilize the optimization. For inverse reinforcement learning, the desired distribution is implicitly given by the expert demonstrations. The resulting optimization can be efficiently solved using state-of-the-art reinforcement learning methods. For reinforcement learning, the log-likelihood of the desired distribution is given by the reward function. The resulting optimization problem corresponds to a standard reinforcement learning formulation, except for an additional objective of maximizing the entropy of the robot behavior. This entropy objective adds little overhead to the optimization, but can lead to better exploration and more diversified policies. Trust-region I-projections are not only useful for training robots, but can also be applied to other machine learning problems. I-projections are typically used for variational inference, in order to approximate an intractable distribution by a simpler model. However, the resulting optimization problems are usually optimized based on stochastic gradient descent which often suffers from high variance in the gradient estimates. As trust-region I-projections where shown to be effective for reinforcement learning and inverse reinforcement learning, this thesis also explores their use for variational inference. More specifically, trust-region I-projections are investigated for the problem of approximating an intractable distribution by a Gaussian mixture model (GMM) with an adaptive number of components. GMMs are highly desirable for variational inference because they can yield arbitrary accurate approximations while inference from GMMs is still relatively cheap. In order to make learning the GMM feasible, we derive a lower bound that enables us to decompose the objective function. The optimization can then be performed by iteratively updating individual components using a technique from reinforcement learning. The resulting method is capable of learning approximations of significantly higher quality than existing variational inference methods. Due to the similarity of the underlying optimization problems, the insights gained from our variational inference method are also useful for IL and IRL. Namely, a similar lower bound can be applied also for the I-projection formulation of imitation learning. However, whereas for variational inference the lower bound serves to decompose the objective function, for imitation learning it allows us to provide a reward signal to the robot that does not depend on its behavior. Compared to reward functions that are relative to the current behavior of the robot---which are typical for popular adversarial methods---behavior-independent reward functions have the advantages that we can show convergence even for greedy optimization. Furthermore, behavior-independent reward functions solve the inverse reinforcement learning problem, thereby closing the gap between imitation learning and IRL. However, algorithms derived from our non-adversarial formulation are actually very similar to existing AIL methods, and we can even show that adversarial inverse reinforcement learning (AIRL) is indeed an instance of our formulation. AIRL was derived from an adversarial formulation, and we point out several problems of that derivation. In contrast, we show that AIRL can be straightforwardly derived from out non-adversarial formulation. Furthermore, we demonstrate that the non-adversarial formulation can be also used to derive novel algorithms by presenting a non-adversarial method for offline imitation learning.

Reinforcement learning (RL) provides a computational model to animal’s autonomous acquisition of behaviors even in an uncertain environment. Inverse reinforcement learning (IRL) is its opposite; given a history of behaviors of an agent, IRL attempts to determine the unknown characteristics, like a reward function, of the agent. Conventional IRL methods usually assume the agent has taken a stationary policy that is optimal in the environment. However, real RL agents do not necessarily take stationary policy, because they are often on the way of adapting to their own environments. Especially when facing an uncertain environment, an intelligent agent should take a mixed (or switching) strategy consisting of an exploitation that is best at the current situation and an exploration to resolve the environmental uncertainty. In this study, we propose a new IRL method that can identify both of a non-stationary policy and a fixed but unknown reward function, based on the behavioral history of a learning agent; in particular, we estimate a change point of the behavior policy from an exploratory one in the agent’s early stage of the learning and an exploitative one in its later learning stage. When applied to a computer simulation during a simple maze task of an agent, our method could identify the change point of the behavior policy and the fixed reward function, only from the agent’s history of behaviors.

We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risksensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology and economics. We propose a gradient-based inverse reinforcement learning algorithm that minimizes a loss function defined on the observed behavior. We demonstrate the performance of the proposed technique on two examples, the first of which is the canonical Grid World example and the second of which is an MDP modeling passengers' decisions regarding ride-sharing. In the latter, we use pricing and travel time data from a ride-sharing company to construct the transition probabilities and rewards of the MDP.

Learning from demonstration, or imitation learning, is the process of learning to act in an environment from examples provided by a teacher. Inverse reinforcement learning (IRL) is a specific form of learning from demonstration that attempts to estimate the reward function of a Markov decision process from examples provided by the teacher. The reward function is often considered the most succinct description of a task. In simple applications, the reward function may be known or easily derived from properties of the system and hard coded into the learning process. However, in complex applications, this may not be possible, and it may be easier to learn the reward function by observing the actions of the teacher. This paper provides a comprehensive survey of the literature on IRL. This survey outlines the differences between IRL and two similar methods - apprenticeship learning and inverse optimal control. Further, this survey organizes the IRL literature based on the principal method, describes applications of IRL algorithms, and provides areas of future research.

Reinforcement learning (RL)-based brain machine interfaces (BMIs) provide a promising solution for paralyzed people. Enhancing the decoding performance of RL-based BMIs relies on the design of effective reward signals. Inverse reinforcement learning (IRL) offers an approach to infer subjects' own evaluation from the observed behavior. However, applying IRL to extract reward information in complex BMI tasks requires consideration of the dynamics of subjects' goal during the control process. This dynamic nature of subjects' evaluation requires the IRL method to be able to estimate a time varying reward function. Previous IRL methods applied in BMI systems only estimated a static reward function. Existing IRL algorithms for dynamic reward estimation employ optimization methods to approximate the reward map for each state at each time, which demands substantial amounts of data to achieve convergence. In this paper, we propose a dynamic IRL method to estimate the feedback-driven reward of subjects during BMI tasks. We utilize a state-observation model to continuously infer the reward value for each state, with sensory feedback serving as the external input to model the transition process of the reward. We evaluate our proposed method on a simulated BMI fetch task, which is a multistep task with a time varying reward function. Our method demonstrates improved reward estimation close to the ground truth value, and it significantly outperforms the existing dynamic IRL method when the map size exceeds 25(p<0.01). These preliminary results suggests that the dynamic IRL method for feedback-driven reward estimation holds potential for improving the design of RL-based BMIs.

We consider the problem of learning the behavior of multiple mobile robots executing fixed trajectories in a common space and possibly interacting with each other in their execution. The mobile robots are observed by a subject robot from a vantage point from which it can observe a portion of their trajectories only. This problem exhibits wide-ranging applications and the specific application we consider here is that of the subject robot who desires to penetrate a simple perimeter patrol by two interacting robots and reach a goal location. Our approach extends single-agent inverse reinforcement learning (IRL) to a multi-robot setting and partial observability, and models the interaction between the mobile robots as equilibrium behavior. IRL provides weights over the features of the robots' reward functions, thereby allowing us to learn their preferences. Subsequently, we derive a Markov decision process based policy for each other robot. We extend a predominant IRL technique and empirically evaluate its performance in our application setting. We show that our approach in the application setting results in significant improvement in the subject's ability to predict the patroller positions at different points in time with a corresponding increase in its successful penetration rate.

Inverse reinforcement learning (IRL) addresses the problem of recovering the unknown reward function for a given Markov decision problem (MDP) given the corresponding optimal policy or a perturbed version thereof. This paper studies the space of possible solutions to the general IRL problem, when the agent is provided with incomplete/imperfect information regarding the optimal policy for the MDP whose reward must be estimated. We focus on scenarios with finite state-action spaces and discuss the constraints imposed on the set of possible solutions when the agent is provided with (i) perturbed policies; (ii) optimal policies; and (iii) incomplete policies. We discuss previous works on IRL in light of our analysis and show that, with our characterization of the solution space, it is possible to determine non-trivial closed-form solutions for the IRL problem. We also discuss several other interesting aspects of the IRL problem that stem from our analysis.

The problem of learning an expert’s unknown reward function using a limited number of demonstrations recorded from the expert’s behavior is investigated in the area of inverse reinforcement learning (IRL). To gain traction in this challenging and underconstrained problem, IRL methods predominantly represent the reward function of the expert as a linear combination of known features. Most of the existing IRL algorithms either assume the availability of a transition function or provide a complex and inefficient approach to learn it. In this paper, we present a model-free approach to IRL, which casts IRL in the maximum likelihood framework. We present modifications of the model-free Q-learning that replace its maximization to allow computing the gradient of the Q-function. We use gradient ascent to update the feature weights to maximize the likelihood of expert’s trajectories. We demonstrate on two problem domains that our approach improves the likelihood compared to previous methods.

This chapter provides an overview of the most popular methods of inverse reinforcement learning (IRL) and imitation learning (IL). These methods solve the problem of optimal control in a data-driven way, similarly to reinforcement learning, however with the critical difference that now rewards are not observed. The problem is rather to learn the reward function from the observed behavior of an agent. As behavioral data without rewards is widely available, the problem of learning from such data is certainly very interesting. This chapter provides a moderate-level technical description of the most promising IRL methods, equips the reader with sufficient knowledge to understand and follow the current literature on IRL, and presents examples that use simple simulated environments to evaluate how these methods perform when the “ground-truth” rewards are known. We then present use cases for IRL in quantitative finance which include applications in trading strategy identification, sentiment-based trading, option pricing, inference of portfolio investors, and market modeling.

ABSTRACTIn this article, the initial excitation‐based inverse reinforcement learning methods are presented for continuous‐time linear non‐zero‐sum games. The policy iteration and value iteration algorithms are presented for the inverse reinforcement learning problems, and an online‐verifiable initial excitation condition is given to guarantee the convergence of the presented algorithms. Comparing with the traditional inverse reinforcement learning algorithms for linear non‐zero‐sum games, the presented algorithms relax the requirement on data‐storage mechanism. Furthermore, the requirement on the initial stabilizing state feedback matrices is relaxed in the presented initial excitation‐based value iteration algorithm. The properties of the presented initial excitation‐based policy iteration and value iteration algorithms are analyzed. Simulation results show the efficiency of the presented algorithms.

**Read paper on the following link:** https://ifaamas.org/Proceedings/aamas2022/pdfs/p253.pdf **Abstract:** The recent mean field game (MFG) formalism has enabled the application of inverse reinforcement learning (IRL) methods in large-scale multi-agent systems, with the goal of inferring reward signals that can explain demonstrated behaviours of large populations. The existing IRL methods for MFGs are built upon reducing an MFG to a Markov decision process (MDP) defined on the collective behaviours and average rewards of the population. However, this paper reveals that the reduction from MFG to MDP holds only for the fully cooperative setting. This flaw invalidates existing IRL methods on MFGs with non-cooperative environments. To measure more general behaviours in large populations, we study the use of individual behaviours to directly infer ground-truth reward functions for MFGs. We propose Mean Field IRL (MFIRL), the first IRL framework for MFGs that can handle both cooperative and non-cooperative environments. Based on this theoretically justified framework, we develop a practical algorithm effective for MFGs with continuous state-action spaces and unknown dynamics. We evaluate MFIRL on both cooperative and mixed cooperative-competitive scenarios with many agents. Results show that MFIRL excels in reward recovery, sample efficiency and robustness to changing dynamics, compared to the method based on the reduction to MDP.

Rewards are critical hyperparameters in reinforcement learning (RL), since in most cases different reward values will lead to greatly different performance. Due to their commercial value, RL rewards become the target of reverse engineering by the inverse reinforcement learning (IRL) algorithm family. Existing efforts typically utilize two metrics to measure the IRL performance: the expected value difference and the mean reward loss, which we call them EVD and MRL respectively. Unfortunately, in some cases, EVD and MRL can give completely opposite results, due to MRL focusing on whole state-space rewards while EVD only considering partly sampled rewards. Such situation naturally rises to one fundamental question: whether current metrics and assessment are sufficient and accurate for more general use. Thus, in this paper, based on the metric called normalized mutual information of reward clusters (C-NMI) we propose a novel IRL assessment; we aim to fill this research gap by considering a middle-granularity state space between the entire state space and the specific sampling space. We utilize the agglomerative nesting algorithm (AGNES) to control dynamical C-NMI computing via a 4-order tensor model with injected manipulated trajectories. With such a model, we can uniformly capture different-dimension values of MRL, EVD, and C-NMI, and perform more comprehensive and accurate assessment and analyses. Extensive experiments on several mainstream IRLs are experimented in Object World, hence revealing that the assessing accuracy of our method increases 110.13&#x0025; and 116.59&#x0025; respectively when compared with the EVD and MRL. Meanwhile, C-NMI is more robust than EVD and MRL under different demonstrations.

Reinforcement learning, characterized by trial-and-error learning and delayed rewards, is central to decision-making processes. Its core component, the reward function, is traditionally handcrafted, but designing these functions is often challenging or impossible in real-world scenarios. Inverse reinforcement learning (IRL) addresses this issue by extracting reward functions from expert demonstrations, facilitating optimal policy derivation and offering a deeper understanding of expert behavior. This comprehensive review focuses on three key aspects: the diverse methodologies employed in IRL, its wide-ranging applications across fields such as robotics, autonomous vehicles, and human intent analysis, and the importance of curated datasets in advancing IRL research. A structured analysis of IRL techniques is provided, applications are categorized by domain, and the role of benchmark datasets in evaluating performance and guiding future developments is emphasized. The unique value of IRL in bridging the gap between human and artificial learning is highlighted, demonstrating its potential to unlock advancements in machine learning, decision making, and explainable AI. By summarizing the current state of IRL research and advocating for future directions, this review serves as a valuable resource for researchers and practitioners seeking to explore and advance the field.

Reinforcement Learning (RL) algorithms depend on well-defined reward functions for policy optimization. Designing such functions is a complex task, even for domain experts. However, valid task demonstrations can still be collected from moderately skilled individuals. This motivates the use of methods such as Imitation Learning (IL) and Inverse Reinforcement Learning (IRL), where an expert provides demonstrations, allowing the algorithm to infer a policy or reward function that aligns with observed behavior. A common assumption in IRL is that demonstrations come from highly skilled experts. While some studies have explored the impact of suboptimal demonstrators, the influence of intentionally malicious demonstrations remains underexplored. This study introduces an adversarial demonstrator framework that systematically perturbs a subset of demonstrations to manipulate the reward function inferred by an IRL algorithm. Additionally, it quantifies the impact of such adversarial manipulations on the learned policy. Our results show that simply altering 10% of the demonstrations can lead the IRL algorithm to learn a faulty reward function, ultimately degrading the performance of the trained policy by up to three times the effect of adding 10% random trajectories to the result.

In this paper, we study the inverse reinforcement learning (IRL) algorithm for semi-Markov decision processes (SMDPs) with average reward based on the performance sensitivity analysis. By analyzing the structural form of the performance difference formula between any two different policies, we utilize the expert policy to transform the IRL problems of SMDPs into convex optimization problems. Based on two different performance difference formulas, two IRL algorithms are proposed and the algorithms are verified on a grid maze with options. The feasibility and effectiveness of algorithms are illustrated in the simulation results.