An inverse reinforcement learning algorithm for semi-Markov decision processes

Energy-efficient and damage-recovery slithering gait design for a snake-like robot based on reinforcement learning and inverse reinforcement learning

This work develops a novel high-dimensional inverse reinforcement learning (IRL) algorithm for human motion analysis in medical, clinical, and robotics applications. The method is based on the assumption that a surgical robot operators’ skill or a patient’s motor skill is encoded into the innate reward function during motion planning and recovered by an IRL algorithm from motion demonstrations. This class of applications is characterized by high-dimensional sensory data, which is computationally prohibitive for most existing IRL algorithms. We propose a novel function approximation framework and reformulate the Bellman optimality equation to handle high-dimensional state spaces efficiently. We compare different function approximators in simulated environments, and adopt a deep neural network as the function approximator. The technique is applied to evaluating human patients with spinal cord injuries under spinal stimulation, and the skill levels of surgical robot operators. The results demonstrate the efficiency and effectiveness of the proposed method.

Reinforcement Learning (RL) algorithms depend on well-defined reward functions for policy optimization. Designing such functions is a complex task, even for domain experts. However, valid task demonstrations can still be collected from moderately skilled individuals. This motivates the use of methods such as Imitation Learning (IL) and Inverse Reinforcement Learning (IRL), where an expert provides demonstrations, allowing the algorithm to infer a policy or reward function that aligns with observed behavior. A common assumption in IRL is that demonstrations come from highly skilled experts. While some studies have explored the impact of suboptimal demonstrators, the influence of intentionally malicious demonstrations remains underexplored. This study introduces an adversarial demonstrator framework that systematically perturbs a subset of demonstrations to manipulate the reward function inferred by an IRL algorithm. Additionally, it quantifies the impact of such adversarial manipulations on the learned policy. Our results show that simply altering 10% of the demonstrations can lead the IRL algorithm to learn a faulty reward function, ultimately degrading the performance of the trained policy by up to three times the effect of adding 10% random trajectories to the result.

Rewards are critical hyperparameters in reinforcement learning (RL), since in most cases different reward values will lead to greatly different performance. Due to their commercial value, RL rewards become the target of reverse engineering by the inverse reinforcement learning (IRL) algorithm family. Existing efforts typically utilize two metrics to measure the IRL performance: the expected value difference and the mean reward loss, which we call them EVD and MRL respectively. Unfortunately, in some cases, EVD and MRL can give completely opposite results, due to MRL focusing on whole state-space rewards while EVD only considering partly sampled rewards. Such situation naturally rises to one fundamental question: whether current metrics and assessment are sufficient and accurate for more general use. Thus, in this paper, based on the metric called normalized mutual information of reward clusters (C-NMI) we propose a novel IRL assessment; we aim to fill this research gap by considering a middle-granularity state space between the entire state space and the specific sampling space. We utilize the agglomerative nesting algorithm (AGNES) to control dynamical C-NMI computing via a 4-order tensor model with injected manipulated trajectories. With such a model, we can uniformly capture different-dimension values of MRL, EVD, and C-NMI, and perform more comprehensive and accurate assessment and analyses. Extensive experiments on several mainstream IRLs are experimented in Object World, hence revealing that the assessing accuracy of our method increases 110.13% and 116.59% respectively when compared with the EVD and MRL. Meanwhile, C-NMI is more robust than EVD and MRL under different demonstrations.

We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risksensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of models of human decision-making having their origins in behavioral psychology and economics. We propose a gradient-based inverse reinforcement learning algorithm that minimizes a loss function defined on the observed behavior. We demonstrate the performance of the proposed technique on two examples, the first of which is the canonical Grid World example and the second of which is an MDP modeling passengers' decisions regarding ride-sharing. In the latter, we use pricing and travel time data from a ride-sharing company to construct the transition probabilities and rewards of the MDP.

ABSTRACTIn this article, the initial excitation‐based inverse reinforcement learning methods are presented for continuous‐time linear non‐zero‐sum games. The policy iteration and value iteration algorithms are presented for the inverse reinforcement learning problems, and an online‐verifiable initial excitation condition is given to guarantee the convergence of the presented algorithms. Comparing with the traditional inverse reinforcement learning algorithms for linear non‐zero‐sum games, the presented algorithms relax the requirement on data‐storage mechanism. Furthermore, the requirement on the initial stabilizing state feedback matrices is relaxed in the presented initial excitation‐based value iteration algorithm. The properties of the presented initial excitation‐based policy iteration and value iteration algorithms are analyzed. Simulation results show the efficiency of the presented algorithms.

<p style='text-indent:20px;'>This paper aims to investigate the maintenance policy for a two-machine one-buffer (2M1B) assembly line system. We assume that the observed quality states of the deteriorating machines in the system are characterized by multiple decreasing yield stages. A semi-Markov decision process (SMDP) model is used for describing the deteriorating process of the system. A heuristically accelerated multi-agent reinforcement learning (HAMRL) method is conducted to solve the problem model. The asynchronous updating rules are introduced in the HAMRL method, and the production time, preventive maintenance (PM) time and corrective repair (CR) time are random, and the deterioration mode of the device is not fixed. Meanwhile, a comparison with a simulated annealing search (SAS) based exploration algorithm and a neighborhood search (NS) based exploration algorithm in reinforcement learning (RL) is presented. The empirical results indicate that the proposed HAMRL algorithm can speed up the learning process, and has a certain advantage for the larger space and the more practical problem. And the maintenance strategy for the 2M1B assembly line system is obtained under the condition of convergent system average cost rate. This paper provides new and practical insights into the application and selection of techniques for maintenance policy of the 2M1B assembly line system.</p>

Learning from demonstration, or imitation learning, is the process of learning to act in an environment from examples provided by a teacher. Inverse reinforcement learning (IRL) is a specific form of learning from demonstration that attempts to estimate the reward function of a Markov decision process from examples provided by the teacher. The reward function is often considered the most succinct description of a task. In simple applications, the reward function may be known or easily derived from properties of the system and hard coded into the learning process. However, in complex applications, this may not be possible, and it may be easier to learn the reward function by observing the actions of the teacher. This paper provides a comprehensive survey of the literature on IRL. This survey outlines the differences between IRL and two similar methods - apprenticeship learning and inverse optimal control. Further, this survey organizes the IRL literature based on the principal method, describes applications of IRL algorithms, and provides areas of future research.

The problem of learning an expert’s unknown reward function using a limited number of demonstrations recorded from the expert’s behavior is investigated in the area of inverse reinforcement learning (IRL). To gain traction in this challenging and underconstrained problem, IRL methods predominantly represent the reward function of the expert as a linear combination of known features. Most of the existing IRL algorithms either assume the availability of a transition function or provide a complex and inefficient approach to learn it. In this paper, we present a model-free approach to IRL, which casts IRL in the maximum likelihood framework. We present modifications of the model-free Q-learning that replace its maximization to allow computing the gradient of the Q-function. We use gradient ascent to update the feature weights to maximize the likelihood of expert’s trajectories. We demonstrate on two problem domains that our approach improves the likelihood compared to previous methods.

Robots had a great impact on the manufacturing industry ever since the early seventies when companies such as KUKA and ABB started deploying their first industrial robots. These robots merely performed very specific tasks in specific ways within well-defined environments. Still, they proved to be very useful as they could exceed human performance at these tasks. However, in order to enable robots to enter our daily life, they need to become more versatile and need to operate in much less structured environments. This thesis is partly devoted to stretching these limitations by means of learning, namely imitation learning (IL) and inverse reinforcement learning (IRL). Reinforcement learning (RL) is a powerful approach to enable robots to solve a task in an unknown environment. The practitioner describes a desired behavior by specifying a reward function and the robot autonomously interacts with the environment in order to find a control policy that generates high accumulated reward. However, RL is not suitable for teaching new tasks by non-experts because specifying appropriate reward functions can be difficult. Demonstrating the desired behavior is often easier for non-experts. Imitation learning can be used in order to enable the robot to reproduce the demonstrations. However, without explicitly inferring and modeling the intentions of the demonstrations, it can become difficult to solve the task for unseen situations. Inverse reinforcement learning (IRL) therefore aims to infer a reward function from the demonstrations, such that optimizing this reward function yields the desired behavior even for different situations. This thesis introduces a unifying approach to solve the inverse reinforcement learning problem in the same way as the reinforcement learning problem. This is achieved by framing both problems as information projection problems, i.e., we strive to minimize the relative entropy between a probabilistic model of the robot behavior and a given desired distribution. Furthermore, a trust region on the robot behavior is used to stabilize the optimization. For inverse reinforcement learning, the desired distribution is implicitly given by the expert demonstrations. The resulting optimization can be efficiently solved using state-of-the-art reinforcement learning methods. For reinforcement learning, the log-likelihood of the desired distribution is given by the reward function. The resulting optimization problem corresponds to a standard reinforcement learning formulation, except for an additional objective of maximizing the entropy of the robot behavior. This entropy objective adds little overhead to the optimization, but can lead to better exploration and more diversified policies. Trust-region I-projections are not only useful for training robots, but can also be applied to other machine learning problems. I-projections are typically used for variational inference, in order to approximate an intractable distribution by a simpler model. However, the resulting optimization problems are usually optimized based on stochastic gradient descent which often suffers from high variance in the gradient estimates. As trust-region I-projections where shown to be effective for reinforcement learning and inverse reinforcement learning, this thesis also explores their use for variational inference. More specifically, trust-region I-projections are investigated for the problem of approximating an intractable distribution by a Gaussian mixture model (GMM) with an adaptive number of components. GMMs are highly desirable for variational inference because they can yield arbitrary accurate approximations while inference from GMMs is still relatively cheap. In order to make learning the GMM feasible, we derive a lower bound that enables us to decompose the objective function. The optimization can then be performed by iteratively updating individual components using a technique from reinforcement learning. The resulting method is capable of learning approximations of significantly higher quality than existing variational inference methods. Due to the similarity of the underlying optimization problems, the insights gained from our variational inference method are also useful for IL and IRL. Namely, a similar lower bound can be applied also for the I-projection formulation of imitation learning. However, whereas for variational inference the lower bound serves to decompose the objective function, for imitation learning it allows us to provide a reward signal to the robot that does not depend on its behavior. Compared to reward functions that are relative to the current behavior of the robot---which are typical for popular adversarial methods---behavior-independent reward functions have the advantages that we can show convergence even for greedy optimization. Furthermore, behavior-independent reward functions solve the inverse reinforcement learning problem, thereby closing the gap between imitation learning and IRL. However, algorithms derived from our non-adversarial formulation are actually very similar to existing AIL methods, and we can even show that adversarial inverse reinforcement learning (AIRL) is indeed an instance of our formulation. AIRL was derived from an adversarial formulation, and we point out several problems of that derivation. In contrast, we show that AIRL can be straightforwardly derived from out non-adversarial formulation. Furthermore, we demonstrate that the non-adversarial formulation can be also used to derive novel algorithms by presenting a non-adversarial method for offline imitation learning.

Reinforcement learning (RL)-based brain machine interfaces (BMIs) provide a promising solution for paralyzed people. Enhancing the decoding performance of RL-based BMIs relies on the design of effective reward signals. Inverse reinforcement learning (IRL) offers an approach to infer subjects' own evaluation from the observed behavior. However, applying IRL to extract reward information in complex BMI tasks requires consideration of the dynamics of subjects' goal during the control process. This dynamic nature of subjects' evaluation requires the IRL method to be able to estimate a time varying reward function. Previous IRL methods applied in BMI systems only estimated a static reward function. Existing IRL algorithms for dynamic reward estimation employ optimization methods to approximate the reward map for each state at each time, which demands substantial amounts of data to achieve convergence. In this paper, we propose a dynamic IRL method to estimate the feedback-driven reward of subjects during BMI tasks. We utilize a state-observation model to continuously infer the reward value for each state, with sensory feedback serving as the external input to model the transition process of the reward. We evaluate our proposed method on a simulated BMI fetch task, which is a multistep task with a time varying reward function. Our method demonstrates improved reward estimation close to the ground truth value, and it significantly outperforms the existing dynamic IRL method when the map size exceeds 25(p<0.01). These preliminary results suggests that the dynamic IRL method for feedback-driven reward estimation holds potential for improving the design of RL-based BMIs.

The development of automated driving vehicles aims to provide safer, comfortable, and more efficient mobility options. However, the decision-making control of autonomous vehicles still faces limitations of human performance mimicry. These limitations become particularly evident in complex and unfamiliar driving scenarios, where weak decision-making abilities and poor adaptation of vehicle behaviour are prominent issues. This paper proposes a game-theoretic decision-making algorithm for human-like driving in the vehicle lane change scenario. Firstly, an inverse reinforcement learning (IRL) model is used to quantitatively analyze the lane change trajectories of the natural driving dataset, establishing the human-like human cost function. Subsequently, joint safety, and comfort to build the comprehensive decision cost function. The combined decision cost function is used to conduct a noncooperative game of vehicle lane changing decisions to solve the optimal decision of host vehicle lane changing. The host vehicle lane-changing decision problem is formulated as a Stackelberg game optimization problem. To verify the feasibility and effectiveness of the algorithm proposed in this study, a lane change test scenario was established. Firstly, we analyze the human-like decision-making model derived from the maximum entropy inverse reinforcement learning algorithm to verify the effectiveness and robustness of the IRL algorithm. Secondly, the human-like game decision-making algorithm in this paper is validated by conducting an interactive lane-changing experiment with obstacle vehicles of different driving styles. The experimental results prove that the human-like driving decision-making model proposed in this study can make lane-changing behaviours in line with human driving patterns in lane-changing scenarios of the expressway.

A complementary learning approach for expertise transference of human-optimized controllers

Inverse reinforcement learning (IRL) is the problem of learning the preferences of an agent from observing its behavior on a task. While this problem is witnessing sustained attention, the related problem of online IRL - where the observations are incrementally accrued, yet the real-time demands of the application often prohibit a full rerun of an IRL method - has received much less attention. We introduce a formal framework for online IRL, called incremental IRL (I2RL), and a new method that advances maximum entropy IRL with hidden variables, to this setting. Our formal analysis shows that the new method has a monotonically improving performance with more demonstration data, as well as probabilistically bounded error, both under full and partial observability. Experiments in a simulated robotic application, which involves learning under occlusion, show the significantly improved performance of online IRL as compared to both batch IRL and an online imitation learning method.

Inverse reinforcement learning (IRL) addresses the problem of recovering the unknown reward function for a given Markov decision problem (MDP) given the corresponding optimal policy or a perturbed version thereof. This paper studies the space of possible solutions to the general IRL problem, when the agent is provided with incomplete/imperfect information regarding the optimal policy for the MDP whose reward must be estimated. We focus on scenarios with finite state-action spaces and discuss the constraints imposed on the set of possible solutions when the agent is provided with (i) perturbed policies; (ii) optimal policies; and (iii) incomplete policies. We discuss previous works on IRL in light of our analysis and show that, with our characterization of the solution space, it is possible to determine non-trivial closed-form solutions for the IRL problem. We also discuss several other interesting aspects of the IRL problem that stem from our analysis.