Moratorium Effect on Estimation Values in Simple Reinforcement Learning

Reinforcement learning is a framing of enabling agents to learn from interaction with environments. It has focused generally on Markov decision process (MDP) domains, but a domain may be non-Markovian in the real world. In this paper, we introduce a new description of macro-actions with tree structure in reinforcement learning. The macro-action is an action control structure which provides an agent with control which applies a collection of related microscopic actions as a single action unit. And we propose a simple method for dynamically acquiring macro-actions from the experiences of agents during reinforcement learning process.

Reinforcement Learning with Foregone Payoff Information in Normal Form Games

Transfer learning is the one of methods to improve the efficiency of training for reinforcement learning (RL) agents. That is the method of transferring knowledge which is acquired in the past tasks or given by experts to similar tasks. However, if the RL agents don't retain knowledge that is effective for improving learning speed or quality, there is a problem in that it causes negative transfer, which reduces learning efficiency. This study proposes a method for deleting knowledge which are less the number of state observation through trial and error by RL agents. Our proposed method was applied to path planning for a two-wheeled mobile robot to verify its effectiveness, and it was confirmed that the proposed method reduced the amount of knowledge by 98% and the learning time by 32%.

The development and maturation of multisensory processing and integration depend on the brain’s ability to detect patterns and adjust behaviour based on feedback. However, the developmental dynamics of these mechanisms are not yet fully understood. This study examines neural and behavioural differences in multisensory statistical and reinforcement learning processes by comparing adults and children, and further investigates these processes across two learning tasks of varying difficulty in children. Using a discriminative choice and a match recognition multisensory associative learning task with embedded statistical stimulus regularities, 28 adults (19.0–30.9 years) and 2 groups of children (N = 28 and N = 29, 8.5–12.8 years) learned novel, non-linguistic associations of audio–visual or tactile–visual stimuli. Behaviourally, accuracy increased and reaction time decreased over time in all groups, and these learning effects were stronger in adults than in children and for audio–visual than for tactile–visual learning. Computational modelling revealed that adults employed more advanced learning strategies, with greater sensitivity to value differences, and more deterministic choice patterns. Behaviourally, reaction times were slower for more surprising trials across all groups. Neuroimaging showed that statistical surprise modulated activity in the lateral prefrontal cortex, intraparietal sulcus, and anterior cingulate cortex, while reward prediction errors were linked to striatal, medial frontal, and hippocampal regions. Reward prediction errors were consistent between adults and children and between task variants, whereas neural surprise processing emerged only in the match recognition task and showed developmental differences, indicating immaturity in children’s ability to detect statistical regularities. These findings highlight the importance of task design when comparing learning differences across developmental groups and showcase how reinforcement and statistical learning can be studied in parallel.

Explainable reinforcement learning (XRL) is an emerging subfield of explainable machine learning that has attracted considerable attention in recent years. The goal of XRL is to elucidate the decision-making process of reinforcement learning (RL) agents in sequential decision-making settings. Equipped with this information, practitioners can better understand important questions about RL agents (especially those deployed in the real world), such as what the agents will do and why. Despite increased interest, there exists a gap in the literature for organizing the plethora of papers—especially in a way that centers the sequential decision-making nature of the problem. In this survey, we propose a novel taxonomy for organizing the XRL literature that prioritizes the RL setting. We propose three high-level categories: feature importance, learning process and Markov decision process, and policy-level. We overview techniques according to this taxonomy, highlighting challenges and opportunities for future work. We conclude by using these gaps to motivate and outline a roadmap for future work.

Reinforcement Learning (RL) agents have great successes in solving tasks with large observation and action spaces from limited feedback. Still, training the agents is data-intensive and there are no guarantees that the learned behavior is safe and does not violate rules of the environment, which has limitations for the practical deployment in real-world scenarios. This paper discusses the engineering of reliable agents via the integration of deep RL with constraint-based augmentation models to guide the RL agent towards safe behavior. Within the constraints set, the RL agent is free to adapt and explore, such that its effectiveness to solve the given problem is not hindered. However, once the RL agent leaves the space defined by the constraints, the outside models can provide guidance to still work reliably. We discuss integration points for constraint guidance within the RL process and perform experiments on two case studies: a strictly constrained card game and a grid world environment with additional combinatorial subgoals. Our results show that constraint-guidance does both provide reliability improvements and safer behavior, as well as accelerated training.

The supervisory control of robots is a very demanding application. In the context of robots control in assistive environments, it is important that the robot user is able to give commands to robots in a way that is easy and intuitive. There are several tasks that can be achieved using robots under assistive environments. It is challenging to efficiently control multiple robots / robots with degrees of freedom with a simple/intuitive interface by a single operator. In this proposal, we propose the use of Reinforcement Learning for intuitive interface mapping. Based on interaction with the environments, we can determine the optimal interface mapping through the process of Reinforcement Learning. The novelty of this paper is the use of changing reward functions based on qualitative performance evaluation for the Reinforcement Learning algorithm. In this paper, we show that the use of proposed reward functions can result in optimal/intuitive interface mapping for multiple robots / robots with degrees of freedom control applications.

Reinforcement learning (RL) in problems with large markov decision processes (MDPs) can result in many action policies. Over time, it becomes non-trivial computationally to search for action policies that are effective to the states. Function approximation can be used to generalize the action policies to improve the search efficiency. Based on the Adaptive Resonance Theory, FASON is a type of self-organizing neural networks that can be used for function approximation. This paper introduces the features that allow FASON to act and learn efficiently in large MDPs. Being structurally compatible to the propositional rules, the performance of FASON can be improved by directly inserting not-easily-learned domain knowledge into it. For speeding up RL, the Knowledge-based Exploration strategy is used to direct exploration. To select cognitive nodes effective to the states, a Reward Vigilance Adaptation strategy is used to adapt the reward vigilance. The performance of FASON is compared with selected benchmark models in three RL problems with large MDPs. Ten test cases were designed to study the effect of four key parameters on FASON. Experiment results of the winning parameters are presented to inform on the best ways of using FASON.

Dopamine signaling is thought to mediate reward-based learning. We tested for a role of dopamine in motor adaptation by administering the dopamine precursor levodopa to healthy participants in two experiments involving reaching movements. Levodopa has been shown to impair reward-based learning in cognitive tasks. Thus, we hypothesized that levodopa would selectively impair aspects of motor adaptation that depend on the reinforcement of rewarding actions. In the first experiment, participants performed two separate tasks in which adaptation was driven either by visual error-based feedback of the hand position or binary reward feedback. We used EEG to measure event-related potentials evoked by task feedback. We hypothesized that levodopa would specifically diminish adaptation and the neural responses to feedback in the reward learning task. However, levodopa did not affect motor adaptation in either task nor did it diminish event-related potentials elicited by reward outcomes. In the second experiment, participants learned to compensate for mechanical force field perturbations applied to the hand during reaching. Previous exposure to a particular force field can result in savings during subsequent adaptation to the same force field or interference during adaptation to an opposite force field. We hypothesized that levodopa would diminish savings and anterograde interference, as previous work suggests that these phenomena result from a reinforcement learning process. However, we found no reliable effects of levodopa. These results suggest that reward-based motor adaptation, savings, and interference may not depend on the same dopaminergic mechanisms that have been shown to be disrupted by levodopa during various cognitive tasks.NEW & NOTEWORTHY Motor adaptation relies on multiple processes including reinforcement of successful actions. Cognitive reinforcement learning is impaired by levodopa-induced disruption of dopamine function. We administered levodopa to healthy adults who participated in multiple motor adaptation tasks. We found no effects of levodopa on any component of motor adaptation. This suggests that motor adaptation may not depend on the same dopaminergic mechanisms as cognitive forms or reinforcement learning that have been shown to be impaired by levodopa.

Autonomous driving technology is constantly developing to a higher level of complex scenes, and there is a growing demand for the utilization of end-to-end data-driven control. However, the end-to-end path tracking process often encounters challenges in learning efficiency and generalization. To address this issue, this paper designs a deep deterministic policy gradient (DDPG)-based reinforcement learning strategy that integrates imitation learning and feedforward exploration in the path following process. In imitation learning, the path tracking control data generated by the model predictive control (MPC) method is used to train an end-to-end steering control model of a deep neural network. Another feedforward exploration behavior is predicted by road curvature and vehicle speed, and adds it and imitation learning to the DDPG reinforcement learning to obtain decision-making experience and action prediction behavior of the path tracking process. In the reinforcement learning process, imitation learning is used to update the pre-training parameters of the actor network, and a feedforward steering technique with random noise is adopted for strategy exploration. In the reward function, a hierarchical progressive reward form and a constrained objective reward function referring to MPC are designed, and the actor-critic network architecture is determined. Finally, the path tracking performance of the designed method is verified by comparing various training results, simulations, and HIL tests. The results show that the designed method can effectively utilize pre-training and feedforward prior experience to obtain optimal path tracking performance of an autonomous vehicle, and has better generalization ability than other methods. This study provides an efficient control scheme for improving the end-to-end control performance of autonomous vehicles.

Most existing person re-identification(Re-ID) approaches achieve superior results based on the assumption that a large amount of pre-labelled data is usually available and can be put into training phrase all at once. However, this assumption is not applicable to most real-world deployment of the Re-ID task. In this work, we propose an alternative reinforcement learning based human-in-the-loop model which releases the restriction of pre-labelling and keeps model upgrading with progressively collected data. The goal is to minimize human annotation efforts while maximizing Re-ID performance. It works in an iteratively updating framework by refining the RL policy and CNN parameters alternately. In particular, we formulate a Deep Reinforcement Active Learning (DRAL) method to guide an agent (a model in a reinforcement learning process) in selecting training samples on-the-fly by a human user/annotator. The reinforcement learning reward is the uncertainty value of each human selected sample. A binary feedback (positive or negative) labelled by the human annotator is used to select the samples of which are used to fine-tune a pre-trained CNN Re-ID model. Extensive experiments demonstrate the superiority of our DRAL method for deep reinforcement learning based human-in-the-loop person Re-ID when compared to existing unsupervised and transfer learning models as well as active learning models.

Reinforcement learning (RL) focuses on an essential aspect of intelligent behavior – how an agent can learn to make good decisions given experience and rewards in a stochastic<br>world. Yet popular RL algorithms that have enabled exciting successes in domains with good simulators (Go, Atari, etc) still often fail to learn in other domains because they rely on<br>simple heuristics for exploration. This provides additional empirical justification for essential questions around RL, specifically around algorithms that learn in a provably efficient manner through strategic exploration in any considered domain. This thesis provides new algorithms<br>and theory that enable good performance with respect to existing theoretical frameworks for evaluating RL algorithms (specifically, probably approximately correct) and introduces new stronger evaluation criteria, that may be particularly of interest as RL is applied to more real world problems.<br>For the first line of work on probably approximately correct (PAC) RL algorithms, we introduce a series of algorithms for episodic tabular domains with substantially better PAC<br>sample complexity bounds that culminate in a new algorithm with close to minimax optimal PAC and regret bounds. Look up tables are required by most sample efficient and computationally tractable algorithms, but cannot represent many practical domains. We therefore also present a new RL algorithm that can learn a good policy in environments with high dimensional observations and hidden deterministic states; unlike predecessors, this algorithm provably<br>explores not only in a statistically but also computationally efficient manner assuming access to function classes with efficient optimization oracles. To make progress it is critical to have the right measures of success. While empirical<br>demonstrations are quite clear, we find that for theoretical properties, two of the most commonly used learning frameworks, PAC guarantees and regret guarantees, each allow undesirable algorithm behavior (e.g. ignoring new observations that could improve the policy). We present<br>a new stronger learning framework called Uniform-PAC that unifies the existing frameworks and prevents undesirable algorithm properties. One caveat of all existing learning frameworks is that for any particular episode, we do not<br>know how well the algorithm will perform. To address this, we introduce the IPOC framework that requires algorithms to provide a certificate before each episode bounding how suboptimal the current policy can be. Such certifications may be of substantial interest in high stakes scenarios when an organization may wish to track or even pause an online RL system should the potential expected performance bound drop below a required expected outcome.

Advanced reinforcement learning (RL) technologies have recently increased the opportunity for automating several tasks in cluster management at scale by exploiting repetitive logs of cluster operation and building a learning model for resource allocation and job scheduling. Yet, this trend of adopting RL in the domain of cluster management has not fully addressed the diversity and heterogeneity of jobs and machines in modern cluster environments. In this paper, we present an RL-based scheduler for a multi-resource cluster, namely SCARL (SCheduler with Attentive Reinforcement Learning), concentrating on intricate cluster operating conditions with different resource requirements and capabilities. Specifically, we employ attentive embedding and factored-action scheduling that together efficiently incorporate time-varying interdependency of jobs and machines in RL processing; they enable an end-to-end scalable policy for scheduling diverse jobs on heterogeneous machines. To the best of our knowledge, we are the first to employ attention mechanism in RL-based cluster resource management. Through experiments, we demonstrate that our approach is competitive with existing heuristic methods under various cluster simulation configurations, e.g., an average 9.2 % enhancement in slowdown over the shortest job first algorithm. Additionally, the approach yields stable performance with our test cluster for running synthetic workloads based on real traces.

Curriculum learning has the potential to solve the problem of sparse rewards, a long-standing challenge in reinforcement learning, with greater sample efficiency than traditional reinforcement learning algorithms because curriculum learning enables agents to learn tasks in a meaningful order: from simple tasks to difficult ones. However, most curriculum learning in RL still relies on fixed hand-designed sequences of tasks. We present a novel scheme of automatic curriculum learning for reinforcement learning agents. A two-level hierarchical reinforcement learning framework, with a high-level policy called the curriculum generator and a low-level policy called the action policy, is proposed. During training, the curriculum generator automatically proposes curricula for the action policy to learn. Our training methods guarantee that the proposed curricula are always moderately difficult for the action policy. Both levels of policies are trained simultaneously and independently. After training, the low-level policy will be able to finish all tasks without the instructions given by the curriculum generator. Experiment results on a wide range of benchmark robotics environments demonstrate that our method accelerates convergence considerably and improves the training quality compared with the method without the curriculum generator.

In EdgeAI embedded devices that exploit reinforcement learning (RL), it is essential to reduce the number of actions taken by the agent in the real world and minimize the compute-intensive policies learning process. Convolutional autoencoders (AEs) has demonstrated great improvement for speeding up the policy learning time when attached to the RL agent, by compressing the high dimensional input data into a small latent representation for feeding the RL agent. Despite reducing the policy learning time, AE adds a significant computational and memory complexity to the model which contributes to the increase in the total computation and the model size. In this article, we propose a model for speeding up the policy learning process of RL agent with the use of AE neural networks, which engages binary and ternary precision to address the high complexity overhead without deteriorating the policy that an RL agent learns. Binary Neural Networks (BNNs) and Ternary Neural Networks (TNNs) compress weights into 1 and 2 bits representations, which result in significant compression of the model size and memory as well as simplifying multiply-accumulate (MAC) operations. We evaluate the performance of our model in three RL environments including DonkeyCar, Miniworld sidewalk, and Miniworld Object Pickup, which emulate various real-world applications with different levels of complexity. With proper hyperparameter optimization and architecture exploration, TNN models achieve near the same average reward, Peak Signal to Noise Ratio (PSNR) and Mean Squared Error (MSE) performance as the full-precision model while reducing the model size by 10x compared to full-precision and 3x compared to BNNs. However, in BNN models the average reward drops up to 12% - 25% compared to the full-precision even after increasing its model size by 4x. We designed and implemented a scalable hardware accelerator which is configurable in terms of the number of processing elements (PEs) and memory data width to achieve the best power, performance, and energy efficiency trade-off for EdgeAI embedded devices. The proposed hardware implemented on Artix-7 FPGA dissipates 250 μJ energy while meeting 30 frames per second (FPS) throughput requirements. The hardware is configurable to reach an efficiency of over 1 TOP/J on FPGA implementation. The proposed hardware accelerator is synthesized and placed-and-routed in 14 nm FinFET ASIC technology which brings down the power dissipation to 3.9 μJ and maximum throughput of 1,250 FPS. Compared to the state of the art TNN implementations on the same target platform, our hardware is 5x and 4.4x (2.2x if technology scaled) more energy efficient on FPGA and ASIC, respectively.