Reinforcement learning transfer via sparse coding

Reinforcement learning agents can successfully learn in a variety of difficult tasks. A fundamental problem is that they may learn slowly in complex environments, inspiring the development of speedup methods such as transfer learning. Transfer improves learning by reusing learned behaviors in similar tasks, usually via an inter-task mapping, which defines how a pair of tasks are related. This paper proposes a novel transfer learning technique to autonomously construct an inter-task mapping by using a novel combinations of sparse coding, sparse projection learning, and sparse pseudo-input gaussian processes. Experiments show successful transfer of information between two very different domains: the mountain car and the pole swing-up task. This paper empirically shows that the learned inter-task mapping can be used to successfully (1) improve the performance of a learned policy on a fixed number of samples, (2) reduce the learning times needed by the algorithms to converge to a policy on a fixed number of samples, and (3) converge faster to a near-optimal policy given a large amount of samples.

Robots had a great impact on the manufacturing industry ever since the early seventies when companies such as KUKA and ABB started deploying their first industrial robots. These robots merely performed very specific tasks in specific ways within well-defined environments. Still, they proved to be very useful as they could exceed human performance at these tasks. However, in order to enable robots to enter our daily life, they need to become more versatile and need to operate in much less structured environments. This thesis is partly devoted to stretching these limitations by means of learning, namely imitation learning (IL) and inverse reinforcement learning (IRL). Reinforcement learning (RL) is a powerful approach to enable robots to solve a task in an unknown environment. The practitioner describes a desired behavior by specifying a reward function and the robot autonomously interacts with the environment in order to find a control policy that generates high accumulated reward. However, RL is not suitable for teaching new tasks by non-experts because specifying appropriate reward functions can be difficult. Demonstrating the desired behavior is often easier for non-experts. Imitation learning can be used in order to enable the robot to reproduce the demonstrations. However, without explicitly inferring and modeling the intentions of the demonstrations, it can become difficult to solve the task for unseen situations. Inverse reinforcement learning (IRL) therefore aims to infer a reward function from the demonstrations, such that optimizing this reward function yields the desired behavior even for different situations. This thesis introduces a unifying approach to solve the inverse reinforcement learning problem in the same way as the reinforcement learning problem. This is achieved by framing both problems as information projection problems, i.e., we strive to minimize the relative entropy between a probabilistic model of the robot behavior and a given desired distribution. Furthermore, a trust region on the robot behavior is used to stabilize the optimization. For inverse reinforcement learning, the desired distribution is implicitly given by the expert demonstrations. The resulting optimization can be efficiently solved using state-of-the-art reinforcement learning methods. For reinforcement learning, the log-likelihood of the desired distribution is given by the reward function. The resulting optimization problem corresponds to a standard reinforcement learning formulation, except for an additional objective of maximizing the entropy of the robot behavior. This entropy objective adds little overhead to the optimization, but can lead to better exploration and more diversified policies. Trust-region I-projections are not only useful for training robots, but can also be applied to other machine learning problems. I-projections are typically used for variational inference, in order to approximate an intractable distribution by a simpler model. However, the resulting optimization problems are usually optimized based on stochastic gradient descent which often suffers from high variance in the gradient estimates. As trust-region I-projections where shown to be effective for reinforcement learning and inverse reinforcement learning, this thesis also explores their use for variational inference. More specifically, trust-region I-projections are investigated for the problem of approximating an intractable distribution by a Gaussian mixture model (GMM) with an adaptive number of components. GMMs are highly desirable for variational inference because they can yield arbitrary accurate approximations while inference from GMMs is still relatively cheap. In order to make learning the GMM feasible, we derive a lower bound that enables us to decompose the objective function. The optimization can then be performed by iteratively updating individual components using a technique from reinforcement learning. The resulting method is capable of learning approximations of significantly higher quality than existing variational inference methods. Due to the similarity of the underlying optimization problems, the insights gained from our variational inference method are also useful for IL and IRL. Namely, a similar lower bound can be applied also for the I-projection formulation of imitation learning. However, whereas for variational inference the lower bound serves to decompose the objective function, for imitation learning it allows us to provide a reward signal to the robot that does not depend on its behavior. Compared to reward functions that are relative to the current behavior of the robot---which are typical for popular adversarial methods---behavior-independent reward functions have the advantages that we can show convergence even for greedy optimization. Furthermore, behavior-independent reward functions solve the inverse reinforcement learning problem, thereby closing the gap between imitation learning and IRL. However, algorithms derived from our non-adversarial formulation are actually very similar to existing AIL methods, and we can even show that adversarial inverse reinforcement learning (AIRL) is indeed an instance of our formulation. AIRL was derived from an adversarial formulation, and we point out several problems of that derivation. In contrast, we show that AIRL can be straightforwardly derived from out non-adversarial formulation. Furthermore, we demonstrate that the non-adversarial formulation can be also used to derive novel algorithms by presenting a non-adversarial method for offline imitation learning.

This doctoral thesis presents a framework for machine based interpretation of Malaysian Sign Language (MSL). The goal is to explore and develop necessary core algorithms that are capable of recognizing hand signs, based on the monocular/2D visual input of a low-cost hardware (Digital Webcam), without the assistance of visual-markers on hands. Successful hand sign interpretation with the aforementioned constrained visual system is challenging. Economical hardware devices often provide noisy/cluttered raw data with limited useful information. Non-involvement of visual markers imposes the requirement of additional strategies to disambiguate occlusions and background complexities. Natural variations in hand posture/gesture performances further convolute the recognition task. Addressing these challenges require computationally heavy algorithms in image processing, machine learning and pattern recognition. Thus vision-based sign language translator frameworks have to be multifaceted and versatile, in order to address innumerable practical concerns. In this research, hand postures interpretation is achieved by statistical projection methods. Analysis of high dimensional hand shape data is a complex task. Thus reducing dimensions via known projection methods aids in alleviating some of these complexities. Locality Preserving Projection (LPP) has been employed to accomplish this goal, while Principal Component Analysis (PCA) has been used as the benchmark method. Both methods were extensively tested against a hand posture database of 24 alphabet signs from MSL. As for the hand gesture interpretation, parameter-less Gaussian process based machine learning methods are proposed, as an alternative to current status quo. A variation of the Gaussian latent variable model – Gaussian Process Dynamic Model (GPDM) – has been employed for hand gesture interpretation. This has been comparatively studied against the established Hidden Markov Model (HMM). These machine learning methods were thoroughly tested against a hand gesture database of 64 signs from MSL. Grammatical sentence construction offers both intelligible sentences, and a mean of identifying erroneous sign classifications. In this thesis, a proof-of-concept on: use of Context Free Grammar (CFG) in successful parsing of meaningful sentences based on recognised gesture words, and how it can be extended in identifying (and correcting) erroneous sign classifications, is demonstrated with 92 sentences and the output of GPDM gesture classification. The novelty and contribution of this research: this is the first undertaking of markerless vision-based translation for MSL. Both LPP and GPDM are introduced to the sign language research field as an outcome of this investigation. The preliminary study and demonstration of grammatical sentence construction for translated gesture signs, is the first of its kind to the best of author's knowledge.

We present an approach for the quantification of the usefulness of transfer in reinforcement learning via regret bounds for a multi-agent setting. Considering a number of $$\varvec{\aleph }$$ agents operating in the same Markov decision process, however possibly with different reward functions, we consider the regret each agent suffers with respect to an optimal policy maximizing its average reward. We show that when the agents share their observations the mutual regret of all agents is smaller by a factor of $$\varvec{\sqrt{\aleph }}$$ compared to the case when each agent has to rely on the information collected by itself. This result demonstrates how considering the regret in multi-agent settings can provide theoretical bounds on the benefit of sharing observations in transfer learning.

Transfer learning is the one of methods to improve the efficiency of training for reinforcement learning (RL) agents. That is the method of transferring knowledge which is acquired in the past tasks or given by experts to similar tasks. However, if the RL agents don't retain knowledge that is effective for improving learning speed or quality, there is a problem in that it causes negative transfer, which reduces learning efficiency. This study proposes a method for deleting knowledge which are less the number of state observation through trial and error by RL agents. Our proposed method was applied to path planning for a two-wheeled mobile robot to verify its effectiveness, and it was confirmed that the proposed method reduced the amount of knowledge by 98% and the learning time by 32%.

Nowadays, computer-based face recognition is a mature and reliable mechanism that is significantly used in many access control scenarios along with other biometric methods. Face recognition consists of two subtasks including Face Verification and Face Identification. By comparing a pair of images, Face Verification determines whether those images are related to one person or not; and Face Identification has to identify a specific face within a set of available faces in the database. There are many challenges in face recognition such as angle, illumination, pose, facial expression, noise, resolution, occlusion and the few number of one-class samples with several classes. In this paper, we are carrying out face recognition by utilizing transfer learning in a siamese network which consists of two similar CNNs. In the siamese network, a pair of two face images is given to the network as input, then the network extracts the features of this pair of images and finally, it determines whether the pair of images belongs to one person or not by using a similarity criterion. The results show that the proposed model is comparable with advanced models that are trained on datasets containing large numbers of samples. furthermore, it improves the accuracy of face recognition in comparison with methods which are trained using datasets with a few number of samples, and the mentioned accuracy is claimed to be 95.62% on LFW dataset.

In this work, we design and evaluate a computational learning model that enables a human-robot team to co-develop joint strategies for performing novel tasks that require coordination. The joint strategies are learned through "perturbation training," a human team-training strategy that requires team members to practice variations of a given task to help their team generalize to new variants of that task. We formally define the problem of human-robot perturbation training and develop and evaluate the first end-to-end framework for such training, which incorporates a multi-agent transfer learning algorithm, human-robot co-learning framework and communication protocol. Our transfer learning algorithm, Adaptive Perturbation Training (AdaPT), is a hybrid of transfer and reinforcement learning techniques that learns quickly and robustly for new task variants. We empirically validate the benefits of AdaPT through comparison to other hybrid reinforcement and transfer learning techniques aimed at transferring knowledge from multiple source tasks to a single target task. We also demonstrate that AdaPT's rapid learning supports live interaction between a person and a robot, during which the human-robot team trains to achieve a high level of performance for new task variants. We augment AdaPT with a co-learning framework and a computational bi-directional communication protocol so that the robot can co-train with a person during live interaction. Results from large-scale human subject experiments (n=48) indicate that AdaPT enables an agent to learn in a manner compatible with a human's own learning process, and that a robot undergoing perturbation training with a human results in a high level of team performance. Finally, we demonstrate that human-robot training using AdaPT in a simulation environment produces effective performance for a team incorporating an embodied robot partner.

The ambitious goal of transfer learning is to accelerate learning on a target task after training on a different, but related, source task. While many past transfer methods have focused on transferring value-functions, this paper presents a method for transferring policies across tasks with different state and action spaces. In particular, this paper utilizes transfer via inter-task mappings for policy search methods (TVITM-PS) to construct a transfer functional that translates a population of neural network policies trained via policy search from a source task to a target task. Empirical results in robot soccer Keepaway and Server Job Scheduling show that TVITM-PS can markedly reduce learning time when full inter-task mappings are available. The results also demonstrate that TVITMPS still succeeds when given only incomplete inter-task mappings. Furthermore, we present a novel method for learning such mappings when they are not available, and give results showing they perform comparably to hand-coded mappings.

In this work we explore the use of latent representations obtained from multiple input sensory modalities (such as images or sounds) in allowing an agent to learn and exploit policies over different subsets of input modalities. We propose a three-stage architecture that allows a reinforcement learning agent trained over a given sensory modality, to execute its task on a different sensory modality---for example, learning a visual policy over image inputs, and then execute such policy when only sound inputs are available. We show that the generalized policies achieve better out-of-the-box performance when compared to different baselines. Moreover, we show this holds in different OpenAI gym and video game environments, even when using different multimodal generative models and reinforcement learning algorithms.

Brain-computer interaction (BCI) and physiological computing are terms that refer to using processed neural or physiological signals to influence human interaction with computers, environment, and each other. A major challenge in developing these systems arises from the large individual differences typically seen in the neural/physiological responses. As a result, many researchers use individually-trained recognition algorithms to process this data. In order to minimize time, cost, and barriers to use, there is a need to minimize the amount of individual training data required, or equivalently, to increase the recognition accuracy without increasing the number of user-specific training samples. One promising method for achieving this is collaborative filtering, which combines training data from the individual subject with additional training data from other, similar subjects. This paper describes a successful application of a collaborative filtering approach intended for a BCI system. This approach is based on transfer learning (TL), active class selection (ACS), and a mean squared difference user-similarity heuristic. The resulting BCI system uses neural and physiological signals for automatic task difficulty recognition. TL improves the learning performance by combining a small number of user-specific training samples with a large number of auxiliary training samples from other similar subjects. ACS optimally selects the classes to generate user-specific training samples. Experimental results on 18 subjects, using both nearest neighbors and support vector machine classifiers, demonstrate that the proposed approach can significantly reduce the number of user-specific training data samples. This collaborative filtering approach will also be generalizable to handling individual differences in many other applications that involve human neural or physiological data, such as affective computing.

Transfer on Reinforcement Learning (RL) is a promising method on learning new skills and adapting new situations for humanoid robots as tasks or environments change. Within the normal process of Transfer Learning in RL, the inter-task mapping is manually defined, which lacks generalization ability. Therefore, how to automatically learning intertask relations becomes a hot topic. Considering the limited computational resource of a physical humanoid robot, high learning efficiency regarding to both fast speed algorithm and low sample complexity should be emphasized in skills transfer. According to this view, in this research, the inter-task relations are modelled using a three-way Restricted Boltzmann Machine (RBM), which is turned out to be a powerful model in capturing the similarity between samples from source task and target task. Since standard Contrastive Divergence (CD) algorithm commonly used for RBM learning suffers from the inputindependent problem and may lead the learning process timeconsuming or inapplicable, a Cyclic Contrastive Divergence (CCD) learning algorithm is employed. In order to evaluate the performance, experiment that transfer the skill of walking on flat surface to the skill walking on slope surface is conducted on our physical robot platform, PKU-HR5.1, and the result indicates that the method is feasible and efficient.

We present a Momentum Re-identification (MoReID) framework that can leverage a very large number of negative samples in training for general re-identification task. The design of this framework is inspired by Momentum Contrast (MoCo), which uses a dictionary to store current and past batches to build a large set of encoded samples. As we find it less effective to use past positive samples which may be highly inconsistent to the encoded feature property formed with the current positive samples, MoReID is designed to use only a large number of negative samples stored in the dictionary. However, if we train the model using the widely used Triplet loss that uses only one sample to represent a set of positive/negative samples, it is hard to effectively leverage the enlarged set of negative samples acquired by the MoReID framework. To maximize the advantage of using the scaled-up negative sample set, we newly introduce Hard-distance Elastic loss (HE loss), which is capable of using more than one hard sample to represent a large number of samples. Our experiments demonstrate that a large number of negative samples provided by MoReID framework can be utilized at full capacity only with the HE loss, achieving the state-of-the-art accuracy on three re-ID benchmarks, VeRi-776, Market-1501, and VeRi-Wild.KeywordsRe-identificationLarge-scale negative samplesMomentum encoderMoReIDHE loss

We propose using a statistical-relational model, the Markov Logic Network, for knowledge transfer in reinforcement learning. Our goal is to extract relational knowledge from a source task and use it to speed up learning in a related target task. We show that Markov Logic Networks are effective models for capturing both source-task Q-functions and source-task policies. We apply them via demonstration, which involves using them for decision making in an initial stage of the target task before continuing to learn. Through experiments in the RoboCup simulated-soccer domain, we show that transfer via Markov Logic Networks can significantly improve early performance in complex tasks, and that transferring policies is more effective than transferring Q-functions.

Transfer in reinforcement learning with indeterminate natural transformations

Reinforcement learning (RL) is a well-studied framework to solve complex decision-making problems in unknown environments. The actor-critic model in RL facilitates autonomy transfer by allowing agents to iteratively update their policies using ongoing dynamic evaluations of value functions via a critic. In this paper, we examine the impact of using different pretrained critics on the performance of actor-critic algorithms. First, in any single given environment, we show that a pretrained critic can be effective in reducing the duration of an initial training phase, thereby accelerating convergence by a factor of up to 2×. In this setting, we identify the critical range of the number of episodes for which a critic will need to be trained in order for it to be an effective pretrained critic. Second, we show that a critic trained in one environment enables transfer of autonomy by aiding learning of behaviors in a different, yet related environment. We carry out extensive experiments on a bipedal locomotion task in the MuJoCo physics engine to verify our hypotheses. Our results in this paper mark the first step towards demonstrating the role and impact of pretrained critics to achieve rapid autonomy transfer for complex reinforcement learning tasks while minimizing costs associated with retraining in new environments.