Multitask reinforcement learning with metadata-guided adaptive routing

In many Multi Agent Systems, under-education agents investigate their environments to discover their targets. Any agent can also learn its strategy. In multitask learning, one agent studies a set of related problems together simultaneously, by a common model. In reinforcement learning exploration phase, it is necessary to introduce a process of trial and error to learn better rewards obtained from environment. To reach this end, anyone can typically employ the uniform pseudorandom number generator in exploration period. On the other hand, it is predictable that chaotic sources also offer a random-like series comparable to stochastic ones. It is useful in multitask reinforcement learning, to use teammate agents' experience by doing simple interactions between each other. We employ the past experiences of agents to enhance performance of multitask learning in a nondeterministic environment. Communications are created by operators of evolutionary algorithm. In this paper we have also employed the chaotic generator in the exploration phase of reinforcement learning in a nondeterministic maze problem. We obtained interesting results in the maze problem.

In many Multi-Agent Systems, under-education agents investigate their environments to discover their target(s). Any agent can also learn its strategy. In multi-task learning, one agent studies a set of related problems together simultaneously, by a common model. In reinforcement learning exploration phase, it is necessary to introduce a process of trial and error to learn better rewards obtained from environment. To reach this end, anyone can typically employ the uniform pseudorandom number generator in exploration period. On the other hand, it is predictable that chaotic sources also offer a random-like series comparable to stochastic ones. It is useful in multi-task reinforcement learning, to use teammate agents' experience by doing simple interactions between each other. We employ the past experiences of agents to enhance performance of multi-task learning in a nondeterministic environment. Communications are created by operators of evolutionary algorithm. In this paper we have also employed the chaotic generator in the exploration phase of reinforcement learning in a nondeterministic maze problem. We obtained interesting results in the maze problem.

Recently, Multitask learning, which can cope with several tasks, has attracted much attention. Multitask Reinforcement Learning introduced by Tanaka et al is a problem class where number of problem instances of Markov Decision Processes sampled from the same probability distributions is sequentially given to reinforcement learning agents. The purpose of solving this problem is to realize adaptive agents for newly given environments by using knowledge acquired from past experience. Evolutionary Algorithms are often used to solve reinforcement learning problems if problem classes are quite different with Markov Decision Processes or state-action space is quite huge. From the viewpoint of Evolutionary Algorithms studies, the Multitask Reinforcement Learning problems are regarded as dynamic problems whose fitness landscape has changed temporally. In this paper, a memory-based Evolutionary Programming which is suitable for Multitask Reinforcement Learning problems is proposed.

Dexterous grasping and manipulation with multifingered robotic hands presents a significant challenge due to their high degrees of freedom and the need for task-specific adaptations. Existing methods usually adopt single-task learning framework or focus on simple stable wrap grasping, limiting their efficiency and generalization ability when encountering new task or precise functional grasping pose. In this article, we introduce MetaGrasp, a novel approach that defines dexterous functional grasping as a multitask reinforcement learning (RL) problem based on hand grasp pose classification. Our method features a unique gradual skill curriculum learning (GSCL) framework, which structures the learning process into three stages: beginner, intermediate, and advanced curriculum learning according to the level of difficulty. MetaGrasp leverages this hierarchical learning structure to develop a versatile, adaptive grasping policy that can grasp objects based on hand grasp pose and object point cloud inputs. Taking five hand grasp types as research cases, the trained policy with our MetaGrasp can be easily adpated to grasp different object instances from different object categories according to functional grasp intentions specified by one expert demonstration without requiring extensive system interaction. We categorize the dexterous functional grasping tasks of a five-fingered robotic hand into multiple tasks based on hand poses for RL, and to combine meta imitation learning (IL) with curriculum learning. The experimental results show that the MetaGrasp has better one-shot generalization ability on new grasp tasks, and outperforms state-of-the-art single-task dexterous grasping methods.

We propose a fully distributed actor-critic architecture, named diffusion-distributed-actor-criticDiff-DAC, with application to multitask reinforcement learning (MRL). During the learning process, agents communicate their value and policy parameters to their neighbours, diffusing the information across a network of agents with no need for a central station. Each agent can only access data from its local task, but aims to learn a common policy that performs well for the whole set of tasks. The architecture is scalable, since the computational and communication cost per agent depends on the number of neighbours rather than the overall number of agents. We derive Diff-DAC from duality theory and provide novel insights into the actor-critic framework, showing that it is actually an instance of the dual-ascent method. We prove almost sure convergence of Diff-DAC to a common policy under general assumptions that hold even for deep neural network approximations. For more restrictive assumptions, we also prove that this common policy is a stationary point of an approximation of the original problem. Numerical results on multitask extensions of common continuous control benchmarks demonstrate that Diff-DAC stabilises learning and has a regularising effect that induces higher performance and better generalisation properties than previous architectures.

Rahul Aralikatte, Heather Lent, Ana Valeria Gonzalez, Daniel Herschcovich, Chen Qiu, Anders Sandholm, Michael Ringaard, Anders Søgaard. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

In this paper, a new distributed multi-agent off-policy Actor-Critic algorithm for collaborative multitask reinforcement learning is proposed. The Critic stage is based on the distributed gradient temporal difference algorithm GTD(1), while the Actor stage is derived from a predefined global criterion function and consists of a complementary consensus-based exact policy gradient algorithm. A proof that the Feller-Markov properties hold for the derived algorithm at the Actor stage is derived. The weak convergence of the algorithm to the set of stationary points of an attached ODE is proved under mild conditions using the two-time-scale stochastic approximation arguments. An experimental verification of the algorithm properties is given, demonstrating its high efficiency and practical applicability.

In pursuit of reinforcement learning systems that could train in physical environments, we investigate multi-task approaches as a means to alleviate the need for massive data acquisition. In a tabular scenario where the Q-functions are collected across tasks, we model our learning problem as optimizing a higher order tensor structure. Recognizing that close-related tasks may require similar actions, our proposed method imposes a low-rank condition on this aggregated Q-tensor. The rationale behind this approach to multi-task learning is that the low-rank structure enforces the notion of similarity, without the need to explicitly prescribe which tasks are similar, but inferring this information from a reduced amount of data simultaneously with the stochastic optimization of the Q-tensor. The efficiency of our low-rank tensor approach to multi-task learning is demonstrated in two numerical experiments, first in a benchmark environment formed by a collection of inverted pendulums, and then into a practical scenario involving multiple wireless communication devices.

Robotic shepherding has become indispensable in animal husbandry and crowd management, offering a modern solution to traditional challenges. Drone Automated Shepherding leverages advanced maneuverability and an extensive field of view to improve the efficiency of these tasks, which are typically labor-intensive and time-consuming. Existing methods for managing large herds face significant challenges due to insufficient coordination among multiple drones and the complexities involved in simultaneously executing diverse shepherding tasks. This paper aims to enhance the execution of multiple shepherding tasks by optimizing drone coordination, designing optimal flight paths, and reducing flight time. To harness the potential of reinforcement learning, we develop a multi-drone collaborative shepherding environment that facilitates efficient drone training using a dense reward system. Additionally, we employ a multi-task deep reinforcement learning algorithm that enhances the sample efficiency and reward performance by leveraging shared information across tasks within this environment. Two specific tasks, driving and collecting, are used to assess the performance of our methodology. The effectiveness of our approach is measured against a classical solution named CTRL, examining metrics such as success rate, completion time, and flight path length. Results indicate that our approach significantly outperforms the CTRL in all measured metrics. Visualization of drone trajectories provides further evidence of our enhanced collaboration and efficiency in shepherding operations. Real-world experiments are conducted on a square of 300 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{m}^{3}$</tex-math></inline-formula>, where two drones utilize our method to guide four small autonomous vehicles from the starting area to the goal area within 30 seconds.

: We consider the problem of multi-task reinforcement learning (MTRL) in multiple partially observable stochastic environments. This model is appropriate for the statistical analysis of electronic systems via standoff sensing. The electronic circuit is partially observable. The framework permits life-long learning, in which the algorithm continually improves with time, as it sees more environments and systems.

In multi-task learning, there are roughly two approaches to discovering representations. The first is to discover task relevant representations, i.e., those that compactly represent solutions to particular tasks. The second is to discover domain relevant representations, i.e., those that compactly represent knowledge that remains invariant across many tasks. In this article, we propose a new approach to multi-task learning that captures domain-relevant knowledge by learning potential-based shaping functions, which augment a task’s reward function with artificial rewards. We address two key issues that arise when deriving potential functions. The first is what kind of target function the potential function should approximate; we propose three such targets and show empirically that which one is best depends critically on the domain and learning parameters. The second issue is the representation for the potential function. This article introduces the notion of \(k\)-relevance, the expected relevance of a representation on a sample sequence of \(k\) tasks, and argues that this is a unifying definition of relevance of which both task and domain relevance are special cases. We prove formally that, under certain assumptions, \(k\)-relevance converges monotonically to a fixed point as \(k\) increases, and use this property to derive Feature Selection Through Extrapolation of k-relevance (FS-TEK), a novel feature-selection algorithm. We demonstrate empirically the benefit of FS-TEK on artificial domains.

In this study, we propose a multitask reinforcement learning (RL) algorithm for foundational policy acquisition to generate novel motor skills. Learning the rich representation of the multitask policy is a challenge in dynamic movement generation tasks because the policy needs to cope with changes in goals or environments with different reward functions or physical parameters. Inspired by human sensorimotor adaptation mechanisms, we developed the learning pipeline to construct the encoder–decoder networks and network selection to facilitate foundational policy acquisition under multiple situations. First, we compared the proposed method with previous multitask RL methods in the standard multilocomotion tasks. The results showed that the proposed approach outperformed the baseline methods. Then, we applied the proposed method to the ball heading task using a monopod robot model to evaluate skill generation performance. The results showed that the proposed method was able to adapt to novel target positions or inexperienced ball restitution coefficients but to acquire a foundational policy network, originally learned for heading motion, which can generate an entirely new overhead kicking skill.

Many robotic tasks are composed of a lot of temporally correlated sub-tasks in a highly complex environment. It is important to discover situational intentions and proper actions by deliberating on temporal abstractions to solve problems effectively. To understand the intention separated from changing task dynamics, we extend an empowerment-based regularization technique to situations with multiple tasks based on the framework of a generative adversarial network. Under the multitask environments with unknown dynamics, we focus on learning a reward and policy from the unlabeled expert examples. In this study, we define situational empowerment as the maximum of mutual information representing how an action conditioned on both a certain state and sub-task affects the future. Our proposed method derives the variational lower bound of the situational mutual information to optimize it. We simultaneously learn the transferable multi-task reward function and policy by adding an induced term to the objective function. By doing so, the multi-task reward function helps to learn a robust policy for environmental change. We validate the advantages of our approach on multi-task learning and multi-task transfer learning. We demonstrate our proposed method has the robustness of both randomness and changing task dynamics. Finally, we prove that our method has significantly better performance and data efficiency than existing imitation learning methods on various benchmarks.

Multi-agent off-policy actor-critic algorithm for distributed multi-task reinforcement learning

Learning a multi-task policy is crucial in multi-agent reinforcement learning (MARL). Recent work has focused on learning in the context of online multi-task reinforcement learning, where a policy is jointly trained from scratch, aiming to generalize well to few-shot or even zero-shot tasks. However, existing online methods require tremendous interactions and are therefore unsuitable for environments where interactions are expensive. In this work, we novelly introduce the modularization for multi-task and multi-agent offline pre-training (M3) to learn high-level transferable policy representations. We claim that the discrete policy representation is critical for multi-task offline learning and accordingly leverage contexts as a task prompt to enhance the adaptability of pre-trained models to various tasks. To disentangle multiple agents of variation under heterogeneous and non-stationary properties even though they receive the same task, we employ an agent-invariant VQ-VAE to identify each of the multiple agents. We encapsulate the pre-trained model as part of an online MARL algorithm and fine-tune it to improve generalization. We also theoretically analyze the generalization error of our method. We test the proposed method on the challenging StarCraft Multi-Agent Challenge (SMAC) tasks, and empirical results show that it can achieve supreme performance in few-shot or even zero-shot settings across multiple tasks over state-of-the-art MARL methods.