Reinforcement learning transfer based on subgoal discovery and subtask similarity

Conventional penalty function-based safe reinforcement learning (RL) algorithms often handle safety and policy difference constraints separately, lacking specialized mechanisms to address scenarios where both constraints are violated simultaneously. This limitation sometimes results in severe performance degradation, underscoring the need for further investigation. To address this issue, a simple yet efficient safe RL algorithm is proposed in this work called coupled penalty-based safe policy optimization (CPSPO). CPSPO operates within the first-order local policy search framework, ensuring ease of implementation in practice. To enhance behavior correction when both constraints are violated, CPSPO introduces coupled penalties, considering the cost and Kullback-Leibler (KL)-divergence constraint violation situation simultaneously, effectively improving both constraints handling. Moreover, instead of the conventional policy ratio clipping mechanism, CPSPO directly incorporates the KL-divergence constraint in the loss function design, providing a more intuitive and practical approach to proximal safe policy optimization. Extensive experiments demonstrate that CPSPO outperforms mainstream safe RL algorithms in reward optimization and safety constraint satisfaction.

Arbitration between model-based (MB) and model-free (MF) reinforcement learning (RL) is key feature of human reinforcement learning. The computational model of arbitration control has been demonstrated to outperform conventional reinforcement learning algorithm, in terms of not only behavioral data but also neural signals. However, this arbitration process does not take full account of contextual changes in environment during learning. By incorporating a Dirichlet process Gaussian mixture model into the arbitration process, we propose a meta-controller for RL that quickly adapts to contextual changes of environment. The proposed model performs better than a conventional model-free RL, model-based RL, and arbitration model.

In this article, we explore how to optimize task allocation for robot swarms in dynamic environments, emphasizing the necessity of formulating robust, flexible, and scalable strategies for robot cooperation. We introduce a novel framework using a decentralized partially observable Markov decision process (Dec-POMDP), specifically designed for distributed robot swarm networks. At the core of our methodology is the local information aggregation multiagent deep deterministic policy gradient (LIA-MADDPG) algorithm, which merges centralized training with distributed execution. During the centralized training phase, a local information aggregation (LIA) module is meticulously designed to gather critical data from neighboring robots, enhancing decision-making efficiency. In the distributed execution phase, a strategy improvement method is proposed to dynamically adjust task allocation based on changing and partially observable environmental conditions. Our empirical evaluations show that the LIA module can be seamlessly integrated into various centralized training and decentralized execution (CTDE)-based multiagent reinforcement learning (MARL) methods, significantly enhancing their performance. Additionally, by comparing LIA-MADDPG with six conventional reinforcement learning algorithms and a heuristic algorithm, we demonstrate its superior scalability, rapid adaptation to environmental changes, and ability to maintain both stability and convergence speed. These results underscore LIA-MADDPG's outstanding performance and its potential to significantly improve dynamic task allocation in robot swarms through enhanced local collaboration and adaptive strategy execution.

The purpose of transfer learning is to utilize the knowledge gained from the existing (source) domain to enhance the performance on a distinct but related (target) domain. Existing works on transfer learning are not capable of optimizing different quality measures (components) such as minimizing the marginal distribution, minimizing the conditional distribution, maximizing the target domain variance, modeling the manifold by utilizing the common geometric properties in the source as well as the target domain at the same time. Moreover, existing transfer learning methods use conventional approaches to determine the appropriate values of their parameters, which is very hectic and time-consuming. Therefore, in order to overcome the drawbacks of existing approaches, we propose a Particle Swarm Optimization based Parameter Selection Approach for Unsupervised Discriminant Analysis (UDATL-PSO) in transfer learning framework. In UDATL-PSO, all the quality measures are considered at the same time, as well as the PSO approach has been used to select the best values of their parameters. Extensive experiments on various transfer learning tasks show that the proposed method has a significant influence on state-of-the-art methods.

In this paper, we apply reinforcement learning(RL) to control a 3-Dof manipulator. Learning is achieved by using an on-policy temporal difference(TD) learning. In order to control the manipulator, the TD agents are in charge of the movements of the robot by adjusting each angle. We first implement the SARSA algorithm by repeatedly updating a hash-table with the current and next state-action values, and the received rewards. However, since the conventional RL algorithms have issues of the heavy computational load, we propose RL algorithms applying multi-agents approach with an assumption and vector values for states. We state the result of the classical RL algorithm and compare it with our approach. For the simulation, we test a three-link planar arm for two cases: positioning the end-effector to a fixed goal point and random goal points. The simulation results show that the robot completes the mission successfully with the improvement in the computational load.

The advancement of autonomous driving technology is becoming increasingly vital in the modern technological landscape, where it promises notable enhancements in safety, efficiency, traffic management, and energy use. Despite these benefits, conventional deep reinforcement learning algorithms often struggle to effectively navigate complex driving environments. To tackle this challenge, we propose a novel network called DynamicNoise, which was designed to significantly boost the algorithmic performance by introducing noise into the deep Q-network (DQN) and double deep Q-network (DDQN). Drawing inspiration from the NoiseNet architecture, DynamicNoise uses stochastic perturbations to improve the exploration capabilities of these models, thus leading to more robust learning outcomes. Our experiments demonstrated a 57.25% improvement in the navigation effectiveness within a 2D experimental setting. Moreover, by integrating noise into the action selection and fully connected layers of the soft actor–critic (SAC) model in the more complex 3D CARLA simulation environment, our approach achieved an 18.9% performance gain, which substantially surpassed the traditional methods. These results confirmed that the DynamicNoise network significantly enhanced the performance of autonomous driving systems across various simulated environments, regardless of their dimensionality and complexity, by improving their exploration capabilities rather than just their efficiency.

Enhanced Dynamic Deep Q-Network for Federated Learning scheduling policies on IoT devices using explanation-driven trust

Reinforcement learning (RL) is a widely investigated intelligent algorithm and proved to be useful in the wireless communication area. However, for optimization problems in large-scale multi-cell networks whose dimension increases exponentially, it is unrealistic to employ a conventional centralized RL algorithm and make decisions for the entire network. Multi-agent RL, which allows distribute decision-making, is expected to solve the scalability problem but with performance issues due to the unknown global information, i.e., non-stationary environment. In this paper, we propose a parameter-sharing multi-agent RL for grouping decisions of coordinated multi-point in a large-scale network, where agents jointly serve users to enhance the cell-edge service. By sharing information via parameters, our theoretical and simulation results show that parameter sharing can largely benefit the multi-agent algorithm with convergence proof and convergence speed analysis. To reduce the effect of biased local heterogeneous experience, we also propose a transfer learning method for the parameter sharing process, whose performance of transfer learning algorithms is verified by the simulation results.

Transfer learning is the task of leveraging the information from labeled examples in some domains to predict the labels for examples in another domain. It finds abundant practical applications, such as sentiment prediction, image classification and network intrusion detection. In this paper, we propose a graph-based transfer learning framework. It propagates the label information from the source domain to the target domain via the example-feature-example tripartite graph, and puts more emphasis on the labeled examples from the target domain via the example-example bi-partite graph. Our framework is semi-supervised and non-parametric in nature and thus more flexible. We also develop an iterative algorithm so that our framework is scalable to large-scale applications. It enjoys the theoretical property of convergence. Compared with existing transfer learning methods, the proposed framework propagates the label information to both the features irrelevant to the source domain and the unlabeled examples in the target omain via the common features in a principled way. Experimental results on 3 real data sets demonstrate the effectiveness of our algorithm.

Transfer learning using computational intelligence: A survey

Transfer learning is the one of methods to improve the efficiency of training for reinforcement learning (RL) agents. That is the method of transferring knowledge which is acquired in the past tasks or given by experts to similar tasks. However, if the RL agents don't retain knowledge that is effective for improving learning speed or quality, there is a problem in that it causes negative transfer, which reduces learning efficiency. This study proposes a method for deleting knowledge which are less the number of state observation through trial and error by RL agents. Our proposed method was applied to path planning for a two-wheeled mobile robot to verify its effectiveness, and it was confirmed that the proposed method reduced the amount of knowledge by 98% and the learning time by 32%.

Sequential decision-making over long timescales and in complex task environments is an important problem in Artificial Intelligence (AI). An effective approach to tackle this problem is to autonomously decompose a long-horizon task into a sequence of simpler subtasks or subgoals. We refer to this approach as Autonomous Task Decomposition (ATD) in the thesis and study it for multi-agent coordination using model-free Hierarchical Reinforcement Learning (HRL), single-agent goal-reaching using model-free HRL, and single-agent goal-reaching using model-based planning. The objective of the thesis is to develop novel methods to address three important challenges related to ATD, which are as follows: 1. Effective multi-agent HRL under sparse global rewards and complex inter-dependencies among agents. 2. Efficient unification of autonomous subgoal discovery and single-agent HRL without slow learning. 3. Learning models for planning-based ATD that produce more rewarding and feasible plans. In this regard, the thesis introduces three novel ATD methods as follows: 1. Inter Subtask Empowerment based Multi-agent Options (ISEMO) is introduced for effective multi-agent HRL by using auxiliary rewards that capture the inter-dependencies among HRL agents and their (handcrafted) subtasks. ISEMO leads to better coordinated performance of the inter-dependent agents on a complex Search & Rescue task, compared to a standard multi-agent HRL method. 2. End-to-End Hierarchical Reinforcement Learning with Integrated Discovery of Salient Subgoals (LIDOSS) is introduced for efficient unification of subgoal discovery and HRL for single-agent goal-reaching, by using a probability-based subgoal discovery heuristic integrated with the subgoal selection policy. LIDOSS accelerates end-to-end learning and leads to higher goal-reaching success rates compared to a state-of-the-art HRL method. 3. Finally, Learning Subgoal Graph using Value-based Subgoal Discovery and Automatic Pruning (LSGVP) is introduced to learn subgoal graph-based planning models that produce more rewarding and feasible plans for single-agent goal-reaching. LSGVP uses cumulative reward-based subgoal discovery and automatic pruning of erroneous connections in the subgoal graph. It achieves higher positive cumulative rewards and higher success rates compared to other state-of-the-art subgoal graph-based planning methods, while also being more data-efficient than model-free HRL.

Reinforcement learning is a learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in reinforcement learning upon the fast development of deep neural networks. Along with the promising prospects of reinforcement learning in numerous domains such as robotics and game-playing, transfer learning has arisen to tackle various challenges faced by reinforcement learning, by transferring knowledge from external expertise to facilitate the efficiency and effectiveness of the learning process. In this survey, we systematically investigate the recent progress of transfer learning approaches in the context of deep reinforcement learning. Specifically, we provide a framework for categorizing the state-of-the-art transfer learning approaches, under which we analyze their goals, methodologies, compatible reinforcement learning backbones, and practical applications. We also draw connections between transfer learning and other relevant topics from the reinforcement learning perspective and explore their potential challenges that await future research progress.

Surface electromyogram (sEMG)-based human-computer interface (HCI) is an effective tool for detecting human movements. Because sEMG-based motion recognition usually requires prolonged data measurements from the user (target), transfer learning reusing pre-measured (source) data from other users and pre-trained classifiers can be applied to sEMG data to reduce the measurement time. However, little knowledge is available regarding the combination of transfer learning methods and classifiers in sEMG data applications. Thus, we investigated the classification accuracy of data- and parameter-space-based transfer learning with shallow or deep classifiers in cross-subject sEMG classification. The dataset contains eight classes of forearm motions recorded from 25 volunteer participants. We used a support vector machine (SVM) as a shallow classifier as well as a deep neural network architecture, referred to as an artificial neural network (ANN), as a deep classifier. In addition, we used style transfer mapping (STM) as a data-space-based transfer learning method and fine-tuning (FT) as a parameter-space-based transfer learning method. Consequently, the classification accuracy of the ANN was higher than that of the SVM, regardless of the combinational use of transfer learning. STM and FT significantly improved the classification accuracy compared with non-transfer cases regardless of the classifier (note that FT can only be used with the ANN). In particular, the combined use of FT and the ANN yielded the best accuracy. These findings suggest that parameter-space-based transfer learning and deep classifiers are suitable for cross-subject sEMG classification. The combined use of parameter-space-based transfer learning and deep classifiers can effectively reduce the data measurement time of sEMG-based HCI applications.

The rapid proliferation of electric vehicles (EVs) presents substantial challenges to effective charging and discharging regulation within distribution networks, particularly under increasingly diverse and dynamic regulatory scenarios. Conventional reinforcement learning (RL) methods, typically limited by task-specific training, struggle to generalize effectively across varying regulatory objectives and heterogeneous spatial EV distributions. To address these limitations, this study proposes a novel offline meta-reinforcement learning (Meta-RL) framework designed to enhance the adaptability of policy networks through a dedicated meta-network for task-specific identification. Specifically, we propose an innovative meta-network enhanced by large language models (LLMs), which simultaneously captures dynamic physical trajectories derived from historical operational data and encodes expert dispatch knowledge extracted from textual instructions. By integrating these physical and semantic modalities, the meta-network effectively generates task-specific embeddings, enabling the downstream policy network to rapidly adapt and formulate context-aware regulatory strategies. Comparative experiments demonstrate that the proposed offline Meta-RL framework achieves a 47.2% average reduction in operation cost compared to conventional single-task RL methods. Ablation studies further confirm that integrating physical and semantic information significantly improves task recognition accuracy and decision-making quality, reducing adaptation time by 68.8% and average operation costs by 18.4%.