Towards more effective skill discovery in reinforcement learning by incorporating state reachability.

Reinforcement learning (RL) is one of the popular methods for intelligent control and decision making in the field of robotics recently. The goal of RL is to learn an optimal policy of the agent by interacting with the environment via trail and error. There are two main algorithms for RL problems, including model-free and model-based methods. Model-free RL is driven by historical trajectories and empirical data of the agent to optimize the policy, which needs to take actions in the environment to collect the trajectory data and may cause the damage of the robot during training in the real environment. The main different between model-based and model-free RL is that a model of the transition probability in the interaction environment is employed. Thus the agent can search the optimal policy through internal simulation. However, the model of the transition probability is usually estimated from historical data in a single environment with statistical errors. Therefore, an issue is faced by the agent is that the optimal policy is sensitive to perturbations in the model of the environment which can lead to serious degradation in performance. Robust RL aims to learn a robust optimal policy that accounts for model uncertainty of the transition probability to systematically mitigate the sensitivity of the optimal policy in perturbed environments. In this overview, we begin with an introduction to the algorithms in RL, then focus on the model uncertainty of the transition probability in robust RL. In parallel, we highlight the current research and challenges of robust RL for robot control. To conclude, we describe some research areas in robust RL and look ahead to the future work about robot control in complex environments.

To address the limitations of the Deep Deterministic Policy Gradient (DDPG) in robot path planning, we propose an improved DDPG method that integrates kinematic analysis and D* algorithm, termed D*-KDDPG. Firstly, the current work promotes the reward function of DDPG to account for the robot’s kinematic characteristics and environment perception ability. Secondly, informed by the global path information provided by the D* algorithm, DDPG successfully avoids getting trapped in local optima within complex environments. Finally, a comprehensive set of simulation experiments is carried out to investigate the effectiveness of D*-KDDPG within various environments. Simulation results indicate that D*-KDDPG completes strategy learning within only 26.7% of the training steps required by the original DDPG, retrieving enhanced navigation performance and promoting safety. D*-KDDPG outperforms D*-DWA with better obstacle avoidance performance in dynamic environments. Despite a 1.8% longer path, D*-KDDPG reduces navigation time by 16.2%, increases safety distance by 72.1%, and produces smoother paths.

Abstract. In the last few years, the 3D GIS domain has developed rapidly, and has become increasingly accessible to different disciplines. 3D Spatial analysis of Built-up areas seems to be one of the most challenging topics in the communities currently dealing with spatial data. One of the most basic problems in spatial analysis is related to visibility computation in such an environment. Visibility calculation methods aim to identify the parts visible from a single point, or multiple points, of objects in the environment. In this work, we present a unique method combining visibility analysis in 3D environments with dynamic motion planning algorithm, named Visibility Velocity Obstacles (VVO) with Markov process defined as spatial visibility analysis for routes in 3D dense city environment. Based on our VVO analysis, we use Reinforcement Learning (RL) method in order to find an optimal action policy in dense 3D city environment described as Markov decision process, navigating in the most visible routes. As far as we know, we present for the first time a Reinforcement Learning (RL) solution to the visibility analysis in 3D dense environment problem, generating a sequence of viewpoints that allows an optimal visibility in different routes in urban city. Our analysis is based on fast and unique solution for visibility boundaries, formulating the problem with RL methods.

Reinforcement Learning (RL) has demonstrated significant potential in fields such as robotic control, autonomous driving, and financial decision-making. However, in complex environments, RL still faces challenges in stability and convergence. This study addresses this core issue by establishing a theoretical analysis framework that combines stochastic approximation theory and Lyapunov stability theory to rigorously analyze the convergence conditions and stability bounds of various RL algorithms in complex environments. Based on the theoretical analysis, we propose an Adaptive Stable RL (ASRL) algorithm, which employs dynamic regularized policy optimization and robust value function estimation to effectively mitigate policy oscillation and training divergence. Systematic experiments conducted in OpenAI Gym, MuJoCo, and customized non-stationary environments demonstrate that ASRL significantly outperforms baseline algorithms such as PPO and SAC in terms of convergence speed, final performance, and stability. Additionally, in industrial control and robotic navigation case studies, ASRL exhibits excellent adaptability and robustness. This research not only provides theoretical support for RL in complex environments but also offers optimization guidelines for algorithm design in practical applications.

In reinforcement learning, the reward function has a significant impact on the performance of the agent. However, determining the appropriate value of this reward function requires many attempts and trials. Although many automated reinforcement learning methods have been proposed to find an appropriate reward function, their proof is lacking in complex environments such as quadrupedal locomotion. In this paper, we propose a method to automatically tune the scale of the dominant reward functions in reinforcement learning of a quadrupedal robot. Reinforcement learning of the quadruped robot is very sensitive to the reward function, and recent outstanding research results have put a lot of effort into reward shaping. In this paper, we propose an automated reward shaping method that automatically adjusts the reward function scale appropriately. We select some dominant reward functions, arrange their weights in a certain unit, and then calculate their gait scores so that we can select the agent with the highest score. This gait score was defined to reflect the stable walking of the quadrupedal robot. Additionally, quadrupedal locomotion learning requires reward functions of different scales depending on the robot’s size and shape. Therefore, we evaluate the performance of the proposed method on two different robots.

Transfer learning is the one of methods to improve the efficiency of training for reinforcement learning (RL) agents. That is the method of transferring knowledge which is acquired in the past tasks or given by experts to similar tasks. However, if the RL agents don't retain knowledge that is effective for improving learning speed or quality, there is a problem in that it causes negative transfer, which reduces learning efficiency. This study proposes a method for deleting knowledge which are less the number of state observation through trial and error by RL agents. Our proposed method was applied to path planning for a two-wheeled mobile robot to verify its effectiveness, and it was confirmed that the proposed method reduced the amount of knowledge by 98% and the learning time by 32%.

Autonomous marine agents find extensive applications in environmental data collection, naval security, and exploration of harsh ocean regions. As intelligent agents, they must perform onboard routing, collect data about their surroundings and update their route to minimize mission travel time, energy, or data collection. While Markov Decision Processes (MDPs) and Reinforcement Learning (RL) are often used for path planning, they are computationally expensive for onboard routing as they need in-mission re-planning. In the present paper, we develop a novel, deep learning method based on the decision transformers for optimal path planning and onboard routing of autonomous marine agents. The transformer architectures convert the RL-based optimal path planning problem into a supervised learning problem via sequence modeling. Before the mission, during the offline planning phase, the environment is first modeled as a stochastic dynamic ocean flow with dynamically orthogonal flow equations. A training dataset for the transformer model is created by solving the stochastic dynamically orthogonal Hamilton-Jacobi level set partial differential equations or a dynamic programming solution for MDPs. These paths are then processed to obtain sequences of states, actions and returns for our transformer models, where the agent's state is typically its spatio-temporal coordinate and other collectible data. We propose and analyze multiple state modeling choices against the agent's state estimation capabilities and scenarios with multiple target locations. We demonstrate that (i) a trained agent learns to infer the surrounding flow and perform optimal onboard routing when the agent's state estimation is accurate,(ii) specifying the target locations (in case of multiple targets) as a part of the state enables a trained agent to route itself to the correct destination, and (iii) a trained agent is robust to limited noise in state transitions and is capable of reaching target locations in completely new flow scenarios. We extensively showcase end-to-end planning and onboard routing in various canonical and idealized ocean flow scenarios. We analyze the predictions of the transformer models and explain the inner mechanics of learning through a novel visualization of self-attention of actions and states on the trajectories.

Summary form only given. Reinforcement learning refers to improving performance through trial-and-error experience. Although, modern computational approaches to reinforcement learning were inspired by animal learning, they have now branched out in a very several different directions. Some researchers are interested in finding high-quality approximate solutions to large-scale stochastic planning problems that are important for industry and government. Others are pursuing the goal of building intelligent, resourceful autonomous agents that, like animals, can succeed while acting in real-time in complex environments. While these goals have much in common - and they both involve the real world - they represent two very different perspectives on reinforcement learning and related methods. After first reviewing the major elements of modern reinforcement learning and its relationship to optimal control, other types of machine learning, and to neuroscience, I present several striking example applications and describe some of the latest research directed toward scaling up to ever more complex problems. I conclude by laying out a view of the future of reinforcement learning research, emphasizing that the issues that need to be addressed depend strongly on which type of reinforcement learning one has in mind.

Autonomous on-ramp merging requires a safe and efficient decision and planning framework to navigate dynamic and complex traffic scenarios. Reinforcement Learning (RL) offers adaptability in such environments but struggles to ensure safety in unseen situations, whereas Model Predictive Control (MPC) can enforce safety constraints but relies on accurate models. To leverage the strengths of both, we propose a hierarchical Safety-Augmented RL and MPC integration approach, i.e., SARMI, for decision and planning in the on-ramp merging scenario. The high-level decision-making layer employs a discrete Soft Actor-Critic (SAC-Discrete) algorithm enhanced with the Augmented Lagrangian method to generate actions. MPC generates reference trajectories based on RL actions and feeds predicted states back to RL for reward and cost design, ensuring consistency between RL and MPC. Our key innovations include: i) incorporating a Gaussian-based risk field model into the cost design of constrained RL, which quantifies collision risk based on MPC-predicted states, enabling agents to make proactive decisions; ii) an Augmented Lagrangian SAC-Discrete method with barrier-like quadratic penalties to promote compliance with safety constraints and alleviate oscillations in dual gradient descent; iii) theoretical analysis proving the equivalence between the optimal solutions of the primal and dual problems in Augmented Lagrangian SAC-Discrete and iv) a dual safety mechanism combining action masking (to filter invalid actions) and action shielding (to replace unsafe actions in RL), enhancing safety during the exploration and execution stages, respectively. Experiments demonstrate the superiority of our method over the baseline SAC-Discrete, showing improved safety and efficiency.

In this paper we consider autonomous control of a snake-like robot using reinforcement learning. Conventional methods of reinforcement learning have significant problems in practical use. That is curse of dimensionally and lack of generality. To solve these problems, we focus on design of the mechanical body of the snake-like robot, and abstract necessary small state-action space from complex environments by utilizing the function of the body. To discuss the function of the body, experiments have been conducted and transition probability has been identified. As the result, we confirmed that by the function of the body, learning machine can observe different complex environments as similar simple environments.

Reinforcement Learning (RL) Markov Decision Processes is studied with an emphasis on the well-studied exploration problem. We first formulate and discuss a definition of “efficient” algorithms that is termed Probably Approximately Correct (PAC) in RL. Next we provide general sufficient conditions for such an algorithm that applies to several different modeling assumptions. The conditions can be used to demonstrate that efficient learning is possible in finite MDPs, with either a model-based or model-free approach, in factored MDPs, and in continuous MDPs with linear dynamics. In the reinforcement-learning (RL) problem (Sutton & Barto 1998), an agent acts in an unknown or incompletely known environment with the goal of maximizing an external reward signal. In the most standard mathematical formulation of the problem, the environment is modeled as a Markov Decision Process (MDP) and the goal of the agent is to obtain near-optimal discounted return. Over the years, many algorithms have been proposed for this problem, but analyses of their performances have been relatively scarce. In fact, until recently, most theoretical guarantees have been that certain algorithms will discover an optimal policy in the limit, after an infinite amount of experience. In contrast, several attempts have been made to study “Probably Approximately Correct” or PAC-MDP algorithms, which exhibit near-optimal behavior in polynomial time and experience. This paper discusses several extensions of those results. We present a theorem that provides sufficient conditions for an algorithm to be PAC-MDP. We examine these conditions and show how they can be applied to prove that efficient learning is possible in three interesting scenerios: finite MDPs (i.e. the “Tabular case”),

Traditional deep reinforcement learning methods suffer from slow convergence speeds and poor adaptability in complex environments and are prone to falling into local optima in AGV system applications. To address these issues, in this paper, an adaptive path planning algorithm with an improved Deep Q Network algorithm called the B-PER DQN algorithm is proposed. Firstly, a dynamic temperature adjustment mechanism is constructed, and the temperature parameters in the Boltzmann strategy are adaptively adjusted by analyzing the change trend of the recent reward window. Next, the Priority experience replay mechanism is introduced to improve the training efficiency and task diversity through experience grading sampling and random obstacle configuration. Then, a refined multi-objective reward function is designed, combined with direction guidance, step punishment, and end point reward, to effectively guide the agent in learning an efficient path. Our experimental results show that, compared with other algorithms, the improved algorithm proposed in this paper achieves a higher success rate and faster convergence in the same environment and represents an efficient and adaptive solution for reinforcement learning for path planning in complex environments.

Task allocation for Unmanned Aerial Vehicle (UAV) swarms is a complex challenge. Effective distribution of tasks among multiple UAVs necessitates the optimization of conflicting objectives, such as task balance, flight distance, cost, and target benefits. This multi-objective optimization problem is inherently intricate due to strong interdependencies, nonlinear relationships, and dynamic environments. Traditional methods, including heuristic algorithms, supervised machine learning, and basic reinforcement learning mechanisms, often suffer from low efficiency and strict requirements for data sources or supervision mechanisms. They also struggle to adapt to problems with large discrete action spaces. To address this challenge, this paper proposes a novel approach that integrates swarm intelligence simulation with reinforcement learning. The core of this approach is a reinforcement learning algorithm based on the Deep Deterministic Policy Gradient (DDPG) framework. A key innovation is a customized DDPG variant specifically designed to handle the large, discrete action spaces inherent in UAV task allocation. Additionally, a behavior tree model is employed to simulate UAV swarm interactions, providing a realistic and verifiable environment for evaluating task allocation strategies. This simulation generates a rich dataset for training and testing reinforcement learning algorithms. The combined approach efficiently addresses the multi-objective optimization problem, enabling optimal task distribution for numerous UAVs. Compared to existing heuristic methods, the proposed approach demonstrates superior computational efficiency and solution quality, while maintaining reasonable convergence stability and environmental adaptability. Simulation-based validation ensures the approach’s robustness in dynamic environments, a critical factor for practical applications. Overall, the integration of simulation and reinforcement learning provides a powerful framework for tackling the complex task allocation problem in UAV swarms.

Reinforcement learning (RL) can provide a basic framework for autonomous robots to learn to control and maximize future cumulative rewards in complex environments. To achieve high performance, RL controllers must consider the complex external dynamics for movements and task (reward function) and optimize control commands. For example, a robot playing tennis and squash needs to cope with the different dynamics of a tennis or squash racket and such dynamic environmental factors as the wind. In addition, this robot has to tailor its tactics simultaneously under the rules of either game. This double complexity of the external dynamics and reward function sometimes becomes more complex when both the multiple dynamics and multiple reward functions switch implicitly, as in the situation of a real (multi-agent) game of tennis where one player cannot observe the intention of her opponents or her partner. The robot must consider its opponent's and its partner's unobservable behavioral goals (reward function). In this article, we address how an RL agent should be designed to handle such double complexity of dynamics and reward. We have previously proposed modular selection and identification for control (MOSAIC) to cope with nonstationary dynamics where appropriate controllers are selected and learned among many candidates based on the error of its paired dynamics predictor: the forward model. Here we extend this framework for RL and propose MOSAIC-MR architecture. It resembles MOSAIC in spirit and selects and learns an appropriate RL controller based on the RL controller's TD error using the errors of the dynamics (the forward model) and the reward predictors. Furthermore, unlike other MOSAIC variants for RL, RL controllers are not a priori paired with the fixed predictors of dynamics and rewards. The simulation results demonstrate that MOSAIC-MR outperforms other counterparts because of this flexible association ability among RL controllers, forward models, and reward predictors.

Technological advancements in robotics and cognitive science are contributing to the development of the field of cognitive robotics. Modern robotic platforms are able to exhibit the ability to learn and reason about complex tasks and to follow behavioural goals in complex environments. Nevertheless, many challenges still exist. One of these great challenges is to equip these robots with cognitive systems that allow them to deal with less constrained situations, beyond constrained scenarios as in industrial robotics. In this work we explore the application of the Reinforcement Learning (RL) paradigm to study the autonomous development of robot controllers without a priori supervised learning. Such a model-based RL architecture is discussed for the cognitive implications of applying RL in humanoid robots. To this end we show a developmental framework for RL in robotics and its implementation and testing for the iCub robotic platform in two novel experimental scenarios. In particular we focus on iCub simulation experiments with comparisons between internal perception-based reward signals and external ones, in order to compare learning performance of the robot guided by its own perception of action's outcomes with the one when the robot has its actions externally evaluated.