RL-DOVS: Reinforcement Learning for Autonomous Robot Navigation in Dynamic Environments

Autonomous navigation in dynamic and unstructured environments presents several challenges, requiring seamless integration of real-time path planning, obstacle avoidance, and precise localization. This paper introduces a novel hybrid framework that synergizes Modified Pure Pursuit (MPP), Vector Field Histogram (VFH), and LiDAR-based Simultaneous Localization and Mapping (SLAM) to address these challenges. The MPP algorithm improves trajectory tracking through adaptive look-ahead distance tuning and curvature smoothing, minimizing oscillations in high-speed navigation. The VFH module processes LiDAR data to dynamically adjust steering directions, enabling real-time obstacle avoidance without global path replanning. Concurrently, the LiDAR-SLAM pipeline constructs high-resolution occupancy grid maps. The system's computational efficiency meets real-time operational requirements, making it suitable for industrial automation, urban logistics, and emergency response robotics. This work advances autonomous navigation by unifying adaptive control, reactive obstacle avoidance, and robust localization into a cohesive architecture, setting a benchmark for deployable solutions in complex, real-world settings.

Autonomous navigation in dynamic environments presents a significant challenge for mobile robotic systems. This paper proposes a novel approach utilizing Convolutional Neural Networks (CNNs) for multi-object detection in 3D space and 2D segmentation using bird’s eye view (BEV) maps derived from 3D Light Detection and Ranging (LiDAR) data. Our method aims to enable mobile robots to localize movable objects and their occupancy, which is crucial for safe and efficient navigation. To address the scarcity of labeled real-world datasets, a synthetic dataset based on a simulation environment is generated to train and evaluate our model. Additionally, we employ a subset of the NVIDIA r2b dataset for evaluation in the real world. Furthermore, we integrate our CNN-based detection and segmentation model into a Robot Operating System 2 (ROS2) framework, facilitating communication between mobile robots and a centralized node for data aggregation and map creation. Our experimental results demonstrate promising performance, showcasing the potential applicability of our approach in future assembly systems. While further validation with real-world data is warranted, our work contributes to advancing perception systems by proposing a solution for multi-source, multi-object tracking and mapping.

A difficult issue in mobile robot navigation or path planning in static or dynamic environment is to find the path from the starting point to the target avoiding collisions with obstacles. This paper presents a new approach for solving the problem of mobile robot navigation in dynamic environment based on tangent circle algorithm. To plan the trajectory of the robot, the algorithm relies on the creation of different types of paths between the robot and the target. In fact, during the motion path, the robot switches between a straight line and/or circular trajectory in order to avoid obstacles and reach the goal. The proposed approach is implemented on different simulation scenarios involving state or dynamic obstacles in the environment. Results show that the novel method is efficient and that the mobile robot succeeds to reach its target in a collision-free path in different cases.

Autonomous navigation in dynamic environment heavily depends on the environment and its topology. Prior knowledge of the environment is not usually accurate as the environment keeps evolving in time. Since robot is continuously evaluating the environment as it proceeds, deciding the optimal way to traverse the environment to get to the goal, computationally efficient yet mathematically adaptive navigation algorithms are needed. In this paper, a navigation scheme for mobile robot, capable of dealing with time variant environment is proposed. This approach consists of a global planner (A*) and local planner (VFH) to assure an optimal and collision-free robot motion. The algorithm is tested both in simulation and experimentation in different environments that are known to result in failures in VFH and ROS navigation stack, for comparison purposes. Overall, the algorithm enables the robot to get to the goal faster and also produces a smoother path while doing so.

Visual simultaneous localization and mapping (SLAM) is a critical technology for autonomous navigation in dynamic environments. However, traditional SLAM algorithms often struggle to maintain accuracy in highly dynamic environments, where elements undergo significant, rapid, and unpredictable changes, leading to asymmetric information acquisition. Aiming to improve the accuracy of the SLAM algorithm in a dynamic environment, a dynamic SLAM algorithm based on deep learning is proposed. Firstly, YOLOv10n is used to improve the front end of the system, and semantic information is added to each frame of the image. Then, ORB-SLAM2 is used to extract feature points in each region of each frame and retrieve semantic information from YOLOv10n. Finally, through the map construction thread, the dynamic object feature points extracted by YOLOv10n are eliminated, and the construction of a static map is realized. The experimental results show that the accuracy of the proposed algorithm is improved by more than 96% compared with the state-of-the-art ORB-SLAM2 in a highly dynamic environment. Compared with other dynamic SLAM algorithms, the proposed algorithm has improved both accuracy and runtime.

3D LiDAR-based obstacle detection and tracking for autonomous navigation in dynamic environments

Scene understanding is critical for autonomous navigation in dynamic environments. Perception tasks in this domain like segmentation and tracking are usually tackled individually. In this letter, we address the problem of 4D panoptic segmentation using LiDAR scans, which requires to assign to each 3D point in a temporal sequence of scans a semantic class and for each object a temporally consistent instance ID. We propose a novel approach that builds on top of an arbitrary single-scan panoptic segmentation network and extends it to the temporal domain by associating instances across time. We propose a contrastive aggregation network that leverages the point-wise features from the panoptic network. It generates an embedding space in which encodings of the same instance at different timesteps lie close together and far from encodings belonging to other instances. The training is inspired by contrastive learning techniques for self-supervised metric learning. Our association module combines appearance and motion cues to associate instances across scans, allowing us to perform temporal perception. We evaluate our proposed method on the SemanticKITTI benchmark and achieve state-of-the-art results even without relying on pose information.

This paper presents an adaptive Kalman filtering localization system that integrates GPS, IMU, and wheel encoder data to improve the positioning accuracy and robustness of wheeled mobile robots in complex environments. The system employs a nine-dimensional state vector to model position, velocity, attitude, and acceleration, with a Kalman filter framework for real-time state estimation. To address intermittent sensor signal loss, a covariance adaptation mechanism and dynamic gain adjustment are introduced, enhancing stability and reliability under weak or obstructed signals. Experimental evaluations under normal operation, GPS loss, IMU drift, and wheel speed error show that the proposed method achieves average positioning errors of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.25 \mathrm{m}, 2.05 \mathrm{m}, 1.72 \mathrm{m}$</tex>, and 1.45 m respectively, compared to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.98 \mathrm{m}, 3.47 \mathrm{m}, 3.01 \mathrm{m}$</tex>, and 2.65 m using a traditional Kalman filter. Furthermore, the system maintains error bounds within 1 m over 1 minute and 4 m over 4 minutes during GPS outages. These results demonstrate superior trajectory tracking and error convergence, outperforming fixed-parameter filters and offering strong adaptability for long-term autonomous navigation in dynamic environments.

In order to enhance the behavior of autonomous service robots, we are exploring multiple paradigms for their planning and execution strategy (the way of interleaving the planning, selection and execution of actions). In this paper we focus on continuous proactive planning with multiple hypotheses in order to continuously generate multiple solution-plans from which an action can be selected when appropriate. To illustrate the concepts, we develop how it could be used for autonomous navigation in dynamic environments, and analyze the tests we realized with several instantiations. We also discuss several aspects and concerns about the proposed strategy, and how integrating more semantic information could enhance the capabilities of service robots for real-world applications.

Deep learning has made it possible for reinforcement learning to scale to issues that were thought insurmountable, for instance learning to play video games by only looking at the pixels. Robotics uses deep reinforcement learning methods that enable control rules to be learnt through sensor inputs in the actual environment. Despite analytical methods are capable of solving the majority of offline compression problems, their online applications have proven to be significantly more difficult, and the effectiveness of the current approaches is frequently not adequate. By combining visual evidence of real-time packaging with distributional knowledge of random issue parameters, we enhance a previous deep reinforcement learning hyper-heuristic approach in this research. The acquired packing tactics are also better understood using a new graphic presentation, which can expose more details than the conventional landscape analysis. In this study, we start with an overview of reinforcement learning for robot navigation as a whole before moving on to the two primary categories of value-based and policy-based approaches. The deep the system, trust area policy optimization, and other key profound reinforcement learning methods will be covered in our research. In addition, we highlight the unique advantages of deep neural networks while emphasizing visual attention through learning by reinforcement. Finally, we outline a number of active research areas in the domain. Recent research has shown that deep reinforcement learning holds great promise for overcoming issues in video games with sequential decision-making situations. A category of uncertain or unpredictable efficiency problems can be solved by selecting or developing algorithms using a broad search architecture known as a hyper-heuristic. The proposed method can serve as a significant reference to tackle other comparable combinatorial optimum issues because visual layout inputs will facilitate learning, as online packing challenges are routinely provided in production situations.

This paper proposes a novel approach, Simultaneous Path Planning and Topological Mapping (SP2ATM), to address the problem of path planning by registering the topology of the perceived dynamic environment as opposed to the conventional grid representation. The local topology is encoded, concurrent and incremental with path planning, by extracting only the admissible free space. The resulting Admissible Space Topological Map (ASTM) then serves as the minimum information to facilitate path planning in the 3D configuration space. Experimental results obtained from our mobile robot X1 in a complex planar environment, validates completeness and optimality of the algorithm.

An end-to-end approach to autonomous navigation that is based on deep reinforcement learning (DRL) with a survival penalty function is proposed in this paper. Two actor-critic (AC) frameworks, namely, deep deterministic policy gradient (DDPG) and twin-delayed DDPG (TD3), are employed to enable a nonholonomic wheeled mobile robot (WMR) to perform navigation in dynamic environments containing obstacles and for which no maps are available. A comprehensive reward based on the survival penalty function is introduced; this approach effectively solves the sparse reward problem and enables the WMR to move toward its target. Consecutive episodes are connected to increase the cumulative penalty for scenarios involving obstacles; this method prevents training failure and enables the WMR to plan a collision-free path. Simulations are conducted for four scenarios-movement in an obstacle-free space, in a parking lot, at an intersection without and with a central obstacle, and in a multiple obstacle space-to demonstrate the efficiency and operational safety of our method. For the same navigation environment, compared with the DDPG algorithm, the TD3 algorithm exhibits faster numerical convergence and higher stability in the training phase, as well as a higher task execution success rate in the evaluation phase.

Mobile robots have been making a significant contribution to the advancement of many sectors including automation of mining, space, surveillance, military, health, agriculture and many more. Safe and efficient navigation is a fundamental requirement of mobile robots, thus, the demand for advanced algorithms rapidly increased. Mobile robot navigation encompasses the following four requirements: perception, localization, pathplanning and motion control. Among those, path-planning is a vital part of a fast, secure operation. During the last couple of decades, many path-planning algorithms were developed. Despite most of the mobile robot applications being in dynamic environments, the number of algorithms capable of navigating robots in dynamic environments is limited. This paper presents a qualitative comparative study of the up-to-date mobile robot pathplanning methods capable of navigating robots in dynamic environments. The paper discusses both classical and heuristic methods including artificial potential field, genetic algorithm, fuzzy logic, neural networks, artificial bee colony, particle swarm optimization, bacterial foraging optimization, ant-colony and Agoraphilic algorithm. The general advantages and disadvantages of each method are discussed. Furthermore, the commonly used state-of-the-art methods are critically analyzed based on six performance criteria: algorithm&#x0027;s ability to navigate in dynamically cluttered areas, moving goal hunting ability, object tracking ability, object path prediction ability, incorporating the obstacle velocity in the decision, validation by simulation and experimentation. This investigation benefits researchers in choosing suitable path-planning methods for different applications as well as identifying gaps in this field.

The rapidly-exploring random trees (RRT) is a sampling-based path planner which utilizes simultaneously kinematics and dynamics of a robot. However, since the RRT has produced a robot path without taking the existence of dynamic obstacles into consideration, RRT-based navigation has the risk of a collision with dynamic obstacles. We proposed a path replanning technique for the RRT applied to robot navigation in dynamic environments, which is named grafting. The proposed technique replans a safe and efficient path in real time instead of the original path which may cause a collision with dynamic obstacles. Moreover, the replanned path can be easily merged into the original RRT path because the grafting technique preserves the property of the RRT. The grafting technique was tested by simulations in various dynamic environments, which revealed that the grafting technique was capable of replanning a safe and efficient path for RRT-based navigation in real time.

Path planning in the presence of dynamic obstacles is a fundamental problem in robotics with widespread applications. A typical approach to such problems is that a robot predicts the trajectories of dynamic obstacles, and plans its path while avoiding them. Such a formulation becomes limiting though for scenarios where an agent cannot complete its task efficiently, without disrupting the movement of dynamic obstacles. For example, when merging in heavy traffic or navigating through crowded corridors. In this paper, we propose a paradigm for planning in dynamic environments, called Disruption-Limited Planning (DLP), that allows a robot to disrupt the motions of dynamic obstacles in order to accomplish its task. DLP builds on the premise that while a robot may have to disrupt others’ trajectories to achieve its goals, it should try to limit the disruption. DLP assumes that it can estimate others’ response to its own actions/plans, and plans its own path while ensuring that no other agents’ disrupted trajectory cost gets worse than w-times their initial trajectory costs. While our formulation is motivated by the Stackelberg competitions, we show that DLP can be both more expressive and computationally more efficient compared to a Stackelberg planner. We present DLP paradigm, develop its efficient implementation based on A*, analyze its theoretical properties, and apply it to multiple planning in dynamic environment problems, including x,y,time planning, planning for self-driving, and planning for arm manipulation. We compare DLP with purely altruistic, purely egocentric, and optimal Stackelberg planners, demonstrating the efficacy of DLP over these alternatives.