Reinforcement learning for autonomous vehicle navigation in Urban environments

Global navigation satellite systems (GNSS) are often integrated with inertial measurement units (IMU) for navigation in urban environments. The ability of these integrated systems to achieve precise positioning and navigation is highly dependent on accurate time synchronization. While the current time synchronization algorithms are capable of achieving millisecond-level synchronization in open environments, their performance deteriorates significantly in complex urban environments due to the impact on GNSS measurements of non-line-of-sight (NLOS) signals and multipath effects. Here we propose a loosely coupled GNSS/IMU integration time synchronization error modeling and compensation algorithm to tackle this problem in the context of urban vehicle navigation. In the algorithm, the time synchronization error is augmented as a state variable in the Robust Extended Kalman Filter (REKF), with the robustness factor threshold being dynamically adjusted based on the time synchronization error. Furthermore, we have designed a system time synchronization detection scheme based on two detection factors, with the aim of achieving high-precision GNSS/IMU time synchronization in complex urban environments. Experimental results show that the proposed algorithm synchronizes data time to the millisecond level in urban positioning environments. This represents an improvement of more than 90% in time synchronization precision compared to the traditional EKF-based time synchronization algorithm, and an improvement of more than 60% compared to the REKF-based time synchronization algorithm. Furthermore, compared to the EKF-based GNSS/IMU fusion algorithm, the proposed algorithm enhances the accuracy of the determination of both position and velocity in the horizontal and 3D directions by more than 50% and heading accuracy by 85%.

Lane detection is an essential visual capability for the autonomous vehicle navigation used in the intelligent transportation system (ITS). Several approaches for lane detection have been suggested in the past. However, there is still one issue about robustness. This paper presents a robust and real-time multi-lane detection method based on road marking feature points (RMFP) for autonomous vehicle navigation in an urban environment. The key idea in this paper is to apply methods from extracting RMFP and the target tracking domain to identify lane information. Then the RMFP is extracted from the gray-scale image and the IPM image. Additionally, the lane line color and structure features are used to sift through the RMFP that meet lane lines. Lastly, the clustering method is used to generate lane lines, and these lines are tracked by frame association and Kalman Filter. The experiment results based on image sequences taken under various visual conditions showed that the proposed system can identify the lane line markings with 98% accuracy. The method shows much robustness under various complicated scenarios and meets the real-time requirement.

Autonomous navigation is a fundamental task in mobile robotics. In the last years, several approaches have been addressing the autonomous navigation in outdoor environments. Lately it has also been extended to robotic vehicles in urban environments. This paper focus in the road identification problem, which is an important capability to autonomous vehicle drive. Our approach is based on image processing, template matching classification, and finite state machines processing. The proposed system allows to train an image segmentation algorithm in order to identify navigable and non-navigable regions (inside/outside roads), generating as output the steering control for an Electric Autonomous Vehicle, that should stay following the road. Several experimental tests have been carried out under different environmental conditions to evaluate the proposed techniques.

A robust road markings detection algorithm is a fundamental component of intelligent vehicles' autonomous navigation in urban environment. This paper presents an algorithm for detecting road markings including zebra crossings, stop lines and lane markings to provide road information for intelligent vehicles. First, to eliminate the impact of the perspective effect, an Inverse Perspective Mapping (IPM) transformation is applied to the images grabbed by the camera; the region of interest (ROI) was extracted from IPM image by a low level processing. Then, different algorithms are adopted to extract zebra crossings, stop lines and lane markings. The experiments on a large number of street scenes in different conditions demonstrate the effectiveness of the proposed algorithm.

Vision-based bio-inspired control strategies offer great promise in demonstrating safe autonomous navigation of aerial microsystems in completely unstructured environments. This paper presents an innovative navigation technique that embeds bio-inspired wide-field processing of instantaneous optic flow patterns within the H∞ loop shaping synthesis framework, resulting in a dynamic controller that enables robust stabilization and command tracking behavior in obstacle-laden environments. The local environment is parameterized as a series of simpler corridor-like environments in the optic flow model, and the loop shaping controller is synthesized to provide robust stability across the range of modeled environments. Experimental validation is provided using a quadrotor aerial vehicle across environments with large variation in local structure, with the loop shaping controller demonstrating better tracking performance than other comparable controllers in straight-line corridors of different widths. The current approach is computationally efficient, as it does not involve explicit extraction of an environment depth map, and makes for an attractive paradigm for aerial microsystem navigation in urban environments.

<div class="section abstract"> <div class="htmlview paragraph">This paper investigates the use of artificial intelligence (AI) to enhance autonomous vehicle (AV) controls in unstructured off-road environments. Recent research developments address the utilization of AI and deep learning (DL) models to share information with AVs, enabling intelligent navigation in urban environments. The suggested research allows AVs to navigate in an off-road environment by utilizing AI-predicted trajectories while avoiding obstacles. Our method aims to enhance vehicle autonomy, enabling it to move promptly and responsively while interacting with other vehicles in an off-road environment. We demonstrate the efficacy of AI-driven trajectory predictions through comprehensive simulation validations. Moreover, the research demonstrates the use of AI prediction to construct a proactive Model Predictive Controller (MPC) for off-road autonomous vehicles. Simulation studies and comparisons with existing approaches validate the effectiveness and advantages of the proposed approach for off-road autonomous vehicle controls.</div> </div>

This paper deals with a problem of multitask agents' navigation in urban environment with obstacles using different algorithmic approaches: neural network models with a system of rewards and penalties, neural networks oriented on distance minimization, as well as models that use direct movement vectors and rectangular route structures. The novelty of the presented method is the creation of a hybrid algorithm that incorporates strict rules for movement in an initially unknown environment and, when obstacles appear, switches to a neural network for obstacle avoidance. This approach combines deterministic movement, which can be adapted to artificial intelligence for safe and reliable navigation in uncharted territory. Experiments showed that thirty neural argents trained purely using rewards and penalties in movement exhibits slow learning and cyclical movement as clusters, which is due to the fact that they optimize the current state without global planning. Models trained solely on distance optimization perform slightly better, but still create clusters of movement toward the goal. Meanwhile, the use of strict linearity rules and rectangular routes allows for effective obstacle avoidance and consistent target achievement with less time under conditions of limited graph information. The results show that rule-based learning yields better results in agent movement than fully autonomous agents. Consequently, the combined path generation method yields the best results in agent navigation along a known path.

There are high expectations for autonomous personal mobility vehicles (PMVs) to support the mobility of older people. Autonomous navigation systems are being developed to assist mobility in public areas with mixed pedestrian traffic, such as airports and shopping malls. For autonomous navigation of PMVs, achieving both comfort and efficiency, even in crowded environments, is important. In this study, we focused on the characteristic of passenger tolerance, in which a passenger’s discomfort is relatively small concerning an approaching pedestrian. The objective was to propose an efficient autonomous navigation method without increasing passenger discomfort, considering the characteristics of passenger tolerance. First, the passenger tolerance characteristics were clarified through data analysis of a previous study’s dataset and a newly collected dataset. Next, a path-planning method considering the characteristics was proposed, and the proposed method was evaluated by numerical simulations. The evaluation results showed that the proposed method has a potential to achieve efficient autonomous navigation in crowded environments.

Rapid advancements in vehicle automation technology have enabled autonomous vehicles (AVs) to move beyond simple, structured highway settings and operate increasingly within more complex urban environments. As AVs become more prevalent, they will inevitably share the road with diverse traffic participants, particularly pedestrians. The high uncertainty and dynamic nature inherent in human behavior significantly complicate the AV’s decision-making process. To ensure both safety and efficiency while exhibiting socially acceptable behavior, AVs must make real-time decisions and continuously adapt their strategies in response to surrounding pedestrian behaviors. This poses a major challenge to existing AV decision-making systems. Therefore, this study focuses on developing the decision-making strategies for AVs in urban pedestrian-involved environments, spanning scenarios from simple interactions with a single pedestrian to complex cases involving multiple pedestrians. The core objective is to uncover the underlying interaction dynamics and to improve the AV’s capability to make decisions that are safe, efficient and socially acceptable in dynamic, human-centric contexts. Initially, this study investigates the critical factors influencing the decision making processes of human drivers and pedestrians during vehicle–pedestrian interactions. A series of controlled experiments were conducted using a virtual reality platform designed to collect realistic behavioral data. The data analysis mainly focused on kinematic variables that are easily measurable in real-world deployments. AdaBoost classification and Partial Dependence Plots were employed to identify and visualize the most influential factors affecting pedestrian crossing intentions and driver approaching behaviors. The results indicate that longitudinal distance and vehicle acceleration are the most influential factors in pedestrian decision-making, while pedestrian speed and longitudinal distance also play a crucial role in determining whether the vehicle yields or not. Based on these insights, a simplified mathematical model was developed to relate observable kinematic parameters to pedestrian crossing intentions, providing a practical tool for dynamically inferring crossing intentions during interactions. Additionally, the study explored driver yielding patterns under varying degrees of pedestrian intention clarity. These findings support the development of interpretable and implementable models for real-time decision-making in vehicle–pedestrian interaction scenarios. In addition, this study examines the decision-making strategies of the AVs when interacting with a single pedestrian in unsignalized intersections. A novel framework was proposed that integrates the Partially Observable Markov Decision Process with behavioral game theory to dynamically model interactive behaviors between the AVs and the pedestrian. Both agents were modeled as dynamic-belief-induced quantal cognitive hierarchy models, considering human reasoning limitations and bounded rationality in the decision-making process. Moreover, a dynamic belief updating mechanism allowed the AV to update its understanding of the opponent’s rationality degree in real-time based on observed behaviors and adapt its strategies accordingly. The analysis results indicate that proposed models effectively simulate vehicle-pedestrian interactions and the AV decision-making approach performs well in safety, efficiency, and smoothness. It captures key patterns of the driving behavior operated by real human drivers in virtual reality experiments and even achieves more comfortable navigation compared to our previous virtual reality experimental data. Finally, this study investigates the decision-making strategies of the AVs operating in pedestrian-rich shared spaces. A novel framework was proposed for modeling interactions between the AV and multiple pedestrians. In this framework, a cognitive process modeling approach inspired by the Free Energy Principle was integrated into both the AV and pedestrian models to simulate more realistic interaction dynamics. Specifically, the proposed pedestrian Cognitive-Risk Social Force Model adjusts goal-directed and repulsive forces using a fused measure of cognitive uncertainty and physical risk to produce human-like trajectories. Meanwhile, the AV leverages this fused risk to construct a dynamic, riskaware adjacency matrix for a Graph Convolutional Network within a Soft Actor-Critic architecture, allowing it to make more reasonable and informed decisions. The qualitative and quantitative results indicate that the proposed strategy effectively improves safety, efficiency, and smoothness of AV navigation compared to the state-of-the-art method. In conclusion, this study addresses key challenges in AV decision-making within human-centric urban environments, providing valuable insights as well as practical solutions to support safe, efficient, and socially aware autonomous navigation.

Advanced land‐vehicle navigation commonly uses integrated systems to counteract global navigation satellite system (GNSS) solution degradation. This occurs mainly in urban environments due to blockage of the satellite signals. This paper presents a loosely coupled inertial navigation system/GNSS navigation system that combines an attitude and heading reference system (AHRS) device with a dual‐frequency dual‐antenna GNSS heading receiver. The integrated navigation system is aided by a low‐cost odometer which replaces external wheel speed sensors usually installed in autonomous vehicles. The proposed odometer extracts the anti‐lock braking system‐generated pulses of rear wheels from vehicle controller area network messages. Following, it converts them into software‐generated signal pulses which are sent to the AHRS device through the serial port. The system platform uses a mobile internet data link to get differential GNSS corrections in real‐time from a public Ntrip—networked transport of Radio Technical Commission for Maritime Services via internet protocol—broadcaster in order to allow the GNSS receiver to operate in differential global positioning system/real‐time kinematic (RTK) modes. Thus, the integrated navigation system provides centimeter‐level positioning accuracy at 100 Hz. Since the positioning accuracy is severely affected by numerous factors, this work proposes a replicated 24 full factorial design with the purpose of evaluating the in‐field obtained positioning performance under different factors combinations. The experimental design chosen allows to know under which conditions it is feasible to replace the available GNSS velocity by the proposed odometry solution, when they are used as navigation aids, and knowing that the proposed odometry has a low resolution. The analysis of 32‐runs factorial design results, using a significance level of .05, demonstrated that the proposed odometry can overcome GNSS/RTK velocity.

Unmanned Aerial Vehicles (UAVs) have a major issue in autonomous navigation, particularly in dynamic or GPS-denied environments. Conventional rule-based or GPS-dependent strategies lack the flexibility and resilience to support real-time decision-making, especially in difficult terrain. We propose a novel deep learning-based framework, DeepFusion-NavNet, that couples reinforcement learning with semantic segmentation to perform end-to-end autonomous UAV navigation in an attempt to overcome these limitations. It employs a Deep Q-Network (DQN) to learn the optimal trajectory of the UAV based on policies learned once spatial features are extracted from aerial images by a U-Net-based semantic segmentation model. The proposed method was experimented with in both urban and forest environments utilizing the AirSim simulation tool. Relative to existing deep reinforcement learning baselines, experimental validation proves that DeepFusion-NavNet results in a navigation success rate of 97.5%, reduces collision incidents by 91.2%, and improves route efficiency by 22.8%. These results indicate that DeepFusion-NavNet significantly enhances UAV safety and autonomy under challenging operating conditions.

<p>Pedestrian detection is one of the important features in autonomous ground vehicle (AGV). It ensures the capability for safety navigation in urban environment. Therefore, the detection accuracy became a crucial part which leads to implementation using Laser Range Finder (LRF) for better data representation. In this study, an improved laser configuration and fusion technique is introduced by implementation of triple LRFs in two layers with Pedestrian Data Analysis (PDA) to recognize multiple pedestrians. The PDA integrates various features from feature extraction process for all clusters and fusion of multiple layers for better recognition. The experiments were conducted in various occlusion scenarios such as intersection, closed-pedestrian and combine scenarios. The analysis of the laser fusion and PDA for all scenarios showed an improvement of detection where the pedestrians were represented by various detection categories which solve occlusion issues when low number<br />of laser data were obtained.</p>

The rapid growth of urban mobility and logistics demands innovative solutions to address congestion, inefficiencies, and sustainability challenges. Unmanned Aerial Systems (UAS) offer a promising alternative for urban transportation, particularly in last-mile deliveries and emergency response. To accommodate the anticipated scale of UAS operations, potentially involving millions of daily flights, the UAS Traffic Management (UTM) ecosystem must support safe, efficient, and fair airspace access while minimizing disruption. UAS operations in dense urban areas must be supported by efficient and scalable path planning and deconflicting mechanisms which compute collision-free trajectories. A dynamic rerouting mechanism allows UAS trajectories to dynamically adapt to real-time deviations caused by environmental factors, communication errors, or dynamic airspace constraints. In this paper, we introduce a UAS Path Planning with Dynamic Rerouting framework that leverages a space-time graph model to ensure safe and efficient variable speed UAS navigation in urban environments. Our approach provides collision-free trajectories by construction and enables real-time path adjustments by dynamically rerouting UAS in case of slowdown events. By incorporating real-time request management and collision-free trajectories, our model enhances the feasibility of integrating UAS into multimodal urban transportation networks. We analyze the computational efficiency of our approach and demonstrate its scalability through simulation experiments using a real-world urban map in very dense scenarios with up to 14000 UAS vehicles. Our results indicate that dynamic rerouting with collision-free trajectory computation improves UAS route availability while maintaining airspace safety and operational efficiency.

GNSS receivers are designed to provide an estimate of three fundamental components for navigation: position, velocity and time. However, GNSS signals can be highly degraded or denied in some challenging environments as urban canyons, forests, mines and indoor environments. In addition to these critical scenarios, no orientation (or attitude) information is provided by standard receivers. Thus, in order to enhance robustness and performance of navigation solutions, it is possible to rely on integration with GNSS of measurements coming from other sensors, such as inertial sensors, barometers, magnetometers, cameras and other systems that can be used for positioning. The objective is to go towards seamless positioning, integrating measurements adaptively based on availability and reliability of the different systems.

In this chapter, we discuss potential opportunities and challenges associated with applications of reinforcement learning (RL) in the aerospace domain. In particular, we focus on problems related to sensor resource management, autonomous navigation, advanced manufacturing, maintenance, repair and overhaul operations, and human-machine collaboration. We present two detailed RL case studies related to sensor tasking for aerial surveillance and robot control in an additive manufacturing application which utilizes different flavors of RL including the more recent deep RL framework. Finally, we highlight some ongoing research developments which could address key challenges in deploying RL in the aerospace domain.