Semantic information based solution for visual SLAM in dynamic environment

Visual Simultaneous Localization and Mapping (VSLAM) has developed as the basic ability of robots in past few decades. There are a lot of open-sourced and impressive SLAM systems. However, the majority of the theories and approaches of SLAM systems at present are based on the static scene assumption, which is usually not practical in reality because moving objects are ubiquitous and inevitable under most circumstances. In this paper the DDL-SLAM (Dynamic Deep Learning SLAM) is proposed, a robust RGB-D SLAM system for dynamic scenarios that, based on ORB-SLAM2, adds the abilities of dynamic object segmentation and background inpainting. We are able to detect moving objects utilizing both semantic segmentation and multi-view geometry. Having a static scene map allows inpainting background of the frame which has been obscured by moving objects, therefore the localization accuracy is greatly improved in the dynamic environment. Experiment with a public RGB-D benchmark dataset, the results clarify that DDL-SLAM can significantly enhance the robustness and stability of the RGB-D SLAM system in the highly-dynamic environment.

The early applications of Visual Simultaneous Localization and Mapping (VSLAM) technology were primarily focused on static environments, relying on the static nature of the environment for map construction and localization. However, in practical applications, we often encounter various dynamic environments, such as city streets, where moving objects are present. These dynamic objects can make it challenging for robots to accurately understand their own position. This paper proposes a real-time localization and mapping method tailored for dynamic environments to effectively deal with the interference of moving objects in such settings. Firstly, depth images are clustered, and they are subdivided into sub-point clouds to obtain clearer local information. Secondly, when processing regular frames, we fully exploit the structural invariance of static sub-point clouds and their relative relationships. Among these, the concept of the sub-point cloud is introduced as novel idea in this paper. By utilizing the results computed based on sub-poses, we can effectively quantify the disparities between regular frames and reference frames. This enables us to accurately detect dynamic areas within the regular frames. Furthermore, by refining the dynamic areas of keyframes using historical observation data, the robustness of the system is further enhanced. We conducted comprehensive experimental evaluations on challenging dynamic sequences from the TUM dataset and compared our approach with state-of-the-art dynamic VSLAM systems. The experimental results demonstrate that our method significantly enhances the accuracy and robustness of pose estimation. Additionally, we validated the effectiveness of the system in dynamic environments through real-world scenario tests.

This paper reviews the latest research progress of visual SLAM in dynamic scenes. The improved algorithms dependent on deep learning and multi-sensor fusion are discussed. First of all, the challenges of traditional visual SLAM in dynamic environments are stated, especially the interference of high dynamic objects on system feature extraction, pose estimation and mapping. Then, the application of several object detection technologies such as YOLOv5 and YOLOv7 in dynamic visual SLAM was analyzed in detail. These methods significantly improve the positioning precision and robustness of the system by identifying and eliminating dynamic feature points in the scene. In addition, this paper explores the visual SLAM system combining semantic segmentation and geometric constraint optimization, and finally introduces the multi-sensor SLAM method combining IMU data, point cloud data and visual information. Experimental results show that these improved algorithms have a significant improvement in positioning accuracy and system robustness compared with traditional visual SLAM algorithms under TUM and other dynamic data sets.

Current visual simultaneous localization and mapping (SLAM) systems have demonstrated commendable efficacy in static environments. However, the presence of dynamic objects in real-world settings frequently leads to system discrepancies, significantly impairing the accuracy and robustness of SLAM systems. Conventional visual SLAM approaches typically utilize epipolar constraints to mitigate the impact of outliers; nevertheless, they encounter limitations when confronted with a substantial number of dynamic or planar moving objects. To tackle these issues, this paper introduces a novel dynamic visual SLAM system, termed DEG-SLAM. Initially, the system employs the YOLOv5 object detection network to identify dynamic objects, subsequently relaying the semantic information to the tracking module. During the tracking phase, both semantic information and epipolar constraints are leveraged to filter out dynamic feature points. To address the challenges posed by the malfunctioning of epipolar constraints in degenerate scenes, DEG-SLAM incorporates a degenerate constraint mechanism aimed at further eliminating dynamic feature points. Furthermore, a reprojection constraint has been introduced to enhance the filtering of absent dynamic feature points that lie outside the detection boxes. Experimental findings reveal that DEG-SLAM significantly improves accuracy and robustness when compared to traditional ORB-SLAM3 in dynamic environments. The performance benefits of DEG-SLAM are particularly evident in degenerate scenarios, thereby affirming its practicality and reliability in complex settings.

Accurate reconstruction of static environments from LiDAR scans of scenes containing dynamic objects, which we refer to as Dynamic to Static Translation (DST), is an important area of research in Autonomous Navigation. This problem has been recently explored for visual SLAM, but to the best of our knowledge no work has been attempted to address DST for LiDAR scans. The problem is of critical importance due to wide-spread adoption of LiDAR in Autonomous Vehicles. We show that state-of the art methods developed for the visual domain when adapted for LiDAR scans perform poorly. We develop DSLR, a deep generative model which learns a mapping between dynamic scan to its static counterpart through an adversarially trained autoencoder. Our model yields the first solution for DST on LiDAR that generates static scans without using explicit segmentation labels. DSLR cannot always be applied to real world data due to lack of paired dynamic-static scans. Using Unsupervised Domain Adaptation, we propose DSLR-UDA for transfer to real world data and experimentally show that this performs well in real world settings. Additionally, if segmentation information is available, we extend DSLR to DSLR-Seg to further improve the reconstruction quality. DSLR gives the state of the art performance on simulated and real-world datasets and also shows at least 4× improvement. We show that DSLR, unlike the existing baselines, is a practically viable model with its reconstruction quality within the tolerable limits for tasks pertaining to autonomous navigation like SLAM in dynamic environments.

Existing visual SLAM (vSLAM) systems fail to perform well in dynamic environments as they cannot effectively ignore moving objects during pose estimation and mapping. We propose a lightweight approach to improve the robustness of existing feature based RGB-D and stereo vSLAM by accurately removing dynamic outliers in the scene that contribute to failures in pose estimation and mapping. First, a novel motion state detection algorithm using the depth and feature flow information is presented to identify regions in the scene with high moving probability. This information is then fused with semantic cues via a probability framework to enable accurate and robust moving object extraction to retain the useful features for pose estimation and mapping. To reduce the computational complexity of extracting semantic information in every frame, we propose to extract semantics only on keyframes with significant changes in image content. Semantic propagation is used to compensate for the changes in the intermediate frames (i.e., non-keyframes). This is achieved by computing the dense transformation map using the available feature flow vectors. The proposed techniques can be integrated into existing vSLAM systems to increase their robustness in dynamic environments without incurring much computation cost. Our work highlights the importance of distinguishing between motion states of potential moving objects for vSLAM in highly dynamic environments. We provide extensive experimental results on four well-known RGB-D and stereo datasets to show that the proposed technique outperforms existing vSLAM methods in indoor and outdoor environments under various dynamic scenarios including crowded scenes. We also perform our experiments on a low-cost embedded platform, i.e., Jetson TX1, to demonstrate the computational efficiency of our method.

Visual simultaneous localization and mapping (VSLAM) is extensively utilized in the field of unmanned driving. Classical VSLAM typically postulates that the surroundings are stationary, and dynamic objects can seriously affect the accuracy of the system. However, detecting and tracking dynamic objects is beneficial for improving the autonomy and intelligence of robotics. In order to solve the problem of VSLAM distinguishing dynamic objects in dynamic scenes, a semantic VSLAM system based on the VDO-SLAM framework is proposed. The approach integrates semantic segmentation into the visual odometry and combines scene flow and geometric constraints to effectively reduce the negative impact of dynamic objects on system performance. Specifically, a transformer-based neural network architecture for optical flow estimation was employed, which effectively enhances the segmentation precision of foreground and background regions. And a geometric constraint was introduced to classify dynamic objects, which can reduction of erroneous segmentation of dynamic objects with effect. The data analysis results on the KITTI dataset demonstrate that our proposed algorithms can effectively enhance precision and stability of VSLAM system, reduce the tracking error of VSLAM in dynamic environments, and demonstrate good potential and application value.

Visual simultaneous localization and mapping (SLAM) in dynamic environments has received significant attention in recent years, and accurate segmentation of real dynamic objects is the key to enhancing the accuracy of pose estimation in such environments. In this study, we propose a visual SLAM approach based on ORB-SLAM3, namely dynamic multiple object tracking SLAM (DMOT-SLAM), which can accurately estimate the camera’s pose in dynamic environments while tracking the trajectories of moving objects. We introduce a spatial point correlation constraint and combine it with instance segmentation and epipolar constraint to identify dynamic objects. We integrate the proposed motion check method into DeepSort, an object tracking algorithm, to facilitate inter-frame tracking of dynamic objects. This integration not only enhances the stability of dynamic features detection but also enables the estimation of global motion trajectories for dynamic objects and the construction of object-level semi-dense semantic maps. We evaluate our approach on the public TUM, Bonn, and KITTI dataset, and the results show that our approach has a significant improvement over ORB-SLAM3 in dynamic scenes and performs better compared to other state-of-the-art SLAM approaches. Moreover, experiments in real-world scenarios further substantiate the effectiveness of our approach.

Simultaneous localisation and mapping (SLAM) is crucial in mobile robotics. Most visual SLAM systems assume that the environment is static. However, in real life, there are many dynamic objects, which affect the accuracy and robustness of these systems. To improve the performance of visual SLAM systems, this study proposes a dynamic visual SLAM (SEG-SLAM) system based on the orientated FAST and rotated BRIEF (ORB)-SLAM3 framework and you only look once (YOLO)v5 deep-learning method. First, based on the ORB-SLAM3 framework, the YOLOv5 deep-learning method is used to construct a fusion module for target detection and semantic segmentation. This module can effectively identify and extract prior information for obviously and potentially dynamic objects. Second, differentiated dynamic feature point rejection strategies are developed for different dynamic objects using the prior information, depth information, and epipolar geometry method. Thus, the localisation and mapping accuracy of the SEG-SLAM system is improved. Finally, the rejection results are fused with the depth information, and a static dense 3D mapping without dynamic objects is constructed using the Point Cloud Library. The SEG-SLAM system is evaluated using public TUM datasets and real-world scenarios. The proposed method is more accurate and robust than current dynamic visual SLAM algorithms.

Simultaneous localization and mapping (SLAM) plays a crucial role in the field of intelligent mobile robots. However, the traditional Visual SLAM (VSLAM) framework is based on strong assumptions about static environments, which are not applicable to dynamic real-world environments. The correctness of re-localization and recall of loop closure detection are both lower when the mobile robot loses frames in a dynamic environment. Thus, in this paper, the re-localization and loop closure detection method with a semantic topology graph based on ORB-SLAM2 is proposed. First, we use YOLOv5 for object detection and label the recognized dynamic and static objects. Secondly, the topology graph is constructed using the position information of static objects in space. Then, we propose a weight expression for the topology graph to calculate the similarity of topology in different keyframes. Finally, the re-localization and loop closure detection are determined based on the value of topology similarity. Experiments on public datasets show that the semantic topology graph is effective in improving the correct rate of re-localization and the accuracy of loop closure detection in a dynamic environment.

Aiming at the problems of low positioning accuracy, poor map readability and weak robustness when the mobile robot implements the SLAM technology due to the existence of dynamic objects in the mobile robot working environment, an SLAM technology algorithm for mobile robot dynamic environment positioning based on semantic information is proposed. Firstly, the front end of the ORB-SLAM2 framework will be used, combined with the YOLO v4 target detection algorithm, to extract the ORB features of the input image. Meanwhile, the YOLO v4 target detection algorithm is used to obtain the dynamic and static areas of the object containing semantic information in the scene image, to obtain the preliminary semantic dynamic and static areas, and to perform semantic segmentation in the image; Then, the dynamic object region is selected by using the image polar equation, and the ORB feature points distributed on the dynamic object are eliminated. Finally, the processed feature points and adjacent key frames are used for inter-frame matching to estimate the camera pose to build a static environment map. The experiment used the open TUM dataset to compare the proposed algorithm with the traditional ORB-SLAM2 test results show that the proposed algorithm improves the pose estimation accuracy by 75% compared with the ORB-SLAM2 in the dynamic environment, and the map construction effect is significantly enhanced. The experimental results show that this algorithm can eliminate the influence of dynamic objects on SLAM technology, improve the positioning accuracy of the system, and expand the application field of the system.

During the continual transitions from lab research to real-world applications of vision-based algorithms, there are significant challenges, e.g. the robustness to adapt to complex environments and the high demands of multi-task, multi-modal learning models. More specifically on visual simultaneous localization and mapping (visual SLAM) for mobile robots, two notable limitations are: (i) the drift issue during pose estimation, especially in dynamic environments, makes the positioning system unstable; (ii) the additional loop detection takes many computational resources for image retrieval and global feature geometry checking. This thesis explores the approaches of visual localization on unmanned ground vehicles by solving those limitations. The conventional feature extraction and matching pipeline for vision-based tasks involve three steps: (i) local feature detection and description, (ii) feature matching, and (iii) outlier rejection. The first contribution of the thesis is adding a feature selection and anticipation stage to reduce tracking drift. We explore the noise model of image features to select a subset of all the observed image features with the best ”contribution” during data association and pose estimation across multiple frames. Conventional SLAM algorithms take a strong assumption of scene rigidity, which limits the application under challenging environments. The second part of the thesis addresses the tough issue in dynamic environments with moving objects. We presented GMC, namely the motion clustering approach, a lightweight dynamic object filtering method. It can distinguish moving objects from static landmarks. Based on the theory of motion coherence within a particular image area, GMC could segment dynamic objects in 3D space. We can provide an efficient and robust correspondence algorithm that can extract dynamic objects from a static background with the method. In this way, we propose a dynamic SLAM system that is real-time and free from expensive GPU processors. In contrast to GMC, the thesis’s third part turns to an object-aware learning-based model for more general dynamic scenarios. We use object detection and tracking as points, lines, planes, etc. We utilize semantic information and extract sparse image features simultaneously to keep track of dynamic objects. The static background and different dynamic objects are jointly optimized in a newly developed bundle adjustment sliding window. The estimated 3D bounding boxes can provide more robust camera tracking and better scene understanding, and better map merging. The fourth part of the thesis leverages the emerging feature learning framework. It proposes a unified self-supervised model called LGDNet to generate both global and local image feature descriptors end-to-end. Global feature descriptors embed the whole image into a compact representation, leading to easier scene comparison. On the other hand, local features focus more on the local region similarities of some salient parts for structure from motion. Our proposed method can directly extract features together with descriptors that encode both local maximum responses and global context information, avoiding duplicate calculations based on different feature extraction criteria.

We present a simultaneous localization and mapping algorithm for a mobile robot in 3D. It allows to construct a map of the environment without artifacts caused by dynamic objects such as trees or people. Experimental results confirm the effectiveness of the algorithm in real world scenarios. We also compare the developed algorithm with previously realized analogs.

Loop closure detection is the core part of the visual simultaneous localization and mapping (VSLAM) for autonomous robots. In the dynamic environments, loop closure detection turns to be a very difficult problem compared to the static scenario. This paper proposes a novel approach based on convolutional autoencoder neural network (CAENN) architecture for extracting images features, and then uses Euclidean loss for minimizing the difference between the extracted feature and the gist feature which has the ability of scene recognition. In order to improve the accuracy and recall rate for loop closure detection in the dynamic scenario, the perspective transformation and dynamic object are applied in the process of the construction of the training set. And by calculating the Manhattan distance between the two image feature vectors, the loop closure is accepted when the distance is smaller than the threshold. The experimental results demonstrate that the proposed method obtains better accuracy and recall rate compared to the commonly used gist feature method and has a lower cost on time and space compared to the BOW method and AlexNet method in loop closure detection.

Simultaneous Localization and Mapping (SLAM) is a crucial technology for intelligent mobile robots to operate successfully in unknown environments. While many excellent SLAM systems have been developed in recent years, most assume that the environment is static, resulting in poor performance in dynamic environments. To address this limitation, we propose multiview stereo (MVS)‐SLAM (MVS‐SLAM), a real‐time semantic RGBD SLAM system with improved‐multiview geometry, built on the RGBD mode of ORB‐SLAM3. MVS‐SLAM tightly integrates semantic and geometric information to tackle the challenges posed by dynamic scenes. To meet the real‐time requirements, the semantic module leverages the latest and fastest object detection network, YOLOv7, to provide semantic prior knowledge for the geometric module. We introduce a novel geometric constraint method that capitalizes on depth images and semantic information to recover three‐dimensional (3D) feature points and initial camera pose. We use a 3D coordinate error threshold to identify dynamic points and remove them using the K‐means clustering algorithm. This approach effectively reduces the impact of dynamic points. We validate MVS‐SLAM using challenging dynamic sequences from the TUM data set, demonstrating that it significantly improves localization accuracy and system robustness in all types of dynamic environments.