RTS-vSLAM: Real-Time Visual Semantic Tracking and Mapping Under Dynamic Environments

Perception of the indoor environment is the basis of mobile robot localization, navigation, and path planning, and it is particularly important to construct semantic maps in real time using minimal resources. The existing methods are too dependent on the graphics processing unit (GPU) for acquiring semantic information about the indoor environment, and cannot build the semantic map in real time on the central processing unit (CPU). To address the above problems, this paper proposes a non-use GPU for lightweight indoor semantic map construction algorithm, named NGLSFusion. In the VO method, ORB features are used for the initialization of the first frame, new keyframes are created by optical flow method, and feature points are extracted by direct method, which speeds up the tracking speed. In the semantic map construction method, a pretrained model of the lightweight network LinkNet is optimized to provide semantic information in real time on devices with limited computing power, and a semantic point cloud is fused using OctoMap and Voxblox. Experimental results show that the algorithm in this paper ensures the accuracy of camera pose while speeding up the tracking speed, and obtains a reconstructed semantic map with complete structure without using GPU.

In order to solve the shortcomings of traditional simultaneous localization and mapping in dynamic environment, which is interfered by moving objects, resulting in low accuracy and poor robustness, a visual simultaneous localization and mapping algorithm combining semantic information for motion detection was proposed. First, the SegNet deep neural network is used to extract the semantic information of the environment, and the prior knowledge is used to determine the static attribute objects and dynamic attribute objects. In the motion detection module, the feature points on the dynamic attribute objects are used to perform motion detection using geometric constraint relationships. Then the building module uses semantic information to build a semantic octo-tree map. In order to analyse the effect of motion detection, a control experiment with a motion detection module removed was set up. Finally, experiments were conducted using TUM datasets, and the experimental results of the two schemes were compared and analysed.

Abstract. Semantic information in images—such as meaningful, recognizable regions or objects that are incongruent with the overall scene—can effectively capture attention. The meaning map is currently the primary method for mapping the distribution of semantic information across visual stimuli. However, this approach relies on subjective human ratings of semantic meaningfulness and requires extensive manual annotation for each image prior to analysis. To address these limitations and enable rapid, efficient, and reproducible generation of semantic distribution maps for any given image, this paper proposes an automated method for constructing semantic information maps using multimodal large language models (MLLMs). Our approach generates two types of semantic maps: local semantic information maps, which quantify the semantic content at each spatial location within an image, and global semantic maps, which assess the contextual relevance of local regions to the overall scene. Additionally, the method can generate distribution maps of visual information associated with specific concepts. We argue that this method significantly enhances the precision and flexibility in controlling, measuring, and manipulating semantic information in visual stimuli, thereby advancing research in visual attention and visual language processing. The method is currently under further refinement.

Visual tracking is an important component in the navigation and control of unmanned aerial vehicles (UAVs). This paper is concerned with the visual tracking problem for the navigation and control of UAVs and focuses on real-time image processing methods. We review previous research based on geometric active contours which are automatic processes to track object boundaries in an image. Test results and some limitations of the previous tracking system are discussed. Then, a fast and robust implementation of the Chan-Vese active contour model is proposed to overcome these limitations.

Being intensively studied, visual tracking has seen great recent advances in either speed (e.g., with correlation filters) or accuracy (e.g., with deep features). Real-time and high accuracy tracking algorithms, however, remain scarce. In this paper we study the problem from a new perspective and present a novel parallel tracking and verifying (PTAV) framework, by taking advantage of the ubiquity of multi-thread techniques and borrowing from the success of parallel tracking and mapping in visual SLAM. Our PTAV framework typically consists of two components, a tracker T and a verifier V, working in parallel on two separate threads. The tracker T aims to provide a super real-time tracking inference and is expected to perform well most of the time; by contrast, the verifier V checks the tracking results and corrects T when needed. The key innovation is that, V does not work on every frame but only upon the requests from T; on the other end, T may adjust the tracking according to the feedback from V. With such collaboration, PTAV enjoys both the high efficiency provided by T and the strong discriminative power by V. In our extensive experiments on popular benchmarks including OTB2013, OTB2015, TC128 and UAV20L, PTAV achieves the best tracking accuracy among all real-time trackers, and in fact performs even better than many deep learning based solutions. Moreover, as a general framework, PTAV is very flexible and has great rooms for improvement and generalization.

Abstract: Path planning is a way to define the motion of an autonomous surface vehicle (ASV) in any existing obstacle environment to enable the vehicle's movement by setting directions to avoid that can react to the obstacles in the vehicle's path. A good, planned path perceives the environment to the extent of uncertainty and tries to build or adapt its change in the path of motion. Efficient path planning algorithms are needed to alleviate deficiencies, which are to be modified using the deterministic path that leads the ASV to reach a goal or a desired location while finding an optimal solution has become a challenge in the field of optimization along with a collision-free path, making path planning a critical thinker. The traditional algorithms have a lot of training and computation, making it difficult in a realistic environment. This review paper explores the different techniques available in path planning and collision avoidance of ASV in a dynamic environment. The objective of good path planning and collision avoidance for a dynamic environment is compared effectively with the existing obstacle’s movement of different vehicles. Different path planning technical approaches are compared with their performance and collision avoidance for unmanned vehicles in marine environments by early researchers. This paper gives us a clear idea for developing an effective path planning technique to overcome marine accidents in the dynamic ocean environment while choosing the shortest, obstacle-free path for Autonomous Surface Vehicles that can reduce risk and enhance the safety of unmanned vehicle movement in a harsh ocean environment.

In view of existing Visual SLAM (VSLAM) algorithms when constructing semantic map of indoor environment, there are problems with low accuracy and low label classification accuracy when feature points are sparse. This paper proposed a 3D semantic VSLAM algorithm called BMASK-RCNN based on Mask Scoring RCNN. Firstly, feature points of images are extracted by Binary Robust Invariant Scalable Keypoints (BRISK) algorithm. Secondly, map points of reference key frame are projected to current frame for feature matching and pose estimation, and an inverse depth filter is used to estimate scene depth of created key frame to obtain camera pose changes. In order to achieve object detection and semantic segmentation for both static objects and dynamic objects in indoor environments and then construct dense 3D semantic map with VSLAM algorithm, a Mask Scoring RCNN is used to adjust its structure partially, where a TUM RGB-D SLAM dataset for transfer learning is employed. Semantic information of independent targets in scenes provides semantic information including categories, which not only provides high accuracy of localization but also realizes the probability update of semantic estimation by marking movable objects, thereby reducing the impact of moving objects on real-time mapping. Through simulation and actual experimental comparison with other three algorithms, results show the proposed algorithm has better robustness, and semantic information used in 3D semantic mapping can be accurately obtained.

Simultaneous localization and mapping (SLAM) plays a crucial role in the field of intelligent mobile robots. However, the traditional Visual SLAM (VSLAM) framework is based on strong assumptions about static environments, which are not applicable to dynamic real-world environments. The correctness of re-localization and recall of loop closure detection are both lower when the mobile robot loses frames in a dynamic environment. Thus, in this paper, the re-localization and loop closure detection method with a semantic topology graph based on ORB-SLAM2 is proposed. First, we use YOLOv5 for object detection and label the recognized dynamic and static objects. Secondly, the topology graph is constructed using the position information of static objects in space. Then, we propose a weight expression for the topology graph to calculate the similarity of topology in different keyframes. Finally, the re-localization and loop closure detection are determined based on the value of topology similarity. Experiments on public datasets show that the semantic topology graph is effective in improving the correct rate of re-localization and the accuracy of loop closure detection in a dynamic environment.

The early applications of Visual Simultaneous Localization and Mapping (VSLAM) technology were primarily focused on static environments, relying on the static nature of the environment for map construction and localization. However, in practical applications, we often encounter various dynamic environments, such as city streets, where moving objects are present. These dynamic objects can make it challenging for robots to accurately understand their own position. This paper proposes a real-time localization and mapping method tailored for dynamic environments to effectively deal with the interference of moving objects in such settings. Firstly, depth images are clustered, and they are subdivided into sub-point clouds to obtain clearer local information. Secondly, when processing regular frames, we fully exploit the structural invariance of static sub-point clouds and their relative relationships. Among these, the concept of the sub-point cloud is introduced as novel idea in this paper. By utilizing the results computed based on sub-poses, we can effectively quantify the disparities between regular frames and reference frames. This enables us to accurately detect dynamic areas within the regular frames. Furthermore, by refining the dynamic areas of keyframes using historical observation data, the robustness of the system is further enhanced. We conducted comprehensive experimental evaluations on challenging dynamic sequences from the TUM dataset and compared our approach with state-of-the-art dynamic VSLAM systems. The experimental results demonstrate that our method significantly enhances the accuracy and robustness of pose estimation. Additionally, we validated the effectiveness of the system in dynamic environments through real-world scenario tests.

To address the dual requirements of real-time performance and robustness in visual target tracking for autonomous driving, this study proposes a real-time visual target tracking algorithm based on lightweight neural networks. By integrating sensor fusion techniques, the KCF algorithm, and Kalman filtering technology, detailed testing was conducted using MATLAB simulation tools to model three core scenarios encountered in autonomous driving environments. Research findings indicate that the algorithm establishes a more reliable perception environment for autonomous vehicles. This will provide higher-quality perception information support for subsequent decision-making planning and control execution modules of autonomous driving systems, thereby driving the advancement of autonomous driving technology toward higher levels and enhanced safety. This holds significant practical importance.

Abstract. Semantic mapping is crucial for intelligent obstacle avoidance and planning in SLAM systems. We proposed an autonomous semantic mapping approach that integrates multimodal semantic segmentation and SLAM techniques to construct a dense 3D semantic map in real time. Multimodal semantic segmentation based on camera images and LiDAR point clouds is performed in each frame, which assigns image segmentation labels to LiDAR points, generating per-frame 3D semantic information. These segmented frames are then incrementally fused within the SLAM framework to produce a globally consistent semantic map of the environment. The proposed approach is validated through real-world experiments conducted around the Star Lake Building at Wuhan University using the Luo-Jia Explorer system. The experimental results show that our method achieves real-time performance with an inference speed of up to 14Hz on an RTX 4070 GPU, effectively processing sensor data on 10Hz while maintaining high segmentation accuracy in both indoor and outdoor scenarios.

Real-time object detection and tracking in video streams has become a fundamental component of modern intelligent systems, enabling applications in surveillance, autonomous driving, robotics, healthcare monitoring, and smart cities. The rapid advancement of deep learning techniques has significantly improved the accuracy and efficiency of detecting and tracking multiple objects under complex and dynamic environments. This study presents a deep learning–based framework for real-time object detection and tracking in continuous video streams, emphasizing both computational efficiency and detection robustness.The proposed system integrates a state-of-the-art convolutional neural network (CNN)-based object detector with a reliable multi-object tracking algorithm. The detection module identifies objects of interest in each frame, while the tracking component maintains consistent identities across consecutive frames by utilizing motion prediction and feature association strategies. To achieve real-time performance, the model is optimized through lightweight network architectures and parallel processing on GPU hardware. Data augmentation and transfer learning techniques are employed to enhance generalization and reduce training time. Performance evaluation is conducted on benchmark video datasets using standard metrics such as precision, recall, mean Average Precision (mAP), and tracking accuracy. Experimental results demonstrate that the proposed framework achieves high detection accuracy while maintaining low latency, making it suitable for real-time deployment. Furthermore, the system effectively handles challenges such as occlusion, scale variation, and illumination changes. This research presents a comprehensive study on real-time object detection and tracking in video streams using deep learning techniques. The proposed framework integrates YOLO-based object detection with Deep SORT multi-object tracking to achieve robust spatial localization and temporal identity preservation. The study evaluates performance using benchmark datasets including COCO, MOT17, and KITTI. Mathematical formulations of detection loss, Kalman filtering, and data association are presented. Experimental analysis using precision, recall, mAP, MOTA, ROC-AUC, and PR-AUC demonstrates that the system achieves high detection accuracy while maintaining real-time performance above 35 frames per second. The findings confirm that deep learning-based frameworks provide scalable and efficient solutions for surveillance, autonomous vehicles, and intelligent monitoring systems.

Designing a real-time visual tracking system to catch goldfishes is a very complex task because a fish’s precise relative 3D positions inside a water tank are difficult to determine. In this paper, we present a new visual servo control system using stereo vision to acquire the 3D positions of goldfishes. The continuously adaptive mean shift (CAMSHIFT) algorithm and optical flow vision tracking methods are applied to provide feedback of a tracked fish’s real-time position and motion trajectory in a water tank at a high frame-rate. Two low-cost, well calibrated cameras are used to track the goldfishes and a 5-axis robot manipulator is used to catch a specified goldfish. The designed visual tracking and servo system is less sensitive to lighting influences and thus performs more efficiently. Experiments of the proposed method yielded very good results as the real-time 3D vision successfully tracked two fishes and guided the manipulator to catching one of them in a net attached to its end-effector.

A compact infrared absorption spectrometer is developed and integrated with a light-duty unmanned aerial vehicle (UAV) to provide high-resolution, real-time spatial mapping of species concentrations in dynamic atmospheric environments. The multi-species optical sensor is contained within a 0.5 ft3 package that is mounted to a custom fabricated quadrotor UAV. The on-board optical diagnostics record CO and CO2 concentrations which are transmitted in real-time at a rate of 1.25 Hz to a remote monitoring station. The sensor results are synchronized with the UAV GPS system to generate time-histories and spatial maps. The aerial spectrometer was deployed in a series of prescribed forest burns at the UC Berkeley Blodgett Forest Research Station, where it successfully captured transient plume behavior and generated three-dimensional spatial profiles on 100 m spatial scales and altitudes up to 100 m. The measurements presented here aim to demonstrate the potential for high-resolution, UAV-borne optical sensors for atmospheric monitoring in harsh, dynamic environments, while serving as a guide for future technological improvements and applications.

Visual simultaneous localization and mapping (SLAM) is a critical technology for autonomous navigation in dynamic environments. However, traditional SLAM algorithms often struggle to maintain accuracy in highly dynamic environments, where elements undergo significant, rapid, and unpredictable changes, leading to asymmetric information acquisition. Aiming to improve the accuracy of the SLAM algorithm in a dynamic environment, a dynamic SLAM algorithm based on deep learning is proposed. Firstly, YOLOv10n is used to improve the front end of the system, and semantic information is added to each frame of the image. Then, ORB-SLAM2 is used to extract feature points in each region of each frame and retrieve semantic information from YOLOv10n. Finally, through the map construction thread, the dynamic object feature points extracted by YOLOv10n are eliminated, and the construction of a static map is realized. The experimental results show that the accuracy of the proposed algorithm is improved by more than 96% compared with the state-of-the-art ORB-SLAM2 in a highly dynamic environment. Compared with other dynamic SLAM algorithms, the proposed algorithm has improved both accuracy and runtime.