Semantic SLAM for dynamic semi-structured environments using object and plane detection

The object detection of remote sensing image often has low accuracy and high missed or false detection rate due to the large number of small objects, instance level noise and cloud occlusion. In this paper, a new object detection model based on SRGAN and YOLOV3 is proposed, which is called SR-YOLO. It solves the problems of SRGAN network sensitivity to hyper-parameters and modal collapse. Meanwhile, The FPN network in YOLOv3 is replaced by PANet, shortened the distance between the lowest and the highest layers, and the SR-YOLO model has strong robustness and high detection ability by using the enhanced path to enrich the characteristics of each layer. The experimental results on the UCAS-High Resolution Aerial Object Detection Dataset showed SR-YOLO has achieved excellent performance. Compared with YOLOv3, the average precision (AP) of SR-YOLO increased from 92.35% to 96.13%, the log-average miss rate (MR-2) decreased from 22% to 14%, and the Recall rate increased from 91.36% to 95.12%.

Robust robot localization with visually adaptive consensus filters in dynamic corridor environments

This research proposes a novel hardware-optimized design to accelerate Convolutional Neural Networks (CNNs) using Verilog HDL. The design is specifically developed for the DARKNET-19 system model, which serves as the backbone of the YOLOv3-tiny algorithm, a widely used framework for real-time object detection in dynamic environments. The CNN architecture was implemented in Verilog HDL and synthesized using Synopsys Design Compiler, with a focus on improving both object detection accuracy and hardware resource efficiency. The proposed design efficiently performs key CNN operations, including convolution, pooling, and activation, enabling faster real-time object detection compared to many existing methods. To improve performance, the hardware design incorporates parallel processing techniques, allowing multiple computations to be executed simultaneously. This significantly reduces the system latency and power consumption. The convolutional layers of the DARKNET-19 architecture are efficiently mapped onto the hardware platform, ensuring optimized data storage and fast memory access, which further enhances processing speed and detection accuracy. An innovative feature of the design is a 2-dimensional image preprocessing module that prepares input images before they are fed into the CNN. This preprocessing stage includes image resizing, brightness normalization, and color adjustment, which helps the CNN process visual data more effectively. After preprocessing, the images pass through several CNN layers. The convolutional layers extract key features from the images, while the pooling and activation layers refine these features to improve detection performance. Finally, the processed data is analyzed by the YOLOv3-tiny algorithm, which identifies and locates objects in the images with high precision. Experimental results demonstrate that the proposed high-speed and resource-efficient hardware architecture is well-suited for real-time object detection applications, particularly in highly dynamic and unpredictable environments.

Multi-Sensor Fusion (MSF) based perception systems have become the foundation supporting many industrial applications and domains, such as self-driving cars, robotic arms, and unmanned aerial vehicles. With the rapid development of data-driven artificial intelligence (AI), the perception capabilities of MSF have been comprehensively enhanced, especially in understanding complex, dynamic external environments. Similar to traditional software, AI-enabled MSF systems also require rigorous testing. However, existing testing methods are still limited to evaluating the frame-level perception capabilities (e.g., object detection in static scenes) of singlesensor systems (e.g., image-based and point cloud-based systems). Given that many safety-critical intelligent systems, such as selfdriving cars, are operated in dynamic environments where perception systems play an important role, there comes an urgent need to assess their dynamic perception capabilities in understanding and responding to external environmental variations in real-time. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">To bridge this gap, we design and implement DRIVENCE, an automated metamorphic testing tool for testing the dynamic perception capabilities of MSF-based systems. DRIVENCE accounts for various real-world physical constraints to generate realistic multi-modal test sequences by inserting multiple dynamic traffic participants into the background image and point cloud driving sequences. To diversify testing sequences, we incorporate six driving patterns derived from real-world common driving behaviors into the testing process. We conduct experiments with five SOTA MSF-based tracking systems to evaluate DRIVENCE from the perspectives of (1) generated test cases’ realism, (2) fault detection capabilities, and (3) test efficiency. The results show that DRIVENCE can generate realistic and modality-consistent test driving sequences and effectively detect various dynamic perception errors within MSF systems.

Accurate object detection is important for the safe navigation of autonomous vehicles (AVs) in dynamic environments. Conventional AVs rely on high-resolution but bulky, topmounted 3D Light Detection And Ranging (LiDAR) sensors, which are impractical for scalable production and computationally expensive to process. To address this, we utilize quasi-2D LiDAR sensors, which offer a compact and scalable solution for AVs. However, point clouds generated by these sensors have a narrow vertical field of view and contain a limited number of points compared to 3D LiDARs, which makes object detection significantly more difficult. To overcome this challenge, we transform the quasi-2D point clouds into Bird's-Eye-View (BEV) images and apply four different input encoding strategies. These encodings enhance the spatial and temporal context of the data, making them suitable for efficient deep-learning-based 2D object detection models to perform real-time detection with minimal computational overhead. In contrast to other approaches for 2D LiDARs, we are able to detect not only vehicles but also pedestrians and cyclists. We use a large real-world dataset to compare our proposed input encodings combined with three different state-of-the-art 2D object detection models. Our results demonstrate that these input encodings enhance object detection performance across all models, significantly reducing false positives, improving heading angle prediction, and boosting the detection accuracy of smaller traffic participants.

Current simultaneous localization and mapping (SLAM) algorithms perform well in static environments but easily fail in dynamic environments. Recent works introduce deep learning-based semantic information to SLAM systems to reduce the influence of dynamic objects. However, it is still challenging to apply a robust localization in dynamic environments for resource-restricted robots. This paper proposes a real-time RGB-D inertial odometry system for resource-restricted robots in dynamic environments named Dynamic-VINS. Three main threads run in parallel: object detection, feature tracking, and state optimization. The proposed Dynamic-VINS combines object detection and depth information for dynamic feature recognition and achieves performance comparable to semantic segmentation. Dynamic-VINS adopts grid-based feature detection and proposes a fast and efficient method to extract high-quality FAST feature points. IMU is applied to predict motion for feature tracking and moving consistency check. The proposed method is evaluated on both public datasets and real-world applications and shows competitive localization accuracy and robustness in dynamic environments. Yet, to the best of our knowledge, it is the best-performance real-time RGB-D inertial odometry for resource-restricted platforms in dynamic environments for now. The proposed system is open source at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/HITSZ-NRSL/Dynamic-VINS.git</uri>

Object detection has become a cornerstone of computer vision, with applications ranging from autonomous driving and robotics to surveillance and augmented reality. While substantial progress has been made in controlled and static settings, real-world environments often pose significant challenges due to dynamic backgrounds, occlusions, illumination variations, and cluttered scenes. This survey provides a comprehensive review of recent advancements in object detection specifically tailored for dynamic and complex environments. We classify existing approaches based on their core methodologies, including traditional feature-based techniques, deep learning models, and hybrid frameworks. Key challenges such as real-time performance, adaptability to environmental changes, and robustness to motion are discussed in depth. Furthermore, we analyze benchmark datasets and evaluation metrics commonly used in this domain, highlighting their limitations and suggesting improvements. Finally, we explore emerging trends and future directions, including the integration of spatiotemporal modeling, sensor fusion, and domain adaptation strategies. This survey aims to serve as a valuable reference for researchers and practitioners seeking to develop or apply object detection systems in real-world, unpredictable environments.

In recent years, object detection has become a crucial component in various applications such as surveillance, autonomous vehicles, medical imaging, and smart cities. This project focuses on object detection using state-of-the-art algorithms YOLOv8 and YOLOv9. Leveraging the power of deep learning, YOLO (You Only Look Once) models are renowned for their speed and accuracy in identifying multiple objects within images and video streams. The proposed system allows users to upload images or videos and utilizes the webcam for live object detection. The frontend is built using Streamlit, ensuring an intuitive and interactive user interface, while the backend is implemented in Python and executed on Google Colab to leverage GPU acceleration. The system processes the input through pre-trained YOLO models, identifies objects, and displays them with bounding boxes and confidence scores. This project demonstrates how advanced object detection algorithms can be integrated into user-friendly applications, promoting accessibility and real-world deployment. Additionally, it serves as a foundation for further enhancements such as object tracking, anomaly detection, and integration with IoT systems for smarter decision-making in dynamic environments.

Simultaneous localization and map building (SLAM) is crucial in autonomous robot navigation. However, existing SLAM systems generally assume a static environment, which makes it difficult to cope with the interference caused by moving objects in dynamic scenes, affecting the system’s localization accuracy and robustness. To address this challenge, this paper proposes YKD-SLAM, a visual SLAM system for indoor dynamic environments, which is based on the ORB-SLAM2 framework and incorporates YOLOv8 target detection, RCF-KMeans (Region-ConstrainedFastK-Means), and epipolar geometric constraints to realize the accurate rejection of dynamic feature points and improve the localization performance in dynamic environments. YKD-SLAM first uses YOLOv8 to detect dynamic objects in the scene, generates a detection frame, optimizes the depth map through open operations, and performs multi-region segmentation of the region within the detection frame by combining RCF-KMeans. Subsequently, through the dynamic feature point rejection strategy based on epipolar geometric constraints, different regions in the detection frame are discriminated into dynamic and static regions, and the feature points in the dynamic region are rejected to improve the localization accuracy and robustness of the system in dynamic environments. The experimental results show that YKD-SLAM performs well in several dynamic scenes in the TUMRGB-D dataset. Compared with ORB-SLAM2, its ATE is reduced by 98.37%; compared with DynaSLAM, the system operation efficiency is improved by 95.35%. In addition, practical experiments conducted in indoor dynamic scenes further validate its potential in real applications.

Simultaneous localization and mapping (SLAM) plays a crucial role in the field of intelligent mobile robots. However, the traditional Visual SLAM (VSLAM) framework is based on strong assumptions about static environments, which are not applicable to dynamic real-world environments. The correctness of re-localization and recall of loop closure detection are both lower when the mobile robot loses frames in a dynamic environment. Thus, in this paper, the re-localization and loop closure detection method with a semantic topology graph based on ORB-SLAM2 is proposed. First, we use YOLOv5 for object detection and label the recognized dynamic and static objects. Secondly, the topology graph is constructed using the position information of static objects in space. Then, we propose a weight expression for the topology graph to calculate the similarity of topology in different keyframes. Finally, the re-localization and loop closure detection are determined based on the value of topology similarity. Experiments on public datasets show that the semantic topology graph is effective in improving the correct rate of re-localization and the accuracy of loop closure detection in a dynamic environment.

This paper discusses the research on the use of Artificial Intelligence in autonomous robot object identification. The specific focus of this research is on a wheeled soccer playing robot. The goal is to recognize a ball as an object using the Single Shot MultiBox Detector MobileNetV2 model. This system has multi-vision inputs such as distance measurements and angle values ​​for object detection. This methodology is based on deep learning with the TensorFlow Object Detection API with the MobileNetV2 SSD model. This model is trained with a dataset of 3707 ball images over 617 thousand steps on Google Collaboratory. It was found that the average measurement error of the ball object is 6.58% for the distance when viewed through the robot's front camera. In addition, the omnidirectional camera is able to detect the ball object and angle values ​​from the front of the robot. What makes this research different is the use of distance and angle measurements for detection and the omnidirectional camera for system performance in dynamic environments. This research aims to address the improvement of AI-based object detection systems for autonomous robotics in the context of real-world use cases.

During the continual transitions from lab research to real-world applications of vision-based algorithms, there are significant challenges, e.g. the robustness to adapt to complex environments and the high demands of multi-task, multi-modal learning models. More specifically on visual simultaneous localization and mapping (visual SLAM) for mobile robots, two notable limitations are: (i) the drift issue during pose estimation, especially in dynamic environments, makes the positioning system unstable; (ii) the additional loop detection takes many computational resources for image retrieval and global feature geometry checking. This thesis explores the approaches of visual localization on unmanned ground vehicles by solving those limitations. The conventional feature extraction and matching pipeline for vision-based tasks involve three steps: (i) local feature detection and description, (ii) feature matching, and (iii) outlier rejection. The first contribution of the thesis is adding a feature selection and anticipation stage to reduce tracking drift. We explore the noise model of image features to select a subset of all the observed image features with the best ”contribution” during data association and pose estimation across multiple frames. Conventional SLAM algorithms take a strong assumption of scene rigidity, which limits the application under challenging environments. The second part of the thesis addresses the tough issue in dynamic environments with moving objects. We presented GMC, namely the motion clustering approach, a lightweight dynamic object filtering method. It can distinguish moving objects from static landmarks. Based on the theory of motion coherence within a particular image area, GMC could segment dynamic objects in 3D space. We can provide an efficient and robust correspondence algorithm that can extract dynamic objects from a static background with the method. In this way, we propose a dynamic SLAM system that is real-time and free from expensive GPU processors. In contrast to GMC, the thesis’s third part turns to an object-aware learning-based model for more general dynamic scenarios. We use object detection and tracking as points, lines, planes, etc. We utilize semantic information and extract sparse image features simultaneously to keep track of dynamic objects. The static background and different dynamic objects are jointly optimized in a newly developed bundle adjustment sliding window. The estimated 3D bounding boxes can provide more robust camera tracking and better scene understanding, and better map merging. The fourth part of the thesis leverages the emerging feature learning framework. It proposes a unified self-supervised model called LGDNet to generate both global and local image feature descriptors end-to-end. Global feature descriptors embed the whole image into a compact representation, leading to easier scene comparison. On the other hand, local features focus more on the local region similarities of some salient parts for structure from motion. Our proposed method can directly extract features together with descriptors that encode both local maximum responses and global context information, avoiding duplicate calculations based on different feature extraction criteria.

Simultaneous Localization and Mapping (SLAM) is a crucial technology for intelligent mobile robots to operate successfully in unknown environments. While many excellent SLAM systems have been developed in recent years, most assume that the environment is static, resulting in poor performance in dynamic environments. To address this limitation, we propose multiview stereo (MVS)‐SLAM (MVS‐SLAM), a real‐time semantic RGBD SLAM system with improved‐multiview geometry, built on the RGBD mode of ORB‐SLAM3. MVS‐SLAM tightly integrates semantic and geometric information to tackle the challenges posed by dynamic scenes. To meet the real‐time requirements, the semantic module leverages the latest and fastest object detection network, YOLOv7, to provide semantic prior knowledge for the geometric module. We introduce a novel geometric constraint method that capitalizes on depth images and semantic information to recover three‐dimensional (3D) feature points and initial camera pose. We use a 3D coordinate error threshold to identify dynamic points and remove them using the K‐means clustering algorithm. This approach effectively reduces the impact of dynamic points. We validate MVS‐SLAM using challenging dynamic sequences from the TUM data set, demonstrating that it significantly improves localization accuracy and system robustness in all types of dynamic environments.

This paper provides a comprehensive study of the security of YOLO (You Only Look Once) model series for object detection, emphasizing their evolution, technical innovations, and performance across the COCO dataset. The robustness of YOLO models under adversarial attacks and image corruption, offering insights into their resilience and adaptability, is analyzed in depth. As real-time object detection plays an increasingly vital role in applications such as autonomous driving, security, and surveillance, this review aims to clarify the strengths and limitations of each YOLO iteration, serving as a valuable resource for researchers and practitioners aiming to optimize model selection and deployment in dynamic, real-world environments. The results reveal that YOLOX models, particularly their large variants, exhibit superior robustness compared to other YOLO versions, maintaining higher accuracy under challenging conditions. Our findings serve as a valuable resource for researchers and practitioners aiming to optimize YOLO models for dynamic and adversarial real-world environments while guiding future research toward developing more resilient object detection systems.

A novel visual SLAM method named OPDS-SLAM is proposed to address the dynamic indoor environments and improve the robot’s understanding of the environment. First of all, OPDS-SLAM utilizes YOLOv11 to identify human-beings in the scene and combines the depth information to remove human-beings’ features. Second, the spherical coordinates are used to represent the planar information in the scene, and the stability of the system is improved by constructing a backend optimization with planar features. Third, the objects are represented by the ellipsoids, and the planar constraints are added to enhance the speed and accuracy of object construction. Finally, the experiments conducted on the TUM RGB-D dataset and a self-built dataset demonstrate that OPDS-SLAM can remove moving human-beings’ feature in the indoor environments in real-time. Meanwhile a semantic map containing object-plane information is constructed.