Time-of-Flight Depth Datasets for Indoor Semantic SLAM

Simultaneous Localization and Mapping (SLAM) plays an important role in the computer vision and robotics field. The traditional SLAM framework adopts a strong static world assumption for analysis convenience. How to cope with dynamic environments is of vital importance and attracts more attentions. Existing SLAM systems toward dynamic scenes either solely utilize semantic information, solely utilize geometry information, or naively combine the results from them in a loosely coupled way. In this paper, we present SOF-SLAM: Semantic Optical Flow SLAM, a visual semantic SLAM system toward dynamic environments, which is built on RGB-D mode of ORB-SLAM2. A new dynamic features detection approach called semantic optical flow is proposed, which is a kind of tightly coupled way and can fully take advantage of feature's dynamic characteristic hidden in semantic and geometry information to remove dynamic features effectively and reasonably. The pixel-wise semantic segmentation results generated by SegNet serve as mask in the proposed semantic optical flow to get a reliable fundamental matrix, which is then used to filter out the truly dynamic features. Only the remaining static features are reserved in the tracking and optimization module to achieve accurate camera pose estimation in dynamic environments. Experiments on public TUM RGB-D dataset and in real-world environment are conducted. Compared with ORB-SLAM2, the proposed SOF-SLAM achieves averagely 96.73% improvements in high-dynamic scenarios. It also outperforms the other four state-of-the-art SLAM systems which cope with the dynamic environments.

Typical visual simultaneous localization and mapping (SLAM) systems rely on purely geometric elements. Recently semantic SLAM has become popular due to its capability to obtain high-level understanding of the environment. However, most semantic SLAM systems merely aim to produce semantic maps rather than exploiting semantic constraints to boost the accuracy performance. In this paper, we propose a new RGB-D SLAM system for the use in indoor environments. Given that a ground plane represents a significant part of most indoor scenes, our new system exploit a semantic ground plane constraint inferred from a deep neural network along with geometric constraints on the state estimation to improve the performance. More specifically, we introduce a semantically meaningful geometric primitive, namely, a global ground plane to the optimization process of visual SLAM. The global ground model is able to provide an extra constraint on the state estimation through which both frame-to-keyframe and frame-to-model tracking processes are combined, and low-texture ground plane is effectively utilized. Experimental results for the TUM dataset and a real scene have demonstrated the improved performance brought by the semantic constraint in our method.

Recently, Simultaneous Localization and Mapping (SLAM) has achieved good performance in a certain environment. However, in a complex indoor environment, the tracking of the position of the mobile robot in Visual SLAM may get lost in some scenes with hardly any features. This paper presents a mobile robot localization method using an asynchronous data fusion formed by a wheel odometer and a visual odometer in the Extended Kalman Filter (EKF) based on the Semantic SLAM system. The first approach uses Semantic SLAM based on ORB-SLAM2 with YOLOv3, to get rid of the dynamic object while localizing and mapping, and build a global point cloud map where keyframes information is stored. Then, wheel odometer information and visual odometer information acquire after using a global point cloud map. Finally, the asynchronous data fusion for mobile robot localization, composed of visual odometer and wheel odometer, is realized at a frequency of about 50Hz in EKF. The experimental results on the real robot validate the method in accuracy and robustness by comparing it with ground truth from the motion capture measurement system in indoor environment.

The accuracy and stability of front-end point cloud registration algorithms are crucial for the mapping and localization precision in laser SLAM (simultaneous localization and mapping) systems. Traditional point-to-line and point-to-plane iterative closest point (ICP) registration algorithms, widely used in SLAM front ends, often suffer from low efficiency, significant data dependency during the matching process, and a propensity for local optima. This registration method exhibits a more pronounced local optimum issue in large-scale SLAM mapping, thereby diminishing matching accuracy and increasing reliance on initial values. To address these limitations, this paper introduces NI-LIO, a novel SLAM algorithm that integrates ICP with normal distributions transform (NDT) to enhance localization accuracy, computational efficiency and robustness. By combining the precision of ICP with the robustness of NDT, the proposed algorithm significantly improves system stability and localization accuracy. The analysis of mapping and localization experiments indicates a significant reduction in errors compared to traditional SLAM algorithms, with experiments showing a REMS value decrease of over 20%. Compared to ALOAM, FAST_LIO2 and Lego-LOAM algorithms, the new NI-LIO algorithm shows improvements in both accuracy and stability, enabling the construction of a more precise and consistent global map. This algorithm exhibits excellent adaptability to various environments.

Simultaneous localization and mapping (SLAM) is a well-known positioning approach in GPS-denied environments such as urban canyons and inside buildings. Autonomous/aided target detection and recognition (ATR) is commonly used in military application to detect threats and targets in outdoor environments. This papers present approaches to combine SLAM with ATR in ways that compensate for the drawbacks in each method. The methods use physical objects that are recognizable by ATR as unambiguous features in SLAM, while SLAM provides the ATR with better position estimates. Landmarks in the form of 3D point features based on normal aligned radial features (NARF) are used in conjunction with identified objects and 3D object models that replace landmarks when possible. This leads to a more compact map representation with fewer landmarks, which partly compensates for the introduced cost of the ATR. We analyze three approaches to combine SLAM and 3D-data; point-point matching ignoring NARF features, point-point matching using the set of points that are selected by NARF feature analysis, and matching of NARF features using nearest neighbor analysis. The first two approaches are is similar to the common iterative closest point (ICP). We propose an algorithm that combines EKF-SLAM and ATR based on rectangle estimation. The intended application is to improve the positioning of a first responder moving through an indoor environment, where the map offers localization and simultaneously helps locate people, furniture and potentially dangerous objects such as gas canisters.

Augmented Reality (AR) has increasingly benefited from the use of Simultaneous Localization and Mapping (SLAM) systems. This technology has enabled developers to create AR markerless applications, but lack semantic understanding of their environment. The inclusion of this information would empower AR applications to better react to the surroundings more realistically. To gain semantic knowledge, in recent years, focus has shifted toward fusing SLAM systems with neural networks, giving birth to the field of Semantic SLAM. Building on existing research, this paper aimed to create a SLAM system that generates a 3D map using ORB-SLAM2 and enriches it with semantic knowledge originated from the Fast-SCNN network. The key novelty of our approach is a new method for improving the predictions of neural networks, employed to balance the loss of accuracy introduced by efficient real-time models. Exploiting sensor information provided by a smartphone, GPS coordinates are utilized to query the OpenStreetMap database. The returned information is used to understand which classes are currently absent in the environment, so that they can be removed from the network’s prediction with the goal of improving its accuracy. We achieved 87.40% Pixel Accuracy with Fast-SCNN on our custom version of COCO-Stuff and showed an improvement by involving GPS data for our self-made smartphone dataset resulting in 90.24% Pixel Accuracy. Having in mind the use on smartphones, the implementation aimed to find a trade-off between accuracy and efficiency, making the system achieve an unprecedented speed. To this end, the system was carefully designed and a strong focus on lightweight neural networks is also fundamental. This enabled the creation of an above real-time Semantic SLAM system that we called EnvSLAM (Environment SLAM). Our extensive evaluation reveals the efficiency of the system features and the operability in above real-time (48.1 frames per second with an input image resolution of 640 × 360 pixels). Moreover, the GPS integration indicates an effective improvement of the network’s prediction accuracy.

Accurate localization and reliable mapping is essential for autonomous navigation of robots. As one of the core technologies for autonomous navigation, Simultaneous Localization and Mapping (SLAM) has attracted widespread attention in recent decades. Based on vision or LiDAR sensors, great efforts have been devoted to achieving real-time SLAM that can support a robot’s state estimation. However, most of the mature SLAM methods generally work under the assumption that the environment is static, while in dynamic environments they will yield degenerate performance or even fail. In this paper, first we quantitatively evaluate the performance of the state-of-the-art LiDAR-based SLAMs taking into account different pattens of moving objects in the environment. Through semi-physical simulation, we observed that the shape, size, and distribution of moving objects all can impact the performance of SLAM significantly, and obtained instructive investigation results by quantitative comparison between LOAM and LeGO-LOAM. Secondly, based on the above investigation, a novel approach named EMO to eliminating the moving objects for SLAM fusing LiDAR and mmW-radar is proposed, towards improving the accuracy and robustness of state estimation. The method fully uses the advantages of different characteristics of two sensors to realize the fusion of sensor information with two different resolutions. The moving objects can be efficiently detected based on Doppler effect by radar, accurately segmented and localized by LiDAR, then filtered out from the point clouds through data association and accurate synchronized in time and space. Finally, the point clouds representing the static environment are used as the input of SLAM. The proposed approach is evaluated through experiments using both semi-physical simulation and real-world datasets. The results demonstrate the effectiveness of the method at improving SLAM performance in accuracy (decrease by 30% at least in absolute position error) and robustness in dynamic environments.

Traditional approaches to simultaneous localization and mapping (SLAM) are unable to extract semantic information from the scene or meet the high-level task requirements of robots, and the efficiency of 3D map construction is low. To solve this problem, a 3D semantic map construction system is proposed in the paper to build a 3D semantic map. Firstly, the current position of the camera is estimated and optimized based on ORB-SLAM algorithm, and the globally consistent trajectory and pose are obtained. Then the semantic segmentation network is designed to predict semantic category of every pixel. The 3D semantic point cloud information is generated by combing the semantic information and the object point cloud. The global consistent camera pose estimated by visual SLAM algorithm is integrated with the semantic point cloud information to generate a 3D semantic map. Finally, we use an Octomap which is applied to navigation projects for map storage to reduce the storage capacity of the map. The experimental result verifies the accuracy and efficiency of this method.

In recent years, dense visual SLAM (Simultaneous Localization And Mapping) has been proved that it has lots of advantages on the accuracy of pose estimation compared to sparse features due to exploiting the more available information in image data. In this paper, we propose a scalable dense frame-to-model SLAM system based on KinectFusion algorithm. We design an effective active volume shift strategy to extend the tracking range without increment on storage resource. Furthermore, we propose three local optimization methods including motion predicting, weighted ICP (Iterative Closest Point) and multi-modelframe combined estimation to improve the tracking stability and decrease the accumulative error. Note that, as the basis of on-line augmented reality application, our work focus on local optimization rather than global closing loop optimization which is a classical problem in SLAM. At last, we evaluate our optimized approaches on publicly available benchmark datasets, and compare it with the original KinectFusion, a similar method and another two frame-to-frame RGBD SLAM methods. The results indicate our approaches yield a certain degree improvement on the performance of tracking accuracy.

Object SLAM introduces the concept of objects into Simultaneous Localization and Mapping (SLAM) and helps understand indoor scenes for mobile robots and object-level interactive applications. The state-of-art object SLAM systems face challenges such as partial observations, occlusions, unobservable problems, limiting the mapping accuracy and robustness. This letter proposes a novel monocular Semantic Object SLAM (SO-SLAM) system that addresses the introduction of object spatial constraints. We explore three representative spatial constraints, including scale proportional constraint, symmetrical texture constraint and plane supporting constraint. Based on these semantic constraints, we propose two new methods - a more robust object initialization method and an orientation fine optimization method. We have verified the performance of the algorithm on the public datasets and an author-recorded mobile robot dataset and achieved a significant improvement on mapping effects. We will release the code here. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

A high efficient multi-robot simultaneous localization and mapping system using partial computing offloading assisted cloud point registration strategy

When firefighters search inside a building that is at risk of collapse due to abandonment or disasters such as fire, they use old architectural drawings or a simple monitoring method involving a video device attached to a robot. However, using these methods, the disaster situation inside a building at risk of collapse is difficult to detect and identify. Therefore, we investigate the generation of digital maps for a disaster site to accurately analyze internal situations. In this study, a robot combined with a low-cost camera and two-dimensional light detection and ranging (2D-lidar) traverses across a floor to estimate the location of obstacles while drawing an internal map of the building. We propose an algorithm that detects the floor and then determines the possibility of entry, tracks collapses, and detects obstacles by analyzing patterns on the floor. The robot’s location is estimated, and a digital map is created based on Hector simultaneous localization and mapping (SLAM). Subsequently, the positions of obstacles are estimated based on the range values detected by 2D-lidar, and the position of the obstacles are identified on the map using the map update method in semantic SLAM. All equipment are implemented using low-specification devices, and the experiments are conducted using a low-cost robot that affords near-real-time performance. The experiments are conducted in various actual internal environments of buildings. In terms of obstacle detection performance, almost all obstacles are detected, and their positions identified on the map with a high accuracy of 89%.

Convolutional neural networks (CNNs) for visual semantic segmentation have been attracting considerable attention recently because of their superior support for many significant tasks, such as autonomous driving, semantic SLAM (simultaneous localization and mapping) and remote sensing surveying and mapping. These kinds of applications generally need to be implemented on the smart terminals, which means that a kind of hardware platform with high energy efficiency and real-time performance is required. However, CNNs for semantic segmentation usually contain some symmetrical encoders and decoders, corresponding to the down-sampling process (e.g., pooling, convolution) and the up-sampling process (e.g., unpooling, deconvolution). All of these processes are computing and storage intensive, which limits their applicability in the resource constrained embedded systems. In this paper, an FPGA-based accelerator programed by OpenCL is proposed. We evaluate its performance on the CamVid dataset. The global accuracy only drops by 2.04% with 8-bit quantization. Additionally, the system shows 48.89 GOPS and 2.4x real-time performance against CPU when running on an Arria-10 GX1150 device.

Localization and mapping are key requirements for autonomous mobile systems to perform navigation and interaction tasks. Iterative Closest Point (ICP) is widely applied for LiDAR scan-matching in the robotic community. In addition, the standard ICP algorithm only considers geometric information when iteratively searching for the nearest point. However, ICP individually cannot achieve accurate point-cloud registration performance in challenging environments such as dynamic environments and highways. Moreover, the computation of searching for the closest points is an expensive step in the ICP algorithm, which is limited to meet real-time requirements, especially when dealing with large-scale point-cloud data. In this paper, we propose a segment-based scan-matching framework for six degree-of-freedom pose estimation and mapping. The LiDAR generates a large number of ground points when scanning, but many of these points are useless and increase the burden of subsequent processing. To address this problem, we first apply an image-based ground-point extraction method to filter out noise and ground points. The point cloud after removing the ground points is then segmented into disjoint sets. After this step, a standard point-to-point ICP is applied into to calculate the six degree-of-freedom transformation between consecutive scans. Furthermore, once closed loops are detected in the environment, a 6D graph-optimization algorithm for global relaxation (6D simultaneous localization and mapping (SLAM)) is employed. Experiments based on publicly available KITTI datasets show that our method requires less runtime while at the same time achieves higher pose estimation accuracy compared with the standard ICP method and its variants.

The robot simultaneous localization and mapping (SLAM) is a very important and useful technology in the robotic field. However, the environmental map constructed by the traditional visual SLAM method contains little semantic information, which cannot satisfy the needs of complex applications. The semantic map can deal with this problem efficiently, which has become a research hot spot. This paper proposed an improved deep residual network- (ResNet-) based semantic SLAM method for monocular vision robots. In the proposed approach, an improved image matching algorithm based on feature points is presented, to enhance the anti-interference ability of the algorithm. Then, the robust feature point extraction method is adopted in the front-end module of the SLAM system, which can effectively reduce the probability of camera tracking loss. In addition, the improved key frame insertion method is introduced in the visual SLAM system to enhance the stability of the system during the turning and moving of the robot. Furthermore, an improved ResNet model is proposed to extract the semantic information of the environment to complete the construction of the semantic map of the environment. Finally, various experiments are conducted and the results show that the proposed method is effective.