Fast 3D Semantic Mapping in Road Scenes

Traditional approaches to simultaneous localization and mapping (SLAM) are unable to extract semantic information from the scene or meet the high-level task requirements of robots, and the efficiency of 3D map construction is low. To solve this problem, a 3D semantic map construction system is proposed in the paper to build a 3D semantic map. Firstly, the current position of the camera is estimated and optimized based on ORB-SLAM algorithm, and the globally consistent trajectory and pose are obtained. Then the semantic segmentation network is designed to predict semantic category of every pixel. The 3D semantic point cloud information is generated by combing the semantic information and the object point cloud. The global consistent camera pose estimated by visual SLAM algorithm is integrated with the semantic point cloud information to generate a 3D semantic map. Finally, we use an Octomap which is applied to navigation projects for map storage to reduce the storage capacity of the map. The experimental result verifies the accuracy and efficiency of this method.

Robots typically possess sensors of different modalities, such as colour cameras, inertial measurement units, and 3D laser scanners. Often, solving a particular problem becomes easier when more than one modality is used. However, while there are undeniable benefits to combine sensors of different modalities the process tends to be complicated. Segmenting scenes observed by the robot into a discrete set of classes is a central requirement for autonomy as understanding the scene is the first step to reason about future situations. Scene segmentation is commonly performed using either image data or 3D point cloud data. In computer vision many successful methods for scene segmentation are based on conditional random fields (CRF) where the maximum a posteriori (MAP) solution to the segmentation can be obtained by inference. In this paper we devise a new CRF inference method for scene segmentation that incorporates global constraints, enforcing the sets of nodes are assigned the same class label. To do this efficiently, the CRF is formulated as a relaxed quadratic program whose MAP solution is found using a gradient-based optimisation approach. The proposed method is evaluated on images and 3D point cloud data gathered in urban environments where image data provides the appearance features needed by the CRF, while the 3D point cloud data provides global spatial constraints over sets of nodes. Comparisons with belief propagation, conventional quadratic programming relaxation, and higher order potential CRF show the benefits of the proposed method.

Surface defect identification is a crucial task in many manufacturing systems, including automotive, aircraft, steel rolling, and precast concrete. Although image-based surface defect identification methods have been proposed, these methods usually have two limitations: images may lose partial information, such as depths of surface defects, and their precision is vulnerable to many factors, such as the inspection angle, light, color, noise, etc. Given that a three-dimensional (3D) point cloud can precisely represent the multidimensional structure of surface defects, we aim to detect and classify surface defects using a 3D point cloud. This has two major challenges: (i) the defects are often sparsely distributed over the surface, which makes their features prone to be hidden by the normal surface and (ii) different permutations and transformations of 3D point cloud may represent the same surface, so the proposed model needs to be permutation and transformation invariant. In this paper, a two-step surface defect identification approach is developed to investigate the defects’ patterns in 3D point cloud data. The proposed approach consists of an unsupervised method for defect detection and a multi-view deep learning model for defect classification, which can keep track of the features from both defective and non-defective regions. We prove that the proposed approach is invariant to different permutations and transformations. Two case studies are conducted for defect identification on the surfaces of synthetic aircraft fuselage and the real precast concrete specimen, respectively. The results show that our approach receives the best defect detection and classification accuracy compared with other benchmark methods.

Converting three-dimensional (3D) point cloud data to two-dimensional (2D) image based on virtual structured-light system is a popular method in 3D data compression because the image format is easier to process by the exiting storing and transmitting methods. When this method is used in the 3D point cloud compression, the quantization error is introduced during the coordinate mapping. To solve this problem, a virtual structured-light 3D point cloud compression algorithm based on geometric reshaping is presented. And a least-square system parameter optimization method is proposed to further improve the data decoding accuracy. In the proposed method, the 3D spatial coordinates are reshaped to a 2D matrix first, and then the 2D matrix is stored as a “Holoimage” by using the parameter optimized virtual structured-light system. The quantization error introduced by coordinate mapping between X and Y coordinates of the 3D point cloud and the 2D image pixel coordinates is suppressed, so the decoding accuracy is improved. In addition, the geometric information of the 3D point cloud is hidden synchronously when the 3D point cloud is compressed, which is of great significance in the copyright protection of the point cloud data. Experiments verify the effectiveness of the proposed algorithm. The decoding root mean square error (RMSE) of the proposed method is decreased by 79.86% on average compared with the traditional one under the same compression ratio, and the compression ratio of the proposed method is 1.96 times bigger than the traditional one under the similar decoding accuracy.

Building Information Modeling (BIM) has gained wider acceptance in the Architecture/Engineering/Construction (A/E/C) industries in the U.S. and internationally. Capability of BIM technologies in early detection of spatial conflicts among various building systems and components concurrently designed by different design team members prior to construction has great potential in reducing construction delays, loss of productivity, and overall construction costs. Recent advancements in 3D point cloud technologies, such as 3D laser scanning, can greatly augment BIM technology applications in the A/E/C industry by providing a rapid three-dimensional capture of the built environment in great detail and accuracy. This paper discusses the current approaches of implementing 3D point cloud data in BIM and virtual design and construction (VDC) applications during various stages of a project lifecycle and the challenges associated with processing the huge amount of 3D point cloud data. Conversion from discrete 3D point cloud raster data to geometric/vector BIM data remains to be a labor-intensive process. The needs for intelligent geometric feature detection/reconstruction algorithms for automated point cloud processing and issues related to data management will be discussed. This paper also presents some innovative approaches for integrating 3D point cloud data with BIM to efficiently augment built environment design, construction, and management.

Simultaneous localization and mapping (SLAM) becomes popular in the autonomous robot research community. SLAM requests a mobile robot, placed at an unknown location in an unknown environment, to incrementally build a consistent map of this environment while simultaneously determining its location within this map. The odometer data is helpful but the noisy error accumulates. Assistant with measurements from the other sensors, SLAM improves the accuracy by filtering. The extended Kalman filter is the most used method to feedback the error. In this thesis, a monocular vision based SLAM for a wheeled robot is proposed. A single web camera is the only measurement input to correct the odometer data. Extract features and match them based on the scale invariant feature transform (SIFT). Absence of the feature depth information is the key issue for the monocular vision in SLAM. The inverse depth parametrization method solves the problem by present the position in inverse coordinate with some reasonable assumptions. Convergence of the feature position is more easily achieved by the EKF due to the linearity. Combining the odometer data with the inverse depth method, this system provides an accurate map by tests of the indoor image sequence.

The objective of this work is to develop and apply a spectral graph theoretic approach for differentiating between (classifying) additive manufactured (AM) parts contingent on the severity of their dimensional variation from laser-scanned coordinate measurements (3D point cloud). The novelty of the approach is in invoking spectral graph Laplacian eigenvalues as an extracted feature from the laser-scanned 3D point cloud data in conjunction with various machine learning techniques. The outcome is a new method that classifies the dimensional variation of an AM part by sampling less than 5% of the 2 million 3D point cloud data acquired (per part). This is a practically important result, because it reduces the measurement burden for postprocess quality assurance in AM—parts can be laser-scanned and their dimensional variation quickly assessed on the shop floor. To realize the research objective, the procedure is as follows. Test parts are made using the fused filament fabrication (FFF) polymer AM process. The FFF process conditions are varied per a phased design of experiments plan to produce parts with distinctive dimensional variations. Subsequently, each test part is laser scanned and 3D point cloud data are acquired. To classify the dimensional variation among parts, Laplacian eigenvalues are extracted from the 3D point cloud data and used as features within different machine learning approaches. Six machine learning approaches are juxtaposed: sparse representation, k-nearest neighbors, neural network, naïve Bayes, support vector machine, and decision tree. Of these, the sparse representation technique provides the highest classification accuracy (F-score > 97%).

Unmanned mining is one of the most effective methods to solve mine safety and low efficiency. However, it is the key to accurate localization and mapping for underground mining environment. A novel graph simultaneous localization and mapping (SLAM) optimization method is proposed, which is based on Generalized Iterative Closest Point (GICP) three-dimensional (3D) point cloud registration between consecutive frames, between consecutive key frames and between loop frames, and is constrained by roadway plane and loop. GICP-based 3D point cloud registration between consecutive frames and consecutive key frames is first combined to optimize laser odometer constraints without other sensors such as inertial measurement unit (IMU). According to the characteristics of the roadway, the innovative extraction of the roadway plane as the node constraint of pose graph SLAM, in addition to automatic removing the noise point cloud to further improve the consistency of the underground roadway map. A lightweight and efficient loop detection and optimization based on rules and GICP is designed. Finally, the proposed method was evaluated in four scenes (such as the underground mine laboratory), and compared with the existing 3D laser SLAM method (such as Lidar Odometry and Mapping (LOAM)). The results show that the algorithm could realize low drift localization and point cloud map construction. This method provides technical support for localization and navigation of underground mining environment.

Development of vehicle- recognition method on water surfaces using LiDAR data: SPD2 (spherically stratified point projection with diameter and distance)

Most existing Simultaneous Localization and Mapping (SLAM) systems rely on the assumption of static environments to achieve reliable and efficient mapping. However, such methods often suffer from degraded localization accuracy and mapping consistency in dynamic settings, as they lack explicit mechanisms to distinguish between static and dynamic elements. To overcome this limitation, we present BMP-SLAM, a vision-based SLAM approach that integrates semantic segmentation and Bayesian motion estimation to robustly handle dynamic indoor scenes. To enable real-time dynamic object detection, we integrate YOLOv5, a semantic segmentation network that identifies and localizes dynamic regions within the environment, into a dedicated dynamic target detection thread. Simultaneously, the data association Bayesian mobile probability proposed in this paper effectively eliminates dynamic feature points and successfully reduces the impact of dynamic targets in the environment on the SLAM system. To enhance complex indoor robotic navigation, the proposed system integrates semantic keyframe information with dynamic object detection outputs to reconstruct high-fidelity 3D point cloud maps of indoor environments. The evaluation conducted on the TUM RGB-D dataset indicates that the performance of BMP-SLAM is superior to that of ORB-SLAM3, with the trajectory tracking accuracy improved by 96.35%. Comparative evaluations demonstrate that the proposed system achieves superior performance in dynamic environments, exhibiting both lower trajectory drift and enhanced positioning precision relative to state-of-the-art dynamic SLAM methods.

This paper introduces a medium-scale point cloud dataset for semantic SLAM (Simultaneous Localization and Mapping) acquired using a SwissRanger time-of-flight camera. An indoor environment with relatively unfluctuating lighting conditions is considered for mapping and localization. The camera is positioned on a mobile tripod and ready to capture images at prearranged locations in the environment. The prearranged locations are in fact used as ground truth for estimating the variance with poses calculated from SLAM, and also as initial pose estimates for the ICP algorithm (Iterative Closest Point). An interesting point is that, in this work, no type of Inertial Measurement Units or visual odometry techniques has been utilized, given the fact that, data from time-of-flight cameras is noisy and sensitive to external conditions (such as lighting, transparent surfaces, parallel overlapping surfaces etc.). Furthermore, a large collection of household objects is made in order to label the scene with semantic information. The whole SLAM dataset with pose files along with the point clouds of household objects is a major contribution in this paper apart from mapping and plane detection using a publicly available toolkit. Also, a novel metric, a context-based similarity score, for evaluating SLAM algorithms is presented.

Object detection in 3D point clouds is essential in fields such as geospatial intelligence and autonomous driving. The common machine learning problem of scarce labeled training data is even more acute with 3D point cloud data. Active learning provides a framework to prioritize the additional effort to manually annotate unlabeled training data. Most active learning methods for deep learning fall into one of two categories: uncertainty methods and diversity methods. Uncertainty methods select data by assessing model outputs for their confidence and consistency and are therefore dependent on the expected output of each deep learning task. These methods tend to select batches of informative yet highly similar samples to label. Diversity-based active learning aims to create a labeled dataset that is both varied and representative of the remaining unlabeled data. Diversity methods operate directly on the feature representations of the inputs and are thus more flexible with respect to the specifics of the deep learning task. Our current work explores applying diversity methods and uncertainty-diversity hybrid methods to 3D object detection. We evaluate various approaches to incorporate diversity, including K-Medoids clustering, core set selection, and furthest nearest neighbors. We address the high dimensionality of the features extracted from a VoxelNet-based object detector by varying the distance metric used in the active learning algorithms. Furthermore, we compare our results to those obtained using only uncertainty methods. We assess the performance and efficiency of each active learning method in addition to the representativeness and diversity of the labeled datasets produced. We find that hybrid uncertainty-diversity methods outperform other methods in terms of object detection AP50 throughout active learning, annotation efficiency, and class balance.

Three-dimensional (3D) semantic segmentation based on point clouds has become a critical technology in the field of intelligent spatial perception and scene understanding. However, most existing methods cannot fully exploit the spatial feature of 3D point cloud data and the orderliness of two-dimensional (2D) image information, resulting in an inability to effectively improve segmentation accuracy. This study proposes a novel 2D image and 3D point cloud fusion method named shared multi-layer perceptron fusion semantic segmentation (SMFusionSeg) for semantic segmentation. The rich features of 2D image segmentation and spatial information of 3D point clouds are integrated by designing a fusion architecture that facilitates the transfer of 2D semantics to 3D semantics. Specifically, the fusion architecture concatenates the features from both 2D and 3D domains, which are then further integrated through a shared multi-layer perceptron (MLP) and optimized using an attention mechanism to enhance feature effectiveness and distinctiveness. Moreover, a knowledge distillation strategy is introduced to improve the learning capability and segmentation accuracy in parsing complex scenes. Experimental results on the SemanticKITTI dataset demonstrate that the proposed SMFusionSeg achieves a mean Intersection over Union ( mIoU ) of 65.4%, significantly outperforming traditional single-modal methods and many existing segmentation techniques, thereby effectively enhancing the precision and robustness of geographic feature recognition.

Automatic object grasping is a challenging problem and has numerous applications in various fields. Currently, researchers have developed models that use only 3D point cloud data which is not sufficient to capture a complete grasping ability (graspability), because many visual features related to objects are missing in the 3D points. So here we propose an auxiliary convolutional neural network pipeline (CNN) for graspability modeling via simultaneously using visual information from RGBD images and 3D point clouds. For training the auxiliary CNN, we have created new dataset where the most graspable object has been placed in class 5, whereas the least graspable object has been placed in class 1. Our graspability modeling includes, 12 object features, where 9 are extracted from elliptic Fourier descriptors, the other 3 features are Euclidian distance from the centroid, compactness of an object and category of an object. We have thoroughly evaluated our proposed approach by incorporating it into state-of-the-art grasping method Graspnet [8], which has further improved the overall average grasp precision. Additionally, we performed an ablation study on various network elements and loss functions (cross entropy, mean square loss) for obtaining the best accuracy and graspability scores.

In the architecture, engineering, and construction (AEC) industry, creating an indoor model of existing buildings has been a challenging task since the introduction of building information modeling (BIM). Because the process of BIM is primarily manual and implies a high possibility of error, the automated creation of indoor models remains an ongoing research. In this paper, we propose a fully automated method to generate 2D floorplan computer-aided designs (CADs) from 3D point clouds. The proposed method consists of two main parts. The first is to detect planes in buildings, such as walls, floors, and ceilings, from unstructured 3D point clouds and to classify them based on the Manhattan-World (MW) assumption. The second is to generate 3D BIM in the industry foundation classes (IFC) format and a 2D floorplan CAD using the proposed line-detection algorithm. We experimented the proposed method on 3D point cloud data from a university building, residential houses, and apartments and evaluated the geometric quality of a wall reconstruction. We also offer the source code for the proposed method on GitHub.