Development of Autonomous Navigation System Using 3D Map with Geometric and Semantic Information

Traditional approaches to simultaneous localization and mapping (SLAM) are unable to extract semantic information from the scene or meet the high-level task requirements of robots, and the efficiency of 3D map construction is low. To solve this problem, a 3D semantic map construction system is proposed in the paper to build a 3D semantic map. Firstly, the current position of the camera is estimated and optimized based on ORB-SLAM algorithm, and the globally consistent trajectory and pose are obtained. Then the semantic segmentation network is designed to predict semantic category of every pixel. The 3D semantic point cloud information is generated by combing the semantic information and the object point cloud. The global consistent camera pose estimated by visual SLAM algorithm is integrated with the semantic point cloud information to generate a 3D semantic map. Finally, we use an Octomap which is applied to navigation projects for map storage to reduce the storage capacity of the map. The experimental result verifies the accuracy and efficiency of this method.

Building accurate and dense 3D maps of indoor environments is a significant task for mobile robotics, with applications in navigation and semantic mapping. Although the current featurepoint SLAM algorithm is relatively mature, the existing SLAM method often can't work because it cannot extract enough feature information in the some scenes with changeable lighting, especially in indoor environment. In this paper, to deal with the problem of image illumination changing, we propose an adaptive threshold feature point extraction algorithm. With such a new solution, our system can work in environments with the varied illumination. A dense 3D mapping system exploits improved feature point method to estimate the pose of the camera, then the three-dimensional dense map is builded by the optimized camera pose. The surfel model and the deformation map are utilized to further fuse and optimize the point-cloud maps. Finally, the ideal 3D maps are obtained. We fully evaluate our method on the public data sets. Experiments show that the system has a good effect to build dense 3D map in the indoor environment.

Accurate positioning is essential for autonomous driving, but localization using 2D maps is challenging due to the domain gap between perspective view and 2D map. While GNSS accuracy is often limited by atmospheric effects, multipath, and signal blockages. We propose a novel positioning method that combines perspective view images with satellite images retrieved based on rough GNSS positions to achieve precise three-degree-of-freedom (3-DoF) pose estimation. Our method leverages the Swin Transformer for satellite image processing and semantic completion for monocular image analysis. By extracting depth and semantic information from monocular images, we convert these to overhead projections, effectively bridging the gap between different viewpoints. This cross-view transformation allows for precise alignment of features from monocular images onto semantically enriched satellite images. Additionally, we integrate a robust global position estimator using the semantic information from satellite images to further enhance accuracy and robustness. The experimental results demonstrate that our method excels in various complex scenarios; we successfully improved the positioning accuracy within 1 m to 80.67% and the heading in 1° to 33.78%. However, longitudinal localization remains more challenging, with higher errors than lateral positioning.

Perception of the indoor environment is the basis of mobile robot localization, navigation, and path planning, and it is particularly important to construct semantic maps in real time using minimal resources. The existing methods are too dependent on the graphics processing unit (GPU) for acquiring semantic information about the indoor environment, and cannot build the semantic map in real time on the central processing unit (CPU). To address the above problems, this paper proposes a non-use GPU for lightweight indoor semantic map construction algorithm, named NGLSFusion. In the VO method, ORB features are used for the initialization of the first frame, new keyframes are created by optical flow method, and feature points are extracted by direct method, which speeds up the tracking speed. In the semantic map construction method, a pretrained model of the lightweight network LinkNet is optimized to provide semantic information in real time on devices with limited computing power, and a semantic point cloud is fused using OctoMap and Voxblox. Experimental results show that the algorithm in this paper ensures the accuracy of camera pose while speeding up the tracking speed, and obtains a reconstructed semantic map with complete structure without using GPU.

Nowadays, 3D city models are used by the increasing number of applications. Most applications require not only geometric information but also semantic information. As a standard and tool for 3D city model, CityGML, provides a method for storing and managing both geometric and semantic information. Moreover, it also provides the multi-scale representation of 3D building model for efficient visualization. In CityGML, building models are represented in five LODs (Level of Detail), start from LOD0, LOD1, LOD2, LOD3, and LOD4. Each level has different accuracy and detail requirement for visualization. Usually, for obtaining multi-LOD of 3D building model, several data sources are integrated. For example, LiDAR data is used for generating LOD0, LOD1, and LOD2 as close-range photogrammetry data is used for generating more detailed model in LOD3 and LOD4. However, using additional data sources is increasing cost and time consuming. Since the development of TLS (Terrestrial Laser Scanner), data collection for detailed model can be conducted in a relative short time compared to photogrammetry. Point cloud generated from TLS can be used for generating the multi-LOD of building model. This paper gives an overview about the representation of 3D building model in CityGML and also method for generating multi-LOD of building from TLS data. An experiment was conducted using TLS. Following the standard in CityGML, point clouds from TLS were processed resulting 3D model of building in different level of details. Afterward, models in different LOD were converted into XML schema to be used in CityGML. From the experiment, final result shows that TLS can be used for generating 3D models of building in LOD1, LOD2, and LOD3.

Some tasks in the construction industry and urban management field such as site selection and fire response management are usually managed by using a Geographical Information System (GIS), as the tasks in these processes require a high level and amount of integrated geospatial information. Recently, a key element of this integrated geospatial information to emerge is detailed geometrical and semantic information about buildings. In parallel, Building Information Models (BIMs) of today have the capacity for storing and representing such detailed geometrical and semantic information. In this context, the research aimed to investigate the applicability of BIMs in geospatial environment by focusing specifically on these two domains; site selection and fire response management. In the first phase of the research two use case scenarios were developed in order to understand the processes in these domains in a more detailed manner and to establish the scope of a possible software development for transferring information from BIMs into the geospatial environment. In the following phase of the research two data models were developed – a Schema-Level Model View and a geospatial data model. The Schema-Level Model View was used in simplifying the information acquired from the BIM, while the geospatial data model acted as the template for creating physical files and databases in the geospatial environment. Following this, three software components to transfer building information into the geospatial environment were designed, developed, and validated. The first component served for acquiring the building information from the BIM, while the latter two served for transforming the information into the geospatial environment. The overall research demonstrated that it is possible to transfer (high level of geometric and semantic) information acquired from BIMs into the geospatial environment. The results also demonstrated that BIMs provide a sufficient level and amount of (geometric and semantic) information (about the building) for the seamless automation of data management tasks in the site selection and fire response management processes.

Abstract. Semantic information in images—such as meaningful, recognizable regions or objects that are incongruent with the overall scene—can effectively capture attention. The meaning map is currently the primary method for mapping the distribution of semantic information across visual stimuli. However, this approach relies on subjective human ratings of semantic meaningfulness and requires extensive manual annotation for each image prior to analysis. To address these limitations and enable rapid, efficient, and reproducible generation of semantic distribution maps for any given image, this paper proposes an automated method for constructing semantic information maps using multimodal large language models (MLLMs). Our approach generates two types of semantic maps: local semantic information maps, which quantify the semantic content at each spatial location within an image, and global semantic maps, which assess the contextual relevance of local regions to the overall scene. Additionally, the method can generate distribution maps of visual information associated with specific concepts. We argue that this method significantly enhances the precision and flexibility in controlling, measuring, and manipulating semantic information in visual stimuli, thereby advancing research in visual attention and visual language processing. The method is currently under further refinement.

Vehicle self-localization techniques, such as GNSS and Velodyne, can benefit greatly by accessing to an accurate 3D building map, especially in urban areas. Though, an accurate automatic 3D building map reconstruction is challenging in dense urban areas, in which buildings are located close to each other. In general, 3D building map can be generated from adding accurate height information of the building roofs to a detailed 2D map. But 2D building maps usually only provide footprint of the buildings which is not enough for 3D shape modeling of complicated buildings in urban areas such as Shinjuku, Tokyo, Japan. On the other hand, with the development of laser scanning technology, airborne laser scanning (ALS) has now become easily accessible for large scale 3D mapping. However, an automatic 3D map reconstruction using only airborne point cloud is not feasible for dense urban areas, because of the difficulty of defining boundary of each building individually. This paper presents an automatic 3D building map reconstruction technique which integrates airborne laser point cloud with 2D boundary map. Results from Shinjuku area is evaluated for the proposed technique.

Lidar SLAM is mainly suitable for static scenes without moving targets. The existence of motion in actual scenes will limit the application of the algorithm. If there are moving targets in the lidar field of view, the 3D point cloud map constructed by the lidar SLAM will include a series of fake point cloud from the moving target. This situation leads to insufficient accuracy of the constructed map and affects the applications such as navigation and route planning. In this paper, a moving target removal algorithm is proposed, which realizes the detection and removal of moving targets through multi-frame accumulation, change detection, and point cloud clustering of point clouds. The experiments base d on measured lidar point cloud show that the proposed algorithm can optimize the mapping results of the existing lidar SLAM, remove the continuous trajectory caused by the moving target, and construct a more accurate 3D map.

Inter-frame matching of laser point clouds is the key to SLAM's autonomous positioning and 3D map construction. Aiming at the failure of inter frame matching of multi-line lidar SLAM during fast field of view conversion, this paper proposes a spatial 3D mapping method using 180 degree large-field-angle lidar to improve A-LOAM. Firstly, the nonground feature points are obtained by dual-threshold ground filtering, and a novel point cloud feature extraction method based on curvature is used to realize the feature point matching of large-field-angle point cloud; Secondly, using the matching methods of edge point to edge line and plane point to plane block, through the robust frame matching of large-field-angle point cloud data, the spatial 3D mapping is realized; Finally, three-dimensional mapping experiments are carried out on the large-field-angle point cloud data collected in different scenes. The results show that the spatial 3D mapping based on robust frame matching of large-field-angle lidar can reach the accuracy of nearly centimeter level in indoor scene and decimeter level in outdoor scene. It has application prospects in 3D mapping and other fields.

To create with an autonomous mobile robot a 3D volumetric map of a scene it is necessary to gage several 3D scans and to merge them into one consistent 3D model. This paper provides a new solution to the simultaneous localization and mapping (SLAM) problem with six degrees of freedom. Robot motion on natural surfaces has to cope with yaw, pitch and roll angles, turning pose estimation into a problem in six mathematical dimensions. A fast variant of the Iterative Closest Points algorithm registers the 3D scans in a common coordinate system and relocalizes the robot. Finally, consistent 3D maps are generated using a global relaxation. The algorithms have been tested with 3D scans taken in the Mathies mine, Pittsburgh, PA. Abandoned mines pose significant problems to society, yet a large fraction of them lack accurate 3D maps.

In view of existing Visual SLAM (VSLAM) algorithms when constructing semantic map of indoor environment, there are problems with low accuracy and low label classification accuracy when feature points are sparse. This paper proposed a 3D semantic VSLAM algorithm called BMASK-RCNN based on Mask Scoring RCNN. Firstly, feature points of images are extracted by Binary Robust Invariant Scalable Keypoints (BRISK) algorithm. Secondly, map points of reference key frame are projected to current frame for feature matching and pose estimation, and an inverse depth filter is used to estimate scene depth of created key frame to obtain camera pose changes. In order to achieve object detection and semantic segmentation for both static objects and dynamic objects in indoor environments and then construct dense 3D semantic map with VSLAM algorithm, a Mask Scoring RCNN is used to adjust its structure partially, where a TUM RGB-D SLAM dataset for transfer learning is employed. Semantic information of independent targets in scenes provides semantic information including categories, which not only provides high accuracy of localization but also realizes the probability update of semantic estimation by marking movable objects, thereby reducing the impact of moving objects on real-time mapping. Through simulation and actual experimental comparison with other three algorithms, results show the proposed algorithm has better robustness, and semantic information used in 3D semantic mapping can be accurately obtained.

Our research objective is construction of 3D map and development of its practical applications. As an application, the authors had proposed 3D control interface for an operator who controls remote control robot. It is well know that an operator can control a robot easily using bird's-eye view. Using 3D control interface, an operator can control the robot at bird's-eye view. The bird's-eye view is constructed from 3D map and robot model. The operator can recognize relationship between the robot and the environment from 3D map and robot model following real robot motion. In this paper, we will describe 3D control interface and its implementation for HELIOS carrier. We will show an experimental result of 3D control interface. From result, we confirmed the effectiveness of the proposed method.

The autonomous navigation system of human beings is more efficient than artificial intelligence technologies, as prior knowledge such as semantic information of the objects and environment is effectively utilized by humans in complicated tasks. Therefore, the ability to build a complete and efficient semantic map system is critical for robot autonomous navigation. In this article, we propose a new framework for building an environment semantic map. Specifically, we construct a 2D semantic map by projecting 3D scene semantic information recognized by convolutional neural network onto a 2D plane. 3D reconstruction of the environment is achieved by RGB-D SLAM 3D space mapping algorithm. We simplify the 3D clustering connectability, using only three cues, to achieve real-time boundary-aware detection of foreground objects. The network trained on the public benchmark dataset is employed to classify the material of the foreground extraction object. Experimental results have demonstrated that the proposed framework is effective in the contraction of semantic map and can be efficiently utilized in the real-time semantic slam system.

The trade-off between better accuracy and efficiency is an important problem in smoke detection. In deep networks, low-level features are generally considered to be poor in semantic information but rich in keeping geometric details, which is the opposite for high-level features. Therefore, feature fusion plays a crucial role in combining semantic and geometric information. The transformer encoder has the advantages of global computations and perfect memory that searches each element of a sequence and updates it by aggregating information from the entire sequence. In this paper, we proposed a block that combines the geometric information of the low-level features and the semantic information of the high-level features of the backbone with the transform encoder. We boost the fuse-transformer YOLOX-Nano to 44.28% AP on VOC0712 training datasets and on VOC07 test datasets with 416 x 416 resolution, outperforming the original YOLOX-Nano 43.32% AP. If converted to a smoke datasets which containing 9726 training images and 2320 test images. We increase the fuse transformer YOLOX-Nano to 31.48% AP on smoke datasets with a resolution of 416 × 416, outperforming the original YOLOX-Nano by 30.9% AP.