A method for automatically generating semantic information distribution maps of images

The robot simultaneous localization and mapping (SLAM) is a very important and useful technology in the robotic field. However, the environmental map constructed by the traditional visual SLAM method contains little semantic information, which cannot satisfy the needs of complex applications. The semantic map can deal with this problem efficiently, which has become a research hot spot. This paper proposed an improved deep residual network- (ResNet-) based semantic SLAM method for monocular vision robots. In the proposed approach, an improved image matching algorithm based on feature points is presented, to enhance the anti-interference ability of the algorithm. Then, the robust feature point extraction method is adopted in the front-end module of the SLAM system, which can effectively reduce the probability of camera tracking loss. In addition, the improved key frame insertion method is introduced in the visual SLAM system to enhance the stability of the system during the turning and moving of the robot. Furthermore, an improved ResNet model is proposed to extract the semantic information of the environment to complete the construction of the semantic map of the environment. Finally, various experiments are conducted and the results show that the proposed method is effective.

A multi-modal 3D object-detection method, based on data from cameras and LiDAR, has become a subject of research interest. PointPainting proposes a method for improving point-cloud-based 3D object detectors using semantic information from RGB images. However, this method still needs to improve on the following two complications: first, there are faulty parts in the image semantic segmentation results, leading to false detections. Second, the commonly used anchor assigner only considers the intersection over union (IoU) between the anchors and ground truth boxes, meaning that some anchors contain few target LiDAR points assigned as positive anchors. In this paper, three improvements are suggested to address these complications. Specifically, a novel weighting strategy is proposed for each anchor in the classification loss. This enables the detector to pay more attention to anchors containing inaccurate semantic information. Then, SegIoU, which incorporates semantic information, instead of IoU, is proposed for the anchor assignment. SegIoU measures the similarity of the semantic information between each anchor and ground truth box, avoiding the defective anchor assignments mentioned above. In addition, a dual-attention module is introduced to enhance the voxelized point cloud. The experiments demonstrate that the proposed modules obtained significant improvements in various methods, consisting of single-stage PointPillars, two-stage SECOND-IoU, anchor-base SECOND, and an anchor-free CenterPoint on the KITTI dataset.

The autonomous navigation system of human beings is more efficient than artificial intelligence technologies, as prior knowledge such as semantic information of the objects and environment is effectively utilized by humans in complicated tasks. Therefore, the ability to build a complete and efficient semantic map system is critical for robot autonomous navigation. In this article, we propose a new framework for building an environment semantic map. Specifically, we construct a 2D semantic map by projecting 3D scene semantic information recognized by convolutional neural network onto a 2D plane. 3D reconstruction of the environment is achieved by RGB-D SLAM 3D space mapping algorithm. We simplify the 3D clustering connectability, using only three cues, to achieve real-time boundary-aware detection of foreground objects. The network trained on the public benchmark dataset is employed to classify the material of the foreground extraction object. Experimental results have demonstrated that the proposed framework is effective in the contraction of semantic map and can be efficiently utilized in the real-time semantic slam system.

To apply information theory to more areas, the author proposed semantic information G theory, which is a natural generalization of Shannon’s information theory. This theory uses the P-T probability framework so that likelihood functions and truth functions (or membership functions), as well as sampling distributions, can be put into the semantic mutual information formula at the same time. Hence, we can connect statistics and (fuzzy) logic. Rate-distortion function R(D) becomes rate-verisimilitude function R(G) (G is the lower limit of the semantic mutual information) when the distortion function is replaced with the semantic information function. Seven difficult tasks are 1) clarifying the relationship between minimum information and maximum entropy in statistical mechanics, 2) compressing images according to visual discrimination, 3) multilabel learning for obtaining truth functions or membership functions from sampling distributions, 4) feature classifications with maximum mutual information criterion, 5) proving the convergence of the expectation-maximization algorithm for mixture models, 6) interpreting Popper’s verisimilitude and reconciling the contradiction between the content approach and the likeness approach, and 7) providing practical confirmation measures and clarifying the raven paradox. This paper simply introduces the mathematical methods for these tasks and the conclusions. The P-T probability framework and the semantic information G theory should have survived the tests. They should have broader applications. Further studies are needed for combining them with neural networks for machine learning.

This paper proposes a method to build a semantic map for robot navigation by using a laser sensor and monocular vision. Laser data is used to construct an environmental grid map, which can guide the robot forward without collision. In this paper, we assume the indoor environment is one of the floors of the building. By using the monocular vision, the mobile robot can automatically search the doorplate of each office room and recognize the room number, which is used as an environmental semantic identifier. We assign the semantic label to the grid map so that the constructed map includes not only the spatial information but also the semantic information. The robot traverses the target floor and completes the construction of the semantic map automatically. The experimental result has shown the robot can be navigated to any given address in the target floor to complete the visit task by using our proposed semantic map.

Perception of the indoor environment is the basis of mobile robot localization, navigation, and path planning, and it is particularly important to construct semantic maps in real time using minimal resources. The existing methods are too dependent on the graphics processing unit (GPU) for acquiring semantic information about the indoor environment, and cannot build the semantic map in real time on the central processing unit (CPU). To address the above problems, this paper proposes a non-use GPU for lightweight indoor semantic map construction algorithm, named NGLSFusion. In the VO method, ORB features are used for the initialization of the first frame, new keyframes are created by optical flow method, and feature points are extracted by direct method, which speeds up the tracking speed. In the semantic map construction method, a pretrained model of the lightweight network LinkNet is optimized to provide semantic information in real time on devices with limited computing power, and a semantic point cloud is fused using OctoMap and Voxblox. Experimental results show that the algorithm in this paper ensures the accuracy of camera pose while speeding up the tracking speed, and obtains a reconstructed semantic map with complete structure without using GPU.

Localization technology is essential for robots. The map created to recognize the location mainly contains metric information. However, in a changing environment, a Semantic Map containing Semantic object information is required a multi-modal sensor composed of multiple types and multiple sensors[RGBD, thermal, night vision, global shutter camera, microphone, 16 channel laser sensor(=Lidar)] was created for semantic information recognition and semantic map creation in various environments, and calibration was performed to integrate the coordinate system. After that, we introduce the method of generating the metric map according to the configuration of the multi-modal sensor. Also, we propose a method to obtain a single accurate location by integrating the location recognition results obtained from various maps. This can be used to specify the position of the semantic object. Finally, it can be expected that the semantic object and the semantic map information obtained through the multi-modal sensor can be used for various different sensor configurations and various types of robots.

Cherry growth modeling based on Prior Distance Embedding contrastive learning: Pre-training, anomaly detection, semantic segmentation, and temporal modeling

The article presents a navigation system that utilizes a semantic map created on a hexagonal grid. The system plans the path by incorporating semantic and metric information while considering the vehicle’s dynamic constraints. The article concludes by discussing a low-level control algorithm used in the system. This solution’s advantages include using a semantic map on a hexagonal grid, which enables more efficient and accurate navigation. Creating a map of allowable speeds based on the semantic map provides an additional layer of information that can help optimize the vehicle’s trajectory. Incorporating both semantic and metric information in the path-planning process leads to a more precise and tailored navigation solution that accounts for the vehicle’s capabilities and the environment it is operating in. Finally, the low-level control algorithm ensures that the vehicle follows the planned trajectory while considering real-time sensor data and other factors affecting its movement. Through this article, we aim to provide insights into the cutting-edge advancements in path planning techniques and shed light on the potential of combining hexagonal grids, vehicle dynamics constraints, and semantic awareness. These innovations have the potential to revolutionize autonomous navigation systems, enabling vehicles to navigate complex environments with greater efficiency, safety, and adaptability.

Large-scale semantic mapping is crucial for outdoor autonomous agents to perform high-level tasks such as planning and navigation. In this paper, we propose a novel method for large-scale 3D semantic reconstruction through implicit representations from posed LiDAR measurements alone. We first leverage an octree-based hierarchical structure to store implicit features, then these implicit features are decoded to signed distance value and semantic information through shallow Multilayer Perceptrons (MLPs). We leverage radial window self-attention networks to predict the semantic labels of point clouds. We then jointly optimize the feature embedding and MLP parameters with a self-supervision paradigm for point-cloud geometry and a pseudo-supervision paradigm for semantic and panoptic labels. Subsequently, geometric structures and object categories for novel points in the unseen area are regressed, and the marching cubes method is exploited to subdivide and visualize scenes in the inferring stage. Experiments on two real-world datasets, SemanticKITTI and SemanticPOSS, demonstrate the superior segmentation efficiency and mapping effectiveness of our framework compared to current state-of-the-art 3D semantic LiDAR mapping methods.

This paper presents an autonomous navigation system. Our system is based on an accurate 3D map, which includes “geometric information” (e.g., curb, wall, street tree) and “semantic information” (e.g., sidewalk, roadway, crosswalk) extracted by environmental recognition. By using the semantic map, we can obtain the suitable area to keep away from undesired places. Furthermore, by comparing the map with real-time 3D geometric information from LIDAR, we obtain the robot position. To show the effectiveness of our system, we conduct a 3D semantic map construction experiment and driving test. The experiment results show that the proposed system enables accurate and highly reproducible localization and stable autonomous mobility.

Power, size and weight are major constraining factors for UAVs, which have caused researchers to investigate alternative information representations, such as semantics, to reduce the computational load onboard UAVs. However, the connection between spatial and semantic information has not been fully explored and the literature on including semantic information for UAV navigation is limited. This paper provides a new taxonomy on spatial and semantic relations through consideration of a selected subset of the literature. Concepts of semantic information are related with spatial maps and two hypotheses for improving algorithms based on the literature are generated. The algorithms are tested in numerical simulations on the task of semantic mapping and the effects of influential parameters are examined. Results indicate that a defined hierarchical structure of semantic information enables improved map reproduction. The proposed algorithm exhibits the potential to improve the use of semantic information in UAV deployment scenarios, such as agriculture, search and rescue and industrial inspections.

In view of existing Visual SLAM (VSLAM) algorithms when constructing semantic map of indoor environment, there are problems with low accuracy and low label classification accuracy when feature points are sparse. This paper proposed a 3D semantic VSLAM algorithm called BMASK-RCNN based on Mask Scoring RCNN. Firstly, feature points of images are extracted by Binary Robust Invariant Scalable Keypoints (BRISK) algorithm. Secondly, map points of reference key frame are projected to current frame for feature matching and pose estimation, and an inverse depth filter is used to estimate scene depth of created key frame to obtain camera pose changes. In order to achieve object detection and semantic segmentation for both static objects and dynamic objects in indoor environments and then construct dense 3D semantic map with VSLAM algorithm, a Mask Scoring RCNN is used to adjust its structure partially, where a TUM RGB-D SLAM dataset for transfer learning is employed. Semantic information of independent targets in scenes provides semantic information including categories, which not only provides high accuracy of localization but also realizes the probability update of semantic estimation by marking movable objects, thereby reducing the impact of moving objects on real-time mapping. Through simulation and actual experimental comparison with other three algorithms, results show the proposed algorithm has better robustness, and semantic information used in 3D semantic mapping can be accurately obtained.

Abstract. Semantic mapping is crucial for intelligent obstacle avoidance and planning in SLAM systems. We proposed an autonomous semantic mapping approach that integrates multimodal semantic segmentation and SLAM techniques to construct a dense 3D semantic map in real time. Multimodal semantic segmentation based on camera images and LiDAR point clouds is performed in each frame, which assigns image segmentation labels to LiDAR points, generating per-frame 3D semantic information. These segmented frames are then incrementally fused within the SLAM framework to produce a globally consistent semantic map of the environment. The proposed approach is validated through real-world experiments conducted around the Star Lake Building at Wuhan University using the Luo-Jia Explorer system. The experimental results show that our method achieves real-time performance with an inference speed of up to 14Hz on an RTX 4070 GPU, effectively processing sensor data on 10Hz while maintaining high segmentation accuracy in both indoor and outdoor scenarios.

Pre-trained DCNN trained on a large-scale image database can be used as a universal feature representation for image classification, which has achieved significant progress in some image recognition tasks. Compared with other image recognition tasks, directly utilizing a single convolutional feature as feature representation for vein recognition task cannot achieve the impressive result due to the sparse distribution of vein information. Therefore, to obtain more representative and discriminative convolutional feature for vein recognition, a novel multi-layer convolutional features' concatenation with semantic feature selector is proposed in this paper. In the pre-trained DCNN, different convolutional layers can encode different-level feature information. High-level convolutional features with vein information cover more semantic information and low-level convolutional features with vein information cover more detail information. However, low-level convolutional features also contain some background information. Therefore, in order to remove the background information of low-level convolutional features, a novel semantic feature selector is presented. First, the proposed local max-pooling of preserving spatial position (LMP-PSP) information is applied on activation map obtained by adding up all feature maps of the high-level convolutional layer to generate the semantic weighting map, which reflects key vein information of high-level convolutional features. Then, semantic weighting map is regarded as a feature selector to discard the background information of the low-level convolutional features and preserve the detail information of low level convolutional features. Finally, low-level convolutional features with vein information are selectively linked to high-level convolutional features with vein information based on the proposed semantic feature selector. A series of rigorous experiments on two lab-made vein databases named CUMT-Hand-Dorsa Vein database and CUMT-Palm Vein database is conducted to verify the effectiveness and feasibility of the proposed model. Besides, additional experiments with PUT Palm Vein database and the subset of PolyU database illustrate its generalization ability and robustness.