Indoor Semantic Map Building for Robot Navigation

Abstract. Semantic information in images—such as meaningful, recognizable regions or objects that are incongruent with the overall scene—can effectively capture attention. The meaning map is currently the primary method for mapping the distribution of semantic information across visual stimuli. However, this approach relies on subjective human ratings of semantic meaningfulness and requires extensive manual annotation for each image prior to analysis. To address these limitations and enable rapid, efficient, and reproducible generation of semantic distribution maps for any given image, this paper proposes an automated method for constructing semantic information maps using multimodal large language models (MLLMs). Our approach generates two types of semantic maps: local semantic information maps, which quantify the semantic content at each spatial location within an image, and global semantic maps, which assess the contextual relevance of local regions to the overall scene. Additionally, the method can generate distribution maps of visual information associated with specific concepts. We argue that this method significantly enhances the precision and flexibility in controlling, measuring, and manipulating semantic information in visual stimuli, thereby advancing research in visual attention and visual language processing. The method is currently under further refinement.

The robot simultaneous localization and mapping (SLAM) is a very important and useful technology in the robotic field. However, the environmental map constructed by the traditional visual SLAM method contains little semantic information, which cannot satisfy the needs of complex applications. The semantic map can deal with this problem efficiently, which has become a research hot spot. This paper proposed an improved deep residual network- (ResNet-) based semantic SLAM method for monocular vision robots. In the proposed approach, an improved image matching algorithm based on feature points is presented, to enhance the anti-interference ability of the algorithm. Then, the robust feature point extraction method is adopted in the front-end module of the SLAM system, which can effectively reduce the probability of camera tracking loss. In addition, the improved key frame insertion method is introduced in the visual SLAM system to enhance the stability of the system during the turning and moving of the robot. Furthermore, an improved ResNet model is proposed to extract the semantic information of the environment to complete the construction of the semantic map of the environment. Finally, various experiments are conducted and the results show that the proposed method is effective.

In recent years, with the rapid development of artificial intelligence, simultaneous localization and mapping technology to solve robot perception problems has been widely used in automatic driving, robot navigation, augmented reality and other fields. In this paper, a method of reconstructing large outdoor 3D dense semantic map based on monocular vision is proposed. Firstly, the motion trajectory of the camera is estimated by visual odometer, and the depth data is obtained by monocular depth estimation algorithm. Meanwhile, a dense conditional random field (CRF) image segmentation system based on deep learning and super-pixel distribution is used for semantic segmentation. Then 2D image semantics are gradually transferred to 3D point clouds by Bayesian progressive label migration strategy and further optimize 3D labels through a novel CRF model. Super pixels are utilized to enforce smoothness and form robust PN high order potential. Finally, dense 3D semantic maps of urban environment is generated by arbitrary length image sequence. The algorithm is tested on KITTI dataset, and the constructed semantic map shows that the proposed algorithm can reconstruct globally consistent semantic map in large-scale outdoor scenes.

The autonomous navigation system of human beings is more efficient than artificial intelligence technologies, as prior knowledge such as semantic information of the objects and environment is effectively utilized by humans in complicated tasks. Therefore, the ability to build a complete and efficient semantic map system is critical for robot autonomous navigation. In this article, we propose a new framework for building an environment semantic map. Specifically, we construct a 2D semantic map by projecting 3D scene semantic information recognized by convolutional neural network onto a 2D plane. 3D reconstruction of the environment is achieved by RGB-D SLAM 3D space mapping algorithm. We simplify the 3D clustering connectability, using only three cues, to achieve real-time boundary-aware detection of foreground objects. The network trained on the public benchmark dataset is employed to classify the material of the foreground extraction object. Experimental results have demonstrated that the proposed framework is effective in the contraction of semantic map and can be efficiently utilized in the real-time semantic slam system.

Perception of the indoor environment is the basis of mobile robot localization, navigation, and path planning, and it is particularly important to construct semantic maps in real time using minimal resources. The existing methods are too dependent on the graphics processing unit (GPU) for acquiring semantic information about the indoor environment, and cannot build the semantic map in real time on the central processing unit (CPU). To address the above problems, this paper proposes a non-use GPU for lightweight indoor semantic map construction algorithm, named NGLSFusion. In the VO method, ORB features are used for the initialization of the first frame, new keyframes are created by optical flow method, and feature points are extracted by direct method, which speeds up the tracking speed. In the semantic map construction method, a pretrained model of the lightweight network LinkNet is optimized to provide semantic information in real time on devices with limited computing power, and a semantic point cloud is fused using OctoMap and Voxblox. Experimental results show that the algorithm in this paper ensures the accuracy of camera pose while speeding up the tracking speed, and obtains a reconstructed semantic map with complete structure without using GPU.

One of most important techniques that plays a key role in elevating a mobile robot’s independence is its ability to construct a map from an unknown surrounding in an unknown initial position, and with the use of onboard sensors, localize itself in this map. This technique is called simultaneous localization and mapping or SLAM. Over the last 30 years, numerous new and interesting inquiries have been raised, with the improvement of new techniques, new computational instruments, and new sensors. However, the big challenges facing mobile robots in the next decade, as in the autonomous urban vehicles, require extended representations that exceed traditional mapping found in classical SLAM systems, i.e. the so-called semantic representation. The main goal of a SLAM system with semantic concepts is to expand mobile robots’ services and strengthen human-robot interaction. Related works reviewed show that the visual-based SLAM or VSLAM has received a great deal of interest in the last decade. This is due to the visual sensors’ capability to provide information of the scene whereas they are low-priced, smaller and lighter than other sensors. Unlike the metric representation, semantic mapping is still immature, and it comes up short on durable formulation. This paper aims to systematically review recent researches related to the semantic VSLAM, including its types, approaches, and challenges. The paper also deals with the classical SLAM system by giving an overview of necessary information before getting into detail. This review also provides new researches in the SLAM domain facilities to further understand the anatomy of modern VSLAM generation, discover recent algorithms, and apprehend some open challenges.

This thesis presents a recent research on the problem of environmental modeling for both localization and map building for wheel-based, differential driven, fully autonomous and self-contained mobile robots. The robots behave in an indoor office environment. They have a multi-sensor setup where the encoders are used for odometry and two exteroperceptive sensors, a 360° laser scanner and a monocular vision system, are employed to perceive the surrounding. The whole approach is feature based meaning that instead of directly using the raw data from the sensor features are firstly extracted. This allows the filtering of noise from the sensors and permits taking account of the dynamics in the environment. Furthermore, a properly chosen feature extraction has the characteristic of better isolating informative patterns. When describing these features care has to be taken that the uncertainty from the measurements is taken into account. The representation of the environment is crucial for mobile robot navigation. The model defines which perception capabilities are required and also which navigation technique is allowed to be used. The presented environmental model is both metric and topological. By coherently combining the two paradigms the advantages of both methods are added in order to face the drawbacks of a single approach. The capabilities of the hybrid approach are exploited to model an indoor office environment where metric information is used locally in structures (rooms, offices), which are naturally defined by the environment itself while the topology of the whole environment is resumed separately thus avoiding the need of global metric consistency. The hybrid model permits the use of two different and complementary approaches for localization, map building and planning. This combination permits the grouping of all the characteristics which enables the following goals to be met: Precision, robustness and practicability. Metric approaches are, per definition, precise. The use of an Extended Kalman Filter (EKF) permits to have a precision which is just bounded by the quality of the sensor data. Topological approaches can easily handle large environments because they do not heavily rely on dead reckoning. Global consistency can, therefore, be maintained for large environments. Consistent mapping, which handle large environments, is achieved by choosing a topological localization approach, based on a Partially Observable Markov Decision Process (POMDP), which is extended to simultaneous localization and map building. The theory can be mathematically proven by making some assumptions. However, as stated during the whole work, at the end the robot itself has to show how good the theory is when used in the real world. For this extensive experimentation for a total of more than 9 km is performed with fully autonomous self-contained robots. These experiments are then carefully analyzed. With the metric approach precision with error bounds of about 1 cm and less than 1 degree is further confirmed by ground truth measurements with a mean error of less than 1 cm. The topological approach is successfully tested by simultaneous localization and map building where the automatically created maps turned out to work better than the a priori maps. Relocation and closing the loop are also successfully tested.

With the increasingly complex application scenarios of indoor mobile robots, traditional navigation methods based on metric maps have been unable to meet people's needs. Mobile robots not only need to perceive the spatial geometric information of the environment, but also need to deeply and comprehensively understand the semantic information of the environment in order to perform tasks such as complex behavioral decision-making and human-computer interaction. In this paper, we propose a semantic mapping system for indoor environments based on a monocular camera and a laser. The semantic mapping system adopts the technique of scene classification to construct the scene semantics of indoor environments, in which the semantic classifier is embedded into a recurrent neural network to better learn the correlation of consecutive frames. Experimental results indicate that the proposed semantic mapping system exhibits great performance in the robustness and accuracy of semantic mapping.

Localization technology is essential for robots. The map created to recognize the location mainly contains metric information. However, in a changing environment, a Semantic Map containing Semantic object information is required a multi-modal sensor composed of multiple types and multiple sensors[RGBD, thermal, night vision, global shutter camera, microphone, 16 channel laser sensor(=Lidar)] was created for semantic information recognition and semantic map creation in various environments, and calibration was performed to integrate the coordinate system. After that, we introduce the method of generating the metric map according to the configuration of the multi-modal sensor. Also, we propose a method to obtain a single accurate location by integrating the location recognition results obtained from various maps. This can be used to specify the position of the semantic object. Finally, it can be expected that the semantic object and the semantic map information obtained through the multi-modal sensor can be used for various different sensor configurations and various types of robots.

This paper presents a novel approach for mobile robot localization using monocular vision. The proposed approach locates a robot relative to the target to which the robot moves. Two points are selected from the target as two featurepoints. Once the coordinates in an image of the two feature points are detected, the position and motion direction of the robot can be determined according to the detected coordinates. Unlike those reported geometry pose estimation orlandmarks matching methods, this approach requires neither artificial landmarks nor an accurate map of indoor environment. It needs less computation and can simplify greatly the localization problem. The validity and flexibility of the proposed approach is demonstrated by experiments performed on real images. The results show that this new approach is not only simple and flexible but also has high localization precision.Keywords: mobile robot, localization, monocular vision, single image, feature points 1. INTRODUCTION

Commonly used robot map models include occupancy grid maps, topological maps, and semantic maps. Among these, an occupancy grid map is mainly represented as a quadrilateral grid. This paper proposes an occupancy information grid for intelligent robots by exploiting the advantages of the occupancy grid map and spatial information grid. In terms of geometric structure, a regular hexagonal grid is used instead of a regular quadrilateral grid. In terms of attribute structure, the single obstacle attribute is replaced by the grid terrain characteristics, grid element attributes, and grid edge attributes. Thus, the occupancy information grid model is transformed into a new data structure describing the spatial environment, and it can be effectively applied to map construction and path planning of intelligent robots. For the map construction application of intelligent robots, this paper describes the basic process of laser sensor-based grid model construction. For the path planning application of intelligent robots, this paper extends the A* algorithm based on a regular hexagonal grid. Additionally, map construction and path planning applications for intelligent robots are experimentally verified. Several experimental results were obtained. First, the experimental results confirmed the theoretical conclusion that the minimum sampling density of the hexagonal structure was 13.4% lower than that of the quadrilateral structure. Second, the regular hexagonal grid is clearly more advantageous in describing environmental scenes, which can ameliorate the "undercompleteness" phenomenon. Third, there were large differences in the planning paths based on two types of grids, as shown by the fact that the distance of the planning paths obtained by the regular hexagonal grid was reduced by at least 10.8% and at most 15.6% compared with the regular quadrilateral grid.

Semantic map is the foundation for mobile robots to understand its environment. By considering the semantic mapping problem as a semi-Markov process, a new hierarchical semi-Markov random field is proposed for this task. The proposed model can use multiple contextual information to label the places and objects in map and can partition the observations into spacial and semantic consistent sub-sequences each of which corresponding to a place. The proposed model is called coupled hidden semi-Markov conditional random fields (CHSM-CRFs). According to the structure of CHSM-CRFs, a piecewise learning algorithm and an approximating online inference algorithm based on Monte Carlo sampling are proposed for it. Experimental results with a mobile robot prove that the proposed method has high precision for labeling the places and objects in sematic mapping.

In a large-scale indoor environment, a mobile robot needs a proper internal representation of the surrounding environment to carry out its tasks. The metric (grid-based) map and topological map are two common internal representations in robotic realm. In order to take advantage of the two kinds of environmental representations, this paper aims to construct a topological map of an indoor environment from its grid map, which is usually acquired by a SLAM method. The cells on the grid map have three types of states: free, unknown and occupied. After grid map is obtained, the improved NJW method will be applied to cluster the free cells on the grid map. Instead of using the traditionally Euclidean distance, we propose to use the wavefront distance transform algorithm to calculate the distance between each two free cells on the grid map. Based on the segmenting results, we make some optimizations on the boundaries between each two sub-regions. Finally, taking the specific application backgrounds into consideration, we make some discussion on the value of the scale parameter σ and the selection of clusters k. The simulation experiment results show that our improved NJW algorithm has a better performance on topological segmentation of an indoor environment.

This paper presents an automated robotic system for generating semantic maps of inventory in retail environments. In retail settings, semantic maps are labeled maps of stores where each discrete section of shelving is assigned a department label describing the types of products on that shelf. Starting from a metric map of the store, the robot autonomously extracts the shelf boundaries, generates a distance-optimal tour of the store to view every shelf, and follows the tour while avoiding unmapped clutter and moving people. The robot creates a point cloud of the store using the data collected from this tour. We introduce a novel soft-object assignment algorithm to create a virtual map and a dynamic programming algorithm to segment this map. These algorithms use a priori information about the products to boost data from laser and camera sensors in order to recognize and semantically label objects. The primary contribution of this paper is the integration of multiple systems for automated path planning, navigation, object recognition, and semantic mapping. This paper represents an important contribution toward deploying mobile robots in dynamic human environments.

Simultaneous localization and mapping (SLAM) becomes popular in the autonomous robot research community. SLAM requests a mobile robot, placed at an unknown location in an unknown environment, to incrementally build a consistent map of this environment while simultaneously determining its location within this map. The odometer data is helpful but the noisy error accumulates. Assistant with measurements from the other sensors, SLAM improves the accuracy by filtering. The extended Kalman filter is the most used method to feedback the error. In this thesis, a monocular vision based SLAM for a wheeled robot is proposed. A single web camera is the only measurement input to correct the odometer data. Extract features and match them based on the scale invariant feature transform (SIFT). Absence of the feature depth information is the key issue for the monocular vision in SLAM. The inverse depth parametrization method solves the problem by present the position in inverse coordinate with some reasonable assumptions. Convergence of the feature position is more easily achieved by the EKF due to the linearity. Combining the odometer data with the inverse depth method, this system provides an accurate map by tests of the indoor image sequence.