USIS: A unified semantic image synthesis model trained on a single or multiple samples

The robot simultaneous localization and mapping (SLAM) is a very important and useful technology in the robotic field. However, the environmental map constructed by the traditional visual SLAM method contains little semantic information, which cannot satisfy the needs of complex applications. The semantic map can deal with this problem efficiently, which has become a research hot spot. This paper proposed an improved deep residual network- (ResNet-) based semantic SLAM method for monocular vision robots. In the proposed approach, an improved image matching algorithm based on feature points is presented, to enhance the anti-interference ability of the algorithm. Then, the robust feature point extraction method is adopted in the front-end module of the SLAM system, which can effectively reduce the probability of camera tracking loss. In addition, the improved key frame insertion method is introduced in the visual SLAM system to enhance the stability of the system during the turning and moving of the robot. Furthermore, an improved ResNet model is proposed to extract the semantic information of the environment to complete the construction of the semantic map of the environment. Finally, various experiments are conducted and the results show that the proposed method is effective.

Semantic mapping is one of the most active and promising research areas within autonomous mobile robotics. Informally, a semantic map associates a high-level human-understandable label (like “office” or “corridor”) to a portion of an environment. Most semantic mapping approaches are based on classifiers that, given some features perceived by robot sensors in a physical place, associate a semantic label to the place. These approaches are often tested on a limited number of homogeneous places (e.g., few rooms within a single building). This line of action seems to hinder the development of methods for constructing semantic maps that can be (re)used in a number of previously unseen environments. In this paper, we aim at contributing to make semantic mapping methods more general. In particular, we focus on indoor environments and we consider the following research question: to what extent are the semantic mapping approaches shown to label rooms in a single building expected to work when applied to different buildings?

Semantic mapping based on the supervised object detectors is sensitive to image distribution. In real-world environments, the object detection and segmentation performance can lead to a major drop, preventing the use of semantic mapping in a wider domain. On the other hand, the development of vision-language foundation models demonstrates a strong zero-shot transferability across data distribution. It provides an opportunity to construct generalizable instance-aware semantic maps. Hence, this work explores how to boost instance-aware semantic mapping from object detection generated from foundation models. We propose a probabilistic label fusion method to predict close-set semantic classes from open-set label measurements. An instance refinement module merges the over-segmented instances caused by inconsistent segmentation. We integrate all the modules into a unified semantic mapping system. Reading a sequence of RGB-D input, our work incrementally reconstructs an instance-aware semantic map. We evaluate the zero-shot performance of our method in ScanNet and SceneNN datasets. Our method achieves 40.3 mean average precision (mAP) on the ScanNet semantic instance segmentation task. It outperforms the traditional semantic mapping method significantly.

This article introduces an adapted semantic map method that was created to account for the semantics of derivational affixation. As the derivatives of a single derivational affix often express a wide range of readings, and different affixes sometimes give rise to very similar, or even identical, readings, a method that is tailored to the needs of derivational affixation is believed to be necessary. The different readings expressed by the derivatives of a single affix can be represented and related to one another on a semantic map—the maps thus visualise the substantial semantic complexity of derivational affixation. They can, however, also be used to compare the semantic structure of derivatives of different affixes. The data for this study come from the British National Corpus (BNC). Derivatives of the English derivational suffixes -age and -ery are analysed and semantic maps are created for each of these morphological categories. A comparison of these maps shows that both affixes are highly polysemous, and that they express a similar range of readings. In spite of this similarity, the semantic structure of the categories is not identical and they should not be considered synonymous. The semantic maps thus prove to be a useful tool for representing the semantic structure of individual morphological categories and also for comparing the semantics of different categories.

In this paper, we present an image manipulation system that enables a user to change the scene of an input image by simple semantic conversion of label, which can “transform” reality. To achieve this, we employed semantic segmentation method and trained a semantic image synthesis model on our landscape dataset of 18 classes. Qualitative results demonstrate the effectiveness of our proposed method.

Perception of the indoor environment is the basis of mobile robot localization, navigation, and path planning, and it is particularly important to construct semantic maps in real time using minimal resources. The existing methods are too dependent on the graphics processing unit (GPU) for acquiring semantic information about the indoor environment, and cannot build the semantic map in real time on the central processing unit (CPU). To address the above problems, this paper proposes a non-use GPU for lightweight indoor semantic map construction algorithm, named NGLSFusion. In the VO method, ORB features are used for the initialization of the first frame, new keyframes are created by optical flow method, and feature points are extracted by direct method, which speeds up the tracking speed. In the semantic map construction method, a pretrained model of the lightweight network LinkNet is optimized to provide semantic information in real time on devices with limited computing power, and a semantic point cloud is fused using OctoMap and Voxblox. Experimental results show that the algorithm in this paper ensures the accuracy of camera pose while speeding up the tracking speed, and obtains a reconstructed semantic map with complete structure without using GPU.

This study aimed to examine the effectiveness of the semantic concept map method in teaching the family concepts of life studies lessons to students with Autism Spectrum Disorder. A multiple probe model with probe phase between subjects, one of the single-subject research methods, was used. The subjects of the study are 10-year-old three boys diagnosed with ASD. Inter-observer reliability and implementation reliability data were also collected in the study. Graphical analysis of the obtained data was made. As a result of the research, it was revealed that the semantic concept map method was effective in teaching the concepts within the theme of our family to students with Autism Spectrum Disorder.

Large-scale semantic mapping is crucial for outdoor autonomous agents to perform high-level tasks such as planning and navigation. In this paper, we propose a novel method for large-scale 3D semantic reconstruction through implicit representations from posed LiDAR measurements alone. We first leverage an octree-based hierarchical structure to store implicit features, then these implicit features are decoded to signed distance value and semantic information through shallow Multilayer Perceptrons (MLPs). We leverage radial window self-attention networks to predict the semantic labels of point clouds. We then jointly optimize the feature embedding and MLP parameters with a self-supervision paradigm for point-cloud geometry and a pseudo-supervision paradigm for semantic and panoptic labels. Subsequently, geometric structures and object categories for novel points in the unseen area are regressed, and the marching cubes method is exploited to subdivide and visualize scenes in the inferring stage. Experiments on two real-world datasets, SemanticKITTI and SemanticPOSS, demonstrate the superior segmentation efficiency and mapping effectiveness of our framework compared to current state-of-the-art 3D semantic LiDAR mapping methods.

In this paper, we focus on the task of generating realistic images given an input semantic layout, which is also called semantic image synthesis. Most of previous methods are based on conditional generative adversarial networks mechanism, which is stacks of convolution, normalization, and non-linearity layers. However, these methods easily generate blurred regions and distorted structures. There are two limits existing: their normalization layers are unable to make a good balance between keeping semantic layout information and geometric changes; and cannot effectively aggregated multi-level feature. To address the above problems, we propose a novel method which incorporates multi-level gate feature aggregation mechanism (GFA) and spatially adaptive batch-instance normalization (SPAda-BIN) for semantic image synthesis. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, in terms of both visual fidelity and quantitative metrics.

To support content based image retrieval, MPEG-7 is developed to define the content interfaces for images. In MPEG-7, Dominant Color Descriptor (DCD) is considered as the most important feature, and is widely used to describe the color features of an image. To support semantic queries from users, we proposed a color feature semantic mapping method in this work, which can translate the DCD values into semantic color names. The semantic mapping method is realized by constructing a mapping table between the DCD values and the semantic color names. To validate the effectiveness of our mapping method, an image retrieval experiment is conducted. From the comparison with the manually indexed description, the proposed mapping method is proved to be effective by the experiment results. Our work is very important to automatically generate the semantic description of an image and then support the users’ semantic retrieval queries.

The language modeling approach centers on the issue of estimating an accurate model by choosing appropriate language models as well as smoothing techniques. In the thesis, we propose a novel context-sensitive semantic smoothing method referred to as a topic signature language model. It extracts explicit topic signatures from a document and then statistically maps them into individual words in the vocabulary. In order to support the new language model, we developed two automated algorithms to extract multiword phrases and ontological concepts, respectively, and an EM-based algorithm to learn semantic mapping knowledge from co-occurrence data. The topic signature language model is applied to three applications: information retrieval, text classification, and text clustering. The evaluations on news collection and biomedical literature prove the effectiveness of the topic signature language model. In the experiment of information retrieval, the topic signature language model consistently outperforms the baseline two-stage language model as well as the context-insensitive semantic smoothing method in all configurations. It also beats the state-of-the-art Okapi models in all configurations. In the experiment of text classification, when the size of training documents is small, the Bayesian classifier with semantic smoothing not only outperforms the classifiers with background smoothing and Laplace smoothing, but it also beats the active learning classifiers and SVM classifiers. On the task of clustering, whether or not the dataset to cluster is small, the model-based k-means with semantic smoothing performs significantly better than both the model-based k-means with background smoothing and Laplace smoothing. It is also superior to the spherical k-means in terms of effectiveness. In addition, we empirically prove that, within the framework of topic signature language models, the semantic knowledge learned from one collection could be effectively applied to other collections. In the thesis, we also compare three types of topic signatures (i.e., words, multiword phrases, and ontological concepts), with respect to their effectiveness and efficiency for semantic smoothing. In general, it is more expensive to extract multiword phrases and ontological concepts than individual words, but semantic mapping based on multiword phrases and ontological concepts are more effective in handling data sparsity than on individual words.

To address perception and navigation challenges in precision agriculture caused by GPS signal loss and weakly structured environments in greenhouses, this study proposes an integrated framework for real-time semantic reconstruction and path planning. This framework comprises three core components: First, it introduces a semantic segmentation method tailored for greenhouse environments, enhancing recognition accuracy of key navigable areas such as furrows. Second, it designs a visual-semantic fusion SLAM point cloud reconstruction algorithm and proposes a semantic point cloud rasterization method. Finally, it develops a semantic-constrained A* path planning algorithm adapted for semantic maps. We collected a segmentation dataset (1083 images, 4 classes) and a reconstruction dataset from greenhouses in Shanghai. Experiments demonstrate that the segmentation algorithm achieves 95.44% accuracy and 87.93% mIoU, with a 3.9% improvement in furrow category recognition accuracy. The reconstructed point cloud exhibits an average relative error of 7.37% on furrows. In practical greenhouse validation, single-frame point cloud fusion took approximately 0.35 s, while path planning was completed in under 1 s. Feasible paths avoiding crops were successfully generated across three structurally distinct greenhouses. Results demonstrate that this framework can stably and in real-time accomplish semantic mapping and path planning, providing effective technical support for digital agriculture.

The problem of robust and invariant representation of places is being addressed. A place recognition technique is proposed followed by an application to a semantic topological mapping. The proposed technique is evaluated on a robot localization database which consists of a large set of images taken under various weather conditions. The results show that the proposed method can robustly recognize the places and is invariant to geometric transformations, brightness changes and noise. The comparative analysis with the state-of-the-art semantic place description methods show that the method outperforms the competing methods and exhibits better average recognition rates.

Visual-based Simultaneous Localization and Mapping (vSLAM) is indispensable for intelligent mobile robots and autonomous vehicles. However, the scene rigidity assumption typical in vSLAM systems limits the widespread use of most vSLAM under dynamic environments in the real world. We propose Real-time Visual Semantic Tracking and Mapping (RTS-vSLAM), implemented based on RGB-D-based ORB-SLAM2, which uses both semantic information and geometric methods to detect moving objects and outlier features. RTS-vSLAM adopts a novel generic system architecture that enables the use of many kinds of semantic segmentation methods on the remote server and can be employed on embedded system devices, head-mounted devices, and low-performance robots, where GPUs are not mounted on, via requesting service from the server. It also generates many kinds of maps to support navigation and more complex tasks in real-time. The proposed system is evaluated using the TUM RGB-D dataset and compared tracking errors with the state-of-the-art vSLAM methods under dynamic environments. RTS-vSLAM outperforms the others in tracking accuracy in a highly dynamic scenario and reduces the time delays greatly.KeywordsSLAMDynamic SLAMTrackingMappingRTS-vSLAM

With the development of enterprise intelligence and automation, the inspection of power distribution room is not limited to manual work. This paper proposes a semantic segmentation method based on improved ORB-SLAM3 and YOLOV5 fusion to eliminate dynamic features. ORB-SLAM3 is used to realize scene reconstruction to generate dense point cloud maps. In this paper, all the differences between ORB-SLAM2 and ORBSLAM3 are compared, and the dense point cloud generation and loop detection modules are added to make the mapping more accurate. In real scenes, the depth sensor is inevitably disturbed by dynamic objects when obtaining real environmental information. Compared with dynamic SLAM for object detection, the semantic segmentation module of YOLOV5 is used in this paper, which can remove dynamic objects while minimizing the loss of effective features. In addition, TensorRT is used to accelerate the construction of the map, and the semantic map is generated, which makes a prerequisite for the subsequent conversion of the map into a two-dimensional occupancy map. Experimental results show that compared with the original ORB-SLAM2/3, the improved ORBSLAM3 greatly improves the accuracy in dynamic scenes. A more real-time and accurate scene reconstruction function of power distribution room is realized.