CaDIS: Cataract dataset for surgical RGB-image segmentation.

Chapter 12 - Semantic scene segmentation for robotics

Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method.

Methods and datasets on semantic segmentation for Unmanned Aerial Vehicle remote sensing images: A review

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

Abstract. Semantic instance segmentation from scenes, serving as a crucial role for 3D modelling and scene understanding. Conducting semantic segmentation before grouping instances is adopted by the existing state-of-the-art methods. However, without additional refinement, semantic errors will fully propagate into the grouping stage, resulting in low overlap with the ground truth instance. Furthermore, the proposed methods focused on indoor level scenes, which are limited when directly applied to large-scale outdoor Airborne Laser Scanning (ALS) point clouds. Numerous instances, significant object density and scale variations make ALS point clouds distinct from indoor data. In order to address the problems, we proposed a geometric characterization-aware semantic instance segmentation network, which utilized both semantic and objectness score to select potential points for grouping. And in point cloud feature learning stage, hand-craft geometry features are taken as input for geometric characterization awareness. Moreover, to address errors propagated from previous modules after grouping, we have additionally designed a per-instance refinement module. To assess semantic instance segmentation, we conducted experiments on an open-source dataset. Additionally, we performed semantic segmentation experiments to evaluate the performance of our proposed point cloud feature learning method.

There has been remarkable progress in the field of Semantic segmentation in recent years. Yet, it remains a challenging problem to apply segmentation to the video-based applications. Videos usually involve significantly larger volume of data compared to images. Particularly, a video contains around 30 frames per second. Segmentation of the similar frames unnecessarily adds to the time required for segmentation of complete video. In this paper, we propose a contour detection-based approach for detection of salient frames for faster semantic segmentation of videos. We propose to detect the salient frames of the video and pass only the salient frames through the segmentation block. Then, the segmented labels of the salient frames are mapped to the non-salient frames. The salient frame is defined by the variation in the pixel values of the background subtracted frames. The background subtraction is done using MOG2 background subtractor algorithm for background subtraction in various lighting conditions. We demonstrate the results using the Pytorch model for semantic segmentation of images. We propose to concatenate the semantic segmentation model to our proposed framework. We evaluate our result by comparing the time taken and the mean Intersection over Union (mIoU) for segmentation of the video with and without passing the video input through our proposed framework. We evaluate the results of Saliency Detection Block using Retention and Condensation ratio as the quality metrics.

A training dataset for semantic segmentation of urban point cloud map for intelligent vehicles

Uncrewed aerial vehicles (UAVs) aerial remote sensing (RS) images offer higher spatial resolution compared with satellite RS images, providing more detailed scene information. The flexibility of UAV-mounted sensors allows for the simultaneous acquisition of multimodal data, including RGB images and depth maps. Depth information, serving as a supplementary cue, has been extensively utilized in semantic segmentation research. Nevertheless, the current investigations on red, green, blue-depth (RGB-D) image semantic segmentation predominantly concentrate on indoor or road scenarios. In UAV RS scenes, there exist numerous objects featuring similar visual texture manifestations. The introduction of depth geometric information can efficaciously enhance the semantic segmentation accuracy of complex ground objects. Concurrently, high-resolution RS images also present challenges to the modeling of global information in large-scale space. To address these challenges, we propose the first framework for semantic segmentation of RGB-D images in UAV RS, namely, SkyMamba. This framework effectively integrates visual texture information and geometric depth information. Simultaneously, it leverages the long-sequence modeling capability of the state-space model to efficiently capture global context information in the RS space, achieving efficient semantic segmentation of RS scenes. Furthermore, we introduce a novel benchmark dataset for semantic segmentation of UAV RS RGB-D images, named comprehensive urban-scale depth, which is divided into two versions based on coarse-grained and fine-grained semantic annotation strategies to provide diverse challenges. Experimental results demonstrate that the proposed method exhibits state-of-the-art performance on both versions of the dataset, particularly in handling imbalanced scene semantic class distributions. The code for this article is available at the following URL: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CapyLin-G/CUS-Depth</uri>

Semantic segmentation is an important task in self-driving cars. The aims of semantic segmentation are to recognize pre-defined objects and its pixel-by-pixel location. The most popular method in semantic segmentation is Deep learning which has considerably improved semantic image segmentation. This work does an overview for semantic segmentation using Deep learning. This works also implement comparisons in term of precision, mean IOU and processing time. Three popular algorithms are PSPNet, FCN and SegNet that are examined carefully. In detail, the aim of this work points out a trade-off between processing time and mean IOU, and also between processing time and precision. Moreover, this paper concentrates on road segmentation for embedded devices, so processing time is significantly important. This work also figures out which method is suitable for embedded devices on road segmentation.

We propose a practical Convolution Neural Network (CNN) model termed the CNN for Semantic Segmentation for driver Assistance system (CSSA). It is a novel semantic segmentation model for probabilistic pixel-wise segmentation, which is able to predict pixel-wise class labels of a given input image. Recently, scene understanding has turned out to be one of the emerging areas of research, and pixel-wise semantic segmentation is a key tool for visual scene understanding. Among future intelligent systems, the Advanced Driver Assistance System (ADAS) is one of the most favorite research topic. The CSSA is a road scene understanding CNN that could be a useful constituent of the ADAS toolkit. The proposed CNN network is an encoder-decoder model, which is built on convolutional encoder layers adopted from the Visual Geometry Group’s VGG-16 net, whereas the decoder is inspired by segmentation network (SegNet). The proposed architecture mitigates the limitations of the existing methods based on state-of-the-art encoder-decoder design. The encoder performs convolution, while the decoder is responsible for deconvolution and un-pooling/up-sampling to predict pixel-wise class labels. The key idea is to apply the up-sampling decoder network, which maps the low-resolution encoder feature maps. This architecture substantially reduces the number of trainable parameters and reuses the encoder’s pooling indices to up-sample to map pixel-wise classification and segmentation. We have experimented with different activation functions, pooling methods, dropout units and architectures to design an efficient CNN architecture. The proposed network offers a significant improvement in performance in segmentation results while reducing the number of trainable parameters. Moreover, there is a considerable improvement in performance in comparison to the benchmark results over PASCAL VOC-12 and the CamVid.

Semantic image segmentation is a core task for autonomous driving, which is performed by deep models. Since training these models draws to a curse of human-based image labeling, the use of synthetic images with automatically generated labels together with unlabeled real-world images is a promising alternative. This implies addressing an unsupervised domain adaptation (UDA) problem. In this paper, we propose a new co-training procedure for synth-to-real UDA of semantic segmentation models. It performs iterations where the (unlabeled) real-world training images are labeled by intermediate deep models trained with both the (labeled) synthetic images and the real-world ones labeled in previous iterations. More specifically, a self-training stage provides two domain-adapted models and a model collaboration loop allows the mutual improvement of these two models. The final semantic segmentation labels (pseudo-labels) for the real-world images are provided by these two models. The overall procedure treats the deep models as black boxes and drives their collaboration at the level of pseudo-labeled target images, i.e., neither modifying loss functions is required, nor explicit feature alignment. We test our proposal on standard synthetic and real-world datasets for onboard semantic segmentation. Our procedure shows improvements ranging from approximately 13 to 31 mIoU points over baselines.

Semantic segmentation has always been a very challenging research topic in computer vision and deep learning and has extensive applications in real-life scenarios. With the development of computing hardware and deep learning technology, researchers have a higher research enthusiasm for semantic segmentation. This work briefly introduces several semantic segmentation models, datasets, and the main issues of traditional semantic segmentation. Afterward, the current mainstream semantic segmentation algorithm and the experimental results were compared. Finally, future research directions of semantic segmentation are presented and discussed.

Autonomous robots are a hot research subject within the fields of science and technology, which has a big impact on social-economic development. The ability of the autonomous robot to perceive and understand its working environment is the basis for solving more complicated issues. In recent years, an increasing number of artificial intelligence-based methods have been proposed in the field of scene understanding for autonomous robots, and deep learning is one of the current key areas in this field. Outstanding gains have been attained in the field of scene understanding for autonomous robots based on deep learning. Thus, this paper presents a review of recent research on the deep learning-based scene understanding for autonomous robots. This survey provides a detailed overview of the evolution of robotic scene understanding and summarizes the applications of deep learning methods in scene understanding for autonomous robots. In addition, the key issues in autonomous robot scene understanding are analyzed, such as pose estimation, saliency prediction, semantic segmentation, and object detection. Then, some representative deep learning-based solutions for these issues are summarized. Finally, future challenges in the field of the scene understanding for autonomous robots are discussed.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.

Automated data-driven method for creating digital building models from dense point clouds and images through semantic segmentation and parametric model fitting