Semantic Road Segmentation using Deep Learning

Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method.

Chapter 12 - Semantic scene segmentation for robotics

Synthetic Aperture Radar (SAR) imagery have been one of the important tools to support earth observations and topographic measurements. It means SAR imagery are essentially rich in structures and some important target categories are difficult to recognize. Optical imagery contains rich and clear spectral information which has a good influence on semantic image segmentation. The success of deep neural networks for semantic segmentation heavily depends on large-scale and well-labeled data sets, which are hard to collect in practice. In this paper, we consider deep transfer learning for semantic segmentation, we propose a deep novel transfer learning method, which transfers a semantically segment model from SAR imagery to SAR and optical fusion imagery. The experimental results show that the method proposed achieves higher mean Intersection over Union (mIoU) with less training time compared with other methods.

Semantic segmentation is a fundamental task in image processing and computer vision domains. Therefore, it is a core component of most emerging industry applications such as medical imaging, autonomous driving, and agriculture. The last five years have witnessed a huge growth in the computer vision domain with the introduction of Generative Adversarial Networks (GANs) due to their ability to learn deep representations by a competitive process between two neural networks called the Generator the Discriminator. Neural Architecture Search (NAS) is undergoing a revolution aimed at automating the architectural engineering process. This paper discusses implementing a novel framework for semantic image segmentation called iSegmentor using NAS and GANs. The proposed approach is the first framework that utilizes NAS in GANs for semantic image segmentation to the best of our knowledge. The iSegmentor framework was evaluated using mean Intersection over Union (mIoU) and Mean Pixel Accuracy (MPA) metrics. The iSegmentor was mainly tested and evaluated for the autonomous driving domain. With the analysis of the past five-year literature, Cityscapes and CamVid datasets were chosen. The iSegmentor achieved 81.73 MPA and 71.90 mIoU for the Cityscapes dataset, whereas CamVid achieved 80.41 MPA and 70.63 mIoU. Further, we present future avenues that we identified with the conducted experiments to assist other researchers to extend the research area of NAS in GANs in semantic segmentation.

One promising approach for pixel-wise semantic segmentation is based on higher-order Conditional Random Fields (CKFs). We aim to selectively choose segments for the higher-order CRFs in semantic segmentation. To this end, we formulate the selection as an optimization problem. We propose three optimization criteria in relation to the selected segments, namely: a) averaged goodness, b) coverage area and c) non-overlapped area. Essentially, we desire to have best segments with maximum coverage area and maximum non-overlapped area. We apply two evolutionary optimization algorithms, namely: the genetic algorithm (GA) and the particle swarm optimization (PSO). The goodness of segments is estimated using the Latent Dirichlet Allocation approach. Experiment results show that semantic segmentation with GA-or-PSO-selected segments yields competitive semantic segmentation accuracy in comparison to that of naively using all segments. Moreover, the fewer number of segments used in semantic segmentation speeds up its computation time up to six times faster.

With the emergence of deep learning, computer vision has witnessed extensive advancement and has seen immense applications in multiple domains. Specifically, image captioning has become an attractive focal direction for most machine learning experts, which includes the prerequisite of object identification, location, and semantic understanding. In this paper, semantic segmentation and image captioning are comprehensively investigated based on traditional and state‐of‐the‐art methodologies. In this survey, we deliberate on the use of deep learning techniques on the segmentation analysis of both 2D and 3D images using a fully convolutional network and other high‐level hierarchical feature extraction methods. First, each domain’s preliminaries and concept are described, and then semantic segmentation is discussed alongside its relevant features, available datasets, and evaluation criteria. Also, the semantic information capturing of objects and their attributes is presented in relation to their annotation generation. Finally, analysis of the existing methods, their contributions, and relevance are highlighted, informing the importance of these methods and illuminating a possible research continuation for the application of semantic image segmentation and image captioning approaches.

Semantic image segmentation based on deep learning is gaining popularity because it is giving promising results in medical image analysis, automated land categorization, remote sensing, and other computer vision applications. Many algorithms have been designed in recent years, yet there is scope for further improvement in computer vision research. We have proposed a unique ensemble method called Ranking and Nonhierarchical Comparison Ensemble (RNCE) for semantic segmentation of landcover images based on the Ranking and Nonhierarchical Comparison methodology. Our approach has been tested on pretrained models showing improved accuracy and mean IoU with respect to the existing method. The code is available at: https://github.com/vekash2021/RNCE.git .

Recent advances in convolutional neural networks have led to considerably accurate semantic segmentation tasks. However, previous works use a pre-trained backbone borrowed from image classification tasks with considerable computational complexity, resulting in large latencies. Many methods have been proposed to reduce the latency of the segmentation network without loss of accuracy. These methods have reported varying experimental results with different computing devices, acceleration techniques, and input image sizes. Although most studies claim that their results are state-of-the-art, it is necessary to reconsider whether they are being compared under the same conditions. We propose a performance evaluation method of real-time semantic segmentation models to compare the performance under the same conditions fairly. In addition, we carry out an empirical study to evaluate the performance of recent real-time semantic segmentation networks and make a comparative analysis between them. We train the segmentation models using the same input data and data augmentation method. Then, the performance of the segmentation methods is analyzed regarding accuracy and speed. In contrast to most studies that exclude the time required for the pre-processing and post-processing steps, we measured the actual processing time needed to perform semantic segmentation with a real dataset. Further, we measured the processing speed and power consumption of the segmentation models in embedded devices in which real-time segmentation is applied, unlike previous studies that measured performance on a PC. Experimental results showed that the real-time semantic segmentation methods could not run in real-time on embedded devices when considering the pre-processing and post-processing steps. By comprehensively considering the inference speed, energy consumption, and processing time of semantic segmentation models, the experimental results show that FasterSeg-S is suited for embedded devices.

Semantic segmentation partitions a given image or 3D model of a scene into semantically meaning parts and assigns predetermined labels to the parts. With well‐established datasets, deep networks have been successfully used for semantic segmentation of RGB and RGB‐D images. On the other hand, due to the lack of annotated large‐scale 3D datasets, semantic segmentation for 3D scenes has not yet been much addressed with deep learning. In this paper, we present a novel framework for generating semantically segmented triangular meshes of reconstructed 3D indoor scenes using volumetric semantic fusion in the reconstruction process. Our method integrates the results of CNN‐based 2D semantic segmentation that is applied to the RGB‐D stream used for dense surface reconstruction. To reduce the artifacts from noise and uncertainty of single‐view semantic segmentation, we introduce adaptive integration for the volumetric semantic fusion and CRF‐based semantic label regularization. With these methods, our framework can easily generate a high‐quality triangular mesh of the reconstructed 3D scene with dense (i.e., per‐vertex) semantic labels. Extensive experiments demonstrate that our semantic segmentation results of 3D scenes achieves the state‐of‐the‐art performance compared to the previous voxel‐based and point cloud‐based methods.

Methods and datasets on semantic segmentation for Unmanned Aerial Vehicle remote sensing images: A review

Semantic segmentation has always been a very challenging research topic in computer vision and deep learning and has extensive applications in real-life scenarios. With the development of computing hardware and deep learning technology, researchers have a higher research enthusiasm for semantic segmentation. This work briefly introduces several semantic segmentation models, datasets, and the main issues of traditional semantic segmentation. Afterward, the current mainstream semantic segmentation algorithm and the experimental results were compared. Finally, future research directions of semantic segmentation are presented and discussed.

The SAR image semantic segmentation using computer vision techniques has gained much popularity in the research community due to their wide applications. Despite the advancements in Deep Learning for image analysis, these models still struggle to segment SAR images due to the existence of speckle noise and a poor feature extractor. Moreover, deep learning models are challenging to train on small datasets and the performance of the model is significantly impacted by the quality of the data. This calls for the development of an effective network that can draw out critical information from the low resolution SAR images. In this regard, the present work proposes a unique Self attention module in U-Net for the semantic segmentation of low resolution SAR images.. The Self Attention Model makes use of Laplacian kernel to highlight the sharp discontinuities in the features that define the boundaries of the objects. The proposed model, employs dilated convolution layers at the initial layers, enabling the model to more effectively capture larger contextual information. With an accuracy of 0.84 and an F1-score of 0.83, the proposed model outperforms the state-of-the-art techniques in semantic segmentation of low resolution SAR images. The results clearly demonstrate the importance of the self attention module and the consideration of dilated convolution layers in the initial layers in semantic segmentation of low resolution SAR images.

Robust perception systems allow farm robots to recognize weeds and vegetation, enabling the selective application of fertilizers and herbicides to mitigate the environmental impact of traditional agricultural practices. Today's perception systems typically rely on deep learning to interpret sensor data for tasks such as distinguishing soil, crops, and weeds. These approaches usually require substantial amounts of manually labeled training data, which is often time-consuming and requires domain expertise. This paper aims to reduce this limitation and propose an automated labeling pipeline for crop-weed semantic image segmentation in managed agricultural fields. It allows the training of deep learning models without or with only limited manual labeling of images. Our system uses RGB images recorded with unmanned aerial or ground robots operating in the field to produce semantic labels exploiting the field row structure for spatially consistent labeling. We use the rows previously detected to identify multiple crop rows, reducing labeling errors and improving consistency. We further reduce labeling errors by assigning an "unknown" class to challenging-to-segment vegetation. We use evidential deep learning because it provides predictions uncertainty estimates that we use to refine and improve our predictions. In this way, the evidential deep learning assigns high uncertainty to the weed class, as it is often less represented in the training data, allowing us to use the uncertainty to correct the semantic predictions. Experimental results suggest that our approach outperforms general-purpose labeling methods applied to crop fields by a large margin and domain-specific approaches on multiple fields and crop species. Using our generated labels to train deep learning models boosts our prediction performance on previously unseen fields with respect to unseen crop species, growth stages, or different lighting conditions. We obtain an IoU of 88.6% on crops, and 22.7% on weeds for a managed field of sugarbeets, where fully supervised methods have 83.4% on crops and 33.5% on weeds and other unsupervised domain-specific methods get 54.6% on crops and 11.2% on weeds. Finally, our method allows fine-tuning models trained in a fully supervised fashion to improve their performance in unseen field conditions up to +17.6% in mean IoU without additional manual labeling.

Semantic segmentation plays an important part in computer vision by assigning semantic labels to each pixel in an image. Many models have been created to enhance segmentation performance as deep learning develops, from traditional models CNNs, FCNs, and U-Net, to the advanced models DeepLab series, GANs, and Transformer-based models. This paper offers a comprehensive overview of semantic segmentation methods based on deep learning. It begins with an introduction to the task and related background concepts, followed by a review of traditional and advanced models. A summary of current issues, including labeled data scarcity, complex object boundary definition, poor model generalization, and multi-scale information handling, is also provided to assist in future study and real-world application. The study also provides a comparative evaluation of the models' computational and performance characteristics. The overall goal of this work is to give a clear overview of the progress made in semantic segmentation using deep learning.

Deep learning has achieved promising performance on semantic segmentation, but few works focus on semantic segmentation at the fine-grained level. Fine-grained semantic segmentation requires recognizing and distinguishing hundreds of sub-categories. Due to the high similarity of different sub-categories and large variations in poses, scales, rotations, and color of the same sub-category in the fine-grained image set, the performance of traditional semantic segmentation methods will decline sharply. To alleviate these dilemmas, a new approach, named Class Guided Channel Weighting Network (CGCWNet), is developed in this paper to enable fine-grained semantic segmentation. For the large intra-class variations, we propose a Class Guided Weighting (CGW) module, which learns the image-level fine-grained category probabilities by exploiting second-order feature statistics, and use them as global information to guide semantic segmentation. For the high similarity between different sub-categories, we specially build a Channel Relationship Attention (CRA) module to amplify the distinction of features. Furthermore, a Detail Enhanced Guided Filter (DEGF) module is proposed to refine the boundaries of object masks by using an edge contour cue extracted from the enhanced original image. Experimental results on PASCAL VOC 2012 and six fine-grained image sets show that our proposed CGCWNet has achieved state-of-the-art results.