Deep Learning-Based Semantic Segmentation in Autonomous Driving

Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method.

Abstract. In this study, deep learning-based semantic segmentation is used to automatically generate CORINE land cover (CLC) Level 2 classes for a test region in Türkiye. This is accomplished by utilizing new datasets and models created from a pilot region in Italy, which exhibits similar land use/land cover (LU/LC) characteristics to the test region in Canakkale/Türkiye. The training and validation datasets for Italy were generated by employing Sentinel-2 images from various months and different band combinations, along with CLC 2018 vector data for labelling. Different datasets were created to investigate the impact of patch sizes (128 and 256 pixels) and seasonal changes in LU/LC. For the semantic segmentation task, the U-Net architecture was selected as the primary deep learning model. Furthermore, the U-Net architecture was used in conjunction with ResNet50 and ResNet101 for transfer learning, enabling the replacement of the encoder section of the U-Net. These models were tested in the Italy region, and the best-performing ones were subsequently applied to the Canakkale test region to automatically generate CLC 2018. The results were compared with published CLC 2018 Level 2 data for the same region, and the accuracy was assessed using the Intersection over Union (IoU) metric. The findings were presented both visually and statistically.

The surface water in the lakeshore zone is the primary area where cyanobacteria bloom floats intensively. In lake water environment monitoring, it has become pressing to accurately identify the distribution and accumulation coverage area of cyanobacteria blooms in the surface water of the lakeshore zone. This study proposes a real-time and dynamic monitoring technology for cyanobacteria blooms in surface water using a shore-based camera monitoring network. The specific work is as follows: Chaohu Lake, a large eutrophic lake in China, is selected as the research object. The multithreading technology is used to dynamically obtain the hourly video images of 43 cameras around Chaohu Lake. The semantic segmentation method is used to identify the cyanobacteria blooms in the video images, calculate the coverage of cyanobacteria blooms, and draw the spatial distribution map of cyanobacteria blooms in the lakeshore zone of Chaohu Lake. To improve the accuracy of cyanobacteria blooms recognition, we use the ResNet-50 network to integrate three semantic segmentation models, namely FCN, U-net, and DeeplabV3+. By comparing the cyanobacteria blooms results identified by the three methods, it is found that the boundary of the cyanobacteria blooms results identified by DeeplabV3+(ResNet-50) is clear, which is more consistent with the real spatial information of the distribution of cyanobacteria blooms and is more suitable for monitoring the hourly dynamic changes of cyanobacteria blooms in the Chaohu Lake lakeshore zone. The results demonstrated that the time requirement of monitoring cyanobacteria blooms in real time on an hourly basis could be met by utilizing technology that uses multiple threads. The OA (Overall Accuracy), MPA (Mean Pixel Accuracy), IOU (Intersection Over Union) of cyanobacteria blooms, and the IOU of water values of the DeeplabV3+(ResNet-50) were the highest, which were 0.83, 0.82, 0.71, and 0.74, and the RMSE between the predicted and real cyanobacterial blooms coverage of 43 cameras was 6.65%. The above values show that DeeplabV3+(ResNet-50) is this technology’s most suitable semantic segmentation model. This technique can provide technical support for the scientific development of a cyanobacteria blooms management plan in the lakeshore zone of Chaohu Lake by calculating the coverage area of cyanobacteria blooms and drawing the spatial distribution map of cyanobacteria blooms in the lakeshore zone.

Identifying agricultural fields that grow cabbage in the highlands of South Korea is critical for accurate crop yield estimation. Only grown for a limited time during the summer, highland cabbage accounts for a significant proportion of South Korea’s annual cabbage production. Thus, it has a profound effect on the formation of cabbage prices. Traditionally, labor-extensive and time-consuming field surveys are manually carried out to derive agricultural field maps of the highlands. Recently, high-resolution overhead images of the highlands have become readily available with the rapid development of unmanned aerial vehicles (UAV) and remote sensing technology. In addition, deep learning-based semantic segmentation models have quickly advanced by recent improvements in algorithms and computational resources. In this study, we propose a semantic segmentation framework based on state-of-the-art deep learning techniques to automate the process of identifying cabbage cultivation fields. We operated UAVs and collected 2010 multispectral images under different spatiotemporal conditions to measure how well semantic segmentation models generalize. Next, we manually labeled these images at a pixel-level to obtain ground truth labels for training. Our results demonstrate that our framework performs well in detecting cabbage fields not only in areas included in the training data but also in unseen areas not included in the training data. Moreover, we analyzed the effects of infrared wavelengths on the performance of identifying cabbage fields. Based on the results of our framework, we expect agricultural officials to reduce time and manpower when identifying information about highlands cabbage fields by replacing field surveys.

The Automated Building Information Modeling (BIM) reconstruction of existing building curtain walls is crucial for promoting digital Operation and Maintenance (O&M). However, existing 3D reconstruction technologies are mainly designed for general architectural scenes, and there is currently a lack of research specifically focused on the BIM reconstruction of curtain walls. This study proposes a BIM reconstruction method from unmanned aerial vehicle (UAV) images based on neural radiance field (NeRF) and deep learning-based semantic segmentation. The proposed method compensates for the lack of semantic information in traditional NeRF methods and could fill the gap in the automatic reconstruction of semantic models for curtain walls. A comprehensive high-rise building is selected as a case study to validate the proposed method. The results show that the overall accuracy (OA) for semantic segmentation of curtain wall point clouds is 71.8%, and the overall dimensional error of the reconstructed BIM model is less than 0.1m, indicating high modeling accuracy. Additionally, this study compares the proposed method with photogrammetry-based reconstruction and traditional semantic segmentation methods to further validate its effectiveness.

At present, most semantic segmentation models rely on the excellent feature extraction capabilities of a deep learning network structure. Although these models can achieve excellent performance on multiple datasets, ways of refining the target main body segmentation and overcoming the performance limitation of deep learning networks are still a research focus. We discovered a pan-class intrinsic relevance phenomenon among targets that can link the targets cross-class. This cross-class strategy is different from the latest semantic segmentation model via context where targets are divided into an intra-class and inter-class. This paper proposes a model for refining the target main body segmentation using multi-target pan-class intrinsic relevance. The main contributions of the proposed model can be summarized as follows: a) The multi-target pan-class intrinsic relevance prior knowledge establishment (RPK-Est) module builds the prior knowledge of the intrinsic relevance to lay the foundation for the following extraction of the pan-class intrinsic relevance feature. b) The multi-target pan-class intrinsic relevance feature extraction (RF-Ext) module is designed to extract the pan-class intrinsic relevance feature based on the proposed multi-target node graph and graph convolution network. c) The multi-target pan-class intrinsic relevance feature integration (RF-Int) module is proposed to integrate the intrinsic relevance features and semantic features by a generative adversarial learning strategy at the gradient level, which can make intrinsic relevance features play a role in semantic segmentation. The proposed model achieved outstanding performance in semantic segmentation testing on four authoritative datasets compared to other state-of-the-art models.

Pine wilt disease (PWD) is the most dangerous biohazard of pine species and poses a serious threat to forest resources. Coupling satellite remote sensing technology and deep learning technology for the accurate monitoring of PWD is an important tool for the efficient prevention and control of PWD. We used Gaofen-2 remote sensing images to construct a dataset of discolored standing tree samples of PWD and selected three semantic segmentation models—DeepLabv3+, HRNet, and DANet—for training and to compare their performance. To build a GAN-based semi-supervised semantic segmentation model for semi-supervised learning training, the best model was chosen as the generator of generative adversarial networks (GANs). The model was then optimized for structural adjustment and hyperparameter adjustment. Aimed at the characteristics of Gaofen-2 images and discolored standing trees with PWD, this paper adopts three strategies—swelling prediction, raster vectorization, and forest floor mask extraction—to optimize the image identification process and results and conducts an application demonstration study in Nanping city, Fujian Province. The results show that among the three semantic segmentation models, HRNet was the optimal conventional semantic segmentation model for identifying discolored standing trees of PWD based on Gaofen-2 images and that its MIoU value was 68.36%. Additionally, the GAN-based semi-supervised semantic segmentation model GAN_HRNet_Semi improved the MIoU value by 3.10%, and its recognition segmentation accuracy was better than the traditional semantic segmentation model. The recall rate of PWD discolored standing tree monitoring in the demonstration area reached 80.09%. The combination of semi-supervised semantic segmentation technology and high-resolution satellite remote sensing technology provides new technical methods for the accurate wide-scale monitoring, prevention, and control of PWD.

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

Detection of artificial spots in fundus images using modified U-Net based semantic segmentation

A deep semantic segmentation model-based method can achieve state-of-the-art accuracy and high computational efficiency in large-scale crop mapping. However, the model cannot be widely used in actual large-scale crop mapping applications, mainly because the annotation of ground truth data for deep semantic segmentation model training is time-consuming. At the operational level, it is extremely difficult to obtain a large amount of ground reference data by photointerpretation for the model training. Consequently, in order to solve this problem, this study introduces a workflow that aims to extract rice distribution information in training sample shortage regions, using a deep semantic segmentation model (i.e., U-Net) trained on pseudo-labels. Based on the time series Sentinel-1 images, Cropland Data Layer (CDL) and U-Net model, the optimal multi-temporal datasets for rice mapping were summarized, using the global search method. Then, based on the optimal multi-temporal datasets, the proposed workflow (a combination of K-Means and random forest) was directly used to extract the rice-distribution information of Jiangsu (i.e., the K–RF pseudo-labels). For comparison, the optimal well-trained U-Net model acquired from Arkansas (i.e., the transfer model) was also transferred to Jiangsu to extract local rice-distribution information (i.e., the TF pseudo-labels). Finally, the pseudo-labels with high confidences generated from the two methods were further used to retrain the U-Net models, which were suitable for rice mapping in Jiangsu. For different rice planting pattern regions of Jiangsu, the final results showed that, compared with the U-Net model trained on the TF pseudo-labels, the rice area extraction errors of pseudo-labels could be further reduced by using the U-Net model trained on the K–RF pseudo-labels. In addition, compared with the existing rule-based rice mapping methods, he U-Net model trained on the K–RF pseudo-labels could robustly extract the spatial distribution information of rice. Generally, this study could provide new options for applying a deep semantic segmentation model to training sample shortage regions.

Recent advances in convolutional neural networks have led to considerably accurate semantic segmentation tasks. However, previous works use a pre-trained backbone borrowed from image classification tasks with considerable computational complexity, resulting in large latencies. Many methods have been proposed to reduce the latency of the segmentation network without loss of accuracy. These methods have reported varying experimental results with different computing devices, acceleration techniques, and input image sizes. Although most studies claim that their results are state-of-the-art, it is necessary to reconsider whether they are being compared under the same conditions. We propose a performance evaluation method of real-time semantic segmentation models to compare the performance under the same conditions fairly. In addition, we carry out an empirical study to evaluate the performance of recent real-time semantic segmentation networks and make a comparative analysis between them. We train the segmentation models using the same input data and data augmentation method. Then, the performance of the segmentation methods is analyzed regarding accuracy and speed. In contrast to most studies that exclude the time required for the pre-processing and post-processing steps, we measured the actual processing time needed to perform semantic segmentation with a real dataset. Further, we measured the processing speed and power consumption of the segmentation models in embedded devices in which real-time segmentation is applied, unlike previous studies that measured performance on a PC. Experimental results showed that the real-time semantic segmentation methods could not run in real-time on embedded devices when considering the pre-processing and post-processing steps. By comprehensively considering the inference speed, energy consumption, and processing time of semantic segmentation models, the experimental results show that FasterSeg-S is suited for embedded devices.

Semantic segmentation has always been a very challenging research topic in computer vision and deep learning and has extensive applications in real-life scenarios. With the development of computing hardware and deep learning technology, researchers have a higher research enthusiasm for semantic segmentation. This work briefly introduces several semantic segmentation models, datasets, and the main issues of traditional semantic segmentation. Afterward, the current mainstream semantic segmentation algorithm and the experimental results were compared. Finally, future research directions of semantic segmentation are presented and discussed.

In the task of image classification of machine tool machining, the coolant water column will block the tool and interfere with the image classification. To solve this problem, firstly, the coolant water column is removed from the image by semantic segmentation. Because the number of samples in the data set is insufficient, the effect of semantic segmentation model is not very good. These results are taken as the input number of image classification model According to, it will directly affect the accuracy of classification, but it is impossible to modify the parameters of semantic segmentation model during classification training. In this case, the image after semantic segmentation processing is combined with the original image according to certain rules, combined with the information of some original images, and then classified. This method can effectively reduce the interference to the classification model when the semantic segmentation model is wrong.

Remote sensing, particularly satellite-based, can play a valuable role in monitoring areas prone to geohazards. The high spatial and temporal coverage provided by satellite data can be used to reconstruct past events and continuously monitor sensitive areas for potential hazards. This paper presents a range of techniques and methods that were applied for in-depth analysis and utilization of Earth observation data, with a particular emphasis on: (1) detecting mining subsidence, where a novel approach is proposed by combining an improved U-Net model and Interferometry Synthetic Aperture Radar (InSAR) technology. The results showed that the Efficient Channel Attention (ECA) U-Net model performed better than the U-Net (baseline) model in terms of Mean Intersection over Union (MIoU) and Intersection over Union (IoU) indicators; (2) monitoring water conservancy and hydropower engineering. The Xiaolangdi multipurpose dam complex was monitored using Small BAsline Subsets (SBAS) InSAR method on Sentinel-1 time series data and four small regions with high deformation rates were identified on the slope of the reservoir bank on the north side. The dam body also showed obvious deformation with a velocity exceeding 60 mm/a; (3) the evaluation of the potential of InSAR results to integrate monitoring and warning systems for valuable heritage and architectural preservation. The overall outcome of these methods showed that the use of Artificial Intelligence (AI) techniques in combination with InSAR data leads to more efficient analysis and interpretation, resulting in improved accuracy and prompt identification of potential hazards; and (4) finally, this study also presents a method for detecting landslides in mountainous regions, using optical imagery. The new temporal landslide detection method is evaluated over a 7-year analysis period and unlike conventional bi-temporal change detection methods, this approach does not depend on any prior-knowledge and can potentially detect landslides over extended periods of time such as decades.

Grasping is an important and fundamental action for the interaction between robots and the environment. However, because grasping is a complex system engineering, there is still much room for development. At present, many studies use regression to solve the problem of grasp detection or use the unstable 3D point cloud as input, which may cause poor results to a certain extent. In this paper, we propose to use semantic segmentation of pixel-level classification to solve the problem of grasp pose detection. We adopt a grasp detection method based on the DeepLabv3+ model, which includes semantic segmentation and post-processing. In the semantic segmentation part, the classification mask of the objects is predicted through the input RGB image, and then the predicted objects of different classifications are fitted with the minimum bounding directed rectangle to obtain the two-dimensional grasp pose, and the final three-dimensional grasping pose is calculated through the conversion of the input depth image. On the validation dataset, we use the indicator of semantic segmentation to evaluate the proposed network and achieve a great result. In addition, the simulation robot experiment further verifies the effectiveness of the network.KeywordsGrasp pose detectionDeepLabv3+Minimum bounding directed rectangle fitting