Relevancy between Objects Based on Common Sense for Semantic Segmentation

Methods and datasets on semantic segmentation for Unmanned Aerial Vehicle remote sensing images: A review

Semantic segmentation is an important task in self-driving cars. The aims of semantic segmentation are to recognize pre-defined objects and its pixel-by-pixel location. The most popular method in semantic segmentation is Deep learning which has considerably improved semantic image segmentation. This work does an overview for semantic segmentation using Deep learning. This works also implement comparisons in term of precision, mean IOU and processing time. Three popular algorithms are PSPNet, FCN and SegNet that are examined carefully. In detail, the aim of this work points out a trade-off between processing time and mean IOU, and also between processing time and precision. Moreover, this paper concentrates on road segmentation for embedded devices, so processing time is significantly important. This work also figures out which method is suitable for embedded devices on road segmentation.

Semantic segmentation has always been a very challenging research topic in computer vision and deep learning and has extensive applications in real-life scenarios. With the development of computing hardware and deep learning technology, researchers have a higher research enthusiasm for semantic segmentation. This work briefly introduces several semantic segmentation models, datasets, and the main issues of traditional semantic segmentation. Afterward, the current mainstream semantic segmentation algorithm and the experimental results were compared. Finally, future research directions of semantic segmentation are presented and discussed.

Chapter 12 - Semantic scene segmentation for robotics

Deep learning (DL)-based approaches are notable for their ability to establish feature associations without relying on physical constraints, unlike traditional strategies that are complex and dependent on expert experience. However, three main challenges hinder the versatility of semantic segmentation models. First, the targets in these images are dense and exist at varying spatial scales, which imposes higher demands on the model for accurate segmentation across scales. Second, the segmentation of small targets in the images is often overlooked, leading to a compromise between fine segmentation and model efficiency. Lastly, the data-intensive nature of remote sensing images and the resource-intensive operations of large-scale networks impose significant communication and computation burdens on edge devices, which may not have sufficient resources to handle them effectively. To address these challenges, this paper proposes a lightweight semantic segmentation method for remote sensing images to achieve high-precision segmentation for multi-scale targets while maintaining low computational complexity. The main components include: (1) embedding the inverted residual block structure to minimize the number of model parameters and computational costs; (2) introducing the parallel irregular space pyramid pooling module to efficiently aggregate multi-scale contextual information for fine-grained recognition of small targets; and (3) embedding transfer learning into the encoder-decoder structure to speed up the convergence rate and improve multi-scale feature fusion capability, thereby reducing semantic information loss. The proposed lightweight method has been extensively tested on real-world high-resolution remote sensing datasets. It achieved PA, MPA, MIoU, and FWIoU scores of 87.90%, 75.76%, 66.29%, and 78.81% on the Vaihingen dataset; 87.03%, 85.31%, 74.85%, and 77.54% on the Potsdam dataset; and 95.37%, 83.33%, 75.70%, and 91.31% on the Aeroscapes dataset. Compared to other popular semantic segmentation models, the proposed method achieved the highest values in all four evaluation indicators, demonstrating its effectiveness and superiority.

Synthetic Aperture Radar (SAR) imagery have been one of the important tools to support earth observations and topographic measurements. It means SAR imagery are essentially rich in structures and some important target categories are difficult to recognize. Optical imagery contains rich and clear spectral information which has a good influence on semantic image segmentation. The success of deep neural networks for semantic segmentation heavily depends on large-scale and well-labeled data sets, which are hard to collect in practice. In this paper, we consider deep transfer learning for semantic segmentation, we propose a deep novel transfer learning method, which transfers a semantically segment model from SAR imagery to SAR and optical fusion imagery. The experimental results show that the method proposed achieves higher mean Intersection over Union (mIoU) with less training time compared with other methods.

Remote sensing images contain a wealth of geospatial information. To accurately identify different geospatial categories and extract relevant data, image semantic segmentation plays a crucial role. In recent years, deep learning technology has brought significant breakthroughs to semantic segmentation of remote sensing images, significantly enhancing its performance. This paper investigates the application of deep learning technologies in remote sensing image semantic segmentation, based on Convolutional Neural Networks (CNN) and Transformer-based semantic segmentation methods. It conducts an in-depth comparison of their structural characteristics and applicable scenarios, summarizes the achievements and shortcomings of existing research, and provides technical references and theoretical support for future studies, thereby contributing to the further development of deep learning technology in the field of remote sensing. Research results indicate that CNN-based semantic segmentation methods still hold advantages in extracting local features and achieving efficient segmentation, whereas Transformers address CNN's limitations in global context modeling and long-range dependency capture. Therefore, the collaborative integration of CNN and Transformers will become an important research direction for enhancing model performance in the future.

The surface water in the lakeshore zone is the primary area where cyanobacteria bloom floats intensively. In lake water environment monitoring, it has become pressing to accurately identify the distribution and accumulation coverage area of cyanobacteria blooms in the surface water of the lakeshore zone. This study proposes a real-time and dynamic monitoring technology for cyanobacteria blooms in surface water using a shore-based camera monitoring network. The specific work is as follows: Chaohu Lake, a large eutrophic lake in China, is selected as the research object. The multithreading technology is used to dynamically obtain the hourly video images of 43 cameras around Chaohu Lake. The semantic segmentation method is used to identify the cyanobacteria blooms in the video images, calculate the coverage of cyanobacteria blooms, and draw the spatial distribution map of cyanobacteria blooms in the lakeshore zone of Chaohu Lake. To improve the accuracy of cyanobacteria blooms recognition, we use the ResNet-50 network to integrate three semantic segmentation models, namely FCN, U-net, and DeeplabV3+. By comparing the cyanobacteria blooms results identified by the three methods, it is found that the boundary of the cyanobacteria blooms results identified by DeeplabV3+(ResNet-50) is clear, which is more consistent with the real spatial information of the distribution of cyanobacteria blooms and is more suitable for monitoring the hourly dynamic changes of cyanobacteria blooms in the Chaohu Lake lakeshore zone. The results demonstrated that the time requirement of monitoring cyanobacteria blooms in real time on an hourly basis could be met by utilizing technology that uses multiple threads. The OA (Overall Accuracy), MPA (Mean Pixel Accuracy), IOU (Intersection Over Union) of cyanobacteria blooms, and the IOU of water values of the DeeplabV3+(ResNet-50) were the highest, which were 0.83, 0.82, 0.71, and 0.74, and the RMSE between the predicted and real cyanobacterial blooms coverage of 43 cameras was 6.65%. The above values show that DeeplabV3+(ResNet-50) is this technology’s most suitable semantic segmentation model. This technique can provide technical support for the scientific development of a cyanobacteria blooms management plan in the lakeshore zone of Chaohu Lake by calculating the coverage area of cyanobacteria blooms and drawing the spatial distribution map of cyanobacteria blooms in the lakeshore zone.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.

Convolutional Neural Network (CNN) is widely used for semantic segmentation and land-use and land-cover (LULC) mapping of very high-resolution (VHR) remote sensing images. The convolution operation is a powerful method for VHR classification, but the loss of high-frequency detail information caused during its operation decreases the classification accuracy, particularly in the boundary. Thus, it is necessary to supply additional boundary information to the CNN for alleviating this situation. In the classification task (and in LULC mapping), providing more effective information generates a better classification result. Current methods regard the boundary of images as the same category object and process it uniformly, which loses a notable amount of useful information because of the different properties, such as ambiguity and transition, between remote sensing images and their boundaries. Thus, a semantic segmentation method with category boundary for LULC mapping is proposed in this paper. First, a multi-task CNN called the category boundary detection network (CBDN) is designed to extract the boundary information of different category objects. Second, this category boundary and VHR images are used for initial semantic segmentation. Finally, the category boundary and the initial semantic segmentation result (ISSR) are fused to obtain the final LULC map by a two-step strategy, including the explicit fusion and the boundary attention loss function. To verify whether category boundary improved the classification accuracy, a set of comparative experiments were conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets. The method in this paper was compared with a semantic segmentation method with no boundary information and a semantic segmentation method with global boundary. The results showed that the proposed method in this paper achieved good performances in the Vaihingen (overall accuracy (OA) = 0.924, Kappa coefficient (K) = 0.898, mean F1 score (mF1) = 0.896 and mean Intersection over Union (mIoU) = 0.817) and Potsdam datasets (OA = 0.890, K = 0.857, mF1 = 0.923, and mIoU = 0.860) based on the eroded labels.

Pine wilt disease (PWD) is the most dangerous biohazard of pine species and poses a serious threat to forest resources. Coupling satellite remote sensing technology and deep learning technology for the accurate monitoring of PWD is an important tool for the efficient prevention and control of PWD. We used Gaofen-2 remote sensing images to construct a dataset of discolored standing tree samples of PWD and selected three semantic segmentation models—DeepLabv3+, HRNet, and DANet—for training and to compare their performance. To build a GAN-based semi-supervised semantic segmentation model for semi-supervised learning training, the best model was chosen as the generator of generative adversarial networks (GANs). The model was then optimized for structural adjustment and hyperparameter adjustment. Aimed at the characteristics of Gaofen-2 images and discolored standing trees with PWD, this paper adopts three strategies—swelling prediction, raster vectorization, and forest floor mask extraction—to optimize the image identification process and results and conducts an application demonstration study in Nanping city, Fujian Province. The results show that among the three semantic segmentation models, HRNet was the optimal conventional semantic segmentation model for identifying discolored standing trees of PWD based on Gaofen-2 images and that its MIoU value was 68.36%. Additionally, the GAN-based semi-supervised semantic segmentation model GAN_HRNet_Semi improved the MIoU value by 3.10%, and its recognition segmentation accuracy was better than the traditional semantic segmentation model. The recall rate of PWD discolored standing tree monitoring in the demonstration area reached 80.09%. The combination of semi-supervised semantic segmentation technology and high-resolution satellite remote sensing technology provides new technical methods for the accurate wide-scale monitoring, prevention, and control of PWD.

Recent advances in convolutional neural networks have led to considerably accurate semantic segmentation tasks. However, previous works use a pre-trained backbone borrowed from image classification tasks with considerable computational complexity, resulting in large latencies. Many methods have been proposed to reduce the latency of the segmentation network without loss of accuracy. These methods have reported varying experimental results with different computing devices, acceleration techniques, and input image sizes. Although most studies claim that their results are state-of-the-art, it is necessary to reconsider whether they are being compared under the same conditions. We propose a performance evaluation method of real-time semantic segmentation models to compare the performance under the same conditions fairly. In addition, we carry out an empirical study to evaluate the performance of recent real-time semantic segmentation networks and make a comparative analysis between them. We train the segmentation models using the same input data and data augmentation method. Then, the performance of the segmentation methods is analyzed regarding accuracy and speed. In contrast to most studies that exclude the time required for the pre-processing and post-processing steps, we measured the actual processing time needed to perform semantic segmentation with a real dataset. Further, we measured the processing speed and power consumption of the segmentation models in embedded devices in which real-time segmentation is applied, unlike previous studies that measured performance on a PC. Experimental results showed that the real-time semantic segmentation methods could not run in real-time on embedded devices when considering the pre-processing and post-processing steps. By comprehensively considering the inference speed, energy consumption, and processing time of semantic segmentation models, the experimental results show that FasterSeg-S is suited for embedded devices.

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

Archaeological site segmentation of ancient city walls based on deep learning and LiDAR remote sensing

<p>Perception is a fundamental task of autonomous driving systems, which gathers all the necessary information about the surrounding environment of the moving vehicle. Then a decision-making system takes the perception data as input and provides the optimum decision given a scenario, which maximizes the safety of the passengers. In this project, we have developed variants of the U-Net model to perform semantic segmentation on urban scene images to understand the surroundings of an autonomous vehicle. The U-Net model and its variants are adopted for semantic segmentation in this project to account for the power of the U-Net in handling large and small datasets. We have also compared the best-performing variant with other commonly used semantic segmentation models. The comparative analysis was performed using three well-known models, including FCN-16, FCN-8, and SegNet. After conducting sensitivity and comparative analysis, it is concluded that the U-Net variants performed the best in terms of the Intersection over Union (IoU) evaluation metric and other quality metrics.</p>