Analysis of Encoder-Decoder Based Deep Learning Architectures for Semantic Segmentation in Remote Sensing Images

Segmenting objects from cluttered backgrounds in single-channel images, such as marine radar echoes, medical images, and remote sensing images, poses significant challenges due to limited texture, color information, and diverse target types. This paper proposes a novel solution: the Onet, an O-shaped assembly of twin U-Net deep neural networks, designed for unsupervised binary semantic segmentation. The Onet, trained with an intensity-complementary image pair and without the need for annotated labels, maximizes the Jensen-Shannon divergence (JSD) between the densely localized features and the class probability maps. By leveraging the symmetry of U-Net, Onet subtly strengthens the dependence between dense local features, global features, and class probability maps during the training process. The design of the complementary input pair aligns with the theoretical requirement that optimizing JSD needs the class probability of negative samples to accurately estimate the marginal distribution. Compared to the current leading unsupervised segmentation methods, the Onet demonstrates superior performance in target segmentation in marine radar frames and cloud segmentation in remote sensing images. Notably, we found that Onet's foreground prediction significantly enhances the signal-to-noise ratio (SNR) of targets amidst marine radar clutter. Onet's source code is publicly accessible at https://github.com/joeyee/Onet.

Remote sensing images are usually characterized by complex backgrounds, scale and orientation variations, and large intraclass variance. General semantic segmentation methods usually fail to fully investigate the above issues, and thus their performances on remote sensing image segmentation are limited. In this article, we propose our LOGCAN++, a semantic segmentation model customized for remote sensing images, which is made up of a global class-aware (GCA) module and several local class-aware (LCA) modules. The GCA module captures global representations for class-level context modeling to reduce the interference of background noise. The LCA module generates local class representations as intermediate perceptual elements to indirectly associate pixels with the global class representations, targeting dealing with the large intraclass variance problem. In particular, we introduce affine transformations in the LCA module for adaptive extraction of local class representations to effectively tolerate scale and orientation variations in remote sensing images. Extensive experiments on three benchmark datasets show that our LOGCAN++ outperforms current mainstream general and remote sensing semantic segmentation methods and achieves a better trade-off between speed and accuracy.

Aircraft segmentation in remote sensing images (RSIs) is an important but challenging problem for both civil and military applications. U-Net and its variants are widely used in RSI detection, but they are not suitable for multi-scale aircraft segmentation in RSIs, due to the aircrafts in RSIs are relatively small with various orientations, different sizes, fuzzy illumination and shadow, obscure boundary and irregular background. To overcome this problem, a multi-scale residual U-Net with attention (MSRAU-Net) model is constructed for multi-scale aircraft segmentation in RSIs. A multi-scale convolutional module, two modified Respaths and two kinds of attention modules are designed and introduced into MSRAU-Net to extract the multi-scale feature and make the feature fusion between the contraction path and the expansion path more efficient. Different from U-Net, MSRAU-Net replaces the convolutional block of U-Net with the Inception residual block to help the U-Net architecture coordinate the features learned from aircrafts with different scales, and the residual module and attention module are introduced into the modified Respath to deepen the network layers and solve the gradient disappearing problem while extracting the more effective feature from RSIs. The experiments on the RSI dataset validate that MSRAU-Net outperforms the other networks, in particular for detecting the small aircrafts. Compared with attention U-Net and MultiMixUNet, the precision of MSRAU-Net is improved by 9.25 and 3.36, respectively.

Semantic segmentation methods based on deep neural networks have achieved great success in recent years. However, training such deep neural networks relies heavily on a large number of images with accurate pixel-level labels, which requires a huge amount of human effort, especially for large-scale remote sensing images. In this paper, we propose a point-based weakly supervised learning framework called the deep bilateral filtering network (DBFNet) for the semantic segmentation of remote sensing images. Compared with pixel-level labels, point annotations are usually sparse and cannot reveal the complete structure of the objects; they also lack boundary information, thus resulting in incomplete prediction within the object and the loss of object boundaries. To address these problems, we incorporate the bilateral filtering technique into deeply learned representations in two respects. First, since a target object contains smooth regions that always belong to the same category, we perform deep bilateral filtering (DBF) to filter the deep features by a nonlinear combination of nearby feature values, which encourages the nearby and similar features to become closer, thus achieving a consistent prediction in the smooth region. In addition, the DBF can distinguish the boundary by enlarging the distance between the features on different sides of the edge, thus preserving the boundary information well. Experimental results on two widely used datasets, the ISPRS 2-D semantic labeling Potsdam and Vaihingen datasets, demonstrate that our proposed DBFNet can achieve a highly competitive performance compared with state-of-the-art fully-supervised methods. Code is available at https://github.com/Luffy03/DBFNet.

Semantic segmentation has been widely researched for high level analysis of High Spatial Resolution (HSR) remote sensing images, where Convolutional Neural Network (CNN) is the mainstream method. However, the transformer with attention mechanism has its unique capacity of extracting global information which is generally ignored by CNN models. In this paper, a Swin Transformer with UPer head (STUP) is proposed to tackle with semantic segmentation problem on a challenging remote sensing land-cover dataset called LoveDA, which owns complex background samples and inconsistent classes distributions. The proposed STUP combines the Swin Transformer with Uper Head in the form of an encoder-decoder structure, to extract features of HSR images for segmentation. Furthermore, Focal Loss is adopted to handle the unbalanced distribution problem in the training step. Experimental results demonstrate that the proposed STUP clearly outperforms several state-of-the-art models.

In recent years, advancements in remote sensing image observation technology have significantly enriched the surface feature information captured in remote sensing images, posing greater challenges for semantic information extraction from remote sensing imagery. While convolutional neural networks (CNNs) excel at understanding relationships between adjacent image regions, processing multidimensional data requires reliance on attention mechanisms. However, due to the inherent complexity of remote sensing images, most attention mechanisms designed for natural images underperform when applied to remote sensing data. To address these challenges in remote sensing image semantic segmentation, we propose a highly generalizable multi-branch attention fusion method based on shallow and deep features. This approach applies pixel-wise, spatial, and channel attention mechanisms to feature maps fused with shallow and deep features, thereby enhancing the network’s semantic information extraction capability. Through evaluations on the Cityscapes, LoveDA, and WHDLD datasets, we validate the performance of our method in processing remote sensing data. The results demonstrate consistent improvements in segmentation accuracy across most categories, highlighting its strong generalization capability. Specifically, compared to baseline methods, our approach achieves average mIoU improvements of 0.42% and 0.54% on the WHDLD and LoveDA datasets, respectively, significantly enhancing network performance in complex remote sensing scenarios.

The imaging process of optical remote sensing images are easily affected by external conditions. Therefore, remote sensing images under different imaging conditions often show color differences, resulting in feature distribution differences between the source and target domain, hindering the migration of semantic segmentation models between domains. Currently, most domain adaptation methods are for single-source and single-target domains. Here, we proposed a novel and concise method, coined MDANet, for the adaptation of patch images of multi-source and multi-target domains and for reducing the distribution differences of different patch images by projecting them onto the virtual center of a mixed-domain. MDANet is a lightweight and self-supervised network that can be grafted with any semantic segmentation model. Our method significantly improved the segmentation accuracy of semantic segmentation models and showed higher stability and competitiveness than existing methods.

Airport runways, as the core part of airports, belong to vital national infrastructure, and the target detection and segmentation of airport runways in remote sensing images using deep learning methods have significant research value. Most of the existing airport target detection methods based on deep learning rely on horizontal bounding boxes for localization, which often contain irrelevant background information. Moreover, when detecting multiple intersecting airport runways in a single remote sensing image, issues such as false positives and false negatives are apt to occur. To address these challenges, this study proposes an end-to-end remote sensing image airport runway detection and segmentation method based on an improved Mask RCNN (CR-Mask RCNN). The proposed method uses a rotated region generation network instead of a non-rotated region generation network, allowing it to generate rotated bounding boxes that fit the shape of the airport runway more closely, thus avoiding the interference of a large amount of invalid background information brought about by horizontal bounding boxes. Furthermore, the method incorporates an attention mechanism into the backbone feature extraction network to allocate attention to different airport runway feature map scales, which enhances the extraction of local feature information, captures detailed information more effectively, and reduces issues of false positives and false negatives when detecting airport runway targets. The results indicate that, when comparing horizontal bounding boxes with rotated bounding boxes for detecting and segmenting airport runways, the latter are more precise for complex backgrounds. Furthermore, incorporating an attention mechanism enhances the accuracy of airport runway recognition, making it highly effective and practical.

Cloud is always the weak and even uninformative area inevitably existing in the remote sensing images, and greatly limits the development of remote sensing application. Accurate and automatic detection and segmentation of clouds in satellite scenes is a key problem for the application of remote sensing images. In this paper, we propose a fully convolutional network (FCN) based on deep learning framework for cloud segmentation from remote sensing images. First, the obtained remote sensing image is pre-processed by using band merging and size cropping. Second the image is input into the FCN to achieve the cloud segmentation. To demonstrate the effectiveness of the method, we conduct experiments on Landsat-8 satellite dataset. The overall accuracy of our proposed method for cloud segmentation is higher than 88%. Experimental results demonstrate that both thin and thick cloud can be well segmented with higher accuracy and robustness by using our method.

As remote sensing images have complex backgrounds and varying object sizes, their semantic segmentation is challenging. This study proposes a multiscale cascaded network (MSCNet) for semantic segmentation. The resolutions employed with respect to the input remote sensing images are 1, 1/2, and 1/4, which represent high, medium, and low resolutions. First, 3 backbone networks extract features with different resolutions. Then, using a multiscale attention network, the fused features are input into the dense atrous spatial pyramid pooling network to obtain multiscale information. The proposed MSCNet introduces multiscale feature extraction and attention mechanism modules suitable for remote sensing land-cover classification. Experiments are performed using the Deepglobe, Vaihingen, and Potsdam datasets; the results are compared with those of the existing classical semantic segmentation networks. The findings indicate that the mean intersection over union (mIoU) of the MSCNet is 4.73% higher than that of DeepLabv3+ with the Deepglobe datasets. For the Vaihingen datasets, the mIoU of the MSCNet is 15.3%, and 6.4% higher than those of a segmented network (SegNet), and DeepLabv3+, respectively. For the Potsdam datasets, the mIoU of the MSCNet is higher than those of a fully convolutional network, Res-U-Net, SegNet, and DeepLabv3+ by 11.18%, 5.89%, 4.78%, and 3.03%, respectively.

Accurate evaluation of orchard areas from remote sensing images is of great importance in economic and ecological aspects. In practice, the differences in distributions between remote sensing images and the lack of data labels make the semantic segmentation model impossible to use in new data. Unsupervised domain adaptation (UDA) methods can improve the performance of the model in the target domain by aligning the source domain and the target domain. However, due to the class mismatch problem and the interference of high-dimensional feature complexity, most UDA methods cannot achieve satisfactory results in orchard areas segmentation task. To address these issues, we propose an UDA model for orchard areas segmentation by developing a class feature aggregate discriminator. The class feature aggregate discriminator is designed to distinguish intra-domain classes and align inter-domain classes, and class feature aggregate can represent class information of different domains, which helps the model to avoid the interference of complex information. In addition, adversarial loss reweighting is introduced to the novel model, which makes the segmentation model pay more attention to the orchard areas. To verify the effectiveness of the proposed method, we conducted extensive experiments in three different remote sensing images around Yichang City. Compared to the baseline model, the proposed approach improves IoU by 27.68%, and we achieve high gains of 6.07% in IoU over other UDA methods. The larger gain indicates that our proposed method has great potential in cross-domain orchard areas segmentation.

With the rapid development of the internet and popularization of intelligent mobile devices, social media is evolving fast and contains rich spatial information, such as geolocated posts, tweets, photos, video, and audio. Those location-based social media data have offered new opportunities for hazards and disaster identification or tracking, recommendations for locations, friends or tags, pay-per-click advertising, etc. Meanwhile, a massive amount of remote sensing (RS) data can be easily acquired in both high temporal and spatial resolution with a multiple satellite system, if RS maps can be provided, to possibly enable the monitoring of our location-based living environments with some devices like charge-coupled device (CCD) cameras but on a much larger scale. To generate the classification maps, usually, labeled RS image pixels should be provided by RS experts to train a classification system. Traditionally, labeled samples are obtained according to ground surveys, image photo interpretation or a combination of the aforementioned strategies. All the strategies should be taken care of by domain experts, in a means which is costly, time consuming, and sometimes of a low quality due to reasons such as photo interpretation based on RS images only. These practices and constraints make it more challenging to classify land-cover RS images using big RS data. In this paper, a new methodology is proposed to classify urban RS images by exploiting the semantics of location-based social media photos (SMPs). To validate the effectiveness of this methodology, an automatic classification system is developed based on RS images as well as SMPs via big data analysis techniques including active learning, crowdsourcing, shallow machine learning, and deep learning. As the labels of RS training data are given by ordinary people with a crowdsourcing technique, the developed system is named Crowd4RS. The quantitative and qualitative experiments confirm the effectiveness of the proposed Crowd4RS system as well as the proposed methodology for automatically generating RS image maps in terms of classification results based on big RS data made up of multispectral RS images in a high spatial resolution and a large amount of photos from social media sites, such as Flickr and Panoramio.

Semantic segmentation of high-resolution optical remote sensing images is an important but challenging task. To solve the problem that many semantic segmentation networks fail to efficiently utilize global and local context information to improve the segmentation performance, this paper proposes a semantic segmentation network based on sparse self-attention (SDANet) to model the global context dependencies. Specifically, the feature maps are first divided into four regions in spatial and channel dimensions, respectively, and the divided feature maps are rearranged to form new regions. Second, the position and channel self-attention operations are performed on the rearranged regions. Third, the feature maps are restored to the original combination and the position together with channel self-attention operations are performed again to obtain the output feature maps. Finally, semantic segmentation is completed based on the output feature maps. Extensive experiments conducted on the ISPRS Vaihingen dataset demonstrate that the proposed method is superior to self-attention-based DANet, CCNet, and other general semantic segmentation networks, such as FCN, Deeplabv3+, HRNet, etc.

Cropland segmentation is one of the critical techniques in the agriculture Remote Sensing (RS). Although the Deep Learning (DL) methods have achieved remarkable performance in the natural vision, the cropland segmentation of RS images still suffers from cropland adhesion due to the interference from the surrounding environment and the cropland cover. To tackle this problem, this letter proposes a two stage DL method with spectral-spatial priors. In the first stage, the Multi-feature Extraction Module (MEM) is designed to predict the boundary, an important spatial prior of the cropland. In the second stage, the spatial prior is further fused with the spectral prior by MEMs to get accurate cropland prediction. To evaluate the effectiveness and robustness of the proposed method, we construct a data set called Jiaxiang Cropland Set (JCS) and propose a region level evaluation indicator namely the Plot Mean Intersection over Union (PMIoU). The experiment results on the JCS demonstrate that the proposed method is both qualitatively and quantitatively competitive compared with the state-of-the-art methods.

This article presents a transformer and convolutional neural network (CNN) hybrid deep neural network for semantic segmentation of very high resolution (VHR) remote sensing imagery. The model follows an encoder–decoder structure. The encoder module uses a new universal backbone Swin transformer to extract features to achieve better long-range spatial dependencies modeling. The decoder module draws on some effective blocks and successful strategies of CNN-based models in remote sensing image segmentation. In the middle of the framework, an atrous spatial pyramid pooling block based on depthwise separable convolution (SASPP) is applied to obtain a multiscale context. A U-shaped decoder is used to gradually restore the size of the feature maps. Three skip connections are built between the encoder and decoder feature maps of the same size to maintain the transmission of local details and enhance the communication of multiscale features. A squeeze-and-excitation (SE) channel attention block is added before segmentation for feature augmentation. An auxiliary boundary detection branch is combined to provide edge constraints for semantic segmentation. Extensive ablation experiments were conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam benchmarks to test the effectiveness of multiple components of the network. At the same time, the proposed method is compared with the current state-of-the-art methods on the two benchmarks. The proposed hybrid network achieved the second highest overall accuracy (OA) on both the Potsdam and Vaihingen benchmarks (code and models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zq7734509/mmsegmentation-</uri> multilayer).