MDANet: Unsupervised, Mixed-Domain Adaptation for Semantic Segmentation of Remote Sensing Images

Accurate evaluation of orchard areas from remote sensing images is of great importance in economic and ecological aspects. In practice, the differences in distributions between remote sensing images and the lack of data labels make the semantic segmentation model impossible to use in new data. Unsupervised domain adaptation (UDA) methods can improve the performance of the model in the target domain by aligning the source domain and the target domain. However, due to the class mismatch problem and the interference of high-dimensional feature complexity, most UDA methods cannot achieve satisfactory results in orchard areas segmentation task. To address these issues, we propose an UDA model for orchard areas segmentation by developing a class feature aggregate discriminator. The class feature aggregate discriminator is designed to distinguish intra-domain classes and align inter-domain classes, and class feature aggregate can represent class information of different domains, which helps the model to avoid the interference of complex information. In addition, adversarial loss reweighting is introduced to the novel model, which makes the segmentation model pay more attention to the orchard areas. To verify the effectiveness of the proposed method, we conducted extensive experiments in three different remote sensing images around Yichang City. Compared to the baseline model, the proposed approach improves IoU by 27.68%, and we achieve high gains of 6.07% in IoU over other UDA methods. The larger gain indicates that our proposed method has great potential in cross-domain orchard areas segmentation.

Deep learning-based building extraction methods have achieved a high accuracy in closed remote sensing datasets. In fact, the distribution bias between the source and target domains can lead to a dramatic decrease in their building extraction effect in the target domain. However, the mainstream domain adaptation methods that specifically address this domain bias problem require the reselection of many unlabeled samples and retraining in other target domains. This is time-consuming and laborious and even impossible at small regions. To address this problem, a novel domain adaptation framework for cross-domain building extraction is proposed from a perspective of anomaly detection. First, the initial extraction results of images in the target domain are obtained by a source domain-based pre-trained model, and then these results are classified into building mixed and non-building layers according to the predicted probability. Second, anomalous objects in the building layer are detected using the isolation forest method. Subsequently, the remaining objects in the building layer and the objects in the non-building layer are used as positive and negative samples, respectively, to reclassify the mixed layer using the random forest classifier. The newly extracted objects are fused with the remaining objects in the building layer as the final result. Four different experiments are performed on different semantic segmentation models and target domains. Some experimental results indicate that our framework can improve cross-domain building extraction compared to the pre-trained model, with an 8.7% improvement in the F1 metric when migrating from the Inria Aerial Image Labeling dataset to the Wuhan University dataset. Furthermore, experimental results show that our framework can be applied to multiple target domains without retraining and can achieve similar results to domain adaptation models based on adversarial learning.

There is a large amount of out-of-distribution data (OOD) in remote sensing, which hinders high-accuracy segmentation models under the assumption of independent identical distribution (i.i.d.) from stable and reliable performance in real-world remote sensing applications. And Domain Adaptation (DA) is presented to seamlessly extend classifiers to the label-scarce target domain in the presence of the label-sufficient source domain with different data distributions. However, given that the domain shift, i.e. the distribution difference between the two domains, is more serious in remote sensing images, the current DA methods for image segmentation in Computer Vision (CV) typically perform unsatisfactorily in remote sensing, even suffering from the negative domain alignment. To this end, this paper proposes the Self-Training Adversarial Domain Adaptation (STADA) method for remote sensing image segmentation, which not only performs adversarial learning to extract domain-invariant features, but also implements Self-Training using pseudo-labels in the target domain denoised by the conditional adversarial loss for classifier adaptation. The ISPRS and WHU datasets are employed to conduct extensive experiments to investigate the effectiveness of STADA and the specific effect of its each DA component. And the experimental results demonstrate that STADA outperforms other state-of-the-art DA methods in the remote sensing image segmentation task.

Remote sensing images are usually characterized by complex backgrounds, scale and orientation variations, and large intraclass variance. General semantic segmentation methods usually fail to fully investigate the above issues, and thus their performances on remote sensing image segmentation are limited. In this article, we propose our LOGCAN++, a semantic segmentation model customized for remote sensing images, which is made up of a global class-aware (GCA) module and several local class-aware (LCA) modules. The GCA module captures global representations for class-level context modeling to reduce the interference of background noise. The LCA module generates local class representations as intermediate perceptual elements to indirectly associate pixels with the global class representations, targeting dealing with the large intraclass variance problem. In particular, we introduce affine transformations in the LCA module for adaptive extraction of local class representations to effectively tolerate scale and orientation variations in remote sensing images. Extensive experiments on three benchmark datasets show that our LOGCAN++ outperforms current mainstream general and remote sensing semantic segmentation methods and achieves a better trade-off between speed and accuracy.

Segmenting objects from cluttered backgrounds in single-channel images, such as marine radar echoes, medical images, and remote sensing images, poses significant challenges due to limited texture, color information, and diverse target types. This paper proposes a novel solution: the Onet, an O-shaped assembly of twin U-Net deep neural networks, designed for unsupervised binary semantic segmentation. The Onet, trained with an intensity-complementary image pair and without the need for annotated labels, maximizes the Jensen-Shannon divergence (JSD) between the densely localized features and the class probability maps. By leveraging the symmetry of U-Net, Onet subtly strengthens the dependence between dense local features, global features, and class probability maps during the training process. The design of the complementary input pair aligns with the theoretical requirement that optimizing JSD needs the class probability of negative samples to accurately estimate the marginal distribution. Compared to the current leading unsupervised segmentation methods, the Onet demonstrates superior performance in target segmentation in marine radar frames and cloud segmentation in remote sensing images. Notably, we found that Onet's foreground prediction significantly enhances the signal-to-noise ratio (SNR) of targets amidst marine radar clutter. Onet's source code is publicly accessible at https://github.com/joeyee/Onet.

Pine wilt disease (PWD) is the most dangerous biohazard of pine species and poses a serious threat to forest resources. Coupling satellite remote sensing technology and deep learning technology for the accurate monitoring of PWD is an important tool for the efficient prevention and control of PWD. We used Gaofen-2 remote sensing images to construct a dataset of discolored standing tree samples of PWD and selected three semantic segmentation models—DeepLabv3+, HRNet, and DANet—for training and to compare their performance. To build a GAN-based semi-supervised semantic segmentation model for semi-supervised learning training, the best model was chosen as the generator of generative adversarial networks (GANs). The model was then optimized for structural adjustment and hyperparameter adjustment. Aimed at the characteristics of Gaofen-2 images and discolored standing trees with PWD, this paper adopts three strategies—swelling prediction, raster vectorization, and forest floor mask extraction—to optimize the image identification process and results and conducts an application demonstration study in Nanping city, Fujian Province. The results show that among the three semantic segmentation models, HRNet was the optimal conventional semantic segmentation model for identifying discolored standing trees of PWD based on Gaofen-2 images and that its MIoU value was 68.36%. Additionally, the GAN-based semi-supervised semantic segmentation model GAN_HRNet_Semi improved the MIoU value by 3.10%, and its recognition segmentation accuracy was better than the traditional semantic segmentation model. The recall rate of PWD discolored standing tree monitoring in the demonstration area reached 80.09%. The combination of semi-supervised semantic segmentation technology and high-resolution satellite remote sensing technology provides new technical methods for the accurate wide-scale monitoring, prevention, and control of PWD.

Semantic segmentation in remote sensing images is a very challenging task. Each pixel in a remote sensing image has a semantic meaning to it and automatic annotation of each pixel remains as an open challenge for the research community due to its high spatial resolution. To address this issue deep learning based encoder-decoder architectures like SegNet and ResNet that is widely used for computer vision dataset is adopted for remote sensing images and its performance is analyzed based on the pixel wise classification accuracy. From the experiment conducted it is inferred that SegNet suffers from degradation problem when the depth of the network is increased with an overall accuracy of about 86.086% whereas the Residual network manages to overcome the degradation effect with an overall accuracy of about 87.747%.

As an important link between human and nature, land use/land cover change (LUCC) has become a research hotspot since the first day of the issue initiated. Remote sensing observation supplies important data source for LUCC research. With the constant improvement of the remote sensed image spatial resolution and the continuous development of computer scientific algorithms and models, the study of semantic segmentation in the field of automatic interpretation of remote sensed images has promoted quickly. There have been a number of achievements focusing on the one-class extraction of LUCC classification system, but limited progress in multi-class classifying mission when facing high resolution remote sensing data by using semantic segmentation model, because of its sophisticated structure, the phenomenon of synonyms spectrum and foreign body with spectrum in images. Additionally, the limitation of the semantic segmentation model is a crucial issue which needs to be solv for precision of classification in class recognition and the perfection of semantic segmentation on pixel level as well. The High-Resolution network (HRnet) model was used and improved in the study for semantic segmentation because it has the advantage of low resolution loss. In the LUCC classification work, the distribution of classes (such as cultivated land, woodland, grassland, other land, water, etc.) are usually extremely imbalanced, that leads to the limited classification recognition results of land types when using HRnet. So a combined Loss function of Lovasz Softmax Loss+Cross Entropy Loss was used in HRnet to classify the land cover of remote sensing images, to meet the requirements of high-resolution image classification not only in terms of low resolution loss but also in terms of class recognition. The combined loss function provided the direction of the model, and the appropriate loss function improved the performance of the model when faced with the imbalanced distribution of land classes. The results are as follows: 1) Compared with Deeplab and Unet, HRnet model performed better in segmentation results; 2) HRnet model with combined Loss function Lovasz Softmax Loss+Cross Entropy Loss (HRnet+) had better performance, and its test accuracy PA(Pixel Accuracy) value was 0.9577, MIOU (Mean Intersection over Union) value was 0.8801, and Kappa value was 0.9455. This indicated that the HRnet model with low loss of resolution could achieve better results in the automatic segmentation of high-resolution remote sensing images, and the combined loss function constructed in this study could effectively solve the problem of imbalanced distribution of classes in remote sensing images, and the edge optimization ability was also more significant.

An improved SegNet semantic segmentation model is proposed to address the issue of traditional classification algorithms and shallow learning algorithms not being suitable for extracting information from high-resolution remote sensing images. During the research process, space remote sensing images obtained from the GF-1 satellite were used as the data source. In order to improve the operational efficiency of the encoding network, the pooling layer in the encoding network is removed and the ordinary convolutional layer is replaced with a depth-wise separable convolution. By decoding the last layer of the network to obtain the reshaped output results, and then calculating the probability of each classification using a Softmax classifier, the classification of pixels can be achieved. The output result of the classifier is the final result of the remote sensing image semantic segmentation model. The results showed that the proposed algorithm had the highest Kappa coefficient of 0.9531, indicating good classification performance.

Abstract. With deepening application of deep learning technology in the field of remote sensing, several challenges persist in the segmentation of remote sensing optical images. These challenges include: (1) insufficient availability of deep learning-based remote sensing semantic segmentation datasets; (2) inadequate utilization of multi-source remote sensing data in the field of semantic segmentation; (3) limited sample size for effective model training, as well as the need to enhance both the speed and accuracy of model training. To address these challenges, this study introduces a multi-source remote sensing dataset consisting of 15000 data pairs, each comprising remote sensing multispectral data, synthetic aperture radar(SAR) images, land use and land cover(LULC) data, digital elevation model (DEM) data as well as the analysis data including Slope, Aspect, and Hillshade. Through the application of an end-to-end network based on pix2pix, effective fusion and feature enhancement of multi-source remote sensing data were achieved. The structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and Spectral Angle Mapper(SAM) values reached 0.84, 23.14, and 0.19, respectively, representing significant improvements over the baseline Pix2pix model’s performance metrics of 0.62, 18.84, and 0.23. In downstream semantic segmentation applications, the enhanced dataset was utilized to train semantic segmentation models for remote sensing image analysis. This approach effectively improved the training speed and segmentation accuracy of the models, with the mean intersection over union (mIoU) increasing from 0.467 to 0.481 and accuracy rising from 0.734 to 0.746. Moreover, the visual representation of remote sensing image segmentation demonstrated noticeable enhancements.

To alleviate the need for large-scale pixel-wise annotations, domain adaptation for semantic segmentation trains segmentation models on synthetic data (source) with computer-generated annotations, which can be then generalized to segment realistic images (target). Recently, self-supervised learning (SSL) with a combination of image-to-image translation shows great effectiveness in adaptive segmentation. The most common practice is to perform SSL along with image translation to well align a single domain (source or target). However, in this single-domain paradigm, unavoidable visual inconsistency raised by image translation may affect subsequent learning. In addition, pseudo labels generated by a single segmentation model aligned in either the source or target domain may be not accurate enough for SSL. In this paper, based on the observation that domain adaptation frameworks performed in the source and target domain are almost complementary, we propose a novel adaptive dual path learning (ADPL) framework to alleviate visual inconsistency and promote pseudo-labeling by introducing two interactive single-domain adaptation paths aligned in source and target domain respectively. To fully explore the potential of this dual-path design, novel technologies such as dual path image translation (DPIT), dual path adaptive segmentation (DPAS), dual path pseudo label generation (DPPLG) and Adaptive ClassMix are proposed. The inference of ADPL is extremely simple, only one segmentation model in the target domain is employed. Our ADPL outperforms the state-of-the-art methods by large margins on GTA5 →Cityscapes, SYNTHIA → Cityscapes and GTA5 →BDD100K scenarios. Code and models are available at https://github.com/royee182/DPL.

Semantic segmentation methods based on deep neural networks have achieved great success in recent years. However, training such deep neural networks relies heavily on a large number of images with accurate pixel-level labels, which requires a huge amount of human effort, especially for large-scale remote sensing images. In this paper, we propose a point-based weakly supervised learning framework called the deep bilateral filtering network (DBFNet) for the semantic segmentation of remote sensing images. Compared with pixel-level labels, point annotations are usually sparse and cannot reveal the complete structure of the objects; they also lack boundary information, thus resulting in incomplete prediction within the object and the loss of object boundaries. To address these problems, we incorporate the bilateral filtering technique into deeply learned representations in two respects. First, since a target object contains smooth regions that always belong to the same category, we perform deep bilateral filtering (DBF) to filter the deep features by a nonlinear combination of nearby feature values, which encourages the nearby and similar features to become closer, thus achieving a consistent prediction in the smooth region. In addition, the DBF can distinguish the boundary by enlarging the distance between the features on different sides of the edge, thus preserving the boundary information well. Experimental results on two widely used datasets, the ISPRS 2-D semantic labeling Potsdam and Vaihingen datasets, demonstrate that our proposed DBFNet can achieve a highly competitive performance compared with state-of-the-art fully-supervised methods. Code is available at https://github.com/Luffy03/DBFNet.

Domain Adaptation (DA) methods are usually carried out by means of simply reducing the marginal distribution differences between the source and target domains, and subsequently using the resultant trained classifier, namely source classifier, for use in the target domain. However, in many cases, the true predictive distributions of the source and target domains can be vastly different especially when their class distributions are skewed, causing the issues of sample selection bias in DA. Hence, DA methods which leverage the source labeled data may suffer from poor generalization in the target domain, resulting in negative transfer. In addition, we observed that many DA methods use either a source classifier or a linear combination of source classifiers with a fixed weighting for predicting the target unlabeled data. Essentially, the labels of the target unlabeled data are spanned by the prediction of these source classifiers. Motivated by these observations, in this paper, we propose to construct many source classifiers of diverse biases and learn the weight for each source classifier by directly minimizing the structural risk defined on the target unlabeled data so as to heal the possible sample selection bias. Since the weights are learned by maximizing the margin of separation between opposite classes on the target unlabeled data, the proposed method is established here as Maximal Margin Target Label Learning (MMTLL), which is in a form of Multiple Kernel Learning problem with many label kernels. Extensive experimental studies of MMTLL against several state-of-the-art methods on the Sentiment and Newsgroups datasets with various imbalanced class settings showed that MMTLL exhibited robust accuracies on all the settings considered and was resilient to negative transfer, in contrast to other counterpart methods which suffered significantly in prediction accuracy.

In recent years, advancements in remote sensing image observation technology have significantly enriched the surface feature information captured in remote sensing images, posing greater challenges for semantic information extraction from remote sensing imagery. While convolutional neural networks (CNNs) excel at understanding relationships between adjacent image regions, processing multidimensional data requires reliance on attention mechanisms. However, due to the inherent complexity of remote sensing images, most attention mechanisms designed for natural images underperform when applied to remote sensing data. To address these challenges in remote sensing image semantic segmentation, we propose a highly generalizable multi-branch attention fusion method based on shallow and deep features. This approach applies pixel-wise, spatial, and channel attention mechanisms to feature maps fused with shallow and deep features, thereby enhancing the network’s semantic information extraction capability. Through evaluations on the Cityscapes, LoveDA, and WHDLD datasets, we validate the performance of our method in processing remote sensing data. The results demonstrate consistent improvements in segmentation accuracy across most categories, highlighting its strong generalization capability. Specifically, compared to baseline methods, our approach achieves average mIoU improvements of 0.42% and 0.54% on the WHDLD and LoveDA datasets, respectively, significantly enhancing network performance in complex remote sensing scenarios.

Unpaired image-to-image translation is the problem of mapping an image in the source domain to one in the target domain, without requiring corresponding image pairs. To ensure the translated images are realistically plausible, recent works, such as Cycle-GAN, demands this mapping to be invertible. While, this requirement demonstrates promising results when the domains are unimodal, its performance is unpredictable in a multi-modal scenario such as in an image segmentation task. This is because, invertibility does not necessarily enforce semantic correctness. To this end, we present a semantically-consistent GAN framework, dubbed Sem-GAN, in which the semantics are defined by the class identities of image segments in the source domain as produced by a semantic segmentation algorithm. Our proposed framework includes consistency constraints on the translation task that, together with the GAN loss and the cycle-constraints, enforces that the images when translated will inherit the appearances of the target domain, while (approximately) maintaining their identities from the source domain. We present experiments on several image-to-image translation tasks and demonstrate that Sem-GAN improves the quality of the translated images significantly, sometimes by more than 20% on the FCN score. Further, we show that semantic segmentation models, trained with synthetic images translated via Sem-GAN, leads to significantly better segmentation results than other variants.