Structure-Aware Weakly Supervised Network for Building Extraction From Remote Sensing Images

In recent years, there has been a significant increase in the production of high-resolution aerial and satellite imagery. Analyzing and extracting urban features from these images, especially building components, is a significant challenge in photogrammetry and remote sensing. Meanwhile, deep convolution neural networks have been used as a powerful model in the semantic segmentation of building features. However, due to the structure of these types of deep convolution networks, accurate retrieval of building boundary information during training will be difficult, and this will lead to ambiguous boundary areas. On the other hand, the use of distributed-based loss functions, such as the binary cross-entropy loss alone cannot improve the segmentation accuracy in the boundary areas of building features. Accordingly, in this paper, a derivative boundary loss function is introduced to optimize and enhance the extraction of buildings in border areas. To calculate the proposed boundary loss function, distance transform image (DTI), is obtained from ground-truth images and predicted images, that derived from the segmentation network. In this paper, a derivative method for calculating DTI is presented. Another advantage of the proposed loss function is its ability to be used in a wide range of deep convolution segmentation networks. The proposed method was investigated and tested using the ISPRS Potsdam and Vaihingen data sets. The results of this implementation show the superiority in the criteria of evaluation and improvement of buildings extraction if the proposed boundary loss function is used along with the distributed-based loss function.

Semantic segmentation of high-resolution remote sensing images remains challenging due to the degradation of high-frequency semantic cues during convolutional encoding and the lack of frequency consistency in multi-stage feature fusion. To address these issues, we propose FreDNet, a frequency-guided denoising network that explicitly enhances frequency-sensitive representations throughout the segmentation process. Specifically, we introduce the Dual-path Residual Block (DRB), which incorporates a Frequency-aware Denoising Module (FDM) and a Frequency-aware Fusion Module (FFM) to suppress frequency-domain noise while preserving edge structures. Furthermore, we design a Frequency-aware Cross-level Fusion Module (FCFM) that leverages frequency intensity response maps to adaptively fuse encoder and decoder features. These components work collaboratively to enhance the frequency robustness and spatial consistency of the segmentation predictions. Extensive experiments on three challenging benchmarks, ISPRS Vaihingen, ISPRS Potsdam, and LoveDA, demonstrate that FreDNet achieves superior performance, surpassing the latest state-of-the-art approaches by up to 0.8% in mean IoU and 0.9% in overall accuracy, while maintaining a lightweight inference cost. In addition, ablation study confirms the contribution of each component of FreDNet.

The clustering assumption is widely adopted in semi-supervised semantic segmentation methods. However, this assumption heavily relies on high-quality feature representation, leading to learning and cognitive biases if it does not hold. Learning bias entails the potential overfitting of labeled data, leading to capturing local features and consequently misclassifying semantic categories. Cognitive bias indicates the network’s susceptibility to interference features. Recently, a consistency-based mechanism has been proposed to address these biases. By subjecting unlabeled data to diverse weak perturbations, it breaks original clustering features, compelling the model to learn more robust and generalized representations. However, when processing complex remote sensing images, these weak perturbations often prove ineffective in the later stages of training. To address this, we propose a bias correction framework (BCF). The BCF begins with a feature consistency enhancement module (CEM) that guides the student model to learn feature representations with greater generalization capability. In the feature decoding stage, we introduce a multidecoder structure with weakly orthogonal weights to maximize feature differences, thereby further reducing learning and cognitive biases. In addition, to improve the confidence of pseudolabels and enhance consistency learning, we design a multidecoder teacher model based on symmetrical knowledge transfer, allowing the diverse and multiangle information learned by the student model to be transferred to the teacher model. Extensive experimental results show that our method significantly outperforms state-of-the-art methods on the ISPRS Potsdam and Vaihingen datasets.

In the context of fast progress in deep learning, convolutional neural networks (CNNs) have been extensively applied to the semantic segmentation of remote sensing images and have achieved significant progress. However, certain limitations exist in capturing global contextual information due to the characteristics of convolutional local properties. Recently, Transformer has become a focus of research in computer vision and has shown great potential in extracting global contextual information, further promoting the development of semantic segmentation tasks. In this paper, we use ResNet50 as an encoder, embed the hybrid attention mechanism into Transformer, and propose a Transformer-based decoder. The Channel-Spatial Transformer Block (CSTB) further aggregates features by integrating the local feature maps extracted by the encoder with their associated global dependencies. At the same time, an adaptive approach is employed to reweight the interdependent channel maps to enhance the feature fusion. The Global Cross-Fusion Module (GCFM) combines the extracted complementary features to obtain more comprehensive semantic information. Extensive comparative experiments were conducted on the ISPRS Potsdam and Vaihingen datasets, where mIoU reached 78.06% and 76.37%, respectively. The outcomes of multiple ablation experiments also validate the effectiveness of the proposed method.

Remote sensing image interpretation faces easily overlooked issues: (1) Ambiguity: mutual occlusion between objects, e.g., buildings are occluded by trees and vice versa, which leads to ambiguity (building or tree) in GIS; (2) Expression conflicts: topology conflicts between the occluding and occluded objects. However, there have been being no reports on the Remote Sensing Ambiguous Interpretation (RSAI) or its Dataset (RSAID). To address these issues, a RSAID was built and an ambiguous-interpretation generalized generative adversarial network (AIG<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>AN) with self-supervised raster-vector semantic segmentation for remote sensing image, is proposed for the first time. With the occluding image generated, AIG<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>AN not only recognizes occlusion regarded as cheating but also restores the occluded semantic labels, which includes three modules: (1) Random occlusion generator automatically generates the occluding image with the occluding and occluded labels for self-supervised learning with avoiding hallucination issue; (2) Ambiguity dual-interpreters, including both occluding and occluded interpretations, is the first to be proposed; (3) Consistency contrastive learning module is to process the inconsistent interpretations in unambiguous region. Furthermore, we propose a novel multispectral cross-modal learning paradigm via self-supervised pseudo-labeling to automatically enhance the generalization of our AIG<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>AN in different sensor modalities; and propose a conflict-free and regularized raster-vector expression for interpretation results of ambiguous objects. Experiments on our manually restored RSAID from ISPRS Vaihingen and Potsdam show that AIG<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>AN improves mIoU and building IoU by 3.28% and 1.80%, with ambiguity elimination index of 85.32% and 76.41% respectively. Moreover, we reveal occlusion restoration mechanism. RSAID and code are available at: https://github.com/wenlailiu/AIG2AN.

With the increasing availability of multimodal remote sensing (RS) data, semantic segmentation that leverages complementary information from true orthophotos (TOP) and digital surface models (DSM) has become essential for urban analysis. Diffusion-based segmentation provides an effective iterative refinement mechanism for modeling complex multimodal distributions; however, conventional pixel-wise supervision emphasizes local accuracy while overlooking global distribution alignment, often leading to inconsistent predictions and blurred object boundaries. Although maximum mean discrepancy (MMD) measures global statistical differences between predicted and ground-truth distributions, its effectiveness in high-dimensional class-probability spaces is limited by directional cancellation effects that reduce sensitivity to complex distribution shifts. To address this issue, we propose a projection-kernel regularized diffusion-based multimodal RS segmentation framework that enforces global statistical alignment through distribution-level regularization rather than modifying the intrinsic diffusion process. The proposed regularization performs multi-directional projections of high-dimensional class-probability vectors onto one-dimensional subspaces and derives a closed-form kernel integration to avoid numerical sampling across projection directions, enabling efficient and stable global distribution matching. In addition, a Cross-Attention Dual-Encoder Fusion (CADEF) module is introduced to alleviate geometry-texture misalignment, and a Hierarchical EMA-Gated Recursive Denoising (HERD) mechanism is designed to stabilize multiscale feature refinement. Experiments on the ISPRS Vaihingen and Potsdam benchmarks demonstrate that the proposed regularization consistently improves segmentation accuracy over state-of-the-art CNN-, Transformer-, and diffusion-based baselines, yielding enhanced global consistency and sharper boundary delineation. Code is available at: https://github.com/tonyy127/PKDiff.

The groundbreaking segment anything model (SAM), built on a vision transformer (ViT) design with millions of parameters and trained on the large SA-1B dataset, acts as a vision foundation model that can be used for various segmentation tasks. However, this model cannot be directly applied to the semantic segmentation of remote sensing images, as it tends to over-segment objects rather than preserving their semantics. Furthermore, it does not align well with object boundaries, which are usually required for the high-accuracy segmentation of high-resolution remote sensing images. To address this issue, we adjust the image encoder and mask decoder of SAM and propose an HL-SAM-Seg network. The image encoder is extensively adjusted to comprise an adapter, a high-resolution (high-res) path, and a low-resolution (low-res) path. The latter two paths extract and update the high-res and low-res features, which are merged in the mask decoder to perform multi-class segmentation. Specifically, we design a highlow- resolution cross-attention (HL) module inserted into the transformer blocks of the low-res path to align and update the low-res and high-res features. Experimental results on the ISPRS Vaihingen, ISPRS Potsdam, and FloodNet datasets show that the proposed HL-SAM-Seg outperformed conventional state-of-theart semantic segmentation algorithms overall, with underperformance in some small sample categories. Moreover, it has fewer trainable parameters, suggesting the potential for leveraging SAM for remote sensing image segmentation.

For semantic segmentation of high-resolution remote-sensing images, digital surface models (DSMs) information is useful for improving the accuracy and robustness of the segmentation models. However, since the feature distributions of spectral and DSM images vary significantly in different scenes, it is difficult to fuse them effectively in popular deep network models. To solve this issue, we propose an attention-based DSM fusion network (ADF-Net) for high-resolution remote-sensing image semantic segmentation. The proposed network makes two contributions. The first is that we design an attention-based feature fusion module, which can selectively gather features from spectral and DSM information by channel attention mechanism, and further combine them to get high-quality fusion features. The second is that we introduce a residual feature refinement module to reduce the redundant information from skip connection adaptively. We evaluate the proposed network on the ISPRS Vaihingen and Potsdam datasets, experimental results demonstrate that our model outperforms state-of-the-art methods.KeywordsHigh-resolution remote-sensing imagesAttention mechanismSemantic segmentationData fusion

Semantic segmentation has been a fundamental task in interpreting remote sensing imagery (RSI) for various downstream applications. Due to the high intra-class variants and inter-class similarities, inflexibly transferring natural image-specific networks to RSI is inadvisable. To enhance the distinguishability of learnt representations, attention modules were developed and applied to RSI, resulting in satisfactory improvements. However, these designs capture contextual information by equally handling all the pixels regardless of whether they around edges. Therefore, blurry boundaries are generated, rising high uncertainties in classifying vast adjacent pixels. Hereby, we propose an edge distribution attention module (EDA) to highlight the edge distributions of leant feature maps in a self-attentive fashion. In this module, we first formulate and model column-wise and row-wise edge attention maps based on covariance matrix analysis. Furthermore, a hybrid attention module (HAM) that emphasizes the edge distributions and position-wise dependencies is devised combing with non-local block. Consequently, a conceptually end-to-end neural network, termed as EDENet, is proposed to integrate HAM hierarchically for the detailed strengthening of multi-level representations. EDENet implicitly learns representative and discriminative features, providing available and reasonable cues for dense prediction. The experimental results evaluated on ISPRS Vaihingen, Potsdam and DeepGlobe datasets show the efficacy and superiority to the state-of-the-art methods on overall accuracy (OA) and mean intersection over union (mIoU). In addition, the ablation study further validates the effects of EDA.

Accurate semantic segmentation of high-resolution remote sensing imagery is challenged by substantial intra-class variability, inter-class similarity, and the limitations of single-modality data. This paper proposes MMFNet, a novel multimodal fusion network that leverages the Mamba architecture to efficiently capture long-range dependencies for semantic segmentation tasks. MMFNet adopts a dual-encoder design, combining ResNet-18 for local detail extraction and VMamba for global contextual modelling, striking a balance between segmentation accuracy and computational efficiency. A Multimodal Feature Fusion Block (MFFB) is introduced to effectively integrate complementary information from optical imagery and digital surface models (DSMs), thereby enhancing multimodal feature interaction and improving segmentation accuracy. Furthermore, a frequency-aware upsampling module (FreqFusion) is incorporated in the decoder to enhance boundary delineation and recover fine spatial details. Extensive experiments on the ISPRS Vaihingen and Potsdam benchmarks demonstrate that MMFNet achieves mean IoU scores of 83.50% and 86.06%, outperforming eight state-of-the-art methods while maintaining relatively low computational complexity. These results highlight MMFNet's potential for efficient and accurate multimodal semantic segmentation in remote sensing applications.

Accurate building extraction from very high-resolution (VHR) remote sensing images plays an important role in urban dynamic monitoring, planning, and management. However, it is still a challenging task to achieve building extraction with high accuracy and integrity due to diverse building appearances and more complex ground background in VHR remote sensing images. Recently, Unity Networking (UNet) has been proven to be capable of feature extraction and semantic segmentation of remote sensing images. However, UNet cannot achieve sufficient multi-scale and multi-level features with larger receptive fields. To address these problems, an improved network based on UNet structure (Refine-UNet) is proposed for extracting buildings from the VHR images. The proposed Refine-UNet mainly consists of encoder module, decoder module, and refine skip connection scheme. The refine skip connection scheme is composed of an atrous spatial convolutional pyramid pooling (ASPP) module and several improved depthwise separable convolution (IDSC) modules. Experimental results on the Jilin-1 VHR datasets with a spatial resolution of 0.75 m demonstrate that compared with UNet, PSPNet, DeepLabV3+, and SegNet, the proposed Refine-UNet can obtain more accurate building extraction results and achieve the best Precision of 95.1% and intersection over union (IoU) of 87.0%, respectively, indicating the great practical potential.

As the rapid development of earth observation technology and deep learning, building extraction from remotely sensed imagery based on deep convolutional neural networks (DCNNs) has attracted wide attention in recent years. However, due to the heterogeneity of building shapes and sizes and the complexity of the surrounding objects, current building extraction methods still have challenges in boundary accuracy and complete building extraction. For these purposes, we proposed low-level feature enhancement and multi-scale attention pyramid aggregation network (LFEMAP-Net) that considers building boundary information and multi-scale feature expression to obtain higher accuracy building extraction. Firstly, low-level feature enhancement model is proposed based on prior edge information to enhance the representation of spatial details, effectively addressing issues related to information loss and fuzzy boundaries. Additionally, a multi-scale attention pyramid aggregation model is developed during the decoding stage to facilitate the fusion of features from different scales, thereby enhancing the extraction of building features. Experimental results on two publicly available datasets validate that LFEMAP-Net can overcome building extraction interruptions and boundary blur in complex scenes, and achieve boundary optimization and complete segmentation of buildings and achieve better performance than other advanced semantic segmentation models.

The semantic segmentation of remote sensing images requires distinguishing local regions of different classes and exploiting a uniform global representation of the same-class instances. Such requirements make it necessary for the segmentation methods to extract discriminative local features between different classes and to explore representative features for all instances of a given class. While common deep convolutional neural networks (DCNNs) can effectively focus on local features, they are limited by their receptive field to obtain consistent global information. In this paper, we propose a memory-augmented transformer (MAT) to effectively model both the local and global information. The feature extraction pipeline of the MAT is split into a memory-based global relationship guidance module and a local feature extraction module. The local feature extraction module mainly consists of a transformer, which is used to extract features from the input images. The global relationship guidance module maintains a memory bank for the consistent encoding of the global information. Global guidance is performed by memory interaction. Bidirectional information flow between the global and local branches is conducted by a memory-query module, as well as a memory-update module, respectively. Experiment results on the ISPRS Potsdam and ISPRS Vaihingen datasets demonstrated that our method can perform competitively with state-of-the-art methods.

Land-use mapping (LUM) using high-spatial resolution remote sensing images (HSR-RSIs) is a challenging and crucial technology. However, due to the characteristics of HSR-RSIs, such as different image acquisition conditions and massive, detailed information, and performing LUM faces unique scientific challenges. With the emergence of new deep learning (DL) algorithms in recent years, methods to LUM with DL have achieved huge breakthroughs, which offer novel opportunities for the development of LUM for HSR-RSIs. This article aims to provide a thorough review of recent achievements in this field. Existing high spatial resolution datasets in the research of semantic segmentation and single-object segmentation are presented first. Next, we introduce several basic DL approaches that are frequently adopted for LUM. After reviewing DL-based LUM methods comprehensively, which highlights the contributions of researchers in the field of LUM for HSR-RSIs, we summarize these DL-based approaches based on two LUM criteria. Individually, the first one has supervised learning, semisupervised learning, or unsupervised learning, while another one is pixel-based or object-based. We then briefly review the fundamentals and the developments of the development of semantic segmentation and single-object segmentation. At last, quantitative results that experiment on the dataset of ISPRS Vaihingen and ISPRS Potsdam are given for several representative models such as fully convolutional network (FCN) and U-Net, following up with a comparison and discussion of the results.

In land use analysis, the extraction of buildings from remote sensing imagery is an important problem. This work is difficult to obtain the spectral features from buildings due to high intra-class and low inter-class variation of buildings. In the paper, a patch-based deep belief network (PBDBN) architecture is used for the extraction of buildings from remote sensing datasets. And low-level building features (e.g compacted contours) of adjacent regions are combined with Deep Belief Network (DBN) features during the post-processing stage for obtaining better performance. The experimental results are demonstrated on Massachusetts buildings dataset to express the performance of PBDBN and it is compared with other method on the same dataset.