HL-SAM-Seg: Complementary High- and Low-Resolution Features Based on SAM for Remote Sensing Image Semantic Segmentation

Accurate semantic segmentation of high-resolution remote sensing imagery is challenged by substantial intra-class variability, inter-class similarity, and the limitations of single-modality data. This paper proposes MMFNet, a novel multimodal fusion network that leverages the Mamba architecture to efficiently capture long-range dependencies for semantic segmentation tasks. MMFNet adopts a dual-encoder design, combining ResNet-18 for local detail extraction and VMamba for global contextual modelling, striking a balance between segmentation accuracy and computational efficiency. A Multimodal Feature Fusion Block (MFFB) is introduced to effectively integrate complementary information from optical imagery and digital surface models (DSMs), thereby enhancing multimodal feature interaction and improving segmentation accuracy. Furthermore, a frequency-aware upsampling module (FreqFusion) is incorporated in the decoder to enhance boundary delineation and recover fine spatial details. Extensive experiments on the ISPRS Vaihingen and Potsdam benchmarks demonstrate that MMFNet achieves mean IoU scores of 83.50% and 86.06%, outperforming eight state-of-the-art methods while maintaining relatively low computational complexity. These results highlight MMFNet's potential for efficient and accurate multimodal semantic segmentation in remote sensing applications.

In the context of fast progress in deep learning, convolutional neural networks (CNNs) have been extensively applied to the semantic segmentation of remote sensing images and have achieved significant progress. However, certain limitations exist in capturing global contextual information due to the characteristics of convolutional local properties. Recently, Transformer has become a focus of research in computer vision and has shown great potential in extracting global contextual information, further promoting the development of semantic segmentation tasks. In this paper, we use ResNet50 as an encoder, embed the hybrid attention mechanism into Transformer, and propose a Transformer-based decoder. The Channel-Spatial Transformer Block (CSTB) further aggregates features by integrating the local feature maps extracted by the encoder with their associated global dependencies. At the same time, an adaptive approach is employed to reweight the interdependent channel maps to enhance the feature fusion. The Global Cross-Fusion Module (GCFM) combines the extracted complementary features to obtain more comprehensive semantic information. Extensive comparative experiments were conducted on the ISPRS Potsdam and Vaihingen datasets, where mIoU reached 78.06% and 76.37%, respectively. The outcomes of multiple ablation experiments also validate the effectiveness of the proposed method.

In the field of remote sensing, using a large amount of labeled image data to supervise the training of fully convolutional networks for the semantic segmentation of images is expensive. However, using a small amount of labeled data can lead to reduced network performance. This paper proposes an unsupervised semantic segmentation method that combines the ImSE-Net model with SLICm superpixel optimization. First, the ImSE-Net model is used to extract semantic features from the image to obtain rough semantic segmentation results. Then, the SLICm superpixel segmentation algorithm is used to segment the input image into superpixel images. Finally, an unsupervised semantic segmentation model (UGLS) is used to combine high-level abstract semantic features with detailed information on superpixels to obtain edge-optimized semantic segmentation results. Experimental results show that compared with other semantic segmentation algorithms, our method more effectively handles unbalanced areas, such as object boundaries, and achieves better segmentation results, with higher semantic consistency.

Chapter 12 - Semantic scene segmentation for robotics

For semantic segmentation of high-resolution remote-sensing images, digital surface models (DSMs) information is useful for improving the accuracy and robustness of the segmentation models. However, since the feature distributions of spectral and DSM images vary significantly in different scenes, it is difficult to fuse them effectively in popular deep network models. To solve this issue, we propose an attention-based DSM fusion network (ADF-Net) for high-resolution remote-sensing image semantic segmentation. The proposed network makes two contributions. The first is that we design an attention-based feature fusion module, which can selectively gather features from spectral and DSM information by channel attention mechanism, and further combine them to get high-quality fusion features. The second is that we introduce a residual feature refinement module to reduce the redundant information from skip connection adaptively. We evaluate the proposed network on the ISPRS Vaihingen and Potsdam datasets, experimental results demonstrate that our model outperforms state-of-the-art methods.KeywordsHigh-resolution remote-sensing imagesAttention mechanismSemantic segmentationData fusion

The use of fully supervised deep learning methods to extract buildings from remote sensing images has shown excellent performance, which requires large amounts of training data with laborious per-pixel labeling. Compared with pixelated intensive labeling, it is much easier to label data using scribbles, which only takes few seconds for one image. In this paper, a one-stage structure-aware weakly supervised network (SAWSN) for building extraction is proposed, and it learns from easily accessible scribbles rather than from densely annotated ground truth. Firstly, to solve the problem that direct training with scribble labels will lead to poor building structures, an auxiliary edge detection task is introduced to localize building edges explicitly. Secondly, a structure aware scribble extension module (SASEM) is designed to recover building structures from scribbles through effective utilization of edge features. Finally, an edge-structure-aware loss is proposed to limit the scope of the restored structure. We perform extensive experiments on three newly labeled benchmark building extraction datasets (WHU, ISPRS Potsdam, and Vaihingen). Experimental results show that our method achieved 91.72%, 92.83%, and 92.22% of F1 using the ISPRS Vaihingen, Potsdam, and WHU datasets, respectively, and outperformed the state-of-the-art scribble-based weakly supervised methods by 3.27% of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IoU</i> .

With the increasing availability of multimodal remote sensing (RS) data, semantic segmentation that leverages complementary information from true orthophotos (TOP) and digital surface models (DSM) has become essential for urban analysis. Diffusion-based segmentation provides an effective iterative refinement mechanism for modeling complex multimodal distributions; however, conventional pixel-wise supervision emphasizes local accuracy while overlooking global distribution alignment, often leading to inconsistent predictions and blurred object boundaries. Although maximum mean discrepancy (MMD) measures global statistical differences between predicted and ground-truth distributions, its effectiveness in high-dimensional class-probability spaces is limited by directional cancellation effects that reduce sensitivity to complex distribution shifts. To address this issue, we propose a projection-kernel regularized diffusion-based multimodal RS segmentation framework that enforces global statistical alignment through distribution-level regularization rather than modifying the intrinsic diffusion process. The proposed regularization performs multi-directional projections of high-dimensional class-probability vectors onto one-dimensional subspaces and derives a closed-form kernel integration to avoid numerical sampling across projection directions, enabling efficient and stable global distribution matching. In addition, a Cross-Attention Dual-Encoder Fusion (CADEF) module is introduced to alleviate geometry-texture misalignment, and a Hierarchical EMA-Gated Recursive Denoising (HERD) mechanism is designed to stabilize multiscale feature refinement. Experiments on the ISPRS Vaihingen and Potsdam benchmarks demonstrate that the proposed regularization consistently improves segmentation accuracy over state-of-the-art CNN-, Transformer-, and diffusion-based baselines, yielding enhanced global consistency and sharper boundary delineation. Code is available at: https://github.com/tonyy127/PKDiff.

Semantic segmentation of high-resolution remote sensing images remains challenging due to the degradation of high-frequency semantic cues during convolutional encoding and the lack of frequency consistency in multi-stage feature fusion. To address these issues, we propose FreDNet, a frequency-guided denoising network that explicitly enhances frequency-sensitive representations throughout the segmentation process. Specifically, we introduce the Dual-path Residual Block (DRB), which incorporates a Frequency-aware Denoising Module (FDM) and a Frequency-aware Fusion Module (FFM) to suppress frequency-domain noise while preserving edge structures. Furthermore, we design a Frequency-aware Cross-level Fusion Module (FCFM) that leverages frequency intensity response maps to adaptively fuse encoder and decoder features. These components work collaboratively to enhance the frequency robustness and spatial consistency of the segmentation predictions. Extensive experiments on three challenging benchmarks, ISPRS Vaihingen, ISPRS Potsdam, and LoveDA, demonstrate that FreDNet achieves superior performance, surpassing the latest state-of-the-art approaches by up to 0.8% in mean IoU and 0.9% in overall accuracy, while maintaining a lightweight inference cost. In addition, ablation study confirms the contribution of each component of FreDNet.

Semantic segmentation has been a fundamental task in interpreting remote sensing imagery (RSI) for various downstream applications. Due to the high intra-class variants and inter-class similarities, inflexibly transferring natural image-specific networks to RSI is inadvisable. To enhance the distinguishability of learnt representations, attention modules were developed and applied to RSI, resulting in satisfactory improvements. However, these designs capture contextual information by equally handling all the pixels regardless of whether they around edges. Therefore, blurry boundaries are generated, rising high uncertainties in classifying vast adjacent pixels. Hereby, we propose an edge distribution attention module (EDA) to highlight the edge distributions of leant feature maps in a self-attentive fashion. In this module, we first formulate and model column-wise and row-wise edge attention maps based on covariance matrix analysis. Furthermore, a hybrid attention module (HAM) that emphasizes the edge distributions and position-wise dependencies is devised combing with non-local block. Consequently, a conceptually end-to-end neural network, termed as EDENet, is proposed to integrate HAM hierarchically for the detailed strengthening of multi-level representations. EDENet implicitly learns representative and discriminative features, providing available and reasonable cues for dense prediction. The experimental results evaluated on ISPRS Vaihingen, Potsdam and DeepGlobe datasets show the efficacy and superiority to the state-of-the-art methods on overall accuracy (OA) and mean intersection over union (mIoU). In addition, the ablation study further validates the effects of EDA.

Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clustering methods that assign a pixel to an instance center. In this work, we present a fully convolution neural network that learns instance segmentation from semantic segmentation and instance contours (boundaries of things). Instance contours along with semantic segmentation yield a boundary aware semantic segmentation of things. Connected component labeling on these results produces instance segmentation. We merge semantic and instance segmentation results to output panoptic segmentation. We evaluate our proposed method on the CityScapes dataset to demonstrate qualitative and quantitative performances along with several ablation studies. Our overview video can be accessed from https://youtu.be/wBtcxRhG3e0.

Semantic segmentation is a fundamental task in remote sensing image understanding. Recently, Deep Convolutional Neural Networks (DCNNs) have considerably improved the performance of the semantic segmentation of natural scenes. However, it is still challenging for very high-resolution remote sensing images. Due to the large and complex scenes as well as the influence of illumination and imaging angle, it is particularly difficult for the existing methods to accurately obtain the category of pixels at object boundaries—the so-called boundary blur. We propose a framework called Boundary-Aware Semi-Supervised Semantic Segmentation Network (BAS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> Net), which obtains more accurate segmentation results without additional annotation workload, especially at the object boundaries. The Channel-weighted Multi-scale Feature (CMF) module balances semantic and spatial information, and the Boundary Attention Module (BAM) weights the features with rich semantic boundary information to alleviate the boundary blur. Additionally, to decrease the amount of difficult and tedious manual labeling of remote sensing images, a discriminator network infers pseudolabels from unlabeled images to assist semisupervised learning, and further improves the performance of the segmentation network. To validate the effectiveness of the proposed framework, extensive experiments have been performed on both the ISPRS Vaihingen dataset, and the novel remote sensing dataset AIR-SEG with more categories, and complex boundaries. The results demonstrate a significant improvement of accuracy especially on boundaries and for small objects.

Aiming at the problem that the semantic segmentation algorithm has poor segmentation results in large depth of field (DOF) images, this paper proposed the concept of depth IoU (dIoU) evaluation standard. This concept based on the effective generalized IoU-loss (GIoU-loss) and the instance-level IoU (iIoU) concept proposed by Cityscapes dataset in the field of target detection. This paper introduced the image depth information into the loss function in the semantic segmentation algorithm. By using dIoU as a evaluation standard, the detection effect of the distant object in the DOF picture will get more weight. It solves the problem of weight reduction caused by the far-reaching effect of the distant object in the traditional evaluation standard.

This paper aims to address the issue of traditional art patterns not fully leveraging contextual information in image semantic segmentation. It proposes a multi-level feature aggregation method for image semantic segmentation based on visual Transformer and coordinate attention adjustment (CA Adjustment). Initially, the input image is segmented into slices for linear projection, incorporating learnable positional embeddings to derive an encoded input sequence. Subsequently, employing a visual Transformer-based encoder, the image is transformed into patches to capture global context across the network. CA Adjustment is then introduced to adaptively merge fusion features and visual backbone features using weighted adjustments. Lastly, a multi-objective grasshopper optimization algorithm is designed based on a clustering evolution mechanism to further enhance algorithm performance. Extensive experimental results demonstrate that our proposed algorithm effectively models global contextual information for image feature extraction. Compared to existing advanced algorithms, our method achieves high segmentation accuracy in semantic segmentation tasks, consistently exceeding 95%. Ablation experiments further validate the efficacy of our approach, underscoring its broad applicability in high-precision image semantic segmentation.

An image semantic segmentation algorithm using fully convolutional network (FCN) integrated with the recently proposed simple linear iterative clustering (SLIC) that is based on boundary term (BSLIC) is developed. To improve the segmentation accuracy, the developed algorithm combines the FCN semantic segmentation results with the superpixel information acquired by BSLIC. During the combination process, the superpixel semantic annotation is newly introduced and realized by the four criteria. The four criteria are used to annotate a superpixel region, according to FCN semantic segmentation result. The developed algorithm can not only accurately identify the semantic information of the target in the image, but also achieve a high accuracy in the positioning of small edges. The effectiveness of our algorithm is evaluated on the dataset PASCAL VOC 2012. Experimental results show that the developed algorithm improved the target segmentation accuracy in comparison with the traditional FCN model. With the BSLIC superpixel information that is involved, the proposed algorithm can get 3.86%, 1.41%, and 1.28% improvement in pixel accuracy (PA) over FCN-32s, FCN-16s, and FCN-8s, respectively.

<p indent=0mm>An important part of autonomous driving is the perception of the driving environment of the car, which has created a strong demand for high precision semantic segmentation algorithms that can be run in real time on low-power mobile devices. However, when analyzing the factors that affect the accuracy and speed of the semantic segmentation network, it can be found that in the structure of the previous semantic segmentation algorithm, spatial information and context features are difficult to take into account at the same time, and using two networks to obtain spatial information and context information separately will increase the amount of calculation and storage. Therefore, a new structure is proposed that divides the spatial path and context path from the network based on the residual structure, and a two-path real-time semantic segmentation network is designed based on this structure. The network contains a feature fusion module and an attention refinement module, which are used to realize the function of fusing the multi-scale features of two paths and optimizing the output results of context path. The network is based on the PyTorch framework and uses NVIDIA 1080Ti graphics cards for experiments. On the road scene data set Cityscapes, mIoU reached 78.8%, and the running speed reached 27.5 fps.