An Union Method Combining the Stitching of Normal Images and the Unsupervised Semantic Segmentation of Stitched Image

In the field of remote sensing, using a large amount of labeled image data to supervise the training of fully convolutional networks for the semantic segmentation of images is expensive. However, using a small amount of labeled data can lead to reduced network performance. This paper proposes an unsupervised semantic segmentation method that combines the ImSE-Net model with SLICm superpixel optimization. First, the ImSE-Net model is used to extract semantic features from the image to obtain rough semantic segmentation results. Then, the SLICm superpixel segmentation algorithm is used to segment the input image into superpixel images. Finally, an unsupervised semantic segmentation model (UGLS) is used to combine high-level abstract semantic features with detailed information on superpixels to obtain edge-optimized semantic segmentation results. Experimental results show that compared with other semantic segmentation algorithms, our method more effectively handles unbalanced areas, such as object boundaries, and achieves better segmentation results, with higher semantic consistency.

Generating pixel-level annotations for semantic segmentation tasks of high-resolution remote sensing images is both time-consuming and labor-intensive, which has led to increased interest in unsupervised methods. Therefore, in this paper, we propose an unsupervised multi-scale hybrid feature extraction network based on the CNN-Transformer architecture, referred to as MSHFE-Net. The MSHFE-Net consists of three main modules: a Multi-Scale Pixel-Guided CNN Encoder, a Multi-Scale Aggregation Transformer Encoder, and a Parallel Attention Fusion Module. The Multi-Scale Pixel-Guided CNN Encoder is designed for multi-scale, fine-grained feature extraction in unsupervised tasks, efficiently recovering local spatial information in images. Meanwhile, the Multi-Scale Aggregation Transformer Encoder introduces a multi-scale aggregation module, which further enhances the unsupervised acquisition of multi-scale contextual information, obtaining global features with stronger feature representation. The Parallel Attention Fusion Module employs an attention mechanism to fuse global and local features in both channel and spatial dimensions in parallel, enriching the semantic relations extracted during unsupervised training and improving the performance of unsupervised semantic segmentation. K-means clustering is then performed on the fused features to achieve high-precision unsupervised semantic segmentation. Experiments with MSHFE-Net on the Potsdam and Vaihingen datasets demonstrate its effectiveness in significantly improving the accuracy of unsupervised semantic segmentation.

Convolutional Neural Network (CNN) is widely used for semantic segmentation and land-use and land-cover (LULC) mapping of very high-resolution (VHR) remote sensing images. The convolution operation is a powerful method for VHR classification, but the loss of high-frequency detail information caused during its operation decreases the classification accuracy, particularly in the boundary. Thus, it is necessary to supply additional boundary information to the CNN for alleviating this situation. In the classification task (and in LULC mapping), providing more effective information generates a better classification result. Current methods regard the boundary of images as the same category object and process it uniformly, which loses a notable amount of useful information because of the different properties, such as ambiguity and transition, between remote sensing images and their boundaries. Thus, a semantic segmentation method with category boundary for LULC mapping is proposed in this paper. First, a multi-task CNN called the category boundary detection network (CBDN) is designed to extract the boundary information of different category objects. Second, this category boundary and VHR images are used for initial semantic segmentation. Finally, the category boundary and the initial semantic segmentation result (ISSR) are fused to obtain the final LULC map by a two-step strategy, including the explicit fusion and the boundary attention loss function. To verify whether category boundary improved the classification accuracy, a set of comparative experiments were conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets. The method in this paper was compared with a semantic segmentation method with no boundary information and a semantic segmentation method with global boundary. The results showed that the proposed method in this paper achieved good performances in the Vaihingen (overall accuracy (OA) = 0.924, Kappa coefficient (K) = 0.898, mean F1 score (mF1) = 0.896 and mean Intersection over Union (mIoU) = 0.817) and Potsdam datasets (OA = 0.890, K = 0.857, mF1 = 0.923, and mIoU = 0.860) based on the eroded labels.

Unsupervised semantic segmentation (USS) aims to discover and recognize meaningful categories without any labels. For a successful USS, two key abilities are required: 1) information compression and 2) clustering capability. Previous methods have relied on feature dimension reduction for information compression, however, this approach may hinder the process of clustering. In this paper, we propose a novel USS framework called Expand-and-Quantize Unsupervised Semantic Segmentation (EQUSS), which combines the benefits of high-dimensional spaces for better clustering and product quantization for effective information compression. Our extensive experiments demonstrate that EQUSS achieves state-of-the-art results on three standard benchmarks. In addition, we analyze the entropy of USS features, which is the first step towards understanding USS from the perspective of information theory.

Efficient training of supervised deep learning models for semantic segmentation requires a massive volume of annotated data. In this paper, we propose an unsupervised semantic segmentation method through the application of a radial transform method in the polar coordinate system to unannotated images. This method generates radial transformed images up to the number of pixels in the input image. Each generated image corresponds to a pixel in the original image, which is a spatial representation of the selected pixel with respect to other pixels, in the polar coordinate system. A dimension reduction method, such as a convolutional autoencoder (CAE), can extract features of these representations for clustering and later, labeling the original image by mapping the labels back to the original image. The advantage of the proposed radial transform technique is that it generates a massive number of training images by sampling pixels in the polar coordinate system from a very limited number of original images in the Cartesian coordinate system. The proposed approach achieved 88.20% accuracy in pixel-level segmentation of left kidney, right kidney, and non-kidney pixels in contrast-enhanced computed tomography (CT) images.

Semantic segmentation in driving scenarios is important for modern autonomous driving technology. While the existing methods have shown promising results in segmenting normal-condition images, their performance in adverse scenes remains unsatisfactory due to limited visual field and lack of annotation. To address this issue, we propose an unsupervised domain adaptation semantic segmentation method with the transformer architecture, namely ACSegFormer, for driving-scene adverse conditions, aiming at mining image features in visually restricted scenes. Three effective training strategies are proposed in ACSegFormer to learn the latent image context relations and to reduce the gaps between different domains: an entropy-based pseudo label correction scheme that refines the target domain predictions with the normal reference predictions, an optimal transport-based inter-domain alignment module that performs domain alignment on the outputs of transformer encoder, and a masked context learning module that enhances the model’s ability to perceive the missing information of target domain image. Our ACSegFormer has no additional training parameters on top of the existing transformer segmentation framework, which can be easily used for self-training-based unsupervised domain adaptation approaches. The experimental results show that our ACSegFormer achieves state-of-the-art performance on driving-scene segmentation benchmarks in adverse conditions, including Dark Zurich and ACDC. Codes and models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/wenyyu/ACSegFormer</uri>.

Unsupervised brain tumor segmentation using a symmetric-driven adversarial network

Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method.

The proposed new method constructs a contrastive learning task through unsupervised image-to-image translation and indirectly extracts domain-invariant features by maximizing mutual information. Specifically, by reusing the semantic segmentation network for image-to-image translation, the probability space of the segmentation output is projected into the RGB space, thereby constructing positive and negative samples to maximize the mutual information between the input image and the semantic segmentation RGB projection map. Meanwhile, the output space of the target domain is transferred to the source domain, and then the robust domain-invariant semantic features are extracted. We develop two adversarial transfer methods and a three-stage training paradigm of pre-training, cross-domain transfer, and self-supervised training. Experiments on benchmarks demonstrate that this method is reasonable and feasible. Comprehensive ablation studies and analyses are also carried out to reveal the advantages and disadvantages of the two designed transfer schemes and the effect and significance of color-invariant semantic information for unsupervised semantic segmentation tasks. The contrastive learning task, which is constructed through unsupervised cross-domain image-to-image translation, provides a new insight for cross-domain transfer learning to overcome the problem of domain shift.KeywordsDomain shiftUnsupervised semantic segmentationUnsupervised image-to-image translationContrastive learning

Methods and datasets on semantic segmentation for Unmanned Aerial Vehicle remote sensing images: A review

A novel unsupervised multi-spectral multiple-segmenter texture segmentation method with unknown number of classes is presented. The unsupervised segmenter is based on a combination of several unsupervised segmentation results, each in different resolution, using the sum rule. Multi-spectral texture mosaics are locally represented by four causal multi-spectral random field models recursively evaluated for each pixel. The single-resolution segmentation part of the algorithm is based on the underlying Gaussian mixture model and starts with an over segmented initial estimation which is adaptively modified until the optimal number of homogeneous texture segments is reached. The performance of the presented method is extensively tested on the Prague segmentation benchmark using the commonest segmentation criteria and compares favourably with several alternative texture segmentation methods.

Remote sensing images contain a wealth of geospatial information. To accurately identify different geospatial categories and extract relevant data, image semantic segmentation plays a crucial role. In recent years, deep learning technology has brought significant breakthroughs to semantic segmentation of remote sensing images, significantly enhancing its performance. This paper investigates the application of deep learning technologies in remote sensing image semantic segmentation, based on Convolutional Neural Networks (CNN) and Transformer-based semantic segmentation methods. It conducts an in-depth comparison of their structural characteristics and applicable scenarios, summarizes the achievements and shortcomings of existing research, and provides technical references and theoretical support for future studies, thereby contributing to the further development of deep learning technology in the field of remote sensing. Research results indicate that CNN-based semantic segmentation methods still hold advantages in extracting local features and achieving efficient segmentation, whereas Transformers address CNN's limitations in global context modeling and long-range dependency capture. Therefore, the collaborative integration of CNN and Transformers will become an important research direction for enhancing model performance in the future.

In recent years, the convolutional neural network (CNN) has made remarkable achievements in semantic segmentation. The method of semantic segmentation has a desirable application prospect. Nowadays, the methods mostly use an encoder-decoder architecture as a way of generating pixel by pixel segmentation prediction. The encoder is for extracting feature maps and decoder for recovering feature map resolution. An improved semantic segmentation method on the basis of the encoder-decoder architecture is proposed. We can get better segmentation accuracy on several hard classes and reduce the computational complexity significantly. This is possible by modifying the backbone and some refining techniques. Finally, after some processing, the framework has achieved good performance in many datasets. In comparison with the traditional architecture, our architecture does not need additional decoding layer and further reuses the encoder weight, thus reducing the complete quantity of parameters needed for processing. In this paper, a modified focal loss function is also put forward, as a replacement for the cross-entropy function to achieve a better treatment of the imbalance problem of the training data. In addition, more context information is added to the decode module as a way of improving the segmentation results. Experiments prove that the presented method can get better segmentation results. As an integral part of a smart city, multimedia information plays an important role. Semantic segmentation is an important basic technology for building a smart city.

<p indent="0mm">Pedestrians and other moving small-targets plays an important role in road scene point cloud. However, they perform poorly for real-time semantic segmentation. Therefore, a small-target semantic segmentation method based on RangeNet++ and optimization of loss function is proposed. Firstly, the Focal Loss function is used to improve the original RangeNet++ which uses cross entropy as loss function. Then, the weight of some rare, important moving small-targets in Focal Loss function is adjusted. The proposed method can improve the detection and segmentation accuracy of important moving small-targets categories such as pedestrians with fewer training sessions under the DarkNet21 which has fewer convolutional layers. Experiments on the SemanticKITTI dataset show that compared with the original RangeNet++, the improved method uses fewer convolutional layers backbone and fewer training sessions, but achieves higher accuracy and precision in semantic segmentation of moving small-targets.

Remote sensing images are usually characterized by complex backgrounds, scale and orientation variations, and large intraclass variance. General semantic segmentation methods usually fail to fully investigate the above issues, and thus their performances on remote sensing image segmentation are limited. In this article, we propose our LOGCAN++, a semantic segmentation model customized for remote sensing images, which is made up of a global class-aware (GCA) module and several local class-aware (LCA) modules. The GCA module captures global representations for class-level context modeling to reduce the interference of background noise. The LCA module generates local class representations as intermediate perceptual elements to indirectly associate pixels with the global class representations, targeting dealing with the large intraclass variance problem. In particular, we introduce affine transformations in the LCA module for adaptive extraction of local class representations to effectively tolerate scale and orientation variations in remote sensing images. Extensive experiments on three benchmark datasets show that our LOGCAN++ outperforms current mainstream general and remote sensing semantic segmentation methods and achieves a better trade-off between speed and accuracy.