Improved Pseudomasks Generation for Weakly Supervised Building Extraction From High-Resolution Remote Sensing Imagery

In weakly supervised semantic segmentation (WSSS) tasks on remote sensing images, it is a common practice to train a classification network from scratch using a large batch of images with a limited number of classes. Subsequently, class activation maps are extracted from the model based on predefined class indices, and these maps are then optimized to obtain pseudolabels. To make this strategy effective when introducing a new class, a substantial amount of data needs to be provided to the model. In this article, we present an innovative framework, RS-TextWS-Seg, designed to efficiently generate high-quality segmentation results for a wide range of remote sensing objects using concise descriptions. Our proposed framework comprises three sequential stages: initially, we undertake parameter fine-tuning of the contrastive language-image pretraining (CLIP) model to swiftly strengthen its capacity for zero-shot detection of a limited number of remote sensing features. Subsequently, we introduce a text-driven background suppression mechanism aimed at deriving class activation maps from the refined CLIP model based on textual cues, while concurrently mitigating background noises. Finally, we use the segment anything model (SAM) to refine the edges of the extracted class activation map. We widely researched the leading-edge methodologies in WSSS and conducted a range of comparative experiments and ablation studies to prove the efficacy of our proposed framework. The research findings underscore that RS-TextWS-Seg outperforms other state-of-the-art methods on renowned datasets such as DLRSD and Potsdam, as well as on bespoke datasets specifically curated for overground petroleum pipelines and oil well fields.

Deep learning-based thyroid ultrasound image segmentation is of great importance in clinical diagnosis. The Weakly Supervised Semantic Segmentation (WSSS) models requiring only Image-level Labels (IL) reduce the dependence on pixel-level labels. However, due to the lack of position of the objects in image-level labels, WSSS methods with IL are generally based on Class Activation Map (CAM) that can locate the objects. Nowadays, channel attention, which is one of the effective tools to extract object features from images, has been widely used in semantic segmentation tasks. Unfortunately, due to the large differences in the size and discriminative features of thyroid nodules, the mainstream channel attention is unable to perform flexible feature extraction for nodules, leading to the problem of under-segmentation or over-segmentation in the predicted results of the model. To overcome this issue, we propose a dynamic channel attention network that can extract features of thyroid nodules adaptively, called the CAM-guided Adaptive Attention Network (CAANet). In detail, CAANet can pay different attention to the overall and discriminative features of nodules based on the information provided by CAM. Finally, to verify the effectiveness of our method, we perform experimental comparisons with recent WSSS methods and the mainstream channel attention methods on the thyroid ultrasound image dataset. The evaluation results show that our method has better performance improvement, with a mean Intersection over Union (IoU) of 52.713%.

Existing weakly supervised semantic segmentation (WSSS) methods based on image‐level labels always rely on class activation maps (CAMs), which measure the relationships between features and classifiers. However, CAMs only focus on the most discriminative regions of images, resulting in their poor coverage performance. We attribute this to the deficiency in the recognition ability of a single classifier and the negative impacts caused by magnitudes during the CAMs normalisation process. To address the aforementioned issues, we propose to construct selective multiple classifiers (SMC). During the training process, we extract multiple prototypes for each class and store them in the corresponding memory bank. These prototypes are divided into foreground and background prototypes, with the former used to identify foreground objects and the latter aimed at preventing the false activation of background pixels. As for the inference stage, multiple prototypes are adaptively selected from the memory bank for each image as SMC. Subsequently, CAMs are generated by measuring the angle between SMC and features. We enhance the recognition ability of classifiers by adaptively constructing multiple classifiers for each image, while only relying on angle measurement to generate CAMs can alleviate the suppression phenomenon caused by magnitudes. Furthermore, SMC can be integrated into other WSSS approaches to help generate better CAMs. Extensive experiments conducted on standard WSSS benchmarks such as PASCAL VOC 2012 and MS COCO 2014 demonstrate the superiority of our proposed method.

Deep learning techniques have shown great potential in medical image processing, particularly through accurate and reliable image segmentation on magnetic resonance imaging (MRI) scans or computed tomography (CT) scans, which allow the localization and diagnosis of lesions. However, training these segmentation models requires a large number of manually annotated pixel-level labels, which are time-consuming and labor-intensive, in contrast to image-level labels that are easier to obtain. It is imperative to resolve this problem through weakly-supervised semantic segmentation models using image-level labels as supervision since it can significantly reduce human annotation efforts. Most of the advanced solutions exploit class activation mapping (CAM). However, the original CAMs rarely capture the precise boundaries of lesions. In this study, we propose the strategy of multi-scale inference to refine CAMs by reducing the detail loss in single-scale reasoning. For segmentation, we develop a novel model named Mixed-UNet, which has two parallel branches in the decoding phase. The results can be obtained after fusing the extracted features from two branches. We evaluate the designed Mixed-UNet against several prevalent deep learning-based segmentation approaches on our dataset collected from the local hospital and public datasets. The validation results demonstrate that our model surpasses available methods under the same supervision level in the segmentation of various lesions from brain imaging.

Weakly supervised semantic segmentation (WSSS) generally utilizes the Class Activation Map (CAM) to synthesize pseudo-labels. However, the current methods of obtaining CAM focus on salient features of a specific layer, resulting in highlighting the most discriminative regions and further leading to rough segmentation results for WSSS. In this paper, we rethink the potential of the ordinary classifier and find that if features of all the layers are applied, the classifier will obtain CAM with complete discriminative regions. Inspired by this, we propose Fully-CAM for WSSS, which can fully exploit the potential of the ordinary classifier and yield more accurate segmentation results. Precisely, Fully-CAM firstly weights feature with their corresponding gradients to yield CAMs of each layer, then fusing these layers’ CAMs could generate an ultimate CAM with complete discriminative regions. Furthermore, Fully-CAM is encapsulated into a plug-in, which can be mounted on any trained ordinary classifier with convolution layer, and it exceeds its previous performance without extra training.

DCAM: Disturbed class activation maps for weakly supervised semantic segmentation

Weakly supervised semantic segmentation (WSSS) models relying on class activation maps (CAMs) have achieved desirable performance comparing to the non-CAMs-based counterparts. However, to guarantee WSSS task feasible, we need to generate pseudo labels by expanding the seeds from CAMs which is complex and time-consuming, thus hindering the design of efficient end-to-end (single-stage) WSSS approaches. To tackle the above dilemma, we resort to the off-the-shelf and readily accessible saliency maps for directly obtaining pseudo labels given the image-level class labels. Nevertheless, the salient regions may contain noisy labels and cannot seamlessly fit the target objects, and saliency maps can only be approximated as pseudo labels for simple images containing single-class objects. As such, the achieved segmentation model with these simple images cannot generalize well to the complex images containing multi-class objects. To this end, we propose an end-to-end multi-granularity denoising and bidirectional alignment (MDBA) model, to alleviate the noisy label and multi-class generalization issues. Specifically, we propose the online noise filtering and progressive noise detection modules to tackle image-level and pixel-level noise, respectively. Moreover, a bidirectional alignment mechanism is proposed to reduce the data distribution gap at both input and output space with simple-to-complex image synthesis and complex-to-simple adversarial learning. MDBA can reach the mIoU of 69.5% and 70.2% on validation and test sets for the PASCAL VOC 2012 dataset. The source codes and models have been made available at https://github.com/NUST-Machine-Intelligence-Laboratory/MDBA.

Deep convolutional neural networks have promoted significant progress in building extraction from high-resolution remote sensing imagery. Although most of such work focuses on modifying existing image segmentation networks in computer vision, we propose a new network in this paper, Deep Encoding Network (DE-Net), that is designed for the very problem based on many lately introduced techniques in image segmentation. Four modules are used to construct DE-Net: the inception-style downsampling modules combining a striding convolution layer and a max-pooling layer, the encoding modules comprising six linear residual blocks with a scaled exponential linear unit (SELU) activation function, the compressing modules reducing the feature channels, and a densely upsampling module that enables the network to encode spatial information inside feature maps. Thus, DE-Net achieves state-of-the-art performance on the WHU Building Dataset in recall, F1-Score, and intersection over union (IoU) metrics without pre-training. It also outperformed several segmentation networks in our self-built Suzhou Satellite Building Dataset. The experimental results validate the effectiveness of DE-Net on building extraction from aerial imagery and satellite imagery. It also suggests that given enough training data, designing and training a network from scratch may excel fine-tuning models pre-trained on datasets unrelated to building extraction.

Weakly supervised semantic segmentation (WSSS) faces challenges in generating reliable pseudo masks from image-level labels due to limited spatial supervision. While prototype-based contrastive learning (PCL) improves class activation maps (CAMs), existing methods struggle with background-foreground ambiguity and incomplete object activation. We propose EPCL, which addresses these issues through two key innovations: (1) a class-background(CB)contrast to better discriminate hard background semantics, and (2) an inter-class (IC) contrast sampling strategy that enhances object region activation. EPCL is architecture-agnostic and integrates into various WSSS baselines. Experiments on two segmentation benchmarks show EPCL outperforms existing methods across different WSSS settings.

Greenhouses are densely distributed across the cultivated land in high-resolution remote sensing imagery, resulting in the problem of dense object extraction. On the one hand, objects tend to be wrongly merged into one object since objects are connected; on the other hand, the existed random sampling scheme does not make good use of the object distribution density to improve the training effect. To meet the demand for dense greenhouse extraction, this paper proposes a novel deep learning-based greenhouse extraction algorithm. To solve the problem that dense objects tend to be wrongly merged, this paper proposes a dual-task learning module, which uses the discriminant object boundary of the pixel-based branch to distinguish the objects of the object-based branch; to take advantage of the object distribution density for effective training, a high-density biased sampler is proposed. Moreover, this paper provides a dataset of manually labeled imagery to train and develop the proposed method. Results on greenhouse extraction in six regions in China achieve peak mIoU and mAP value, surpassing the state-of-the-art methods. Finally, a product of a greenhouse map in China is provided for analysis.

Recently, many excellent weakly supervised semantic segmentation (WSSS) works are proposed based on class activation mapping (CAM). However, there are few works that consider the characteristics of medical images. In this paper, we find that there are mainly two challenges of medical images in WSSS: i) the boundary of object foreground and background is not clear; ii) the co-occurrence phenomenon is very severe in training stage. We thus propose a Causal CAM (C-CAM) method to overcome the above challenges. Our method is motivated by two cause-effect chains including category-causality chain and anatomy-causality chain. The category-causality chain represents the image content (cause) affects the category (effect). The anatomy-causality chain represents the anatomical structure (cause) affects the organ segmentation (effect). Extensive experiments were conducted on three public medical image data sets. Our C-CAM generates the best pseudo masks with the DSC of 77.26%, 80.34% and 78.15% on ProMRI, ACDC and CHAOS compared with other CAM-like methods. The pseudo masks of C-CAM are further used to improve the segmentation performance for organ segmentation tasks. Our C-CAM achieves DSC of 83.83% on ProMRI and DSC of 87.54% on ACDC, which outperforms state-of-the-art WSSS methods. Our code is available at https://github.com/Tian-lab/C-CAM.

Image-level weakly supervised semantic segmentation (WSSS) reduces the dependence on high-quality data annotation, which plays a crucial role in computational pathology. Benefit from the ability to localize the objects with only binary labels, Class Activation Map (CAM) is a widely used method to initial pseudo masks. However, due to the low contrast among different tissues in histopathological images, most existing CAM-based methods perform poorly in gland segmentation. We retrospect this process and find that class consistency and semantic consistency can guide the network to effectively distinguish confusing pixels and generate fine-grained pseudo masks. Specifically, for class consistency, we propose Consistency Correlation Attention (CCA) to encourage the network to focus on the contribution of class features to semantic dependencies. For semantic consistency, we propose Multi-scale Pyramid Fusion Pooling (MPFP) to aggregate coarse-to-fine global semantic information from CAMs at multiple spatial resolutions, thus identifying class localization. Additionally, we introduce a Purified Labels Filtration (PLF) strategy during the segmentation phase to mitigate the noisy supervision signal and improve the segmentation quality of the model. Extensive experiments show that the our method achieves new state-of-the-art results on three publicly available gland datasets. Furthermore, our method demonstrates impressive domain adaptation capability, achieving satisfactory results with only a small portion of samples when faced with unseen domain data.

Precise tumor localization and sub-region identification are critical for disease diagnosis. However, current Weakly Supervised Semantic Segmentation (WSSS) methods for brain tumor segmentation are primarily single-class, neglecting differences between tumor sub-regions. We observed that when mainstream transformer-based WSSS methods are applied to multi-class brain tumor segmentation, they encounter two major challenges: sub-region discrimination errors and over-segmentation of small lesions. To address these challenges and advance multi-class WSSS methods for brain tumor analysis, this paper proposes Channel-gated Transformers with Affinity CAM (CTAC). CTAC first employs channel-gated multi-head self-attention to overcome the over-smoothing tendency of the transformer, thereby enhancing inter-class discriminability and improving the model's subclass differentiation capability. Then, CTAC uses multi-scale smoothed affinity to adaptively suppress low-confidence responses in the Class Activation Map (CAM), mitigating over-activation in the CAM, and alleviating the over-segmentation phenomena of small lesions. The proposed CTAC significantly outperformed the baseline method on the BraTS2021 glioma and BraTS2023-MEN meningioma datasets. On Brats2021, it achieved a multi-class mean IoU (mIoU) of 61.718%, an increase of 4.964 percentage points (pp), with the whole-tumor mIoU reaching 79.798% (+6.882 pp). On Brats2023-MEN, CTAC attained 72.887% mIoU (+4.676 pp) for multi-class segmentation and 75.394% (+7.839 pp) for whole-tumor. Furthermore, CTAC surpasses recent state-of-the-art methods. Code is available at https://github.com/yhan94-lab/CTAC.

Weakly supervised learning, releasing deep learning from highly labor-intensive pixel-wise annotations, has gained great attention, especially for medical image segmentation. With only image-level labels, pixel-wise segmentation/localization usually is achieved based on class activation maps (CAMs) containing the most discriminative regions. One common consequence of CAM-based approaches is incomplete foreground segmentation, i.e. under-segmentation/false negatives. Meanwhile, suffering from relatively limited medical imaging data, class-irrelevant tissues can hardly be suppressed during classification, resulting in incorrect background identification, i.e. over-segmentation/false positives. The above two issues are determined by the loose-constraint nature of image-level labels penalizing on the entire image space, and thus how to develop pixel-wise constraints based on image-level labels is the key for performance improvement which is under-explored. In this paper, based on unsupervised clustering, we propose a new paradigm called cluster-re-supervision to evaluate the contribution of each pixel in CAMs to final classification and thus generate pixel-wise supervision (i.e., clustering maps) for CAMs refinement on both over- and under-segmentation reduction. Furthermore, based on self-supervised learning, an inter-modality image reconstruction module, together with random masking, is designed to complement local information in feature learning which helps stabilize clustering. Experimental results on two popular public datasets demonstrate the superior performance of the proposed weakly-supervised framework for medical image segmentation. More importantly, cluster-re-supervision is independent of specific tasks and highly extendable to other applications.

Anomaly-guided weakly supervised lesion segmentation on retinal OCT images