Weakly Supervised Gland Segmentation with Class Semantic Consistency and Purified Labels Filtration

Existing weakly supervised semantic segmentation (WSSS) methods based on image‐level labels always rely on class activation maps (CAMs), which measure the relationships between features and classifiers. However, CAMs only focus on the most discriminative regions of images, resulting in their poor coverage performance. We attribute this to the deficiency in the recognition ability of a single classifier and the negative impacts caused by magnitudes during the CAMs normalisation process. To address the aforementioned issues, we propose to construct selective multiple classifiers (SMC). During the training process, we extract multiple prototypes for each class and store them in the corresponding memory bank. These prototypes are divided into foreground and background prototypes, with the former used to identify foreground objects and the latter aimed at preventing the false activation of background pixels. As for the inference stage, multiple prototypes are adaptively selected from the memory bank for each image as SMC. Subsequently, CAMs are generated by measuring the angle between SMC and features. We enhance the recognition ability of classifiers by adaptively constructing multiple classifiers for each image, while only relying on angle measurement to generate CAMs can alleviate the suppression phenomenon caused by magnitudes. Furthermore, SMC can be integrated into other WSSS approaches to help generate better CAMs. Extensive experiments conducted on standard WSSS benchmarks such as PASCAL VOC 2012 and MS COCO 2014 demonstrate the superiority of our proposed method.

Weakly supervised semantic segmentation (WSSS) generally utilizes the Class Activation Map (CAM) to synthesize pseudo-labels. However, the current methods of obtaining CAM focus on salient features of a specific layer, resulting in highlighting the most discriminative regions and further leading to rough segmentation results for WSSS. In this paper, we rethink the potential of the ordinary classifier and find that if features of all the layers are applied, the classifier will obtain CAM with complete discriminative regions. Inspired by this, we propose Fully-CAM for WSSS, which can fully exploit the potential of the ordinary classifier and yield more accurate segmentation results. Precisely, Fully-CAM firstly weights feature with their corresponding gradients to yield CAMs of each layer, then fusing these layers’ CAMs could generate an ultimate CAM with complete discriminative regions. Furthermore, Fully-CAM is encapsulated into a plug-in, which can be mounted on any trained ordinary classifier with convolution layer, and it exceeds its previous performance without extra training.

DCAM: Disturbed class activation maps for weakly supervised semantic segmentation

In weakly supervised semantic segmentation (WSSS) tasks on remote sensing images, it is a common practice to train a classification network from scratch using a large batch of images with a limited number of classes. Subsequently, class activation maps are extracted from the model based on predefined class indices, and these maps are then optimized to obtain pseudolabels. To make this strategy effective when introducing a new class, a substantial amount of data needs to be provided to the model. In this article, we present an innovative framework, RS-TextWS-Seg, designed to efficiently generate high-quality segmentation results for a wide range of remote sensing objects using concise descriptions. Our proposed framework comprises three sequential stages: initially, we undertake parameter fine-tuning of the contrastive language-image pretraining (CLIP) model to swiftly strengthen its capacity for zero-shot detection of a limited number of remote sensing features. Subsequently, we introduce a text-driven background suppression mechanism aimed at deriving class activation maps from the refined CLIP model based on textual cues, while concurrently mitigating background noises. Finally, we use the segment anything model (SAM) to refine the edges of the extracted class activation map. We widely researched the leading-edge methodologies in WSSS and conducted a range of comparative experiments and ablation studies to prove the efficacy of our proposed framework. The research findings underscore that RS-TextWS-Seg outperforms other state-of-the-art methods on renowned datasets such as DLRSD and Potsdam, as well as on bespoke datasets specifically curated for overground petroleum pipelines and oil well fields.

Weakly supervised semantic segmentation (WSSS) models relying on class activation maps (CAMs) have achieved desirable performance comparing to the non-CAMs-based counterparts. However, to guarantee WSSS task feasible, we need to generate pseudo labels by expanding the seeds from CAMs which is complex and time-consuming, thus hindering the design of efficient end-to-end (single-stage) WSSS approaches. To tackle the above dilemma, we resort to the off-the-shelf and readily accessible saliency maps for directly obtaining pseudo labels given the image-level class labels. Nevertheless, the salient regions may contain noisy labels and cannot seamlessly fit the target objects, and saliency maps can only be approximated as pseudo labels for simple images containing single-class objects. As such, the achieved segmentation model with these simple images cannot generalize well to the complex images containing multi-class objects. To this end, we propose an end-to-end multi-granularity denoising and bidirectional alignment (MDBA) model, to alleviate the noisy label and multi-class generalization issues. Specifically, we propose the online noise filtering and progressive noise detection modules to tackle image-level and pixel-level noise, respectively. Moreover, a bidirectional alignment mechanism is proposed to reduce the data distribution gap at both input and output space with simple-to-complex image synthesis and complex-to-simple adversarial learning. MDBA can reach the mIoU of 69.5% and 70.2% on validation and test sets for the PASCAL VOC 2012 dataset. The source codes and models have been made available at https://github.com/NUST-Machine-Intelligence-Laboratory/MDBA.

Weakly supervised semantic segmentation (WSSS) has employed Class Activation Maps (CAMs) to localize the objects. However, the CAMs typically do not fit along the object boundaries and highlight only the most-discriminative regions. To resolve the problems, we propose a Gated Pyramid Pooling (GPP) layer which is a substitute for a Global Average Pooling (GAP) layer, and an Adversarial Erasing Framework via Triplet (AEFT). In the GPP layer, a feature pyramid is obtained by pooling the CAMs at multiple spatial resolutions, and then be aggregated into an attention for class prediction by gated convolution. With the process, CAMs are trained not only to capture the global context but also to preserve fine-details from the image. Meanwhile, the AEFT targets an over-expansion, a chronic problem of Adversarial Erasing (AE). Although AE methods expand CAMs by erasing the discriminative regions, they usually suffer from the over-expansion due to an absence of guidelines on when to stop erasing. We experimentally verify that the over-expansion is due to rigid classification, and metric learning can be a flexible remedy for it. AEFT is devised to learn the concept of erasing with the triplet loss between the input image, erased image, and negatively sampled image. With the GPP and AEFT, we achieve new state-of-the-art both on the PASCAL VOC 2012 val/test and MS-COCO 2014 val set by 70.9%/71.7% and 44.8% in mIoU, respectively.KeywordWeakly supervised semantic segmentation

Recently, many excellent weakly supervised semantic segmentation (WSSS) works are proposed based on class activation mapping (CAM). However, there are few works that consider the characteristics of medical images. In this paper, we find that there are mainly two challenges of medical images in WSSS: i) the boundary of object foreground and background is not clear; ii) the co-occurrence phenomenon is very severe in training stage. We thus propose a Causal CAM (C-CAM) method to overcome the above challenges. Our method is motivated by two cause-effect chains including category-causality chain and anatomy-causality chain. The category-causality chain represents the image content (cause) affects the category (effect). The anatomy-causality chain represents the anatomical structure (cause) affects the organ segmentation (effect). Extensive experiments were conducted on three public medical image data sets. Our C-CAM generates the best pseudo masks with the DSC of 77.26%, 80.34% and 78.15% on ProMRI, ACDC and CHAOS compared with other CAM-like methods. The pseudo masks of C-CAM are further used to improve the segmentation performance for organ segmentation tasks. Our C-CAM achieves DSC of 83.83% on ProMRI and DSC of 87.54% on ACDC, which outperforms state-of-the-art WSSS methods. Our code is available at https://github.com/Tian-lab/C-CAM.

Weakly supervised semantic segmentation (WSSS) faces challenges in generating reliable pseudo masks from image-level labels due to limited spatial supervision. While prototype-based contrastive learning (PCL) improves class activation maps (CAMs), existing methods struggle with background-foreground ambiguity and incomplete object activation. We propose EPCL, which addresses these issues through two key innovations: (1) a class-background(CB)contrast to better discriminate hard background semantics, and (2) an inter-class (IC) contrast sampling strategy that enhances object region activation. EPCL is architecture-agnostic and integrates into various WSSS baselines. Experiments on two segmentation benchmarks show EPCL outperforms existing methods across different WSSS settings.

Deep learning-based thyroid ultrasound image segmentation is of great importance in clinical diagnosis. The Weakly Supervised Semantic Segmentation (WSSS) models requiring only Image-level Labels (IL) reduce the dependence on pixel-level labels. However, due to the lack of position of the objects in image-level labels, WSSS methods with IL are generally based on Class Activation Map (CAM) that can locate the objects. Nowadays, channel attention, which is one of the effective tools to extract object features from images, has been widely used in semantic segmentation tasks. Unfortunately, due to the large differences in the size and discriminative features of thyroid nodules, the mainstream channel attention is unable to perform flexible feature extraction for nodules, leading to the problem of under-segmentation or over-segmentation in the predicted results of the model. To overcome this issue, we propose a dynamic channel attention network that can extract features of thyroid nodules adaptively, called the CAM-guided Adaptive Attention Network (CAANet). In detail, CAANet can pay different attention to the overall and discriminative features of nodules based on the information provided by CAM. Finally, to verify the effectiveness of our method, we perform experimental comparisons with recent WSSS methods and the mainstream channel attention methods on the thyroid ultrasound image dataset. The evaluation results show that our method has better performance improvement, with a mean Intersection over Union (IoU) of 52.713%.

Benefiting from free labeling pixel-level samples, weakly supervised semantic segmentation (WSSS) is making progress in automatically extracting building from high-resolution (HR) remote sensing (RS) imagery. For WSSS methods, generating high-quality pseudomasks is crucial for accurate building extraction.To improve the performance of generating pseudomasks by using image-level labels, this article proposes a weakly supervised building extraction method by combining adversarial climbing and gated convolution. The proposed method optimizes class activation maps (CAMs) by using adversarial climbing strategy, generates accurate class boundary maps by introducing a gated convolution module, and further refines building pseudomasks by fusing pairing semantic affinities and CAMs with a random walk strategy. Experimental results on three datasets—two ISPRS datasets and a self-annotated dataset—demonstrate that the proposed approach outperformed SOTA WSSS methods, leading to improvement of building extraction from HR RS imager. This article provides a new approach for optimizing pseudomasks generation, and a methodological reference for the applications of weakly supervised on RS images.

Precise tumor localization and sub-region identification are critical for disease diagnosis. However, current Weakly Supervised Semantic Segmentation (WSSS) methods for brain tumor segmentation are primarily single-class, neglecting differences between tumor sub-regions. We observed that when mainstream transformer-based WSSS methods are applied to multi-class brain tumor segmentation, they encounter two major challenges: sub-region discrimination errors and over-segmentation of small lesions. To address these challenges and advance multi-class WSSS methods for brain tumor analysis, this paper proposes Channel-gated Transformers with Affinity CAM (CTAC). CTAC first employs channel-gated multi-head self-attention to overcome the over-smoothing tendency of the transformer, thereby enhancing inter-class discriminability and improving the model's subclass differentiation capability. Then, CTAC uses multi-scale smoothed affinity to adaptively suppress low-confidence responses in the Class Activation Map (CAM), mitigating over-activation in the CAM, and alleviating the over-segmentation phenomena of small lesions. The proposed CTAC significantly outperformed the baseline method on the BraTS2021 glioma and BraTS2023-MEN meningioma datasets. On Brats2021, it achieved a multi-class mean IoU (mIoU) of 61.718%, an increase of 4.964 percentage points (pp), with the whole-tumor mIoU reaching 79.798% (+6.882 pp). On Brats2023-MEN, CTAC attained 72.887% mIoU (+4.676 pp) for multi-class segmentation and 75.394% (+7.839 pp) for whole-tumor. Furthermore, CTAC surpasses recent state-of-the-art methods. Code is available at https://github.com/yhan94-lab/CTAC.

Deep learning techniques have shown great potential in medical image processing, particularly through accurate and reliable image segmentation on magnetic resonance imaging (MRI) scans or computed tomography (CT) scans, which allow the localization and diagnosis of lesions. However, training these segmentation models requires a large number of manually annotated pixel-level labels, which are time-consuming and labor-intensive, in contrast to image-level labels that are easier to obtain. It is imperative to resolve this problem through weakly-supervised semantic segmentation models using image-level labels as supervision since it can significantly reduce human annotation efforts. Most of the advanced solutions exploit class activation mapping (CAM). However, the original CAMs rarely capture the precise boundaries of lesions. In this study, we propose the strategy of multi-scale inference to refine CAMs by reducing the detail loss in single-scale reasoning. For segmentation, we develop a novel model named Mixed-UNet, which has two parallel branches in the decoding phase. The results can be obtained after fusing the extracted features from two branches. We evaluate the designed Mixed-UNet against several prevalent deep learning-based segmentation approaches on our dataset collected from the local hospital and public datasets. The validation results demonstrate that our model surpasses available methods under the same supervision level in the segmentation of various lesions from brain imaging.

PurposeIn surgical computer vision applications, data privacy and expert annotation challenges impede the acquisition of labeled training data. Unpaired image-to-image translation techniques have been explored to automatically generate annotated datasets by translating synthetic images into a realistic domain. The preservation of structure and semantic consistency, i.e., per-class distribution during translation, poses a significant challenge, particularly in cases of semantic distributional mismatch.MethodThis study empirically investigates various translation methods for generating data in surgical applications, explicitly focusing on semantic consistency. Through our analysis, we introduce a novel and simple combination of effective approaches, which we call ConStructS. The defined losses within this approach operate on multiple image patches and spatial resolutions during translation.ResultsVarious state-of-the-art models were extensively evaluated on two challenging surgical datasets. With two different evaluation schemes, the semantic consistency and the usefulness of the translated images on downstream semantic segmentation tasks were evaluated. The results demonstrate the effectiveness of the ConStructS method in minimizing semantic distortion, with images generated by this model showing superior utility for downstream training.ConclusionIn this study, we tackle semantic inconsistency in unpaired image translation for surgical applications with minimal labeled data. The simple model (ConStructS) enhances consistency during translation and serves as a practical way of generating fully labeled and semantically consistent datasets at minimal cost. Our code is available at https://gitlab.com/nct_tso_public/constructs.

The costly process of obtaining semantic segmentation labels has driven research towards weakly supervised semantic segmentation (WSSS) methods, using only image-level, point, or box labels. Such annotations introduce limitations and challenges that results in overly-tuned methods specialized in specific domains or scene types. The over reliance of image-level based methods on generation of high quality class activation maps (CAMs) results in limited applicable dataset complexity range, mostly focusing on object centric scenes. Additionally, the lack of dense annotations requires methods to increase network complexity to obtain additional semantic information, often done through multiple stages of training and refinement. Here, we present a single-stage approach generalizable to a wide range of dataset complexities, that is trainable from scratch, without any dependency on pre-trained backbones, classification, or separate refinement tasks. We utilize point annotations to generate reliable, on-the-fly pseudo-masks through refined and spatially filtered features. We are to demonstrate SOTA performance on benchmark datasets (PascalVOC 2012), as well as significantly outperform other SOTA WSSS methods on recent real-world datasets (CRAID, CityPersons, IAD, ADE20K, CityScapes) with up to 28.1% and 22.6% performance boosts compared to our single-stage and multi-stage baselines respectively.

Weakly Supervised Semantic Segmentation (WSSS) has received increasing attentions recently, which leverages only limited supervision to train the segmentation networks. Meanwhile, Transformer utilizes multi-head attention mechanism to overcome the limited receptive fields of convolution neural networks (CNN). As a result, a large amount of WSSS work has been built upon Transformer architecture. However, most of them overlook over-smoothing issue (i.e. the patch tokens in deep layers tend to be similar, indicating that the learned information is nearly identical), which will affect the accuracy of Class Activation Map (CAM) in representing the pixel level semantic information. In this paper, an Attention-based Layer Fusion scheme with Token Masking is proposed to address the above issue. The core modules include a Weighted Layer Attention Aggregation (WLA) module and a Random-Masked Class Token Refinement (RMTR) module. The former integrates the attention information of different layers in an adaptive way, which refines CAM to obtain a refined object contour. The latter introduces random masks to aggregate higher-level semantic information. Extensive experiments have been conducted on public datasets, in which our network manifests superior performances than other state-of-the-arts methods.