Mixed-UNet: Refined class activation mapping for weakly-supervised semantic segmentation with multi-scale inference

Deep learning-based thyroid ultrasound image segmentation is of great importance in clinical diagnosis. The Weakly Supervised Semantic Segmentation (WSSS) models requiring only Image-level Labels (IL) reduce the dependence on pixel-level labels. However, due to the lack of position of the objects in image-level labels, WSSS methods with IL are generally based on Class Activation Map (CAM) that can locate the objects. Nowadays, channel attention, which is one of the effective tools to extract object features from images, has been widely used in semantic segmentation tasks. Unfortunately, due to the large differences in the size and discriminative features of thyroid nodules, the mainstream channel attention is unable to perform flexible feature extraction for nodules, leading to the problem of under-segmentation or over-segmentation in the predicted results of the model. To overcome this issue, we propose a dynamic channel attention network that can extract features of thyroid nodules adaptively, called the CAM-guided Adaptive Attention Network (CAANet). In detail, CAANet can pay different attention to the overall and discriminative features of nodules based on the information provided by CAM. Finally, to verify the effectiveness of our method, we perform experimental comparisons with recent WSSS methods and the mainstream channel attention methods on the thyroid ultrasound image dataset. The evaluation results show that our method has better performance improvement, with a mean Intersection over Union (IoU) of 52.713%.

Recently, many excellent weakly supervised semantic segmentation (WSSS) works are proposed based on class activation mapping (CAM). However, there are few works that consider the characteristics of medical images. In this paper, we find that there are mainly two challenges of medical images in WSSS: i) the boundary of object foreground and background is not clear; ii) the co-occurrence phenomenon is very severe in training stage. We thus propose a Causal CAM (C-CAM) method to overcome the above challenges. Our method is motivated by two cause-effect chains including category-causality chain and anatomy-causality chain. The category-causality chain represents the image content (cause) affects the category (effect). The anatomy-causality chain represents the anatomical structure (cause) affects the organ segmentation (effect). Extensive experiments were conducted on three public medical image data sets. Our C-CAM generates the best pseudo masks with the DSC of 77.26%, 80.34% and 78.15% on ProMRI, ACDC and CHAOS compared with other CAM-like methods. The pseudo masks of C-CAM are further used to improve the segmentation performance for organ segmentation tasks. Our C-CAM achieves DSC of 83.83% on ProMRI and DSC of 87.54% on ACDC, which outperforms state-of-the-art WSSS methods. Our code is available at https://github.com/Tian-lab/C-CAM.

Weakly supervised semantic segmentation (WSSS) models relying on class activation maps (CAMs) have achieved desirable performance comparing to the non-CAMs-based counterparts. However, to guarantee WSSS task feasible, we need to generate pseudo labels by expanding the seeds from CAMs which is complex and time-consuming, thus hindering the design of efficient end-to-end (single-stage) WSSS approaches. To tackle the above dilemma, we resort to the off-the-shelf and readily accessible saliency maps for directly obtaining pseudo labels given the image-level class labels. Nevertheless, the salient regions may contain noisy labels and cannot seamlessly fit the target objects, and saliency maps can only be approximated as pseudo labels for simple images containing single-class objects. As such, the achieved segmentation model with these simple images cannot generalize well to the complex images containing multi-class objects. To this end, we propose an end-to-end multi-granularity denoising and bidirectional alignment (MDBA) model, to alleviate the noisy label and multi-class generalization issues. Specifically, we propose the online noise filtering and progressive noise detection modules to tackle image-level and pixel-level noise, respectively. Moreover, a bidirectional alignment mechanism is proposed to reduce the data distribution gap at both input and output space with simple-to-complex image synthesis and complex-to-simple adversarial learning. MDBA can reach the mIoU of 69.5% and 70.2% on validation and test sets for the PASCAL VOC 2012 dataset. The source codes and models have been made available at https://github.com/NUST-Machine-Intelligence-Laboratory/MDBA.

Existing weakly supervised semantic segmentation (WSSS) methods based on image‐level labels always rely on class activation maps (CAMs), which measure the relationships between features and classifiers. However, CAMs only focus on the most discriminative regions of images, resulting in their poor coverage performance. We attribute this to the deficiency in the recognition ability of a single classifier and the negative impacts caused by magnitudes during the CAMs normalisation process. To address the aforementioned issues, we propose to construct selective multiple classifiers (SMC). During the training process, we extract multiple prototypes for each class and store them in the corresponding memory bank. These prototypes are divided into foreground and background prototypes, with the former used to identify foreground objects and the latter aimed at preventing the false activation of background pixels. As for the inference stage, multiple prototypes are adaptively selected from the memory bank for each image as SMC. Subsequently, CAMs are generated by measuring the angle between SMC and features. We enhance the recognition ability of classifiers by adaptively constructing multiple classifiers for each image, while only relying on angle measurement to generate CAMs can alleviate the suppression phenomenon caused by magnitudes. Furthermore, SMC can be integrated into other WSSS approaches to help generate better CAMs. Extensive experiments conducted on standard WSSS benchmarks such as PASCAL VOC 2012 and MS COCO 2014 demonstrate the superiority of our proposed method.

Anomaly-guided weakly supervised lesion segmentation on retinal OCT images

Benefiting from free labeling pixel-level samples, weakly supervised semantic segmentation (WSSS) is making progress in automatically extracting building from high-resolution (HR) remote sensing (RS) imagery. For WSSS methods, generating high-quality pseudomasks is crucial for accurate building extraction.To improve the performance of generating pseudomasks by using image-level labels, this article proposes a weakly supervised building extraction method by combining adversarial climbing and gated convolution. The proposed method optimizes class activation maps (CAMs) by using adversarial climbing strategy, generates accurate class boundary maps by introducing a gated convolution module, and further refines building pseudomasks by fusing pairing semantic affinities and CAMs with a random walk strategy. Experimental results on three datasets—two ISPRS datasets and a self-annotated dataset—demonstrate that the proposed approach outperformed SOTA WSSS methods, leading to improvement of building extraction from HR RS imager. This article provides a new approach for optimizing pseudomasks generation, and a methodological reference for the applications of weakly supervised on RS images.

Weakly supervised semantic segmentation (WSSS) generally utilizes the Class Activation Map (CAM) to synthesize pseudo-labels. However, the current methods of obtaining CAM focus on salient features of a specific layer, resulting in highlighting the most discriminative regions and further leading to rough segmentation results for WSSS. In this paper, we rethink the potential of the ordinary classifier and find that if features of all the layers are applied, the classifier will obtain CAM with complete discriminative regions. Inspired by this, we propose Fully-CAM for WSSS, which can fully exploit the potential of the ordinary classifier and yield more accurate segmentation results. Precisely, Fully-CAM firstly weights feature with their corresponding gradients to yield CAMs of each layer, then fusing these layers’ CAMs could generate an ultimate CAM with complete discriminative regions. Furthermore, Fully-CAM is encapsulated into a plug-in, which can be mounted on any trained ordinary classifier with convolution layer, and it exceeds its previous performance without extra training.

DCAM: Disturbed class activation maps for weakly supervised semantic segmentation

In weakly supervised semantic segmentation (WSSS) tasks on remote sensing images, it is a common practice to train a classification network from scratch using a large batch of images with a limited number of classes. Subsequently, class activation maps are extracted from the model based on predefined class indices, and these maps are then optimized to obtain pseudolabels. To make this strategy effective when introducing a new class, a substantial amount of data needs to be provided to the model. In this article, we present an innovative framework, RS-TextWS-Seg, designed to efficiently generate high-quality segmentation results for a wide range of remote sensing objects using concise descriptions. Our proposed framework comprises three sequential stages: initially, we undertake parameter fine-tuning of the contrastive language-image pretraining (CLIP) model to swiftly strengthen its capacity for zero-shot detection of a limited number of remote sensing features. Subsequently, we introduce a text-driven background suppression mechanism aimed at deriving class activation maps from the refined CLIP model based on textual cues, while concurrently mitigating background noises. Finally, we use the segment anything model (SAM) to refine the edges of the extracted class activation map. We widely researched the leading-edge methodologies in WSSS and conducted a range of comparative experiments and ablation studies to prove the efficacy of our proposed framework. The research findings underscore that RS-TextWS-Seg outperforms other state-of-the-art methods on renowned datasets such as DLRSD and Potsdam, as well as on bespoke datasets specifically curated for overground petroleum pipelines and oil well fields.

Weakly supervised learning, releasing deep learning from highly labor-intensive pixel-wise annotations, has gained great attention, especially for medical image segmentation. With only image-level labels, pixel-wise segmentation/localization usually is achieved based on class activation maps (CAMs) containing the most discriminative regions. One common consequence of CAM-based approaches is incomplete foreground segmentation, i.e. under-segmentation/false negatives. Meanwhile, suffering from relatively limited medical imaging data, class-irrelevant tissues can hardly be suppressed during classification, resulting in incorrect background identification, i.e. over-segmentation/false positives. The above two issues are determined by the loose-constraint nature of image-level labels penalizing on the entire image space, and thus how to develop pixel-wise constraints based on image-level labels is the key for performance improvement which is under-explored. In this paper, based on unsupervised clustering, we propose a new paradigm called cluster-re-supervision to evaluate the contribution of each pixel in CAMs to final classification and thus generate pixel-wise supervision (i.e., clustering maps) for CAMs refinement on both over- and under-segmentation reduction. Furthermore, based on self-supervised learning, an inter-modality image reconstruction module, together with random masking, is designed to complement local information in feature learning which helps stabilize clustering. Experimental results on two popular public datasets demonstrate the superior performance of the proposed weakly-supervised framework for medical image segmentation. More importantly, cluster-re-supervision is independent of specific tasks and highly extendable to other applications.

Weakly supervised semantic segmentation (WSSS) faces challenges in generating reliable pseudo masks from image-level labels due to limited spatial supervision. While prototype-based contrastive learning (PCL) improves class activation maps (CAMs), existing methods struggle with background-foreground ambiguity and incomplete object activation. We propose EPCL, which addresses these issues through two key innovations: (1) a class-background(CB)contrast to better discriminate hard background semantics, and (2) an inter-class (IC) contrast sampling strategy that enhances object region activation. EPCL is architecture-agnostic and integrates into various WSSS baselines. Experiments on two segmentation benchmarks show EPCL outperforms existing methods across different WSSS settings.

Weakly Supervised Semantic Segmentation (WSSS) has received increasing attentions recently, which leverages only limited supervision to train the segmentation networks. Meanwhile, Transformer utilizes multi-head attention mechanism to overcome the limited receptive fields of convolution neural networks (CNN). As a result, a large amount of WSSS work has been built upon Transformer architecture. However, most of them overlook over-smoothing issue (i.e. the patch tokens in deep layers tend to be similar, indicating that the learned information is nearly identical), which will affect the accuracy of Class Activation Map (CAM) in representing the pixel level semantic information. In this paper, an Attention-based Layer Fusion scheme with Token Masking is proposed to address the above issue. The core modules include a Weighted Layer Attention Aggregation (WLA) module and a Random-Masked Class Token Refinement (RMTR) module. The former integrates the attention information of different layers in an adaptive way, which refines CAM to obtain a refined object contour. The latter introduces random masks to aggregate higher-level semantic information. Extensive experiments have been conducted on public datasets, in which our network manifests superior performances than other state-of-the-arts methods.

Image-level weakly supervised semantic segmentation (WSSS) reduces the dependence on high-quality data annotation, which plays a crucial role in computational pathology. Benefit from the ability to localize the objects with only binary labels, Class Activation Map (CAM) is a widely used method to initial pseudo masks. However, due to the low contrast among different tissues in histopathological images, most existing CAM-based methods perform poorly in gland segmentation. We retrospect this process and find that class consistency and semantic consistency can guide the network to effectively distinguish confusing pixels and generate fine-grained pseudo masks. Specifically, for class consistency, we propose Consistency Correlation Attention (CCA) to encourage the network to focus on the contribution of class features to semantic dependencies. For semantic consistency, we propose Multi-scale Pyramid Fusion Pooling (MPFP) to aggregate coarse-to-fine global semantic information from CAMs at multiple spatial resolutions, thus identifying class localization. Additionally, we introduce a Purified Labels Filtration (PLF) strategy during the segmentation phase to mitigate the noisy supervision signal and improve the segmentation quality of the model. Extensive experiments show that the our method achieves new state-of-the-art results on three publicly available gland datasets. Furthermore, our method demonstrates impressive domain adaptation capability, achieving satisfactory results with only a small portion of samples when faced with unseen domain data.

Precise tumor localization and sub-region identification are critical for disease diagnosis. However, current Weakly Supervised Semantic Segmentation (WSSS) methods for brain tumor segmentation are primarily single-class, neglecting differences between tumor sub-regions. We observed that when mainstream transformer-based WSSS methods are applied to multi-class brain tumor segmentation, they encounter two major challenges: sub-region discrimination errors and over-segmentation of small lesions. To address these challenges and advance multi-class WSSS methods for brain tumor analysis, this paper proposes Channel-gated Transformers with Affinity CAM (CTAC). CTAC first employs channel-gated multi-head self-attention to overcome the over-smoothing tendency of the transformer, thereby enhancing inter-class discriminability and improving the model's subclass differentiation capability. Then, CTAC uses multi-scale smoothed affinity to adaptively suppress low-confidence responses in the Class Activation Map (CAM), mitigating over-activation in the CAM, and alleviating the over-segmentation phenomena of small lesions. The proposed CTAC significantly outperformed the baseline method on the BraTS2021 glioma and BraTS2023-MEN meningioma datasets. On Brats2021, it achieved a multi-class mean IoU (mIoU) of 61.718%, an increase of 4.964 percentage points (pp), with the whole-tumor mIoU reaching 79.798% (+6.882 pp). On Brats2023-MEN, CTAC attained 72.887% mIoU (+4.676 pp) for multi-class segmentation and 75.394% (+7.839 pp) for whole-tumor. Furthermore, CTAC surpasses recent state-of-the-art methods. Code is available at https://github.com/yhan94-lab/CTAC.

The rapid development of medical practices and imaging technology tools creates substantial growth in the amount of medical image data each year in our present era. This research aims to develop a hybrid approach that integrates Machine Learning (ML) and Deep Learning (DL) techniques to enhance the accuracy and reliability of medical image classification for diagnostic purposes. Medical imaging data complexity and growing volume serve as the research motivation, which leads to an investigation of standalone ML or DL limitations and their combination into a single framework. The medical image processing starts with normalization, then noise reduction, and continues to grayscale conversion before performing histogram equalization. This research uses VGG16 and ResNet50 alongside MobileNet and InceptionV3 for feature extraction, then applies ten different ML algorithms, including SVM and MLP, and Ran-dom Forest, for classification. Five public medical image datasets from Kaggle are used: COVID-19 chest X-rays, melanoma skin lesions, pneumonia chest X-rays, acute stroke facial images, and various eye diseases. Hybrid models display superior performance compared to stand-alone ML or DL models based on accuracy, precision, recall, and F1-score evaluation measures. Multiple datasets demonstrate that the MobileNet+MLP combination de-livers the most accurate results, which demonstrates its reliable and efficient performance. The developed AI diagnostic tool presents a scalable system alongside accuracy and interpretability to enhance clinical decision outcomes.