A Comparative Review of U-Net-Based Enhanced Polyp Segmentation Models: From Architectural Evolution to Multimodal Fusion

Colorectal cancer is a common malignant tumor. Colorectal cancer is primarily caused by the cancerization of an adenomatous polyp. Segmentation of polyps in computer-assisted enteroscopy images is helpful for doctors to diagnose and treat the disease accurately. In this study, a segmentation algorithm of colonoscopy images based on multi-scale feature fusion is proposed. The proposed algorithm adopts ResNet50 as the backbone network to extract features. The shallow features are processed using the cross extraction module, thus increasing the receptive field, retaining the texture information, and fusing the processed shallow features and deep features at different proportions based on a multi-proportion fusion module. The proposed algorithm is capable of suppressing redundant information, removing background noise, and sharpening boundaries while acquiring considerable semantic information. As revealed by the results of the experiments on the published Kvasir-SEG dataset of intestinal polyps, the mean Dice coefficient and mean intersection over union were obtained as 0.9192 and 0.8873, better than that of existing mainstream algorithms. The result verifies the effectiveness of the proposed network and provides a reference for deep learning concerning the image processing and analysis of intestinal polyps.

Ensuring precise segmentation of colorectal polyps holds critical importance in the early diagnosis and treatment of colorectal cancer. Nevertheless, existing deep learning-based segmentation methods are fully supervised, requiring extensive, precise, manual pixel-level annotation data, which leads to high annotation costs. Additionally, it remains challenging to train large-scale segmentation models when confronted with limited colonoscopy data. To address these issues, we introduce the general segmentation foundation model—the Segment Anything Model (SAM)—into the field of medical image segmentation. Fine-tuning the foundation model is an effective approach to tackle sample scarcity. However, current SAM fine-tuning techniques still rely on precise annotations. To overcome this limitation, we propose WSPolyp-SAM, a novel weakly supervised approach for colonoscopy polyp segmentation. WSPolyp-SAM utilizes weak annotations to guide SAM in generating segmentation masks, which are then treated as pseudo-labels to guide the fine-tuning of SAM, thereby reducing the dependence on precise annotation data. To improve the reliability and accuracy of pseudo-labels, we have designed a series of enhancement strategies to improve the quality of pseudo-labels and mitigate the negative impact of low-quality pseudo-labels. Experimental results on five medical image datasets demonstrate that WSPolyp-SAM outperforms current fully supervised mainstream polyp segmentation networks on the Kvasir-SEG, ColonDB, CVC-300, and ETIS datasets. Furthermore, by using different amounts of training data in weakly supervised and fully supervised experiments, it is found that weakly supervised fine-tuning can save 70% to 73% of annotation time costs compared to fully supervised fine-tuning. This study provides a new perspective on the combination of weakly supervised learning and SAM models, significantly reducing annotation time and offering insights for further development in the field of colonoscopy polyp segmentation.

The fully convolutional networks (FCNs) have been widely applied in numerous medical image segmentation tasks. However, tissue regions usually have large variations of shape and scale, so the ability of neural networks to learn multi-scale features is important to the segmentation performance. In this paper, we improve the network for multi-scale feature fusion, in the medical image segmentation by introducing two feature fusion modules: i) global attention multi-scale feature fusion module (GMF); ii) local dense multi-scale feature fusion module (LMF). GMF aims to use global context information to guide the recalibration of low-level features from both spatial and channel aspects, so as to enhance the utilization of effective multi-scale features and suppress the noise of low-level features. LMF adopts bottom-up top-down structure to capture context information, to generate semantic features, and to fuse feature information at different scales. LMF can integrate local dense multi-scale context features layer by layer in the network, thus improving the ability of network to encode interdependent relationships among boundary pixels. Based on the above two modules, we propose a novel medical image segmentation framework (GLF-Net). We evaluated the proposed network and modules on challenging brain tumor segmentation and pancreas segmentation datasets, and very competitive performance has been achieved.

Accurate segmentation of colorectal polyps is essential for the early detection and prevention of colorectal cancer, as it directly influences the accuracy of risk assessment, treatment planning, and long-term patient outcomes. Federated learning (FL) provides a privacy-preserving approach to polyp segmentation. However, most existing methods rely on aggregation mechanisms that average the weights across clients, which fail to account for the non-independent and identically distributed (non-IID) data from different clients or the dynamic nature of clinical applications. This paper presents an Adaptive Weight Aggregation Federated Learning Framework (FedDW) for polyp segmentation tasks. Specifically, we first propose a medical value evaluation method based on fuzzy membership mapping, which achieves fair quantification of the contribution differences of different medical institutions through normalization of multidimensional indicators and the application of membership function mapping. Subsequently, we design a clinically-driven dual-path weight fusion mechanism based on real-time contribution membership degrees, which switches between client collaboration and lesion optimization strategies according to clinical needs, ensuring better adaptability to varying clinical scenarios. Finally, we develop a dynamic federated collaborative optimization strategy for global model aggregation, which fairly quantifies the clinical value of local model updates to dynamically generate server-side aggregation weights. It strengthens updates from high-quality clients while suppressing interference from low-quality clients, achieving coordination between the personalization of local models and the enhancement of global model performance. Experimental results demonstrate that the proposed framework significantly outperforms comparative methods in both segmentation accuracy and institutional consistency on polyp datasets, achieving a 2.19% higher Intersection over Union (IoU), a 2.04 mm lower Average Symmetric Surface Distance (ASSD), and a one-point reduction in the standard deviation of IoU compared to baseline approaches.

Polyps are very common abnormalities in human gastrointestinal regions. Their early diagnosis may help in reducing the risk of colorectal cancer. Vision-based computer-aided diagnostic systems automatically identify polyp regions to assist surgeons in their removal. Due to their varying shape, color, size, texture, and unclear boundaries, polyp segmentation in images is a challenging problem. Existing deep learning segmentation models mostly rely on convolutional neural networks that have certain limitations in learning the diversity in visual patterns at different spatial locations. Further, they fail to capture inter-feature dependencies. Vision transformer models have also been deployed for polyp segmentation due to their powerful global feature extraction capabilities. But they too are supplemented by convolution layers for learning contextual local information. In the present paper, a polyp segmentation model CoInNet is proposed with a novel feature extraction mechanism that leverages the strengths of convolution and involution operations and learns to highlight polyp regions in images by considering the relationship between different feature maps through a statistical feature attention unit. To further aid the network in learning polyp boundaries, an anomaly boundary approximation module is introduced that uses recursively fed feature fusion to refine segmentation results. It is indeed remarkable that even tiny-sized polyps with only 0.01% of an image area can be precisely segmented by CoInNet. It is crucial for clinical applications, as small polyps can be easily overlooked even in the manual examination due to the voluminous size of wireless capsule endoscopy videos. CoInNet outperforms thirteen state-of-the-art methods on five benchmark polyp segmentation datasets.

U-Net is a deep learning technique that has made significant contributions to medical image segmentation. Although the accomplishments of deep learning algorithms in terms of image processing are evident, many challenges still need to be overcome to achieve human-like performance. One of the main challenges in building deeper U-Nets is black-box problems, such as vanishing gradients. Overcoming this problem allows us to develop neural networks with advanced network designs. We propose three U-Net variants, namely efficient R2U-Net, efficient dense U-Net, and efficient fractal U-Net, that can create highly accurate segmentation maps. The first part of our contribution makes use of EfficientNet to distribute resources in the network efficiently. The second part of our work applies the following layer connections to design the U-Net decoders: residual connections, dense connections, and fractal expansion. We apply EfficientNet as the encoder to our three decoders to design three conceivable models. The aforementioned three proposed deep learning models were tested on four benchmark datasets, including the CHASE DB1 and digital retinal images for vessel extraction (DRIVE) retinal image databases and the ISIC 2018 and HAM10000 dermoscopy image databases. We obtained the highest Dice coefficient of 0.8013, 0.8808, 0.8019, and 0.9295 for CHASE DB1, ISIC 2018, DRIVE, and HAM10000, respectively, and a Jaccard (JAC) score of 0.6686, 0.7870, 0.6694, and 0.8683 for CHASE DB1, ISIC 2018, DRIVE, and HAM10000, respectively. Statistical analysis revealed that the proposed deep learning models achieved better segmentation results compared with the state-of-the-art models. U-Net is quite an adaptable deep learning framework and can be integrated with other deep learning techniques. The use of recurrent feedback connections, dense convolution, residual skip connections, and fractal convolutional expansions allow for the design of improved deeper U-Net models. With the addition of EfficientNet, we can now leverage the performance of an optimally scaled classifier for U-Net encoders.

Salient regions usually carry important information in images. Existing feature encryption algorithms aim at extracting edge features as significant information rather than salient regions for encryption purpose. Moreover, most of them protect significant information by transforming the input image into texture-like or noise-like encrypted image which is obviously a visual sign of encrypted image, and thus can be easily attacked. In this paper, we propose a salient regions encryption scheme to generate visually meaningful ciphertext. First, salient regions are efficiently extracted by a saliency detection model in the compressed domain. Then we pre-encrypt these salient regions by a chaos-based encryption algorithm. With optical encryption theory, the pre-encrypted salient regions are finally transformed into a visually meaningful ciphertext. To the best of our knowledge, it is the first time to use salient regions as important visual information for encryption to obtain cipertext in images. The experimental results demonstrate that the salient regions can be largely hidden with the proposed method.

In recent years, generative adversarial networks (GANs) have gained tremendous popularity for various imaging related tasks such as artificial image generation to support AI training. GANs are especially useful for medical imaging-related tasks where training datasets are usually limited in size and heavily imbalanced against the diseased class. We present a systematic review, following the PRISMA guidelines, of recent GAN architectures used for medical image analysis to help the readers in making an informed decision before employing GANs in developing medical image classification and segmentation models. We have extracted 54 papers that highlight the capabilities and application of GANs in medical imaging from January 2015 to August 2020 and inclusion criteria for meta-analysis. Our results show four main architectures of GAN that are used for segmentation or classification in medical imaging. We provide a comprehensive overview of recent trends in the application of GANs in clinical diagnosis through medical image segmentation and classification and ultimately share experiences for task-based GAN implementations.

The U-Net model for lung area segmentation in medical images is considered in this work. A preprocessed and enhanced dataset was used to train the U-Net model across 70 epochs using an encoder-decoder architecture with skip connections. To estimate the accuracy of the model, important performance indicators such as the Dice coefficient, Intersection over Union (IOU), and training/validation loss were observed. The coefficient for the dice went up from 0.5 to 0.9 in the results, while the IOU value calmed at 0.9, representing the model’s efficiency in proper segmentation. Strong generalization to previously unseen data and minimal overfitting were shown by the loss metrics’ in accordance reduction. In agreement to the study, U-Net has the potential to be used in real-world medical applications. Further study is recommended to enhance performance through the application of transfer learning and consideration devices.

Prevention of colorectal cancer (CRC) by inspecting and removing colorectal polyps has become a global health priority because CRC is one of the most frequent cancers in the world. Although recent U-Net-based convolutional neural networks (CNNs) with deep feature representation and skip connections have shown to segment polyps effectively, U-Net-based approaches still have limitations in modeling explicit global contexts, due to the intrinsic nature locality of convolutional operations. To overcome these problems, this study proposes a novel deep learning model, SwinE-Net, for polyp segmentation that effectively combines a CNN-based EfficientNet and Vision Transformer (ViT)-based Swin Ttransformer. The main challenge is to conduct accurate and robust medical segmentation in maintaining global semantics without sacrificing low-level features of CNNs through Swin Transformer. First, the multidilation convolutional block generates refined feature maps to enhance feature discriminability for multilevel feature maps extracted from CNN and ViT. Then, the multifeature aggregation block creates intermediate side outputs from the refined polyp features for efficient training. Finally, the attentive deconvolutional network-based decoder upsamples the refined and combined feature maps to accurately segment colorectal polyps. We compared the proposed approach with previous state-of-the-art methods by evaluating various metrics using five public datasets (Kvasir, ClinicDB, ColonDB, ETIS, and EndoScene). The comparative evaluation, in particular, proved that the proposed approach showed much better performance in the unseen dataset, which shows the generalization and scalability in conducting polyp segmentation. Furthermore, an ablation study was performed to prove the novelty and advantage of the proposed network. The proposed approach outperformed previous studies.

Medical image segmentation is the technique that helps doctor view and has a precise diagnosis, partic-ularly in Colorectal Cancer. Specifically, with the increase in cases, the diagnosis and identification need to be faster and more accurate for many patients; in endoscopic images, the segmentation task has been vital to helping the doctor identify the position of the polyps or the ache in the system correctly. As a result, many efforts have been made to apply deep learning to automate polyp segmentation, mostly to ameliorate the U -shape structure. However, the simple skip connection scheme in UNet leads to the deficient context information and the semantic gap between feature maps from the encoder and decoder. To deal with this problem, we propose a novel framework composed of ConvNeXt backbone and Multi Kernel Positional Embedding block. Thanks to the suggested module, our method can attain better accuracy and generalization in the polyps segmen-tation task. Extensive experiments show that our model achieves the Dice coefficient of 0.8818 and the IOU score of 0.8163 on the Kvasir-SEG dataset. Furthermore, on various datasets, we make competitive achievement results with other previous state-of-the-art methods.

Polyp segmentation is crucial in the diagnosis of colorectal cancer. The introduction of the Segment Any-thing Model (SAM) provides powerful pretraining capabilities for polyp segmentation, but it faces two main challenges when applied to endoscopic images: first, its Transformer- based architecture tends to overlook local details, leading to feature bias; second, its performance on out-of-distribution (OOD) data is suboptimal, affecting prediction accuracy and confidence estimation. To address these issues, we propose an enhanced approach based on SAM that incorporates a Local Feature Enhancement Module (LFEM) and a Channel Attention Enhancement Module (CAEM). LFEM improves the capture of high-frequency information by enhancing local detail features, playing a crucial role in extracting polyp edges and textures. CAEM introduces a channel attention mechanism that dynamically adjusts the weight of feature channels, thereby enhancing the model's sensitivity and generalization ability. Additionally, we draw on the design principles of the Cross-Branch Feature Enhancement module and the Uncertainty- Guided Prediction Regularization (UPR) module to further improve SAM's performance in multi-scale feature extraction and on out-of-distribution data. Experimental results demon-strate that the inclusion of LFEM and CAEM significantly improves the model's segmentation accuracy on multiple public polyp datasets, particularly excelling in complex background scenarios.

Detecting steel surface defects is crucial in industrial production, as it enhances product quality and safety while fostering technological advancements. However, this task remains challenging due to difficulties in edge feature extraction, multi-scale feature fusion, and the limited availability of training data. In this study, we tackle these challenges by proposing an advanced object detection network, SDT-Net. The proposed network incorporates an Edge Information Pyramid Generator (EIPG) module and an Adaptive Enhanced Fusion (AEF) module, effectively minimizing parameter redundancy while preserving rich edge information.
These modules enhance feature expressiveness, thereby significantly boosting overall network performance. To better capture multi-scale characteristics, we also design a CSP-Hierarchical Multi-scale Split Fusion module (CSP-HMSSF), enabling more effective feature fusion across resolutions and improving detection accuracy for defects of various sizes. Furthermore, we develop a Lightweight Efficient detection head(LWEHead) that strengthens the model’s detection capabilities while maintaining inference efficiency. Finally, we construct SDT-DET, a defect detection dataset comprising ten categories. Experimental evaluations demonstrate that our approach attains 75.1% mAP on the NEU-DET dataset and 66.4% mAP on the SDT-DET dataset. The experimental results confirm that SDT-Net is a highly effective steel surface defect detection model. It can greatly improve efficiency and reduce the cost of inspection and has great potential in industrial applications.

Segmentation of polyps based on pyramid vision transformers and residual block for real-time endoscopy imaging

Deep learning techniques have achieved significant advancements in medical image segmentation in recent years. However, model generalization remains severely constrained by domain shift, particularly in cross-modal medical image segmentation tasks. Traditional segmentation models struggle to generalize effectively to unseen target domains due to differences in joint probability distributions across medical imaging modalities. Existing methods primarily focus on unsupervised domain adaptation (UDA) and domain generalization (DG) techniques. While UDA methods face practical limitations due to difficulties in obtaining target domain data, current DG approaches often overlook inherent anatomical priors in medical images as well as the heterogeneity and sparsity of lesion regions. To address these challenges, this paper proposes a cross-modal medical image segmentation framework that integrates the Vision Mamba model with dynamic domain generalization. The framework achieves cross-domain feature alignment and multi-scale feature fusion by leveraging bidirectional state-space sequence modeling, Bezier curve-style enhancement, and a dual-normalization strategy. Additionally, the VEBlock module is introduced, which effectively combines the dynamic sequence modeling capabilities of the Mamba model with non-local attention mechanisms to better capture cross-modal global dependencies. Experimental results on the BraTS 2018 and cross-modal cardiac datasets demonstrate significant improvements in cross-modal segmentation. For example, in T2 → T1 tasks, our framework achieves an average Dice score of 56.22%, outperforming baseline methods by 1.78% while reducing Hausdorff distance for tumor boundaries to 13.26mm. Furthermore, in the Cardiac CT → MRI tasks, the Hausdorff distance is optimized to 27.34mm, validating the proposed framework's strong generalization capability for complex anatomical structures.