A Hybrid Medical Image Semantic Segmentation Network Based on Novel Mamba and Transformer

Accurate lesion segmentation is critical for cancer diagnosis and treatment. Convolutional neural networks (CNNs) are widely used for medical image segmentation but struggle to capture long-range dependencies. Transformers mitigate this limitation but come with high computational costs. Mamba, a state-space model (SSM), efficiently models long-range dependencies but lacks precision in fine details. To address these challenges, this study aimed to develop a novel segmentation approach that combines the strengths of CNNs, Transformers, and Mamba, enhancing both global context understanding and local feature extraction in medical image segmentation. We propose AttmNet, a U-shaped network designed for medical image segmentation, which incorporates a novel structure called MAM (Multiscale-Convolution, Self-Attention, and Mamba). The MAM block integrates multi-layer convolution for multi-scale feature learning with an Att-Mamba component that combines self-attention and Mamba to effectively capture global context while preserving fine details. We evaluated AttmNet on four public datasets for breast, skin, and lung lesion segmentation. AttmNet outperformed state-of-the-art methods in terms of intersection over union (IoU) and Dice similarity coefficients. On the breast ultrasound (BUS) dataset, AttmNet achieved a 3.38% improvement in IoU and a 4.54% increase in Dice over the next best method. On the breast ultrasound images (BUSI) dataset, AttmNet's IoU and Dice coefficients were 1.17% and 3.21% higher than the closest competitor, respectively. In the PH2 Dermoscopy Image dataset, AttmNet surpassed the next best model by 0.25% in both IoU and Dice. On the larger coronavirus disease 2019 (COVID-19) Lung dataset, AttmNet maintained strong performance, achieving higher IoU and Dice scores than the next best models, SegMamba and TransUNet. AttmNet is a powerful and efficient tool for medical image segmentation, addressing the limitations of existing methods through its advanced design. The MAM block significantly enhances segmentation accuracy while maintaining computational efficiency, making AttmNet highly suitable for clinical applications. The code is available at https://github.com/hyb2840/AttmNet.

In recent years, multilayer perceptron (MLP)-based networks have gained significant attention in the field of medical image segmentation. However, achieving high accuracy in breast ultrasound image segmentation remains challenging due to the difficulty in capturing sufficient target features. Shallow networks have proven insufficient for addressing the complexities involved in ultrasound image segmentation tasks. We propose a deep MLP-based network for ultrasound breast segmentation that incorporates boundary awareness. The architecture of the network comprises seven layers, with the initial three layers utilizing convolutional neural network (CNN), while the subsequent four layers are based on tokenized MLP(Tok-MLP). The CNN is utilized for extracting shallow features, while the Tok-MLP is employed to capture deeper features. Additionally, breast ultrasound images are often influenced by various factors that result in blurred tissue boundaries, which complicates the segmentation process. To address this issue, we introduce the Boundary Aware Module (BAM), designed to enhance boundary detection through two distinct strategies. This module is integrated with the original up-sampled feature maps to ensure that boundary features receive equal attention as internal features. Experimental results on two breast ultrasound datasets demonstrate that the proposed DCMNet significantly outperforms six state-of-the-art methods, as evidenced by superior Dice and Intersection over Union (IoU) scores. Specifically, the BUSI dataset achieved IoU and Dice scores of 67.11% and 80.01%, respectively.

Breast UltraSound (BUS) imaging is a commonly used diagnostic tool in the field of counter fighting breast diseases, especially for early detection and diagnosis of breast cancer. Due to the inherent characteristics of ultrasound images such as blurry boundaries and diverse tumor morphologies, it is challenging for doctors to manually segment breast tumors. In recent years, the Convolutional Neural Network (CNN) technology has been widely applied to automatically segment BUS images. However, due to the inherent limitations of CNNs in capturing global contextual information, it is difficult to capture the full context. To address this issue, the paper proposes a novel BGRD-TransUNet model for breast lesion segmentation, based on TransUNet. The proposed model, first, replaces the original ResNet50 backbone network of TransUNet with DenseNet121 for initial feature extraction. Next, newly designed Residual Multi-Scale Feature Modules (RMSFMs) are employed to extract features from various layers of DenseNet121, thus capturing richer features within specific layers. Thirdly, a Boundary Guidance (BG) network is added to enhance the contour information of BUS images. Additionally, newly designed Boundary Attentional Feature Fusion Modules (BAFFMs) are used to integrate edge information and features extracted through RMSFMs. Finally, newly designed Parallel Channel and Spatial Attention Modules (PCSAMs) are used to refine feature extraction using channel and spatial attention. An extensive experimental testing performed on two public datasets demonstrates that the proposed BGRD-TransUNet model outperforms all state-of-the-art medical image segmentation models, participating in the experiments, according to all evaluation metrics used (except for few separate cases), including the two most important and widely used metrics in the field of medical image segmentation, namely the Intersection over Union (IoU) and Dice Similarity Coefficient (DSC). More specifically, on the BUSI dataset and dataset B, BGRD-TransUNet achieves IoU values of 76.77% and 86.61%, and DSC values of 85.08% and 92.47%, respectively, which are higher by 7.27 and 3.64, and 5.81 and 2.54 percentage points, than the corresponding values achieved by the baseline (TransUNet).

Medical image segmentation is crucial for improving healthcare outcomes. Convolutional neural networks (CNNs) have been widely applied in medical image analysis; however, their inherent inductive biases limit their ability to capture global contextual information. Vision transformer (ViT) architectures address this limitation by leveraging attention mechanisms to model global relationships; however, they typically require large-scale datasets for effective training, which is challenging in the field of medical imaging due to limited data availability. This study aimed to integrate the advantages of CNN and ViT architectures to improve segmentation performance on small-scale medical image datasets. In this study, we established a U-shaped network architecture based on a Transformer-assisted convolutional neural network (TAC-UNet). The TAC-UNet is primarily composed of a hybrid structure integrating CNN and Transformer components. Specifically, the hybrid architecture follows a dual-path design in which the Transformer branch continuously conveys global contextual information to the CNN backbone. This allows the CNN backbone to enhance its global perception while building on the local features it extracts, thereby improving its ability to comprehend complex image structures. A channel cross-attention (CCA) module is also incorporated as a bridge between the encoder and decoder to better reconcile the semantic discrepancies between them. Detailed experiments on three public datasets were conducted. Specifically, our model was trained on 30 images from the Multi-organ Nucleus Segmentation (MoNuSeg) training dataset, 85 images from the Gland Segmentation (GlaS) training dataset, and 551 images from the Computer Vision Center Colorectal Cancer-Clinic Database (CVC-ClinicDB) dataset. We evaluated the performance of our model on the corresponding test sets. Our TAC-UNet achieved the best Dice scores (80.36%, 90.70%, and 91.81% on the MoNuSeg, GlaS, and CVC-ClinicDB datasets, respectively) of all the models. Compared to other CNN-based, Transformer-based, and hybrid methods, the TAC-UNet demonstrated significantly superior segmentation performance. Our TAC-UNet model showed advanced segmentation performance on small-scale medical image datasets. The detailed experimental results showed the effectiveness of the method. Our model's code is available at: https://github.com/hejlhello/TAC-UNet.

Due to the low contrast, blurry boundaries, and large amount of shadows in breast ultrasound (BUS) images, automatic tumor segmentation remains a challenging task. Deep learning provides a solution to this problem, since it can effectively extract representative features from lesions and the background in BUS images. A novel automatic tumor segmentation method is proposed by combining a dilated fully convolutional network (DFCN) with a phase-based active contour (PBAC) model. The DFCN is an improved fully convolutional neural network with dilated convolution in deeper layers, fewer parameters, and batch normalization techniques; and has a large receptive field that can separate tumors from background. The predictions made by the DFCN are relatively rough due to blurry boundaries and variations in tumor sizes; thus, the PBAC model, which adds both region-based and phase-based energy functions, is applied to further improve segmentation results. The DFCN model is trained and tested in dataset 1 which contains 570BUS images from 89 patients. In dataset 2, a 10-fold support vector machine (SVM) classifier is employed to verify the diagnostic ability using 460 features extracted from the segmentation results of the proposed method. Advantages of the present method were compared with three state-of-the-art networks; the FCN-8s, U-net, and dilated residual network (DRN). Experimental results from 170BUS images show that the proposed method had a Dice Similarity coefficient of 88.97±10.01%, a Hausdorff distance (HD) of 35.54±29.70 pixels, and a mean absolute deviation (MAD) of 7.67±6.67 pixels, which showed the best segmentation performance. In dataset 2, the area under curve (AUC) of the 10-fold SVM classifier was 0.795 which is similar to the classification using the manual segmentation results. The proposed automatic method may be sufficiently accurate, robust, and efficient for medical ultrasound applications.

Rheumatoid arthritis (RA) is a chronic autoimmune disease that can cause severe joint damage and functional impairment. Ultrasound imaging has shown promise in providing real-time assessment of synovium inflammation associated with the early stages of RA. Accurate segmentation of the synovium region and quantification of inflammation-specific imaging biomarkers are crucial for assessing and grading RA. However, automatic segmentation of the synovium in 3D ultrasound is challenging due to ambiguous boundaries, variability in synovium shape, and inhomogeneous intensity distribution. In this work, we introduce a novel network architecture, Swin Transformers with Deep Attentive Features for 3D segmentation (SwinDAF3D), which integrates Swin Transformers into a Deep Attentive Features framework. The developed architecture leverages the hierarchical structure and shifted windows of Swin Transformers to capture rich, multi-scale and attentive contextual information, improving the modeling of long-range dependencies and spatial hierarchies in 3D ultrasound images. In a six-fold cross-validation study with 3D ultrasound images of RA patients' finger joints (n = 72), our SwinDAF3D model achieved the highest performance with a Dice Score (DSC) of 0.838 ± 0.013, an Intersection over Union (IoU) of 0.719 ± 0.019, and Surface Dice Score (SDSC) of 0.852 ± 0.020, compared to 3D UNet (DSC: 0.742 ± 0.025; IoU: 0.589 ± 0.031; SDSC: 0.661 ± 0.029), DAF3D (DSC: 0.813 ± 0.017; IoU: 0.689 ± 0.022; SDSC: 0.817 ± 0.013), Swin UNETR (DSC: 0.808 ± 0.025; IoU: 0.678 ± 0.032; SDSC: 0.822 ± 0.039), UNETR++ (DSC: 0.810 ± 0.014; IoU: 0.684 ± 0.018; SDSC: 0.829 ± 0.027) and TransUNet (DSC: 0.818 ± 0.013; IoU: 0.692 ± 0.017; SDSC: 0.815 ± 0.016) models. This ablation study demonstrates the effectiveness of combining a Swin Transformers feature pyramid with a deep attention mechanism, improving the segmentation accuracy of the synovium in 3D ultrasound. This advancement shows great promise in enabling more efficient and standardized RA screening using ultrasound imaging.

U-net-based segmentation of foreign bodies and ghost images in panoramic radiographs.

While convolutional neural networks (CNNs) and vision transformers (ViTs) have advanced medical image segmentation, they face inherent limitations such as local receptive fields in CNNs and high computational complexity in ViTs. This paper introduces Deconver, a novel network that integrates traditional deconvolution techniques from image restoration as a core learnable component within a U -shaped architecture. Deconver replaces computationally expensive attention mechanisms with efficient nonnegative deconvolution (NDC) operations, enabling the restoration of high-frequency details while suppressing artifacts. Key innovations include a backpropagation-friendly NDC layer based on a provably monotonic update rule and a parameter-efficient design. Evaluated across five datasets (ISLES'22, Spleen, BraTS'23, GlaS, and FIVES) covering both 2D and 3D segmentation tasks, Deconver achieves state-of-the-art performance in Dice scores and Hausdorff distance while reducing computational costs (FLOPs) by up to 90% compared to leading baselines. By bridging traditional image restoration with deep learning, this work offers a practical solution for high-precision segmentation in resource-constrained clinical workflows.

Real time 2D phase contrast (RTPC) MRI is useful for flow quantification in atrial fibrillation (AF) patients, but data analysis requires time-consuming anatomical contouring for many cardiac time frames. Our goal was to develop a convolutional neural network (CNN) for fully automated left atrial (LA) flow quantification. Forty-four AF patients underwent cardiac MRI including LA RTPC, collecting a median of 358 timeframes per scan. 15,307 semi-manual derived RTPC LA contours comprised ground truth for CNN training, validation, and testing. CNN vs. human performance was assessed using Dice scores (DSC), Hausdorff distance (HD), and flow measures (stasis, velocities, flow). LA contour DSC across all patients were similar to human inter-observer DSC (0.90 vs. 0.93) and a median 4.6 mm [3.5–5.9 mm] HD. There was no impact of heart rate variability on contouring quality (low vs. high variability DSC: 0.92 ± 0.05 vs. 0.91 ± 0.03, p = 0.95). CNN based LA flow quantification showed good to excellent agreement with semi-manual analysis (r > 0.90) and small bias in Bland-Altman analysis for mean velocity (-0.10 cm/s), stasis (1%), and net flow (-2.4 ml/s). This study demonstrated the feasibility of CNN based LA flow analysis with good agreements in LA contours and flow measures and resilience to heartbeat variability in AF.

Automatic and accurate segmentation of medical images plays an essential role in disease diagnosis and treatment planning. Convolution neural networks have achieved remarkable results in medical image segmentation in the past decade. Meanwhile, deep learning models based on Transformer architecture also succeeded tremendously in this domain. However, due to the ambiguity of the medical image boundary and the high complexity of physical organization structures, implementing effective structure extraction and accurate segmentation remains a problem requiring a solution. In this paper, we propose a novel Dual Encoder Network named DECTNet to alleviate this problem. Specifically, the DECTNet embraces four components, which are a convolution-based encoder, a Transformer-based encoder, a feature fusion decoder, and a deep supervision module. The convolutional structure encoder can extract fine spatial contextual details in images. Meanwhile, the Transformer structure encoder is designed using a hierarchical Swin Transformer architecture to model global contextual information. The novel feature fusion decoder integrates the multi-scale representation from two encoders and selects features that focus on segmentation tasks by channel attention mechanism. Further, a deep supervision module is used to accelerate the convergence of the proposed method. Extensive experiments demonstrate that, compared to the other seven models, the proposed method achieves state-of-the-art results on four segmentation tasks: skin lesion segmentation, polyp segmentation, Covid-19 lesion segmentation, and MRI cardiac segmentation.

This study was aimed to explore the adoption value of magnetic resonance imaging (MRI) under convolutional neural networks (CNN) in the therapeutic effect of nasopharyngeal carcinoma (NPC) radiotherapy. A total of 54 NPC patients were recruited. CNN was employed to perform 3D visualization processing on magnetic resonance (MR) images of NPC patients. MRI changes were analyzed before and after the patient received radiotherapy. The image segmentation and radiotherapy effects of CNN were evaluated by the Recall, intersection over union (IOU), postoperative apparent diffusion coefficient (ADC), and diagnostic coincidence rate. Moreover, gradient vector flow (GVF) algorithm, fuzzy c-means (FCM), and SegNet were adopted for comparative evaluation. Recall of CNN was 94.89% and the IOU was 84.16%, which was remarkably different from other algorithms (P < 0.05). After analysis of the MRI images of patients receiving radiotherapy, ADC of local residual patients was 1.108 ± 0.097 measured by CNN, the ADC was 1.826 ± 0.115, and the missed diagnosis rate was only 7.14%. In summary, CNN had a good effect on the localization and segmentation of NPC patients, and can accurately evaluate the effect of patients receiving radiotherapy, which can assist clinical diagnosis and treatment of NPC.

EH-former: Regional easy-hard-aware transformer for breast lesion segmentation in ultrasound images

Objective.Ultrasound shear wave imaging enables noninvasive, quantitative assessment of tissue pathology with mechanical elasticity measurements. However, shear wave elastography (SWE) reconstructions are challenged by noise sensitivity, inefficient multi-push strategies for scalable region of interest coverage, and limited annotated data, leading to suboptimal reconstruction and unreliable inclusion segmentation.Approach.In this work, we present a novel two-stage deep learning framework that addresses these limitations through a convolutional neural network (CNN)-based multi-nested-LSTM reconstruction network followed by a compound-loss-driven CNN-denoiser. The reconstruction stage begins with a ResNet3D-encoder that extracts spatiotemporal features from sequential multi-push acoustic radiation force data. These features are temporally windowed with Nested CNN-LSTM, converted from 3D to 2D with temporal attention module, and enhanced by fast Fourier transform-based frequency attention. The resulting 2D maps are subsequently decoded into primary 2D elasticity reconstructions. To mitigate data-scarcity and improve generalization, a patch-based training regime is also proposed. The second stage introduces a dual-decoder denoising network that separately processes inclusion and background stiffness features, followed by a fusion module that produces a denoised modulus map and a segmentation mask. A multi-objective compound loss is designed to accommodate the denoising, fusing, and mask generation. The method is validated on sequential multi-push (simulation and experimental) SWE motion data with multiple overlapping regions.Results.The method was tested on simulated and CIRS phantom datasets with four overlapping push regions, yielding 26.33 dB peak-signal-to-noise-ratio (PSNR), 30.73 dB contrast-to-noise-ratio (CNR), and 0.813 intersection over union (IoU) in simulation, and 22.44 dB PSNR, 36.88 dB CNR, and 0.781 IoU experimentally. Evaluation on anex vivoswine liver confirmed elasticity estimates within reported biological stiffness ranges. Compared to DSWE-Net and spatio-temporal CNNs, our approach shows superior reconstruction, segmentation, and noise insensitivity.Significance.This framework provides a robust approach to SWE reconstruction and inclusion segmentation, demonstrating strong potential for clinical translation.

Nowadays, inspired by the great success of Transformers in Natural Language Processing, many applications of Vision Transformers (ViTs) have been investigated in the field of medical image analysis including breast ultrasound (BUS) image segmentation and classification. In this paper, we propose an efficient multi-task framework to segment and classify tumors in BUS images using hybrid convolutional neural networks (CNNs)-ViTs architecture and Multi-Perceptron (MLP)-Mixer. The proposed method uses a two-encoder architecture with EfficientNetV2 backbone and an adapted ViT encoder to extract tumor regions in BUS images. The self-attention (SA) mechanism in the Transformer encoder allows capturing a wide range of high-level and complex features while the EfficientNetV2 encoder preserves local information in image. To fusion the extracted features, a Channel Attention Fusion (CAF) module is introduced. The CAF module selectively emphasizes important features from both encoders, improving the integration of high-level and local information. The resulting feature maps are reconstructed to obtain the segmentation maps using a decoder. Then, our method classifies the segmented tumor regions into benign and malignant using a simple and efficient classifier based on MLP-Mixer, that is applied for the first time, to the best of our knowledge, for the task of lesion classification in BUS images. Experimental results illustrate the outperformance of our framework compared to recent works for the task of segmentation by producing 83.42% in terms of Dice coefficient as well as for the classification with 86% in terms of accuracy.

BackgroundThe study of plant photosynthesis is essential for productivity and yield. Thanks to the development of high-throughput phenotyping (HTP) facilities, based on chlorophyll fluorescence imaging, photosynthetic traits can be measured in a reliable, reproducible and efficient manner. In most state-of-the-art HTP platforms, these traits are automatedly analyzed at individual plant level, but information at leaf level is often restricted by the use of manual annotation. Automated leaf tracking over time is therefore highly desired. Methods for tracking individual leaves are still uncommon, convoluted, or require large datasets. Hence, applications and libraries with different techniques are required. New phenotyping platforms are initiated now more frequently than ever; however, the application of advanced computer vision techniques, such as convolutional neural networks, is still growing at a slow pace. Here, we provide a method for leaf segmentation and tracking through the fine-tuning of Mask R-CNN and intersection over union as a solution for leaf tracking on top-down images of plants. We also provide datasets and code for training and testing on both detection and tracking of individual leaves, aiming to stimulate the community to expand the current methodologies on this topic.ResultsWe tested the results for detection and segmentation on 523 Arabidopsis thaliana leaves at three different stages of development from which we obtained a mean F-score of 0.956 on detection and 0.844 on segmentation overlap through the intersection over union (IoU). On the tracking side, we tested nine different plants with 191 leaves. A total of 161 leaves were tracked without issues, accounting to a total of 84.29% correct tracking, and a Higher Order Tracking Accuracy (HOTA) of 0.846. In our case study, leaf age and leaf order influenced photosynthetic capacity and photosynthetic response to light treatments. Leaf-dependent photosynthesis varies according to the genetic background.ConclusionThe method provided is robust for leaf tracking on top-down images. Although one of the strong components of the method is the low requirement in training data to achieve a good base result (based on fine-tuning), most of the tracking issues found could be solved by expanding the training dataset for the Mask R-CNN model.