Medical Image Segmentation Using Transformer Networks.

DSEU-net: A novel deep supervision SEU-net for medical ultrasound image segmentation

Medical image segmentation, the process of precisely delineating regions of interest (e.g. organs, lesions, cells) within medical images, is a pivotal technique in medical image analysis. It finds widespread application in computer‐aided diagnosis, surgical planning, radiation therapy, and pathological analysis, thus playing a crucial role in enabling precision medicine and enhancing the quality of clinical care. Traditional medical image segmentation methods often rely on hand‐crafted features and rule‐based approaches, which struggle to handle the inherent complexity and variability of medical imagery, leading to limitations in segmentation accuracy and robustness. Recently, deep learning methodologies, driven by their powerful capabilities in automatic feature learning and non‐linear modelling, have overcome the limitations of traditional methods and achieved significant advancements in the field of medical image segmentation. This review provides a comprehensive overview and summary of recent progress in deep learning‐based medical image segmentation, with a particular focus on fully supervised learning paradigms leveraging convolutional neural networks, transformers, and the segment anything model. We delve into the underlying principles, network architectures, advantages, and limitations of these approaches. Furthermore, we systematically compare their performance across diverse imaging modalities, anatomical structures, and pathological targets. We also present a curated compilation of commonly used datasets, evaluation metrics, and loss functions relevant to medical image segmentation. Finally, we discuss future research directions and potential challenges, offering insights into the evolving landscape of this critical field.

Machine learning-based algorithms using fully convolutional networks (FCNs) have been a promising option for medical image segmentation. However, such deep networks silently fail if input samples are drawn far from the training data distribution, thus causing critical problems in automatic data processing pipelines. To overcome such out-of-distribution (OoD) problems, we propose a novel OoD score formulation and its regularization strategy by applying an auxiliary add-on classifier to an intermediate layer of an FCN, where the auxiliary module is helfpul for analyzing the encoder output features by taking their class information into account. Our regularization strategy train the module along with the FCN via the principle of outlier exposure so that our model can be trained to distinguish OoD samples from normal ones without modifying the original network architecture. Our extensive experiment results demonstrate that the proposed approach can successfully conduct effective OoD detection without loss of segmentation performance. In addition, our module can provide reasonable explanation maps along with OoD scores, which can enable users to analyze the reliability of predictions.

As the aging of concrete structures becomes increasingly severe, crack detection has become a crucial aspect of maintaining their sustainable use. This paper investigates an automatic detection and analysis method for concrete cracks based on deep convolutional neural network architecture. The study utilized drones equipped with high-definition cameras to capture images of concrete structures such as bridges and roads, collecting over 10,000 original images containing cracks. Using Labelme software for point-by-point image annotation and applying data augmentation techniques like rotation and scaling, the dataset was expanded to 30,000 images to meet the requirements of deep learning model training. An improved AlexNet model was developed, replacing the fully connected layer with global average pooling to enhance robustness and reduce overfitting. The model achieved an average accuracy of 95.42% on the test set, outperforming traditional methods by 15%. The model also proved effective in noisy environments. Additionally, a semantic segmentation model based on Fully Convolutional Network (FCN) was introduced, incorporating atrous convolution and spatial pyramid pooling, achieving a pixel-level accuracy of 97.6%, surpassing benchmark models such as FCN and U-Net. The model accurately estimated crack width, with an error rate within ±2% compared to field measurements. This method improves detection efficiency and accuracy while reducing manual intervention, providing strong support for the maintenance of concrete structures.

In this work, we present a memory-efficient fully convolutional network (FCN)\nincorporated with several memory-optimized techniques to reduce the run-time\nGPU memory demand during training phase. In medical image segmentation tasks,\nsubvolume cropping has become a common preprocessing. Subvolumes (or small\npatch volumes) were cropped to reduce GPU memory demand. However, small patch\nvolumes capture less spatial context that leads to lower accuracy. As a pilot\nstudy, the purpose of this work is to propose a memory-efficient FCN which\nenables us to train the model on full size CT image directly without subvolume\ncropping, while maintaining the segmentation accuracy. We optimize our network\nfrom both architecture and implementation. With the development of computing\nhardware, such as graphics processing unit (GPU) and tensor processing unit\n(TPU), now deep learning applications is able to train networks with large\ndatasets within acceptable time. Among these applications, semantic\nsegmentation using fully convolutional network (FCN) also has gained a\nsignificant improvement against traditional image processing approaches in both\ncomputer vision and medical image processing fields. However, unlike general\ncolor images used in computer vision tasks, medical images have larger scales\nthan color images such as 3D computed tomography (CT) images, micro CT images,\nand histopathological images. For training these medical images, the large\ndemand of computing resource become a severe problem. In this paper, we present\na memory-efficient FCN to tackle the high GPU memory demand challenge in organ\nsegmentation problem from clinical CT images. The experimental results\ndemonstrated that our GPU memory demand is about 40% of baseline architecture,\nparameter amount is about 30% of the baseline.\n

The fully convolutional networks (FCNs) have been widely applied in numerous medical image segmentation tasks. However, tissue regions usually have large variations of shape and scale, so the ability of neural networks to learn multi-scale features is important to the segmentation performance. In this paper, we improve the network for multi-scale feature fusion, in the medical image segmentation by introducing two feature fusion modules: i) global attention multi-scale feature fusion module (GMF); ii) local dense multi-scale feature fusion module (LMF). GMF aims to use global context information to guide the recalibration of low-level features from both spatial and channel aspects, so as to enhance the utilization of effective multi-scale features and suppress the noise of low-level features. LMF adopts bottom-up top-down structure to capture context information, to generate semantic features, and to fuse feature information at different scales. LMF can integrate local dense multi-scale context features layer by layer in the network, thus improving the ability of network to encode interdependent relationships among boundary pixels. Based on the above two modules, we propose a novel medical image segmentation framework (GLF-Net). We evaluated the proposed network and modules on challenging brain tumor segmentation and pancreas segmentation datasets, and very competitive performance has been achieved.

Deep learning has achieved great success in the field of computer vision, and the precision in image classification and image detection has surpassed humans. Therefore, this paper combines deep learning and medical image segmentation, focusing on how to improve the accuracy and speed of segmentation algorithm of medical exercise rehabilitation image. Aiming at the shortcomings of traditional medical image recognition methods, a medical exercise rehabilitation image segmentation algorithm based on hierarchical features of convolutional neural networks is proposed, this paper calls it as hierarchical features of convolutional neural networks (HFCNN). The algorithm firstly samples the convolution output of multiple layers in the convolutional neural network to a unified scale and combines them to construct a hierarchical feature. This hierarchical feature combines the structural information of objects contained in the shallow layer of the network with the semantic information of objects contained in the deep layers of the network, so it has a strong ability to express. Secondly, the image can be segmented into multiple super pixels by the super pixel segmentation algorithm. The classifier is trained using the hierarchical features of the super pixel, and then the classification result is mapped back to the pixel. Finally, a fully connected conditional random field algorithm including one-potential potential energy and paired potential energy is constructed. The corresponding energy function is used to smooth the classification result of the pixel, and the regional consistency and continuity of the pixel mark are improved. Compared with many classical convolutional neural network algorithms, this algorithm not only accelerates the network convergence speed, shortens the training time, but also significantly improves the accuracy of segmentation algorithm of medical exercise rehabilitation image, showing good practical value.

One of the most common tasks in medical imaging is semantic segmentation. Achieving this segmentation automatically has been an active area of research, but the task has been proven very challenging due to the large variation of anatomy across different patients. However, recent advances in deep learning have made it possible to significantly improve the performance of image recognition and semantic segmentation methods in the field of computer vision. Due to the data driven approaches of hierarchical feature learning in deep learning frameworks, these advances can be translated to medical images without much difficulty. Several variations of deep convolutional neural networks have been successfully applied to medical images. Especially fully convolutional architectures have been proven efficient for segmentation of 3D medical images. In this article, we describe how to build a 3D fully convolutional network (FCN) that can process 3D images in order to produce automatic semantic segmentations. The model is trained and evaluated on a clinical computed tomography (CT) dataset and shows state-of-the-art performance in multi-organ segmentation.

In recent years, generative adversarial networks (GANs) have gained tremendous popularity for various imaging related tasks such as artificial image generation to support AI training. GANs are especially useful for medical imaging-related tasks where training datasets are usually limited in size and heavily imbalanced against the diseased class. We present a systematic review, following the PRISMA guidelines, of recent GAN architectures used for medical image analysis to help the readers in making an informed decision before employing GANs in developing medical image classification and segmentation models. We have extracted 54 papers that highlight the capabilities and application of GANs in medical imaging from January 2015 to August 2020 and inclusion criteria for meta-analysis. Our results show four main architectures of GAN that are used for segmentation or classification in medical imaging. We provide a comprehensive overview of recent trends in the application of GANs in clinical diagnosis through medical image segmentation and classification and ultimately share experiences for task-based GAN implementations.

Adaptive optics (AO) retinal imaging has enabled the visualization of cellular-level changes in the living human eye. However, imaging tissue-level lesions with such high resolution introduces unique challenges. At a fine spatial scale, intralesion features can resemble cells, effectively serving as camouflage and making it difficult to delineate the boundary of lesions. The size discrepancy between the tissue-level lesions and retinal cells is also highly variable, ranging from a difference of several-fold to greater than an order-of-magnitude. Here, we introduce a hybrid-transformer based on the combination of a convolutional LinkNet and a fully axial attention transformer network to consider both local and global image features, which excels at identifying tissue-level lesions within a cellular landscape. After training the hybrid transformer on 489 manually-annotated AO images, accurate lesion segmentation was achieved on a separate test dataset consisting of 75 AO images for validation. The segmentation accuracy achieved using the hybrid transformer was superior to the use of convolutional neural networks alone (U-Net and LinkNet) or transformer-based networks alone (AxialDeepLab and Medical Transformer) (p<0.05). These experimental results demonstrate that the combination of convolution and transformer networks are an efficient way to utilize both local and global image features for the purpose of lesion segmentation in medical imaging and may be important for computer-aided diagnosis that relies on accurate lesion segmentation.

Medical image segmentation is a fundamental task in medical image analysis, playing a crucial role in disease diagnosis, treatment planning, and clinical decision-making. Accurate segmentation of anatomical structures, such as blood vessels, organs, and lesions, is essential for reliable medical interpretations. However, existing segmentation models often face challenges in effectively capturing both local and global features within medical images. Traditional convolutional neural networks (CNNs) excel at extracting fine-grained local features but struggle to model long-range dependencies due to their limited receptive fields. Conversely, transformer-based models can capture global contextual relationships but often fail to preserve detailed local structures, leading to suboptimal segmentation performance. The purpose of this study is to develop SEFormer, a novel hybrid network that integrates SENet, ResNet, and Transformer to enhance medical image segmentation. SEFormer aims to effectively capture both local and global features, addressing the limitations of traditional CNNs and transformer-based models.SENet is incorporated to recalibrate feature maps by adaptively emphasizing informative features, thereby compensating for the global representation limitations of CNNs. Meanwhile, the ResNet backbone ensures deep feature extraction while maintaining computational efficiency. Additionally, the Transformer module is introduced to capture long-range dependencies and enhance contextual awareness, complementing the local feature extraction capabilities of CNNs. This hybrid approach allows SEFormer to balance fine-grained spatial details with broader contextual information, leading to more precise segmentation results.Furthermore, to mitigate information loss during feature extraction and ensure a more complete and hierarchical representation of image information, we introduce a feature pyramid structure inspired by multi-scale image pyramid models. By progressively increasing the receptive field across different scales and employing skip connections, SEFormer effectively fuses multi-scale local and global features at each stage of the pyramid. This hierarchical fusion mechanism ensures a richer and more robust feature representation, which is particularly beneficial for segmenting complex medical structures such as blood vessels. We evaluate SEFormer on the CHASE_DB1 dataset, a widely used benchmark for retinal vessel segmentation. Experimental results demonstrate that SEFormer outperforms existing state-of-the-art segmentation methods, achieving a 3.25% improvement in segmentation accuracy. Additionally, we conduct ablation studies to verify the contributions of different components within SEFormer. The results show that incorporating SENet enhances feature recalibration and channel attention, while the Transformer module significantly improves global context awareness. The feature pyramid structure further contributes to performance gains by ensuring a multi-scale representation of vascular structures. Compared to conventional CNN-based methods, our approach achieves better segmentation accuracy with improved robustness to variations in vessel thickness, noise, and image contrast. Furthermore, SEFormer maintains computational efficiency, making it suitable for real-world clinical applications where both accuracy and processing speed are critical. SEFormer provides an efficient and accurate approach for medical image segmentation by leveraging the complementary strengths of CNNs, SENet, and Transformers while integrating a feature pyramid structure to maximize feature representation. Future research directions include extending SEFormer to other medical imaging modalities, such as CT and MRI scans, as well as exploring further optimization techniques to reduce computational costs while maintaining high segmentation performance. Additionally, integrating SEFormer with active learning frameworks could further improve segmentation performance by leveraging expert-annotated data more efficiently.

Medical image segmentation is an essential process in disease analysis and diagnosis, in which medical images are divided into specific areas of anatomic or pathological structures. However, this process has challenges, such as the quality, variability, and complexity of medical images. Thus, we propose a medical image segmentation method using the Convolutional Neural Network (CNN) and the Histogram Equalization (HE) image preprocessing. We evaluated medical image segmentation using 2 secondary datasets, namely Lung CT-Scan and Chest X-Rays, with 267 and 3616 images, respectively. The HE method uses an averaging of the optimal cumulative distribution function (CDF) values with a range of 0 to 39. The experimental results show that adding the HE method can increase the accuracy of CNN on the Lung CT-Scan and Chest X-rays data sets, respectively, by 0.73 percentage points from 91.28% to 92.08 % and 1.23 percentage points from 95.56% to 96.85% for the training process. The results of the testing process (validation) have an increase in accuracy of 2.82 percentage points from 89.77% to 92.59% and 0.91 percentage points from 96.06% to 96.97%. Implementing the HE method increases accuracy in CNN medical image segmentation based on the value of dice similarity coefficient (DSC) and structural similarity index measurement (SSIM) close to 0.95.

Accurate image segmentation is crucial in medical imaging for quantifying diseases, assessing prognosis, and evaluating treatment outcomes. However, existing methods often fall short in integrating global and local features in a meaningful way, failing to give sufficient attention to abnormal regions and boundary details in medical images. These limitations hinder the effectiveness of segmentation techniques in clinical settings. To address these issues, we propose a novel deep learning-based approach, MIPC-Net, designed for precise boundary segmentation in medical images. Our approach, inspired by radiologists' working patterns, introduces two distinct modules: 1. Mutual Inclusion of Position and Channel Attention (MIPC) Module: To improve boundary segmentation precision, we present the MIPC module. This module enhances the focus on channel information while extracting position features and vice versa, effectively enhancing the segmentation of boundaries in medical images. 2. Skip-Residue Module: To optimize the restoration of medical images, we introduce Skip-Residue, a global residual connection. This module improves the integration of the encoder and decoder by filtering out irrelevant information and recovering the most crucial information lost during the feature extraction process. We evaluate the performance of MIPC-Net on three publicly accessible datasets: Synapse, ISIC2018-Task, and Segpc. The evaluation uses metrics such as the Dice coefficient (DSC) and Hausdorff Distance (HD). Our ablation study confirms that each module contributes to the overall improvement of segmentation quality. Notably, with the integration of both modules, our model outperforms state-of-the-art methods across all metrics. Specifically, MIPC-Net achieves a 2.23 mm reduction in Hausdorff Distance on the Synapse dataset, highlighting the model's enhanced capability for precise image boundary segmentation. The introduction of the novel MIPC and Skip-Residue modules significantly improves feature extraction accuracy, leading to better boundary recognition in medical image segmentation tasks. Our approach demonstrates substantial improvements over existing methods, as evidenced by the results on benchmark datasets.

Hybrid CNN-transformer network for interactive learning of challenging musculoskeletal images

An image semantic segmentation algorithm using fully convolutional network (FCN) integrated with the recently proposed simple linear iterative clustering (SLIC) that is based on boundary term (BSLIC) is developed. To improve the segmentation accuracy, the developed algorithm combines the FCN semantic segmentation results with the superpixel information acquired by BSLIC. During the combination process, the superpixel semantic annotation is newly introduced and realized by the four criteria. The four criteria are used to annotate a superpixel region, according to FCN semantic segmentation result. The developed algorithm can not only accurately identify the semantic information of the target in the image, but also achieve a high accuracy in the positioning of small edges. The effectiveness of our algorithm is evaluated on the dataset PASCAL VOC 2012. Experimental results show that the developed algorithm improved the target segmentation accuracy in comparison with the traditional FCN model. With the BSLIC superpixel information that is involved, the proposed algorithm can get 3.86%, 1.41%, and 1.28% improvement in pixel accuracy (PA) over FCN-32s, FCN-16s, and FCN-8s, respectively.