Transformers for colorectal cancer segmentation in CT imaging.

Stroke is a major factor in causing disability and fatalities. Doctors use computerized tomography (CT) and magnetic resonance imaging (MRI) scans to assess the severity of a stroke. Automatic image segmentation can help doctors diagnose strokes more quickly and accurately, but it is challenging due to the variability of stroke lesions and the limited availability of labeled data. Deep learning is the cutting-edge technique of machine learning and artificial intelligence, which needs an extensive labeled dataset for effective training. Unfortunately, in the medical domain, the availability of labeled data is severely limited, posing a challenge for conventional deep- learning approaches. In this article, we introduce a system that utilizes deep learning in the form of fusing transformer-based and convolutional neural network (CNN)-based features and few-shot learning techniques to segment ischemic strokes in multimedia MRIs. To accomplish this, we employ two different methods. The first method involves parallel fusion, where we combine CNN-based and transformer-based features. The second method utilizes serial fusion, combining CNN-based and transformer models using few-shot learning. Through the integration of transformer and CNN models, we can extract both global and local features and enhance the system's performance. Moreover, we tackle the issue of limited labeled data by integrating few-shot learning techniques. Additionally, our system optimizes efficiency by selecting only the slices with lesions, disregarding unlesioned slices. The system under consideration is trained with the BraTS2020 dataset, evaluated on the ISLES 2015 dataset, and contrasted the performance with cutting-edge systems. The suggested system attains a dice coefficient score of 0.76, surpassing the scores of previous cutting-edge systems by a substantial margin.

PurposeSemantic segmentation is a fundamental part of the surgical application of deep learning. Traditionally, segmentation in vision tasks has been performed using convolutional neural networks (CNNs), but the transformer architecture has recently been introduced and widely investigated. We aimed to investigate the performance of deep learning models in segmentation in robot-assisted radical prostatectomy (RARP) and identify which of the architectures is superior for segmentation in robotic surgery.Materials and MethodsIntraoperative images during RARP were obtained. The dataset was randomly split into training and validation data. Segmentation of the surgical instruments, bladder, prostate, vas and seminal vesicle was performed using three CNN models (DeepLabv3, MANet, and U-Net++) and three transformers (SegFormer, BEiT, and DPT), and their performances were analyzed.ResultsThe overall segmentation performance during RARP varied across different model architectures. For the CNN models, DeepLabV3 achieved a mean Dice score of 0.938, MANet scored 0.944, and U-Net++ reached 0.930. For the transformer architectures, SegFormer attained a mean Dice score of 0.919, BEiT scored 0.916, and DPT achieved 0.940. The performance of CNN models was superior to that of transformer models in segmenting the prostate, vas, and seminal vesicle.ConclusionsDeep learning models provided accurate segmentation of the surgical instruments and anatomical structures observed during RARP. Both CNN and transformer models showed reliable predictions in the segmentation task; however, CNN models may be more suitable than transformer models for organ segmentation and may be more applicable in unusual cases. Further research with large datasets is needed.

HomeRadiology: Imaging CancerVol. 2, No. 2 Previous Research HighlightsFree AccessA New Approach for Automated Image Segmentation of Organs at Risk in Cervical CancerMeredith Jones, Lacey R. McNallyMeredith Jones, Lacey R. McNallyMeredith JonesLacey R. McNallyPublished Online:Mar 27 2020https://doi.org/10.1148/rycan.2020204010MoreSectionsPDF ToolsImage ViewerAdd to favoritesCiteTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked InEmail Take-Away Points■ Major Focus: Development of an automated multiclass image segmentation model to identify organs at risk (OARs) on CT scans in patients with locally advanced cervical cancer, prior to radiation therapy.■ Key Results: The convolutional neural network model outperformed model-based, intensity-based, and atlas-based segmentation algorithms, with comparable results to other deep learning segmentation models while also decreasing the time needed for image segmentation by several thousand-fold.■ Impact: Reduction of intra- and interuser variability imposed by manual OAR delineation can lead to more accurate and efficient radiation therapy planning.The recent article by Liu et al developed an automated multiclass image segmentation model to identify organs at risk (OARs) on CT scans. To plan effective and safe radiation therapy in the context of cervical cancer, a CT scan is often acquired prior to the first treatment. Currently, a radiation oncologist will segment the CT images by hand, which is a time-consuming process, subject to intra- and interobserver variability, to identify OARs. To overcome these potential limitations and biases, automated image segmentation techniques can be utilized to improve accuracy and speed of annotating each image. A major problem in creation of deep learning image segmentation models for medical images is the lack of available datasets for model training and validation. Liu et al developed a two-dimensional image segmentation model based off the established convolutional neural network architecture, U-Net. U-Net was utilized because it is feasible to train from scratch with a small training set.This study looked at the CT scans of 105 patients with locally advanced cervical cancer. OAR segmentation for cervical cancer is a particularly challenging problem due to variability in organ size and frequent unclear boundaries. Even an experienced radiologist will have trouble. Seven organs were identified as OARs for cervical cancer radiation therapy: bladder, bone marrow, femoral head left, femoral head right, rectum, small intestine, and spinal cord.The model uses context aggregation blocks where U-Net has convolution layers. These context aggregation blocks use dilation convolution layers with different dilated rates and normal convolution layers with different kernel sizes to yield a low false-positive rate and accurate feature extraction in a large receptive field. A squeeze-extract block was used to ensure proper consideration of organs that are small. A class-weighted cross-entropy loss function was also used to overcome the imbalance in the datasets created by the differences in size and shape of each OAR.A ground truth was available based on the radiation oncologist’s annotations, allowing a Dice similarity coefficient (DSC) and 95th Hausdorff distance (HD) to be calculated and used as an evaluation metric. The DSC and the HD metrics indicated a comparable performance to other deep learning image segmentation models in most of the organs analyzed; inferior results were seen in the rectum and superior results in the intestines. All results were analyzed by an oncologist who determined that greater than 90% of results produced by this model were highly acceptable to be used during radiation therapy planning. To make the model more robust and generalizable, it is necessary to acquire a larger dataset with scans from different sources. Translation of this model to three dimensional may ultimately improve clinical translation.Highlighted ArticleLiu Z, Liu X, Xiao B, et al. Segmentation of organs-at-risk in cervical cancer CT images with a convolutional neural network. Physica Medica 2020;69:184-191. doi: https://doi.org/10.1016/j.ejmp.2019.12.008Highlighted ArticleLiu Z, Liu X, Xiao B, et al. Segmentation of organs-at-risk in cervical cancer CT images with a convolutional neural network. Physica Medica 2020;69:184-191. doi: https://doi.org/10.1016/j.ejmp.2019.12.008 Crossref, Medline, Google ScholarArticle HistoryPublished online: Mar 27 2020 FiguresReferencesRelatedDetailsCited ByJournal of Biomimetics, Biomaterials and Biomedical Engineering, Vol. 49Mathematical Problems in Engineering, Vol. 2020Recommended Articles nnU-Net: Further Automating Biomedical Image AutosegmentationRadiology: Imaging Cancer2021Volume: 3Issue: 1Deep Learning: A Primer for RadiologistsRadioGraphics2017Volume: 37Issue: 7pp. 2113-2131Generative Adversarial Networks: A Primer for RadiologistsRadioGraphics2021Volume: 41Issue: 3pp. 840-857Current Applications and Future Impact of Machine Learning in RadiologyRadiology2018Volume: 288Issue: 2pp. 318-328Deep Generative Adversarial Networks: Applications in Musculoskeletal ImagingRadiology: Artificial Intelligence2021Volume: 3Issue: 3See More RSNA Education Exhibits A Multidisciplinary Approach for Program Development with Artificial Intelligence in Pancreatic Cancer: How We Fit InDigital Posters2019Abnormal Changes After Radiation Therapy: What Radiologists Should KnowDigital Posters2018Artificial Intelligence in Diagnostic Imaging: Current Applications and Future PerspectiveDigital Posters2019 RSNA Case Collection Acute Radiation EnteropathyRSNA Case Collection2021Small Bowel LymphomaRSNA Case Collection2021Hirschsprung DiseaseRSNA Case Collection2021 Vol. 2, No. 2 Metrics Downloaded 310 times Altmetric Score

Domain- and task-specific transfer learning for medical segmentation tasks

To assess whether octree representation and octree-based convolutional neural networks (CNNs) improve segmentation accuracy of three-dimensional images. Cardiac CT angiographic examinations from 100 patients (mean age, 67 years ± 17 [standard deviation]; 60 men) performed between June 2012 and June 2018 with semantic segmentations of the left ventricular (LV) and left atrial (LA) blood pools at the end-diastolic and end-systolic cardiac phases were retrospectively evaluated. Image quality (root mean square error [RMSE]) and segmentation fidelity (global Dice and border Dice coefficients) metrics of the octree representation were compared with spatial downsampling for a range of memory footprints. Fivefold cross-validation was used to train an octree-based CNN and CNNs with spatial downsampling at four levels of image compression or spatial downsampling. The semantic segmentation performance of octree-based CNN (OctNet) was compared with the performance of U-Nets with spatial downsampling. Octrees provided high image and segmentation fidelity (median RMSE, 1.34 HU; LV Dice coefficient, 0.970; LV border Dice coefficient, 0.843) with a reduced memory footprint (87.5% reduction). Spatial downsampling to the same memory footprint had lower data fidelity (median RMSE, 12.96 HU; LV Dice coefficient, 0.852; LV border Dice coefficient, 0.310). OctNet segmentation improved the border segmentation Dice coefficient (LV, 0.612; LA, 0.636) compared with the highest performance among U-Nets with spatial downsampling (Dice coefficients: LV, 0.579; LA, 0.592). Octree-based representations can reduce the memory footprint and improve segmentation border accuracy.Keywords CT, Cardiac, Segmentation, Supervised Learning, Convolutional Neural Network (CNN), Deep Learning Algorithms, Machine Learning Algorithms© RSNA, 2021.

Rapid advancements in artificial intelligence and generative artificial intelligence have enabled the creation of fake images and videos that appear highly realistic. According to a report published in 2022, approximately 71% of people rely on fake videos and become victims of blackmail. Moreover, these fake videos and images are used to tarnish the reputation of popular public figures. This has increased the demand for deepfake detection techniques. The accuracy of the techniques proposed in the literature so far varies with changes in fake content generation techniques. Additionally, these techniques are computationally intensive. The techniques discussed in the literature are based on convolutional neural networks, Linformer models, or transformer models for deepfake detection, each with its advantages and disadvantages. In this manuscript, a hybrid architecture combining transformer and Linformer models is proposed for deepfake detection. This architecture converts an image into patches and performs position encoding to retain spatial relationships between patches. Its encoder captures the contextual information from the input patches, and Gaussian Error Linear Unit resolves the vanishing gradient problem. The Linformer component reduces the size of the attention matrix. Thus, it reduces the execution time to half without compromising accuracy. Moreover, it utilizes the unique features of transformer and Linformer models to enhance the robustness and generalization of deepfake detection techniques. The low computational requirement and high accuracy of 98.9% increase the real-time applicability of the model, preventing blackmail and other losses to the public. The proposed hybrid model utilizes the strength of the transformer model in capturing complex patterns in data. It uses the self-attention potential of the Linformer model and reduces the computation time without compromising the accuracy. Moreover, the models were implemented on patch sizes of 6 and 11. It is evident from the obtained results that increasing the patch size improves the performance of the model. This allows the model to capture fine-grained features and learn more effectively from the same set of videos. The larger patch size also enables the model to better preserve spatial details, which contributes to improved feature extraction.

Eye diseases can affect vision and well-being, so early, accurate diagnosis is crucial to prevent serious impairment. Deep learning models have shown promise for automating the diagnosis of eye diseases from images. However, current methods mostly use single-model architectures, including convolutional neural networks (CNNs), which might not adequately capture the long-range spatial correlations and local fine-grained features required for classification. To address these limitations, this study proposes a multi-stage framework for eye diseases (MST-EDS), including two stages: hybrid and stacking models in the categorization of eye illnesses across four classes: normal, diabetic_retinopathy, glaucoma, and cataract, utilizing a benchmark dataset from Kaggle. Hybrid models are developed based on Transformer models: Vision Transformer (ViT), Data-efficient Image Transformer (DeiT), and Swin Transformer are used to extract deep features from images, Principal Component Analysis (PCA) is used to reduce the complexity of extracted features, and Machine Learning (ML) models are used as classifiers to enhance performance. In the stacking model, the outputs of the best hybrid models are stacked, and they are used to train and evaluate meta-learners to improve classification performance. The experimental results show that the MST-EDS-RF model recorded the best performance compared to individual Transformer and hybrid models, with 97.163% accuracy.

Kidneys are most easily segmented by convolutional neural networks (CNN) on contrast enhanced CT (CECT) images, but their segmentation accuracy may be reduced when only non-contrast CT (NCCT) images are available. The purpose of this work was to investigate the improvement in segmentation accuracy when implementing a generative adversarial network (GAN) to create virtual contrast enhanced (vCECT) images from non-contrast inputs. A 2D cycleGAN model, incorporating an additional idempotent loss function to restrict the GAN from making unnecessary modifications to data already in the translated domain, was trained to generate virtual contrast enhanced images on 286 paired non-contrast and contrast enhanced inputs. A 3D CNN trained on contrast enhanced images was applied to segment the kidneys in a test set of 20 paired non-contrast and contrast enhanced images. The non-contrast images were converted to virtual contrast enhanced images, then kidneys in both image conditions were segmented by the CNN. Segmentation results were compared to analyst annotations on non-contrast images visually and by Dice Coefficient (DC). Segmentation on virtual contrast enhanced images were more complete with fewer extraneous detections compared to non-contrast images in 16/20 cases. Mean(±SD) DC was 0.88(±0.80), 0.90(±0.03), and 0.95(±0.05) for non-contrast, virtual contrast enhanced, and real contrast enhanced, respectively. Virtual contrast enhancement visually improved segmentation quality, poor performing cases had their performance improved resulting in an overall reduction in DC variation, and the minimum DC increased from 0.65 to 0.85. This work provides preliminary results demonstrating the potential effectiveness of using a GAN for virtual contrast enhancement to improve CNN-based kidney segmentation on non-contrast images.

Segmentation of organs-at-risk in cervical cancer CT images with a convolutional neural network

Medical Image Semantic Segmentation (MISS), the process of assigning a semantic label to each pixel in an image, is a foundational task in computational medicine, critical for quantitative diagnostics and treatment planning. However, developing robust MISS models faces two intertwined challenges. First, there is an architectural dilemma: Convolutional Neural Networks (CNNs), like U-Net, excel at learning local features but are limited by their receptive fields, failing to capture global context essential for segmenting organs with large deformations. Conversely, Vision Transformers (ViTs) effectively model long-range dependencies but lack the inductive biases of CNNs, leading to poor generalization on the small datasets typical in medicine without extensive pre-training. Second, the prohibitive cost and expertise required for creating pixel-level annotations create a severe data scarcity bottleneck. While Semi-Supervised Learning (SSL) aims to mitigate this by leveraging unlabeled data, existing methods often fail to learn high-level semantic relations and are susceptible to confirmation bias from noisy pseudo-labels, class imbalance, and suboptimal contrastive sample selection. This thesis presents a comprehensive investigation to systematically address these challenges, delivering a cohesive suite of novel deep learning frameworks. The contributions are four-fold: First, to resolve the architectural trade-off, this work introduces CS-Unet, a pure Transformer network built upon a U-Net-like architecture. Its core innovation is the Convolutional Swin Transformer (CST) block, which integrates convolutions directly within the Multi-Head Self-Attention and Feed-Forward Network modules. This design imbues the Transformer with inherent localized spatial context and strong inductive biases, enabling it to efficiently learn both local and global features. Without pre-training, CS-Unet outperforms existing Transformer and CNN-based models on multi-organ and cardiac datasets, achieving state-of-the-art performance with fewer parameters. Second, to address data scarcity, a novel Multi-Scale Cross Supervised Contrastive Learning (MCSC) framework for SSL is developed. MCSC jointly trains CNN and Transformer models, using a cross-teaching paradigm where each network provides pseudo-labels for the other. Crucially, it moves beyond simple output consistency by applying a contrastive loss to feature maps at multiple scales, enforcing hierarchical semantic consistency. To handle the class imbalance endemic to medical imaging, a class-prevalence-aware loss is used to ensure features for infrequent classes are learned robustly. Third, to fortify SSL against noisy pseudo-labels, a certainty-guided contrastive learning strategy is proposed. This approach mitigates the impact of inaccurate pseudo-labels by using a certainty metric to guide the selection of samples for contrastive learning. The framework’s computational efficiency is enhanced through novel sampling strategies that select a few representative samples for contrasting, and a negative memory bank is used to increase sample diversity and eliminate dependence on batch size. Fourth, this thesis introduces a new paradigm for SSL by leveraging external anatomical priors through the Contrastive Cross-Teaching with Registration (CCT-R) framework. CCT-R is the first method to integrate spatial registration transforms into the learning process. It features two novel modules: a Registration Supervision Loss (RSL), which uses transforms between labeled and unlabeled volumes to generate an additional, highly reliable source of pseudo-labels, and Registration-Enhanced Positive Sampling (REPS), which uses registration to identify anatomically-corresponding positive pairs across volumes for contrastive learning. Overall, these contributions provide a powerful toolkit that significantly alleviates the annotation bottleneck in medical AI. The proposed methods demonstrate state-of-the-art performance on challenging segmentation benchmarks, delivering a pathway to develop accurate, data-efficient models for real-world clinical applications and opening new avenues for research into fusing geometric priors with semantic segmentation.

Purpose:To develop a head and neck normal structures auto-contouring toolthat could be used to automatically detect the errors in auto-contours froma clinically-validated auto-contouring tool.Methods:An auto-contouring tool based on convolutional neural networks (CNN)was developed for 16 normal structures of the head and neck and tested toidentify the contour errors from a clinically-validated multi-atlas-basedauto-contouring system (MACS). The CT scans and clinical contours from 3495patients were semi-automatically curated and used to train and validate theCNN-based auto-contouring tool. The final accuracy of the tool was evaluatedby calculating the Sørensen-Dice similarity coefficients (DSC) andHausdorff distances between the automatically generated contours andphysician-drawn contours on 174 internal and 24 external CT scans. Lastly,the CNN-based tool was evaluated on 60 patients’ CT scans toinvestigate the possibility to detect contouring failures. The contouringfailures on these patients were classified as either minor or major errors.The criteria to detect contouring errors were determined by analyzing theDSC between the CNN- and MACS-based contours under two independentscenarios: 1. contours with minor error are clinically acceptable and 2.contours with minor errors are clinically unacceptable.Results:The average DSC and Hausdorff distance of our CNN-based tool were98.4%/1.23cm for brain, 89.1%/0.42cm for eyes, 86.8%/1.28cm for mandible,86.4%/0.88cm for brainstem, 83.4%/0.71cm for spinal cord, 82.7%/1.37cm forparotids, 80.7%/1.08cm for esophagus, 71.7%/0.39cm for lenses, 68.6%/0.72for optic nerves, 66.4%/0.46cm for cochleas, and 40.7%/0.96cm for opticchiasm. With the error detection tool, the proportions of the clinicallyunacceptable MACS contours that were correctly detected were 0.99/0.80 onaverage except for the optic chiasm, when contours with minor errors areclinically acceptable/unacceptable respectively. The proportions of theclinically acceptable MACS contours that were correctly detected were0.81/0.60 on average except for the optic chiasm, when contours with minorerrors are clinically acceptable/unacceptable respectively.Conclusion:Our CNN-based auto-contouring tool performed well on both thepublically-available and the internal datasets. Furthermore, our resultsshow that CNN-based algorithms are able to identify ill-defined contoursfrom a clinically-validated and used multi-atlas-based auto-contouring tool.Therefore, our CNN-based tool can effectively perform automatic verificationof MACS contours.

Stiffened panels are widely used in a range of engineering domains to enhance load-bearing capacity, reduce buckling, and improve overall structural integrity. Stiffener disbonding is a common failure mode, making it essential to detect these disbonds before they reach a critical level that could lead to structural failure. In this work, a transformer architecture has been implemented to detect the presence and extent of disbonding between the stiffener and the host aluminium plate. The research uses a dataset of Lamb waves obtained from an experimental setup with four types of specimens, ranging from no disbonding to disbonding of varying widths induced by Teflon strips. The problem is framed as a multi-class classification task with labels representing different disbonding levels. The transformer model utilizes a self-attention mechanism to analyze the Lamb wave responses and ascertain the presence and extent of disbanding. Operating in encoder-only mode, the tailored transformer accepts Lamb wave responses as input and outputs a class probability score for each of the four predefined disbonding levels using a softmax activation function. The developed model has been able to predict the state of the stiffened panels with an accuracy of 100%. To demonstrate the effectiveness of the transformer model, it has been compared to the convolutional neural network (CNN) model. When tested on an unseen noisy dataset, the transformer model achieved an accuracy of 100%, outperforming the CNN model, which achieved an accuracy of 89%.

Automatic cervical lymphadenopathy segmentation from CT data using deep learning

Transformer based architectures have emerged as powerful deep learning models compared to traditional convolutional neural networks (CNNs) for complex tasks like image classification and segmentation. In this study, the performance of transformer architectures on analysis of medical imaging datasets is investigated through quantitative comparison of three state-of-the-art transformer models which are Vision Transformer (ViT), SegFormer, and MaskFormer, with respect to CNN-based architectures. The models are evaluated on benchmark MRI datasets of human brain across two core tasks: classification and semantic segmentation for three tumor types namely glioma, meningioma and pituitary tumor. Our analysis takes into account multiple performance indicators, including accuracy, segmentation quality, model complexity, and model interpretability. The results reveal notable differences in trade-offs between accuracy, interpretability, and resource demands, offering practical guidance on the relative strengths and suitability of transformer and CNN models in clinical brain tumor diagnostics.

Convolutional neural networks (CNNs) show potential for delineating cancers on contrast-enhanced MRI (ce-MRI) but there are clinical scenarios in which administration of contrast is not desirable. We investigated performance of the CNN for delineating primary nasopharyngeal carcinoma (NPC) on non-contrast-enhanced images and compared the performance to that on ce-MRI. We retrospectively analyzed primary NPC in 195 patients using a well-established CNN, U-Net, for tumor delineation on the non-contrast-enhanced fat-suppressed (fs)-T2W, ce-T1W and ce-fs-T1W images. The CNN-derived delineations were compared to manual delineations to obtain Dice similarity coefficient (DSC) and average surface distance (ASD). The DSC and ASD on fs-T2W were compared to those on ce-MRI. Primary tumor volumes (PTVs) of CNN-derived delineations were compared to that of manual delineations. The CNN for NPC delineation on fs-T2W images showed similar DSC (0.71 ± 0.09) and ASD (0.21 ± 0.48cm) to those on ce-T1W images (0.71 ± 0.09 and 0.17 ± 0.19cm, respectively) (p > 0.05), and lower DSC but similar ASD to ce-fs-T1W images (0.73 ± 0.09, p < 0.001; and 0.17 ± 0.20cm, p > 0.05). The CNN overestimated PTVs on all sequences (p < 0.001). The CNN showed promise for NPC delineation on fs-T2W images in cases where it is desirable to avoid contrast agent injection. The CNN overestimated PTVs on all sequences.