A Semi-Automatic Magnetic Resonance Imaging Annotation Algorithm Based on Semi-Weakly Supervised Learning.

Contrastive learning, a successful form of representational learning, has shown promising results in pretraining deep learning (DL) models for downstream tasks. When working with limited annotation data, as in medical image segmentation tasks, learning domain-specific local representations can further improve the performance of DL models. In this work, we extend the contrastive learning framework to utilize domain-specific contrast information from unlabeled Magnetic Resonance (MR) images to improve the performance of downstream MR image segmentation tasks in the presence of limited labeled data. The contrast in MR images is controlled by underlying tissue properties (e.g., T1 or T2) and image acquisition parameters. We hypothesize that learning to discriminate local representations based on underlying tissue properties should improve subsequent segmentation tasks on MR images. We propose a novel constrained contrastive learning (CCL) strategy that uses tissue-specific information via a constraint map to define positive and negative local neighborhoods for contrastive learning, embedding this information in the representational space during pretraining. For a given MR contrast image, the proposed strategy uses local signal characteristics (constraint map) across a set of related multi-contrast MR images as a surrogate for underlying tissue information. We demonstrate the utility of the approach for downstream: (1) multi-organ segmentation tasks in T2-weighted images where a DL model learns T2 information with constraint maps from a set of 2D multi-echo T2-weighted images (n=101) and (2) tumor segmentation tasks in multi-parametric images from the public brain tumor segmentation (BraTS) (n=80) dataset where DL models learn T1 and T2 information from multi-parametric BraTS images. Performance is evaluated on downstream multi-label segmentation tasks with limited data in (1) T2-weighted images of the abdomen from an in-house Radial-T2 (Train/Test=30/20), (2) public Cartesian-T2 (Train/Test=6/12) dataset, and (3) multi-parametric MR images from the public brain tumor segmentation dataset (BraTS) (Train/Test=40/50). The performance of the proposed CCL strategy is compared to state-of-the-art self-supervised contrastive learning techniques. In each task, a model is also trained using all available labeled data for supervised baseline performance. The proposed CCL strategy consistently yielded improved Dice scores, Precision, and Recall metrics, and reduced HD95 values across all segmentation tasks. We also observed performance comparable to the baseline with reduced annotation effort. The t-SNE visualization of features for T2-weighted images demonstrates its ability to embed T2 information in the representational space. On the BraTS dataset, we also observed that using an appropriate multi-contrast space to learn T1+T2, T1, or T2 information during pretraining further improved the performance of tumor segmentation tasks. Learning to embed tissue-specific information that controls MR image contrast with the proposed constrained contrastive learning improved the performance of DL models on subsequent segmentation tasks compared to conventional self-supervised contrastive learning techniques. The use of such domain-specific local representations could help understand, improve performance, and mitigate the scarcity of labeled data in MR image segmentation tasks.

A Validation Framework for Brain Tumor Segmentation

Residual dense network for medical magnetic resonance images super-resolution

Background: Magnetic resonance (MR) image guided radiation therapy system, enables real time MR guided radiotherapy (RT) without additional radiation exposure to patients during treatment. However, MR image lacks electron density information required for dose calculation. Image fusion algorithm with deformable registration between MR and computed tomography (CT) was developed to solve this issue. However, delivered dose may be different due to volumetric changes during image registration process. In this respect, synthetic CT generated from the MR image would provide more accurate information required for the real time RT. Materials and Methods: We analyzed 1,209 MR images from 16 patients who underwent MR guided RT. Structures were divided into five tissue types, air, lung, fat, soft tissue and bone, according to the Hounsfield unit of deformed CT. Using the deep learning model (U-NET model), synthetic CT images were generated from the MR images acquired during RT. This synthetic CT images were compared to deformed CT generated using the deformable registration. Pixelto- pixel match was conducted to compare the synthetic and deformed CT images. Results and Discussion: In two test image sets, average pixel match rate per section was more than 70% (67.9 to 80.3% and 60.1 to 79%; synthetic CT pixel/deformed planning CT pixel) and the average pixel match rate in the entire patient image set was 69.8%. Conclusion: The synthetic CT generated from the MR images were comparable to deformed CT, suggesting possible use for real time RT. Deep learning model may further improve match rate of synthetic CT with larger MR imaging data.

BackgroundDistant metastases is the main failure mode of nasopharyngeal carcinoma. However, early prediction of distant metastases in NPC is extremely challenging. Deep learning has made great progress in recent years. Relying on the rich data features of radiomics and the advantages of deep learning in image representation and intelligent learning, this study intends to explore and construct the metachronous single-organ metastases (MSOM) based on multimodal magnetic resonance imaging.Patients and methodsThe magnetic resonance imaging data of 186 patients with nasopharyngeal carcinoma before treatment were collected, and the gross tumor volume (GTV) and metastatic lymph nodes (GTVln) prior to treatment were defined on T1WI, T2WI, and CE-T1WI. After image normalization, the deep learning platform Python (version 3.9.12) was used in Ubuntu 20.04.1 LTS to construct automatic tumor detection and the MSOM prediction model.ResultsThere were 85 of 186 patients who had MSOM (including 32 liver metastases, 25 lung metastases, and 28 bone metastases). The median time to MSOM was 13 months after treatment (7–36 months). The patients were randomly assigned to the training set (N = 140) and validation set (N = 46). By comparison, we found that the overall performance of the automatic tumor detection model based on CE-T1WI was the best (6). The performance of automatic detection for primary tumor (GTV) and lymph node gross tumor volume (GTVln) based on the CE-T1WI model was better than that of models based on T1WI and T2WI (AP@0.5 is 59.6 and 55.6). The prediction model based on CE-T1WI for MSOM prediction achieved the best overall performance, and it obtained the largest AUC value (AUC = 0.733) in the validation set. The precision, recall, precision, and AUC of the prediction model based on CE-T1WI are 0.727, 0.533, 0.730, and 0.733 (95% CI 0.557–0.909), respectively. When clinical data were added to the deep learning prediction model, a better performance of the model could be obtained; the AUC of the integrated model based on T2WI, T1WI, and CE-T1WI were 0.719, 0.738, and 0.775, respectively. By comparing the 3-year survival of high-risk and low-risk patients based on the fusion model, we found that the 3-year DMFS of low and high MSOM risk patients were 95% and 11.4%, respectively (p < 0.001).ConclusionThe intelligent prediction model based on magnetic resonance imaging alone or combined with clinical data achieves excellent performance in automatic tumor detection and MSOM prediction for NPC patients and is worthy of clinical application.

Simple SummaryThe integration of advanced magnetic resonance imaging (MRI) has the potential to enable the improved prediction of the molecular diagnosis of adult-type gliomas. In this context, this study investigated whether deep learning-based predictive models can benefit from adding multi-shell diffusion MRI to conventional MRI. We evaluated the performance of an exemplar deep learning model for differentiating (1) isocitrate dehydrogenase (IDH)-mutation versus IDH-wildtype; (2) 1p/19q codeletion versus 1p/19q non-codeletion; and (3) IDH-mutation with or without 1p/19q codeletion, and IDH-wildtype. The model achieved the best prediction performance in our cohort of 146 patients in all three tasks when multi-shell diffusion MRI and conventional MRI are combined. These results demonstrate the specific added value provided by advanced diffusion MRI, extending the current literature on building deep learning models based on multiple MRI modalities.The WHO classification since 2016 confirms the importance of integrating molecular diagnosis for prognosis and treatment decisions of adult-type diffuse gliomas. This motivates the development of non-invasive diagnostic methods, in particular MRI, to predict molecular subtypes of gliomas before surgery. At present, this development has been focused on deep-learning (DL)-based predictive models, mainly with conventional MRI (cMRI), despite recent studies suggesting multi-shell diffusion MRI (dMRI) offers complementary information to cMRI for molecular subtyping. The aim of this work is to evaluate the potential benefit of combining cMRI and multi-shell dMRI in DL-based models. A model implemented with deep residual neural networks was chosen as an illustrative example. Using a dataset of 146 patients with gliomas (from grade 2 to 4), the model was trained and evaluated, with nested cross-validation, on pre-operative cMRI, multi-shell dMRI, and a combination of the two for the following classification tasks: (i) IDH-mutation; (ii) 1p/19q-codeletion; and (iii) three molecular subtypes according to WHO 2021. The results from a subset of 100 patients with lower grades gliomas (2 and 3 according to WHO 2016) demonstrated that combining cMRI and multi-shell dMRI enabled the best performance in predicting IDH mutation and 1p/19q codeletion, achieving an accuracy of 75 ± 9% in predicting the IDH-mutation status, higher than using cMRI and multi-shell dMRI separately (both 70 ± 7%). Similar findings were observed for predicting the 1p/19q-codeletion status, with the accuracy from combining cMRI and multi-shell dMRI (72 ± 4%) higher than from each modality used alone (cMRI: 65 ± 6%; multi-shell dMRI: 66 ± 9%). These findings remain when we considered all 146 patients for predicting the IDH status (combined: 81 ± 5% accuracy; cMRI: 74 ± 5%; multi-shell dMRI: 73 ± 6%) and for the diagnosis of the three molecular subtypes according to WHO 2021 (combined: 60 ± 5%; cMRI: 57 ± 8%; multi-shell dMRI: 56 ± 7%). Together, these findings suggest that combining cMRI and multi-shell dMRI can offer higher accuracy than using each modality alone for predicting the IDH and 1p/19q status and in diagnosing the three molecular subtypes with DL-based models.

This study investigates the value of magnetic resonance imaging (MRI) based on a deep learning algorithm in the diagnosis of diabetic macular edema (DME) patients. A total of 96 patients with DME were randomly divided into the experimental group (N = 48) and the control group (N = 48). A deep learning 3D convolutional neural network (3D-CNN) algorithm for MRI images of patients with DME was designed. The application value of this algorithm was comprehensively evaluated by MRI image segmentation Dice value, sensitivity, specificity, and other indicators and diagnostic accuracy. The results showed that the quality of MRI images processed by the 3D-CNN algorithm based on deep learning was significantly improved, and the Dice value, sensitivity, and specificity index data were significantly better than those of the traditional CNN algorithm (P < 0.05). In addition, the diagnostic accuracy of MRI images processed by this algorithm was 93.78 ± 5.32%, which was significantly better than the diagnostic accuracy of 64.25 ± 10.24% of traditional MRI images in the control group (P < 0.05). In summary, the 3D-CNN algorithm based on deep learning can significantly improve the accuracy and sensitivity of MRI image recognition and segmentation in patients with DME, can significantly improve the diagnostic accuracy of MRI in patients with DME, and has a good clinical application value.

The study is aimed at exploring the application of artificial intelligence algorithm-based magnetic resonance imaging (MRI) in the diagnosis of acute cerebral infarction, expected to provide a reference for diagnosis and effect evaluation of acute cerebral infarction. In this study, 80 patients diagnosed with suspected acute cerebral infarction per Diagnostic Criteria for Cerebral Infarction were selected as the research subjects. MRI images were reconstructed by deep dictionary learning to improve their recognition ability. At the same time, the same diagnostic operation was performed by Computed Tomography (CT) images to compare with MRI. The results of the interalgorithm comparison showed the image reconstruction effect of the deep dictionary learning model is significantly better than SAE reconstruction, single-layer dictionary reconstruction model, and KAVD reconstruction. After comparison, the results of MRI based on artificial intelligence algorithm and CT evaluation were statistically significant (P < 0.05). In the lesion image, the diameter of MRI lesions (3.81 ± 0.77 cm) based on artificial intelligence algorithm and the diameter of lesions in CT (3.66 ± 1.65 cm) also had significant statistical significance (P < 0.05). The results showed that MRI based on deep learning was more sensitive than CT imaging for diagnosis and evaluation of patients with acute cerebral infarction, with only 1 case misdiagnosed. The rate of disease detection and lesion image quality had a higher improvement. The results can provide effective support for the clinical application of MRI based on artificial intelligence algorithm in the diagnosis of acute cerebral infarction.

The use of 4D-[(18)F]fluorodeoxyglucose (FDG) PET/CT in combination with respiratory gated magnet resonance imaging (MRI) in target volume definition for stereotactic radiation of liver metastases was investigated. A total of 18patients received respiration gated FDG-PET/CT and MRI. Data were fused using a rigid co-registration algorithm. The quality of the co-registration was rated on a scale from 1 (excellent) to 5 (poor) for co-registration of MRI with gated PET and ungated PET. Gross tumor volume (GTV) was delineated in CT (GTV (CT)), MRI (GTV(MRI)), and PET (GTV(PET)). MRI- and PET-based GTVs were defined by three observers each. Interobserver variability was calculated for all patients as well as for subgroups with and without previous treatment of liver metastases. All GTVs were compared for all patients and separately for patients with previous local therapy. In addition, a semiautomatic segmentation algorithm was applied on the PET images. Co-registration between MR and PET images was rated with 3.3 in average when non-gated PET was used and improved significantly (p < 0.01) to 2.1 using gated PET. The average GTV(CT) was 51.5ml, GTV(MRI) 51.8ml, and the average GTV(PET) 48.1ml. Volumes delineated in MRI were 9.9% larger compared to those delineated in CT. Volumes delineated in PET were 13.8% larger than in MRI. The differences between the GTVs were more pronounced in patients with previous treatment. The GTVs defined in MRI showed an interobserver variability of 47.9% (84.1% with previous treatment and 26.2% without previous treatment). The PET-defined GTVs showed an interobserver variability of 21% regardless of previous treatment. Semiautomatic segmentation did not provide satisfying results. FDG-PET can distinguish vital tumor tissue and scar tissue, and therefore alters the GTV especially in patients with previous local treatment. In addition, it reduces the interobserver variability significantly compared to MRI. However, respiratory gated PET is necessary for good co-registration of PET and MRI.

The deep learning method is an efficient solution for improving the quality of undersampled magnetic resonance (MR) image reconstruction while reducing lengthy data acquisition. Most deep learning methods neglect the mutual constraints between the real and imaginary components of complex-valued k-space data. In this paper, a new complex-valued convolutional neural network, namely, Dense-U-Dense Net (DUD-Net), is proposed to interpolate the undersampled k-space data and reconstruct MR images. The proposed network comprises dense layers, U-Net, and other dense layers in sequence. The dense layers are used to simulate the mutual constraints between real and imaginary components, and U-Net performs feature sparsity and interpolation estimation for k-space data. Two MRI datasets were used to evaluate the proposed method: brain magnitude-only MR images and knee complex-valued k-space data. Several operations were conducted for data preprocessing. First, the complex-valued MR images were synthesized by phase modulation on magnitude-only images. Second, a radial trajectory based on the golden angle was used for k-space undersampling, whereby a reversible normalization method was proposed to balance the distribution of positive and negative values in k-space data. The optimal performance of DUD-Net was demonstrated based on a quantitative evaluation of inter-method and intra-method comparisons. When compared with other methods, significant improvements were achieved, PSNRs were increased by 10.78 and 5.74dB, whereas RMSEs were decreased by 71.53% and 30.31% for magnitude and phase image, respectively. It is concluded that DUD-Net significantly improves the performance of MR image reconstruction.

This thesis focuses on finding ways to differentiate the gray matter (GM) and white matter (WM) in magnetic resonance (MR) images of the human spinal cord (SC). The aim of this project is to quantify tissue loss in these compartments to study their implications on the progression of multiple sclerosis (MS). To this end, we propose segmentation algorithms that we evaluated on MR images of healthy volunteers. Segmentation of GM and WM in MR images can be done manually by human experts, but manual segmentation is tedious and prone to intra- and inter-rater variability. Therefore, a deterministic automation of this task is necessary. On axial 2D images acquired with a recently proposed MR sequence, called AMIRA, we experiment with various automatic segmentation algorithms. We first use variational model-based segmentation approaches combined with appearance models and later directly apply supervised deep learning to train segmentation networks. Evaluation of the proposed methods shows accurate and precise results, which are on par with manual segmentations. We test the developed deep learning approach on images of conventional MR sequences in the context of a GM segmentation challenge, resulting in superior performance compared to the other competing methods. To further assess the quality of the AMIRA sequence, we apply an already published GM segmentation algorithm to our data, yielding higher accuracy than the same algorithm achieves on images of conventional MR sequences. On a different topic, but related to segmentation, we develop a high-order slice interpolation method to address the large slice distances of images acquired with the AMIRA protocol at different vertebral levels, enabling us to resample our data to intermediate slice positions. From the methodical point of view, this work provides an introduction to computer vision, a mathematically focused perspective on variational segmentation approaches and supervised deep learning, as well as a brief overview of the underlying project's anatomical and medical background.

The using of deep learning method belongs to the application and research of artificial intelligence technology in medical field for assisting medical image information processing. Due to the larger difference between the image underlying character features and high-level semantic concepts described, the existing image annotation algorithms perform not very ideal, therefore, the we propose an image annotation framework based on multimodule deep transfer learning method, aiming at the application of deep convolution neural network technology to optimize the neural network parameters, and improve the tagging accuracy in detail. The proposed multimodule deep learning annotation framework uses two stages to optimiz neural network parameters: (1) using deep neural network to optimize the single-mode condition parameters; (2) using the phase correlation, to realize the multi-mode state, the experiment of the optimal combination of public data sets shows that the scheme can effectively improve the performance of image annotation.

597 Background: Automated breast tumor identification and segmentation in magnetic resonance imaging (MRI) is a difficult and crucial area of study in breast cancer research. Artificial intelligence (AI) models are increasingly being developed for automated localization of lesions in imaging studies to facilitate quantitative assessment of features for improved diagnostic, prognostic and predictive performance. Such models have had success in detecting breast cancers in mammography, ultrasound and CT, but few have achieved three-dimensional (3D) volumetric tumor segmentation from breast MRI. The purpose of this study was to apply two state-of-the-art AI – specifically, deep learning (DL) - algorithms to 3D MRI breast cancer data and identify the higher performing algorithm for precise segmentation of breast tumors. Methods: We evaluated pre-treatment, T1 post-gadolinium contrast enhanced breast MRI from 222 patients with known breast cancers (n = 262). Images were split into training (n = 142), validation (n = 36), and hold-out test (n = 44) datasets. Two DL algorithms, U-Net and VAE-UNet, were trained to classify tumors on the training dataset across 1000 epochs. The output for each is a precise localization and segmentation of each tumor at the pixel level from every MRI image. We evaluated the performance of each algorithm using 5-fold cross-validation and testing on the validation and test sets. We calculated a dice accuracy score for each model as the performance comparison metric. Results: The highest dice accuracy score achieved on the validation dataset by generic U-Net was 83.38%, with an average across 1000 epochs of 62.41%. The highest dice accuracy achieved on the validation dataset by VAE-UNet was 82.62%, with an average across epochs of 61.28%. On our test dataset, the highest dice accuracy score achieved by U-Net was 93.09%, with an average across epochs of 66.31%, and the highest accuracy score for VAE-UNet was 90.98%, with average across epochs of 50.47%. Although U-Net appeared to perform slightly better than VAE-Unet for most cases, there were distinct cases where VAE-UNet outperformed U-Net (dice score up to 59% better than U-Net). Subsequent analysis indicated that VAE-UNet preferentially outperforms U-Net for tumors with low sphericity (p = 0.001). Conclusions: Our results suggest that U-Net is well suited for segmenting breast tumors from breast MRI in most cases, but that VAE-UNet outperforms U-Net when the tumor shapes are less spherical. Our findings could inform the choice of DL algorithms in research and clinical endeavors that rely on accurate breast cancer tumor segmentation. In particular, these two tools could be configured to facilitate tumor assessment from breast MRI in the clinical setting for: breast cancer screening in high-risk patient populations, pre-surgical planning, and monitoring of treatment response.

Objectives:Rotator cuff tears (RCT) are common injuries that increase in frequency with age. Deep learning is a subfield of artificial intelligence that uses artificial neural networks to perform a variety of tasks, including image analysis. Several previous studies have investigated the utility of deep learning for identifying fatty infiltration, muscle atrophy, and tear size of the supraspinatus muscle on shoulder magnetic resonance imaging (MRI). However, limited data exists on the use of deep learning for image analysis to determine the presence of supraspinatus pathology versus normal anatomy on shoulder MRI. RadImageNet, a recently published database consisting of 1.35 million annotated computed tomography (CT), MRI, and ultrasound (US) images from greater than 130,000 patients, contains thousands of shoulder MRI images. No research to date has investigated whether deep learning algorithms can accurately detect supraspinatus pathology on MRIs in the RadImageNet database.The objective of the present study was to determine if deep learning algorithms can accurately differentiate shoulder MRI images as those with supraspinatus pathology versus normal anatomy in the RadImageNet database.Methods:The RadImageNet database was queried for all MRI images containing supraspinatus pathology. Both supraspinatus tears (partial and full thickness) and tendinosis images are categorized under the same “supraspinatus pathology” label in the database. In the comparison cohort with normal anatomy, coronal and sagittal images capturing the supraspinatus muscle were included. Although the supraspinatus was still visualized, all sagittal images medial to the glenoid were excluded to better represent the supraspinatus pathology images. The training/validation/test datasets were created using a random 80/10/10 split of the entire cohort. Data were pre-processed using standard data augmentation, including random cropping, rotation, and translation of images. ConvNeXt, a pre-trained deep learning model for the purposes of image analysis, was chosen as the base architecture for transfer learning to optimize both accuracy and computing speed. The ConvNeXt model was trained using training (N=4,937 images) and validation (N=619 images) sets for 15 epochs. Algorithm performance was assessed on an independent test cohort (N=619 images) not used during model training. Performance metrics included classification accuracy, sensitivity, specificity, and positive and negative predictive values. Model training was performed in PyTorch on an 8-gigabyte NIVIDIA Quadro M4000 Cloud Graphics Processing Unit.Results:A total of 5,241 shoulder MRIs with supraspinatus pathology and 932 without any pathology were obtained for comparison. The overall accuracy, sensitivity, specificity, positive predictive value, and negative predictive value of the classifier on the independent test set was 98.9% (95% confidence interval: 97.7% – 99.5%), 98.7%, 100.0%, 100.0%, and 93.1%, respectively. A confusion matrix for model predictions on the test set is displayed in Figure 1. Interpretation plots displaying six examples of incorrect model predictions are shown in Figure 2. Class 1 and 2 designate images without pathology and those with supraspinatus pathology, respectively. All six images were cases where the model predicted “normal” when the ground truth label of the image was supraspinatus pathology. Figure 3 demonstrates an example of class activation mapping, which visually indicates the areas on an image that influenced the model prediction the most. Bright yellow coloration indicates highly influential regions whereas purple coloration corresponds to regions that were less influential. In Figure 3, which is an example image of supraspinatus pathology, one of the most influential regions appears to localize to the supraspinatus tendon.Conclusions:Deep learning is a viable method for differentiating supraspinatus pathology from normal anatomy on shoulder MRIs. The ConvNeXt model demonstrated excellent performance in predicting images with supraspinatus pathology on shoulder MRIs in the RadImageNet database. Future research using deep learning techniques to classify different rotator cuff tear types and sizes is warranted.

Electrocardiogram (ECG) is acquired during Magnetic Resonance Imaging (MRI) to monitor patients and synchronize image acquisition with the heart motion. ECG signals are highly distorted during MRI due to the complex electromagnetic environment. Automated ECG analysis is therefore complicated in this context and there is no reference technique in MRI to classify pathological heartbeats. Imaging arrhythmic patients is hence difficult in MRI. Deep Learning based heartbeat classifier have been suggested but require large databases whereas existing annotated sets of ECG in MRI are very small. We proposed a Siamese network to leverage a large database of unannotated ECGs outside MRI. This was used to develop an efficient representation of ECG signals, further used to develop a heartbeat classifier. We extensively tested several data augmentations and self-supervised learning (SSL) techniques and assessed the generalization of the obtained classifier to ECG signals acquired in MRI. These augmentations included random noises and a model simulating MRI specific artefacts. SSL pretraining improved the generalizability of heartbeat classifiers in MRI (F1=0.75) compared to Deep Learning not relying on SSL (F1=0.46) and another classical machine learning approach (F1=0.40). These promising results seem to indicate that the use of SSL techniques can learn efficient ECG signal representation, and are useful for the development of Deep Learning models even when only scarce annotated medical data are available.