Denoising diffusion probabilistic models for 3D medical image generation

International audience

Synthesis of a large set of high-quality medical images with variability in anatomical representation and image appearance has the potential to provide solutions for tackling the scarcity of properly annotated data in medical image analysis research. In this paper, we propose a novel framework consisting of image segmentation and synthesis based on mask-conditional GANs for generating high-fidelity and diverse Cardiac Magnetic Resonance (CMR) images. The framework consists of two modules: i) a segmentation module trained using a physics-based simulated database of CMR images to provide multi-tissue labels on real CMR images, and ii) a synthesis module trained using pairs of real CMR images and corresponding multi-tissue labels, to translate input segmentation masks to realistic-looking cardiac images. The anatomy of synthesized images is based on labels, whereas the appearance is learned from the training images. We investigate the effects of the number of tissue labels, quantity of training data, and multi-vendor data on the quality of the synthesized images. Furthermore, we evaluate the effectiveness and usability of the synthetic data for a downstream task of training a deep-learning model for cardiac cavity segmentation in the scenarios of data replacement and augmentation. The results of the replacement study indicate that segmentation models trained with only synthetic data can achieve comparable performance to the baseline model trained with real data, indicating that the synthetic data captures the essential characteristics of its real counterpart. Furthermore, we demonstrate that augmenting real with synthetic data during training can significantly improve both the Dice score (maximum increase of 4%) and Hausdorff Distance (maximum reduction of 40%) for cavity segmentation, suggesting a good potential to aid in tackling medical data scarcity.

BackgroundBecause of superior soft tissue contrast, the use of magnetic resonance imaging (MRI) as a complement to computed tomography (CT) in the target definition procedure for radiotherapy is increasing. To keep the workflow simple and cost effective and to reduce patient dose, it is natural to strive for a treatment planning procedure based entirely on MRI. In the present study, we investigate the dose calculation accuracy for different treatment regions when using bulk density assignments on MRI data and compare it to treatment planning that uses CT data.MethodsMR and CT data were collected retrospectively for 40 patients with prostate, lung, head and neck, or brain cancers. Comparisons were made between calculations on CT data with and without inhomogeneity corrections and on MRI or CT data with bulk density assignments. The bulk densities were assigned using manual segmentation of tissue, bone, lung, and air cavities.ResultsThe deviations between calculations on CT data with inhomogeneity correction and on bulk density assigned MR data were small. The maximum difference in the number of monitor units required to reach the prescribed dose was 1.6%. This result also includes effects of possible geometrical distortions.ConclusionsThe dose calculation accuracy at the investigated treatment sites is not significantly compromised when using MRI data when adequate bulk density assignments are made. With respect to treatment planning, MRI can replace CT in all steps of the treatment workflow, reducing the radiation exposure to the patient, removing any systematic registration errors that may occur when combining MR and CT, and decreasing time and cost for the extra CT investigation.

Deep learning has resulted in a huge advancement in computer vision. However, deep models require an enormous amount of manually annotated data, which is a laborious and time-consuming task. Large amounts of images demand the availability of target objects for acquisition. This is a kind of luxury we usually do not have in the context of automatic inspection of complex mechanical assemblies, such as in the aircraft industry. We focus on using deep convolutional neural networks (CNN) for automatic industrial inspection of mechanical assemblies, where training images are limited and hard to collect. Computer-aided design model (CAD) is a standard way to describe mechanical assemblies; for each assembly part we have a three-dimensional CAD model with the real dimensions and geometrical properties. Therefore, rendering of CAD models to generate synthetic training data is an attractive approach that comes with perfect annotations. Our ultimate goal is to obtain a deep CNN model trained on synthetic renders and deployed to recognize the presence of target objects in never-before-seen real images collected by commercial RGB cameras. Different approaches are adopted to close the domain gap between synthetic and real images. First, the domain randomization technique is applied to generate synthetic data for training. Second, domain invariant features are utilized while training, allowing to use the trained model directly in the target domain. Finally, we propose a way to learn better representative features using augmented autoencoders, getting performance close to our baseline models trained with real images.

e17105 Background: In patients suspected of prostate cancer, the decision to perform a biopsy hinges on available clinical data. Magnetic resonance imaging (MRI) and digital rectal exam (DRE) data are informative but may not be available. We created accurate models for clinically significant prostate cancer (csPCa), flexible to MRI and DRE data availability. Methods: Optimized ensembles of calibrated random forest models predicting csPCa (Grade Group ≥2) used total PSA, free PSA, prior negative biopsy status, and age, with or without DRE and MRI data (prostate volume and PI-RADS score). Risk models were derived (training cohorts n=1257 to 2191) and validated (validation cohorts n=317 to 1257) from different clinical sites. Models were evaluated by the area under the receiver operating characteristic curve (ROC AUC), sensitivity, specificity, positive predictive value, and negative predictive value, using thresholds providing ~ 95% sensitivity. Feature importance was determined by SHAP analysis. Results: All models had an AUC of at least 0.80, showing that predicting csPCa can be accurate without MRI or DRE data. Including MRI data significantly increased the AUC in the validation cohort (ClarityDX Prostate 0.80 vs ClarityDX Prostate +MRI 0.87). DRE had moderate value for models without MRI data (ClarityDX Prostate 0.80 vs ClarityDX Prostate +DRE 0.82) and minor value with MRI data (ClarityDX Prostate +MRI vs ClarityDX Prostate +DRE+MRI; AUC 0.87 vs 0.87; specificity 45% vs 47%, Table). Mean absolute SHAP values were highest for PI-RADS and prostate volume. Conclusions: These optimized risk models provide high accuracy for predicting csPCa in various clinical settings. Including MRI data greatly increases model accuracy, while DRE has a smaller effect on model accuracy. [Table: see text]

Deep learning has become the main way of the object detection task for autonomous vehicles. Meanwhile, this method typically requires vast amounts of training data to reach their full potential. However, collecting the data from real world and labeling manually is an expensive, time-consuming and error-prone process. Synthetic image has the potential to replace real image for training neural networks, because image creation and labeling annotations are free in this way. For the network trained by synthetic images, the reality gap between real and synthetic images is the main obstacle to use it in the real world. And most previous works are only devoted to generate synthetic images with a good performance on model training, but lack of analysis of the domain gap that affects the performance. This work designs a method of generating the real and synthetic images to analyze and reduce the reality gap between synthetic and real images for car detection. Firstly, this work put one single car with no-background in a random background image to generate real single car and synthetic single car images. In order to further reduce the domain gap in content level, this method keeps the car distribution in synthetic images is similar with the distribution of car in real world. For the purpose of reducing the domain gap in appearance level, the parameters of camera model are same as the camera parameters of image collecting cars and the image is rendered by using the PBRT(Physically Based Ray Tracing) when we generated the synthetic images. Secondly, by training the neural network of instance segmentation with different datasets, the across validation result proves that the reality gap between synthetic and real images is no more than the domain gap between real images. Thirdly, the training results of datasets with different samples diversity show that the diversity of the samples yields better generalization between different datasets for car detection which can effectively reduce the domain gap.

Magnetic resonance imaging (MRI) provides noninvasive evaluation of patient's anatomy without using ionizing radiation. Due to this and the high soft-tissue contrast, MRI use has increased and has potential for use in longitudinal studies where changes in patients' anatomy or tumors at different time points are compared. Deformable image registration can be useful for these studies. Here, we describe two datasets that can be used to evaluate the registration accuracy of systems for MR images, as it cannot be assumed to be the same as that measured on CT images. Two sets of images were created to test registration accuracy. (a) A porcine phantom was created by placing ten 0.35-mm gold markers into porcine meat. The porcine phantom was immobilized in a plastic container with movable dividers. T1-weighted, T2-weighted, and CT images were acquired with the porcine phantom compressed in four different ways. The markers were not visible on the MR images, due to the selected voxel size, so they did not interfere with the measured registration accuracy, while the markers were visible on the CT images and were used to identify the known deformation between positions. (b) Synthetic images were created using 28 head and neck squamous cell carcinoma patients who had MR scans pre-, mid-, and postradiotherapy treatment. An inter- and intrapatient variation model was created using these patient scans. Four synthetic pretreatment images were created using the interpatient model, and four synthetic post-treatment images were created for each of the pretreatment images using the intrapatient model. The T1-weighted, T2-weighted, and CT scans of the porcine phantom in the four positions are provided. Four T1-weighted synthetic pretreatment images each with four synthetic post-treatment images, and four T2-weighted synthetic pretreatment images each with four synthetic post-treatment images are provided. Additionally, the applied deformation vector fields to generate the synthetic post-treatment images are provided. The data are available on TCIA under the collection MRI-DIR. The proposed database provides two sets of images (one acquired and one computer generated) for use in evaluating deformable image registration accuracy. T1- and T2-weighted images are available for each technique as the different image contrast in the two types of images may impact the registration accuracy.

Large annotated datasets are required for training deep learning models, but in medical imaging data sharing is often complicated due to ethics, anonymization and data protection legislation. Generative AI models, such as generative adversarial networks (GANs) and diffusion models, can today produce very realistic synthetic images, and can potentially facilitate data sharing. However, in order to share synthetic medical images it must first be demonstrated that they can be used for training different networks with acceptable performance. Here, we therefore comprehensively evaluate four GANs (progressive GAN, StyleGAN 1–3) and a diffusion model for the task of brain tumor segmentation (using two segmentation networks, U-Net and a Swin transformer). Our results show that segmentation networks trained on synthetic images reach Dice scores that are 80%–90% of Dice scores when training with real images, but that memorization of the training images can be a problem for diffusion models if the original dataset is too small. Our conclusion is that sharing synthetic medical images is a viable option to sharing real images, but that further work is required. The trained generative models and the generated synthetic images are shared on AIDA data hub.

Magnetic resonance imaging (MRI) of breast tissue is often used to definitively diagnose breast cancer due to its high soft-tissue contrast. However, its high cost and limited accessibility make it unsuitable for real-time screening, unlike ultrasound. Machine learning models that synthesize MRI from ultrasound could enable low-cost screening and triage by providing MRI-like tissue characterization during routine ultrasound exams. However, machine learning models trained to generate synthetic MRIs from ultrasound often struggle due to the absence of paired datasets across imaging modalities. This lack of paired datasets is a particular struggle in the field of model building for medical imaging purposes. We propose a physics-based method for generating synthetic ultrasound images from MRI volumes using mathematical modeling of acoustic beam propagation and tissue composition. We use a singular MRI example as a proof-of-concept of this physics simulation, which is based on pre-existing ultrasound physics. Each MRI slice was divided into a 2 × 2 grid to identify local beam entry regions. Tangent lines were calculated along each grid edge to estimate perpendicular ultrasound beam paths in a pinwheel fashion around the breast tissue. Initial simulations indicate that the generated ultrasound-like images reproduce plausible intensity gradients and textural variation consistent with known acoustic-tissue interactions. In resource-limited settings, access to MRI for breast cancer workup remains a significant barrier when ultrasound findings are inconclusive. To enable development of cross-modal synthesis models, we present a physics-based simulation framework that creates paired MRI-ultrasound training datasets by synthesizing realistic ultrasound images from available MRI data. This supports the feasibility of using physics-driven methods to generate synthetic ultrasound from MRI data. Because contrast enhancement and vascular permeability are not modeled, this framework should be interpreted as a non-contrast anatomical MRI to conventional B-mode ultrasound simulation rather than a complete cross-modal transformation that captures DCE-MRI or contrast-enhanced ultrasound behavior.

Background: This study evaluates the ability to predict the response of locally advanced breast cancers to neoadjuvant therapy (NAT) using patient-specific magnetic resonance imaging (MRI) data and a biophysical mathematical model. The 3D mathematical model consists of three parts: tumor cell proliferation, tumor spread (diffusion), and treatment. In particular, the tumor cells proliferate according to logistic growth, and the diffusion term is coupled to the mechanical properties of the surrounding fibroglandular and adipose tissues to inform individual tumor growth patterns (specific to each patient’s anatomy). The model’s treatment term accounts for tumor cell reduction according to approximate local drug delivery for each patient. Methods: Patients (N = 21) with intermediate to high grade invasive breast cancers with varying receptor status, who were eligible for NAT as a component of their clinical care, were recruited. Each patient was treated with standard-of-care consisting of one or two NAT regimens in sequence followed by surgical resection of any residual tumor. MRI data are acquired at four time points: 1) prior to initiation of NAT, 2) after 1 cycle of NAT, 3) after 2-4 cycles of NAT, and 4) 1 cycle after scan 3. The MRI data is processed and evaluated using our semi-automated pipeline. Specifically, diffusion-weighted MRI data is utilized to characterize the cellularity throughout the tumor tissue, and dynamic contrast-enhanced (DCE-) MRI data is used to segment the breast tissue and analyze the local drug delivery using pharmacokinetic analysis and population-derived plasma curves of drug concentrations. The model’s predictive ability is assessed using three different strategies. First, the model is calibrated using each patient’s first two scans to enable predictions of the total tumor cellularity, volume, and longest axis that are directly compared to the values measured from their third scan. Second, the model’s predictions for tumor response are compared to the corresponding response evaluation criteria in solid tumors (RECIST) results. Third, the model is re-calibrated using scans 3 and 4 and simulated to the time of surgery to compare the model’s predictions to each patient’s response status determined by surgical pathology. Results: Calibrating the model with MRI data for one cycle of therapy yields predictions strongly correlated with tumor response measured from each patient’s third scan, concordance correlation coefficients of 0.91, 0.90, and 0.86 for total cellularity, volume, and longest axis, respectively (p < 0.01, N = 18). The model’s predictions are significantly (p < 0.01) correlated with tumor response as designated by RECIST for the cohort. Specifically, the model predicts greater percent reduction in the longest axis for the RECIST designated responder group (i.e., complete response and partial response) compared to non-responders. At the time of surgery, the model predicts changes in total tumor cellularity from baseline that are significantly (p < 0.01) correlated with pathological response status—an area under the receiver operator characteristic curve of 0.92 and a sensitivity and specificity of 1.0 and 0.74, respectively. Discussion: These preliminary results suggest that this clinical-mathematical approach can be predictive of tumor response very early in the course of NAT on a patient-specific basis. Moreover, the study was performed in the community-care setting across a heterogenous group of patients, indicating the approach may be practical for wide-spread application. NCI U01 CA174706, NCI U01 CA154602, CPRIT RR160005, ACS-RSG-18-006-01-CCE, R01CA240589, NCI-U24CA226110, CPRIT RR160093 Citation Format: Angela M Jarrett, David A. Hormuth, II, Anum K Syed, Chengyue Wu, John Virostko, Anna G Sorace, Julie C DiCarlo, Jeanne Kowalski, Debra Patt, Boone Goodgame, Sarah Avery, Thomas E Yankeelov. Predicting breast cancer response to neoadjuvant therapies using a mathematical model individualized with patient-specific magnetic resonance imaging data: Preliminary Results [abstract]. In: Proceedings of the 2020 San Antonio Breast Cancer Virtual Symposium; 2020 Dec 8-11; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2021;81(4 Suppl):Abstract nr PS13-18.

Background: Magnetic resonance (MR) image guided radiation therapy system, enables real time MR guided radiotherapy (RT) without additional radiation exposure to patients during treatment. However, MR image lacks electron density information required for dose calculation. Image fusion algorithm with deformable registration between MR and computed tomography (CT) was developed to solve this issue. However, delivered dose may be different due to volumetric changes during image registration process. In this respect, synthetic CT generated from the MR image would provide more accurate information required for the real time RT. Materials and Methods: We analyzed 1,209 MR images from 16 patients who underwent MR guided RT. Structures were divided into five tissue types, air, lung, fat, soft tissue and bone, according to the Hounsfield unit of deformed CT. Using the deep learning model (U-NET model), synthetic CT images were generated from the MR images acquired during RT. This synthetic CT images were compared to deformed CT generated using the deformable registration. Pixelto- pixel match was conducted to compare the synthetic and deformed CT images. Results and Discussion: In two test image sets, average pixel match rate per section was more than 70% (67.9 to 80.3% and 60.1 to 79%; synthetic CT pixel/deformed planning CT pixel) and the average pixel match rate in the entire patient image set was 69.8%. Conclusion: The synthetic CT generated from the MR images were comparable to deformed CT, suggesting possible use for real time RT. Deep learning model may further improve match rate of synthetic CT with larger MR imaging data.

Unpaired multi-modal training and single-modal testing for detecting signs of endometriosis.

Deep learning architectures can automatically learn complex features and patterns associated with glaucomatous optic neuropathy (GON). However, developing robust algorithms requires a large number of data sets. We sought to train an adversarial model for generating high-quality optic disc images from a large, diverse data set and then assessed the performance of models on generated synthetic images for detecting GON. A total of 17,060 (6874 glaucomatous and 10,186 healthy) fundus images were used to train deep convolutional generative adversarial networks (DCGANs) for synthesizing disc images for both classes. We then trained two models to detect GON, one solely on these synthetic images and another on a mixed data set (synthetic and real clinical images). Both the models were externally validated on a data set not used for training. The multiple classification metrics were evaluated with 95% confidence intervals. Models' decision-making processes were assessed using gradient-weighted class activation mapping (Grad-CAM) techniques. Following receiver operating characteristic curve analysis, an optimal cup-to-disc ratio threshold for detecting GON from the training data was found to be 0.619. DCGANs generated high-quality synthetic disc images for healthy and glaucomatous eyes. When trained on a mixed data set, the model's area under the receiver operating characteristic curve attained 99.85% on internal validation and 86.45% on external validation. Grad-CAM saliency maps were primarily centered on the optic nerve head, indicating a more precise and clinically relevant attention area of the fundus image. Although our model performed well on synthetic data, training on a mixed data set demonstrated better performance and generalization. Integrating synthetic and real clinical images can optimize the performance of a deep learning model in glaucoma detection. Optimizing deep learning models for glaucoma detection through integrating DCGAN-generated synthetic and real-world clinical data can be improved and generalized in clinical practice.

Image synthesis in magnetic resonance (MR) imaging has been an active area of research for more than ten years. MR image synthesis can be used to create images that were not acquired or replace images that are corrupted by artifacts, which can be of great benefit in automatic image analysis. Although synthetic images have been used with success in many applications, it is quite often true that they do not look like real images. In practice, an expert can usually distinguish synthetic images from real ones. Generative adversarial networks (GANs) have significantly improved the realism of synthetic images. However, we argue that further improvements can be made through the introduction of noise in the synthesis process, which better models the actual imaging process. Accordingly, we propose a novel approach that incorporates randomness into the model in order to better approximate the distribution of real MR images. Results show that the proposed method has comparable accuracy with the state-of-the-art approaches as measured by multiple similarity measurements while also being able to control the noise level in synthetic images. To further demonstrate the superiority of this model, we present results from a human observer study on synthetic images, which shows that our results capture the essential features of real MR images.

Spinal robotic guidance platforms, which rely on computed tomography (CT) or cone beam CT (CBCT)-based navigation, enable precise preoperative planning and intraoperative placement of pedicle screws. However, they also result in significant radiation exposure to both patients and clinical staff. A recently FDA- and CE-approved high-resolution synthetic computed tomography (sCT) technology allows the generation of CT-equivalent images from magnetic resonance imaging (MRI) data, potentially reducing radiation exposure. To assess the feasibility of using MRI-derived sCT images for pedicle screw planning and robotic navigation, and to compare patient radiation doses between standard CT-based and sCT-based workflows. Twelve patients scheduled for thoracolumbar pedicle screw fixation underwent 3T MRI with additional T1-weighted 3D dual-echo gradient-echo (GRE) sequences. These GRE datasets were converted into sCT images using BoneMRI® v1.8 (MRIguidance B.V.) and imported into the MAZOR™ robotic guidance system for preoperative planning, sCT-to-fluoroscopy fusion, and intraoperative navigation. Radiation doses were estimated using Monte Carlo-based NCICT and NCIRF dosimetry models, and compared to those from conventional CT-based workflows. sCT images were visually comparable to conventional CT and fully compatible with the robotic navigation platform for both planning and execution. Replacing CT with sCT reduced patient radiation exposure by 98%, with residual dose primarily resulting from intraoperative fluoroscopy used for image fusion and verification. MRI-derived sCT offers a radiation-free alternative to conventional CT for spinal robotic guidance, enabling pedicle screw planning and navigation while eliminating CT-associated radiation exposure.