Deformable image-based motion compensation for interventional cone-beam CT with a learned autofocus metric

Deformable motion is one of the main challenges to image quality in interventional cone beam CT (CBCT). Autofocus methods have been successfully applied for deformable motion compensation in CBCT, using multi-region joint optimization approaches that leverage the moderately smooth spatial variation motion of the deformable motion field with a local neighborhood. However, conventional autofocus metrics enforce images featuring sharp image-appearance, but do not guarantee the preservation of anatomical structures. Our previous work (DL-VIF) showed that deep convolutional neural networks (CNNs) can reproduce metrics of structural similarity (visual information fidelity - VIF), removing the need for a matched motion-free reference, and providing quantification of motion degradation and structural integrity. Application of DL-VIF within local neighborhoods is challenged by the large variability of local image content across a CBCT volume, and requires global context information for successful evaluation of motion effects. In this work, we propose a novel deep autofocus metric, based on a context-aware, multi-resolution, deep CNN design. In addition to the inclusion of contextual information, the resulting metric generates a voxel-wise distribution of reference-free VIF values. The new metric, denoted CADL-VIF, was trained on simulated CBCT abdomen scans with deformable motion at random locations and with amplitude up to 30 mm. The CADL-VIF achieved good correlation with the ground truth VIF map across all test cases with R2 = 0.843 and slope = 0.941. When integrated into a multi-ROI deformable motion compensation method, CADL-VIF consistently reduced motion artifacts, yielding an average increase in SSIM of 0.129 in regions with severe motion and 0.113 in regions with mild motion. This work demonstrated the capability of CADL-VIF to recognize anatomical structures and penalize unrealistic images, which is a key step in developing reliable autofocus for complex deformable motion compensation in CBCT.

Purpose: Complex, involuntary, non-periodic, deformable motion presents a confounding factor to cone-beam CT (CBCT) image quality due to long (>10 s) scan times. We report and demonstrate an image-based deformable motion compensation method for CBCT, including phantom, cadaver, and animal studies as precursors to clinical studies. Methods: The method corrects deformable motion in CBCT scan data by solving for a motion vector field (MVF) that optimizes a sharpness criterion in the 3D image (viz., gradient entropy). MVFs are estimated by interpolating M locally rigid motion trajectories across N temporal nodes and are incorporated in a modified 3D filtered backprojection approach. The method was evaluated in a cervical spine phantom under flexion, and a cadaver undergoing variable magnitude of complex motion while imaged on a mobile C-arm (Cios Spin 3D, Siemens Healthineers, Forchheim, Germany). Further assessment was performed on a preclinical animal study using a clinical fixed-room C-arm (Artis Zee, Siemens Healthineers, Forchheim, Germany). Results: In phantom studies, the algorithm resolved visibility of cervical vertebrae under situations of strong flexion, reducing the root-mean-square error by 60% when compared to a motion-free reference. Reduced motion artifacts (blurring, streaks, and loss of soft-tissue edges) were evident in abdominal CBCT of a cadaver imaged during small, medium, and large motion-induced deformation. The animal study demonstrated reduction of streaks from complex motion of bowel gas during the scan. Conclusion: Overall, the studies demonstrate the robustness of the algorithm to a broad range of motion amplitudes, frequencies, data sources (i.e., mobile or fixed-room C-arms) and other confounding factors in real (not simulated) experimental data (e.g., truncation and scatter). These preclinical studies successfully demonstrate reduction of motion artifacts in CBCT and support translation of the method to clinical studies in interventional body radiology.

We describe an investigation of the use of the Insight Toolkit (ITK), a widely used open-source image segmentation and registration software, to obtain the motion transform from the MR images for both rigid and deformable PET motion corrections for simultaneous PET/MR scans. A human brain PET/MR FDG study and two human cardiac PET/MR FDG studies were acquired on Siemens' Biograph-mMR. ITK's 3D-Versor registration was used to obtain the rigid motion transform of the brain from the moved and the reference T2 weighted MR images. Registrations based on B-Spline transform and Demons transform were used to obtain the deformable motion fields from the TWIST (Time-resolved angiography With Interleaved Stochastic Trajectories) and CINE MR images. The motion transforms derived from the registrations were then applied to the PET reconstructions. It has been observed that 3D-Versor registration produces accurate rigid motion transform. With regard to deformable motions, Demons transform performs slightly better than B-Spline transform in terms of %RMSE in the registered TWIST MR images. However, the motion field obtained from the TWIST MR images using Demons transform was observed to introduce irregular distortion in the PET image. On the other hand, Demons motion transforms obtained from the CINE images do not introduce any artifact in the PET image, whereas the B-Spline method does not register the CINE MR images very well. The motion corrected PET image has better signal-to-noise ratio than the gated PET image as expected.

This paper proposes a method to improve the prediction accuracy of motion compensation (MC) in video coding using block-wise and pixel-wise motion with little additional information. In general MC, block-wise motion vectors (MVs) are computed by block matching (BM), but these are not always suitable for motions such as local scaling and deformation. To improve the accuracy of such motions, the proposed method estimates the pixel-wise motion precisely from two decoded frames with little additional information, and extrapolates the frame to be encoded using the pixel-wise motion and an interpolation method. This is first-stage MC. In second-stage MC, the encoder adaptively chooses the extrapolated or previous frames as a reference frame for each MC block. The block-wise MVs are then computed using BM to prevent loss of accuracy. Experimental results show that the proposed method outperforms BM for local scaling and deformation motions.

Purpose: Compare the localization accuracy of 3D cone beam computed tomography (CBCT) and 4D CBCT for a dynamic respiratory phantom, considering cases with and without respiratory variations between planning and treatment. Methods and Materials: A respiratory phantom was imaged using 3D CBCT and 4D CBCT. Measurements of the amplitude of motion in 3D CBCT and 4D CBCT and size of the sphere at its exhale position in 4D CBCT were acquired. The phantom was then misaligned by known distances relative to a reference position. 3D CBCT and 4D CBCT were used to re-align the phantom by comparing localization images to reference images. Because the ideal 4D CBCT registration approach was to be determined, multiple localization-reference pairs of images were examined. Comparison was performed for multiple respiratory waveforms at two amplitudes as well as for waveforms where amplitudes were increased or decreased by 30%, simulating changes in respiratory behavior between simulation and localization. Error of localization trials was defined as the difference between the recommended table shifts and the known displacement. Results: Measurements of the respiratory amplitude were underestimated for all cases. Measurement of the sphere diameter exhibited < 2 mm error. Localization errors were less than 2.6 mm for all cases. When the motion amplitude was the same during localization as planning, automatic registration of the exhale frame from the 4D CBCT to the reference exhale frame had the smallest error in localization of 0.54 ± 0.03 mm. However, automatic registration of the 4D CBCT AIP to the reference AIP had the smallest errors when the motion amplitude changed between planning and localization, 1.59 ± 0.12 mm for the +30% amplitude change and 0.69 ± 0.02 mm for the -30% amplitude change. Conclusion: Because lung tumor motion amplitude throughout treatment may vary between planning and treatment, the use of 4D CBCT AIP registered to reference AIP was recommended for localization. 4D CBCT provided additional benefits compared to 3D CBCT of substantial reduction in target blurring and verification of respiratory motion characteristics prior to treatment.

To assess whether octree representation and octree-based convolutional neural networks (CNNs) improve segmentation accuracy of three-dimensional images. Cardiac CT angiographic examinations from 100 patients (mean age, 67 years ± 17 [standard deviation]; 60 men) performed between June 2012 and June 2018 with semantic segmentations of the left ventricular (LV) and left atrial (LA) blood pools at the end-diastolic and end-systolic cardiac phases were retrospectively evaluated. Image quality (root mean square error [RMSE]) and segmentation fidelity (global Dice and border Dice coefficients) metrics of the octree representation were compared with spatial downsampling for a range of memory footprints. Fivefold cross-validation was used to train an octree-based CNN and CNNs with spatial downsampling at four levels of image compression or spatial downsampling. The semantic segmentation performance of octree-based CNN (OctNet) was compared with the performance of U-Nets with spatial downsampling. Octrees provided high image and segmentation fidelity (median RMSE, 1.34 HU; LV Dice coefficient, 0.970; LV border Dice coefficient, 0.843) with a reduced memory footprint (87.5% reduction). Spatial downsampling to the same memory footprint had lower data fidelity (median RMSE, 12.96 HU; LV Dice coefficient, 0.852; LV border Dice coefficient, 0.310). OctNet segmentation improved the border segmentation Dice coefficient (LV, 0.612; LA, 0.636) compared with the highest performance among U-Nets with spatial downsampling (Dice coefficients: LV, 0.579; LA, 0.592). Octree-based representations can reduce the memory footprint and improve segmentation border accuracy.Keywords CT, Cardiac, Segmentation, Supervised Learning, Convolutional Neural Network (CNN), Deep Learning Algorithms, Machine Learning Algorithms© RSNA, 2021.

Video super-resolution aims at generating a high-resolution video from its low-resolution counterpart. With the rapid rise of deep learning, many recently proposed video super-resolution methods use convolutional neural networks in conjunction with explicit motion compensation to capitalize on statistical dependencies within and across low-resolution frames. Two common issues of such methods are noteworthy. Firstly, the quality of the final reconstructed HR video is often very sensitive to the accuracy of motion estimation. Secondly, the warp grid needed for motion compensation, which is specified by the two flow maps delineating pixel displacements in horizontal and vertical directions, tends to introduce additional errors and jeopardize the temporal consistency across video frames. To address these issues, we propose a novel dynamic local filter network to perform implicit motion estimation and compensation by employing, via locally connected layers, sample-specific and position-specific dynamic local filters that are tailored to the target pixels. We also propose a global refinement network based on ResBlock and autoencoder structures to exploit non-local correlations and enhance the spatial consistency of super-resolved frames. The experimental results demonstrate that the proposed method outperforms the state-of-the-art, and validate its strength in terms of local transformation handling, temporal consistency as well as edge sharpness.

Recent studies on video super-resolution (SR) has shown that convolutional neural network (CNN) combined with motion compensation (MC) is able to merge information from multiple low-resolution (LR) frames to generate high quality images. However, Most SR and MC modules based on deep CNN simply increase the depth of the network, which cannot make full use of hierarchical and multi-scale features, thereby achieving relatively low performance. To address the above problem, a novel multi-scale residual dense Network (MSRDN) is proposed in this paper. A multi-scale residual density block (MSRDB) is first designed to extract abundant local features through dense convolution layer, which helps to adaptively detect image features of different scales with convolution kernels of different scales. Then, we redesign SR module and MC module with MSRDB, which adaptively learns more effective features from local features and uses global feature fusion to jointly and adaptively learn global hierarchical features. Comparative results on Vid4 dataset demonstrate that MSRDB can make more full use of feature information, which helps to effectively improve the reconstruction performance of video SR.

Between March 2008 and September 2009, 17 sequential patients underwent SBRT for either primary lung cancer or isolated metastatic lung disease.All patients had a tumor size limited to ≤ 5 cm in maximum dimension.Fourteen patients had peripheral tumor.

Real-time motion-adaptive radiotherapy of intrahepatic tumors needs to account for motion and deformations of the liver and the target location within. Phantoms representative of anatomical deformations are required to investigate and improve dynamic treatments. A deformable phantom capable of testing motion detection and motion mitigation techniques is presented here. The dynamically dEformable Liver PHAntom (ELPHA) was designed to fulfill three main constraints: First, a reproducibly deformable anatomy is required. Second, the phantom should provide multimodality imaging contrast for motion detection. Third, a time-resolved dosimetry system to measure temporal effects should be provided. An artificial liver with vasculature was casted from soft silicone mixtures. The silicones allow for deformation and radiographic image contrast, while added cellulose provides ultrasonic contrast. An actuator was used for compressing the liver in the inferior direction according to a prescribed respiratory motion trace. Electromagnetic (EM) transponders integrated in ELPHA help provide ground truth motion traces. They were used to quantify the motion reproducibility of the phantom and to validate motion detection based on ultrasound imaging. A two-dimensional ultrasound probe was used to follow the position of the vessels with a template-matching algorithm. This detected vessel motion was compared to the EM transponder signal by calculating the root-mean-square error (RMSE). ELPHA was then used to investigate the dose deposition of dynamic treatment deliveries. Two dosimetry systems, radio-chromic film and plastic scintillation dosimeters (PSD), were integrated in ELPHA. The PSD allow for time-resolved measurement of the delivered dose, which was compared to a time-resolved dose of the treatment planning system. Film and PSD were used to investigate dose delivery to the deforming phantom without motion compensation and with treatment-couch tracking for motion compensation. ELPHA showed densities of 66 and 45HU in the liver and the surrounding tissues. A high motion reproducibility with a submillimeter RMSE (<0.32mm) was measured. The motion of the vasculature detected with ultrasound agreed well with the EM transponder position (RMSE<1mm). A time-resolved dosimetry system with a 1Hz time resolution was achieved with the PSD. The agreement of the planned and measured dose to the PSD decreased with increasing motion amplitude: A dosimetric RMSE of 1.2, 2.1, and 2.7cGy/s was measured for motion amplitudes of 8, 16, and 24mm, respectively. With couch tracking as motion compensation, these values decreased to 1.1, 1.4, and 1.4cGy/s. This is closer to the static situation with 0.7cGy/s. Film measurements showed that couch tracking was able to compensate for motion with a mean target dose within 5% of the static situation (-5% to +1%), which was higher than in the uncompensated cases (-41% to -1%). ELPHA is a deformable liver phantom with high motion reproducibility. It was demonstrated to be suitable for the verification of motion detection and motion mitigation modalities. Based on the multimodality image contrast, a high accuracy of ultrasound based motion detection was shown. With the time-resolved dosimetry system, ELPHA is suitable for performance assessment of real-time motion-adaptive radiotherapy, as was shown exemplary with couch tracking.

Acute esophagitis (AE) occurs among a significant number of patients with locally advanced lung cancer treated with radiotherapy. Early prediction of AE, indicated by esophageal wall expansion, is critical, as it can facilitate the redesign of treatment plans to reduce radiation-induced esophageal toxicity in an adaptive radiotherapy (ART) workflow. We have developed a novel machine learning framework to predict the patient-specific spatial presentation of the esophagus in the weeks following treatment, using magnetic resonance imaging (MRI)/ cone-beam CT (CBCT) scans acquired earlier in the 6 week radiotherapy course. Our algorithm captures the response patterns of the esophagus to radiation on a patch level, using a convolutional neural network. A recurrence neural network then parses the evolutionary patterns of the selected features in the time series, and produces a predicted esophagus-or-not label for each individual patch over future weeks. Finally, the esophagus is reconstructed, using all the predicted labels. The algorithm is trained and validated by means of ∼ 250 000 patches taken from MRI scans acquired weekly from a variety of patients, and tested using both weekly MRI and CBCT scans under a leave-one-patient-out scheme. In addition, our approach is externally validated using a publicly available dataset (Hugo ). Using the first three weekly scans, the algorithm can predict the condition of the esophagus over the succeeding 3 weeks with a Dice coefficient of 0.83 ± 0.04, estimate esophagus volume highly (0.98), correlated with the actual volume, using our institutional MRI/CBCT data. When evaluated using the external weekly CBCT data, the averaged Dice coefficient is 0.89 ± 0.03. Our novel algorithm may prove useful in enabling radiation oncologists to monitor and detect AE in its early stages, and could potentially play an important role in the ART decision-making process.

Background: Precise identification of the Inferior Alveolar Nerve Canal (IANC) on the Cone Beam Computed Tomography (CBCT) is essential to avoid surgical complications. Manual identification is labor-intensive and operator-dependent. Deep learning-based Convolutional Neural Networks (CNNs) hold promise, yet their clinical usefulness is unknown. Objective: To systematically assess the effectiveness, reliability, and clinical usefulness of CNNs for IANC detection on CBCT scans, emphasizing limitations and directions for future research. Methods: A systematic review of PubMed, Science Direct, and Google Scholar was performed for studies published between 2014 and 2024. All studies that applied CNNs for the detection of IANC were included, measuring the most important metrics like accuracy, sensitivity, and specificity. Bias was determined with the Risk of Bias 2 (RoB2) tool, and data regarding CNN architecture, dataset size, and performance were extracted. Two reviewers selected the studies and extracted the data independently, and disagreements were resolved by a third reviewer. Inter-reviewer agreement was determined by using Cohen’s Kappa. Results: Nine studies using U-Net, ResNet, and DenseNet achieved high accuracy (85%–97%), sensitivity (84%–95%), and specificity (85%–96%), better than manual detection. Small sample sizes, dataset constraints, and risk of overfitting compromised generalizability. Conclusion: CNNs enhance IANC detection accuracy and reliability but need dataset heterogeneity, standardized metrics for evaluation, and external validation for clinical use.

4D digital tomosynthesis (DTS) techniques for image-guided radiation therapy (IGRT) are able to reduce radiation dose, scan and reconstruction time compared to 4D cone-beam computed tomography (CBCT). In spite of these benefits, the 4D DTS techniques cause the degradation of image quality due to an intrinsic imaging strategy and consequently reduce treatment accuracy. In this study, a deep learning-based convolutional neural network (CNN) framework was proposed for 4D DTS imaging. The proposed CNN framework consisted of the data restoration network based on a U-Net and the denoising network combined with a 2D wavelet transform, and the network training was implemented with clinical images. The quality of the 4D DTS images obtained from the proposed model was evaluated in terms of quantitative accuracy, spatial resolution and noise property. The results showed that the proposed CNN framework improved the quantitative accuracy of 4D DTS images by 3–19%, and the spatial resolution and noise for the proposed CNN framework were reduced by 2.24–7.33% and 8.92–40.07%, respectively, in comparison to other imaging models. These results represented that the degradation of the 4D DTS image quality can be recovered using the proposed CNN framework, and the proposed model is suitable for maintaining spatial resolution as well as suppressing noise and artifacts. In conclusion, the proposed CNN framework can be potentially used to improve the quality of 4D DTS images for the IGRT.

To develop and evaluate a deep learning (DL)-based artificial intelligence (AI) framework for the comprehensive assessment of impacted maxillary canines, including positional classification, buccal-palatal localization, and detection of adjacent tooth root resorption, using panoramic radiographs (PRs) validated against cone beam computed tomography (CBCT) reference standards. This retrospective study included PRs and CBCT scans of 458 patients (581 impacted maxillary canines) acquired between 2020 and 2024. CBCT images served as the reference standard for buccal-palatal localization and root resorption, while canine position was determined based on beta angle measurements on PRs. Regions of interest (ROIs) were manually annotated on PRs, and cropped PR images were used as model inputs. CBCT images were used exclusively as the reference standard for labeling. A total of twelve pretrained convolutional neural network (CNN) architectures were evaluated across three diagnostic tasks. For each specific task, the top six performing architectures were selected to form a majority voting-based fusion model to enhance diagnostic accuracy. Model performance was assessed using accuracy, precision, recall, and F1-score derived from confusion matrix analyses. For three-class positional classification (horizontal, mesioangular, vertical), the best individual CNN achieved an accuracy of 88.61%, while the fusion model improved accuracy to 90.90%. In buccal-palatal localization, individual model accuracy reached 74.85%, increasing to 80.40% with fusion. For detection of adjacent tooth root resorption, the highest individual accuracy was 83.02%, and the fusion model achieved an accuracy of 88.59%. DL models can accurately evaluate the position, buccal-palatal localization, and root resorption associated with impacted maxillary canines when trained on ROIs extracted from PRs using CBCT-based annotations. The use of multiple CNN architectures combined with a fusion strategy significantly enhances diagnostic performance. These findings suggest that AI-based automated analysis may serve as a reliable complementary tool for improving diagnostic consistency and treatment planning in patients with impacted maxillary canines.

We propose a phase-to-amplitude resampling (PTAR) method to reduce motion blurring in motion-compensated (MoCo) 4D cone-beam CT (CBCT) image reconstruction, without increasing the computational complexity of the motion vector field (MVF) estimation approach. PTAR is able to improve the image quality in reconstructed 4D volumes, including both regular and irregular respiration patterns. The PTAR approach starts with a robust phase-gating procedure for the initial MVF estimation and then switches to a phase-adapted amplitude gating method. The switch implies an MVF-resampling, which makes them amplitude-specific. PTAR ensures that the MVFs, which have been estimated on phase-gated reconstructions, are still valid for all amplitude-gated reconstructions. To validate the method, we use an artificially deformed clinical CT scan with a realistic breathing pattern and several patient data sets acquired with a TrueBeamTM integrated imaging system (Varian Medical Systems, Palo Alto, CA, USA). Motion blurring, which still occurs around the area of the diaphragm or at small vessels above the diaphragm in artifact-specific cyclic motion compensation (acMoCo) images based on phase-gating, is significantly reduced by PTAR. Also, small lung structures appear sharper in the images. This is demonstrated both for simulated and real patient data. A quantification of the sharpness of the diaphragm confirms these findings. PTAR improves the image quality of 4D MoCo reconstructions compared to conventional phase-gated MoCo images, in particular for irregular breathing patterns. Thus, PTAR increases the robustness of MoCo reconstructions for CBCT. Because PTAR does not require any additional steps for the MVF estimation, it is computationally efficient. Our method is not restricted to CBCT but could rather be applied to other image modalities.