A Dual-Stream Mamba With Contrastive Representation for Multimodal Deformable Registration.

With respect to the demands of adaptive and 4D-radiotherapy applications, an algorithm is proposed for a fully automatic, multimodality deformable registration that follows the concept of translational relocation of regularly distributed image subvolumes governed by local anatomical features. Thereby, the problem of global deformable registration is broken down to multiple independent local registration steps which allows for straightforward parallelization of the algorithm. In a subsequent step, possible local misregistrations are corrected for by minimization of the elastic energy of the displacement field under consideration of image information. The final displacement field results from interpolation of the subvolume shift vectors. The algorithm can employ as a similarity measure both the correlation coefficient and mutual information. The latter allows the application to intermodality deformable registration problems. The typical calculation time on a modern multiprocessor PC is well below 1 min, which facilitates almost-interactive, "online" usage. CT-to-MRI and CT-to-cone-beam-CT registrations of head-and-neck data sets are presented, as well as inhale-to-exhale registrations of lung CT data sets. For quantitative evaluation of registration accuracy, a virtual thorax phantom was developed; additionally, a landmark-based evaluation on four lung respiratory-correlated CT data sets was performed. This consistently resulted in average registration residuals on the order of the voxel size or less (3D-residuals approximately 1-2 mm). Summarizing, the presented algorithm allows an accurate multimodality deformable registration with calculation times well below 1 min, and thus bears promise as a versatile basic tool in adaptive and 4D-radiotherapy applications.

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

PurposeKilovoltage cone beam computed tomography (kVCBCT)‐guided adaptive radiation therapy (ART) uses daily deformed CT (dCT), which is generated automatically through deformable registration methods. These registration methods may perform poorly in reproducing volumes of the target organ, rectum, and bladder during treatment. We analyzed the registration errors between the daily kVCBCTs and corresponding dCTs for these organs using the default optical flow algorithm and two registration procedures. We validated the effectiveness of these registration methods in replicating the geometry for dose calculation on kVCBCT for ART.MethodsWe evaluated three deformable image registration (DIR) methods to assess their registration accuracy and dose calculation effeciency in mapping target and critical organs. The DIR methods include (1) default intensity‐based deformable registration, (2) hybrid deformable registration, and (3) a two‐step deformable registration process. Each technique was applied to a computerized imaging reference system (CIRS) phantom (Model 062 M) and to five patients who received volumetric modulated arc therapy to the prostate. Registration accuracy was assessed using the 95% Hausdorff distance (HD95) and Dice similarity coefficient (DSC), and each method was compared with the intensity‐based registration method. The improvement in the dCT image quality of the CIRS phantom and five patients was assessed by comparing dCT with kVCBCT. Image quality quantitative metrics for the phantom included the signal‐to‐noise ratio (SNR), uniformity, and contrast‐to‐noise ratio (CNR), whereas those for the patients included the mean absolute error (MAE), mean error, peak signal‐to‐noise ratio (PSNR), and structural similarity index measure (SSIM). To determine dose metric differences, we used a dose‐volume histogram (DVH) and 3.0%/0.3 mm gamma analysis to compare planning computed tomography (pCT) and kVCBCT recalculations with restimulated CT images used as a reference.ResultsThe dCT images generated by the hybrid (dCTH) and two‐step (dCTC) registration methods resulted in significant improvements compared to kVCBCT in the phantom model. Specifically, the SNR improved by 107% and 107.2%, the uniformity improved by 90% and 75%, and the CNR improved by 212.2% and 225.6 for dCTH and dCTC methods, respectively. For the patient images, the MAEs improved by 98% and 94%, the PSNRs improved by 16.3% and 22.9%, and the SSIMs improved by 1% and 1% in the dCTH and dCTC methods, respectively. For the geometric evaluation, only the two‐step registration method improved registration accuracy. The dCTH method yielded an average HD95 of 12 mm and average DSC of 0.73, whereas dCTC yielded an average HD95 of 2.9 mm and average DSC of 0.902. The DVH showed that the dCTC‐based dose calculations differed by <2% from the expected results for treatment targets and volumes of organs at risk. Additionally, gamma indices for dCTC‐based treatment plans were >95% at all points, whereas they were <95% for kVCBCT‐based treatment plans.ConclusionThe two‐step registration method outperforms the intensity‐based and hybrid registration methods. While the hybrid and two‐step‐based methods improved the image quality of kVCBCT in a linear accelerator, only the two‐step method improved the registration accuracy of the corresponding structures among the pCT and kVCBCT datasets. A two‐step registration process is recommended for applying kVCBCT to ART, which achieves better registration accuracy for local and global image structures. This method appears to be beneficial for radiotherapy dose calculation in patients with pelvic cancer.

To develop and evaluate a tool that can improve the accuracy of contour transfer between different image modalities under challenging conditions of low image contrast and large image deformation, comparing to a few commonly used methods, for radiation treatment planning. The software tool includes the following steps and functionalities: (1) accepting input of images of different modalities, (2) converting existing contours on reference images (e.g., MRI) into delineated volumes and adjusting the intensity within the volumes to match target images (e.g., CT) intensity distribution for enhanced similarity metric, (3) registering reference and target images using appropriate deformable registration algorithms (e.g., B-spline, demons) and generate deformed contours, (4) mapping the deformed volumes on target images, calculating mean, variance, and center of mass as the initialization parameters for consecutive fuzzy connectedness (FC) image segmentation on target images, (5) generate affinity map from FC segmentation, (6) achieving final contours by modifying the deformed contours using the affinity map with a gradient distance weighting algorithm. The tool was tested with the CT and MR images of four pancreatic cancer patients acquired at the same respiration phase to minimize motion distortion. Dice's Coefficient was calculated against direct delineation on target image. Contours generated by various methods, including rigid transfer, auto-segmentation, deformable only transfer and proposed method, were compared. Fuzzy connected image segmentation needs careful parameter initialization and user involvement. Automatic contour transfer by multi-modality deformable registration leads up to 10% of accuracy improvement over the rigid transfer. Two extra proposed steps of adjusting intensity distribution and modifying the deformed contour with affinity map improve the transfer accuracy further to 14% averagely. Deformable image registration aided by contrast adjustment and fuzzy connectedness segmentation improves the contour transfer accuracy between multi-modality images, particularly with large deformation and low image contrast.

Image registration is an elementary task in medical image processing and analysis, which can be divided into monomodal and multimodal. Direct 3D multimodal registration in volumetric medical images can provide more insight into the interpretation of subsequent image processing applications than 2D methods. This paper is dedicated to the development of a 3D multimodal image registration algorithm based on a viscous fluid model associated with the Bhattacharyya distance. In our approach, a modified Navier-Stoke's equation is exploited as the foundation of the multimodal image registration framework. The hopscotch method is numerically implemented to solve the velocity field, whose values at the explicit locations are first computed and the values at the implicit positions are solved by transposition. The differential of the Bhattacharyya distance is incorporated into the body force function, which is the main driving force for deformation, to enable multimodal registration. A variety of simulated and real brain MR images were utilized to assess the proposed 3D multimodal image registration system. Preliminary experimental results indicated that our algorithm produced high registration accuracy in various registration scenarios and outperformed other competing methods in many multimodal image registration tasks.Clinical Relevance- This facilitates the disease diagnosis and treatment planning that requires accurate 3D multimodal image registration without massive image data and extensive training regardless of the imaging modality.

The aim of this study is to evaluate anatomic lung tumor changes and dosimetric consequences utilizing the deformable daily kilovolt (KV) cone-beam computer tomography (CBCT) image registration. Five patients diagnosed with NSCLC were treated with three-dimensional conformal radiotherapy (3D CRT) and 10 daily KV CBCT image sets were acquired for each patient. Each CBCT image and plan CT were imported into the deformable image registration (DIR) system. The plan CT image was deformed by the DIR system and a new contour on CBCT was obtained by using the auto-contouring function of the DIR. These contours were individually marked as CBCT f1, CBCT f2,..., and CBCT f10, and imported into a treatment planning system (TPS). The daily CBCT plan was individually generated with the same planning criteria based on new contours. These plans were individually marked as CBCTp1, CBCTp2,..., and CBCTp10, followed by generating a dose accumulation plan (DA plan) in original pCT image contour sets by adding all CBCT plans using Varian Eclipse TPS. The maximum, minimum and mean doses to the plan target volume (PTV) in the 5 DA plans were the same with the CT plans. However, the volume of radiation 5, 10, 20, 30, and 50 Gy of the total lungs in DA plans were less than those of the CT plans. The maximum dose of the spinal cord in the DA plans were average 27.96% less than the CT plans. The mean dose for the left, right, and total lungs in the DA plans were reduced by 13.80%, 23.65%, and 12.96%, respectively. The adaptive 3D CRT based on the deformable registration can reduce the dose to the lung and the spinal cord with the same PTV dose coverage. Moreover, it provides a method for further adaptive radiotherapy exploration.

Deformable image registration (DIR) is a crucial step in adaptive radiation therapy (ART) to deform the planning CT to the current CBCT for dose calculation and for contour propagation. CBCT images are sometimes truncated in the axial plane due to limited field of view (FOV). DIR between CT and truncated CBCT often leads to unphysical results, especially in and near missing regions, which may result in significant errors in dose calculation afterwards. The purpose of this work is to develop and evaluate a method to improve existing DIR algorithms to solve the truncation problem. The radius of FOV of CBCT is first estimated. A recently developed robust CT-CBCT DIR algorithm, called Deformation with Intensity Simultaneously Corrected (DISC), is used for deformation field calculation. At each iteration of DISC, the calculated deformation vector field outside the FOV is replaced by a smooth propagation of the deformation field inside the FOV. Six head-and-neck cancer cases and two prostate cancer cases are used for evaluation. Dose calculation is performed to test the impacts on the resulting dose distribution. In terms of DIR accuracy, it is found that the average normalized mutual information (NMI), normalized cross correlation (NCC) and feature similarity index (FSIM) increase from 0.638, 0.948 and 0.917, to 0.641, 0.951 and 0.919, respectively, when compared with DISC without propagation. In terms of dose distribution, the relative L2 distance of dose (inside the FOV) between the ground truth and that calculated on deformed CT with propagation reduces from 9.25%to 1.41%, compared with those without propagation. We have developed a deformation field propagation method for DIR to register the planning CT and the CBCT image with small FOV. Tests on head-and-neck and prostate cancer cases have demonstrated that our algorithm can generate more accurate registration results for dose calculation. This work is supported in part by the University of California Lab Fees Research Program, the Master Research Agreement from Varian Medical Systems, Inc., and the grants from the National Natural Science Foundation of China (No.30970866).

Background and purpose3D image registration is now common in many medical domains. Multimodal registration implies the use of different imaging modalities, which results in lower accuracy compared to monomodal registration. The aim of this study was to propose a novel approach for deformable image registration (DIR) that incorporates an unsupervised deep learning (DL)-based generation step. The objective was to reduce the challenge of multimodal registration to monomodal registration. Material and methodsTwo datasets from prostate radiotherapy patients were used to evaluate the proposed method. The first dataset consisted of Computed Tomography (CT)/ Cone Beam Computed Tomography (CBCT) pairs from 23 patients using different CBCT devices. The second dataset included Magnetic Resonance Imaging (MRI)/CT pairs from two different care centers, utilizing different MRI devices (0.35 T MRIdian MR-Linac, 1.5 T GE lightspeed MRI). Following a preprocessing step essential for ensuring DL synthesis accuracy and standardizing the database, synthetic CTs (sCTreg) were generated using an unsupervised conditional Generative Adversarial Network (cGAN). The generated sCTs from CBCT or MRI were then utilized for deformable registration with CT scans. This registration method was compared to three standard methods: rigid registration, Elastix registration based on BSplines, and VoxelMorph-based registration (applied exclusively to CBCT/CT). The endpoints of comparison were the dice coefficients calculated between delineated structures for both datasets. ResultsFor both datasets, intermediary sCT generation provided the highest dice coefficients. Dices reached 0.85, 0.85 and 0.75 for the prostate, bladder and rectum for the dataset 1 and 0.90, 0.95 and 0.87 respectively for the dataset 2. When the sCT were not used, dices reached 0.66, 0.78, 0.66 for the dataset 1 and 0.93, 0.87 and 0.84 for the dataset 2. Furthermore, the evaluation of the impact of registration on sCT generation showed that lower Mean Absolute Errors were obtained when the registration was conducted with a sCT. ConclusionsUsing unsupervised deep learning to synthesize intermediate sCT has led to improved registration accuracy in radiotherapy applications employing two distinct imaging modalities.

Fast, accurate, deformable image registration is an important aspect of image-guided interventions. Among the factors that can confound registration is the presence of additional material in the intraoperative image - e.g., contrast bolus or a surgical implant - that was not present in the prior image. Existing deformable registration methods generally fail to account for tissue excised between image acquisitions and typically simply "move" voxels within the images with no ability to account for tissue that is removed or introduced between scans. We present a variant of the Demons algorithm to accommodate such content mismatch. The approach combines segmentation of mismatched content with deformable registration featuring an extra pseudo-spatial dimension representing a reservoir from which material can be drawn into the registered image. Previous work tested the registration method in the presence of tissue excision ("missing tissue"). The current paper tests the method in the presence of additional material in the target image and presents a general method by which either missing or additional material can be accommodated. The method was tested in phantom studies, simulations, and cadaver models in the context of intraoperative cone-beam CT with three examples of content mismatch: a variable-diameter bolus (contrast injection); surgical device (rod), and additional material (bone cement). Registration accuracy was assessed in terms of difference images and normalized cross correlation (NCC). We identify the difficulties that traditional registration algorithms encounter when faced with content mismatch and evaluate the ability of the proposed method to overcome these challenges.

To measure the sensitivity of deformable image registration to image noise. Deformable image registration can be used to map organ contours and other treatment planning data from one CT to another. These CT studies can be acquired with either conventional fan-beam CT systems or more novel cone-beam CT techniques. However, cone-beam CT images can have higher noise levels than fan-beam CT, which might reduce registration accuracy. We have investigated the effect of image quality differences on the deformable registration of fan-beam CTs and CTs with simulated cone-beam noise. Our study used three CT studies for each of five prostate patients. Each CT was contoured by three experienced radiation oncologists. For each patient, one CT was designated the source image and the other two were target images. A deformable image registration process was used to register each source CT to each target CT and then transfer the manually drawn treatment planning contours from the source CT to the target CTs. The accuracy of the automatically transferred contours (and thus of the deformable registration process) was assessed by comparing them to the manual contours on the target CTs, with the differences evaluated with respect to interobserver variability in the manual contours. Then each of the target CTs was modified to include increased noise characteristic of cone-beam CT and the tests were repeated. Changes in registration accuracy due to increased noise were detected by monitoring changes in the automatically transferred contours. We found that the additional noise caused no significant loss of registration accuracy at magnitudes that exceeded what would normally be found in an actual cone-beam CT. We conclude that noise levels in cone-beam CTs that might reduce manual contouring accuracy do not reduce image registration and automatic contouring accuracy.

A prerequisite for adaptive dose-tracking in radiotherapy is the assessment of the deformable image registration (DIR) quality. In this work, various metrics that quantify DIR uncertainties are investigated using realistic deformation fields of 26 head and neck and 12 lung cancer patients. Metrics related to the physiologically feasibility (the Jacobian determinant, harmonic energy (HE), and octahedral shear strain (OSS)) and numerically robustness of the deformation (the inverse consistency error (ICE), transitivity error (TE), and distance discordance metric (DDM)) were investigated. The deformable registrations were performed using a B-spline transformation model. The DIR error metrics were log-transformed and correlated (Pearson) against the log-transformed ground-truth error on a voxel level. Correlations of r ⩾ 0.5 were found for the DDM and HE. Given a DIR tolerance threshold of 2.0 mm and a negative predictive value of 0.90, the DDM and HE thresholds were 0.49 mm and 0.014, respectively. In conclusion, the log-transformed DDM and HE can be used to identify voxels at risk for large DIR errors with a large negative predictive value. The HE and/or DDM can therefore be used to perform automated quality assurance of each CT-based DIR for head and neck and lung cancer patients.

Image registration is a fundamental requirement for medical image analysis. Deep registration methods based on deep learning have been widely recognized for their capabilities to perform fast end-to-end registration. Many deep registration methods achieved state-of-the-art performance by performing coarse-to-fine registration, where multiple registration steps were iterated with cascaded networks. Recently, Non-Iterative Coarse-to-finE (NICE) registration methods have been proposed to perform coarse-to-fine registration in a single network and showed advantages in both registration accuracy and runtime. However, existing NICE registration methods mainly focus on deformable registration, while affine registration, a common prerequisite, is still reliant on time-consuming traditional optimization-based methods or extra affine registration networks. In addition, existing NICE registration methods are limited by the intrinsic locality of convolution operations. Transformers may address this limitation for their capabilities to capture long-range dependency, but the benefits of using transformers for NICE registration have not been explored. In this study, we propose a Non-Iterative Coarse-to-finE Transformer network (NICE-Trans) for image registration. Our NICE-Trans is the first deep registration method that (i) performs joint affine and deformable coarse-to-fine registration within a single network, and (ii) embeds transformers into a NICE registration framework to model long-range relevance between images. Extensive experiments with seven public datasets show that our NICE-Trans outperforms stateof-the-art registration methods on both registration accuracy and runtime.

In-depth understanding of the multimodal deformable image registration tool of a 0.35 T MR-linac.

Landmark-guided deformable image registration for accurate cumulative dose evaluation in lung reirradiation.

Optical coherence tomography (OCT) of the macula has become increasingly important in the investigation of retinal pathology. However, deformable image registration, which is used for aligning subjects for pairwise comparisons, population averaging, and atlas label transfer, has not been well-developed and demonstrated on OCT images. In this paper, we present a deformable image registration approach designed specifically for macular OCT images. The approach begins with an initial translation to align the fovea of each subject, followed by a linear rescaling to align the top and bottom retinal boundaries. Finally, the layers within the retina are aligned by a deformable registration using one-dimensional radial basis functions. The algorithm was validated using manual delineations of retinal layers in OCT images from a cohort consisting of healthy controls and patients diagnosed with multiple sclerosis (MS). We show that the algorithm overcomes the shortcomings of existing generic registration methods, which cannot be readily applied to OCT images. A successful deformable image registration algorithm for macular OCT opens up a variety of population based analysis techniques that are regularly used in other imaging modalities, such as spatial normalization, statistical atlas creation, and voxel based morphometry. Examples of these applications are provided to demonstrate the potential benefits such techniques can have on our understanding of retinal disease. In particular, included is a pilot study of localized volumetric changes between healthy controls and MS patients using the proposed registration algorithm.