Deep learning based data-adaptive descriptor for non-rigid multi-modal medical image registration

Notwithstanding the widespread use of image guided neurosurgery systems in recent years, the accuracy of these systems is strongly limited by the intra-operative deformation of the brain tissue, the so-called brain shift. Intra-operative ultrasound (iUS) imaging as an effective solution to compensate complex brain shift phenomena update patients coordinate during surgery by registration of the intra-operative ultrasound and the pre-operative MRI data that is a challenging problem.In this work a non-rigid multimodal image registration technique based on co-sparse analysis model is proposed. This model captures the interdependency of two image modalities; MRI as an intensity image and iUS as a depth image. Based on this model, the transformation between the two modalities is minimized by using a bimodal pair of analysis operators which are learned by optimizing a joint co-sparsity function using a conjugate gradient.Experimental validation of our algorithm confirms that our registration approach outperforms several of other state-of-the-art registration methods quantitatively. The evaluation was performed using seven patient dataset with the mean registration error of only 1.83 mm. Our intensity-based co-sparse analysis model has improved the accuracy of non-rigid multimodal medical image registration by 15.37% compared to the curvelet based residual complexity as a powerful registration method, in a computational time compatible with clinical use.

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

Nonrigid multimodal image registration remains a challenging task in medical image processing and analysis. The structural representation (SR)-based registration methods have attracted much attention recently. However, the existing SR methods cannot provide satisfactory registration accuracy due to the utilization of hand-designed features for structural representation. To address this problem, the structural representation method based on the improved version of the simple deep learning network named PCANet is proposed for medical image registration. In the proposed method, PCANet is firstly trained on numerous medical images to learn convolution kernels for this network. Then, a pair of input medical images to be registered is processed by the learned PCANet. The features extracted by various layers in the PCANet are fused to produce multilevel features. The structural representation images are constructed for two input images based on nonlinear transformation of these multilevel features. The Euclidean distance between structural representation images is calculated and used as the similarity metrics. The objective function defined by the similarity metrics is optimized by L-BFGS method to obtain parameters of the free-form deformation (FFD) model. Extensive experiments on simulated and real multimodal image datasets show that compared with the state-of-the-art registration methods, such as modality-independent neighborhood descriptor (MIND), normalized mutual information (NMI), Weber local descriptor (WLD), and the sum of squared differences on entropy images (ESSD), the proposed method provides better registration performance in terms of target registration error (TRE) and subjective human vision.

Medical image registration is an integral part of various clinical applications including image guidance, motion tracking, therapy assessment and diagnosis. We present a robust approach for mono-modal and multi-modal medical image registration. To this end, we propose the novel shape operator based local image distance (SOLID) which estimates the similarity of images by comparing their second-order curvature information. Our similarity metric is rigorously tailored to be suitable for comparing images from different medical imaging modalities or image contrasts. A critical element of our method is the extraction of local features using higher-order shape information, enabling the accurate identification and registration of smaller structures. In order to assess the efficacy of the proposed similarity metric, we have implemented a variational image registration algorithm that relies on the principle of matching the curvature information of the given images. The performance of the proposed algorithm has been evaluated against various alternative state-of-the-art variational registration algorithms. Our experiments involve mono-modal as well as multi-modal and cross-contrast co-registration tasks in a broad variety of anatomical regions. Compared to the evaluated alternative registration methods, the results indicate a very favorable accuracy, precision and robustness of the proposed SOLID method in various highly challenging registration tasks.

In non-rigid image registration there are some distinct outliers introduced by the insertion or removal of some image contents, abrupt intensity changes, and local non-smooth deformations in the two images to register. Recently, non-rigid registration of images with outliers has received a great deal of attention in medical imaging research. This work presents a framework for robust non-rigid registration based on the theory of optical flow, within which we use outlier detection to implement locally-refined control point setup in multilevel free- form deformation (FFD) model. We also introduce robust M-estimator to weaken the effect of outliers in global optimization to get improved registration result. We have successfully applied our proposed method to a number of registration tasks to demonstrate its applicability and robustness to outliers.

Non-rigid image registration is an essential tool required for overcoming the inherent local anatomical variations that exist between medical images acquired from different individuals or atlases, among others. This type of registration defines a deformation field that gives a translation or mapping for every pixel in the image. One popular local approach for estimating this deformation field, known as block matching, is where a grid of control points are defined on an image and are each taken as the centre of a small window. These windows are then translated in the second image to maximise a local similarity criterion. This generates two corresponding sets of control points for the two images, yielding a sparse deformation field. This sparse field can then be propagated to the entire image using well known methods such as the thin-plate spline warp or simple Gaussian convolution. Previous block matching procedures all utilise uniformly distributed grid points. This results in the generation of a sparse deformation field containing displacement estimates at uniformly spaced locations. This neglects to make use of the evidence that block matching results are dependent on the amount of local information content. That is, results are better in regions of high information when compared to regions of low information. Consequently, this paper presents a solution to this drawback by proposing the use of a Reversible Jump Markov Chain Monte Carlo (RJMCMC) statistical procedure to optimally select grid points of interest. These grid points have a greater concentration in regions of high information and a lower concentration in regions of small information. Results show that non-rigid registration can by improved by using optimally selected grid points of interest.

Deep learning-based affine medical image registration for multimodal minimal-invasive image-guided interventions – A comparative study on generalizability

CoCycleReg: Collaborative cycle-consistency method for multi-modal medical image registration

Joint segmentation and registration of images is a focused area of research nowadays. Jointly segmenting and registering noisy images and images having weak boundaries/intensity inhomogeneity is a challenging task. In medical image processing, joint segmentation and registration are essential methods that aid in distinguishing structures and aligning images for precise diagnosis and therapy. However, these methods encounter challenges, such as computational complexity and sensitivity to variations in image quality, which may reduce their effectiveness in real-world applications. Another major issue is still attaining effective joint segmentation and registration in the presence of artifacts or anatomical deformations. In this paper, a new nonparametric joint model is proposed for the segmentation and registration of multi-modality images having weak boundaries/noise. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is more efficient to segmenting and registering multi-modality images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is also tested on the images obtained from the freely available CHOAS dataset and compare the results of the proposed model with the other existing models using statistical measures such as the Jaccard similarity index, relative reduction, Dice similarity coefficient and Hausdorff distance. It can be seen that the proposed model outperforms the other existing models in terms of quantitatively and qualitatively.

This research proposes a novel non-rigid registration method for ultrasound images. The most predominant anatomical features in medical images are tissue boundaries, which appear as edges. In ultrasound images, however, other features can be identified as well due to the specular reflections that appear as bright lines superimposed on the ideal edge location. In this work, an image’s local phase information (via the frequency domain) is used to find the ideal edge location. The generalized relaxation labeling process is then formulated to align the feature points extracted from the ideal edge location. In this work, the original relaxation labeling method was generalized by taking n compatibility coefficient values to improve non-rigid registration performance. This contextual information combined with a relaxation labeling process is used to search for a correspondence. Then the transformation is calculated by the thin plate spline (TPS) model. These two processes are iterated until the optimal correspondence and transformation are found. We have tested our proposed method and the state-of-the-art algorithms with synthetic data and bladder ultrasound images of in vivo human subjects. Experiments show that the proposed method improves registration performance significantly, as compared to other state-of-the-art non-rigid registration algorithms.

Background[18F]fluoro-2-deoxy-D-glucose ([18F]FDG) positron emission tomography (PET) is a valuable tool for monitoring response to therapy in oncology. In longitudinal studies, however, patients are not scanned in exactly the same position. Rigid and non-rigid image registration can be applied in order to reuse baseline volumes of interest (VOI) on consecutive studies of the same patient. The purpose of this study was to investigate the impact of various image registration strategies on standardized uptake value (SUV) and metabolic volume test-retest variability (TRT).MethodsTest-retest whole-body [18F]FDG PET/CT scans were collected retrospectively for 11 subjects with advanced gastrointestinal malignancies (colorectal carcinoma). Rigid and non-rigid image registration techniques with various degrees of locality were applied to PET, CT, and non-attenuation corrected PET (NAC) data. VOI were drawn independently on both test and retest scans. VOI drawn on test scans were projected onto retest scans and the overlap between projected VOI and manually drawn retest VOI was quantified using the Dice similarity coefficient (DSC). In addition, absolute (unsigned) differences in TRT of SUVmax, SUVmean, metabolic volume and total lesion glycolysis (TLG) were calculated in on one hand the test VOI and on the other hand the retest VOI and projected VOI. Reference values were obtained by delineating VOIs on both scans separately.ResultsNon-rigid PET registration showed the best performance (median DSC: 0.82, other methods: 0.71-0.81). Compared with the reference, none of the registration types showed significant absolute differences in TRT of SUVmax, SUVmean and TLG (p > 0.05). Only for absolute TRT of metabolic volume, significant lower values (p < 0.05) were observed for all registration strategies when compared to delineating VOIs separately, except for non-rigid PET registrations (p = 0.1). Non-rigid PET registration provided good volume TRT (7.7%) that was smaller than the reference (16%).ConclusionIn particular, non-rigid PET image registration showed good performance similar to delineating VOI on both scans separately, and with smaller TRT in metabolic volume estimates.

A non-rigid medical image registration method based on improved linear elastic model

The Spiking Cortical Model Based Structural Representations for Non-Rigid Multi-Modal Medical Image Registration

In this thesis, we focus on two important medical image processing and analysis problems: multimodal image registration and cerebrovascular segmentation. First, for the purpose of improving the robustness of image registration, we propose two novel registration objective functions and a new technique for better optimization performance. The first novel objective function is a combination of the multi-dimensional mutual information and an angle measure on the maximum distance-gradient (MDG) vector field. The MDG vector field is a new spatial feature field designed for registration tasks. It encodes both local edge information and globally defined spatial information related to the intensity difference, the distance, and the direction of a voxel to a certain object boundary point. This objective function then integrates both the magnitude and the orientation information of the MDG feature into the image registration process. The other novel objective function can be obtained if the statistical joint intensity mappings estimated from the pre-aligned training image pairs are available. This a priori knowledge can then be used as reference to guide the multi-resolution based registration process, by minimizing a new Kullback-Leibler distance (KLD) based similarity measure between the observed joint intensity distribution and the reference joint intensity distribution. To achieve better optimization, which evidently affects the robustness of a registration process, we propose a general technique based on random transformation disturbance to assist optimization procedures to avoid local optima, and hence to achieve robust optimization results. The technique is based on different properties of the local neighborhood of a global optimum and those of local optima. Second, this thesis presents a statistical segmentation technique for the extraction of vasculatures in three-dimensional rotational angiography (3D-RA). This method uses maximum intensity projections (MIP) to improve the accuracy and robustness of threshold estimation using the expectation maximization (EM) algorithm. It is fully automatic and computationally efficient. Finally, encouraging experimental results of the proposed approaches on both synthetic and clinical medical images are presented in the thesis.

Background: Non-rigid registration of medical images with intensity distortions is a difficult problem due to the change in pixel intensity. It is caused by contrast agent or intensity bias field. Methods: In some cases, this problem can be solved using Residual Complexity (RC) method. However, relative modification of parameter in residual complexity would result in completely different experimental effect. Another drawback is sensitivity to noise. To handle this problem, a new intensity-based similarity measure, Weighted Residual Complexity (WRC) has been proposed for effective medical image registration in this paper. Specifically, the local entropy image of two images is computed to be aligned respectively. Then, a weighting function using a function of the local entropy difference is modeled. The weighting function is used to weight the residual image in residual complexity adaptively. The residual image is defined as the difference between reference image and warped floating image. Results: The weighting function assigns smaller weight to residual image if the corresponding pixel value is larger in local entropy difference. The proposed technique was applied to simulative and real medical images. The contrast experiments were made with mutual information, diffeomorphic demons and residual complexity. Conclusion: Also, the analysis of experimental results was made qualitatively and quantitatively, which indicates that this new approach gives a better performance than diffeomorphic demons, mutual information and residual complexity. Keywords: Weighting function, local entropy, residual complexity, intensity distortion, image registration, WRC.