Deep learning-based affine medical image registration for multimodal minimal-invasive image-guided interventions – A comparative study on generalizability

Deep learning based data-adaptive descriptor for non-rigid multi-modal medical image registration

Improvement of medical diagnosis, aided computer surgeries and tumor identification requires an accurate image registration approaches. The registration of multimodal medical images is more complicated than the registration of unimodal medical images due to the variation in luminance between the images. In this paper, an accurate multimodal image registration approach using artificial immune system (AIS) is proposed and the affine transformation model is used in contrast to the most of the related works which assumed rigid transformation model or similarity transformation model. In the proposed approach the LL bands of the discrete wavelet transform (DWT) for the images are used and the normalized mutual information (NMI) is used as a fitness function. The proposed approach achieves good result in the case of noiseless images, noisy images and partial data loss from one of the images. Moreover, the proposed approach does not need any feature extraction or refinement step. To demonstrate the robustness of the proposed approach, it has been compared with two multimodal medical image registration approaches.

Medical image registration is an integral part of various clinical applications including image guidance, motion tracking, therapy assessment and diagnosis. We present a robust approach for mono-modal and multi-modal medical image registration. To this end, we propose the novel shape operator based local image distance (SOLID) which estimates the similarity of images by comparing their second-order curvature information. Our similarity metric is rigorously tailored to be suitable for comparing images from different medical imaging modalities or image contrasts. A critical element of our method is the extraction of local features using higher-order shape information, enabling the accurate identification and registration of smaller structures. In order to assess the efficacy of the proposed similarity metric, we have implemented a variational image registration algorithm that relies on the principle of matching the curvature information of the given images. The performance of the proposed algorithm has been evaluated against various alternative state-of-the-art variational registration algorithms. Our experiments involve mono-modal as well as multi-modal and cross-contrast co-registration tasks in a broad variety of anatomical regions. Compared to the evaluated alternative registration methods, the results indicate a very favorable accuracy, precision and robustness of the proposed SOLID method in various highly challenging registration tasks.

In this thesis, we focus on two important medical image processing and analysis problems: multimodal image registration and cerebrovascular segmentation. First, for the purpose of improving the robustness of image registration, we propose two novel registration objective functions and a new technique for better optimization performance. The first novel objective function is a combination of the multi-dimensional mutual information and an angle measure on the maximum distance-gradient (MDG) vector field. The MDG vector field is a new spatial feature field designed for registration tasks. It encodes both local edge information and globally defined spatial information related to the intensity difference, the distance, and the direction of a voxel to a certain object boundary point. This objective function then integrates both the magnitude and the orientation information of the MDG feature into the image registration process. The other novel objective function can be obtained if the statistical joint intensity mappings estimated from the pre-aligned training image pairs are available. This a priori knowledge can then be used as reference to guide the multi-resolution based registration process, by minimizing a new Kullback-Leibler distance (KLD) based similarity measure between the observed joint intensity distribution and the reference joint intensity distribution. To achieve better optimization, which evidently affects the robustness of a registration process, we propose a general technique based on random transformation disturbance to assist optimization procedures to avoid local optima, and hence to achieve robust optimization results. The technique is based on different properties of the local neighborhood of a global optimum and those of local optima. Second, this thesis presents a statistical segmentation technique for the extraction of vasculatures in three-dimensional rotational angiography (3D-RA). This method uses maximum intensity projections (MIP) to improve the accuracy and robustness of threshold estimation using the expectation maximization (EM) algorithm. It is fully automatic and computationally efficient. Finally, encouraging experimental results of the proposed approaches on both synthetic and clinical medical images are presented in the thesis.

Mutual information (MI) is a very important concept in medical image registration for a diagnosis. During the past decade, there have been a large number of studies on multimodal image registration. This extensive attention is due to its high performance in terms of accuracy and robustness. However, there are so many different perspectives on MI that have been discussed and proposed during the last two decades. As a result, it is very difficult to have a holistic view of the field and to select an appropriate method to solve a specific problem in medical image analysis. In order to solve this problem, this paper proposes a novel taxonomy of MI approaches used in medical image registration. In our work, we extensively and comprehensively investigate works proposed in the literature and systematically classify them based on methods of computational processing. Our taxonomy will help to enhance an understanding of various principles of mutual information.

CoCycleReg: Collaborative cycle-consistency method for multi-modal medical image registration

Joint segmentation and registration of images is a focused area of research nowadays. Jointly segmenting and registering noisy images and images having weak boundaries/intensity inhomogeneity is a challenging task. In medical image processing, joint segmentation and registration are essential methods that aid in distinguishing structures and aligning images for precise diagnosis and therapy. However, these methods encounter challenges, such as computational complexity and sensitivity to variations in image quality, which may reduce their effectiveness in real-world applications. Another major issue is still attaining effective joint segmentation and registration in the presence of artifacts or anatomical deformations. In this paper, a new nonparametric joint model is proposed for the segmentation and registration of multi-modality images having weak boundaries/noise. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is more efficient to segmenting and registering multi-modality images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is also tested on the images obtained from the freely available CHOAS dataset and compare the results of the proposed model with the other existing models using statistical measures such as the Jaccard similarity index, relative reduction, Dice similarity coefficient and Hausdorff distance. It can be seen that the proposed model outperforms the other existing models in terms of quantitatively and qualitatively.

Nonrigid multimodal image registration remains a challenging task in medical image processing and analysis. The structural representation (SR)-based registration methods have attracted much attention recently. However, the existing SR methods cannot provide satisfactory registration accuracy due to the utilization of hand-designed features for structural representation. To address this problem, the structural representation method based on the improved version of the simple deep learning network named PCANet is proposed for medical image registration. In the proposed method, PCANet is firstly trained on numerous medical images to learn convolution kernels for this network. Then, a pair of input medical images to be registered is processed by the learned PCANet. The features extracted by various layers in the PCANet are fused to produce multilevel features. The structural representation images are constructed for two input images based on nonlinear transformation of these multilevel features. The Euclidean distance between structural representation images is calculated and used as the similarity metrics. The objective function defined by the similarity metrics is optimized by L-BFGS method to obtain parameters of the free-form deformation (FFD) model. Extensive experiments on simulated and real multimodal image datasets show that compared with the state-of-the-art registration methods, such as modality-independent neighborhood descriptor (MIND), normalized mutual information (NMI), Weber local descriptor (WLD), and the sum of squared differences on entropy images (ESSD), the proposed method provides better registration performance in terms of target registration error (TRE) and subjective human vision.

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

A relatively large class of information theoretical measures, including e.g. mutual information or normalized entropy, has been used in multi-modal medical image registration. Even though the mathematical foundations of the different measures were very similar, the final expressions turned out to be surprisingly different. Therefore one of the main aims of this paper is to enlight the relationship of different objective functions by introducing a mathematical framework from which several known optimization objectives can be deduced. Furthermore we extend existing measures in order to be applicable on image features different than image intensities and introduce efficiency as a very general concept to qualify such features. The presented framework is very general and not at all restricted to medical images. Still we want to discuss the possible impact of our theoretical framework for the particular problem of medical image registration, where the feature space has traditionally been fixed to image intensities. Our theoretical approach is very general though and can be used for any kind of multi-modal signals, such as for the broad field of multi-media applications.

Analysis of multispectral or multitemporal images requires proper geometric alignment of the images to compare corresponding regions in each image volume. Retrospective three-dimensional alignment or registration of multimodal medical images based on features intrinsic to the image data itself is complicated by their different photometric properties, by the complexity of the anatomical objects in the scene and by the large variety of clinical applications in which registration is involved. While the accuracy of registration approaches based on matching of anatomical landmarks or object surfaces suffers from segmentation errors, voxel-based approaches consider all voxels in the image without the need for segmentation. The recent introduction of the criterion of maximization of mutual information, a basic concept from information theory, has proven to be a breakthrough in the field. While solutions for intrapatient affine. registration based on this concept are already commercially available, current research in the field focuses on interpatient nonrigid matching.

Image registration is an elementary task in medical image processing and analysis, which can be divided into monomodal and multimodal. Direct 3D multimodal registration in volumetric medical images can provide more insight into the interpretation of subsequent image processing applications than 2D methods. This paper is dedicated to the development of a 3D multimodal image registration algorithm based on a viscous fluid model associated with the Bhattacharyya distance. In our approach, a modified Navier-Stoke's equation is exploited as the foundation of the multimodal image registration framework. The hopscotch method is numerically implemented to solve the velocity field, whose values at the explicit locations are first computed and the values at the implicit positions are solved by transposition. The differential of the Bhattacharyya distance is incorporated into the body force function, which is the main driving force for deformation, to enable multimodal registration. A variety of simulated and real brain MR images were utilized to assess the proposed 3D multimodal image registration system. Preliminary experimental results indicated that our algorithm produced high registration accuracy in various registration scenarios and outperformed other competing methods in many multimodal image registration tasks.Clinical Relevance- This facilitates the disease diagnosis and treatment planning that requires accurate 3D multimodal image registration without massive image data and extensive training regardless of the imaging modality.

Notwithstanding the widespread use of image guided neurosurgery systems in recent years, the accuracy of these systems is strongly limited by the intra-operative deformation of the brain tissue, the so-called brain shift. Intra-operative ultrasound (iUS) imaging as an effective solution to compensate complex brain shift phenomena update patients coordinate during surgery by registration of the intra-operative ultrasound and the pre-operative MRI data that is a challenging problem.In this work a non-rigid multimodal image registration technique based on co-sparse analysis model is proposed. This model captures the interdependency of two image modalities; MRI as an intensity image and iUS as a depth image. Based on this model, the transformation between the two modalities is minimized by using a bimodal pair of analysis operators which are learned by optimizing a joint co-sparsity function using a conjugate gradient.Experimental validation of our algorithm confirms that our registration approach outperforms several of other state-of-the-art registration methods quantitatively. The evaluation was performed using seven patient dataset with the mean registration error of only 1.83 mm. Our intensity-based co-sparse analysis model has improved the accuracy of non-rigid multimodal medical image registration by 15.37% compared to the curvelet based residual complexity as a powerful registration method, in a computational time compatible with clinical use.

Mutual Information (MI) has been extensively used as a similarity measure in image registration and motion estimation, and it is particularly robust for 3D multimodal medical image registration. However, MI estimators are known i) to have a high variance and ii) to be computationally costly. In order to overcome these drawbacks, we propose a new similarity measure based on an Edgeworth-based third order expansion of MI and named 3-EMI in the following. This kind of approximation is well known in signal processing, and especially in Independent Components Analysis (ICA), but its computation is easier since data can be prewhitened contrary to images in registration. The performance of affine and non-rigid registrations based on the 3-EMI metric is studied through computer results in the context of cardiac multislice computed tomography. In fact, an estimate of the 3-EMI metric using sample statistics is compared to a histogram-based estimate of the standard normalized MI, showing a better robustness of the 3-EMI measure with respect to the range of the searched deformation. In addition, in practice, one part of the floating image may be missing regarding the reference image. Computer results show that our approach is less sensitive to such a practical problem.

Over the past few decades, fast<strong>-</strong>growth has occurred in the area of medical image acquisition devices, and physicians now rely on the utilization of medical images for the diagnosis, treatment plans, and surgical guidance. Researchers have classified medical images according to two structures: anatomical and functional structures. Due to this classification, the data obtained from two or more images of the same object frequently provide complementary and more abundant information through a process known as multimodal medical model registration. Image registration is spatially mapping the coordinate system of the two images obtained from a different viewpoint and utilizing various sensors. Several automatic multimodal medical image registration algorithms have been introduced based on types of medical images and their applications to increase the reliability, robustness, and accuracy. Due to the diversity in imaging and the different demands<strong> </strong>for applications, there is no single registration algorithm that can do that. This paper introduces a novel method for developing a multimodal medical image registration system that can select the most accepted registration algorithm from a group of registration algorithms for a variety of input datasets. The method described here is based on a machine learning technique that selects the most promising candidate. Several experiments have been conducted, and the results reveal that the novel approach leads to considerably faster reliability, accuracy, and more robustness registration algorithm selection.