Multi-modality Image Registration Using Gradient Vector Flow Intensity

Maximization of mutual information (MI) (MMI) has a wide application in multimodal image registration. The conventional MMI algorithm is based on the statistics derived from image sampling and does not use spatial features. In this paper, we proposed an improved MMI registration algorithm that combines the spatial information through a feature-based selection mechanism. A Harris corner filter is chosen to classify the pixels. The classification results are used to focus sample selection, joint entropy estimation, and weighted MI information calculation. Although features play an important role, conventional feature matching is not used, which removes an important challenge to the use of feature-based information. The advantages of the proposed algorithm are improved robustness by incorporation of spatial features and significant reduction in computing time for registration. Experiments with challenging synthetic images and multimodal airborne infrared images are provided to demonstrate the improvement.

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

The multiresolution approach is commonly used to speed up the mutual-information (MI) based registration process. Conventionally, a Gaussian pyramid is often used as a multiresolution representation. However, in multi-modal medical image registration, MI-based methods with Gaussian pyramid may suffer from the problem of short capture ranges especially at the lower resolution levels. This paper proposes a novel and straightforward multimodal image registration method based on wavelet representation, in which two matching criteria are used including sum of difference (SAD) for improving the registration robustness and MI for assuring the registration accuracy. Experimental results show that the proposed method obtains a longer capture range than the traditional MI-based Gaussian pyramid method meanwhile maintaining comparable accuracy.

It is common for patients to undergo multiple tomographic imaging which provide complementary information. But variations in patient orientation, and differences in resolution and contrast of the modalities make it difficult for a clinician to mentally fuse all the image information accurately. There has been considerable interest in using image registration techniques to transfer all the information into a common coordinate frame. In this paper a maximization of mutual information based multi-modality medical image registration method is described. Mutual information (MI) is usually used to measure the statistical dependence between two random variables, or the amount of information that one variable contains about the other. The method applies mutual information to measure the information redundancy between the intensities of corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned. There exist many important technical issues to be solved about the method such as how to compute MI more accurately and how to obtain the maximization of MI, which are seldom mentioned in published papers. In this paper we provide some implementation issues, for example, subsampling, PV interpolation, outlier strategy. The combination of these computation techniques and searching strategy leads to a fast and accurate multi-modality image registration. The registration results of 3D human brain volume data of 41 CT-MR and 35 PET-MR from seven patients are validated to be subvoxel. The registration method is promising in clinical use.

In this thesis, we focus on two important medical image processing and analysis problems: multimodal image registration and cerebrovascular segmentation. First, for the purpose of improving the robustness of image registration, we propose two novel registration objective functions and a new technique for better optimization performance. The first novel objective function is a combination of the multi-dimensional mutual information and an angle measure on the maximum distance-gradient (MDG) vector field. The MDG vector field is a new spatial feature field designed for registration tasks. It encodes both local edge information and globally defined spatial information related to the intensity difference, the distance, and the direction of a voxel to a certain object boundary point. This objective function then integrates both the magnitude and the orientation information of the MDG feature into the image registration process. The other novel objective function can be obtained if the statistical joint intensity mappings estimated from the pre-aligned training image pairs are available. This a priori knowledge can then be used as reference to guide the multi-resolution based registration process, by minimizing a new Kullback-Leibler distance (KLD) based similarity measure between the observed joint intensity distribution and the reference joint intensity distribution. To achieve better optimization, which evidently affects the robustness of a registration process, we propose a general technique based on random transformation disturbance to assist optimization procedures to avoid local optima, and hence to achieve robust optimization results. The technique is based on different properties of the local neighborhood of a global optimum and those of local optima. Second, this thesis presents a statistical segmentation technique for the extraction of vasculatures in three-dimensional rotational angiography (3D-RA). This method uses maximum intensity projections (MIP) to improve the accuracy and robustness of threshold estimation using the expectation maximization (EM) algorithm. It is fully automatic and computationally efficient. Finally, encouraging experimental results of the proposed approaches on both synthetic and clinical medical images are presented in the thesis.

Joint segmentation and registration of images is a focused area of research nowadays. Jointly segmenting and registering noisy images and images having weak boundaries/intensity inhomogeneity is a challenging task. In medical image processing, joint segmentation and registration are essential methods that aid in distinguishing structures and aligning images for precise diagnosis and therapy. However, these methods encounter challenges, such as computational complexity and sensitivity to variations in image quality, which may reduce their effectiveness in real-world applications. Another major issue is still attaining effective joint segmentation and registration in the presence of artifacts or anatomical deformations. In this paper, a new nonparametric joint model is proposed for the segmentation and registration of multi-modality images having weak boundaries/noise. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is more efficient to segmenting and registering multi-modality images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is also tested on the images obtained from the freely available CHOAS dataset and compare the results of the proposed model with the other existing models using statistical measures such as the Jaccard similarity index, relative reduction, Dice similarity coefficient and Hausdorff distance. It can be seen that the proposed model outperforms the other existing models in terms of quantitatively and qualitatively.

Mutual information (MI) has been commonly used in multi-modal image registration. In this paper, we presented an optimal region selection method for MI based multi-modal image registration. Firstly, image preprocessing and initial registration were applied on the two images. Then we applied two region selection procedures to improve the performance of MI-based registration. The first procedure is hierarchical region detection that selects regions with more structure information in each of the two images. In this procedure, the images were divided into sub-images progressively, and candidate image blocks with high usability were selected based on entropy. The second procedure is optimal matching region selection based on local registration using MI. In this procedure, the regions selected in the first procedure were registered with MI, then the results are used to choose optimal matching regions with high reliability. Finally, the optimal regions were used for the entire image registration by computing the MI of the regions. The experiment results showed that this method could improve the efficiency and accuracy for MI-based image registration.

BackgroundMultimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality.ResultsWe developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system.ConclusionOur software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.

CoCycleReg: Collaborative cycle-consistency method for multi-modal medical image registration

Some similarity measures used in state-of-the-art multimodality image registration algorithms, (e.g., mutual information (MI)) have been shown to be suitable anatomical priors for maximum a posteriori reconstruction in emission tomography. Therefore, it is reasonable to assume that some originally designed anatomical priors may also be well suited for multimodality image registration. In this work, we evaluate the registration performance of three variants of an anatomical Markov prior, previously proposed by Bowsher et al. First, simulated data are used to verify whether the suggested registration criteria yield an optimum when an FDG positron emission tomography (PET) image and a T1-weighted magnetic resonance (MR) image of a human brain are perfectly aligned. Next, the registration accuracy of the proposed criteria is assessed for PET to MR and MR to PET registration of simulated human brain images, and compared to the accuracy reached by MI. Last, the new methods are applied to challenging measured rat and mouse brain data sets, consisting of low resolution FDG microPET images and high resolution microMR images with a strong bias field. It was shown that the anatomy-based Markov priors indeed yield a well-defined optimum for aligned PET-MR images and that similar registration accuracy can be achieved as with MI, especially for registration to MR images suffering from a bias field. Nevertheless, in contrast to MI, the new criteria usually require a good initial guess of the transformation parameters in order not to get stuck in a local optimum. The proposed methods are shown to be superior to MI for registering measured microMR brain images with a strong bias field to FDG microPET images if a good initialization is provided.

A particular problem in image registration arises for multi-modal images taken from different imaging devices and/or modalities. Starting in 1995, mutual information has shown to be a very successful distance measure for multi-modal image registration. Therefore, mutual information is considered to be the state-of-the-art approach to multi-modal image registration. However, mutual information has also a number of well-known drawbacks. Its main disadvantage is that it is known to be highly non-convex and has typically many local maxima. This observation motivates us to seek a different image similarity measure which is better suited for optimization but as well capable to handle multi-modal images. In this work, we investigate an alternative distance measure which is based on normalized gradients. As we show, the alternative approach is deterministic, much simpler, easier to interpret, fast and straightforward to implement, faster to compute, and also much more suitable to numerical optimization.

Deep learning-based affine medical image registration for multimodal minimal-invasive image-guided interventions – A comparative study on generalizability

This paper addresses the problem of estimating the 3D rigid pose of a CT volume of an object from its 2D X-ray projections. We use maximization of mutual information, an accurate similarity measure for multi-modal and mono-modal image registration tasks. However, it is known that the standard mutual information measure only takes intensity values into account without considering spatial information and its robustness is questionable. In this paper, instead of directly maximizing mutual information, we propose to use a variational approximation derived from the Kullback-Leibler bound. Spatial information is then incorporated into this variational approximation using a Markov random field model. The newly derived similarity measure has a least-squares form and can be effectively minimized by a multi-resolution Levenberg-Marquardt optimizer. Experimental results are presented on X-ray and CT datasets of a plastic phantom and a cadaveric spine segment.

A new approach to the problem of multimodality medical image registration is proposed, using a basic concept from information theory, mutual information (MI), or relative entropy, as a new matching criterion. The method presented in this paper applies MI to measure the statistical dependence or information redundancy between the image intensities of corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned. Maximization of MI is a very general and powerful criterion, because no assumptions are made regarding the nature of this dependence and no limiting constraints are imposed on the image content of the modalities involved. The accuracy of the MI criterion is validated for rigid body registration of computed tomography (CT), magnetic resonance (MR), and photon emission tomography (PET) images by comparison with the stereotactic registration solution, while robustness is evaluated with respect to implementation issues, such as interpolation and optimization, and image content, including partial overlap and image degradation. Our results demonstrate that subvoxel accuracy with respect to the stereotactic reference solution can be achieved completely automatically and without any prior segmentation, feature extraction, or other preprocessing steps which makes this method very well suited for clinical applications.

Mutual information (MI) is a very important concept in medical image registration for a diagnosis. During the past decade, there have been a large number of studies on multimodal image registration. This extensive attention is due to its high performance in terms of accuracy and robustness. However, there are so many different perspectives on MI that have been discussed and proposed during the last two decades. As a result, it is very difficult to have a holistic view of the field and to select an appropriate method to solve a specific problem in medical image analysis. In order to solve this problem, this paper proposes a novel taxonomy of MI approaches used in medical image registration. In our work, we extensively and comprehensively investigate works proposed in the literature and systematically classify them based on methods of computational processing. Our taxonomy will help to enhance an understanding of various principles of mutual information.