A Spatial-Feature-Enhanced MMI Algorithm for Multimodal Airborne Image Registration

It is common for patients to undergo multiple tomographic imaging which provide complementary information. But variations in patient orientation, and differences in resolution and contrast of the modalities make it difficult for a clinician to mentally fuse all the image information accurately. There has been considerable interest in using image registration techniques to transfer all the information into a common coordinate frame. In this paper a maximization of mutual information based multi-modality medical image registration method is described. Mutual information (MI) is usually used to measure the statistical dependence between two random variables, or the amount of information that one variable contains about the other. The method applies mutual information to measure the information redundancy between the intensities of corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned. There exist many important technical issues to be solved about the method such as how to compute MI more accurately and how to obtain the maximization of MI, which are seldom mentioned in published papers. In this paper we provide some implementation issues, for example, subsampling, PV interpolation, outlier strategy. The combination of these computation techniques and searching strategy leads to a fast and accurate multi-modality image registration. The registration results of 3D human brain volume data of 41 CT-MR and 35 PET-MR from seven patients are validated to be subvoxel. The registration method is promising in clinical use.

Similarity measure plays a critical role in image registration. Mutual information (MI) has been proved to be a promising measure used widely in multi-modality image registration. However, applying mutual information to original intensities only takes statistical information into consideration, while spatial information is not even considered. In this paper, a novel approach is proposed to incorporate spatial information into MI through gradient vector flow (GVF). Mutual information now is calculated from the GVF-intensity (GVFI) map of the original images instead of their intensity values. Multi-modality brain image registration was performed to test the accuracy and robustness of the proposed method. Experimental results showed that the success rate of our method is higher than that of traditional MI-based registration.

A new approach to the problem of multimodality medical image registration is proposed, using a basic concept from information theory, mutual information (MI), or relative entropy, as a new matching criterion. The method presented in this paper applies MI to measure the statistical dependence or information redundancy between the image intensities of corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned. Maximization of MI is a very general and powerful criterion, because no assumptions are made regarding the nature of this dependence and no limiting constraints are imposed on the image content of the modalities involved. The accuracy of the MI criterion is validated for rigid body registration of computed tomography (CT), magnetic resonance (MR), and photon emission tomography (PET) images by comparison with the stereotactic registration solution, while robustness is evaluated with respect to implementation issues, such as interpolation and optimization, and image content, including partial overlap and image degradation. Our results demonstrate that subvoxel accuracy with respect to the stereotactic reference solution can be achieved completely automatically and without any prior segmentation, feature extraction, or other preprocessing steps which makes this method very well suited for clinical applications.

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

Over the past few decades, fast<strong>-</strong>growth has occurred in the area of medical image acquisition devices, and physicians now rely on the utilization of medical images for the diagnosis, treatment plans, and surgical guidance. Researchers have classified medical images according to two structures: anatomical and functional structures. Due to this classification, the data obtained from two or more images of the same object frequently provide complementary and more abundant information through a process known as multimodal medical model registration. Image registration is spatially mapping the coordinate system of the two images obtained from a different viewpoint and utilizing various sensors. Several automatic multimodal medical image registration algorithms have been introduced based on types of medical images and their applications to increase the reliability, robustness, and accuracy. Due to the diversity in imaging and the different demands<strong> </strong>for applications, there is no single registration algorithm that can do that. This paper introduces a novel method for developing a multimodal medical image registration system that can select the most accepted registration algorithm from a group of registration algorithms for a variety of input datasets. The method described here is based on a machine learning technique that selects the most promising candidate. Several experiments have been conducted, and the results reveal that the novel approach leads to considerably faster reliability, accuracy, and more robustness registration algorithm selection.

To investigate direct volumetric in vivo correspondence of calcified atherosclerotic plaque lesions in MRI and CT images of the thoracic aorta by multimodality image registration and fusion. Twelve CT (11 noncontrast and one contrast) and MRI (TruFISP, contrast T1-weighted volumetric interpolated breath-hold examination (VIBE)) data sets were co-registered by approximate segmentation of the aorta and subsequent automatic co-registration by maximization of mutual information (MI). We quantitatively assessed 22 co-registered calcified plaque lesions on CT and MRI. The three-dimensional registration consistency and accuracy were 1.74 +/- 1.3 mm, and 2.42 +/- 1.65 mm, respectively. The ratio of CT/MRI calcified plaque volume decreased asymptotically with MRI volume, and correlated with average CT lesion density (r = 0.72) for small lesions (<25 mm(3)). The average calcified plaque volume, circumferential extent, and maximal radial width by MRI were significantly smaller compared to CT (35%, 68%, and 53%, respectively; P < 0.05). Software co-registration allowed precise, direct, and voxel-based comparison of calcified atherosclerotic plaque lesions imaged by MRI and CT. In comparison with co-registered MRI, overestimation of calcified plaque in aortic CT due to "blooming" correlates with the average lesion density for small plaques, and is greater for small plaques.

In this thesis, we focus on two important medical image processing and analysis problems: multimodal image registration and cerebrovascular segmentation. First, for the purpose of improving the robustness of image registration, we propose two novel registration objective functions and a new technique for better optimization performance. The first novel objective function is a combination of the multi-dimensional mutual information and an angle measure on the maximum distance-gradient (MDG) vector field. The MDG vector field is a new spatial feature field designed for registration tasks. It encodes both local edge information and globally defined spatial information related to the intensity difference, the distance, and the direction of a voxel to a certain object boundary point. This objective function then integrates both the magnitude and the orientation information of the MDG feature into the image registration process. The other novel objective function can be obtained if the statistical joint intensity mappings estimated from the pre-aligned training image pairs are available. This a priori knowledge can then be used as reference to guide the multi-resolution based registration process, by minimizing a new Kullback-Leibler distance (KLD) based similarity measure between the observed joint intensity distribution and the reference joint intensity distribution. To achieve better optimization, which evidently affects the robustness of a registration process, we propose a general technique based on random transformation disturbance to assist optimization procedures to avoid local optima, and hence to achieve robust optimization results. The technique is based on different properties of the local neighborhood of a global optimum and those of local optima. Second, this thesis presents a statistical segmentation technique for the extraction of vasculatures in three-dimensional rotational angiography (3D-RA). This method uses maximum intensity projections (MIP) to improve the accuracy and robustness of threshold estimation using the expectation maximization (EM) algorithm. It is fully automatic and computationally efficient. Finally, encouraging experimental results of the proposed approaches on both synthetic and clinical medical images are presented in the thesis.

The multiresolution approach is commonly used to speed up the mutual-information (MI) based registration process. Conventionally, a Gaussian pyramid is often used as a multiresolution representation. However, in multi-modal medical image registration, MI-based methods with Gaussian pyramid may suffer from the problem of short capture ranges especially at the lower resolution levels. This paper proposes a novel and straightforward multimodal image registration method based on wavelet representation, in which two matching criteria are used including sum of difference (SAD) for improving the registration robustness and MI for assuring the registration accuracy. Experimental results show that the proposed method obtains a longer capture range than the traditional MI-based Gaussian pyramid method meanwhile maintaining comparable accuracy.

Joint segmentation and registration of images is a focused area of research nowadays. Jointly segmenting and registering noisy images and images having weak boundaries/intensity inhomogeneity is a challenging task. In medical image processing, joint segmentation and registration are essential methods that aid in distinguishing structures and aligning images for precise diagnosis and therapy. However, these methods encounter challenges, such as computational complexity and sensitivity to variations in image quality, which may reduce their effectiveness in real-world applications. Another major issue is still attaining effective joint segmentation and registration in the presence of artifacts or anatomical deformations. In this paper, a new nonparametric joint model is proposed for the segmentation and registration of multi-modality images having weak boundaries/noise. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is more efficient to segmenting and registering multi-modality images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is also tested on the images obtained from the freely available CHOAS dataset and compare the results of the proposed model with the other existing models using statistical measures such as the Jaccard similarity index, relative reduction, Dice similarity coefficient and Hausdorff distance. It can be seen that the proposed model outperforms the other existing models in terms of quantitatively and qualitatively.

BackgroundMultimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality.ResultsWe developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system.ConclusionOur software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.

CoCycleReg: Collaborative cycle-consistency method for multi-modal medical image registration

This paper addresses the problem of estimating the 3D rigid pose of a CT volume of an object from its 2D X-ray projections. We use maximization of mutual information, an accurate similarity measure for multi-modal and mono-modal image registration tasks. However, it is known that the standard mutual information measure only takes intensity values into account without considering spatial information and its robustness is questionable. In this paper, instead of directly maximizing mutual information, we propose to use a variational approximation derived from the Kullback-Leibler bound. Spatial information is then incorporated into this variational approximation using a Markov random field model. The newly derived similarity measure has a least-squares form and can be effectively minimized by a multi-resolution Levenberg-Marquardt optimizer. Experimental results are presented on X-ray and CT datasets of a plastic phantom and a cadaveric spine segment.

Standard Mutual Information function contains local maxima, which make against to convergence of registration transformation parameters for automated multimodality image registration problems. We proposed Feature Potential Mutual Information (FPMI) to increases the smoothness of the registration measure function and use Particle Swarm Optimization to search the optimal registration transformation parameter in this paper. At first, Edges of images are detected. Next, edge feature potential is defined by expanding edges to the neighborhood region using potential function. Each edge point influences the whole potential field, just like the particle of physics in the gravitation field space. FPMI is computed on the edge feature potential of two images. It substitutes the edge feature potential values for gray values in images. It can avoid great change of joint probability distribution and has less local maxima. The registration accuracy of FPMI is analyzed under different edge detection cases. It is shown that the registration accuracy of FPMI is more accurate and more robust than that of MI. Maximization of FPMI is done by PSO. PSO combines local search methods with global search methods, attempting to balance exploration and exploitation. Its complex behavior follows from a few simple rules and has less computational complexity. Multimodal medical images are used to compare the response of FPMI and MI to translation and rotation. Experiments show that FPMI is smoother and has less local fluctuations than that of MI. Registration results show that PSO do it better than Powell’s method to search the optimal registration parameters.

Some similarity measures used in state-of-the-art multimodality image registration algorithms, (e.g., mutual information (MI)) have been shown to be suitable anatomical priors for maximum a posteriori reconstruction in emission tomography. Therefore, it is reasonable to assume that some originally designed anatomical priors may also be well suited for multimodality image registration. In this work, we evaluate the registration performance of three variants of an anatomical Markov prior, previously proposed by Bowsher et al. First, simulated data are used to verify whether the suggested registration criteria yield an optimum when an FDG positron emission tomography (PET) image and a T1-weighted magnetic resonance (MR) image of a human brain are perfectly aligned. Next, the registration accuracy of the proposed criteria is assessed for PET to MR and MR to PET registration of simulated human brain images, and compared to the accuracy reached by MI. Last, the new methods are applied to challenging measured rat and mouse brain data sets, consisting of low resolution FDG microPET images and high resolution microMR images with a strong bias field. It was shown that the anatomy-based Markov priors indeed yield a well-defined optimum for aligned PET-MR images and that similar registration accuracy can be achieved as with MI, especially for registration to MR images suffering from a bias field. Nevertheless, in contrast to MI, the new criteria usually require a good initial guess of the transformation parameters in order not to get stuck in a local optimum. The proposed methods are shown to be superior to MI for registering measured microMR brain images with a strong bias field to FDG microPET images if a good initialization is provided.

Mutual information (MI) has been commonly used in multi-modal image registration. In this paper, we presented an optimal region selection method for MI based multi-modal image registration. Firstly, image preprocessing and initial registration were applied on the two images. Then we applied two region selection procedures to improve the performance of MI-based registration. The first procedure is hierarchical region detection that selects regions with more structure information in each of the two images. In this procedure, the images were divided into sub-images progressively, and candidate image blocks with high usability were selected based on entropy. The second procedure is optimal matching region selection based on local registration using MI. In this procedure, the regions selected in the first procedure were registered with MI, then the results are used to choose optimal matching regions with high reliability. Finally, the optimal regions were used for the entire image registration by computing the MI of the regions. The experiment results showed that this method could improve the efficiency and accuracy for MI-based image registration.