Multimodal Image Registration Using a Viscous Fluid Model with the Bhattacharyya Distance.

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

Multi-modal image registration has been a challenging task in medical images because of the complex intensity relationship between images to be aligned. Registration methods often rely on the statistical intensity relationship between the images which suffers from problems such as statistical insufficiency. The proposed registration method works based on extracting structural features by utilizing the complex phase and gradient-based information. By employing structural relationships between different modalities instead of complex similarity measures, the multi-modal registration problem is converted into a mono-modal one. Therefore, conventional mono-modal similarity measures can be utilized to evaluate the registration results. This new registration paradigm has been tested on magnetic resonance (MR) brain images of different modes. The method has been evaluated based on target registration error (TRE) to determine alignment accuracy. Quantitative results demonstrate that the proposed method is capable of achieving comparable registration accuracy compared to the conventional mutual information.

Multimodal medical image deformable registration is a critical foundation for liver tumor interventional therapy, assisting doctors in path planning and ablation efficacy evaluation. Research on this topic is still limited, especially regarding the challenges in liver multimodal deformable registration, such as significant image intensity differences and large organ deformations. In this paper, we propose DMCR--a novel multimodal multiscale image registration model based on a dual-stream Mamba with contrastive representation. During the encoding stage, images from different modalities are fed into distinct feature branches constructed using the Mamba architecture. In the registration stage, we propose a multiscale registration approach based on a decoding strategy, wherein the deformation field is progressively propagated and fused across multiple scales. Furthermore, we introduce a modality-invariant contrastive loss to guide the model in capturing intrinsic image features during the encoding stage, while disregarding modality-specific details, thus enhancing the effectiveness of subsequent registration. We trained and tested our model on multimodal liver ablation datasets from three different medical centers. The results demonstrate that our proposed model achieves superior registration performance compared to state-of-the-art registration methods and also exhibits better generalization capability.Clinical relevance- This model effectively performs multi-modal image deformable registration, assisting physicians in facilitating interventional procedures and evaluating multimodal therapeutic outcomes.

Over the past few decades, fast<strong>-</strong>growth has occurred in the area of medical image acquisition devices, and physicians now rely on the utilization of medical images for the diagnosis, treatment plans, and surgical guidance. Researchers have classified medical images according to two structures: anatomical and functional structures. Due to this classification, the data obtained from two or more images of the same object frequently provide complementary and more abundant information through a process known as multimodal medical model registration. Image registration is spatially mapping the coordinate system of the two images obtained from a different viewpoint and utilizing various sensors. Several automatic multimodal medical image registration algorithms have been introduced based on types of medical images and their applications to increase the reliability, robustness, and accuracy. Due to the diversity in imaging and the different demands<strong> </strong>for applications, there is no single registration algorithm that can do that. This paper introduces a novel method for developing a multimodal medical image registration system that can select the most accepted registration algorithm from a group of registration algorithms for a variety of input datasets. The method described here is based on a machine learning technique that selects the most promising candidate. Several experiments have been conducted, and the results reveal that the novel approach leads to considerably faster reliability, accuracy, and more robustness registration algorithm selection.

Real-time multi-modal rigid registration based on a novel symmetric-SIFT descriptor

textabstractMultimodal groupwise registration has been of growing interest to the image processing community due to developments in scanner technologies (e.g. multiparametric MRI, DCE-CT or PET-MR) that increased both the number of modalities and number of images under consideration. In this work a novel methodology is presented for multimodal groupwise registration that is based on Laplacian eigenmaps, a nonlinear dimensionality reduction technique. Compared to recently proposed dissimilarity metrics based on principal component analysis, the proposed metric should enable a better capture of the intensity relationships between different images in the group. The metric is constructed to be the second smallest eigenvalue from the eigenvector problem defined in Laplacian eigenmaps. The method was validated in three distinct experiments: a non-linear synthetic registration experiment, the registration of quantitative MRI data of the carotid artery, and the registration of multimodal data of the brain (RIRE). The results show increased accuracy and robustness compared to other state-of-the-art groupwise registration methodologies.

We present an instrument based on commodity embedded hardware, that implements an automatic procedure for early skin-cancer screening using dynamic thermal imaging. The procedure leverages image segmentation in the visible range and real-time multimodal registration to compute the temperature recovery curve (TRC) of suspicious skin lesions using thermal infrared video. The instrument implements two algorithms that infer the malignancy of the lesion from the computed TRCs. The first algorithm assumes that the TRCs are deterministic and infers the malignancy from the distance between the TRC of the suspicious lesion and its surrounding skin, which is assumed to be healthy tissue. The second algorithm models the TRC of the lesion as a random process and uses detection theory to statistically infer its malignancy from the eigenfunctions and corresponding eigenvalues of its covariance function. We built a prototype of the instrument using a Raspberry Pi 3 model B+ board, which acquires a visible-range image of the lesion at the beginning of the procedure and performs image segmentation in 62ms. Operating on a 400×400-pixel region-of-interest within the infrared video, the board performs frame-by-frame multimodal image registration and generates the TRCs in real time at more than 37 frames per second, thus eliminating the need to store video data for off-line processing. The statistical detection algorithm, which yields the best results, runs in 1.07s at the end of the procedure, and achieves a sensitivity of 98% and a specificity of 95% on a dataset of 116 volunteer subjects.

The synthesis of pseudo-CT images is of particular interest in the development of hybrid PET-MRI devices and MRI-guided radiotherapy. These images can be used for attenuation correction during PET image reconstruction. Furthermore, using MRI-based radiotherapy planning would enable a more accurate dosimetry planning due to the superior soft tissue contrast of the scans. The previously proposed methods for pseudo-CT synthesis are characterised by mainly two drawbacks. First, most proposed methods are limited to the head and neck region and therefore not feasible in case of whole body applications. Second, the presence of aligned training pairs of both MRI and CT scans for a number of subjects is assumed. In this work, we present preliminary results for atlas-based approaches using multiple CT atlas scans (from different patients) to synthesise a pseudo-CT image for a new patient using only their MRI data. This application requires accurate and robust deformable multimodal registration. We employed a recent discrete optimisation registration framework together with a self-similarity-based metric to accurately match the CT atlases to the anatomy of the patient. The registered atlases are then jointly combined by means of local fusion strategies. We apply our method to different 3D whole body MRI scans and a total of 18 3D whole body CT atlases. In addition to intensity fusion, the proposed methods can also be used for label fusion. Since evaluation based directly on synthesised intensity values is problematic, we use the Dice overlap after the fusion of segmentation labels as a proxy measure. Our proposed new method, which uses MIND descriptors for multimodal label fusion shows overall the best results.

Multimodal registration is a challenging problem in visual computing, commonly faced during medical image-guided interventions, data fusion and 3D object retrieval. The main challenge of multimodal registration is finding accurate correspondence between modalities, since different modalities do not exhibit the same characteristics. This paper explores how the coherence of different modalities can be utilized for the challenging task of 3D multimodal registration. A novel deep learning multimodal registration framework is proposed by introducing a siamese deep learning architecture, especially designed for aligning and fusing modalities of different structural and physical principles. The cross-modal attention blocks lead the network to establish correspondences between features of different modalities. The proposed framework focuses on the alignment of 3D point clouds and the micro-CT 3D volumes of the same object. A multimodal dataset consisting of real micro-CT scans and their synthetically generated 3D models (point clouds) is presented and utilized for evaluating our methodology.

With respect to the demands of adaptive and 4D-radiotherapy applications, an algorithm is proposed for a fully automatic, multimodality deformable registration that follows the concept of translational relocation of regularly distributed image subvolumes governed by local anatomical features. Thereby, the problem of global deformable registration is broken down to multiple independent local registration steps which allows for straightforward parallelization of the algorithm. In a subsequent step, possible local misregistrations are corrected for by minimization of the elastic energy of the displacement field under consideration of image information. The final displacement field results from interpolation of the subvolume shift vectors. The algorithm can employ as a similarity measure both the correlation coefficient and mutual information. The latter allows the application to intermodality deformable registration problems. The typical calculation time on a modern multiprocessor PC is well below 1 min, which facilitates almost-interactive, "online" usage. CT-to-MRI and CT-to-cone-beam-CT registrations of head-and-neck data sets are presented, as well as inhale-to-exhale registrations of lung CT data sets. For quantitative evaluation of registration accuracy, a virtual thorax phantom was developed; additionally, a landmark-based evaluation on four lung respiratory-correlated CT data sets was performed. This consistently resulted in average registration residuals on the order of the voxel size or less (3D-residuals approximately 1-2 mm). Summarizing, the presented algorithm allows an accurate multimodality deformable registration with calculation times well below 1 min, and thus bears promise as a versatile basic tool in adaptive and 4D-radiotherapy applications.

Multi-modal registration is a fundamental step for many medical imaging procedures. In this paper, the sum of conditional variance (SCV) similarity measure is proposed for 3D multi-modal medical image registration. The SCV similarity measure is based on minimizing the sum of conditional variances that are calculated using the joint histogram of the two images to be registered. Standard Gauss-Newton optimization is used to automatically minimize this measure which allows fast computational time and high accuracy. Experimental results show that our proposed approach is robust, computationally efficient and also more accurate when compared with the standard mutual information (MI) based approach and also the recently proposed sum-of-squared-difference on entropy images (eSSD) approach.

This work reports a deep-learning based registration algorithm that aligns multi-modal retinal images collected from longitudinal clinical studies to achieve accuracy and robustness required for analysis of structural changes in large-scale clinical data. Deep-learning networks that mirror the architecture of conventional feature-point-based registration were evaluated with different networks that solved for registration affine parameters, image patch displacements, and patch displacements within the region of overlap. The ground truth images for deep learning-based approaches were derived from successful conventional feature-based registration. Cross-sectional and longitudinal affine registrations were performed across color fundus photography (CFP), fundus autofluorescence (FAF), and infrared reflectance (IR) image modalities. For mono-modality longitudinal registration, the conventional feature-based registration method achieved mean errors in the range of 39-53 µm (depending on the modality) whereas the deep learning method with region overlap prediction exhibited mean errors in the range 54-59 µm. For cross-sectional multi-modality registration, the conventional method exhibited gross failures with large errors in more than 50% of the cases while the proposed deep-learning method achieved robust performance with no gross failures and mean errors in the range 66-69 µm. Thus, the deep learning-based method achieved superior overall performance across all modalities. The accuracy and robustness reported in this work provide important advances that will facilitate clinical research and enable a detailed study of the progression of retinal diseases such as age-related macular degeneration.

In linear registration, a floating image is spatially aligned with a reference image after performing a series of linear metric transformations. Additionally, linear registration is mainly considered a preprocessing version of nonrigid registration. To better accomplish the task of finding the optimal transformation in pairwise intensity-based medical image registration, in this work, we present an optimization algorithm called the normal vibration distribution search-based differential evolution algorithm (NVSA), which is modified from the Bernstein search-based differential evolution (BSD) algorithm. We redesign the search pattern of the BSD algorithm and import several control parameters as part of the fine-tuning process to reduce the difficulty of the algorithm. In this study, 23 classic optimization functions and 16 real-world patients (resulting in 41 multimodal registration scenarios) are used in experiments performed to statistically investigate the problem solving ability of the NVSA. Nine metaheuristic algorithms are used in the conducted experiments. When compared to the commonly utilized registration methods, such as ANTS, Elastix, and FSL, our method achieves better registration performance on the RIRE dataset. Moreover, we prove that our method can perform well with or without its initial spatial transformation in terms of different evaluation indicators, demonstrating its versatility and robustness for various clinical needs and applications. This study establishes the idea that metaheuristic-based methods can better accomplish linear registration tasks than the frequently used approaches; the proposed method demonstrates promise that it can solve real-world clinical and service problems encountered during nonrigid registration as a preprocessing approach.The source code of the NVSA is publicly available at https://github.com/PengGui-N/NVSA.

Multimodal registration of remotely sensed images based on Jeffrey’s divergence

Due to significant differences in imaging mechanisms between multimodal images, registration methods have difficulty in achieving the ideal effect in terms of time consumption and matching precision. Therefore, this paper puts forward a rapid and robust method for multimodal image registration by exploiting local edge information. The method is based on the framework of SURF and can simultaneously achieve real time and accuracy. Due to the unpredictability of multimodal images’ textures, the local edge descriptor is built based on the edge histogram of neighborhood around keypoints. Moreover, in order to increase the robustness of the whole algorithm and maintain the SURF’s fast characteristic, saliency assessment of keypoints and the concept of self-similar factor are presented and introduced. Experimental results show that the proposed method achieves higher precision and consumes less time than other multimodality registration methods. In addition, the robustness and stability of the method are also demonstrated in the presence of image blurring, rotation, noise, and luminance variations.