SAMmorph: A Weakly‐Supervised Multimodal Medical Image Registration With SAM‐Based Segmentation and Bidirectional Mechanism

A relatively large class of information theoretical measures, including e.g. mutual information or normalized entropy, has been used in multi-modal medical image registration. Even though the mathematical foundations of the different measures were very similar, the final expressions turned out to be surprisingly different. Therefore one of the main aims of this paper is to enlight the relationship of different objective functions by introducing a mathematical framework from which several known optimization objectives can be deduced. Furthermore we extend existing measures in order to be applicable on image features different than image intensities and introduce efficiency as a very general concept to qualify such features. The presented framework is very general and not at all restricted to medical images. Still we want to discuss the possible impact of our theoretical framework for the particular problem of medical image registration, where the feature space has traditionally been fixed to image intensities. Our theoretical approach is very general though and can be used for any kind of multi-modal signals, such as for the broad field of multi-media applications.

Medical image registration is an integral part of various clinical applications including image guidance, motion tracking, therapy assessment and diagnosis. We present a robust approach for mono-modal and multi-modal medical image registration. To this end, we propose the novel shape operator based local image distance (SOLID) which estimates the similarity of images by comparing their second-order curvature information. Our similarity metric is rigorously tailored to be suitable for comparing images from different medical imaging modalities or image contrasts. A critical element of our method is the extraction of local features using higher-order shape information, enabling the accurate identification and registration of smaller structures. In order to assess the efficacy of the proposed similarity metric, we have implemented a variational image registration algorithm that relies on the principle of matching the curvature information of the given images. The performance of the proposed algorithm has been evaluated against various alternative state-of-the-art variational registration algorithms. Our experiments involve mono-modal as well as multi-modal and cross-contrast co-registration tasks in a broad variety of anatomical regions. Compared to the evaluated alternative registration methods, the results indicate a very favorable accuracy, precision and robustness of the proposed SOLID method in various highly challenging registration tasks.

Brain tumor detection and medical image registration are critical tasks in neuroimaging, enabling accurate diagnosis, treatment planning, and follow-up monitoring. Magnetic Resonance Imaging (MRI) remains the preferred modality for visualizing tumor morphology, while image registration facilitates the alignment of multimodal scans for enhanced clinical interpretation. This survey reviews recent advances in deep learning-based methods for brain tumor classification and medical image registration, focusing on Convolutional Neural Networks (CNNs), transfer learning architectures (e.g., VGG-19, ResNet, DenseNet201), and modern registration frameworks including supervised, unsupervised, GAN-based, and reinforcement learning approaches. A comprehensive analysis of 50+ peer-reviewed studies highlights state-of-the-art performance, with brain tumor classification achieving up to 97.5% precision (CNN + FCM) and 96.97% accuracy (DWT+PCA+DNN), while multimodal MRI registration methods report Dice Similarity Coefficient (DSC) values exceeding 0.90 and reduced Target Registration Error (TRE) compared to traditional deformable registration techniques. Trends indicate a shift toward hybrid models combining segmentation and registration, domain adaptation for cross-dataset generalization, and lightweight frameworks for real-time clinical deployment. In conclusion, deep learning approaches have significantly advanced the accuracy, efficiency, and automation of brain tumor detection and medical image registration. However, challenges such as limited annotated multimodal datasets, robust similarity measures for heterogeneous modalities, and the integration of clinically interpretable AI models remain open for future research.

Deep learning-based affine medical image registration for multimodal minimal-invasive image-guided interventions – A comparative study on generalizability

Improvement of medical diagnosis, aided computer surgeries and tumor identification requires an accurate image registration approaches. The registration of multimodal medical images is more complicated than the registration of unimodal medical images due to the variation in luminance between the images. In this paper, an accurate multimodal image registration approach using artificial immune system (AIS) is proposed and the affine transformation model is used in contrast to the most of the related works which assumed rigid transformation model or similarity transformation model. In the proposed approach the LL bands of the discrete wavelet transform (DWT) for the images are used and the normalized mutual information (NMI) is used as a fitness function. The proposed approach achieves good result in the case of noiseless images, noisy images and partial data loss from one of the images. Moreover, the proposed approach does not need any feature extraction or refinement step. To demonstrate the robustness of the proposed approach, it has been compared with two multimodal medical image registration approaches.

Over the past few decades, fast<strong>-</strong>growth has occurred in the area of medical image acquisition devices, and physicians now rely on the utilization of medical images for the diagnosis, treatment plans, and surgical guidance. Researchers have classified medical images according to two structures: anatomical and functional structures. Due to this classification, the data obtained from two or more images of the same object frequently provide complementary and more abundant information through a process known as multimodal medical model registration. Image registration is spatially mapping the coordinate system of the two images obtained from a different viewpoint and utilizing various sensors. Several automatic multimodal medical image registration algorithms have been introduced based on types of medical images and their applications to increase the reliability, robustness, and accuracy. Due to the diversity in imaging and the different demands<strong> </strong>for applications, there is no single registration algorithm that can do that. This paper introduces a novel method for developing a multimodal medical image registration system that can select the most accepted registration algorithm from a group of registration algorithms for a variety of input datasets. The method described here is based on a machine learning technique that selects the most promising candidate. Several experiments have been conducted, and the results reveal that the novel approach leads to considerably faster reliability, accuracy, and more robustness registration algorithm selection.

Analysis of multispectral or multitemporal images requires proper geometric alignment of the images to compare corresponding regions in each image volume. Retrospective three-dimensional alignment or registration of multimodal medical images based on features intrinsic to the image data itself is complicated by their different photometric properties, by the complexity of the anatomical objects in the scene and by the large variety of clinical applications in which registration is involved. While the accuracy of registration approaches based on matching of anatomical landmarks or object surfaces suffers from segmentation errors, voxel-based approaches consider all voxels in the image without the need for segmentation. The recent introduction of the criterion of maximization of mutual information, a basic concept from information theory, has proven to be a breakthrough in the field. While solutions for intrapatient affine. registration based on this concept are already commercially available, current research in the field focuses on interpatient nonrigid matching.

Nonrigid multimodal image registration remains a challenging task in medical image processing and analysis. The structural representation (SR)-based registration methods have attracted much attention recently. However, the existing SR methods cannot provide satisfactory registration accuracy due to the utilization of hand-designed features for structural representation. To address this problem, the structural representation method based on the improved version of the simple deep learning network named PCANet is proposed for medical image registration. In the proposed method, PCANet is firstly trained on numerous medical images to learn convolution kernels for this network. Then, a pair of input medical images to be registered is processed by the learned PCANet. The features extracted by various layers in the PCANet are fused to produce multilevel features. The structural representation images are constructed for two input images based on nonlinear transformation of these multilevel features. The Euclidean distance between structural representation images is calculated and used as the similarity metrics. The objective function defined by the similarity metrics is optimized by L-BFGS method to obtain parameters of the free-form deformation (FFD) model. Extensive experiments on simulated and real multimodal image datasets show that compared with the state-of-the-art registration methods, such as modality-independent neighborhood descriptor (MIND), normalized mutual information (NMI), Weber local descriptor (WLD), and the sum of squared differences on entropy images (ESSD), the proposed method provides better registration performance in terms of target registration error (TRE) and subjective human vision.

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

Joint segmentation and registration of images is a focused area of research nowadays. Jointly segmenting and registering noisy images and images having weak boundaries/intensity inhomogeneity is a challenging task. In medical image processing, joint segmentation and registration are essential methods that aid in distinguishing structures and aligning images for precise diagnosis and therapy. However, these methods encounter challenges, such as computational complexity and sensitivity to variations in image quality, which may reduce their effectiveness in real-world applications. Another major issue is still attaining effective joint segmentation and registration in the presence of artifacts or anatomical deformations. In this paper, a new nonparametric joint model is proposed for the segmentation and registration of multi-modality images having weak boundaries/noise. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is more efficient to segmenting and registering multi-modality images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is also tested on the images obtained from the freely available CHOAS dataset and compare the results of the proposed model with the other existing models using statistical measures such as the Jaccard similarity index, relative reduction, Dice similarity coefficient and Hausdorff distance. It can be seen that the proposed model outperforms the other existing models in terms of quantitatively and qualitatively.

Purpose:To assess the performance of the deformable image registration algorithm used for MRI‐guided adaptive radiation therapy using image feature analysis.Methods:MR images were collected from five patients treated on the MRIdian (ViewRay, Inc., Oakwood Village, OH), a three head Cobalt‐60 therapy machine with an 0.35 T MR system. The images were acquired immediately prior to treatment with a uniform 1.5 mm resolution. Treatment sites were as follows: head/neck, lung, breast, stomach, and bladder. Deformable image registration was performed using the ViewRay software between the first fraction MRI and the final fraction MRI, and the DICE similarity coefficient (DSC) for the skin contours was reported. The SIFT and Harris feature detection and matching algorithms identified point features in each image separately, then found matching features in the other image. The target registration error (TRE) was defined as the vector distance between matched features on the two image sets. Each deformation was evaluated based on comparison of average TRE and DSC.Results:Image feature analysis produced between 2000–9500 points for evaluation on the patient images. The average (± standard deviation) TRE for all patients was 3.3 mm (±3.1 mm), and the passing rate of TRE<3 mm was 60% on the images. The head/neck patient had the best average TRE (1.9 mm±2.3 mm) and the best passing rate (80%). The lung patient had the worst average TRE (4.8 mm±3.3 mm) and the worst passing rate (37.2%). DSC was not significantly correlated with either TRE (p=0.63) or passing rate (p=0.55).Conclusions:Feature matching provides a quantitative assessment of deformable image registration, with a large number of data points for analysis. The TRE of matched features can be used to evaluate the registration of many objects throughout the volume, whereas DSC mainly provides a measure of gross overlap.We have a research agreement with ViewRay Inc.

Registration of multi-modal medical images is an essential pre-processing step, for example, for fusion or image guided-interventions. However, the alignment process is prone to high variability in tissue appearance between modalities, in addition to local intensity variations and artefacts. This work introduces a robust multi-modal registration approach that mitigates the undesirable effect of such variability. Robustness is achieved using Huber’s loss function for the data fidelity and regularization terms. We propose a novel approach using Huber’s criterion, which enables a jointly convex estimation of the motions and the associated scale parameters. We formulate the problem as a complex 2D transformation estimation and investigate a robust total-variation smoothing, as well as a dictionary learning-based data fidelity term. Experiments are conducted using two datasets of multi-contrast MR brain images.

Deep learning based data-adaptive descriptor for non-rigid multi-modal medical image registration

Purpose:To perform validation and commissioning of a commercial deformable image registration (DIR) algorithm (Velocity, Varian Medical Systems) for numerous clinical sites using single and multi‐modality images.Methods:In this retrospective study, the DIR algorithm was evaluated for 10 patients in each of the following body sites: head and neck (HN), prostate, liver, and gynecological (GYN). HN DIRs were evaluated from planning (p)CT to re‐pCT and pCTs to daily CBCTs using dice similarity coefficients (DSC) of corresponding anatomical structures. Prostate DIRs were evaluated from pCT to CBCTs using DSC and target registration error (TRE) of implanted RF beacons within the prostate. Liver DIRs were evaluated from pMR to pCT using DSC and TRE of vessel bifurcations. GYN DIRs were evaluated between fractionated brachytherapy MRIs using DSC of corresponding anatomical structures.Results:Analysis to date has given average DSCs for HN pCT‐to‐(re)pCT DIR for the brainstem, cochleas, constrictors, spinal canal, cord, esophagus, larynx, parotids, and submandibular glands as 0.88, 0.65, 0.67, 0.91, 0.77, 0.69, 0.77, 0.87, and 0.71, respectively. Average DSCs for HN pCT‐to‐CBCT DIR for the constrictors, spinal canal, esophagus, larynx, parotids, and submandibular glands were 0.64, 0.90, 0.62, 0.82, 0.75, and 0.69, respectively. For prostate pCT‐to‐CBCT DIR the DSC for the bladder, femoral heads, prostate, and rectum were 0.71, 0.82, 0.69, and 0.61, respectively. Average TRE using implanted beacons was 3.35 mm. For liver pCT‐to‐pMR, the average liver DSC was 0.94 and TRE was 5.26 mm. For GYN MR‐to‐MR DIR the DSC for the bladder, sigmoid colon, GTV, and rectum were 0.79, 0.58, 0.67, and 0.76, respectively.Conclusion:The Velocity DIR algorithm has been evaluated over a number of anatomical sites. This work functions to document the uncertainties in the DIR in the commissioning process so that these can be accounted for in the development of downstream clinical processes.This work was supported in part by a co‐development agreement with Varian Medical Systems

Multimodality image registration using local linear embedding and hybrid entropy