PCANet-Based Structural Representation for Nonrigid Multimodal Medical Image Registration.

Deep learning based data-adaptive descriptor for non-rigid multi-modal medical image registration

Objective. This paper proposes a novel approach for the longitudinal registration of PET imaging acquired for the monitoring of patients with metastatic breast cancer. Unlike with other image analysis tasks, the use of deep learning (DL) has not significantly improved the performance of image registration. With this work, we propose a new registration approach to bridge the performance gap between conventional and DL-based methods: medical image registration method regularized by architecture (MIRRBA). Approach. MIRRBA is a subject-specific deformable registration method which relies on a deep pyramidal architecture to parametrize the deformation field. Diverging from the usual deep-learning paradigms, MIRRBA does not require a learning database, but only a pair of images to be registered that is used to optimize the network's parameters. We applied MIRRBA on a private dataset of 110 whole-body PET images of patients with metastatic breast cancer. We used different architecture configurations to produce the deformation field and studied the results obtained. We also compared our method to several standard registration approaches: two conventional iterative registration methods (ANTs and Elastix) and two supervised DL-based models (LapIRN and Voxelmorph). Registration accuracy was evaluated using the Dice score, the target registration error, the average Hausdorff distance and the detection rate, while the realism of the registration obtained was evaluated using Jacobian's determinant. The ability of the different methods to shrink disappearing lesions was also computed with the disappearing rate. Main results. MIRRBA significantly improved all metrics when compared to DL-based approaches. The organ and lesion Dice scores of Voxelmorph improved by 6% and 52% respectively, while the ones of LapIRN increased by 5% and 65%. Regarding conventional approaches, MIRRBA presented comparable results showing the feasibility of our method. Significance. In this paper, we also demonstrate the regularizing power of deep architectures and present new elements to understand the role of the architecture in DL methods used for registration.

Brain tumor detection and medical image registration are critical tasks in neuroimaging, enabling accurate diagnosis, treatment planning, and follow-up monitoring. Magnetic Resonance Imaging (MRI) remains the preferred modality for visualizing tumor morphology, while image registration facilitates the alignment of multimodal scans for enhanced clinical interpretation. This survey reviews recent advances in deep learning-based methods for brain tumor classification and medical image registration, focusing on Convolutional Neural Networks (CNNs), transfer learning architectures (e.g., VGG-19, ResNet, DenseNet201), and modern registration frameworks including supervised, unsupervised, GAN-based, and reinforcement learning approaches. A comprehensive analysis of 50+ peer-reviewed studies highlights state-of-the-art performance, with brain tumor classification achieving up to 97.5% precision (CNN + FCM) and 96.97% accuracy (DWT+PCA+DNN), while multimodal MRI registration methods report Dice Similarity Coefficient (DSC) values exceeding 0.90 and reduced Target Registration Error (TRE) compared to traditional deformable registration techniques. Trends indicate a shift toward hybrid models combining segmentation and registration, domain adaptation for cross-dataset generalization, and lightweight frameworks for real-time clinical deployment. In conclusion, deep learning approaches have significantly advanced the accuracy, efficiency, and automation of brain tumor detection and medical image registration. However, challenges such as limited annotated multimodal datasets, robust similarity measures for heterogeneous modalities, and the integration of clinically interpretable AI models remain open for future research.

Symmetric transformer-based network for unsupervised image registration

Improvement of medical diagnosis, aided computer surgeries and tumor identification requires an accurate image registration approaches. The registration of multimodal medical images is more complicated than the registration of unimodal medical images due to the variation in luminance between the images. In this paper, an accurate multimodal image registration approach using artificial immune system (AIS) is proposed and the affine transformation model is used in contrast to the most of the related works which assumed rigid transformation model or similarity transformation model. In the proposed approach the LL bands of the discrete wavelet transform (DWT) for the images are used and the normalized mutual information (NMI) is used as a fitness function. The proposed approach achieves good result in the case of noiseless images, noisy images and partial data loss from one of the images. Moreover, the proposed approach does not need any feature extraction or refinement step. To demonstrate the robustness of the proposed approach, it has been compared with two multimodal medical image registration approaches.

Accurate and fast localization of a predefined target region inside the patient is an important component of many image-guided therapy procedures. This problem is commonly solved by registration of intraoperative 2-D projection images to 3-D preoperative images. If the patient is not fixed during the intervention, the 2-D image acquisition is repeated several times during the procedure, and the registration problem can be cast instead as a 3-D tracking problem. To solve the 3-D problem, we propose in this paper to apply 2-D region tracking to first recover the components of the transformation that are in-plane to the projections. The 2-D motion estimates of all projections are backprojected into 3-D space, where they are then combined into a consistent estimate of the 3-D motion. We compare this method to intensity-based 2-D to 3-D registration and a combination of 2-D motion backprojection followed by a 2-D to 3-D registration stage. Using clinical data with a fiducial marker-based gold-standard transformation, we show that our method is capable of accurately tracking vertebral targets in 3-D from 2-D motion measured in X-ray projection images. Using a standard tracking algorithm (hyperplane tracking), tracking is achieved at video frame rates but fails relatively often (32% of all frames tracked with target registration error (TRE) better than 1.2 mm, 82% of all frames tracked with TRE better than 2.4 mm). With intensity-based 2-D to 2-D image registration using normalized mutual information (NMI) and pattern intensity (PI), accuracy and robustness are substantially improved. NMI tracked 82% of all frames in our data with TRE better than 1.2 mm and 96% of all frames with TRE better than 2.4 mm. This comes at the cost of a reduced frame rate, 1.7 s average processing time per frame and projection device. Results using PI were slightly more accurate, but required on average 5.4 s time per frame. These results are still substantially faster than 2-D to 3-D registration. We conclude that motion backprojection from 2-D motion tracking is an accurate and efficient method for tracking 3-D target motion, but tracking 2-D motion accurately and robustly remains a challenge.

Although a considerable amount of research has been devoted to the registration of medical images, a good proportion of this has been performed on rigid regions of the anatomy. This has tended to limit the practical value of such research. The registration of medical images which are non-rigid in nature is a seemingly difficult and not fully investigated problem. This paper presents an elastic transformation method which is used to perform matching of non-rigid medical images, that is, two-dimensional images of a region of the body such as the thorax or abdomen which exhibits a high level of elasticity. Some forms of elastic transformation may be more localized in scope than others, which leads to the categorization of elastic transformation methods into local and semi-local methods. This paper demonstrates the use of local elastic spatial transformations in matching non-rigid medical images. These transformations are illustrated by matching corresponding computed tomography and magnetic resonance images of the thorax.

In this paper, we introduce SAMmorph, a novel multimodal medical image registration framework addressing challenges in aligning cross‐modality images due to significant appearance differences and heavy reliance on expensive annotations. It comprises three key components: the VoxelSAM module for generating pseudo‐labels, the FusionLayer which takes moving/fixed images and their pseudo‐labels as input, integrates them via two optional schemes (stack: concatenating along the channel dimension to form a 4‐channel tensor; enhance: modulating image intensity with mask confidence to generate a 3‐channel tensor) to produce combined features for registration, and the deformable registration network which includes two architectures (joint and independent registration networks) to implement the bidirectional registration strategy. Experiments demonstrate that SAMmorph outperforms state‐of‐the‐art methods across various metrics on the ‐RegPro dataset, achieving a dice similarity coefficient (DSC) of 0.906, target registration error (TRE) of 6.06 mm, compared to those being 0.789 and 7.632 mm achieved by the strongest tested baseline method CorrMLP, with statistical significance confirmed by Wilcoxon signed‐rank tests (). This work presents a promising approach for improving medical image registration accuracy and efficiency, with potential applications in diverse medical imaging scenarios. The source codes have been released at: https://github.com/qilin‐world/SAMmorph .

Purpose:Stereotactic radiosurgery (SRS), which delivers a potent dose of highly conformal radiation to the target in a single fraction, requires accurate tumor delineation for treatment planning. We present an automatic segmentation strategy, that synergizes intensity histogram thresholding, super‐voxel clustering, and level‐set based contour evolving methods to efficiently and accurately delineate SRS brain tumors on contrast‐enhance T1‐weighted (T1c) Magnetic Resonance Images (MRI).Methods:The developed auto‐segmentation strategy consists of three major steps. Firstly, tumor sites are localized through 2D slice intensity histogram scanning. Then, super voxels are obtained through clustering the corresponding voxels in 3D with reference to the similarity metrics composited from spatial distance and intensity difference. The combination of the above two could generate the initial contour surface. Finally, a localized region active contour model is utilized to evolve the surface to achieve the accurate delineation of the tumors. The developed method was evaluated on numerical phantom data, synthetic BRATS (Multimodal Brain Tumor Image Segmentation challenge) data, and clinical patients’ data. The auto‐segmentation results were quantitatively evaluated by comparing to ground truths with both volume and surface similarity metrics.Results:DICE coefficient (DC) was performed as a quantitative metric to evaluate the auto‐segmentation in the numerical phantom with 8 tumors. DCs are 0.999±0.001 without noise, 0.969±0.065 with Rician noise and 0.976±0.038 with Gaussian noise. DC, NMI (Normalized Mutual Information), SSIM (Structural Similarity) and Hausdorff distance (HD) were calculated as the metrics for the BRATS and patients’ data. Assessment of BRATS data across 25 tumor segmentation yield DC 0.886±0.078, NMI 0.817±0.108, SSIM 0.997±0.002, and HD 6.483±4.079mm. Evaluation on 8 patients with total 14 tumor sites yield DC 0.872±0.070, NMI 0.824±0.078, SSIM 0.999±0.001, and HD 5.926±6.141mm.Conclusion:The developed automatic segmentation strategy, which yields accurate brain tumor delineation in evaluation cases, is promising for its application in SRS treatment planning.

Image registration aims to establish spatial correspondence across pairs, or groups of images, and is a cornerstone of medical image computing and computer-assisted-interventions. Currently, most deep learning-based registration methods assume that the desired deformation fields are globally smooth and continuous, which is not always valid for real-world scenarios, especially in medical image registration (e.g. cardiac imaging and abdominal imaging). Such a global constraint can lead to artefacts and increased errors at discontinuous tissue interfaces. To tackle this issue, we propose a weakly-supervised Deep Discontinuity-preserving Image Registration network (DDIR), to obtain better registration performance and realistic deformation fields. We demonstrate that our method achieves significant improvements in registration accuracy and predicts more realistic deformations, in registration experiments on cardiac magnetic resonance (MR) images from UK Biobank Imaging Study (UKBB), than state-of-the-art approaches.

Soft-tissue deformation presents a confounding factor to rigid image registration by introducing image content inconsistent with the underlying motion model, presenting non-correspondent structure with potentially high power, and creating local minima that challenge iterative optimization. In this paper, we introduce a model for registration performance that includes deformable soft tissue as a power-law noise distribution within a statistical framework describing the Cramer-Rao lower bound (CRLB) and root-mean-squared error (RMSE) in registration performance. The model incorporates both cross-correlation and gradient-based similarity metrics, and the model was tested in application to 3D-2D (CT-to-radiograph) and 3D-3D (CT-to-CT) image registration. Predictions accurately reflect the trends in registration error as a function of dose (quantum noise), and the choice of similarity metrics for both registration scenarios. Incorporating soft-tissue deformation as a noise source yields important insight on the limits of registration performance with respect to algorithm design and the clinical application or anatomical context. For example, the model quantifies the advantage of gradient-based similarity metrics in 3D-2D registration, identifies the low-dose limits of registration performance, and reveals the conditions for which the registration performance is fundamentally limited by soft-tissue deformation.

Registration of medical imaging is essential for aligning in time and space different modalities and hence consolidating their strengths for enhanced diagnosis and for the effective planning of treatment or therapeutic interventions. The primary objective of this study is to develop an integrated registration method that is effective for registering both brain and whole-body images. We seek in the proposed method to combine in one setting the excellent registration results that FMRIB Software Library (FSL) produces with brain images and the excellent results of Statistical Parametric Mapping (SPM) when registering whole-body images. To assess attainment of these objectives, the following registration tasks were performed: (1) FDG_CT with FLT_CT images, (2) pre-operation MRI with intra-operation CT images, (3) brain only MRI with corresponding PET images, and (4) MRI T1 with T2, T1 with FLAIR, and T1 with GE images. Then, the results of the proposed method will be compared to those obtained using existing state-of-the-art registration methods such as SPM and FSL. Initially, three slices were chosen from the reference image, and the normalized mutual information (NMI) was calculated between each of them for every slice in the moving image. The three pairs with the highest NMI values were chosen. The wavelet decomposition method is applied to minimize the computational requirements. An initial search applying a genetic algorithm is conducted on the three pairs to obtain three sets of registration parameters. The Powell method is applied to reference and moving images to validate the three sets of registration parameters. A linear interpolation method is then used to obtain the registration parameters for all remaining slices. Finally, the aligned registered image with the reference image were displayed to show the different performances of the 3 methods, namely the proposed method, SPM and FSL by gauging the average NMI values obtained in the registration results. Visual observations are also provided in support of these NMI values. For comparative purposes, tests using different multi-modal imaging platforms are performed.

Medical image segmentation is a fundamental task in medical image analysis, playing a crucial role in disease diagnosis, treatment planning, and clinical decision-making. Accurate segmentation of anatomical structures, such as blood vessels, organs, and lesions, is essential for reliable medical interpretations. However, existing segmentation models often face challenges in effectively capturing both local and global features within medical images. Traditional convolutional neural networks (CNNs) excel at extracting fine-grained local features but struggle to model long-range dependencies due to their limited receptive fields. Conversely, transformer-based models can capture global contextual relationships but often fail to preserve detailed local structures, leading to suboptimal segmentation performance. The purpose of this study is to develop SEFormer, a novel hybrid network that integrates SENet, ResNet, and Transformer to enhance medical image segmentation. SEFormer aims to effectively capture both local and global features, addressing the limitations of traditional CNNs and transformer-based models.SENet is incorporated to recalibrate feature maps by adaptively emphasizing informative features, thereby compensating for the global representation limitations of CNNs. Meanwhile, the ResNet backbone ensures deep feature extraction while maintaining computational efficiency. Additionally, the Transformer module is introduced to capture long-range dependencies and enhance contextual awareness, complementing the local feature extraction capabilities of CNNs. This hybrid approach allows SEFormer to balance fine-grained spatial details with broader contextual information, leading to more precise segmentation results.Furthermore, to mitigate information loss during feature extraction and ensure a more complete and hierarchical representation of image information, we introduce a feature pyramid structure inspired by multi-scale image pyramid models. By progressively increasing the receptive field across different scales and employing skip connections, SEFormer effectively fuses multi-scale local and global features at each stage of the pyramid. This hierarchical fusion mechanism ensures a richer and more robust feature representation, which is particularly beneficial for segmenting complex medical structures such as blood vessels. We evaluate SEFormer on the CHASE_DB1 dataset, a widely used benchmark for retinal vessel segmentation. Experimental results demonstrate that SEFormer outperforms existing state-of-the-art segmentation methods, achieving a 3.25% improvement in segmentation accuracy. Additionally, we conduct ablation studies to verify the contributions of different components within SEFormer. The results show that incorporating SENet enhances feature recalibration and channel attention, while the Transformer module significantly improves global context awareness. The feature pyramid structure further contributes to performance gains by ensuring a multi-scale representation of vascular structures. Compared to conventional CNN-based methods, our approach achieves better segmentation accuracy with improved robustness to variations in vessel thickness, noise, and image contrast. Furthermore, SEFormer maintains computational efficiency, making it suitable for real-world clinical applications where both accuracy and processing speed are critical. SEFormer provides an efficient and accurate approach for medical image segmentation by leveraging the complementary strengths of CNNs, SENet, and Transformers while integrating a feature pyramid structure to maximize feature representation. Future research directions include extending SEFormer to other medical imaging modalities, such as CT and MRI scans, as well as exploring further optimization techniques to reduce computational costs while maintaining high segmentation performance. Additionally, integrating SEFormer with active learning frameworks could further improve segmentation performance by leveraging expert-annotated data more efficiently.

Dynamic Positron Emission Tomography is a powerful tool for quantitative imaging of in vivo biological processes. The long scan durations necessitate motion correction, to maintain the validity of the dynamic measurements, which can be particularly challenging due to the low signal-to-noise ratio (SNR) and spatial resolution, as well as the complex tracer behaviour in the dynamic PET data. In this paper we develop a novel automated expectation-maximisation image registration framework that incorporates temporal tracer kinetic information to correct for inter-frame subject motion during dynamic PET scans. We employ the Zubal human brain phantom to simulate dynamic PET data using SORTEO (a Monte Carlo-based simulator), in order to validate the proposed method for its ability to recover imposed rigid motion. We have conducted a range of simulations using different noise levels, and corrupted the data with a range of rigid motion artefacts. The performance of our motion correction method is compared with pairwise registration using normalised mutual information as a voxel similarity measure (an approach conventionally used to correct for dynamic PET inter-frame motion based solely on intensity information). To quantify registration accuracy, we calculate the target registration error across the images. The results show that our new dynamic image registration method based on tracer kinetics yields better realignment of the simulated datasets, halving the target registration error when compared to the conventional method at small motion levels, as well as yielding smaller residuals in translation and rotation parameters. We also show that our new method is less affected by the low signal in the first few frames, which the conventional method based on normalised mutual information fails to realign.

Medical image segmentation and registration are two tasks to analyze the anatomical structures in clinical research. Still, deep-learning solutions utilizing the connections between segmentation and registration remain underdiscovered. This article designs a joint learning framework named RSegNet that can realize concurrent deformable registration and segmentation by minimizing an integrated loss function, including three parts: diffeomorphic registration loss, segmentation similarity loss, and dual-consistency supervision loss. The probabilistic diffeomorphic registration branch could benefit from the auxiliary segmentations available from the segmentation branch to achieve anatomical consistency and better deformation regularity by dual-consistency supervision. Simultaneously, the segmentation performance could also be improved by data augmentation based on the registration with well-behaved diffeomorphic guarantees. Experiments on the human brain 3-D magnetic resonance images have been implemented to demonstrate the effectiveness of our approach. We trained and validated RSegNet with 1000 images and tested its performances on four public datasets, which shows that our method successfully yields concurrent improvements of both segmentation and registration compared with separately trained networks. Specifically, our method can increase the accuracy of segmentation and registration by 7.0% and 1.4%, respectively, in terms of Dice scores. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Registration and segmentation of medical images are two significant tasks in medical research and clinical application. However, most existing approaches consider these two tasks independently while neglecting the potential association between them. Therefore, we suggest a new approach that combines these two tasks into one joint deep learning framework, boosting registration, and segmentation performance by introducing dual-consistency supervision. Besides, our framework could generate outputs within 1 s by taking an affinely aligned medical image pair as input, which is suitable for time-critical requirements in a clinic. We tested it on four public datasets and achieved state-of-the-art performance to demonstrate the proposed method’s feasibility and robustness. Furthermore, our proposed RSegNet is a general learning framework suitable for various image modalities and anatomical structures. Hence, we expect our framework to serve as a practical clinical tool to speed up medical image analysis procedures and improve diagnostic accuracy.