Imaging coupled filtering: A unified multi-channel framework for multimodal medical image registration and fusion

Human visual system inspired multi-modal medical image fusion framework

This paper presents a novel multimodal medical image fusion framework (M3IF-NSST-CNN) combining the Non-Subsampled Shearlet Transform (NSST) and deep convolutional neural network (CNN) methods to enhance diagnostic accuracy by effectively integrating Computed Tomography (CT) and Magnetic Resonance Imaging (MRI). Initially, both CT and MRI images undergo decomposition via NSST, producing distinct low-pass (LPSB) and high-pass subbands (HPSB). The LPSB fusion leverages a correlation-based approach, ensuring preservation of structural integrity and maximization of complementary information. Concurrently, the HPSB are merged through a Universal Image Quality Index (UIQI)-based fusion rule, prioritizing edge and detail retention critical for precise medical diagnosis. Subsequently, an inverse NSST transforms these fused sub-bands, generating a preliminary fused image. This image is then further refined through a CNN-based noise reduction module comprising convolutional layers, batch normalization, and Rectified Linear Unit (ReLU) activation functions. The deep CNN architecture progressively refines the fused output, significantly reducing artifacts and noise, thus improving the overall clarity and interpretability of the resultant image. This approach addresses common limitations in image fusion, such as loss of edge details, distortion, and the inability to preserve critical information. The hybrid M3IF-NSST-CNN method improves both the visual quality and parametric accuracy of the fusion, outperforming comparable techniques. By preserving more detail and edge information, this method offers significant advancements in multi-modal medical image fusion for enhanced diagnostic insights.

DFENet: A dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion

Abstract: Multi-modality medical or clinical image fusion is a field of study aimed at enhancing diagnostic accuracy and aid in decisions to be taken by medical professional. Various fusion techniques such as pixel-based, region-based, and transformbased approaches are applied in image fusion to provide accurate fusion. Different devices which take scans of body such as MRI, CT, PET, SPECT, Ultrasound hold and carry different features, and different medical sensors obtain different information of the particular part of the body. Each of these imaging modalities offer only specific information that is used for the detection and analysis of specific problem. The idea behind fusion is to achieve and get better contrast and better fused image. The algorithm is making use of the common pyramid type and similarity type fusion algorithm with the neural networks model to achieve a better and more flexible fusion method. The advantages of image fusion medically are widespread. It plays a pivotal role in tumour localization, surgical planning and in treatment assessment.

Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation

Multimodal medical image fusion is to extract information from different modal images into a single one and obtain the organizational characteristics of source images. Different medical imaging measures such as CT and MRI often bring different visual morphology, however, the salient features of tissues are basically the same from the perspective of human eyes. According to this characteristic, an improved image fusion algorithm based on visual salience detection is proposed in this paper. First, the GBVS algorithm was introduced to calculate visual salience of two registered source images, and then decompose the source images in NSST domain to obtain their low-frequency and high-frequency sub-bands. For the low-frequency sub-bands, local energy and GBVS graph are input into fuzzy logic system to obtain the respective weights for the fused low-frequency sub-band. For the high-frequency sub-bands, the NSML values of each sub-band were calculated and compared to obtain the fused high-frequency sub-band. The final fused image was obtained by using the inverse NSST transformation. Applying this method to multimodal medical image fusion, the visual quality of the image can be enhanced effectively and the salient features of tissues can be preserved well. Experiments on multimodal fusion of different gray-scale medical images show that the proposed method has advantages in retention of image salient features and the overall image contrast, and has better objective index than the comparison models.

Multimodal medical image fusion plays an important role in clinical applications. However, multimodal medical image fusion methods ignore the feature dependence among modals, and the feature fusion ability with different granularity is not strong. A Long-Range Correlation-Guided Dual-Encoder Fusion Network for Medical Images is proposed in this paper. The main innovations of this paper are as follows: Firstly, A Cross-dimension Multi-scale Feature Extraction Module (CMFEM) is designed in the encoder, by extracting multi-scale features and aggregating coarse-to-fine features, the model realizes fine-grained feature enhancement in different modalities. Secondly, a Long-range Correlation Fusion Module (LCFM) is designed, by calculating the long-range correlation coefficient between local features and global features, the same granularity features are fused by the long-range correlation fusion module. long-range dependencies between modalities are captured by the model, and different granularity features are aggregated. Finally, this paper is validated on clinical multimodal lung medical image dataset and brain medical data dataset. On the lung medical image dataset, IE, AG, {textbf {Q}}^{{textbf {AB/F}}}, and EI metrics are improved by 4.53%, 4.10%, 6.19%, and 6.62% respectively. On the brain medical image dataset, SF, VIF, and {textbf {Q}}^{{textbf {AB/F}}} metrics are improved by 3.88%, 15.71%, and 7.99% respectively. This model realizes better fusion performance, which plays an important role in the fusion of multimodal medical images.

An image quality enhancement scheme employing adolescent identity search algorithm in the NSST domain for multimodal medical image fusion

Wavelet transforms have emerged as a powerful tool in image fusion. However, the study and analysis of medical image fusion is still a challenging area of research. Therefore, in this paper, we propose a multiscale fusion of multimodal medical images in wavelet domain. Fusion of medical images has been performed at multiple scales varying from minimum to maximum level using maximum selection rule which provides more flexibility and choice to select the relevant fused images. The experimental analysis of the proposed method has been performed with several sets of medical images. Fusion results have been evaluated subjectively and objectively with existing state-of-the-art fusion methods which include several pyramid- and wavelet-transform-based fusion methods and principal component analysis (PCA) fusion method. The comparative analysis of the fusion results has been performed with edge strength (Q), mutual information (MI), entropy (E), standard deviation (SD), blind structural similarity index metric (BSSIM), spatial frequency (SF), and average gradient (AG) metrics. The combined subjective and objective evaluations of the proposed fusion method at multiple scales showed the effectiveness and goodness of the proposed approach.

Multimodal medical image fusion based on visual saliency map and multichannel dynamic threshold neural P systems in sub-window variance filter domain

Medical image fusion (MIF) significantly improves diagnosis accuracy by combining complementary data from several imaging modalities. However, the conventional fusion methods often fail to maintain the edge details, uniformity of texture and most relevant characteristics. To overcome these challenges, a novel multi-modality medical image fusion using Triple generator network (MIMO-TGAN) is proposed for accurate brain abnormality detection. The proposed MIMO-TGAN uses the multi-modality images such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and Positron Emission Tomography (PET) images for MIF process. Initially, the gathered MRI and CT images are denoised with scalable range-based adaptive trilateral (SCRAT) filter and PET images are processed by multi-scale retinex (MSR) technique. The proposed MIMO-TGAN model contains three generators with a discriminator and an abnormality detection phase. The Generator-1 consists of grouped convolutional with different attention blocks for retrieving the relevant features from MRI and CT images to generate a synthesized image. Simultaneously, Generator-2 retrieves the most important voxel features from PET images and MRI images to generate another synthesized image. Finally, Generator-3 is designed to combine the both synthesized images from two generators by using four fusion rules to accurately detect brain abnormalities. The experimental evaluations demonstrate that the proposed MIMO-TGAN achieves an accuracy of 99.43% and 98.24% for dataset-1and dataset-2 respectively. Moreover, the proposed MIMO-TGAN achieves the entropy values of 7.05 and 6.84 for dataset-1 and dataset-2, and overall SSIM of 0.94 that indicates superior fusion quality results compared to existing methods.

Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients' data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients' data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework.

The role of medical image fusion in different applications like disorder treatment, surgical navigation, and diagnosis is critically increasing. Alzheimer’s disease has been achieved by clear study of structural and functional changes that are present in brain image. Multimodal medical image fusion considers merging of complementary information while removing the surplus artifacts to produce a solitary image which provides details of both spectral and spatial data. In this paper, multimodal medical image fusion method is based on rolling guidance filter and parameter adaptive pulse coupled neural networks. First, the source images are decomposed into base and detail images using rolling GF. The final detail image is obtained by using PA-PCNN, and final base image is obtained by using maximum fusion rule, combining them gives fusion image. The superiority of the proposed method is presented and is justified with respect to quality metrics. KeywordsMedical image fusionAlzheimer diseaseParameter adaptive PCNNRolling guidance filter

Image processing technology requires moreover the full image or the part of image which is to be processed from the user’s point of view like the radius of object etc. The main purpose of fusion is to diminish dissimilar error between the fused image and the input images. With respect to the medical diagnosis, the edges and outlines of the concerned objects is more important than extra information. So preserving the edge features of the image is worth for investigating the image fusion. The image with higher contrast contains more edge-like features. Here we propose a new medical image fusion scheme namely Local Energy Match NSCT based on discrete contourlet transformation, which is constructive to give the details of curve edges. It is used to progress the edge information of fused image by dropping the distortion. This transformation lead to crumbling of multimodal image addicted to finer and coarser details and finest details will be decayed into unusual resolution in dissimilar orientation. The input multimodal images namely CT and MRI images are first transformed by Non Sub sampled Contourlet Transformation (NSCT) which decomposes the image into low frequency and high frequency elements. In our system, the Low frequency coefficient of the image is fused by image averaging and Gabor filter bank algorithm. The processed High frequency coefficients of the image are fused by image averaging and gradient based fusion algorithm. Then the fused image is obtained by inverse NSCT with local energy match based coefficients. To evaluate the image fusion accuracy, Peak Signal to Noise Ratio (PSNR), Root Mean Square Error (RMSE) and Correlation Coefficient parameters are used in this work .

The emergence of multimodal medical imaging technology greatly increases the accuracy of clinical diagnosis and etiological analysis. Nevertheless, each medical imaging modal unavoidably has its own limitations, so the fusion of multimodal medical images may become an effective solution. In this paper, a novel fusion method on the multimodal medical images exploiting convolutional neural network (CNN) and extreme learning machine (ELM) is proposed. As a typical representative in deep learning, CNN has been gaining more and more popularity in the field of image processing. However, CNN often suffers from several drawbacks, such as high computational costs and intensive human interventions. To this end, the model of convolutional extreme learning machine (CELM) is constructed by incorporating ELM into the traditional CNN model. CELM serves as an important tool to extract and capture the features of the source images from a variety of different angles. The final fused image can be obtained by integrating the significant features together. Experimental results indicate that, the proposed method is not only helpful to enhance the accuracy of the lesion detection and localization, but also superior to the current state-of-the-art ones in terms of both subjective visual performance and objective criteria.