Multimodal medical image fusion using convolutional neural network and extreme learning machine

Medical image fusion is very important for the diagnosis and treatment of diseases. In recent years, there have been a number of different multi-modal medical image fusion algorithms that can provide delicate contexts for disease diagnosis more clearly and more conveniently. Recently, nuclear norm minimization and deep learning have been used effectively in image processing. A multi-modality medical image fusion method using a rolling guidance filter (RGF) with a convolutional neural network (CNN) based feature mapping and nuclear norm minimization (NNM) is proposed. At first, we decompose medical images to base layer components and detail layer components by using RGF. In the next step, we get the basic fused image through the pretrained CNN model. The CNN model with pre-training is used to obtain the significant characteristics of the base layer components. And we can compute the activity level measurement from the regional energy of CNN-based fusion maps. Then, a detail fused image is gained by NNM. That is, we use NNM to fuse the detail layer components. At last, the basic and detail fused images are integrated into the fused result. From the comparison with the most advanced fusion algorithms, the results of experiments indicate that this fusion algorithm has the best effect in visual evaluation and objective standard. The fusion algorithm using RGF and CNN-based feature mapping, combined with NNM, can improve fusion effects and suppress artifacts and blocking effects in the fused results.

Multimodal medical image fusion plays an important role in clinical applications. However, multimodal medical image fusion methods ignore the feature dependence among modals, and the feature fusion ability with different granularity is not strong. A Long-Range Correlation-Guided Dual-Encoder Fusion Network for Medical Images is proposed in this paper. The main innovations of this paper are as follows: Firstly, A Cross-dimension Multi-scale Feature Extraction Module (CMFEM) is designed in the encoder, by extracting multi-scale features and aggregating coarse-to-fine features, the model realizes fine-grained feature enhancement in different modalities. Secondly, a Long-range Correlation Fusion Module (LCFM) is designed, by calculating the long-range correlation coefficient between local features and global features, the same granularity features are fused by the long-range correlation fusion module. long-range dependencies between modalities are captured by the model, and different granularity features are aggregated. Finally, this paper is validated on clinical multimodal lung medical image dataset and brain medical data dataset. On the lung medical image dataset, IE, AG, {textbf {Q}}^{{textbf {AB/F}}}, and EI metrics are improved by 4.53%, 4.10%, 6.19%, and 6.62% respectively. On the brain medical image dataset, SF, VIF, and {textbf {Q}}^{{textbf {AB/F}}} metrics are improved by 3.88%, 15.71%, and 7.99% respectively. This model realizes better fusion performance, which plays an important role in the fusion of multimodal medical images.

Medical image fusion based on convolutional neural networks and non-subsampled contourlet transform

Abstract: Multi-modality medical or clinical image fusion is a field of study aimed at enhancing diagnostic accuracy and aid in decisions to be taken by medical professional. Various fusion techniques such as pixel-based, region-based, and transformbased approaches are applied in image fusion to provide accurate fusion. Different devices which take scans of body such as MRI, CT, PET, SPECT, Ultrasound hold and carry different features, and different medical sensors obtain different information of the particular part of the body. Each of these imaging modalities offer only specific information that is used for the detection and analysis of specific problem. The idea behind fusion is to achieve and get better contrast and better fused image. The algorithm is making use of the common pyramid type and similarity type fusion algorithm with the neural networks model to achieve a better and more flexible fusion method. The advantages of image fusion medically are widespread. It plays a pivotal role in tumour localization, surgical planning and in treatment assessment.

DFENet: A dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion

Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation

This paper presents a novel multimodal medical image fusion framework (M3IF-NSST-CNN) combining the Non-Subsampled Shearlet Transform (NSST) and deep convolutional neural network (CNN) methods to enhance diagnostic accuracy by effectively integrating Computed Tomography (CT) and Magnetic Resonance Imaging (MRI). Initially, both CT and MRI images undergo decomposition via NSST, producing distinct low-pass (LPSB) and high-pass subbands (HPSB). The LPSB fusion leverages a correlation-based approach, ensuring preservation of structural integrity and maximization of complementary information. Concurrently, the HPSB are merged through a Universal Image Quality Index (UIQI)-based fusion rule, prioritizing edge and detail retention critical for precise medical diagnosis. Subsequently, an inverse NSST transforms these fused sub-bands, generating a preliminary fused image. This image is then further refined through a CNN-based noise reduction module comprising convolutional layers, batch normalization, and Rectified Linear Unit (ReLU) activation functions. The deep CNN architecture progressively refines the fused output, significantly reducing artifacts and noise, thus improving the overall clarity and interpretability of the resultant image. This approach addresses common limitations in image fusion, such as loss of edge details, distortion, and the inability to preserve critical information. The hybrid M3IF-NSST-CNN method improves both the visual quality and parametric accuracy of the fusion, outperforming comparable techniques. By preserving more detail and edge information, this method offers significant advancements in multi-modal medical image fusion for enhanced diagnostic insights.

With the rapid development of medical imaging methods, multimodal medical image fusion techniques have caught the interest of researchers. The aim is to preserve information from diverse sensors using various models to generate a single informative image. The main challenge is to derive a trade-off between the spatial and spectral qualities of the resulting fused image and the computing efficiency. This article proposes a fast and reliable method for medical image fusion depending on multilevel Guided edge-preserving filtering (MLGEPF) decomposition rule. First, each multimodal medical image was divided into three sublayer categories using an MLGEPF decomposition scheme: small-scale component, large-scale component and background component. Secondly, two fusion strategies-pulse-coupled neural network based on the structure tensor and maximum based-are applied to combine the three types of layers, based on the layers' various properties. The three different types of fused sublayers are combined to create the fused image at the end. A total of 40 pairs of brain images from four separate categories of medical conditions were tested in experiments. The pair of images includes various case studies including magnetic resonance imaging (MRI) , TITc, single-photon emission computed tomography (SPECT) and positron emission tomography (PET). We included qualitative analysis to demonstrate that the visual contrast between the structure and the surrounding tissue is increased in our proposed method. To further enhance the visual comparison, we asked a group of observers to compare our method's outputs with other methods and score them. Overall, our proposed fusion scheme increased the visual contrast and received positive subjective review. Moreover, objective assessment indicators for each category of medical conditions are also included. Our method achieves a high evaluation outcome on feature mutual information (FMI), the sum of correlation of differences (SCD), Qabf and Qy indexes. This implies that our fusion algorithm has better performance in information preservation and efficient structural and visual transferring.

Classification can significantly impact treatment decisions and patient outcomes. This study evaluates and compares the performance of three machine learning models Extreme Learning Machine (ELM), Long Short-Term Memory (LSTM), and Convolutional Neural Networks (CNN) in breast cancer classification. ELM, known for its fast-learning speed and strong generalization, is compared with LSTM, which is effective in capturing long-term dependencies in sequential data, and CNN, which is renowned for its ability to automatically extract features from images and structured data. The models were trained and tested on a breast cancer dataset, focusing on accuracy and computational efficiency. The results revealed that while CNNs demonstrated better accuracy in feature-rich data, LSTMs excelled in handling sequential data patterns. On the other hand, ELM offers a good balance between training speed and classification performance. This comparative analysis provides valuable insights into the strengths and limitations of each model, contributing to the development of more effective breast cancer diagnostic tools. In this case, LSTM outperformed ELM by 0.91%, outperformed CNN significantly by 3.72%, and outperformed Improved LSTM by 0.91%. This indicate that the LSTM model shows higher accuracy in breast cancer classification

In recent years, Convolutional Neural Networks (CNN) have revealed noteworthy enactment in classification of image jobs. Because of limited availability of annotated brain tumor MRI data, overfitting and lack of generalization are common challenges faced by traditional CNN models. To deal with these challenges, our studies explore the practice of Pre-Trained model of CNN and Transfer learning practices to expand the correctness of cataloguing the brain tumor through the data of MRI. Our experimental results demonstrate that the incorporation of Pre-Trained model of CNN with Transfer learning significantly improves the rain tumor’s classification accuracy on MRI data compared to traditional CNN models trained from scratch. Moreover, the research outcomes highlight potential of transfer learning as an operative approach for improving generalization capabilities of CNN models in medical imaging tasks with limited annotated data. By incorporating pre-trained CNN replicas and transfer erudition techniques, the objective of this studies aims to elevate brain tumor taxonomy accurateness with the assistance of MRI data.

Human visual system inspired multi-modal medical image fusion framework

A method for feature level image fusion for multimodal medical images in second generation wavelet domain (lifting wavelet transform domain) is proposed. The feature fused is edge and boundary information of input images that is extracted using wavelet transform modulus maxima criterion. The image fusion performance is evaluated by standard deviation, entropy, cross entropy and gradient parameters. Experimental results show that the proposed method gives better results for image fusion as image contrast, average information content and detail information of fused image are increased. This method has further advantages of fast implementation, flexibility, saving of auxiliary memory, property of perfect reconstruction and simplicity as we have used lifting wavelet transform. The reduction in computational complexity has been achieved by a factor of two as compared to the nonlifted wavelet transform.

In medical science, the role of medical images is very crucial to identify the diagnosis. Variety of medical images such as CT (Computed Tomography) images, MRI (magnetic resonance imaging) images are available but many times these different modalities need more informatics images. Therefore, multi-modality medical image fusion technique is one of the solutions which can provide more informatics details by fusing multi-modality medical images. In this paper, a new approach of multi-modality medical image fusion approach is proposed where fuzzy local information based C-means clustering and local entropy fusion approach is performed in SWT (stationary wavelet transform) Domain. To measure the performance of proposed scheme, a comparative study is also performed over the existing methods. The result analysis is performed in both visually and also by performance metrics such as mutual information, edge based similarity and blind image quality. From experimental evaluation, it can be analyzed that proposed scheme gives better outcomes in compare to existing methods.

Medical image fusion (MIF) significantly improves diagnosis accuracy by combining complementary data from several imaging modalities. However, the conventional fusion methods often fail to maintain the edge details, uniformity of texture and most relevant characteristics. To overcome these challenges, a novel multi-modality medical image fusion using Triple generator network (MIMO-TGAN) is proposed for accurate brain abnormality detection. The proposed MIMO-TGAN uses the multi-modality images such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and Positron Emission Tomography (PET) images for MIF process. Initially, the gathered MRI and CT images are denoised with scalable range-based adaptive trilateral (SCRAT) filter and PET images are processed by multi-scale retinex (MSR) technique. The proposed MIMO-TGAN model contains three generators with a discriminator and an abnormality detection phase. The Generator-1 consists of grouped convolutional with different attention blocks for retrieving the relevant features from MRI and CT images to generate a synthesized image. Simultaneously, Generator-2 retrieves the most important voxel features from PET images and MRI images to generate another synthesized image. Finally, Generator-3 is designed to combine the both synthesized images from two generators by using four fusion rules to accurately detect brain abnormalities. The experimental evaluations demonstrate that the proposed MIMO-TGAN achieves an accuracy of 99.43% and 98.24% for dataset-1and dataset-2 respectively. Moreover, the proposed MIMO-TGAN achieves the entropy values of 7.05 and 6.84 for dataset-1 and dataset-2, and overall SSIM of 0.94 that indicates superior fusion quality results compared to existing methods.

The synthesis of visual information from multiple medical imaging inputs to a single fused image without any loss of detail and distortion is known as multimodal medical image fusion. It improves the quality of biomedical images by preserving detailed features to advance the clinical utility of medical imaging meant for the analysis and treatment of medical disorders. This study develops a novel approach to fuse multimodal medical images utilizing anisotropic diffusion (AD) and non-subsampled contourlet transform (NSCT). First, the method employs anisotropic diffusion for decomposing input images to their base and detail layers to coarsely split two features of input images such as structural and textural information. The detail and base layers are further combined utilizing a sum-based fusion rule which maximizes noise filtering contrast level by effectively preserving most of the structural and textural details. NSCT is utilized to further decompose these images into their low and high-frequency coefficients. These coefficients are then combined utilizing the principal component analysis/Karhunen-Loeve (PCA/KL) based fusion rule independently by substantiating eigenfeature reinforcement in the fusion results. An NSCT-based multiresolution analysis is performed on the combined salient feature information and the contrast-enhanced fusion coefficients. Finally, an inverse NSCT is applied to each coefficient to produce the final fusion result. Experimental results demonstrate an advantage of the proposed technique using a publicly accessible dataset and conducted comparative studies on three pairs of medical images from different modalities and health. Our approach offers better visual and robust performance with better objective measurements for research development since it excellently preserves significant salient features and precision without producing abnormal information in the case of qualitative and quantitative analysis.