FMSRdiff: Efficient latent diffusion framework combining consistency model and flow matching for super-resolution

CMISR: Circular medical image super-resolution

(Conférencier invité)

This paper provides an in-depth review of recent advancements in image super-resolution (SR) techniques using deep learning, with a specific focus on their applications in medical imaging. In the medical field, high-resolution images are crucial for accurate diagnosis and treatment planning. However, due to hardware limitations, it is often challenging to obtain high-resolution (HR) medical images. Traditional image enhancement techniques are not always sufficient to recover fine details. Deep learning-based methods, particularly those leveraging convolutional neural networks (CNNs), generative adversarial networks (GANs), and transformer-based models, have shown remarkable success in overcoming these challenges. The review explores the evolution of these techniques, their advantages, and the challenges involved in medical image super-resolution. The paper also highlights the future directions of research to improve SR for medical imaging, addressing concerns related to image quality, data availability, and computational efficiency.

Low-resolution medical images can seriously interfere with the medical diagnosis, and poor image quality can lead to loss of detailed information. Therefore, improving the quality of medical images and accelerating the reconstruction is of particular importance for diagnosis. To solve this problem, we propose a wavelet-based mini-grid network medical image super-resolution (WMSR) method, which is similar to the three-layer hidden-layer-based super-resolution convolutional neural network (SRCNN) method. Due to the amplification characteristics of wavelets, a stationary wavelet transform (SWT) is used instead of a discrete wavelet transform (DWT). Also, due to the nature of redundant (scale-by-scale) wavelets, it is possible to retain additional information about the image and restore high-resolution images in detail. For a large amount of training data, wavelet sub-band images, including approximation and frequency subbands are combined into a predefined full-scale factor. The mapping between the wavelet sub-band image and its approximate image is then determined. In order to ensure the reproducibility of the image, a method of adding a sub-pixel layer is proposed to realize the hidden layer, and replacing the small mini-grid-network on the hidden layer is of considerable significance to speed up the image recovery speed. Experimental results on the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) show that the model has better performance.

Medical image super-resolution using a relativistic average generative adversarial network

Single image super-resolution (SISR) is of great importance as a low-level computer vision task. The fast development of Generative Adversarial Network (GAN) based deep learning architectures realises an efficient and effective SISR to boost the spatial resolution of natural images captured by digital cameras. However, the SISR for medical images is still a very challenging problem. This is due to (1) compared to natural images, in general, medical images have lower signal to noise ratios, (2) GAN based models pre-trained on natural images may synthesise unrealistic patterns in medical images which could affect the clinical interpretation and diagnosis, and (3) the vanilla GAN architecture may suffer from unstable training and collapse mode that can also affect the SISR results. In this paper, we propose a novel lesion focused SR (LFSR) method, which incorporates GAN to achieve perceptually realistic SISR results for brain tumour MRI images. More importantly, we test and make comparison using recently developed GAN variations, e.g., Wasserstein GAN (WGAN) and WGAN with Gradient Penalty (WGAN-GP), and propose a novel multi-scale GAN (MS-GAN), to achieve a more stabilised and efficient training and improved perceptual quality of the super-resolved results. Based on both quantitative evaluations and our designed mean opinion score, the proposed LFSR coupled with MS-GAN has performed better in terms of both perceptual quality and efficiency.

Medical Image Super-Resolution has always been a subject of interest in medical image processing. However, super-resolved retinal images are a requisite tool for doctors to properly diagnose and treat ophthalmic diseases. The acquisition of high-quality images is challenging owing to several factors including technical hardware limitations, high cost, operator skills, data compatibility, and maintenance issues. This paper proposes Retinal-ESRGAN, a novel hybrid unsupervised GAN model particularly designed for retinal image super-resolution. The model incorporates architectural modifications with respect to generator and discriminator network using Google Colaboratory and TensorFlow 2.0 facilitating limited resource usage. To address resource constraints, a training strategy involving pausing and resuming in batches is implemented. The experiments conducted have demonstrated Retinal-ESRGAN’s potential that achieved an average PSNR of 35.22 dB and SSIM of 0.916, outperforming both SRGAN and ESRGAN showing PSNR metric improvement of 4.8% over SRGAN and 10.5% improvement over ESRGAN. Also, a 5.7% improvement over SRGAN and 22.4% improvement over ESRGAN in SSIM metric, Inception score of 6.02, Fréchet Inception Distance of 25.31 and accuracy of 94.98% utilizing significantly less training time and computational resources.

This paper proposes a new medical image super-resolution (SR) network, namely deep multi-scale network (DMSN), in the uniform discrete curvelet transform (UDCT) domain. DMSN is made up of a set of cascaded multi-scale fushion (MSF) blocks. In each MSF block, we use convolution kernels of different sizes to adaptively detect the local multi-scale feature, and then local residual learning (LRL) is used to learn effective feature from preceding MSF block and current multi-scale features. After obtaining multi-scale features of different MSF block, we use global feature fusion (GFF) to jointly and adaptively learn global hierarchical features in a holistic manner. Finally, compared with other prediction methods in spatial domain, we applied DMSN in UDCT domain, which enables a better representation of global topological structure and local texture detail of HR images. DM-SN shows superior performance over other state-of-the-art medical image SR methods.

To solve the problems of image detail blurring and insufficient utilization of global information in the existing medical image super-resolution reconstruction, this paper proposes a dual-branch fusion network based on residual Transformer network and dynamic convolutional neural network (CTGFSR). The network consists of two branches, one is the global branch based on residual Transformer network, and the other is the local branch based on dynamic convolutional neural network. The global branch uses the self-attention mechanism of Transformer network, which can effectively mine the large-scale global information in the image and improve the overall quality of the image. The local branch uses the characteristic of dynamic convolution to adaptively adjust the convolution kernel parameters, which can enhance the feature extraction ability of convolutional neural network for multi-scale information and improve the detail restoration ability of the image without significantly increasing the network model size. The network uses residual skip connections to preserve the detail information in medical image super-resolution reconstruction. Finally, through the bidirectional gated attention mechanism, the two branches are fused to obtain the final super-resolution reconstruction image. This paper evaluates the performance of the network on two medical image datasets, namely ACDC abdominal MR related to medical image segmentation and L2R2022 lung CT related to registration. The experimental results show that compared with the mainstream super-resolution algorithms, CTGFSR has better overall performance. When the magnification factor is 2 or 4, compared with the convolutional neural network based CFIPC, PDCNCF, ESPCN, FSRCNN, VDSR and the Transformer network based ESRT, SwinIR, the structural similarity SSIM and peak signal-to-noise ratio PSNR have a certain improvement.

Image resolution enhancement is a problem of considerable interest in all medical imaging modalities. Unlike general purpose imaging or video processing, for a very long time, medical image resolution enhancement has been based on optimization of the imaging devices. Although some recent works purport to deal with image postprocessing, much remains to be done regarding medical image enhancement via postprocessing, especially in ultrasound imaging. We face a resolution improvement issue in the case of medical ultrasound imaging. We propose to investigate this problem using multidimensional autoregressive (AR) models. Noting that the estimation of the envelope of an ultrasound radio frequency (RF) signal is very similar to the estimation of classical Fourier-based power spectrum estimation, we theoretically show that a domain change and a multidimensional AR model can be used to achieve super-resolution in ultrasound imaging provided the order is estimated correctly. Here, this is done by means of a technique that simultaneously estimates the order and the parameters of a multidimensional model using relevant regression matrix factorization. Doing so, the proposed method specifically fits ultrasound imaging and provides an estimated envelope. Moreover, an expression that links the theoretical image resolution to both the image acquisition features (such as the point spread function) and a postprocessing feature (the AR model) order is derived. The overall contribution of this work is threefold. First, it allows for automatic resolution improvement. Through a simple model and without any specific manual algorithmic parameter tuning, as is used in common methods, the proposed technique simply and exclusively uses the ultrasound RF signal as input and provides the improved B-mode as output. Second, it allows for the a priori prediction of the improvement in resolution via the knowledge of the parametric model order before actual processing. Finally, to achieve the previous goal, while classical parametric methods would first estimate the model order and then the model parameters, our approach estimates the model parameters and the order simultaneously. The effectiveness of the methodology is validated using two-dimensional synthetic and in vivo data. We show that, compared to other techniques, our method provides better results from a qualitative and a quantitative viewpoint.

Medical image super-resolution reconstruction algorithms based on deep learning: A survey

<p>Pneumonia is a severe lung infection, chest X-ray (CXR) image preferred to find infection. Real images lost its quality, resolution and other feature due to transmission. So good qualitative datasets are very limited. Quality enhancement in medical images is challenging task for researchers. And quality in clinical diagnosis of any disease in deep learning play a very important role. So, this paper presents an aspect with importance of quality in medical images CXR of a particular dataset and how to enhance and create new images with high quality resolution, that is re-used for classification in deep learning. Super resolution convolutional neural netwok (SRCNN) is deep learning based method, which is used for improving resolution in image. Super resolution means low resolution (LR) images from dataset is to be reconstructed or magnified into high resolution (HR). The objective behind this study is to measure the effect of super resolution with quality index, peak signal-to-noise ratio (PSNR), mean squared error (MSE), and structural similarity index measure (SSIM). This experinment performed on 200 images with 10 batches, each batch has 20 images from Kermany dataset, select LR images and converted into HR with SRCNN. Then we find PSNR value of image is increase upto 2 to 5 DB, and MSE of qood quality images is near to zero and MSE decrease up to 20-25, SSIM value have little variation due to same pattern is found in input and output images. Enhancement means highlight or improve the region of interest of pneumonic images. Main goal of this study is to preapare a modified dataset which is further used for classification.</p>

Medical image super-resolution (SR) is an active research area that has many potential applications, including reducing scan time, bettering visual understanding, increasing robustness in downstream tasks, etc. However, applying deep-learning-based SR approaches for clinical applications often encounters issues of domain inconsistency, as the test data may be acquired by different machines or on different organs. In this work, we present a novel algorithm called domain adaptable volumetric super-resolution (DA-VSR) to better bridge the domain inconsistency gap. DA-VSR uses a unified feature extraction backbone and a series of network heads to improve image quality over different planes. Furthermore, DA-VSR leverages the in-plane and through-plane resolution differences on the test data to achieve a self-learned domain adaptation. As such, DA-VSR combines the advantages of a strong feature generator learned through supervised training and the ability to tune to the idiosyncrasies of the test volumes through unsupervised learning. Through experiments, we demonstrate that DA-VSR significantly improves super-resolution quality across numerous datasets of different domains, thereby taking a further step toward real clinical applications.

This paper proposes a novel super-resolution method that exploits the sparse representation and non-local similarity of patches for the effective reconstruction of images. Highresolution images are reconstructed from low resolution observations with an efficient technique based on the alternating direction method of multipliers (ADMM). A robust iterative back-projection approach is used in a post-processing step to remove residual noise and artifacts in the reconstructed image. Experiments on benchmark medical images illustrate the advantage of our method, in terms of PSNR and SSIM, compared to state of the art approaches.

Generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) play an increasingly important role in medical image analysis. The latent spaces of these models often show semantically meaningful directions corresponding to human-interpretable image transformations. However, until now, their exploration for medical images has been limited due to the requirement of supervised data. Several methods for unsupervised discovery of interpretable directions in GAN latent spaces have shown interesting results on natural images. This work explores the potential of applying these techniques on medical images by training a GAN and a VAE on thoracic CT scans and using an unsupervised method to discover interpretable directions in the resulting latent space. We find several directions corresponding to non-trivial image transformations, such as rotation or breast size. Furthermore, the directions show that the generative models capture 3D structure despite being presented only with 2D data. The results show that unsupervised methods to discover interpretable directions in GANs generalize to VAEs and can be applied to medical images. This opens a wide array of future work using these methods in medical image analysis. The code and animations of the discovered directions are available online at https://github.com/julschoen/Latent-Space-Exploration-CT.KeywordsGenerative modelsUnsupervised learningInterpretabilityCT