Medical X-ray images enhancement based on super resolution convolution neural network

The thermal camera systems can be used in all kinds of applications that require the detection of heat change, but thermal imaging systems are highly costly systems. In recent years, developments in the field of deep learning have increased the success by obtaining quality results compared to traditional methods. In this paper, thermal images of neonates (healthy - unhealthy) obtained from a high-resolution thermal camera were used and these images were evaluated as high resolution (ground truth) images. Later, these thermal images were downscaled at 1/2, 1/4, 1/8 ratios, and three different datasets consisting of low-resolution images in different sizes were obtained. In this way, super-resolution applications have been carried out on the deep network model developed based on generative adversarial networks (GAN) by using three different datasets. The successful performance of the results was evaluated with PSNR (peak signal to noise ratio) and SSIM (structural similarity index measure). In addition, healthy - unhealthy classification application was carried out by means of a classifier network developed based on convolutional neural networks (CNN) to evaluate the super-resolution images obtained using different datasets. The obtained results show the importance of combining medical thermal imaging with super-resolution methods.

Deep Learning image processing methods are gradually gaining popularity in a number of areas including medical imaging. Classification, segmentation, and denoising of images are some of the most demanded tasks. In this study, we aim at enhancing optic nerve head images obtained by Optical Coherence Tomography (OCT). However, instead of directly applying noise reduction techniques, we use multiple state-of-the-art image Super-Resolution (SR) methods. In SR, the low-resolution (LR) image is upsampled to match the size of the high-resolution (HR) image. With respect to image enhancement, the upsampled LR image can be considered as low quality, noisy image, and the HR image would be the desired enhanced version of it. We experimented with several image SR architectures, such as super-resolution Convolutional Neural Network (SRCNN), very deep Convolutional Network (VDSR), deeply recursive Convolutional Network (DRCN), and enhanced super-resolution Generative Adversarial Network (ESRGAN). Quantitatively, in terms of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), the SRCNN, VDSR, and DRCN significantly improved the test images. Although the ERSGAN showed the worst PSNR and SSIM, qualitatively, it was the best one.

Super-resolution (SR) can be defined as a method of reconstructing high resolution(HR) image by increasing spatial resolution of one or more low resolution(LR) image counterparts. SR is widely applied in areas like satellite imaging, drones, surveillance and medical imaging etc. SR techniques can be broadly classified as external database driven or internal database driven approaches. The first approach is time consuming during training phase, in learning LR-HR patch relationship but computes fast during testing phase. The second approach directly constructs the super resolved image from existing LR image without the need of any training procedures but found to be computationally intensive during test time. This paper focuses on reducing the computation time of transformed self exemplar based internal database driven technique [1] that uses 3D scene geometry and internal image characteristics for super resolution by exploiting its highly parallelizable nature. On parallelizing the original algorithm we achieved a speedup of 6 on multi-core, upto 10 on GPU, around 14 on hybrid platform of multi-core and GPU. Using our hybrid implementation we can achieve 32x super resolution factor. We compare our results with SRCNN[2] (Super Resolution Convolution Neural Network) an external database SR technique which has a very good performance among the current start-of-the-art SR methods.

Low-resolution medical images can seriously interfere with the medical diagnosis, and poor image quality can lead to loss of detailed information. Therefore, improving the quality of medical images and accelerating the reconstruction is of particular importance for diagnosis. To solve this problem, we propose a wavelet-based mini-grid network medical image super-resolution (WMSR) method, which is similar to the three-layer hidden-layer-based super-resolution convolutional neural network (SRCNN) method. Due to the amplification characteristics of wavelets, a stationary wavelet transform (SWT) is used instead of a discrete wavelet transform (DWT). Also, due to the nature of redundant (scale-by-scale) wavelets, it is possible to retain additional information about the image and restore high-resolution images in detail. For a large amount of training data, wavelet sub-band images, including approximation and frequency subbands are combined into a predefined full-scale factor. The mapping between the wavelet sub-band image and its approximate image is then determined. In order to ensure the reproducibility of the image, a method of adding a sub-pixel layer is proposed to realize the hidden layer, and replacing the small mini-grid-network on the hidden layer is of considerable significance to speed up the image recovery speed. Experimental results on the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) show that the model has better performance.

Lake surface water temperature (LSWT) is an important factor of water ecological environment. As global warming, LSWT is also on the rise. Research on the main reasons of LSWT rising is the basis for controlling and improving the regional ecological environment. However, it is difficult for the existing remote sensing images to take into account the temporal and spatial resolution. Low-resolution images have a serious impact on data accuracy and even produce incorrect results. Therefore, obtaining high temporal and spatial resolution images by downscaling is of great significance to more accurately analyze the temporal and spatial characteristics of LSWT. In this article, Dianchi Lake is selected as research area, and the high spatial resolution image (Landsat) and high temporal resolution image (MODIS) are taken as data. Based on the downscaling algorithm of statistics and learning, DisTrad– super-resolution convolutional neural network (SRCNN) downscaling model is proposed, and the monthly average dataset of LSWT with 50 m resolution is constructed. The results showed 1) DisTrad–SRCNN can reflect the most distribution characteristics of LSWT (SSIMday = 0.96, PSNRday = 23.97; SSIMnight = 0.95, PSNRnight = 24.99). 2) LSWT had an overall upward trend (CRday = 0.22 °C/10a, CRnight = 0.21 °C/10a), showing a cyclical change of cold–warm–cold about 4 years. 3) The northern and lakeshore area were basically in the high temperature, and the whole lake presents a 4–5-year warm–cold–warm periodic change; the LSWT closer to the urban and residential areas and its change rate were relatively high, which indirectly reflected the serious impact of human activities on LSWT.

This paper presents a new quality assessment parameter for the evaluation of deep learning based super resolution techniques applied on thermal images. Three widely used deep learning-based models namely Super-Resolution Convolutional Neural Network (SRCNN), Thermal Enhancement Network (TEN) and Very Deep Super Resolution (VDSR) have been implemented for achieving super resolution of thermal images. Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM) have been the most widely used conventional evaluation metrics for performance measurement of super resolution algorithms. Since these parameters require a reference image for the evaluation of the resultant images, we propose a new quality assessment parameter based on strength of the edges. Edge detection of the super resoluted image is performed utilizing Canny Edge Detection method and the entropy of the edge detection image is computed to provide a new parameter, Edge Detection Entropy Score (EDES). For the comparison and validation of the proposed image quality assessment techniques, Mean Opinion Score (MOS) of the target images have been obtained to be used as a benchmark. The obtained results indicate that the proposed EDES of the super resoluted images has high degree of correlation with the MOS as well as PSNR and SSIM.

The size of pixels in a digital recording device, such as a CCD array, limits the spatial resolution in images obtained by an optical imaging system, thereby degrading the image quality. Digital super-resolution (DSR) techniques are used to reconstruct a high-resolution (HR) image from multiple sub-pixel-shifted low-resolution images in order to improve the image quality. In this article, we formulate a mathematical framework for DSR using the generalized sampling theorem (GST). The GST-based DSR method’s performance is evaluated by comparing it to existing resolution enhancement methods on the basis of evaluation metrics like percentage mean square error (%MSE), structural similarity index measure (SSIM), and peak signal-to-noise ratio (PSNR). The GST DSR method exhibits an overall superior quality image reconstruction based on quantitative analysis (with a near zero %MSE, SSIM around one, and improved PSNR in dB) and qualitative comparison. The robustness of the GST DSR method is further demonstrated in the presence of frame-to-frame shift estimation error using %MSE and SSIM by comparing it with multi-frame interpolation approaches.

Tropospheric Black Carbon (BC) aerosols are short lived positive radiative forcer with critical impacts on cardiovascular and pulmonary health. The MRI-ESM2(CMIP6) delivers the BC monthly global surface concentration from 1950-2100 at coarse special resolution of 1.875° × 1.875° which does not capture the city level BC hotspots at the global scale. The BC hotspots are essential for reducing the BC aerosols induced health burden, implementing air quality management policies and regional planning. The downscaling algorithm is an enhanced U-net with attention-based convolution neural network (Super Resolution Convolution Neural Network (SRCNN)). The SRCNN model executes downscaling of MRI-ESM2(CMIP6) BC monthly surface concentration with special resolution of 1.875° × 1.875° to NASA’s MERRA2 reanalysis BC monthly average surface concentration at a spatial resolution of 0.5° × 0.625°, thus achieving 3.6 times downscaling for identification city level BC-hotspots and cold spots at a global scale. The SRCNN model is trained on global monthly average BC surface concentration data from MRI-ESM2(CMIP6) and NASA’s MERRA2 reanalysis product. The model training is spread from 1980-2012(31 years) with validation from 2013-2016(4 years) and testing for 2017-2020(4 years). We have also examined the effects of Channel Based Attention Module (CBAM) with and without Residual Block (RB) and their effectiveness and efficacy in climate data downscaling with data-scarce condition. The training results showes that CBAM with RB based CNN outperforms then both (CNN without CBAM and without CBAM & RB) in the benchmarks such as stability, overfitting, validation losses etc. The training results for SRCNN (with CBAM and RB) shows a final validation losses of 0.0028, final R² value of 0.7162, final Pearson-r value of 0.8467 with Structural Similarity Index Measure (SSIM) at 0.9954. The SRCNN (with CBAM & RB) model testing reveals it performs exceptionally well in the identification of hotspots and cold-spots, with final testing RSME at 0.0015, final R2 at 0.88, final Pearson-r values at 0.94 and final SSIM at 0.99. Furthermore, testing outputs of SRCNN with attention module and residual blocks shows close fidelity with MERRA2 reanalysis vis-à-vis MRI-ESM2(CMIP6) at both seasonal and annual temporal resolution thus reducing systematic bias between ground truth and global climate models.

Deep Feature Blend Attention: A New Frontier in Super Resolution Image Generation

Super-resolution (SR) has received extensive attention in recent years for satellite image processing in a wide range of application scenarios, such as land classification, identification of changes, the discovery of resources, etc. Satellite images from satellite sensors are mostly low-resolution (LR) images, so they do not completely fulfill object detection and analysis criteria. SR has multiple residual network frameworks in deep learning that have improved performance and can extend thousands of layers in the system. However, each layer improves accuracy by doubling the number of layers, although training thousands of layers are too expensive, the process is slow, and there are functional recovery issues. We proposed a transferred wide residual Single Image Super-Resolution (SISR) remote sensing deep neural network model (WRSR). By increasing the width and reducing the residual network depth, the proposed approach has dramatically reduced memory costs. As a result, our model reduced memory costs by 21% in Enhanced Deep Residual Super-Resolution (EDSR) and 34% in SRResNet as a direct consequence of the in-depth reduction. The proposed architecture improves the efficiency of training loss by performing weight normalization instead of augmentation technology. We compared our method to five recent existing super-resolution (SR) deep neural network methods, tested over three public satellite image datasets and a standard reference (PRIM) dataset. Experiment analysis is evaluated in peak to signal noise ratio (PSNR) and structural similarity index measure (SSIM).

Medical image super-resolution reconstruction algorithms based on deep learning: A survey

Deep learning techniques not only have surpassed humans in several computer vision tasks, but also have achieved state-of-the-art performance on the super-resolution (SR) task, which enhances the resolution of reconstructed images from the observed low-resolution (LR) images. Very Deep Super- Resolution (VDSR) is a popular architecture with decent performance on the SR task. In general, the deeper the VDSR network, the better the performance. Say, on a computing device with limited computational resources, the SR task must support multiple resolutions or multiple output rates. For this multi-rate SR task, the state-of-the-art architectures require multiple networks with varying depths. However as the number of supported rates increases, so does the number of networks imposing incremental computational burden on the computing device. We propose a depth-controllable network and training principles for the multi-rate SR task. The proposed network is configured as a single network regardless of the number of supported rates. The inverse auxiliary loss and contiguous/progressive skip connections are presented to train the network end-to-end throughout the varying number of layers without biasing the performances of specific depths. With three data sets and three scaling factors, the proposed network is compared to the baseline network VDSR, along with the Super-Resolution Convolutional Neural Network (SRCNN). Our network not only requires a single network at varying rates, but also performs as well as the baseline networks in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). Our model outperforms the baseline networks when the depth of layers is more than eleven.

We use images in several important areas such as military, health, security, and science. Images can be distorted during the capturing, recording, processing, and storing. Image quality metrics are the techniques to measure the quality and quality accuracy level of the images and videos. Most of the quality measurement algorithms does not affect by small distortions in the image. Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and Ultrasonic Imaging (UI) are widely used in the health sector. Because of several reasons it might be artifacts in the medical images. Doctor decisions might be affected by these image artifacts. Image quality measurement is an important and challenging area to work on. There are several metrics that have been done in the literature such as mean square error, peak signal-noise ratio, gradient similarity measure, structural similarity index, and universal image quality. Patient information can be an embedded corner of the medical image as a watermark. Watermark can be considered one of the image distortions types. The most common objective evaluation algorithms are simple pixel based which are very unreliable, resulting in poor correlation with the human visual system. In this work, we proposed a new image quality metric which is a Measure of Singular Value Decomposition (M-SVD). Experimental results show that novel M-SVD algorithm gives very promising results against Peak Signal to Noise Ratio (PSNR), the Mean Square Error (MSE), Structural Similarity Index Measures (SSIM), and 3.4. Universal Image Quality (UIQ) assessments in watermarked and distorted images such as histogram equalization, JPEG compression, Gamma Correction, Gaussian Noise, Image Denoising, and Contrast Change.

Image super-resolution is of great interest in medical imaging field. However, different from natural images studied in computer vision field, the low-resolution (LR) medical imaging data is often a stack of high-resolution (HR) 2D slices with large slice thickness. Consequently, the goal of super-resolution for medical imaging data is to reconstruct the missing slice(s) between any two consecutive slices. Since some modalities (e.g., T1-weighted MR image) are often acquired with high-resolution (HR) image, it is intuitive to harness the prior self-similarity information in the HR image for guiding the super-resolution of LR image (e.g., T2-weighted MR image). The conventional way is to find the profile of patchwise self-similarity in the HR image and then use it to reconstruct the missing information at the same location of LR image. However, the local morphological patterns could vary significantly across the LR and HR images, due to the use of different imaging protocols. Therefore, such direct (un-supervised) adaption of self-similarity profile from HR image is often not effective in revealing the actual information in the LR image. To this end, we propose to employ the existing image information in the LR image to supervise the estimation of self-similarity profile by requiring it not only being optimal in representing patches in the HR image, but also producing less reconstruction errors for the existing image information in the LR image. Moreover, to make the anatomical structures spatially consistent in the reconstructed image, we simultaneously estimate the self-similarity profiles for a stack of patches across consecutive slices by solving a group sparse patch representation problem. We have evaluated our proposed super-resolution method on both simulated brain MR images and real patient images with multiple sclerosis lesion, achieving promising results with more anatomical details and sharpness.

Most of the medical images are usually affected by different types of noises during acquisition, storage, and transmission. These images need to be free from noise for better diagnosis, decision, and results. Thus, denoising technique plays an important role in medical image analysis. This paper presents a method of noise removal for brain magnetic resonance imaging (MRI) image using curvelet transform thresholding technique combined with the Wiener filter and compares the result with the curvelet and wavelet-based denoising techniques. To assess the quality of denoised image, the values of peak signal-to-noise ratio (PSNR), mean square error (MSE), and structural similarity index measure (SSIM) are considered. The experimental results show that curvelet denoising method depicts better result than wavelet denoising method, but the combined method of curvelet with Wiener filtering technique is more effective than the wavelet- and curvelet-based denoising method in terms of PSNR, MSE, and SSIM.