Blind single image super resolution with low computational complexity

Investigating single image super-resolution algorithm with deep learning using convolutional neural network for chest digital tomosynthesis

Special Section on Enabling Technologies for Network-Based Applications

Depth images play an important role and are popularly used in many computer vision tasks recently. However, the limited resolution of the depth image has been hindering its further applications. To address this problem, we propose a novel dual sparsity model based single depth image super resolution algorithm, with a single low-resolution depth image as input. We formulate this problem by combining the recently developed analysis model and synthesis model exploiting the sparsity of analyzed vectors and the sparse coefficients respectively. The analysis operator and dictionaries are trained over extensive samples separately. We show that our model clearly outperforms state-of-the-art methods on the widely used Middlebury 2007 datasets both quantitatively and visually.

In this paper, we propose a novel single image super-resolution algorithm that integrates a model-based approach with self-learning deep networks. The proposed method can be adapted to low-resolution (LR) images obtained with real acquisition devices where the point spread function is Gaussian-like. By modelling natural image lines as piece-wise smooth functions and approximating the blurring kernel with B-splines, an intermediate high-resolution (HR) image can be first obtained based on Finite Rate of Innovation theory. A self-supervised deep recursive residual network is then applied to further enhance the reconstruction quality. From the simulation results, our algorithm outperforms other self-learning algorithms and achieves state-of-the-art performance.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

In recent years, keypoints-based features, such as speeded up robust features (SURF), capture the attention of researchers in the area of image copy-move forgery detection. They are fast to compute and are quite robust to most inter-processing and post-processing operations. However, they will fail to detect the forgery if not enough keypoints are detected, and that is the case when the forgery size is small. A small size forgery is possible if the object is far away from the camera, or if the image is scaled down as a post-processing step. In this paper, our focus is on detecting forgeries of small size using an improved SURF based copy-move forgery detection (CMFD) method. Our method improves keypoints detection by preprocessing the image using a single image super resolution (SISR) algorithm. The proposed approach has been evaluated and compared against original SURF through a comprehensive set of experiments using a dataset of small size forgeries. Experimental results indicate that our method outperforms SURF especially when the forgery size is small.

Single image super-resolution (SR) algorithms based on joint dictionaries and sparse representations of image patches have received significant attention in the literature and deliver the state-of-the-art results. Recently, Gaussian mixture models (GMMs) have emerged as favored prior for natural image patches in various image restoration problems. In this paper, we approach the single image SR problem by using a joint GMM learnt from concatenated vectors of high and low resolution patches sampled from a large database of pairs of high resolution and the corresponding low resolution images. Covariance matrices of the learnt Gaussian models capture the inherent correlations between high and low resolution patches, which are utilized for inferring high resolution patches from given low resolution patches. The proposed joint GMM method can be interpreted as the GMM analogue of joint dictionary-based algorithms for single image SR. We study the performance of the proposed joint GMM method by comparing with various competing algorithms for single image SR. Our experiments on various natural images demonstrate the competitive performance obtained by the proposed method at low computational cost.

Single image super-resolution (SISR) is of great importance as a low-level computer vision task. The fast development of Generative Adversarial Network (GAN) based deep learning architectures realises an efficient and effective SISR to boost the spatial resolution of natural images captured by digital cameras. However, the SISR for medical images is still a very challenging problem. This is due to (1) compared to natural images, in general, medical images have lower signal to noise ratios, (2) GAN based models pre-trained on natural images may synthesise unrealistic patterns in medical images which could affect the clinical interpretation and diagnosis, and (3) the vanilla GAN architecture may suffer from unstable training and collapse mode that can also affect the SISR results. In this paper, we propose a novel lesion focused SR (LFSR) method, which incorporates GAN to achieve perceptually realistic SISR results for brain tumour MRI images. More importantly, we test and make comparison using recently developed GAN variations, e.g., Wasserstein GAN (WGAN) and WGAN with Gradient Penalty (WGAN-GP), and propose a novel multi-scale GAN (MS-GAN), to achieve a more stabilised and efficient training and improved perceptual quality of the super-resolved results. Based on both quantitative evaluations and our designed mean opinion score, the proposed LFSR coupled with MS-GAN has performed better in terms of both perceptual quality and efficiency.

Gradient boosting for single image super-resolution

Convolutional neural networks (CNN) have given rise to a new generation of video super-resolution (SR) technique. However, most existing CNN-based video SR algorithms treat the consecutive frames as a series of feature maps, just as the procedure performed in single image SR algorithms. We propose an end-to-end three-dimensional (3-D) CNN video SR framework. The input frames are considered as a cube in our framework. 3-D convolution is performed on it to extract features along spatial and temporal dimension. Image prior knowledge, such as optical flows, is introduced in reconstruction. A combination of mean square error loss and multiscale structure similarity index (MS-SSIM) loss is used to optimize the model. Experimental results show that the proposed method reconstructs high-resolution frames with more accurate and visually pleasant structures compared with state-of-the-art video SR algorithms. We also achieve comparable PSNR/SSIM results with less computation time.

Image super-resolution based on half quadratic splitting

The single image super resolution recovers missing high resolution details so as to reconstruct a high resolution image from a single low resolution image. This paper proposes a novel directionally adaptive, learning-based, single image super resolution method using multiple direction wavelet transform, called directionlets. Here, critically sampled directionlets are used to capture directional features effectively and to extract edge information along different directions from a set of available high resolution images. This information is used as the training set for super resolving a low resolution input image. The directionlet coefficients at finer scales of its high resolution image are learned locally from this training set and the inverse directionlet transform recovers the super resolved high resolution image. The simulation results showed that the proposed directionlet approach outperforms standard interpolation techniques like cubic spline interpolation as well as standard wavelet-based learning, both visually and in terms of the mean squared error (MSE) values. The SNR scores for cubic spline interpolation, wavelet and directionlet method are 13.6998 dB, 23.8324 dB and 30.8654 dB respectively for Barbara.

Image inpainting, a technique of completing missing or corrupted image regions in undetected form, is an open problem in digital image processing. Inpainting of large regions using Deep Convolutional Generative Adversarial Nets (DCGAN) is a new and powerful approach. In described approaches the size of generated image and size of input image should be the same. In this paper we propose a new method where the size of input image with corrupted region can be up to 4 times larger than generated image.

Super-resolution (SR) has become more vital, because of its capability to generate high-quality ultra-high definition (UHD) high-resolution (HR) images from low-resolution (LR) input images. Conventional SR methods entail high computational complexity, which makes them difficult to be implemented for up-scaling of full-high-definition input images into UHD-resolution images. Nevertheless, our previous super-interpolation (SI) method showed a good compromise between Peak-Signal-to-Noise Ratio (PSNR) performances and computational complexity. However, since SI only utilizes simple linear mappings, it may fail to precisely reconstruct HR patches with complex texture. In this paper, we present a novel SR method, which inherits the large-to-small patch conversion scheme from SI but uses global regression based on local linear mappings (GLM). Thus, our new SR method is called GLM-SI. In GLM-SI, each LR input patch is divided into 25 overlapped subpatches. Next, based on the local properties of these subpatches, 25 different local linear mappings are applied to the current LR input patch to generate 25 HR patch candidates, which are then regressed into one final HR patch using a global regressor. The local linear mappings are learned cluster-wise in our off-line training phase. The main contribution of this paper is as follows: Previously, linear-mapping-based conventional SR methods, including SI only used one simple yet coarse linear mapping to each patch to reconstruct its HR version. On the contrary, for each LR input patch, our GLM-SI is the first to apply a combination of multiple local linear mappings, where each local linear mapping is found according to local properties of the current LR patch. Therefore, it can better approximate nonlinear LR-to-HR mappings for HR patches with complex texture. Experiment results show that the proposed GLM-SI method outperforms most of the state-of-the-art methods, and shows comparable PSNR performance with much lower computational complexity when compared with a super-resolution method based on convolutional neural nets (SRCNN15). Compared with the previous SI method that is limited with a scale factor of 2, GLM-SI shows superior performance with average 0.79 dB higher in PSNR, and can be used for scale factors of 3 or higher.

Nowadays, super-resolution is becoming more and more important in most optical imaging systems and image processing applications due to current resolution limit of charged couple device (CCD) and complementary metal-oxide semiconductor (CMOS). In this paper, we proposed a novel single-image super-resolution algorithm, which combines collaborative representation into manifold-preserving approach. The main contributions of our work can be summarized into two points. First, supporting bases which are used to calculate the mapping relationship between low-resolution (LR) images and high-resolution (HR) images are obtained by applying collaborative representation on the neighborhoods of the dictionary atoms, where the neighborhoods are clustered from the whole training sample pools. Second, to achieve a balance between execution speed and reconstruction quality, a global solution for our framework is constructed by transferring the online calculating process to offline. We demonstrate better results on commonly used datasets, showing both better visual performance and higher index values compared to other methods.