Detail warping based video super-resolution using image guides

Many consumer digital cameras support dual shooting mode of both low-resolution (LR) video and high-resolution (HR) image. By periodically switching between the video and image modes, this type of cameras make it possible to super-resolve the LR video with the assistance of neighboring HR still images. We propose a model-based video super-resolution (VSR) technique for the above dual-mode cameras. A HR video frame is modeled as a 2D piecewise autoregressive (PAR) process. The PAR model parameters are learnt from the HR still images inserted between LR video frames. By registering the LR video frames and the HR still images, we base the learning on sample statistics that matches the scene to be constructed. The resulting PAR model is more accurate and robust than if the model parameters are estimated from the LR video frames without referring to the HR images or from a training set. Aided by the powerful scene-matched model the LR video frame is upsampled to the resolution of the HR image via adaptive interpolation. As such, the proposed VSR technique does not require explicit motion estimation of subpixel precision nor the solution of a large-scale inverse problem. The new VSR technique is competitive in visual quality against existing techniques with a fraction of the computational cost.

With the popular of stereo video and free-viewpoint video, binocular and multi-view video enhancement has attracted increasing attention. Current binocular video enhancement methods mainly focus on stereo super-resolution. In this paper, we tend to discuss a new binocular video resolution and frame-rate enhancement scenario to fully utilize the cross-view complementary information. Specifically, one view is captured with high resolution (HR) and low frame-rate (LFR), while the other viewpoint records low resolution (LR) and high frame-rate (HFR) video. Then, a binocular video joint enhancement network, which adopts dual-branch structure with cross-view guidance, is proposed to jointly reconstruct HR and HFR stereo videos. The proposed framework can reduce the capture, storage, compression, and transmission cost of normal HR and HFR stereo videos. Compared with single-view super-resolution and video frame interpolation techniques, the proposed method can recover more realistic HR details and intermediate motion by using cross-view reference. Experimental results on stereo video datasets demonstrate the effectiveness of the proposed joint resolution and frame-rate enhancement framework.

Super-resolution (SR) is a classic problem in image processing. Super-resolution of Video (SRV) aims to build video with high resolution from the one with low resolution. In this paper, we presented a novel SRV scheme on the base of r/KPSO. After modeling the relationship between high and low resolution video, the r/KPSO algorithm was utilized to solve the relationship model to determine the high resolution one. With the consideration of rebuilt video’s quality, objective function of the model was designed to contain gradient and second-order differential, for smoothness and fine details. Experimental results of our proposed method showed higher objective performance. And besides objective criteria, the proposed method could also obtain better subjective quality with finer details and higher sharpness.

In video surveillance, low resolution in face recognition is a major problem. Various Super Resolution (SR) approaches are introduced to perform the high resolution of face video recognition from low resolution videos. However, enhancing the resolution of face videos and reconstructing the high frequency data is a major problem in research area. Therefore, an effective face video Super Resolution method-based on Deep Convolutional Neural Network (Deep CNN) is introduced in this paper to achieve the face resolution effectively. Initially, the input video collected from the database is passed into the frame extraction stage, where the video frames are extracted and the face detection is carried out using the Viola Jones algorithm. Moreover, the detected image frame is processed by the Deep Convolutional Neural Network (Deep CNN) to enhance the image resolution. Deep CNN is highly effective in performing super resolution in face videos. However, the proposed Deep Convolutional Neural Network attains better performance using the metrics, like Second Derivative like Measure of Enhancement (SDME) as 0.9743 using video-1, and Feature SIMilarity index (FSIM) as 53.843 for video-4, respectively.

Video or image super-resolution technology is designed to recovery a high-resolution image from a low-resolution video or image. In recent years, deep neural networks have developed rapidly and have been applied in many digtal image processing tasks. In this paper, we choose an optical flow network to effectively exploit temporal relation within multiple consecutive video frames. In addition, we propose a weight distribution network which gives weight images of different high-resolution images obtained by various super-resolution network methods. This architecture combines advantages of different methods and provides a more accurate high-resolution image. We build a dataset with high-definition video, and use this dataset to train and test our networks. We compare our algorithm with other super-resolution methods and show that it performs a state-of-the-art results.

Most of the previous sparse coding (SC) based super resolution (SR) methods partition the image into overlapped patches, and process each patch separately. These methods, however, ignore the consistency of pixels in overlapped patches, which is a strong constraint for image reconstruction. In this paper, we propose a convolutional sparse coding (CSC) based SR (CSC-SR) method to address the consistency issue. Our CSC-SR involves three groups of parameters to be learned: (i) a set of filters to decompose the low resolution (LR) image into LR sparse feature maps, (ii) a mapping function to predict the high resolution (HR) feature maps from the LR ones, and (iii) a set of filters to reconstruct the HR images from the predicted HR feature maps via simple convolution operations. By working directly on the whole image, the proposed CSC-SR algorithm does not need to divide the image into overlapped patches, and can exploit the image global correlation to produce more robust reconstruction of image local structures. Experimental results clearly validate the advantages of CSC over patch based SC in SR application. Compared with state-of-the-art SR methods, the proposed CSC-SR method achieves highly competitive PSNR results, while demonstrating better edge and texture preservation performance.

In various unmanned aerial vehicle (UAV) imaging applications, the multisensor super-resolution (SR) technique has become a chronic problem and attracted increasing attention. Multisensor SR algorithms utilize multispectral low-resolution (LR) images to make a higher resolution (HR) image to improve the performance of the UAV imaging system. The primary objective of the paper is to develop a multisensor SR method based on the existing multispectral imaging framework instead of using additional sensors. In order to restore image details without noise amplification or unnatural post-processing artifacts, this paper presents an improved regularized SR algorithm by combining the directionally-adaptive constraints and multiscale non-local means (NLM) filter. As a result, the proposed method can overcome the physical limitation of multispectral sensors by estimating the color HR image from a set of multispectral LR images using intensity-hue-saturation (IHS) image fusion. Experimental results show that the proposed method provides better SR results than existing state-of-the-art SR methods in the sense of objective measures.

Recently, the demand for face recognition to identify persons from surveillance video cameras has rapidly increased. Since surveillance cameras are usually placed at positions far from a person's face, the quality of face images captured by the cameras tends to be low. This degrades the recognition accuracy. Therefore, aiming to improve the accuracy of the low-resolution-face recognition, we propose a video-based super-resolution method. The proposed method can generate a high-resolution face image from low-resolution video frames including non-rigid deformations caused by changes of face poses and expressions without using any positional information of facial feature points. Most existing techniques use the facial feature points for image alignment between the video frames. However, it is difficult to obtain the accurate positions of the feature points from low-resolution face images. To achieve the alignment, the proposed method uses a free-form deformation method that flexibly aligns each local region between the images. This enables super-resolution of face images from low-resolution videos. Experimental results demonstrated that the proposed method improved the performance of super-resolution for actual videos in terms of both image quality and face recognition accuracy.

BackgroundThe emotional state of individuals is difficult to identify and it is developing now a days because of vast interest in recognition. Many technologies have been developed to identify this emotional expression based on facial expressions, vocal expressions, physiological signals, and body expressions. Among these, facial emotion is very expressive for recognition using multimodalities. Understanding facial emotions has applications in mental well‐being, decision‐making, and even social change, as emotions play a crucial role in our lives. This recognition is complicated by the high dimensionality of data and non‐linear interactions across modalities. Moreover, the way emotion is expressed by people varies and these feature identification remains challenging, where these limitations are overcome by Deep learning models.MethodsThis research work aims at facial emotion recognition through the utilization of a deep learning model, named the proposed Residual Fused‐Graph Convolution Network (RF‐GCN). Here, multimodal data included is video as well as an Electroencephalogram (EEG) signal. Also, the Non‐Local Means (NLM) filter is used for pre‐processing input video frames. Here, the feature selection process is carried out using chi‐square, after feature extraction, which is done in both pre‐processed video frames and input EEG signals. Finally, facial emotion recognition and its types are determined by RF‐GCN, which is a combination of both the Deep Residual Network (DRN) and Graph Convolutional Network (GCN).ResultsFurther, RF‐GCN is evaluated for performance by metrics such as accuracy, recall, and precision, with superior values of 91.6%, 96.5%, and 94.7%.ConclusionsRF‐GCN captures the nuanced relationships between different emotional states and improves recognition accuracy. The model is trained and evaluated on the dataset and reflects real‐world conditions.

In recent years, research on increasing the spatial resolution and enhancing the quality of satellite images using the deep learning-based super-resolution (SR) method has been actively conducted. In a remote sensing field, conventional SR methods required high-quality satellite images as the ground truth. However, in most cases, high-quality satellite images are difficult to acquire because many image distortions occur owing to various imaging conditions. To address this problem, we propose an adaptive image quality modification method to improve SR image quality for the KOrea Multi-Purpose Satellite-3 (KOMPSAT-3). The KOMPSAT-3 is a high performance optical satellite, which provides 0.7-m ground sampling distance (GSD) panchromatic and 2.8-m GSD multi-spectral images for various applications. We proposed an SR method with a scale factor of 2 for the panchromatic and pan-sharpened images of KOMPSAT-3. The proposed SR method presents a degradation model that generates a low-quality image for training, and a method for improving the quality of the raw satellite image. The proposed degradation model for low-resolution input image generation is based on Gaussian noise and blur kernel. In addition, top-hat and bottom-hat transformation is applied to the original satellite image to generate an enhanced satellite image with improved edge sharpness or image clarity. Using this enhanced satellite image as the ground truth, an SR network is then trained. The performance of the proposed method was evaluated by comparing it with other SR methods in multiple ways, such as edge extraction, visual inspection, qualitative analysis, and the performance of object detection. Experimental results show that the proposed SR method achieves improved reconstruction results and perceptual quality compared to conventional SR methods.

Single image super resolution (SR) aims to estimate high resolution (HR) image from the low resolution (LR) one, and estimating accuracy of HR image gradient is very important for edge directed image SR methods. In this paper, we propose a novel edge directed image SR method by learning based gradient estimation. In proposed method, the gradient of HR image is estimated by using the example based ridge regression model. Recognizing that the training samples of the given sub-set for regression should have similar local geometric structure based on clustering, we employ high frequency of LR image patches with removing the mean value to perform such clustering. Moreover, the precomputed projective matrix of the ridge regression can reduce the computational complexity further. Experimental results suggest that the proposed method can achieve better gradient estimation of HR image and competitive SR quality compared with other SR methods.

Video super-resolution (SR) aims at restoring fine details and enhancing visual experience for low-resolution (LR) videos. In this paper, we propose a very deep non-simultaneous fully recurrent convolutional network for video SR. To make full use of temporal information, we employ motion compensation, very deep fully recurrent convolutional layers and late fusion in our system. Residual connection is also employed in our recurrent structure for more accurate SR. Finally a new model ensemble strategy is used to combine our method with single-image SR method. Experimental results demonstrate that the proposed method is better than state-of-the-art SR methods on quantitative visual quality assessment.

Nowadays Super Resolution (SR) is a trending term in many fields of image processing area where resolution describes the degree of details an image holds like, pixel count, sharpness and clarity. In many applications captured images are of low quality known as low resolution (LR) image, these images hold very less details. Therefore there is a need to convert such LR images to High resolution (HR) images which can be obtained by various SR methods like, interpolation methods, frequency domain based methods, reconstruction based methods and learning based methods. HR images obtained from interpolation methods contain reconstruction artifacts like edge blurs, edge halos and ringing effects, whereas learning based methods produce good results but predict unknown HR pixel values based only on input LR/HR image pairs. Thus, to overcome above mentioned disadvantages we proposed modified interpolation method for obtaining HR image, which predicts unknown HR pixel values from interpolated LR patch and corresponding HR patch using multilayer perceptron (MLP) and discrete wavelet transform (DWT). Experimental results subjectively and objectively show that for considered test images corresponding HR images obtained by our proposed method are of better quality than classical bicubic, Chopade et al. and Man et al. method.

Multimodal magnetic resonance images (e.g., T1-weighted image (TIWI) and T2-weighted image (T2WI)) are used for accurate medical imaging analysis. Different modal images have different resolution depending on pulse sequence parameters under limited data acquisition time. Therefore, interpolation methods are used to match the low-resolution (LR) image with the high-resolution (HR) image. However, the interpolation causes blurring that affects analysis accuracy. Although some recent works such as non-local-means (NLM) filter have manifested impressive super-resolution (SR) performance with available HR modal images, the filter has high computational cost. Therefore, we propose a fast SR framework with iterative-guided back projection, which incorporates iterative back projection with a guided filter (GF) method for resolution enhancement of LR images (e.g., T2WI) by referring HR images in another modality image (e.g., T1WI). The proposed method not only achieves both high accuracy than conventional interpolation methods and original GF and computational efficiency by applying an integral 3D image technique. In addition, although the proposed method is slightly inferior in accuracy visually than the state-of-the-art NLM filter, it can run 22 times faster than the state-of-theart method in expanding three times in the slice-select direction from 180 × 216 × 60 voxels to 180 × 216 × 180 voxels. The computational time of our method is about 1 min only. Therefore, the proposed method will be applied to various applications in practice, including not only multimodal MR images but also multimodal image analysis such as computed tomography (CT) and positron emission tomography (PET).

Arbitrary scale super-resolution diffusion model for brain MRI images