End-To-End Trainable Video Super-Resolution Based on a New Mechanism for Implicit Motion Estimation and Compensation

Recent studies on video super-resolution (SR) has shown that convolutional neural network (CNN) combined with motion compensation (MC) is able to merge information from multiple low-resolution (LR) frames to generate high quality images. However, Most SR and MC modules based on deep CNN simply increase the depth of the network, which cannot make full use of hierarchical and multi-scale features, thereby achieving relatively low performance. To address the above problem, a novel multi-scale residual dense Network (MSRDN) is proposed in this paper. A multi-scale residual density block (MSRDB) is first designed to extract abundant local features through dense convolution layer, which helps to adaptively detect image features of different scales with convolution kernels of different scales. Then, we redesign SR module and MC module with MSRDB, which adaptively learns more effective features from local features and uses global feature fusion to jointly and adaptively learn global hierarchical features. Comparative results on Vid4 dataset demonstrate that MSRDB can make more full use of feature information, which helps to effectively improve the reconstruction performance of video SR.

Recently, Video super-resolution (VSR) has become more crucial as the resolution of display has been grown. The majority of deep learning-based VSR methods combine the convolutional neural networks (CNN) with motion compensation or alignment module to estimate a high-resolution (HR) frame from low-resolution (LR) frames. However, most of the previous methods deal with the spatial features equally and may result in the misaligned temporal features by the pixel-based motion compensation and alignment module. It can lead to the damaging effect on the accuracy of the estimated HR feature. In this paper, we propose a wavelet attention embedding network (WAEN), including wavelet embedding network (WENet) and attention embedding network (AENet), to fully exploit the spatio-temporal informative features. The WENet is operated as a spatial feature extractor of individual low and high-frequency information based on 2-D Haar discrete wavelet transform. The meaningful temporal feature is extracted in the AENet through utilizing the weighted attention map between frames. Experimental results verify that the proposed method achieves superior performance compared with state-of-the-art methods.

Cardiac x-ray CT imaging is still challenging due to heart motion, which cannot be ignored even with the current rotation speed of the equipment. In response, many algorithms have been developed to compensate remaining motion artifacts by estimating the motion using projection data or reconstructed images. In these algorithms, accurate motion estimation is critical to the compensated image quality. In addition, since the scan range is directly related to the radiation dose, it is preferable to minimize the scan range in motion estimation. In this paper, the authors propose a novel motion estimation and compensation algorithm using a sinogram with a rotation angle of less than 360°. The algorithm estimates the motion of the whole heart area using two opposite 3D partial angle reconstructed (PAR) images and compensates the motion in the reconstruction process. A CT system scans the thoracic area including the heart over an angular range of 180° + α + β, where α and β denote the detector fan angle and an additional partial angle, respectively. The obtained cone-beam projection data are converted into cone-parallel geometry via row-wise fan-to-parallel rebinning. Two conjugate 3D PAR images, whose center projection angles are separated by 180°, are then reconstructed with an angular range of β, which is considerably smaller than a short scan range of 180° + α. Although these images include limited view angle artifacts that disturb accurate motion estimation, they have considerably better temporal resolution than a short scan image. Hence, after preprocessing these artifacts, the authors estimate a motion model during a half rotation for a whole field of view via nonrigid registration between the images. Finally, motion-compensated image reconstruction is performed at a target phase by incorporating the estimated motion model. The target phase is selected as that corresponding to a view angle that is orthogonal to the center view angles of two conjugate PAR images. To evaluate the proposed algorithm, digital XCAT and physical dynamic cardiac phantom datasets are used. The XCAT phantom datasets were generated with heart rates of 70 and 100 bpm, respectively, by assuming a system rotation time of 300 ms. A physical dynamic cardiac phantom was scanned using a slowly rotating XCT system so that the effective heart rate will be 70 bpm for a system rotation speed of 300 ms. In the XCAT phantom experiment, motion-compensated 3D images obtained from the proposed algorithm show coronary arteries with fewer motion artifacts for all phases. Moreover, object boundaries contaminated by motion are well restored. Even though object positions and boundary shapes are still somewhat different from the ground truth in some cases, the authors see that visibilities of coronary arteries are improved noticeably and motion artifacts are reduced considerably. The physical phantom study also shows that the visual quality of motion-compensated images is greatly improved. The authors propose a novel PAR image-based cardiac motion estimation and compensation algorithm. The algorithm requires an angular scan range of less than 360°. The excellent performance of the proposed algorithm is illustrated by using digital XCAT and physical dynamic cardiac phantom datasets.

Video super-resolution aims to increase the resolution of videos by exploiting the intra-frame and inter-frame dependencies of the low-resolution video sequences. There are usually two dependent steps in the video super-resolution: the motion compensation and the super-resolution reconstruction. In this paper, we propose a new deep learning framework without the explicit motion estimation by utilizing the self-attention model to exploit the full receptive field of the input video frames. In other words, the proposed deep neural network extracts the local features at all spatial–temporal locations for combining into global features using the self-attention networks in order to reconstruct the high-resolution video frame. The proposed global-aware network outperforms the state-of-the-art deep learning-based image and video super-resolution algorithms in terms of subjective and objective quality with less computational operations, as verified by extensive experiments on public image and video datasets, including Set5, Set14, B100, Urban100, and Vid4.

Aiming at the higher bit-rate occupation of motion vector encoding and more time load of full-searching strategies, a multi-resolution motion estimation and compensation algorithm based on adjacent prediction of frame difference was proposed. Differential motion detection was employed to image sequences and proper threshold was adopted to identify the connected region. Then the motion region was extracted to carry out motion estimation and motion compensation on it. The experiment results show that the encoding efficiency of motion vector is promoted, the complexity of motion estimation is reduced and the quality of the reconstruction image at the same bit-rate as Multi-Resolution Motion Estimation (MRME) is improved.

In the video application, slow motion is visually attractive and gets more attention in video super resolution. To generate the high-resolution (HR) slow motion video frames from the low-resolution (LR) frames, two sub-tasks are required, including video super-resolution (VSR) and video frame interpolation (VFI). However, the interpolation approach is not successful to extract low level feature attention to get the maximum advantage from the property of space-time relation. To this extent, we propose a deep consecutive attention network-based method. The multi-head attention and an attentive temporal feature module are designed to achieve better prediction of interpolation feature frame. Bi-directional deformable ConvLSTM module aggregates and aligns with the information from the multi-head attention and temporal feature block to improve the quality of video frames. This method synthesizes the HR video frames from LR video frames. The experimental results in terms of PSNR show the proposed method of deep consecutive attention outperforms 0.27 dB and 0.31 dB for Vid4 and SPMC datasets respectively, in average of PSNR compared to state-of-the-art baseline method.

Motion estimation and motion compensation (MEMC) has been applied inside the wavelet-transformed domain for video compression at low bit rates. However, pixel-based motion estimation in the wavelet domain is not as straightforward as in the spatial domain due to the shift variance property caused by decimation. The paper finds a new wavelet-based motion estimation and compensation method, named as the sub-decimation decomposition method, to perform shift invariant motion estimation inside the wavelet transformed domain, so that better matching blocks are compensated to achieve more efficient video compression.

Video super-resolution via mixed spatial-temporal convolution and selective fusion

This paper introduces a novel scheme of motion estimation (ME) and motion compensation (MC) in wavelet transform (WT) video coding. The scheme is unique in that half-pixel resolution ME and MC are applied on filtered coefficients ahead of decimation to preserve motion consistency and yet achieve minimal prediction residue. An efficient motion-compensated WT video coding system is built based on the proposed ME and MC. In the coding scheme, according to the property of embedded zerotree wavelet (EZW) coding, the mean square error (MSE) criterion is used in ME instead of normally adopted sum of absolute difference, and an overlapped MC algorithm is developed to combat the block effect introduced by block-based ME and improve peak signal-to-noise ratio (PSNR) performance. The superior performance of the proposed scheme is demonstrated with experimental results.

Convolutional neural networks (CNNs)-based video quality enhancement generally employs optical flow for pixelwise motion estimation and compensation, followed by utilizing motion-compensated frames and jointly exploring the spatiotemporal correlation across frames to facilitate the enhancement. This method, called the optical-flow-based method (OPT), usually achieves high accuracy at the expense of high computational complexity. In this article, we develop a new framework, referred to as biprediction-based multiframe video enhancement (PMVE), to achieve a one-pass enhancement procedure. PMVE designs two networks, that is, the prediction network (Pred-net) and the frame-fusion network (FF-net), to implement the two steps of synthesization and fusion, respectively. Specifically, the Pred-net leverages frame pairs to synthesize the so-called virtual frames (VFs) for those low-quality frames (LFs) through biprediction. Afterward, the slowly fused FF-net takes the VFs as the input to extract the correlation across the VFs and the related LFs, to obtain an enhanced version of those LFs. Such a framework allows PMVE to leverage the cross-correlation between successive frames for enhancement, hence capable of achieving high accuracy performance. Meanwhile, PMVE effectively avoids the explicit operations of motion estimation and compensation, hence greatly reducing the complexity compared to OPT. The experimental results demonstrate that the peak signal-to-noise ratio (PSNR) performance of PMVE is fully on par with that of OPT while its computational complexity is only 1% of OPT. Compared with other state-of-the-art methods in the literature, PMVE is also confirmed to achieve superior performance in both objective quality and visual quality at a reasonable complexity level. For instance, PMVE can surpass its best counterpart method by up to 0.42 dB in PSNR.

In recent years, video super resolution techniques becomes mandatory requirements to get high resolution videos. Many super resolution techniques researched but still video super resolution or scaling is a vital challenge. In this paper, we have presented a real-time video scaling based on convolution neural network architecture to eliminate the blurriness in the images and video frames and to provide better reconstruction quality while scaling of large datasets from lower resolution frames to high resolution frames. We compare our outcomes with multiple exiting algorithms. Our extensive results of proposed technique RemCNN (Reconstruction error minimization Convolution Neural Network) shows that our model outperforms the existing technologies such as bicubic, bilinear, MCResNet and provide better reconstructed motioning images and video frames. The experimental results shows that our average PSNR result is 47.80474 considering upscale-2, 41.70209 for upscale-3 and 36.24503 for upscale-4 for Myanmar dataset which is very high in contrast to other existing techniques. This results proves our proposed model real-time video scaling based on convolution neural network architecture’s high efficiency and better performance.

A technique is presented that incorporates an irregular triangle mesh into wavelet-domain motion-estimation and motion-compensation using a shift-invariant redundant wavelet transform. Triangle vertices are identified by a simple correlation operator locating image edges in the wavelet subbands, while motion compensation takes place through an affine transformation mapping triangles from one frame to the next. The motion-compensated residual is downsampled to a non-redundant form which is then coded using any wavelet-based still-image coder. Experimental results indicate that the combined approach outperforms either technique applied separately,, in addition, the proposed method outperforms a variety of motion-estimation and motion-compensation approaches operating in both the spatial and wavelet domains.

This chapter shows how neural fuzzy systems can improve motion estimation and compensation for video compression. Motion estimation and compensation are key parts of video compression. They help remove temporal redundancies in images. Most motion estimation algorithms neglect the strong-temporal correlations within the motion field. The search windows stay the same through the image sequences and the estimation needs heavy computation. We use an unsupervised neural vector quantizer system can use the temporal correlation of the motion field to estimate the motion vectors. First-order and second-order statistics of the motion vectors give ellipsoidal search windows. This algorithm reduces the search area and entropy and gives clustered motion fields. Motion-compensated video coding further assumes that each block of pixels moves with uniform translational motion. This often does not hold and can produce block artifacts. We use a supervised neural fuzzy system to compensate for the overlapped block motion. This fuzzy system uses the motion vectors of neighboring blocks to map the prior frame’s pixel values to the current pixel value. The neural fuzzy system used 196 rules that came from the prior decoded frame. The fuzzy system learns and updates its rules as it decodes the image. The fuzzy system also improved the compensation accuracy. The appendix derives both the fuzzy system and the supervised neural-like learning laws that tune its parameters.

In this paper we propose a new method of using motion estimation for the purpose of offline recognition of machine-print Hindi digits. The recognition process can be summarized as follows: an image of each of the ten numerals (numbers 0 to 9) is stored, these are called reference images. The differences between the image of the numeral to be recognized and the reference image are considered motions. The motions are estimated and then compensated. The resultant image after motion compensation is compared to the reference image and the difference between the two (treated as error) is calculated. The process of motion estimation and compensation is done for each of the ten numerals. The numeral with the minimum error is chosen as the recognized digit. While this proposed recognition system is applied to Hindi digits in this paper, it can be generalized to the recognition of any numerals in any language and can be extended to text and voice recognition. The proposed method is simple, fast, accurate, and reliable.

The present paper introduces an original way to perform motion estimation and compensation. The scheme is asymmetric in the sense that the compensation stage is not merely the reverse operation of the estimation process. Motion estimation is computed using a classical block matching algorithm and the set of vectors that compose the motion information are classically transmitted to the decoder. The motion compensation uses them to deform a wireframe structure whose vertices are displaced according to the motion vectors. Conditions must be set up on the number of wireframe vertices and on their location. In comparison with a block-based compensation, the main advantage is the possibility to take into account the object information, which results in perceptually more pleasant predicted pictures.