Mesh-based motion estimation and compensation in the wavelet domain using a redundant transform

Aiming at the higher bit-rate occupation of motion vector encoding and more time load of full-searching strategies, a multi-resolution motion estimation and compensation algorithm based on adjacent prediction of frame difference was proposed. Differential motion detection was employed to image sequences and proper threshold was adopted to identify the connected region. Then the motion region was extracted to carry out motion estimation and motion compensation on it. The experiment results show that the encoding efficiency of motion vector is promoted, the complexity of motion estimation is reduced and the quality of the reconstruction image at the same bit-rate as Multi-Resolution Motion Estimation (MRME) is improved.

Motion estimation and motion compensation (MEMC) has been applied inside the wavelet-transformed domain for video compression at low bit rates. However, pixel-based motion estimation in the wavelet domain is not as straightforward as in the spatial domain due to the shift variance property caused by decimation. The paper finds a new wavelet-based motion estimation and compensation method, named as the sub-decimation decomposition method, to perform shift invariant motion estimation inside the wavelet transformed domain, so that better matching blocks are compensated to achieve more efficient video compression.

Cardiac x-ray CT imaging is still challenging due to heart motion, which cannot be ignored even with the current rotation speed of the equipment. In response, many algorithms have been developed to compensate remaining motion artifacts by estimating the motion using projection data or reconstructed images. In these algorithms, accurate motion estimation is critical to the compensated image quality. In addition, since the scan range is directly related to the radiation dose, it is preferable to minimize the scan range in motion estimation. In this paper, the authors propose a novel motion estimation and compensation algorithm using a sinogram with a rotation angle of less than 360°. The algorithm estimates the motion of the whole heart area using two opposite 3D partial angle reconstructed (PAR) images and compensates the motion in the reconstruction process. A CT system scans the thoracic area including the heart over an angular range of 180° + α + β, where α and β denote the detector fan angle and an additional partial angle, respectively. The obtained cone-beam projection data are converted into cone-parallel geometry via row-wise fan-to-parallel rebinning. Two conjugate 3D PAR images, whose center projection angles are separated by 180°, are then reconstructed with an angular range of β, which is considerably smaller than a short scan range of 180° + α. Although these images include limited view angle artifacts that disturb accurate motion estimation, they have considerably better temporal resolution than a short scan image. Hence, after preprocessing these artifacts, the authors estimate a motion model during a half rotation for a whole field of view via nonrigid registration between the images. Finally, motion-compensated image reconstruction is performed at a target phase by incorporating the estimated motion model. The target phase is selected as that corresponding to a view angle that is orthogonal to the center view angles of two conjugate PAR images. To evaluate the proposed algorithm, digital XCAT and physical dynamic cardiac phantom datasets are used. The XCAT phantom datasets were generated with heart rates of 70 and 100 bpm, respectively, by assuming a system rotation time of 300 ms. A physical dynamic cardiac phantom was scanned using a slowly rotating XCT system so that the effective heart rate will be 70 bpm for a system rotation speed of 300 ms. In the XCAT phantom experiment, motion-compensated 3D images obtained from the proposed algorithm show coronary arteries with fewer motion artifacts for all phases. Moreover, object boundaries contaminated by motion are well restored. Even though object positions and boundary shapes are still somewhat different from the ground truth in some cases, the authors see that visibilities of coronary arteries are improved noticeably and motion artifacts are reduced considerably. The physical phantom study also shows that the visual quality of motion-compensated images is greatly improved. The authors propose a novel PAR image-based cardiac motion estimation and compensation algorithm. The algorithm requires an angular scan range of less than 360°. The excellent performance of the proposed algorithm is illustrated by using digital XCAT and physical dynamic cardiac phantom datasets.

This paper introduces a novel scheme of motion estimation (ME) and motion compensation (MC) in wavelet transform (WT) video coding. The scheme is unique in that half-pixel resolution ME and MC are applied on filtered coefficients ahead of decimation to preserve motion consistency and yet achieve minimal prediction residue. An efficient motion-compensated WT video coding system is built based on the proposed ME and MC. In the coding scheme, according to the property of embedded zerotree wavelet (EZW) coding, the mean square error (MSE) criterion is used in ME instead of normally adopted sum of absolute difference, and an overlapped MC algorithm is developed to combat the block effect introduced by block-based ME and improve peak signal-to-noise ratio (PSNR) performance. The superior performance of the proposed scheme is demonstrated with experimental results.

This chapter shows how neural fuzzy systems can improve motion estimation and compensation for video compression. Motion estimation and compensation are key parts of video compression. They help remove temporal redundancies in images. Most motion estimation algorithms neglect the strong-temporal correlations within the motion field. The search windows stay the same through the image sequences and the estimation needs heavy computation. We use an unsupervised neural vector quantizer system can use the temporal correlation of the motion field to estimate the motion vectors. First-order and second-order statistics of the motion vectors give ellipsoidal search windows. This algorithm reduces the search area and entropy and gives clustered motion fields. Motion-compensated video coding further assumes that each block of pixels moves with uniform translational motion. This often does not hold and can produce block artifacts. We use a supervised neural fuzzy system to compensate for the overlapped block motion. This fuzzy system uses the motion vectors of neighboring blocks to map the prior frame’s pixel values to the current pixel value. The neural fuzzy system used 196 rules that came from the prior decoded frame. The fuzzy system learns and updates its rules as it decodes the image. The fuzzy system also improved the compensation accuracy. The appendix derives both the fuzzy system and the supervised neural-like learning laws that tune its parameters.

Video super-resolution aims at generating a high-resolution video from its low-resolution counterpart. With the rapid rise of deep learning, many recently proposed video super-resolution methods use convolutional neural networks in conjunction with explicit motion compensation to capitalize on statistical dependencies within and across low-resolution frames. Two common issues of such methods are noteworthy. Firstly, the quality of the final reconstructed HR video is often very sensitive to the accuracy of motion estimation. Secondly, the warp grid needed for motion compensation, which is specified by the two flow maps delineating pixel displacements in horizontal and vertical directions, tends to introduce additional errors and jeopardize the temporal consistency across video frames. To address these issues, we propose a novel dynamic local filter network to perform implicit motion estimation and compensation by employing, via locally connected layers, sample-specific and position-specific dynamic local filters that are tailored to the target pixels. We also propose a global refinement network based on ResBlock and autoencoder structures to exploit non-local correlations and enhance the spatial consistency of super-resolved frames. The experimental results demonstrate that the proposed method outperforms the state-of-the-art, and validate its strength in terms of local transformation handling, temporal consistency as well as edge sharpness.

This paper presents an algorithm to predict interpolative motion compensation (MC) mode without computing interpolative SADs (Sum of Absolute Difference) for B-picture coding in an MPEG-2 video encoder. Firstly, an initial MC mode is selected among forward frame/field MC and backward frame/field MC for each macroblock (MB) in a Bpicture. This is accomplished by comparing four minimum SADs corresponding to the above-mentioned MC modes, and choosing a minimum SAD among them. Secondly, if the SAD corresponding to the selected mode is less than a specific threshold, we set the selected MC mode to the final MC mode. Otherwise, an MC mode of the current MB is determined between interpolative frame MC and interpolative field MC. If the sum of SADs of forward frame MC mode and backward frame MC mode is less than that of SADs of forward field MC mode and backward field MC mode, the final MC mode is set to an interpolative frame MC. Otherwise, the final MC mode is set to an interpolative field MC mode. Experimental results show that the proposed algorithm is comparable with as brute-force MC mode selection method as in conventional MPEG-2 video encoding. In addition, the proposed algorithm may noticeably reduce the memory bandwidth in implementation of MPEG-2 video encoder.

Motion estimation (ME) and motion compensation (MC) based deinterlacing has been become the mainstream technology of industry. The existing ME/MC algorithms are mostly based on block matching (BM) strategy, in which various BM algorithms are proposed with different tradeoff between the matching accuracy and the CPU time. Aiming at the dilemma between the performance and the efficiency of the available deinterlacing algorithms, two adaptive ME/MC algorithms are presented in this paper. Compared with the full search BM algorithm, the proposed algorithms greatly reduce the computational amount with keeping the performance approximately. However, the inherent drawbacks of the BM limit the farther improvement of the deinterlacing performance. For this purpose, a new deinterlacing algorithm based on motion object is developed, in which it is natural motion object rather than contrived block that is taken as the basic cell for ME/MC. In the new algorithm, a more reliable technique is adopted for accurate motion region detection, and immune clonal selection algorithm is employed to accelerate the matched object searching. Meanwhile, many advanced deinterlacing methods, such as motion compensation, median filtering, Weave and Bob, are integrated into the new algorithm, so it is more adaptive to the various video sequences. The experimental results with several available test databases illustrate that the new algorithm based on motion object is better than those based on motion block on the whole.

In this paper we propose a new method of using motion estimation for the purpose of offline recognition of machine-print Hindi digits. The recognition process can be summarized as follows: an image of each of the ten numerals (numbers 0 to 9) is stored, these are called reference images. The differences between the image of the numeral to be recognized and the reference image are considered motions. The motions are estimated and then compensated. The resultant image after motion compensation is compared to the reference image and the difference between the two (treated as error) is calculated. The process of motion estimation and compensation is done for each of the ten numerals. The numeral with the minimum error is chosen as the recognized digit. While this proposed recognition system is applied to Hindi digits in this paper, it can be generalized to the recognition of any numerals in any language and can be extended to text and voice recognition. The proposed method is simple, fast, accurate, and reliable.

The present paper introduces an original way to perform motion estimation and compensation. The scheme is asymmetric in the sense that the compensation stage is not merely the reverse operation of the estimation process. Motion estimation is computed using a classical block matching algorithm and the set of vectors that compose the motion information are classically transmitted to the decoder. The motion compensation uses them to deform a wireframe structure whose vertices are displaced according to the motion vectors. Conditions must be set up on the number of wireframe vertices and on their location. In comparison with a block-based compensation, the main advantage is the possibility to take into account the object information, which results in perceptually more pleasant predicted pictures.

Delogne et al. (see IEEE Transactions on Image Processing, vol.3, no.5, p.482-91, 1994) have shown how the motion estimation and compensation have to be carried out when interlaced images are processed under the assumption of a translational motion. Due to the finite accuracy of the motion vectors, the prediction error is nonzero after motion compensation. We therefore investigate the influence of quantizing the motion vectors on the statistical properties of the prediction error signal. The power spectral density and the variance of the prediction error are also computed.

An information-lossy method for contour image sequence coding is described in this paper. It is based on motion estimation, compensation and morphological filters. In order to exploit the high temporal redundancy of the contour image sequence, motion estimation and compensation are used for the correspondent label image sequence. The first frame is directly coded by the chain-code method. All the following frames are coded in motion compensation mode. Within each frame, labels are divided into foreground labels and background labels. Only the prediction error for the foreground labels are coded. Morphological filters are applied to clean the prediction error for the foreground labels. Afterwards, an efficient prediction error coding algorithm is introduced based on the modified chain-code method. Experimental results have demonstrated outstanding performance for the coding of a contour image sequence.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

This paper presents a low-complexity motion compensation (MC) aided motion adaptive (MA) de-interlacing method, aiming at effectively leveraging the fact human eyes are not sensitive to high spatial frequency with rapid motion. The basic idea is to enhance MA de-interlacing by applying MC de-interlacing to process slow-moving objects. Compared with conventional MC de-interlacing, the motion estimation only needs to search in a much smaller area and can adaptively switch between the same parity and opposite parity fields according to the motion vector. Simulation demonstrates that the proposed method only incurs less than 5% of computational complexity of general MC de-interlacing, and meanwhile can improve the PSNR by up to 10 dB compared with conventional MA methods. Its silicon cost and energy consumption are further evaluated by an ASIC design at 0.13 mum technology node.

This paper presents an efficient architecture for Motion Estimation (ME) based on the Diamond Search (DS) Algorithm. This architecture also includes the Motion Compensation (MC) for luminance samples, reusing internal ME results and avoiding the additional external memory accesses for the MC operation. The proposed architecture also generates inputs for a Fractional Motion Estimation (FME) for quarter sample precision, as proposed by the H.264/AVC standard. The main goal of this work is to design a low cost ME architecture for the DS algorithm with reduced number of external memory accesses and ready to be integrated with a fractional interpolator. The designed architecture was synthesized to Altera Stratix 4 family of FPGAs and in the worst case scenario, this architecture is able to process Full HD (1080p) videos in real time.

In this work, we proposed an improved frame rate up-conversion (FRUC) algorithm based on dual motion estimation (DME), which combines the unidirectional motion estimation and bi-directional motion estimation. Considering that edge of the image has a significant impact on the interpolated frame and in order to heighten the accuracy of motion estimation at edge area, edge information is added to the motion estimation process. Besides, the overlapped block motion estimation (OBME) method is applied to further improve the accuracy of motion vectors. Then an median filter is used to detect abnormal vector and correct it to obtain the final motion vector field. In the phase of motion compensation, this algorithm introduces block boundary distortion match to improve the lacking of traditional overlapped block motion compensation method (OBMC). The experimental results verify that the FRUC algorithm proposed in this paper outstands other algorithms in subjective and objective quality.