Constrained motion compensation for very low bit rate video compression

We present a semi-fixed-length motion vector coding method for H.263-based low bit rate video compression. The method exploits structural constraints within the motion field. The motion vectors are encoded using semi-fixed-length codes, yielding essentially the same levels of rate-distortion performance and subjective quality achieved by H.263's Huffman-based variable length codes in a noiseless environment. However, such codes provide substantially higher error resilience in a noisy environment.

This paper presents a fast image sequence stabilization system that uses one bit transform (IBT) matching based global motion estimation and Kalman filtering based global motion correction. In order to facilitate a fast motion estimation process, image frames are initially converted into a single bit-plane representation using the IBT, and then motion vectors are evaluated matching the IBTs of two consecutive frames using Boolean operations. Because a binary representation is established, the computational complexity of motion estimation is significantly reduced providing a proportionate reduction in power consumption of custom hardware. The global motion vector is computed using local motion vectors of sub-image blocks to further increase the motion estimation speed. For the motion correction part, global motion vectors are accumulated to yield absolute frame displacements and Kalman filtering is employed to achieve stabilization.

New solutions to Very Low Bit Rate speech coding have been recently proposed based on speech recognition and speech synthesis technologies, [1,2,3,4,5,7,8]. In the continuation of the work described in [8], this paper presents a complete encoding scheme around 500 bits/sec. The proposed solution is based on automatic recognition of elementary acoustical units using HMM modelling. An unsupervised training phase is used to build the HMM models and the codebook of synthesis units. The decoded speech is then obtained by concatenating the corresponding synthesis units based on a HNM-like decomposition of speech. A new unit selection process is proposed integrating some prosody constraints. Through this approach, the size of the synthesis codebook is independent of the targeted bit rate. A complete description of the unit selection process and of the associated prosody modelling is given, together with the quantisation scheme of the overall set of encoded parameters.

The aim of our effort is to reach higher quality of resulting speech coded by very low bit rate (VLBR) segmental coder. In already existing VLBR coder, we want to improve the determination of acoustical units. Furthermore, better analysis-synthesis technique for the synthesis part (Harmonic-Noise Model) instead of LPCC is going to be used. The VLBR coder consists of a recognition system followed by a speech synthesizer. The recognizer identifies recognition acoustic units (RU). On the other hand, the synthesizer concatenates synthesis acoustic units (SU). However, the two kinds of acoustic unit can be identical or different and then can be modeled in different ways such Hidden Markov Model for the RU and Harmonic-Noise model for the SU. Both kinds of units are obtained automatically from a training database of raw speech that does not contain any transcription. In the original version of the coder, the quality of the synthetic speech was not sufficient for these two main reasons: the SU units were too short and difficult to concatenate and the synthesis was done using basic LPCC analysis-synthesis. In order to remove first drawback, three methods of re-segmentation were used. Afterwards, the basic LPCC analysis-synthesis was replaced by HNM.KeywordsHide Markov ModelSpeech SignalVector QuantizationDynamic Time WarpingSegment BoundaryThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A multiple global affine motion model is proposed for low bit rate video compression. Block-wise motion segmentation is first performed with the number of motion objects L predefined. The affine motion models for multiple MOs are estimated and coded in the frame header. The scaling parameters a1, a2, a4 and a5 are coded with a 4-dimensional vector-quantizer (VQ), whose 16 most recently used code words are maintained on line and searched for VQ match, and the 300-word long main code book stored offline. The translational parameters a3 and a6 are coded predicatively as a classical motion vector. L new macro-block modes are added to the standard’s list of 7 intra and inter modes. No segmentation information is transmitted, for the mode already indicates that if one of the affine modes is selected by Lagrange rate-distortion optimization. A metric S is defined to measure locality of the motion and will disable use of affine models when a threshold is surpassed. Simulation shows that more than 50% of the MB’s choose one of the affine modes. When 100kbps or lower bandwidths are available, the proposed codec not only saves 1~18% bit rate, but also enhances error-resilience in multiple slice frames and reduces blocking artifacts notably.

In this paper, we propose a fast block motion estimation scheme by exploiting the correlations of motion vectors (MV) existing in spatially- and temporally-adjacent as well as hierarchically-related blocks. The basic idea is to use the information obtained from the corresponding block at a coarser resolution level and spatio-temporal neighboring blocks at the same level in order to select a good set of initial MV candidates and then perform further local search to find the best matching MV. In order to further reduce computational complexity, the sign pattern vector (SPV) is defined and used for block matching, as opposed to the pixel intensity values used in the conventional block matching methods. By using the SPV definition, a data block can be presented by a mean and a set of binary sign patterns with a much reduced data set. The block matching motion estimation is then divided into mean matching and binary phase matching. The proposed technique enables a significant reduction in computational complexity compared with the conventional block matching motion estimation (ME) because binary phase matching only involves Boolean logic operations. This scheme also significantly reduces the data transfer time between the frame buffer and motion estimator. Our test results indicate that the performance of the proposed scheme is comparable with the full-search block matching under the same search ranges, however, the proposed scheme has a speed-up factor ranging from 250 to 370 in comparison with full search.

This dissertation reports on rate-distortion optimized and computation-constrained video coding algorithms which yield effective and efficient block-based video encoders. We first propose a new cost function for motion estimation which can achieve good tradeoffs among motion vector bit rate, compensation distortion, and number of computations. Based on the new cost function, we propose a predictive motion estimation algorithm which can effectively determine the size of the search region according to the statistics of the motion field. The proposed motion estimation algorithm allows control of the motion vector bit rate and the computational cost, simultaneously. We also develop two efficient rate-distortion optimized coding mode selection algorithms, based on thresholding and finite state machine (FSM) methods, respectively. The thresholding method can eliminate consideration of the computationally expensive modes. The FSM method determines the test orders of the modes and associated parameters by exploiting the local statistics of the input video sequence. The proposed two methods can reduce substantially the computation requirements as compared to the full search mode selection method. Based on the proposed motion estimation and mode selection algorithms, we implement an H.263-based video coder and an MPEG-2, compliant video coder for very low bit rate video applications, and high bit rate interlaced video applications, respectively. The resulting video coders achieve excellent tradeoffs among bit rate, quality, and computational complexity. In fact, experimental results show that our video coders outperform the best known video coders in terms of compression performance and encoding speed.

Publication in the conference proceedings of EUSIPCO, Antalya, Turkey, 2005

Matching pursuits is a dictionary-based coding method, which has recently been demonstrated to outperform standard techniques, such as H.263, at low bit rate video compression. Unfortunately, the method involves an extremely high computational cost, compared to the DCT-based H.263. The contribution of this paper relies on reporting three new dictionaries for matching pursuits video coding, whose advantages include both a reduced computational cost and improved PSNR performance, compared to dictionaries reported to date.

A new rate control scheme is used to calculate the target bit rate for each frame based on a quadratic formulation of the rate distortion function. The distortion measure are assumed to be either the average quantization scale of a frame or mean square error. The rate distortion function is modeled as a second order function of the inverse of the distortion measure. We presented a closed form solution for the target bit allocation which applies to both the digital storage media (DSM) applications (e.g., MPEG-2) and very low bit rate (VLBR) applications (e.g., MPEG-4). We perform experiments on a wavelet-based coder to demonstrate the versatility of our scheme. Model parameters are estimated using statistical linear regression analysis. Since the estimation uses the past encoded frames of the same picture prediction type (I, P and B pictures), the proposed approach is a single pass rate control technique. For the MPEG-2 coder the fluctuations of the bit counts are significantly reduced by 20 percent to 65 percent in the standard deviation of the bit count while the picture quality remains the same. For the MPEG-4 VM coder, the PSNR is improved by almost one dB as compared to the best MPEG-4 VM rate control using exhaustive combinations of quantization scale parameters.

As Moore's law approaches the low nanometer range, predictions have been made that computing via future technology nodes may no longer be correct due to, for example, the presence of noise sources such as thermal noise. Probabilistic computing, a term coined for computing via devices that yield a probabilistically correct output, has earlier been shown as a means for realizing energy efficient computing in signal processing applications. In this paper, we explore the application of probabilistic computing for low power motion estimation. It is observed that motion estimation itself exhibits high resilience to erroneous computation. Simulations show that energy savings up to 44% can be achieved in motion estimation using full search block matching with probabilistic computing with a minor impact (under 0.5 dB) on the required quality of the output. A scheme of error correction based on the motion vector distribution is proposed that further increases the possible energy savings with full search block matching up to 57%.

Block-matching algorithm is the most common technique applied in block-based motion estimation technique. There are several block-matching algorithm based on block-based motion estimation techniques have been developed. Full search (FS), three step search (TSS), new three step search (NTSS), diamond search (DS) and hexagon based search (HS) are the most well known block-matching algorithm. These techniques are applied to video sequences to remove the temporal redundancy for compression purposes and to gauge the motion vector estimation. In addition, the mentioned block-matching algorithms are the baseline techniques that have been used to further develop all the enhanced or improved algorithms. In order to develop the proposed methods, the baseline techniques are studied to develop the proposed algorithms. This chapter proposes modelling of fast unrestricted hexagon search (FUHS16) and unrestricted hexagon-diamond search (UHDS16) algorithms for motion vector estimation, which is based on the theory and application of block-based motion estimation. Both of these algorithms are designed using 16 × 16 block size. In particular, the motion vector estimation, quality performance, computational complexity, and elapsed processing time are emphasised. These parameters have been used to measure the experimental results. It is the aim of this study that this work provides a common framework with which to evaluate and understand block-based matching motion estimation performance. On the theoretical side, four fundamental issues are explored: (1) division of frame, (2) basic block-based matching, (3) motion vector estimation, and (4) block-matching algorithm development. Various existing block-matching motion estimation algorithms have been analysed to develop the fundamental research. Based on the theoretical and fundamental research analysis the FUHS16 and UHDS16 algorithms using 16 × 16 block-based motion estimation formulations were developed. To improve the UHDS16 algorithm, 8 × 8 block-matching technique has been tested. The 8 × 8 block-matching technique is known as UHDS8. The results show positive improvements. From an application perspective, the UHDS8 algorithm efficiently captured the motion vectors in many video sequences. For example, in video compression, the use of motion vectors on individual macro-blocks optimized the motion vector information. The UHDS8 algorithm also offers improvement in terms of image quality performance, computational complexity and elapsed processing time. Thus, this chapter offers contributions in certain areas such as reducing the mechanism of computational complexity in estimating the motion from the video sequences. In particular, the FUHS16, UHDS16 and UHDS8 algorithms were developed to estimate the motion vectors field in the video sequences. Theoretical analysis block-based matching criteria are adapted to FUHS16, UHDS16 and UHDS8 algorithms, which are based on search points technique. Basically, the proposed of FUHS16, UHDS16 and UHDS8 algorithm produces the best motion vector estimation finding based on the block-based matching criteria. Besides that, the UHDS8 algorithm also improves the image quality performances and the search points in terms of the computational complexity. Overall, the study shows that the UHDS8 algorithm produces better results compared to the FUHS16 and UHDS16 algorithm.

This chapter shows how neural fuzzy systems can improve motion estimation and compensation for video compression. Motion estimation and compensation are key parts of video compression. They help remove temporal redundancies in images. Most motion estimation algorithms neglect the strong-temporal correlations within the motion field. The search windows stay the same through the image sequences and the estimation needs heavy computation. We use an unsupervised neural vector quantizer system can use the temporal correlation of the motion field to estimate the motion vectors. First-order and second-order statistics of the motion vectors give ellipsoidal search windows. This algorithm reduces the search area and entropy and gives clustered motion fields. Motion-compensated video coding further assumes that each block of pixels moves with uniform translational motion. This often does not hold and can produce block artifacts. We use a supervised neural fuzzy system to compensate for the overlapped block motion. This fuzzy system uses the motion vectors of neighboring blocks to map the prior frame’s pixel values to the current pixel value. The neural fuzzy system used 196 rules that came from the prior decoded frame. The fuzzy system learns and updates its rules as it decodes the image. The fuzzy system also improved the compensation accuracy. The appendix derives both the fuzzy system and the supervised neural-like learning laws that tune its parameters.

Motion estimation is the vital step in video compression and several algorithms have been proposed for its acceleration. In this paper we present an innovative two pass motion estimation technique with reduced search to minimize the computational load. We adopt the block based motion estimator, which first finds the matching block using the trace calculation and only on match, it calculates the mean absolute difference (MAD) to ensure best motion prediction. In addition, since the motion vectors themselves are temporal, we reduce the search range by approximating the next motion vector from the previous predictions. This method remarkably reduces the computational load and provides better results comparable with more expensive full search MAD techniques.

Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.