A new temporal error concealment method for H.264 using adaptive block sizes

The H.264 video coding standard adopts many new coding tools, such as variable block size, multiple reference frames, quarter-pixel-accuracy motion estimation, intra prediction, loop filter, etc. Using these coding tools, H.264 achieves significant performance. Especially for reference frame, block size, and predication direction, it can adaptively do the selection. However, the encoding complexity increases tremendously. Among these tools, the macro block modes selection and the motion estimation contributes most to total encoding complexity. This paper focuses on complexity reduction in macro block modes selection. On the basis of analyzing some fast mode selection algorithms, a new algorithm is presented, in order to reduce the time of encoding, in the precondition of reducing the quality of encoding obviously.

In this paper, a new error concealment algorithm for the new coding standard H.264 is presented. The algorithm consists of a block size determination step to determine the size type of the lost block and a motion vector recovery step to find the lost motion vector from multiple reference frames. The main feature of this algorithm are as follows. In the block size determination step, we propose a criterion to determine the size type of the lost block from the current frame. In the motion vector recovery step, the optimal motion vector for the lost block chosen from multiple previous reference frames with the minimum value of the side match distortion. The proposed algorithm not only can determine the most correct mode for the lost block, but also can save much more computation time for motion vector recovery. Experimental results show that the proposed algorithm achieves 0.4/spl sim/7 dB improvement over the conventional VM method.

H.264 adopts new coding tools such as intra-prediction, variable block size, motion estimation with quarter-pixel-accuracy, loop filter, etc. The adoption of these tools enables an H.264-coded bitstream to have more information compared with previous standards. In this paper we proposed an effective spatial error concealment method with low complexity. Among the information included in an H.264-coded bitstream, we use prediction modes of intra-blocks for recovering a damaged block. This is because a prediction direction in each prediction mode is highly correlated to the edge direction. We first estimate the edge direction of a damaged block using prediction modes of intra-blocks adjacent to a damaged block and classify the area inside a damaged block into the edge and the flat area. And then our method recovers pixel values in the edge area using edge-directed interpolation, and recovers pixel values in the flat area using weighted interpolation. Simulation results show the proposed method yields better video quality than conventional approaches.

In the new video coding standard H.264/AVC, motion estimation (ME) is allowed to search multiple reference frames. Therefore, the required computation is highly increased, and it is in proportion to the number of searched reference frames. However, the reduction in prediction residues is mostly dependent on the nature of sequences, not on the number of searched frames. Sometimes the prediction residues can be greatly reduced, but frequently a lot of computation is wasted without achieving any better coding performance. In this paper, we propose a context-based adaptive method to speed up the multiple reference frames ME. Statistical analysis is first applied to the available information for each macroblock (MB) after intra-prediction and inter-prediction from the previous frame. Context-based adaptive criteria are then derived to determine whether it is necessary to search more reference frames. The reference frame selection criteria are related to selected MB modes, inter-prediction residues, intra-prediction residues, motion vectors of subpartitioned blocks, and quantization parameters. Many available standard video sequences are tested as examples. The simulation results show that the proposed algorithm can maintain competitively the same video quality as exhaustive search of multiple reference frames. Meanwhile, 76 %-96 % of computation for searching unnecessary reference frames can be avoided. Moreover, our fast reference frame selection is orthogonal to conventional fast block matching algorithms, and they can be easily combined to achieve further efficient implementations.

Fast Interframe mode decision algorithm based on mode mapping and MB activity for MPEG-2 to H.264/AVC transcoding

Motion estimation using multiple reference frames is widely used as the basis for recent video coding standards (eg. H.264/AVC) to achieve increased coding efficiency. However, this increases the complexity of the encoding process. In this paper, a new technique for efficient motion estimation is proposed. A combination of multiple reference frame selection and image residue-based mode selection is used to improve motion estimation time. By dynamic selection of an initial reference frame in advance, the number of reference frames to be considered is reduced. In addition, from examination of the residue between the current block and reconstructed blocks in preceding frames, variable block size mode decisions are made. Modified initial motion vector estimation and early stop condition detection are also adopted to speed up the motion estimation procedure. Experimental results compare the performance of the proposed algorithm with a state of the art motion estimation algorithm and demonstrate significantly reduced motion estimation time while maintaining PSNR performance.

Distributed Video Coding (DVC) is a new paradigm for video compression based on the information theoretical results of Slepian and Wolf (SW), and Wyner and Ziv (WZ). While conventional coding has a rigid complexity allocation as most of the complex tasks are performed at the encoder side, DVC enables a flexible complexity allocation between the encoder and the decoder. The most novel and interesting case is low complexity encoding and complex decoding, which is the opposite of conventional coding. While the latter is suitable for applications where the cost of the decoder is more critical than the encoder's one, DVC opens the door for a new range of applications where low complexity encoding is required and the decoder's complexity is not critical. This is interesting with the deployment of small and battery-powered multimedia mobile devices all around in our daily life. Further, since DVC operates as a reversed-complexity scheme when compared to conventional coding, DVC also enables the interesting scenario of low complexity encoding and decoding between two ends by transcoding between DVC and conventional coding. More specifically, low complexity encoding is possible by DVC at one end. Then, the resulting stream is decoded and conventionally re-encoded to enable low complexity decoding at the other end. Multiview video is attractive for a wide range of applications such as free viewpoint television, which is a system that allows viewing the scene from a viewpoint chosen by the viewer. Moreover, multiview can be beneficial for monitoring purposes in video surveillance. The increased use of multiview video systems is mainly due to the improvements in video technology and the reduced cost of cameras. While a multiview conventional codec will try to exploit the correlation among the different cameras at the encoder side, DVC allows for separate encoding of correlated video sources. Therefore, DVC requires no communication between the cameras in a multiview scenario. This is an advantage since communication is time consuming (i.e. more delay) and requires complex networking. Another appealing feature of DVC is the fact that it is based on a statistical framework. Moreover, DVC behaves as a natural joint source-channel coding solution. This results in an improved error resilience performance when compared to conventional coding. Further, DVC-based scalable codecs do not require a deterministic knowledge of the lower layers. In other words, the enhancement layers are completely independent from the base layer codec. This is called the codec-independent scalability feature, which offers a high flexibility in the way the various layers are distributed in a network. This thesis addresses the following topics: First, the theoretical foundations of DVC as well as the practical DVC scheme used in this research are presented. The potential applications for DVC are also outlined. DVC-based schemes use conventional coding to compress parts of the data, while the rest is compressed in a distributed fashion. Thus, different conventional codecs are studied in this research as they are compared in terms of compression efficiency for a rich set of sequences. This includes fine tuning the compression parameters such that the best performance is achieved for each codec. Further, DVC tools for improved Side Information (SI) and Error Concealment (EC) are introduced for monoview DVC using a partially decoded frame. The improved SI results in a significant gain in reconstruction quality for video with high activity and motion. This is done by re-estimating the erroneous motion vectors using the partially decoded frame to improve the SI quality. The latter is then used to enhance the reconstruction of the finally decoded frame. Further, the introduced spatio-temporal EC improves the quality of decoded video in the case of erroneously received packets, outperforming both spatial and temporal EC. Moreover, it also outperforms error-concealed conventional coding in different modes. Then, multiview DVC is studied in terms of SI generation, which differentiates it from the monoview case. More specifically, different multiview prediction techniques for SI generation are described and compared in terms of prediction quality, complexity and compression efficiency. Further, a technique for iterative multiview SI is introduced, where the final SI is used in an enhanced reconstruction process. The iterative SI outperforms the other SI generation techniques, especially for high motion video content. Finally, fusion techniques of temporal and inter-view side informations are introduced as well, which improves the performance of multiview DVC over monoview coding. DVC is also used to enable scalability for image and video coding. Since DVC is based on a statistical framework, the base and enhancement layers are completely independent, which is an interesting property called codec-independent scalability. Moreover, the introduced DVC scalable schemes show a good robustness to errors as the quality of decoded video steadily decreases with error rate increase. On the other hand, conventional coding exhibits a cliff effect as the performance drops dramatically after a certain error rate value. Further, the issue of privacy protection is addressed for DVC by transform domain scrambling, which is used to alter regions of interest in video such that the scene is still understood and privacy is preserved as well. The proposed scrambling techniques are shown to provide a good level of security without impairing the performance of the DVC scheme when compared to the one without scrambling. This is particularly attractive for video surveillance scenarios, which is one of the most promising applications for DVC. Finally, a practical DVC demonstrator built during this research is described, where the main requirements as well as the observed limitations are presented. Furthermore, it is defined in a setup being as close as possible to a complete real application scenario. This shows that it is actually possible to implement a complete end-to-end practical DVC system relying only on realistic assumptions. Even though DVC is inferior in terms of compression efficiency to the state of the art conventional coding for the moment, strengths of DVC reside in its good error resilience properties and the codec-independent scalability feature. Therefore, DVC offers promising possibilities for video compression with transmission over error-prone environments requirement as it significantly outperforms conventional coding in this case.

H.264/AVC 비디오 압축 표준은 압축 효율을 높이기 위해 다양한 크기의 블록을 사용하여 화면 사이의 움직임 예측을 수행한다. H.264/AVC는 가변적인 블록 크기의 움직임 보상을 통해 세밀한 영역의 움직임까지 예측할 수 있어 잔여 영상을 나타내는 정보량을 효과적으로 줄일 수 있다. 복호를 위해서는 각 블록의 움직임 벡터를 전송해야 하는데, 저비트율 환경에서는 움직임 벡터 정보가 전체 비트스트림의 약 40%를 차지한다. 움직임 벡터 정보량을 줄이기 위해 비디오 부호화 전문가 그룹(VCEG)에서는 다양한 움직임 벡터 예측(Motion Vector Competition) 방법을 제안하였다. 다양한 예측 움직임 벡터를 사용하여 실제 전송해야 할 움직임 벡터 차분값(Motion Vector Difference, MVD)의 크기를 줄이기 때문에 압축 효율을 높일 수 있다. 그러나 다양한 예측 움직임 벡터를 사용하기 때문에 선택된 예측 움직임 벡터의 인덱스 정보를 복호기로 전송해야 한다. 이 논문에서는 인덱스 정보를 효율적으로 전송하기 위해 Phased-in 코드를 기반으로 한 새로운 코드워드 표를 제안했다. 실험을 통해 제안한 방법을 이용하여 동일한 화질에서 평균 약 7.24%의 비트율을 절감할 수 있었고, 동일한 비트율에서는 평균 약 0.36dB의 화질을 향상시킬 수 있었다. The H.264/AVC video coding standard performs inter prediction using variable block sizes to improve coding efficiency. Since we predict not only the motion of homogeneous regions but also the motion of non-homogeneous regions accurately using variable block sizes, we can reduce residual information effectively. However, each motion vector should be transmitted to the decoder. In low bit rate environments, motion vector information takes approximately 40% of the total bitstream. Thus, motion vector competition was proposed to reduce the amount of motion vector information. Since the size of the motion vector difference is reduced by motion vector competition, it requires only a small number of bits for motion vector information. However, we need to send the corresponding index of the best motion vector predictor for decoding. In this paper, we propose a new codeword table based on the phased-in code to encode the index of motion vector predictor efficiently. Experimental results show that the proposed algorithm reduces the average bit rate by 7.24% for similar PSNR values, and it improves the average image quality by 0.36dB at similar bit rates.

H.264/AVC, the latest video coding standard, adopts rate-distortion optimization (RDO) technique to obtain the best intra- and inter-prediction, while maximizing visual quality and minimizing the required bitrate. However, the full RD cost calculation for all intra-prediction modes, the exhaustive searches for finding optimal motion vectors for all block sizes, and the multiple references frame procedure considerably increase its computational complexity with the allowed number of prediction modes. The authors have previously proposed an approach for both inter- and intra-mode decisions that takes into account the several effective parameters, image content type, quantization parameter, correlation of motion vectors and motion details of video objects. In order to reduce the complexity, here we propose a new two-step adaptive search range decision by developing a combination of previous algorithm. The first step of our adaptive search range decision determines the search range for current MB according to the motion vectors of the previous MBs. In the second step, the motion vector obtained in the current frame is used to adaptively searching the window size of the previous reference frame for multi-frame motion estimation. As such, only a subset of inter- and intra-modes is chosen for RDO calculation. Experimental results show that the proposed algorithm reduces the total encoding cost with negligible loss in PSNR and a slightly increase in the required bitrate, when compared to our previous algorithm and other algorithms reported in the literature.

The variable block size motion estimation (ME) and compensation (MC) using multiple reference frames is adopted in H.264/AVC to improve coding efficiency. However, the computational complexity for ME/MC increases proportional to the number of reference frames. In this paper, we propose a new efficient reference frame selection algorithm to reduce the complexity. The proposed algorithm selects suitable reference frames by employing the spatial and temporal correlation of video sequence. The experimental results show that the proposed algorithm decreases video encoding time while keeping the similar visual quality and bit rates.

Consecutive corrupted MBs or slice errors are commonly seen in modern video transmission systems. Temporal error concealment is an effective approach to reduce the impact of errors. Conventional temporal error concealment techniques recover slice errors on a MB basis. We propose a new novel temporal scheme for slice error concealment based on a size-adaptive region basis. Size-adaptive region boundary matching can significantly reduce the misalignments or side mismatches that are commonly seen in MB-based approaches. Instead of using MSE or MAE as the matching criteria, We propose a matching criterion based on structural similarity (SSIM). Our proposed matching criterion can achieve better structural alignments and better visual perceptual quality. We also propose a method to detect motion vector outliers at the last stage of the error concealment scheme, which can further improve the visual quality of the concealed videos. Our proposed scheme conceals slice errors with good accuracy as well as low computation complexity, and it can be applied for real-time applications.

Modern video coding standards such as High Efficiency Video Coding (HEVC) and H.264/MPEG‐4 advanced video coding (AVC) supersede the previous coding standards because of their improved coding efficiency. These standards adopt variable block sizes in frame coding ranging from 4 × 4 to 16 × 16 and 4 × 4 to 64 × 64 for H.264/AVC and HEVC, respectively. The use of variable block sizes for inter prediction provides a significant coding gain compared to coding a macroblock (MB) using regular block size. However, this new feature greatly increases the computational complexity of the encoder when brute‐force rate distortion optimization (RDO) algorithm is used for coding parameter selection. This paper proposes an efficient inter prediction mode selection scheme based on motion homogeneity and residual complexity measures of an MB to speed up the encoding process. The motion homogeneity is assessed through the normalized motion vector (MV) field, and residual complexity is evaluated by the sum of absolute difference (SAD). To acquire the MVs and SADs, motion estimation at 8 × 8 block size is performed using a lightweight recursive motion estimator in which the vector field tends toward true object motion. Based on motion homogeneity and residual complexity of an MB, only a small number of inter prediction modes are selected for the RDO process. The experimental results for H.264/AVC show that the proposed scheme reduces the encoding time by 64% on average without any significant degradation of coding efficiency. © 2016 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

The new emerging video coding standard H.264 employs variable block size motion compensation using multiple references with quarter-pel motion vector accuracy. This scheme is a key feature to accomplish higher coding gain, however, also a decisive factor that increases overall computational complexity. To overcome this, we propose a novel fast mode decision scheme suited for variable block sizes by classifying coding modes based on rate-distortion cost. The experimental results show that the proposed method provides significant reduction in computational complexity without any noticeable coding loss and additional operations.

Motion Estimation is the most crucial and time-consuming part of the H.264/AVC video compression standard. The introduction of motion search of variable block sizes in multiple reference frames has significantly increased the computational complexity. This paper proposes a fast motion estimation algorithm, Most Used Reference First (MURF), based on the usage history of reference frames. The algorithm rearranges the search order of the reference frames based on the selection probability of the reference frames in coding the current frame. The experimental results show that the proposed algorithm, when compared to the best algorithm in H.264 reference software, achieves on average 60% reduction in search points and 52% reduction in motion estimation time with comparable video quality and negligible increase in bit-rate and memory.

The H.264 video compression standard uses enhanced Motion Estimation (ME) features to improve both the compression ratio and the quality of compressed video. The two primary enhancements are the use of Variable Block Size Motion Estimation (VBSME) and multiple reference frames. These two additions greatly increase the computational complexity of the ME algorithm, to the point where a software based real-time (30 frames per second (fps)) implementation is not possible on present microprocessors. Thus hardware acceleration of the H.264 ME algorithm is necessary in order to achieve real-time performance for the implementation of the VBSME and multiple reference frames features. This thesis presents a scalable FPGA-based ME architecture that supports real-time H.264 ME for a wide range of video resolutions ─ from 640x480 VGA to 1920x1088 High Definition (HD). The architecture contains innovations in both the data-path design and memory organization to achieve scalability and real-time performance on FPGAs. At 37% FPGA device utilization, the architecture is able to achieve 31 fps performance for encoding full 1920x1088 progressive HDTV video.