Bi-direction Motion Vector retrieval based error concealment scheme for H.264/AVC

Error protection and concealment of motion vectors are of prime concern when video is transmitted over variable-bandwidth error-prone channels, such as wireless channels. In this paper, we investigate the influence of corrupted motion vectors in video coding based on motion-compensated temporal filtering, and develop various error protection and concealment mechanisms for this class of codecs. The experimental results show that our proposed motion vector coding technique significantly increases the robustness against transmission errors and generates performance gains of up to 7 dB compared with the original coding technique at the cost of less than 4% in terms of rate. It is also shown that our proposed spatial error-concealment mechanism leads to additional performance gains of up to 4 dB.

Compressed video bit-streams are extremely sensitive to packet loss over error-prone channels. Error concealment (EC) has been considered as one of error control techniques to improve the reconstructed picture quality against transmission errors. However, EC methods show poor image reconstruction performance due to unavailable scene change information of the video sequence, especially when an abrupt scene change occurs. In this paper, we propose an effective EC method based on scene change detection algorithm (SCDA), which provides information to decide whether spatial or temporal EC is better to be used for intra and inter frames respectively. The simulation results show that the proposed method highly improves the subjective quality of incorrectly decoded frames and obtains an average gain of 0.7dB compared with the H.264/AVC Joint Model reference software.

Distributed Video Coding (DVC) is a new paradigm for video compression based on the information theoretical results of Slepian and Wolf (SW), and Wyner and Ziv (WZ). While conventional coding has a rigid complexity allocation as most of the complex tasks are performed at the encoder side, DVC enables a flexible complexity allocation between the encoder and the decoder. The most novel and interesting case is low complexity encoding and complex decoding, which is the opposite of conventional coding. While the latter is suitable for applications where the cost of the decoder is more critical than the encoder's one, DVC opens the door for a new range of applications where low complexity encoding is required and the decoder's complexity is not critical. This is interesting with the deployment of small and battery-powered multimedia mobile devices all around in our daily life. Further, since DVC operates as a reversed-complexity scheme when compared to conventional coding, DVC also enables the interesting scenario of low complexity encoding and decoding between two ends by transcoding between DVC and conventional coding. More specifically, low complexity encoding is possible by DVC at one end. Then, the resulting stream is decoded and conventionally re-encoded to enable low complexity decoding at the other end. Multiview video is attractive for a wide range of applications such as free viewpoint television, which is a system that allows viewing the scene from a viewpoint chosen by the viewer. Moreover, multiview can be beneficial for monitoring purposes in video surveillance. The increased use of multiview video systems is mainly due to the improvements in video technology and the reduced cost of cameras. While a multiview conventional codec will try to exploit the correlation among the different cameras at the encoder side, DVC allows for separate encoding of correlated video sources. Therefore, DVC requires no communication between the cameras in a multiview scenario. This is an advantage since communication is time consuming (i.e. more delay) and requires complex networking. Another appealing feature of DVC is the fact that it is based on a statistical framework. Moreover, DVC behaves as a natural joint source-channel coding solution. This results in an improved error resilience performance when compared to conventional coding. Further, DVC-based scalable codecs do not require a deterministic knowledge of the lower layers. In other words, the enhancement layers are completely independent from the base layer codec. This is called the codec-independent scalability feature, which offers a high flexibility in the way the various layers are distributed in a network. This thesis addresses the following topics: First, the theoretical foundations of DVC as well as the practical DVC scheme used in this research are presented. The potential applications for DVC are also outlined. DVC-based schemes use conventional coding to compress parts of the data, while the rest is compressed in a distributed fashion. Thus, different conventional codecs are studied in this research as they are compared in terms of compression efficiency for a rich set of sequences. This includes fine tuning the compression parameters such that the best performance is achieved for each codec. Further, DVC tools for improved Side Information (SI) and Error Concealment (EC) are introduced for monoview DVC using a partially decoded frame. The improved SI results in a significant gain in reconstruction quality for video with high activity and motion. This is done by re-estimating the erroneous motion vectors using the partially decoded frame to improve the SI quality. The latter is then used to enhance the reconstruction of the finally decoded frame. Further, the introduced spatio-temporal EC improves the quality of decoded video in the case of erroneously received packets, outperforming both spatial and temporal EC. Moreover, it also outperforms error-concealed conventional coding in different modes. Then, multiview DVC is studied in terms of SI generation, which differentiates it from the monoview case. More specifically, different multiview prediction techniques for SI generation are described and compared in terms of prediction quality, complexity and compression efficiency. Further, a technique for iterative multiview SI is introduced, where the final SI is used in an enhanced reconstruction process. The iterative SI outperforms the other SI generation techniques, especially for high motion video content. Finally, fusion techniques of temporal and inter-view side informations are introduced as well, which improves the performance of multiview DVC over monoview coding. DVC is also used to enable scalability for image and video coding. Since DVC is based on a statistical framework, the base and enhancement layers are completely independent, which is an interesting property called codec-independent scalability. Moreover, the introduced DVC scalable schemes show a good robustness to errors as the quality of decoded video steadily decreases with error rate increase. On the other hand, conventional coding exhibits a cliff effect as the performance drops dramatically after a certain error rate value. Further, the issue of privacy protection is addressed for DVC by transform domain scrambling, which is used to alter regions of interest in video such that the scene is still understood and privacy is preserved as well. The proposed scrambling techniques are shown to provide a good level of security without impairing the performance of the DVC scheme when compared to the one without scrambling. This is particularly attractive for video surveillance scenarios, which is one of the most promising applications for DVC. Finally, a practical DVC demonstrator built during this research is described, where the main requirements as well as the observed limitations are presented. Furthermore, it is defined in a setup being as close as possible to a complete real application scenario. This shows that it is actually possible to implement a complete end-to-end practical DVC system relying only on realistic assumptions. Even though DVC is inferior in terms of compression efficiency to the state of the art conventional coding for the moment, strengths of DVC reside in its good error resilience properties and the codec-independent scalability feature. Therefore, DVC offers promising possibilities for video compression with transmission over error-prone environments requirement as it significantly outperforms conventional coding in this case.

Error concealment is a well-known technique to improve reproduced picture quality in cases which parts of the coded picture are not available at a decoder for reconstruction. The conventional error concealment methods generally make use of the correlation between a damaged marcoblock and its adjacent macroblocks in the same frame and/or the previous frame. However, performance results of concealment methods are different each other and dependent on how much damaged frames and macroblocks are correlated to surrounding ones temporally and spatially. Moreover, most of them can not conceal effectively the consecutively damaged frames or macro blocks caused by wireless bursty packet loss features, since motion vectors and correlation information between the damaged images are already destroyed. The proposed error concealment aware error resilient video coding is a new method which has an intelligent error concealment selector by using pre-estimation technique of the possibly damaged motion vectors at the encoder. The method recommends the best error concealment per macroblock by transmitting additionally its error concealment selection codes to the decoder. The chosen error concealment helps to conceal the damaged MB at the decoder even when losses of frames are bursty. The experimental results show the substantial effectiveness to improve video quality at the cost of a little bit overhead at the transmitted bit streams.

Consecutive corrupted MBs or slice errors are commonly seen in modern video transmission systems. Temporal error concealment is an effective approach to reduce the impact of errors. Conventional temporal error concealment techniques recover slice errors on a MB basis. We propose a new novel temporal scheme for slice error concealment based on a size-adaptive region basis. Size-adaptive region boundary matching can significantly reduce the misalignments or side mismatches that are commonly seen in MB-based approaches. Instead of using MSE or MAE as the matching criteria, We propose a matching criterion based on structural similarity (SSIM). Our proposed matching criterion can achieve better structural alignments and better visual perceptual quality. We also propose a method to detect motion vector outliers at the last stage of the error concealment scheme, which can further improve the visual quality of the concealed videos. Our proposed scheme conceals slice errors with good accuracy as well as low computation complexity, and it can be applied for real-time applications.

The three-dimensional video (3DV) is composed of variable-length stream sequences captured via diversified cameras surrounding an object. Thus, it is an urgent task to accomplish sufficient encoding to be compatible with incoming bandwidth demands, while achieving a recommended 3DV reception performance. In the 3DV compression framework, the lost macro-blocks (MBs) might propagate into the following frames and the adjoining views. Therefore, it is obligatory to avoid error propagation by concealing the corrupted MBs at the decoder through the utilization of appropriate post-processing error concealment (EC) techniques. The existing EC algorithms fundamentally exploit the temporal, inter-view, and spatial matching within the 3DV frames and views to reconstruct the disparity vectors (DVs) and motion vectors (MVs) of the corrupted MBs. Unluckily, in the state of high severe corruptions and heavily erroneous MBs, these concealment algorithms are predominantly unreliable and might give unreliable 3DV quality. Thence, in this work, we suggest the utilization of the outer block boundary matching algorithm to estimate the MVs and the directional interpolation EC algorithm to estimate the DVs of the erroneous MBs. After that, the Bayesian Kalman filter (BKF) is employed because of its efficiency to filter out the inherent errors in the previously predicted DVs and MVs to accomplish better 3D video performance. Experimental results on standard 3DV sequences demonstrate that the suggested BKF-based EC scheme is more powerful with heavy losses. It subjectively and objectively outperforms the traditional concealment techniques at severely random and bursty packet loss rates (PLRs).

Compressed video streams are very exposed to channel losses when they are transmitted through an unreliable network. The predictive encoding scheme is efficient but even a single packet loss can cause decoding error propagation through all the frames following the erasure. An error-resilience approach to tackle this problem is the multiple description coding (MDC) technique. Another important issue connected to this approach is the problem of the error-concealment (EC) used to reconstruct the information lost during the transmission. In this paper, we deal with a sequence-based EC algorithm that performs the error-concealment by using the information coming from a certain number of frames in the future to minimize the effect of error propagation. The target of this work is applying such an EC strategy to a balanced MDC system. Simulation results show the great improvement that a sequence-based EC algorithm can give compared to classical EC approaches.

An Adaptive Spatio-Temporal Error Concealment (ASTEC) scheme based on H.264/AVC is proposed in the paper to improve the quality of the rebuilt frame in H.264/AVC decoder side. Combining the spatial error concealment method with the temporal one, a new scheme is proposed to conceal an erroneous macro block according to its encoded mode. The temporal error concealment scheme adopts the weighted Boundary Matching Algorithm to recover the erroneous macro block. The spatial error concealment scheme recovers the erroneous macro block by exploiting the directional interpolation which subdivides the macro block according to the number and the direction of edges going through it. Experiments show that the proposed concealment scheme can restrain the diffusion of erroneous macro block and improve the quality of the rebuilt frame subjectively as well as objectively.

Error concealment at a decoder is an efficient method to reduce the degradation of visual quality caused by channel errors. In this paper, we propose a novel spatio-temporal error concealment algorithm based on the spatial-temporal fading (STF) scheme which has been recently introduced. Although STF achieves good performance for the error concealment, several drawbacks including blurring still remain in the concealed blocks. To alleviate these drawbacks, in the proposed method, hybrid approaches with adaptive weights are proposed. First, the boundary matching algorithm and the decoder motion vector estimation which are well-known temporal error concealment methods are adaptively combined to compensate for the defect of each other. Then, an edge preserved method is utilized to reduce the blurring effects caused by the bilinear interpolation for spatial error concealment. Finally, two concealed results obtained by the hybrid spatial and temporal error concealment are pixel-wisely blended with adaptive weights. Experimental results exhibit that the proposed method outperforms conventional methods including STF in terms of the PSNR performance as well as subjective visual quality, and the computational complexity of the proposed method is similar to that of STF.

We propose an error concealment method for robust video transmission based on watermarking technique. In this method, the motion vector information is used as the watermark and embedded in the transform coefficients before quantization. A proper embedding algorithm is designed with the characteristics of the video coder taken into consideration, so that the embedding watermark has little influence on the video quality and the coding efficiency. At the decoder, the extracted watermark is used to effectively detect errors and restore the corrupted motion vectors. Then the restored motion vectors are utilized in error concealment. Simulation results demonstrate the advantages of the proposed method in error prone environment. The proposed method is applicable to all the block-transform based video coding schemes.

Motion-vector-based error concealment algorithms recover lost blocks by motion compensation with the estimated motion vectors. Traditionally, the estimated motion vectors for the lost block are derived by the direct usage of the neighboring available motion vectors. In this work, the residual signal in a block encoded in the video bitstream is used to measure the reliability of the corresponding motion vector. The residual information is applied to weight the neighboring available motion vectors differently, to further improve a newly proposed mathematical formulation between encoded motion vectors and extrapolated motion vectors. The proposed error concealment method using the residual information outperforms a state-of-the-art error concealment algorithm by up to 2.12 dB in sequence PSNR, and outperforms a classic DMVE method by up to 8.27 dB.

We present an error-concealed embedded wavelet (ECEW) video coding system for transmission over Internet or wireless networks. This system consists of two types of frames: intra (I) frames and inter, or predicted (P), frames. Inter frames are constructed by the residual frames formed by variable block-size multiresolution motion estimation (MRME). Motion vectors are compressed by arithmetic coding. The image data of intra frames and residual frames are coded by error-resilient embedded zerotree wavelet (ER-EZW) coding. The ER-EZW coding partitions the wavelet coefficients into several groups and each group is coded independently. Therefore, the error propagation effect resulting from an error is only confined in a group. In EZW coding any single error may result in a totally undecodable bitstream. To further reduce the error damage, we use the error concealment at the decoding end. In intra frames, the erroneous wavelet coefficients are replaced by neighbors. In inter frames, erroneous blocks of wavelet coefficients are replaced by data from the previous frame. Simulations show that the performance of ECEW is superior to ECEW without error concealment by 7 to approximately 8 dB at the error-rate of 10<SUP>-3</SUP> in intra frames. The improvement still has 2 to approximately 3 dB at a higher error-rate of 10<SUP>-2</SUP> in inter frames.

H.264 adopts new coding tools such as variable block size, quarter-pixel-accuracy motion estimation/compensation, multiple reference frames, loop filter, etc. The adoption of these tools enables a macroblock to have more information compared with previous standards. In H.264 each macroblock can have up to sixteen motion vectors, four reference frames, and a mode of the macroblock, all of which can be used for temporal error concealment. The H.264 error concealment in the informative part, however, uses a similar method to prior coding standards that consider only motion vectors of macroblocks adjacent to the lost macroblock. In this paper, we propose an effective temporal error concealment algorithm for H.264-coded video. It uses not only motion vectors and reference frames but the modes of macroblocks adjacent to the lost macroblock as well. Depending on the modes of neighboring macroblocks, each lost macroblock is concealed on the basis of different block sizes. Simulation results show the proposed method yields better video quality than conventional approaches.

Video transmission in error-prone networks suffers bit errors and packet losses that degrade the decoded video quality. In this paper, a video error concealment scheme based on H.264/AVC is proposed to enhance the error robustness using reversible data hiding. At the encoder, motion vectors of macro blocks belonging to the region of interest (ROI) are embedded into the background region within the same frame. The histogram modification technique based on quantized DCT coefficients of coded macro blocks is used to achieve loss less recovery after data extraction. If a macro block in the ROI is found missing at the decoder, the embedded motion vector can be extracted from the corresponding host macro block for the recovery of the missing macro block. This proposed scheme makes a tradeoff between error concealment performance and compression efficiency. Experimental results show the better reconstructed video quality can be achieved compared with the related error concealment methods.

In temporal error concealment (EC), the sum of absolute difference (SAD) is commonly used to identify the best replacement macroblock. Even though the use of SAD ensures spatial continuity and produces visually good results, it is insufficient to ensure edge alignment. Other distortion criteria based solely on structural alignment may also perform poorly in the absence of strong edges. In this paper, we propose a spatio-temporal EC search algorithm using an edge weighted SAD distortion criterion. This distortion criterion ensures both edge alignment and spatial continuity. We assume the loss of motion information and use zero motion vector as the starting search point. We show that the proposed algorithm outperforms the use of unweighted SAD in general. Most importantly, the perceptual quality of EC is improved due to edge alignment while ensuring spatial continuity.