Error control and concealment for video communication: a review

Distributed Video Coding (DVC) is a new paradigm for video compression based on the information theoretical results of Slepian and Wolf (SW), and Wyner and Ziv (WZ). While conventional coding has a rigid complexity allocation as most of the complex tasks are performed at the encoder side, DVC enables a flexible complexity allocation between the encoder and the decoder. The most novel and interesting case is low complexity encoding and complex decoding, which is the opposite of conventional coding. While the latter is suitable for applications where the cost of the decoder is more critical than the encoder's one, DVC opens the door for a new range of applications where low complexity encoding is required and the decoder's complexity is not critical. This is interesting with the deployment of small and battery-powered multimedia mobile devices all around in our daily life. Further, since DVC operates as a reversed-complexity scheme when compared to conventional coding, DVC also enables the interesting scenario of low complexity encoding and decoding between two ends by transcoding between DVC and conventional coding. More specifically, low complexity encoding is possible by DVC at one end. Then, the resulting stream is decoded and conventionally re-encoded to enable low complexity decoding at the other end. Multiview video is attractive for a wide range of applications such as free viewpoint television, which is a system that allows viewing the scene from a viewpoint chosen by the viewer. Moreover, multiview can be beneficial for monitoring purposes in video surveillance. The increased use of multiview video systems is mainly due to the improvements in video technology and the reduced cost of cameras. While a multiview conventional codec will try to exploit the correlation among the different cameras at the encoder side, DVC allows for separate encoding of correlated video sources. Therefore, DVC requires no communication between the cameras in a multiview scenario. This is an advantage since communication is time consuming (i.e. more delay) and requires complex networking. Another appealing feature of DVC is the fact that it is based on a statistical framework. Moreover, DVC behaves as a natural joint source-channel coding solution. This results in an improved error resilience performance when compared to conventional coding. Further, DVC-based scalable codecs do not require a deterministic knowledge of the lower layers. In other words, the enhancement layers are completely independent from the base layer codec. This is called the codec-independent scalability feature, which offers a high flexibility in the way the various layers are distributed in a network. This thesis addresses the following topics: First, the theoretical foundations of DVC as well as the practical DVC scheme used in this research are presented. The potential applications for DVC are also outlined. DVC-based schemes use conventional coding to compress parts of the data, while the rest is compressed in a distributed fashion. Thus, different conventional codecs are studied in this research as they are compared in terms of compression efficiency for a rich set of sequences. This includes fine tuning the compression parameters such that the best performance is achieved for each codec. Further, DVC tools for improved Side Information (SI) and Error Concealment (EC) are introduced for monoview DVC using a partially decoded frame. The improved SI results in a significant gain in reconstruction quality for video with high activity and motion. This is done by re-estimating the erroneous motion vectors using the partially decoded frame to improve the SI quality. The latter is then used to enhance the reconstruction of the finally decoded frame. Further, the introduced spatio-temporal EC improves the quality of decoded video in the case of erroneously received packets, outperforming both spatial and temporal EC. Moreover, it also outperforms error-concealed conventional coding in different modes. Then, multiview DVC is studied in terms of SI generation, which differentiates it from the monoview case. More specifically, different multiview prediction techniques for SI generation are described and compared in terms of prediction quality, complexity and compression efficiency. Further, a technique for iterative multiview SI is introduced, where the final SI is used in an enhanced reconstruction process. The iterative SI outperforms the other SI generation techniques, especially for high motion video content. Finally, fusion techniques of temporal and inter-view side informations are introduced as well, which improves the performance of multiview DVC over monoview coding. DVC is also used to enable scalability for image and video coding. Since DVC is based on a statistical framework, the base and enhancement layers are completely independent, which is an interesting property called codec-independent scalability. Moreover, the introduced DVC scalable schemes show a good robustness to errors as the quality of decoded video steadily decreases with error rate increase. On the other hand, conventional coding exhibits a cliff effect as the performance drops dramatically after a certain error rate value. Further, the issue of privacy protection is addressed for DVC by transform domain scrambling, which is used to alter regions of interest in video such that the scene is still understood and privacy is preserved as well. The proposed scrambling techniques are shown to provide a good level of security without impairing the performance of the DVC scheme when compared to the one without scrambling. This is particularly attractive for video surveillance scenarios, which is one of the most promising applications for DVC. Finally, a practical DVC demonstrator built during this research is described, where the main requirements as well as the observed limitations are presented. Furthermore, it is defined in a setup being as close as possible to a complete real application scenario. This shows that it is actually possible to implement a complete end-to-end practical DVC system relying only on realistic assumptions. Even though DVC is inferior in terms of compression efficiency to the state of the art conventional coding for the moment, strengths of DVC reside in its good error resilience properties and the codec-independent scalability feature. Therefore, DVC offers promising possibilities for video compression with transmission over error-prone environments requirement as it significantly outperforms conventional coding in this case.

The problem of application-layer error control for real-time video transmission over packet lossy networks is commonly addressed by joint source-channel coding (JSCC). The traditional JSCC approaches solve this problem in a sequential manner, where source coding and channel coding are not fully integrated. In this paper, we present an integrated joint source-channel coding (IJSCC) framework, where error resilient source coding, channel coding and error concealment are jointly considered in an integrated manner. We show through both analysis and simulations the advantages of the proposed IJSCC approach, in comparison to a sequential JSCC approach.

Compressed video bit-streams are extremely sensitive to packet loss over error-prone channels. Error concealment (EC) has been considered as one of error control techniques to improve the reconstructed picture quality against transmission errors. However, EC methods show poor image reconstruction performance due to unavailable scene change information of the video sequence, especially when an abrupt scene change occurs. In this paper, we propose an effective EC method based on scene change detection algorithm (SCDA), which provides information to decide whether spatial or temporal EC is better to be used for intra and inter frames respectively. The simulation results show that the proposed method highly improves the subjective quality of incorrectly decoded frames and obtains an average gain of 0.7dB compared with the H.264/AVC Joint Model reference software.

We study error control for an MPEG2 digital video compression scheme. The compressed video is transmitted over an asynchronous transfer mode (ATM) network. We propose to combine error control coding with error concealment. Specifically we use classical error detection to locate errors in the compressed domain, translate this information to the uncompressed domain and then use standard error concealment in the error locations. The solving of the error location problem in this way is the main contribution of this work.

This work addresses the problem of error concealment in video transmission systems over noisy channels employing Bregman divergences along with regularization. Error concealment intends to improve the effects of disturbances at the reception due to bit-errors or cell loss in packet networks. Bregman regularization gives accurate answers after just some iterations with fast convergence, better accuracy and stability. This technique has an adaptive nature: the regularization functional is updated according to Bregman functions that change from iteration to iteration according to the nature of the neighborhood under study at iteration n. Numerical experiments show that high-quality regularization parameter estimates can be obtained. The convergence is sped up while turning the regularization parameter estimation less empiric, and more automatic.KeywordsInverse problemsoptical flowBregman divergencesexponential distributionserror concealmentcomputer visionregularization

This paper presents an efficient error concealment method for multiview video plus depth (MVD) delivery over error-prone channels based on the view synthesis information. Specifically, we propose an enhanced temporal error concealment (TEC) method by estimating the probable partition and missing MVs of a corrupted macroblock (MB) using the synthesized information. Furthermore, the view synthesis error concealment (VSEC) approach that directly uses the synthesized pixels to conceal the corrupted MBs is also adaptively utilized together with the TEC approach. Experimental results have shown the effectiveness of the proposed method compared with the existing methods in terms of both objective and subjective measures.

Compressed bitstreams are, in general, very sensitive to channel errors. For instance, a single bit error in a coded video bitstream may cause severe degradation on picture quality. When bit errors occur during transmission and cannot be corrected by an error correction scheme, error concealment is needed to conceal the corrupted image at the receiver. Error concealment algorithms attempt to repair damaged portions of the picture by exploiting both the spatial and the temporal redundancies in the received and reconstructed video signal. We discuss several temporal, spatial, and transform-domain error concealment techniques for MPEG coded pictures, and propose a new scheme based on directional interpolation. We also compare the performance of these techniques by computer simulation.

We present an error-concealed embedded wavelet (ECEW) video coding system for transmission over Internet or wireless networks. This system consists of two types of frames: intra (I) frames and inter, or predicted (P), frames. Inter frames are constructed by the residual frames formed by variable block-size multiresolution motion estimation (MRME). Motion vectors are compressed by arithmetic coding. The image data of intra frames and residual frames are coded by error-resilient embedded zerotree wavelet (ER-EZW) coding. The ER-EZW coding partitions the wavelet coefficients into several groups and each group is coded independently. Therefore, the error propagation effect resulting from an error is only confined in a group. In EZW coding any single error may result in a totally undecodable bitstream. To further reduce the error damage, we use the error concealment at the decoding end. In intra frames, the erroneous wavelet coefficients are replaced by neighbors. In inter frames, erroneous blocks of wavelet coefficients are replaced by data from the previous frame. Simulations show that the performance of ECEW is superior to ECEW without error concealment by 7 to approximately 8 dB at the error-rate of 10<SUP>-3</SUP> in intra frames. The improvement still has 2 to approximately 3 dB at a higher error-rate of 10<SUP>-2</SUP> in inter frames.

In a packet erasure network environment, video streaming benefits from error control in two ways to achieve graceful degradation. The first approach is application-level (or the link-level) forward error-correction (FEC) to provide erasure protection. The second error control approach is error concealment at the decoder end to compensate lost packets. A large amount of research work has been done in the above two areas. More recently, network coding (NC) techniques have been proposed for efficient data multicast over networks. It was shown in our previous work that multicast video streaming benefits from NC for its throughput improvement. An algebraic model is given to analyze the performance in this work. By exploiting the linear combination of video packets along nodes in a network and the SVC video format, the system achieves path diversity automatically and enables efficient video delivery to heterogeneous receivers in packet erasure channels. The application of network coding can protect video packets against the erasure network environment. However, the rank defficiency problem of random linear network coding makes the error concealment inefficiently. It is shown by computer simulation that the proposed NC video multicast scheme enables heterogenous receiving according to their capacity constraints. But it needs special designing to improve the video transmission performance when applying network coding.

Error concealment algorithms for robust decoding of MPEG compressed video

Scalable video coding (SVC) has aroused a wide interest in the areas of video coding and transmission technology, since it provides desirable features for heterogeneous error-prone network environments. The layered video structure allows not only adaptation to the available bandwidth but also a device adaptation capability via multiple decodable sub-streams. In this paper, we focus on SVC Medium- Grain Scalability (MGS) and investigate the end user's quality of experience (QoE) in error-prone transmission conditions. We describe our error detection and concealment implementation on the JSVM 9.15 reference decoder and assess its effectiveness through several quality indicators. Results show an improvement of the proposed concealment strategy when compared to the standard Frame Copy (FC) and Interpolation, in terms of Peak Signal-to-Signal Noise Ratio (PSNR) and Structural Similarity Index Metric (SSIM) quality metrics. This improvement is evident in the case of concealment on quality enhancement layers, since video fluidity is preserved assuring an acceptable QoE to the end users.

In this paper, we propose an error concealment technique for H.264 coded videos. The algorithm is targeted at compressed videos degraded after packet losses caused by transmission over an unreliable channel. We use a combination of watermarking and half toning (dithering) techniques. At the encoder side, a dithered version of each video frame is embedded into the video using a watermarking technique. The watermarking technique used by the proposed algorithm is a modified version of the Quantization Index Modulation (QIM) algorithm, which provides a good data hiding capacity. At the decoder side, the algorithm identifies which packets of the video were lost, extracts the corresponding mark, and applies an inverse half toning technique to estimate the original content. The algorithm is fast and has a good performance, being able to restore content with a better quality than the default H.264 error concealment algorithm.

The three-dimensional video (3DV) is composed of variable-length stream sequences captured via diversified cameras surrounding an object. Thus, it is an urgent task to accomplish sufficient encoding to be compatible with incoming bandwidth demands, while achieving a recommended 3DV reception performance. In the 3DV compression framework, the lost macro-blocks (MBs) might propagate into the following frames and the adjoining views. Therefore, it is obligatory to avoid error propagation by concealing the corrupted MBs at the decoder through the utilization of appropriate post-processing error concealment (EC) techniques. The existing EC algorithms fundamentally exploit the temporal, inter-view, and spatial matching within the 3DV frames and views to reconstruct the disparity vectors (DVs) and motion vectors (MVs) of the corrupted MBs. Unluckily, in the state of high severe corruptions and heavily erroneous MBs, these concealment algorithms are predominantly unreliable and might give unreliable 3DV quality. Thence, in this work, we suggest the utilization of the outer block boundary matching algorithm to estimate the MVs and the directional interpolation EC algorithm to estimate the DVs of the erroneous MBs. After that, the Bayesian Kalman filter (BKF) is employed because of its efficiency to filter out the inherent errors in the previously predicted DVs and MVs to accomplish better 3D video performance. Experimental results on standard 3DV sequences demonstrate that the suggested BKF-based EC scheme is more powerful with heavy losses. It subjectively and objectively outperforms the traditional concealment techniques at severely random and bursty packet loss rates (PLRs).

The problem of efficient global error estimation and control is studied in embedded nested implicit Runge–Kutta pairs of Gauss and Lobatto types as applied to stiff ordinary differential equations (ODEs). Stiff problems may arise in many areas of engineering, and their accurate numerical solution is an important issue of computational and applied mathematics. A cheap global error estimation technique designed recently for the mentioned Runge–Kutta pairs can severely overestimate the global error when applied to stiff ODEs and, hence, this reduces the efficiency of those solvers. In the present paper, we explain the cause of that error overestimation and show how to improve the mentioned computation techniques for stiff problems. Such modifications not only boost the efficiency of numerical integration of stiff ODEs, but also make the embedded nested implicit Runge–Kutta pairs with scaled modified local and global error controls superior to stiff built-in MATLAB ODE solvers with only local error control when applied to important test examples.

Error resilient coding and error concealment techniques have been introduced in scalable video coding to reduce the quality degradation impact of transmission errors. Redundant picture is one of the error resilient tools, appropriate for delay sensitive applications where re-transmission is not possible and channel loss rates such as in packet-based wireless networks, are not a priori known. In this paper, combination of redundant-key-picture error resilient coding with motion-copy error concealment at the decoder is proposed based on experimental evaluation. Experimental results demonstrate that significant PSNR gain can be achieved using key picture error concealment with motion-copy for various tested sequences under different packet loss rates compared to JVT-W049.