Hybrid spatio-temporal error concealment technique for image/video transmission

Compressed bitstreams are, in general, very sensitive to channel errors. For instance, a single bit error in a coded video bitstream may cause severe degradation on picture quality. When bit errors occur during transmission and cannot be corrected by an error correction scheme, error concealment is needed to conceal the corrupted image at the receiver. Error concealment algorithms attempt to repair damaged portions of the picture by exploiting both the spatial and the temporal redundancies in the received and reconstructed video signal. We discuss several temporal, spatial, and transform-domain error concealment techniques for MPEG coded pictures, and propose a new scheme based on directional interpolation. We also compare the performance of these techniques by computer simulation.

The three-dimensional video (3DV) is composed of variable-length stream sequences captured via diversified cameras surrounding an object. Thus, it is an urgent task to accomplish sufficient encoding to be compatible with incoming bandwidth demands, while achieving a recommended 3DV reception performance. In the 3DV compression framework, the lost macro-blocks (MBs) might propagate into the following frames and the adjoining views. Therefore, it is obligatory to avoid error propagation by concealing the corrupted MBs at the decoder through the utilization of appropriate post-processing error concealment (EC) techniques. The existing EC algorithms fundamentally exploit the temporal, inter-view, and spatial matching within the 3DV frames and views to reconstruct the disparity vectors (DVs) and motion vectors (MVs) of the corrupted MBs. Unluckily, in the state of high severe corruptions and heavily erroneous MBs, these concealment algorithms are predominantly unreliable and might give unreliable 3DV quality. Thence, in this work, we suggest the utilization of the outer block boundary matching algorithm to estimate the MVs and the directional interpolation EC algorithm to estimate the DVs of the erroneous MBs. After that, the Bayesian Kalman filter (BKF) is employed because of its efficiency to filter out the inherent errors in the previously predicted DVs and MVs to accomplish better 3D video performance. Experimental results on standard 3DV sequences demonstrate that the suggested BKF-based EC scheme is more powerful with heavy losses. It subjectively and objectively outperforms the traditional concealment techniques at severely random and bursty packet loss rates (PLRs).

Error resilient coding and error concealment techniques have been introduced in scalable video coding to reduce the quality degradation impact of transmission errors. Redundant picture is one of the error resilient tools, appropriate for delay sensitive applications where re-transmission is not possible and channel loss rates such as in packet-based wireless networks, are not a priori known. In this paper, combination of redundant-key-picture error resilient coding with motion-copy error concealment at the decoder is proposed based on experimental evaluation. Experimental results demonstrate that significant PSNR gain can be achieved using key picture error concealment with motion-copy for various tested sequences under different packet loss rates compared to JVT-W049.

In this paper, the error concealment techniques as motion vector extrapolation (MVE), boundary matching (BM), side matching (SM), 16/spl times/16-MB-based and 8/spl times/8-block-based concealment are implemented and compared with feedback-based no waiting ARQ (NWARQ) error recovery technique. A new error concealment method using preprocessing (PREP) and search is proposed and implemented. This new method tries to use the information of surrounding MV as much as possible, and find a more precise motion vector using additional search processing in a search window centered by the estimated motion vector. Simulation results, based JM1.4 codec of JVT, show that both NWARQ and PREP can significantly improve the reconstruction video quality. And the combined PREP, SEARCH and NWARQ techniques can greatly achieve improved performance. From the simulation results, the SM and 16/spl times/16-MB-based concealment methods show better performance than BM and 8/spl times/8-block-based concealment.

We use particle filtering for correcting the erroneous motion vectors (MVs), which are derived from the boundary matching algorithm (BMA) in packet video communications. Assuming a two-state Markov channel model for transmission, the error of the extracted MVs by BMA is shown to be modeled by the Gaussian mixture distribution. Formulating the problem in the state space, we deploy particle filtering for denoising the erroneous MVs. The main challenge of using particle filters is high computational complexity that is directly related to the number of particles. The proposed particle filtering scheme is efficient even if the number of particles is decreased. Experimental results are provided to show the efficiency of this filtering approach compared to a recent scheme based on Kalman filtering. The experiments show meaningful increase in the quality of the recovered video sequences in terms of PSNR up to 3 dB compared with the other error concealment techniques. Also, the computational complexity of the proposed scheme is discussed.

Distributed Video Coding (DVC) is a new paradigm for video compression based on the information theoretical results of Slepian and Wolf (SW), and Wyner and Ziv (WZ). While conventional coding has a rigid complexity allocation as most of the complex tasks are performed at the encoder side, DVC enables a flexible complexity allocation between the encoder and the decoder. The most novel and interesting case is low complexity encoding and complex decoding, which is the opposite of conventional coding. While the latter is suitable for applications where the cost of the decoder is more critical than the encoder's one, DVC opens the door for a new range of applications where low complexity encoding is required and the decoder's complexity is not critical. This is interesting with the deployment of small and battery-powered multimedia mobile devices all around in our daily life. Further, since DVC operates as a reversed-complexity scheme when compared to conventional coding, DVC also enables the interesting scenario of low complexity encoding and decoding between two ends by transcoding between DVC and conventional coding. More specifically, low complexity encoding is possible by DVC at one end. Then, the resulting stream is decoded and conventionally re-encoded to enable low complexity decoding at the other end. Multiview video is attractive for a wide range of applications such as free viewpoint television, which is a system that allows viewing the scene from a viewpoint chosen by the viewer. Moreover, multiview can be beneficial for monitoring purposes in video surveillance. The increased use of multiview video systems is mainly due to the improvements in video technology and the reduced cost of cameras. While a multiview conventional codec will try to exploit the correlation among the different cameras at the encoder side, DVC allows for separate encoding of correlated video sources. Therefore, DVC requires no communication between the cameras in a multiview scenario. This is an advantage since communication is time consuming (i.e. more delay) and requires complex networking. Another appealing feature of DVC is the fact that it is based on a statistical framework. Moreover, DVC behaves as a natural joint source-channel coding solution. This results in an improved error resilience performance when compared to conventional coding. Further, DVC-based scalable codecs do not require a deterministic knowledge of the lower layers. In other words, the enhancement layers are completely independent from the base layer codec. This is called the codec-independent scalability feature, which offers a high flexibility in the way the various layers are distributed in a network. This thesis addresses the following topics: First, the theoretical foundations of DVC as well as the practical DVC scheme used in this research are presented. The potential applications for DVC are also outlined. DVC-based schemes use conventional coding to compress parts of the data, while the rest is compressed in a distributed fashion. Thus, different conventional codecs are studied in this research as they are compared in terms of compression efficiency for a rich set of sequences. This includes fine tuning the compression parameters such that the best performance is achieved for each codec. Further, DVC tools for improved Side Information (SI) and Error Concealment (EC) are introduced for monoview DVC using a partially decoded frame. The improved SI results in a significant gain in reconstruction quality for video with high activity and motion. This is done by re-estimating the erroneous motion vectors using the partially decoded frame to improve the SI quality. The latter is then used to enhance the reconstruction of the finally decoded frame. Further, the introduced spatio-temporal EC improves the quality of decoded video in the case of erroneously received packets, outperforming both spatial and temporal EC. Moreover, it also outperforms error-concealed conventional coding in different modes. Then, multiview DVC is studied in terms of SI generation, which differentiates it from the monoview case. More specifically, different multiview prediction techniques for SI generation are described and compared in terms of prediction quality, complexity and compression efficiency. Further, a technique for iterative multiview SI is introduced, where the final SI is used in an enhanced reconstruction process. The iterative SI outperforms the other SI generation techniques, especially for high motion video content. Finally, fusion techniques of temporal and inter-view side informations are introduced as well, which improves the performance of multiview DVC over monoview coding. DVC is also used to enable scalability for image and video coding. Since DVC is based on a statistical framework, the base and enhancement layers are completely independent, which is an interesting property called codec-independent scalability. Moreover, the introduced DVC scalable schemes show a good robustness to errors as the quality of decoded video steadily decreases with error rate increase. On the other hand, conventional coding exhibits a cliff effect as the performance drops dramatically after a certain error rate value. Further, the issue of privacy protection is addressed for DVC by transform domain scrambling, which is used to alter regions of interest in video such that the scene is still understood and privacy is preserved as well. The proposed scrambling techniques are shown to provide a good level of security without impairing the performance of the DVC scheme when compared to the one without scrambling. This is particularly attractive for video surveillance scenarios, which is one of the most promising applications for DVC. Finally, a practical DVC demonstrator built during this research is described, where the main requirements as well as the observed limitations are presented. Furthermore, it is defined in a setup being as close as possible to a complete real application scenario. This shows that it is actually possible to implement a complete end-to-end practical DVC system relying only on realistic assumptions. Even though DVC is inferior in terms of compression efficiency to the state of the art conventional coding for the moment, strengths of DVC reside in its good error resilience properties and the codec-independent scalability feature. Therefore, DVC offers promising possibilities for video compression with transmission over error-prone environments requirement as it significantly outperforms conventional coding in this case.

H.263 based video codec for real-time visual communications over wireless radio networks

Error concealment algorithms for robust decoding of MPEG compressed video

Capturing or acquiring video data from advanced gadgets, like cell phone cameras, smart phones, camcorders, etc., is very recent and up growing trend. With the advancement of the video camera as the integrated part of these multimedia devices generates necessity for video stabilization. Digital video stabilization or video frame alignment techniques are used to improve quality of input video by removing the undesired camera motions jitters; undesirable jiggles etc. Many video stabilization techniques are developed till now but only fast and low computational complexity approaches can be applied on the embedded devices. The paper presents the review of fast block matching local motion vector estimation and alignment algorithm. Effectiveness of the algorithm is shown by quantitative evaluation. The proposed work will also help to boost the performance of various error concealment, error resilience and video quality improvement techniques in H.264/HEVC video codec.

A robust image transmission scheme deploying distributed-Alamouti space-time code (STC) over wireless relay networks is proposed. Specifically, we propose a cross-layer design combining application, networking, and physical layer to enhance the quality of multimedia data over error-prone wireless channels. At the application layer, a coded image sequence based on the wavelet transform is divided into two priorities. The higher priority contains the most significant information of image and the lower priority conveys less important information. At the physical layer, two pre-assigned relays cooperate to form a distributed-Alamouti code which help the source to transmit the low priority sequence of the image over error-prone wireless channels. At the networking layer, among the whole relay nodes in the networks we select two relay nodes which provides the maximum signal-to-noise ratio to convey the higher priority. With this strategy, we implicitly achieve unequal error protection for image transmission without controlling power or varying modulation and coding at the transmitter at the expense of increasing network's overhead (i.e. feedback information). The selection of relays needs limited feedback, hence, enabling our proposed scheme feasible in practical applications. Analytical and simulation results are provided to verify our proposal. It has been shown that the image quality is remarkably enhanced in terms of objective quality using the proposed scheme compared to the conventional transmission schemes.

Transmission of digital contents in unavoidable noise-prone environments demands sophisticated error detection and concealment techniques to restore the perceptual qualities. Thus, robust transmission with error resilience has become an indispensable component in many multimedia coding standards including MPEG2/4 and JPEG2000. A variety of error detection/concealment techniques have been reported in the literature. However there has been little work done by employing the advantages of data hiding, which is emerging as a powerful tool for efficient protection and authentication of digital contents. In this paper, we propose a new data hiding technique to offer error resilience of transmitted images in wireless environments. First, the authentication information, which is generated from structural digital signature, will be used for error detection. Second, the recovery information, which is an approximate version of its original data, will be used for error concealment. Quantitative simulation results are provided to demonstrate the performance of the proposed method.

Error concealment is a well-known technique to improve reproduced picture quality in cases which parts of the coded picture are not available at a decoder for reconstruction. The conventional error concealment methods generally make use of the correlation between a damaged marcoblock and its adjacent macroblocks in the same frame and/or the previous frame. However, performance results of concealment methods are different each other and dependent on how much damaged frames and macroblocks are correlated to surrounding ones temporally and spatially. Moreover, most of them can not conceal effectively the consecutively damaged frames or macro blocks caused by wireless bursty packet loss features, since motion vectors and correlation information between the damaged images are already destroyed. The proposed error concealment aware error resilient video coding is a new method which has an intelligent error concealment selector by using pre-estimation technique of the possibly damaged motion vectors at the encoder. The method recommends the best error concealment per macroblock by transmitting additionally its error concealment selection codes to the decoder. The chosen error concealment helps to conceal the damaged MB at the decoder even when losses of frames are bursty. The experimental results show the substantial effectiveness to improve video quality at the cost of a little bit overhead at the transmitted bit streams.

The problem of error control and concealment in video communication is becoming increasingly important because of the growing interest in video delivery over unreliable channels such as wireless networks and the Internet. This paper reviews the techniques that have been developed for error control and concealment. These techniques are described in three categories according to the roles that the encoder and decoder play in the underlying approaches. Forward error concealment includes methods that add redundancy at the source end to enhance error resilience of the coded bit streams. Error concealment by postprocessing refers to operations at the decoder to recover the damaged areas based on characteristics of image and video signals. Last, interactive error concealment covers techniques that are dependent on a dialogue between the source and destination. Both current research activities and practice in international standards are covered.

In spatial error concealment (SEC), methods like bilinear interpolation (BI) and directional interpolation (DI) are commonly used to estimate the missing pixel values resulting from losses occurring in video streams. Despite being able to preserve spatial smoothness, BI produces a blurring effect and is unable to preserve structural information. DI produces spurious edges in regions with no strong edges, resulting in visible artefacts. In this paper, we propose a SEC algorithm that addresses the above drawbacks by formulating a weighted sum of candidate macroblocks produced from DI and BI, with weights derived adaptively through local information. We demonstrate that the proposed algorithm offers visual improvements over both DI based algorithms and the SEC algorithm based on BI in the H.264/AVC reference software JM 12.0. Most importantly, this unique approach preserves edge information and spatial smoothness in the error concealed macroblock due to the integration of both BI and DI.

Distributed Video Coding (DVC) is based on a new paradigm in coding, which has received many interests recently. Among other features, DVC is said to provide better error resiliency. This paper proposes a hybrid spatial and temporal error concealment scheme to conceal errors in Wyner-Ziv (WZ) frames. We first use a spatial concealment based on edge directed filter. This step is exploited to improve the performance of subsequent temporal concealment. An enhanced temporal concealment based on motion compensated temporal interpolation is proposed, including motion vector refinement and smoothing, optimal compensation mode selection, and a new matching criterion for motion estimation. Simulation results show that the objective qualities as well as the perceptual qualities of the corrupted sequences are significantly improved by the hybrid error concealment, outperforming both spatial and temporal concealments alone.