Adaptive Video Error Concealment Using Reversible Data Hiding

The three-dimensional video (3DV) is composed of variable-length stream sequences captured via diversified cameras surrounding an object. Thus, it is an urgent task to accomplish sufficient encoding to be compatible with incoming bandwidth demands, while achieving a recommended 3DV reception performance. In the 3DV compression framework, the lost macro-blocks (MBs) might propagate into the following frames and the adjoining views. Therefore, it is obligatory to avoid error propagation by concealing the corrupted MBs at the decoder through the utilization of appropriate post-processing error concealment (EC) techniques. The existing EC algorithms fundamentally exploit the temporal, inter-view, and spatial matching within the 3DV frames and views to reconstruct the disparity vectors (DVs) and motion vectors (MVs) of the corrupted MBs. Unluckily, in the state of high severe corruptions and heavily erroneous MBs, these concealment algorithms are predominantly unreliable and might give unreliable 3DV quality. Thence, in this work, we suggest the utilization of the outer block boundary matching algorithm to estimate the MVs and the directional interpolation EC algorithm to estimate the DVs of the erroneous MBs. After that, the Bayesian Kalman filter (BKF) is employed because of its efficiency to filter out the inherent errors in the previously predicted DVs and MVs to accomplish better 3D video performance. Experimental results on standard 3DV sequences demonstrate that the suggested BKF-based EC scheme is more powerful with heavy losses. It subjectively and objectively outperforms the traditional concealment techniques at severely random and bursty packet loss rates (PLRs).

Distributed Video Coding (DVC) is a new paradigm for video compression based on the information theoretical results of Slepian and Wolf (SW), and Wyner and Ziv (WZ). While conventional coding has a rigid complexity allocation as most of the complex tasks are performed at the encoder side, DVC enables a flexible complexity allocation between the encoder and the decoder. The most novel and interesting case is low complexity encoding and complex decoding, which is the opposite of conventional coding. While the latter is suitable for applications where the cost of the decoder is more critical than the encoder's one, DVC opens the door for a new range of applications where low complexity encoding is required and the decoder's complexity is not critical. This is interesting with the deployment of small and battery-powered multimedia mobile devices all around in our daily life. Further, since DVC operates as a reversed-complexity scheme when compared to conventional coding, DVC also enables the interesting scenario of low complexity encoding and decoding between two ends by transcoding between DVC and conventional coding. More specifically, low complexity encoding is possible by DVC at one end. Then, the resulting stream is decoded and conventionally re-encoded to enable low complexity decoding at the other end. Multiview video is attractive for a wide range of applications such as free viewpoint television, which is a system that allows viewing the scene from a viewpoint chosen by the viewer. Moreover, multiview can be beneficial for monitoring purposes in video surveillance. The increased use of multiview video systems is mainly due to the improvements in video technology and the reduced cost of cameras. While a multiview conventional codec will try to exploit the correlation among the different cameras at the encoder side, DVC allows for separate encoding of correlated video sources. Therefore, DVC requires no communication between the cameras in a multiview scenario. This is an advantage since communication is time consuming (i.e. more delay) and requires complex networking. Another appealing feature of DVC is the fact that it is based on a statistical framework. Moreover, DVC behaves as a natural joint source-channel coding solution. This results in an improved error resilience performance when compared to conventional coding. Further, DVC-based scalable codecs do not require a deterministic knowledge of the lower layers. In other words, the enhancement layers are completely independent from the base layer codec. This is called the codec-independent scalability feature, which offers a high flexibility in the way the various layers are distributed in a network. This thesis addresses the following topics: First, the theoretical foundations of DVC as well as the practical DVC scheme used in this research are presented. The potential applications for DVC are also outlined. DVC-based schemes use conventional coding to compress parts of the data, while the rest is compressed in a distributed fashion. Thus, different conventional codecs are studied in this research as they are compared in terms of compression efficiency for a rich set of sequences. This includes fine tuning the compression parameters such that the best performance is achieved for each codec. Further, DVC tools for improved Side Information (SI) and Error Concealment (EC) are introduced for monoview DVC using a partially decoded frame. The improved SI results in a significant gain in reconstruction quality for video with high activity and motion. This is done by re-estimating the erroneous motion vectors using the partially decoded frame to improve the SI quality. The latter is then used to enhance the reconstruction of the finally decoded frame. Further, the introduced spatio-temporal EC improves the quality of decoded video in the case of erroneously received packets, outperforming both spatial and temporal EC. Moreover, it also outperforms error-concealed conventional coding in different modes. Then, multiview DVC is studied in terms of SI generation, which differentiates it from the monoview case. More specifically, different multiview prediction techniques for SI generation are described and compared in terms of prediction quality, complexity and compression efficiency. Further, a technique for iterative multiview SI is introduced, where the final SI is used in an enhanced reconstruction process. The iterative SI outperforms the other SI generation techniques, especially for high motion video content. Finally, fusion techniques of temporal and inter-view side informations are introduced as well, which improves the performance of multiview DVC over monoview coding. DVC is also used to enable scalability for image and video coding. Since DVC is based on a statistical framework, the base and enhancement layers are completely independent, which is an interesting property called codec-independent scalability. Moreover, the introduced DVC scalable schemes show a good robustness to errors as the quality of decoded video steadily decreases with error rate increase. On the other hand, conventional coding exhibits a cliff effect as the performance drops dramatically after a certain error rate value. Further, the issue of privacy protection is addressed for DVC by transform domain scrambling, which is used to alter regions of interest in video such that the scene is still understood and privacy is preserved as well. The proposed scrambling techniques are shown to provide a good level of security without impairing the performance of the DVC scheme when compared to the one without scrambling. This is particularly attractive for video surveillance scenarios, which is one of the most promising applications for DVC. Finally, a practical DVC demonstrator built during this research is described, where the main requirements as well as the observed limitations are presented. Furthermore, it is defined in a setup being as close as possible to a complete real application scenario. This shows that it is actually possible to implement a complete end-to-end practical DVC system relying only on realistic assumptions. Even though DVC is inferior in terms of compression efficiency to the state of the art conventional coding for the moment, strengths of DVC reside in its good error resilience properties and the codec-independent scalability feature. Therefore, DVC offers promising possibilities for video compression with transmission over error-prone environments requirement as it significantly outperforms conventional coding in this case.

In this paper, we propose a joint temporal error control (JTEC) method for H.264 which combines RDO-based macroblock (MB) classification at the encoder and adaptive partition size error concealment at the decoder. The encoder classifies the MBs by evaluating the sensitivity of the MBs as the RD cost between the concealment error and the bits needed for the additional motion information. Additional motion information can be transmitted for the error sensitive MBs such as the original motion vector or motion vector index. The decoder utilizes the additional motion information for the sensitive blocks. Non-sensitive blocks are concealed by the Adaptive Partition Size (APS) method. APS selects the best partition mode for error concealment by minimizing the Weighted Double-Side External Boundary Matching Error (WDS-EBME), which jointly measures the inter-MB boundary discontinuity, inter-partition boundary discontinuity and intra-partition block artifacts in the recovered MB. A progressive concealment method is developed for the 4x4 partition mode.

Region-based error concealment of right-view frames for stereoscopic video transmission

We present a region of interest (ROI) based rate control for H.263 compatible video conferencing. A face detection and tracking scheme with very low complexity is proposed for segmentation. By analyzing quadratic rate models in frame layer, video object plane (VOP) layer, and macroblock (MB) layer extracted from the test data, a quadratic rate model at the MB layer with a modified physical meaning is proposed to improve the model accuracy. The basic idea is to use a group of uncoded MBs in the current VOP instead of individual MBs to update model parameters. A joint VOP layer and MB layer rate control algorithm is proposed. The VOP layer rate control assigns target bit rate for each VOP based on the coding complexity and visual importance, and determines an average quantization parameter (QP) for each VOP. Some new features of MB layer rate control are designed to utilize both average statistics of a VOP and individual statistics of MB together. The performance is compared with conventional TMN8 and object-based VM8 rate control, better peak SNR (PSNR) for ROI, and more accurate rate control can be achieved for various video sequences. The proposed rate control algorithm can be extended for H.264 ROI-based scalable video coding.

Both perceptually and statistically, compressed video with large or disordered motion is sensitive to errors. In this paper, we propose a novel fast error-resilient video coding scheme, which is based on significant macroblock (MB) determination and protection. The scheme takes three impact factors (inter-block mode, motion vector difference and SAD value) to build a statistical model. The model takes error concealment (EC) into consideration in advance and generates several parameters for further significant degree (SD) evaluation for MBs. During encoding, we build an SD table for each frame based on the parameters and pick up those MBs with the largest SD values as significant MBs (SMBs). Few additional computations are induced into SMB determination, thus make our scheme practical in real time video coding scenarios. Simulations show that the scheme has an acceptable SMB determination accuracy and the corresponding protection method can prevent errors effectively.

Error concealment is a well-known technique to improve reproduced picture quality in cases which parts of the coded picture are not available at a decoder for reconstruction. The conventional error concealment methods generally make use of the correlation between a damaged marcoblock and its adjacent macroblocks in the same frame and/or the previous frame. However, performance results of concealment methods are different each other and dependent on how much damaged frames and macroblocks are correlated to surrounding ones temporally and spatially. Moreover, most of them can not conceal effectively the consecutively damaged frames or macro blocks caused by wireless bursty packet loss features, since motion vectors and correlation information between the damaged images are already destroyed. The proposed error concealment aware error resilient video coding is a new method which has an intelligent error concealment selector by using pre-estimation technique of the possibly damaged motion vectors at the encoder. The method recommends the best error concealment per macroblock by transmitting additionally its error concealment selection codes to the decoder. The chosen error concealment helps to conceal the damaged MB at the decoder even when losses of frames are bursty. The experimental results show the substantial effectiveness to improve video quality at the cost of a little bit overhead at the transmitted bit streams.

Historically, multiple description coding (MDC) and post-processing error concealment (ECN) algorithms for error-resilient video transmission have evolved separately. Although multiple description coding provides a way to make the video quality degrade gracefully in the presence of packet losses, the resulting quality may be unsatisfactory when only one description is used to reconstruct the signal. This paper proposes a joint description transform coding (MDTC) and statistical error concealment algorithm that adopts a maximum a posteriori (MAP) estimation approach to compensate for the loss of descriptions. This coordination results in significant improvements in reconstructed video quality over independent use of MDTC.

We consider the problem of delivering region of interest (ROI)-coded mobile video streams using limited radio resources. Under the conditions of limited bandwidth and time-varying channel status, the goal is to optimize the transmission latency, while ensuring the quality of the ROI parts. Multi-homing support enables the terminals to establish multiple connections for transmission performance improvement. In this paper, we propose a novel framework for ROI-based video transmission in heterogeneous wireless networks with multi-homed terminals. The framework contains the modules of ROI detector and frame splitter, where macroblocks are categorized based on ROI detection and encapsulated into transforming units. It also includes a channel monitor that keeps track of the status of each communication path and sends feedback signals to the streaming controller for packet-scheduling control; a deep learning method is proposed for channel status prediction. To address the delivery problem, we propose a scheduling approach based on the formulated network model and the rate-distortion model. The scheduling method makes a tradeoff between the transmission delay and the distortion. It also guarantees that packets with ROI content are delivered on paths with sufficient bandwidths and low loss rates. Through comparisons with other scheduling methods, we find that the proposed scheme outperforms the other scheduling methods in terms of improving the quality (peak signal-to-noise ratio), balancing the end-to-end delay, and maintaining the playback fluency.

This paper presents an evaluation of the latest MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) for video streaming applications using best effort protocols. LCEVC is a new video standard by MPEG, which enhances any base codec through an additional low bitrate stream, improving both video compression efficiency and and transmission. However, there is an interplay between packetization, packet loss visibility, choice of codec and video quality, which implies that prior studies with other codecs may be not as relevant. The contributions of this paper is, therefore in twofold: It evaluates the compression performance of LCEVC and then the impact of packet loss on its video quality when compared to H.264 and HEVC.The results from this evaluation suggest that, regarding compression, LCEVC outperformed its base codecs, overall in terms average encoding bitrate savings when using the constant rate factor (CRF) rate control. For example at a CRF of 19, the average encoding bitrate was reduced by 18.7% and 15.8% when compared with the base H.264 and HEVC codecs respectively. Furthermore, LCEVC produced better visual quality across the packet loss range compared to its base codecs and the quality only started to decrease once packet loss exceeded 0.8-1%, and decreases at a slower pace compared to its equivalent base codecs. This suggests that the LCEVC enhancement layer also provides error concealment. The results presented in this paper will be of interest to those considering the LCEVC standard and expected video quality in error-prone environments

We propose an error concealment method for robust video transmission based on watermarking technique. In this method, the motion vector information is used as the watermark and embedded in the transform coefficients before quantization. A proper embedding algorithm is designed with the characteristics of the video coder taken into consideration, so that the embedding watermark has little influence on the video quality and the coding efficiency. At the decoder, the extracted watermark is used to effectively detect errors and restore the corrupted motion vectors. Then the restored motion vectors are utilized in error concealment. Simulation results demonstrate the advantages of the proposed method in error prone environment. The proposed method is applicable to all the block-transform based video coding schemes.

Esta tesis doctoral aborda el problema de la transmisión de vídeo, en sistemas de transmisión de televisión y vídeo bajo demanda en tiempo real y de videoconferencia, a través de redes IP, haciendo un énfasis especial cuando la conexión se establece entre dispositivos móviles. En la primera parte del trabajo se introduce la transmisión de video en redes IP y se hace especial incapie en la videoconferencia, pasando posteriormente a presentar el estado del arte en ambos ámbitos. Tratamos y estudiamos cuestiones relacionadas con la compresión de vídeo, la calidad del vídeo recibido, la calidad de experiencia (QoE), sistemas para la mejora de la QoE para la videoconferencia y también tecnologías, como Software Defined Networking (SDN), que nos permiten mejorar dinámicamente la configuración de los dispositivos de red para mejorar la QoE percibida por los usuarios finales. Posteriormente, pasamos a describir el diseño de la arquitectura y el protocolo para la transmisión de vídeo en redes IP, con distintos objetivos. Proponemos una arquitectura y un protocolo para mejorar la QoE de los usuarios en la transmisión de vídeo en Internet Protocol Television (IPTV). También lo hacemos para la transmisión de vídeo en Redes Heterogéneas usando HTML5. Por último proponemos un algoritmo y protocolo para mejorar QoE de la transmisión de vídeo en redes empleadas para la monitorización ambiental, basándonos principalmente en un estudio fundamentado en la codificación en función del color predominante de las imágenes del vídeo. En el siguiente capítulo pasamos a proponer una nueva arquitectura y un protocolo para la mejora de la calidad de experiencia de extremo a extremo (E2E QoE) de los usuarios conectados a una videoconferencia. Definimos el proceso del sistema, la máquina de estados finita y el algoritmo necesario para su correcto funcionamiento. Por último, presentamos las distintas pruebas experimentales que se han realizado para estudiar y determinar los mejores sistemas de codificación, y su comportamiento al utilizar tecnologías como SDN en redes de dispositivos móviles, en Redes Heterogéneas usando HTML5 y en videoconferencia.

Multilevel region of interest guided bit allocation for multiview video coding

Capturing or acquiring video data from advanced gadgets, like cell phone cameras, smart phones, camcorders, etc., is very recent and up growing trend. With the advancement of the video camera as the integrated part of these multimedia devices generates necessity for video stabilization. Digital video stabilization or video frame alignment techniques are used to improve quality of input video by removing the undesired camera motions jitters; undesirable jiggles etc. Many video stabilization techniques are developed till now but only fast and low computational complexity approaches can be applied on the embedded devices. The paper presents the review of fast block matching local motion vector estimation and alignment algorithm. Effectiveness of the algorithm is shown by quantitative evaluation. The proposed work will also help to boost the performance of various error concealment, error resilience and video quality improvement techniques in H.264/HEVC video codec.

Fast Interframe mode decision algorithm based on mode mapping and MB activity for MPEG-2 to H.264/AVC transcoding