Performance of Spatial and Temporal Error Concealment Method for 3D DWT Video Coding in Packet Loss Channel

In this paper, we propose a novel spatial image error concealment (EC) method based on deep neural network. Considering that the natural images have local correlation and non-local self-similarity, we use the local information to predict the missing pixels and the non-local information to correct the predictions. The deep neural network we utilize can be divided into two parts: the prediction part and the auto-encoder (AE) part. The first part utilizes the local correlation among pixels to predict the missing ones. The second part extracts image features, which are used to collect similar samples from the whole image. In addition, a novel adaptive scan order based on the joint credibility of the support area and reconstruction is also proposed to alleviate the error propagation problem. The experimental results show that the proposed method can reconstruct corrupted images effectively and outperform the compared state-of-the-art methods in terms of objective and perceptual metrics.

Error concealment at a decoder is an efficient method to reduce the degradation of visual quality caused by channel errors. In this paper, we propose a novel spatio-temporal error concealment algorithm based on the spatial-temporal fading (STF) scheme which has been recently introduced. Although STF achieves good performance for the error concealment, several drawbacks including blurring still remain in the concealed blocks. To alleviate these drawbacks, in the proposed method, hybrid approaches with adaptive weights are proposed. First, the boundary matching algorithm and the decoder motion vector estimation which are well-known temporal error concealment methods are adaptively combined to compensate for the defect of each other. Then, an edge preserved method is utilized to reduce the blurring effects caused by the bilinear interpolation for spatial error concealment. Finally, two concealed results obtained by the hybrid spatial and temporal error concealment are pixel-wisely blended with adaptive weights. Experimental results exhibit that the proposed method outperforms conventional methods including STF in terms of the PSNR performance as well as subjective visual quality, and the computational complexity of the proposed method is similar to that of STF.

The deficiencies of traditional shape error concealment method based on Bezier curve were analyzed,which included the complex process of calculating additional control points and the influence of additional control points to accuracy.In this paper,an error concealment method was proposed based on cubic B-spline interpolation(CBI).The matrix formula of CBI was modified to smooth the object's boundary.The known boundary points were interpolated directly and no additional control points needed.Experimental results show that the proposed method is simple and easy implement,and is of great promise in practice.

Video steganography approach enables hiding chunks of secret information inside video sequences. The features of video sequences including high capacity as well as complex structure make them more preferable for choosing as cover media over other media such as image, text, or audio. Video steganography is a prominent as well as the evolving field in the information security domain and significant number of video steganography methods are proposed in recent years. This article provides a comprehensive review of video steganography methods proposed in the literature. This article initially reviews various raw domain-based video steganography methods. In particular, the raw domain-based methods include spatial domain approaches such as least significant bits (LSB), transform domain-based methods such as discrete wavelet transform, discrete cosine transform, etc. Furthermore, the article looks into various compressed domain steganography methods. A critical comparative analysis is included in the article to analyze and contrast the steganography methods proposed in the literature. A brief description of various evaluation matrices for video steganography methods is provided in this article. Moreover, a brief introduction to steganalysis and video steganalysis is provided. The article concludes with a discussion focused on the limitations and challenges of the video steganography methods. Further, a brief insight into future directions in video steganography systems is provided.

An Adaptive Spatio-Temporal Error Concealment (ASTEC) scheme based on H.264/AVC is proposed in the paper to improve the quality of the rebuilt frame in H.264/AVC decoder side. Combining the spatial error concealment method with the temporal one, a new scheme is proposed to conceal an erroneous macro block according to its encoded mode. The temporal error concealment scheme adopts the weighted Boundary Matching Algorithm to recover the erroneous macro block. The spatial error concealment scheme recovers the erroneous macro block by exploiting the directional interpolation which subdivides the macro block according to the number and the direction of edges going through it. Experiments show that the proposed concealment scheme can restrain the diffusion of erroneous macro block and improve the quality of the rebuilt frame subjectively as well as objectively.

In this paper a hybrid error concealment algorithm using linear interpolation is proposed. In the proposed hybrid algorithm, the selective motion field interpolation (SMFI) is employed to conceal the erroneous macroblock. The spatial and temporal boundary-matched errors are then used to check whether the SMFI conceals the erroneous macroblock properly. If the temporally recovered macroblock is reconstructed incorrectly, the spatial error concealment using linear interpolation is employed to conceal the damaged macroblock instead of SMFI.It can achieve better performance subjectively as well as objectively over other error concealment techniques such as AMV, FBBM and MFI. Simulation was conducted on the H.263 codec and the experimental results reveal that the proposed algorithm significantly outperforms other concealment techniques.

Compressed bitstreams are, in general, very sensitive to channel errors. For instance, a single bit error in a coded video bitstream may cause severe degradation on picture quality. When bit errors occur during transmission and cannot be corrected by an error correction scheme, error concealment is needed to conceal the corrupted image at the receiver. Error concealment algorithms attempt to repair damaged portions of the picture by exploiting both the spatial and the temporal redundancies in the received and reconstructed video signal. We discuss several temporal, spatial, and transform-domain error concealment techniques for MPEG coded pictures, and propose a new scheme based on directional interpolation. We also compare the performance of these techniques by computer simulation.

The H.264/AVC Multi-View Coding (MVC) compresses more than one viewpoint of a scene captured by different cameras. Redundancies between views and frames are used for inter-view and inra-view prediction ,respectively, in encoding and in Error Concealment (EC) in decoding. As MVC has high error sensitivity, EC algorithms are developed to improve the quality of decoded frames. In this paper, efficient multi-view EC algorithm is proposed for intra MBs and inters MBs exploiting the spatial, inter-view and inra-view information of neighboring MBs. For Intra MBs, Spatial-Inter View (SIV) algorithm is proposed. For inter MBs, the proposed algorithm is adaptively changed according to size of lost MB. In addition, the algorithm dynamically candidates MBs according to the current used view in EC operation. Then the algorithm self generates more candidate MBs depending on best matched MB to get higher matched MB. It is shown that the proposed algorithm leads to better subjective quality in addition to higher PSNR values. By applying the proposed algorithm, the gain is improved by up to 12.4 dB and 0.73 dB compared with the case of not applying any EC algorithms and the case of applying normal EC algorithm respectively at 22% Loss Rate (LR). Keywords-Multi view Coding, Spatial and Temporal Error Concealment, Outer Boundary Matching Algorithm.

In spatial error concealment (SEC), methods like bilinear interpolation (BI) and directional interpolation (DI) are commonly used to estimate the missing pixel values resulting from losses occurring in video streams. Despite being able to preserve spatial smoothness, BI produces a blurring effect and is unable to preserve structural information. DI produces spurious edges in regions with no strong edges, resulting in visible artefacts. In this paper, we propose a SEC algorithm that addresses the above drawbacks by formulating a weighted sum of candidate macroblocks produced from DI and BI, with weights derived adaptively through local information. We demonstrate that the proposed algorithm offers visual improvements over both DI based algorithms and the SEC algorithm based on BI in the H.264/AVC reference software JM 12.0. Most importantly, this unique approach preserves edge information and spatial smoothness in the error concealed macroblock due to the integration of both BI and DI.

In information and communication networks an essential part of rapidly growing traffic is occupied by streaming video and specialized video data designed for comprehensive analysis by the methods of computer vision. In information and communication networks video compression methods are subject to strict requirements of coding/decoding speed for ensuring the qualitative operation in realtime mode. The paper offers the palette method for effective digital video coding, which uses adaptive quantization of video frames in the color space and their representation in the single byte palette graphical format. The compression level of the proposed method constitutes, on average, 2.98 times. The developed method for video coding ensures the preservation of good quality of visual perception with a PSNR value of 46.1 dB. The encoded video can be displayed using existing software, i.e. there is no need to change the decoder

This paper presents an efficient error concealment method for multiview video plus depth (MVD) delivery over error-prone channels based on the view synthesis information. Specifically, we propose an enhanced temporal error concealment (TEC) method by estimating the probable partition and missing MVs of a corrupted macroblock (MB) using the synthesized information. Furthermore, the view synthesis error concealment (VSEC) approach that directly uses the synthesized pixels to conceal the corrupted MBs is also adaptively utilized together with the TEC approach. Experimental results have shown the effectiveness of the proposed method compared with the existing methods in terms of both objective and subjective measures.

H.264 adopts new coding tools such as intra-prediction, variable block size, motion estimation with quarter-pixel-accuracy, loop filter, etc. The adoption of these tools enables an H.264-coded bitstream to have more information compared with previous standards. In this paper we proposed an effective spatial error concealment method with low complexity. Among the information included in an H.264-coded bitstream, we use prediction modes of intra-blocks for recovering a damaged block. This is because a prediction direction in each prediction mode is highly correlated to the edge direction. We first estimate the edge direction of a damaged block using prediction modes of intra-blocks adjacent to a damaged block and classify the area inside a damaged block into the edge and the flat area. And then our method recovers pixel values in the edge area using edge-directed interpolation, and recovers pixel values in the flat area using weighted interpolation. Simulation results show the proposed method yields better video quality than conventional approaches.

The increasing processing speed of computers has made software-based video coding possible. Several reports have been published on increasing the processing speed of such video coding through improved software development techniques, and on increasing video coding speed by efficiently omitting the video coding process itself. However, the actual speed of software-based video coding depends on the implementation environments. Accordingly, under some environments of lower processing speed, the processor can not encode video data in real time, even if real-time coding is necessary. This has created a need to control a video coding process with depending on the processing speed. To address this need, we propose a new scheme for controlling motion estimation. This scheme, which we call “time adaptive motion estimation” (TAME), is based on the Successive Elimination Algorithm (SEA) method. It enables motion estimation to be completed within a pre-determined target time. Computer simulation results demonstrate that the proposed method achieves nearly the same quality as in the case where coding time is unlimited.

360 Video has become popular in recent years, as commercial interests in deploying Virtual Reality (VR) applications rise. This type of video is usually captured using multi-camera arrays, such as the GoPro Omni camera rig. After separate video streams are captured from multiple cameras, image stitching is applied to obtain a spherical representation of the scene, which spans 360 degrees horizontally and 180 degrees vertically, hence the name 360 video. In the existing workflow of 360 spherical video coding, the 360 video is projected onto the 2D plane with a projection format, such as equirectangular (ERP), cubemap (CMP), equal-area (EAP), octahedron (OHP), etc. Most, if not all, of the currently available 360 video content are provided in ERP format defined in longitude and latitude. Projection format conversion may be performed to convert the native ERP format to another format before coding is applied. Some projection formats contain more than one face, for example, CMP projects the sphere onto a cube of six faces or OHP projects the sphere onto an octahedron of eight faces. For these multi-face projection formats, the faces are packed onto a 2D rectangular picture with a frame packing method. For example, the six faces of CMP can be packed with 4x3 configuration, or 3x2 configuration. Finally, the frame packed picture is coded as a 2D conventional video. Existing video codecs are designed only considering conventional 2D video captured on a plane. When motion compensated prediction uses any samples outside of a reference picture's padding will be performed by simply copying the sample values from the picture boundaries. This repetitive padding method is referred as conventional 2D padding method, which is widely used in video coding standards such as H.264, High Efficiency Video Coding (HEVC). However, a 360 video encompasses video information on the whole sphere, and thus intrinsically has a cyclic property. When considering this cyclic property, the reference pictures of a 360 video no longer have boundaries, as the information they contain is all wrapped around a sphere. This cyclic property holds regardless of which projection format or which frame packing is used to represent the 360 video on a 2D plane. The paper presents a new geometry padding method for motion compensated prediction in 360 video coding. Unlike the conventional padding method for 2D video coding, the proposed geometry padding method extends samples outside of a 2D picture's boundaries using neighboring samples on the sphere. The geometry projection format is considered when performing padding. The corresponding sample outside of a face's boundary (which may come from another side in the same face or from another face), is derived with rectilinear projection. Each face is extended with geometry padding separately. When visualized, the extended faces using geometry padding show continuous texture representing natural extension of the texture inside the face. The proposed geometry padding method is implemented in the HEVC reference software HM-16.12 for the ERP and CMP projection formats. In the simulation, a total of sixteen 4K ERP video and eight 8K ERP video are used. For 8K ground truth 8K video, they are converted to 4K video in ERP and CMP projection formats, coded, and converted back to reconstructed 8K video in ERP format. For 4K ground truth video, they are directly coded in 4K ERP, or converted to CMP consisting of 75% of effective samples, coded, and converted back to reconstructed 4K video in ERP format. Then, the end-to-end spherical PSNR (S-PSNR) is calculated between the original 8K or 4K and the reconstructed 8K or 4K ERP video. BD-rate is calculated between the reference unmodified HEVC, which uses the conventional padding method, and HEVC modified with the proposed geometry padding method. Simulation results showed that geometry padding performs better. For 8K sequences, the proposed geometry padding gives on average luma (Y) BD-rate reduction of 0.3% for ERP and 0.8% for CMP, for 4K sequences, the proposed geometry padding gives on average Y BD-rate reduction of 0.2% for ERP and 1.0% for CMP. Comparing the gains in ERP format with the gains in CMP format, the improvement for CMP is larger. This is because CMP has six faces, therefore the improvement from geometry padding method affects more out-of-boundary samples. The proposed geometry padding method is also especially effective for sequences with fast motion. For example, it achieves BD rate reductions of 4.3%, 2.7%, 1.9%, and 2.5% for Glacier, Chairlift, Sb_in_lot, and Driving, respectively. These four sequences are all captured using moving cameras and have fast moving objects. As a result, the sequences contain a lot of across-the-face-boundary motion which can benefit from improved padding method. Detailed simulation results can be found in JVET contribution JVET-D0075 available at http://phenix.int-evry.fr/jvet/doc_end_user/documents/4_Chengdu/wg11/JVET-D0075-v3.zip.

Distributed Video Coding (DVC) is based on a new paradigm in coding, which has received many interests recently. Among other features, DVC is said to provide better error resiliency. This paper proposes a hybrid spatial and temporal error concealment scheme to conceal errors in Wyner-Ziv (WZ) frames. We first use a spatial concealment based on edge directed filter. This step is exploited to improve the performance of subsequent temporal concealment. An enhanced temporal concealment based on motion compensated temporal interpolation is proposed, including motion vector refinement and smoothing, optimal compensation mode selection, and a new matching criterion for motion estimation. Simulation results show that the objective qualities as well as the perceptual qualities of the corrupted sequences are significantly improved by the hybrid error concealment, outperforming both spatial and temporal concealments alone.