Edge Weighted Spatio-Temporal Search for Error Concealment

The emerging standard of High Efficiency Video Coding (HEVC) provides a 50% improved compression performance compared to its predecessor H.264/AVC at the expense of increasing its computational complexity which is due to a large number of modes involved in the decision process where it would increase encoding time and provide hardware limitations. In this paper, the reduction of intra-coding complexity is proposed by utilization of Rate-Distortion Optimization (RDO) in preselecting a mode with the best RD cost, among all 35 intra-modes. Two cost functions – Sum of Absolute Difference (SAD) and Sum of Absolute Transform Difference (SATD, available in the HEVC Test Model (HM) Reference Software, are determined by making encoder decisions. This paper compares the performance and complexity cost of SAD only for intra-mode, with SATD-based intra-mode by modifying the HM reference encoder through forced selection of the mentioned modes. Results show that SAD consistently produces the least encoding time among all modes, along with sensible values of bitrate. Furthermore, error concealment was introduced in decoding the video sequences with the use of IEEE 802.22 Standard which only allowed power gains resulting to packet error rates (PER) ranging from 1% to 30% for the analysis on the behavior of each video sequence in an environment where error is introduced.

The sum of absolute difference (SAD) is generally adopted as a cost function in motion estimation (ME) and temporal error concealment (TEC) algorithm owing to its efficiency. The hardware architecture of SAD also consumes considerable power dissipation in video codec chip. Hence, the switching-activity analysis on SAD is quite essential from algorithm/architecture perspectives. This work develops the estimation formulas for switching-activity dedicated for SAD engine according to probability theory. The experiment results reveal that the probability error rate (PER) of the SAD engine is as minor as 5.61%. Consequently, this leads to a precise switching-activity estimation of the SAD-based algorithm/architecture for video signal processing.

Consecutive corrupted MBs or slice errors are commonly seen in modern video transmission systems. Temporal error concealment is an effective approach to reduce the impact of errors. Conventional temporal error concealment techniques recover slice errors on a MB basis. We propose a new novel temporal scheme for slice error concealment based on a size-adaptive region basis. Size-adaptive region boundary matching can significantly reduce the misalignments or side mismatches that are commonly seen in MB-based approaches. Instead of using MSE or MAE as the matching criteria, We propose a matching criterion based on structural similarity (SSIM). Our proposed matching criterion can achieve better structural alignments and better visual perceptual quality. We also propose a method to detect motion vector outliers at the last stage of the error concealment scheme, which can further improve the visual quality of the concealed videos. Our proposed scheme conceals slice errors with good accuracy as well as low computation complexity, and it can be applied for real-time applications.

Error concealment at a decoder is an efficient method to reduce the degradation of visual quality caused by channel errors. In this paper, we propose a novel spatio-temporal error concealment algorithm based on the spatial-temporal fading (STF) scheme which has been recently introduced. Although STF achieves good performance for the error concealment, several drawbacks including blurring still remain in the concealed blocks. To alleviate these drawbacks, in the proposed method, hybrid approaches with adaptive weights are proposed. First, the boundary matching algorithm and the decoder motion vector estimation which are well-known temporal error concealment methods are adaptively combined to compensate for the defect of each other. Then, an edge preserved method is utilized to reduce the blurring effects caused by the bilinear interpolation for spatial error concealment. Finally, two concealed results obtained by the hybrid spatial and temporal error concealment are pixel-wisely blended with adaptive weights. Experimental results exhibit that the proposed method outperforms conventional methods including STF in terms of the PSNR performance as well as subjective visual quality, and the computational complexity of the proposed method is similar to that of STF.

Distributed Video Coding (DVC) is a new paradigm for video compression based on the information theoretical results of Slepian and Wolf (SW), and Wyner and Ziv (WZ). While conventional coding has a rigid complexity allocation as most of the complex tasks are performed at the encoder side, DVC enables a flexible complexity allocation between the encoder and the decoder. The most novel and interesting case is low complexity encoding and complex decoding, which is the opposite of conventional coding. While the latter is suitable for applications where the cost of the decoder is more critical than the encoder's one, DVC opens the door for a new range of applications where low complexity encoding is required and the decoder's complexity is not critical. This is interesting with the deployment of small and battery-powered multimedia mobile devices all around in our daily life. Further, since DVC operates as a reversed-complexity scheme when compared to conventional coding, DVC also enables the interesting scenario of low complexity encoding and decoding between two ends by transcoding between DVC and conventional coding. More specifically, low complexity encoding is possible by DVC at one end. Then, the resulting stream is decoded and conventionally re-encoded to enable low complexity decoding at the other end. Multiview video is attractive for a wide range of applications such as free viewpoint television, which is a system that allows viewing the scene from a viewpoint chosen by the viewer. Moreover, multiview can be beneficial for monitoring purposes in video surveillance. The increased use of multiview video systems is mainly due to the improvements in video technology and the reduced cost of cameras. While a multiview conventional codec will try to exploit the correlation among the different cameras at the encoder side, DVC allows for separate encoding of correlated video sources. Therefore, DVC requires no communication between the cameras in a multiview scenario. This is an advantage since communication is time consuming (i.e. more delay) and requires complex networking. Another appealing feature of DVC is the fact that it is based on a statistical framework. Moreover, DVC behaves as a natural joint source-channel coding solution. This results in an improved error resilience performance when compared to conventional coding. Further, DVC-based scalable codecs do not require a deterministic knowledge of the lower layers. In other words, the enhancement layers are completely independent from the base layer codec. This is called the codec-independent scalability feature, which offers a high flexibility in the way the various layers are distributed in a network. This thesis addresses the following topics: First, the theoretical foundations of DVC as well as the practical DVC scheme used in this research are presented. The potential applications for DVC are also outlined. DVC-based schemes use conventional coding to compress parts of the data, while the rest is compressed in a distributed fashion. Thus, different conventional codecs are studied in this research as they are compared in terms of compression efficiency for a rich set of sequences. This includes fine tuning the compression parameters such that the best performance is achieved for each codec. Further, DVC tools for improved Side Information (SI) and Error Concealment (EC) are introduced for monoview DVC using a partially decoded frame. The improved SI results in a significant gain in reconstruction quality for video with high activity and motion. This is done by re-estimating the erroneous motion vectors using the partially decoded frame to improve the SI quality. The latter is then used to enhance the reconstruction of the finally decoded frame. Further, the introduced spatio-temporal EC improves the quality of decoded video in the case of erroneously received packets, outperforming both spatial and temporal EC. Moreover, it also outperforms error-concealed conventional coding in different modes. Then, multiview DVC is studied in terms of SI generation, which differentiates it from the monoview case. More specifically, different multiview prediction techniques for SI generation are described and compared in terms of prediction quality, complexity and compression efficiency. Further, a technique for iterative multiview SI is introduced, where the final SI is used in an enhanced reconstruction process. The iterative SI outperforms the other SI generation techniques, especially for high motion video content. Finally, fusion techniques of temporal and inter-view side informations are introduced as well, which improves the performance of multiview DVC over monoview coding. DVC is also used to enable scalability for image and video coding. Since DVC is based on a statistical framework, the base and enhancement layers are completely independent, which is an interesting property called codec-independent scalability. Moreover, the introduced DVC scalable schemes show a good robustness to errors as the quality of decoded video steadily decreases with error rate increase. On the other hand, conventional coding exhibits a cliff effect as the performance drops dramatically after a certain error rate value. Further, the issue of privacy protection is addressed for DVC by transform domain scrambling, which is used to alter regions of interest in video such that the scene is still understood and privacy is preserved as well. The proposed scrambling techniques are shown to provide a good level of security without impairing the performance of the DVC scheme when compared to the one without scrambling. This is particularly attractive for video surveillance scenarios, which is one of the most promising applications for DVC. Finally, a practical DVC demonstrator built during this research is described, where the main requirements as well as the observed limitations are presented. Furthermore, it is defined in a setup being as close as possible to a complete real application scenario. This shows that it is actually possible to implement a complete end-to-end practical DVC system relying only on realistic assumptions. Even though DVC is inferior in terms of compression efficiency to the state of the art conventional coding for the moment, strengths of DVC reside in its good error resilience properties and the codec-independent scalability feature. Therefore, DVC offers promising possibilities for video compression with transmission over error-prone environments requirement as it significantly outperforms conventional coding in this case.

This paper presents an efficient error concealment method for multiview video plus depth (MVD) delivery over error-prone channels based on the view synthesis information. Specifically, we propose an enhanced temporal error concealment (TEC) method by estimating the probable partition and missing MVs of a corrupted macroblock (MB) using the synthesized information. Furthermore, the view synthesis error concealment (VSEC) approach that directly uses the synthesized pixels to conceal the corrupted MBs is also adaptively utilized together with the TEC approach. Experimental results have shown the effectiveness of the proposed method compared with the existing methods in terms of both objective and subjective measures.

Compressed video bit-streams are extremely sensitive to packet loss over error-prone channels. Error concealment (EC) has been considered as one of error control techniques to improve the reconstructed picture quality against transmission errors. However, EC methods show poor image reconstruction performance due to unavailable scene change information of the video sequence, especially when an abrupt scene change occurs. In this paper, we propose an effective EC method based on scene change detection algorithm (SCDA), which provides information to decide whether spatial or temporal EC is better to be used for intra and inter frames respectively. The simulation results show that the proposed method highly improves the subjective quality of incorrectly decoded frames and obtains an average gain of 0.7dB compared with the H.264/AVC Joint Model reference software.

The H.264/AVC Multi-View Coding (MVC) compresses more than one viewpoint of a scene captured by different cameras. Redundancies between views and frames are used for inter-view and inra-view prediction ,respectively, in encoding and in Error Concealment (EC) in decoding. As MVC has high error sensitivity, EC algorithms are developed to improve the quality of decoded frames. In this paper, efficient multi-view EC algorithm is proposed for intra MBs and inters MBs exploiting the spatial, inter-view and inra-view information of neighboring MBs. For Intra MBs, Spatial-Inter View (SIV) algorithm is proposed. For inter MBs, the proposed algorithm is adaptively changed according to size of lost MB. In addition, the algorithm dynamically candidates MBs according to the current used view in EC operation. Then the algorithm self generates more candidate MBs depending on best matched MB to get higher matched MB. It is shown that the proposed algorithm leads to better subjective quality in addition to higher PSNR values. By applying the proposed algorithm, the gain is improved by up to 12.4 dB and 0.73 dB compared with the case of not applying any EC algorithms and the case of applying normal EC algorithm respectively at 22% Loss Rate (LR). Keywords-Multi view Coding, Spatial and Temporal Error Concealment, Outer Boundary Matching Algorithm.

Compressed bitstreams are, in general, very sensitive to channel errors. For instance, a single bit error in a coded video bitstream may cause severe degradation on picture quality. When bit errors occur during transmission and cannot be corrected by an error correction scheme, error concealment is needed to conceal the corrupted image at the receiver. Error concealment algorithms attempt to repair damaged portions of the picture by exploiting both the spatial and the temporal redundancies in the received and reconstructed video signal. We discuss several temporal, spatial, and transform-domain error concealment techniques for MPEG coded pictures, and propose a new scheme based on directional interpolation. We also compare the performance of these techniques by computer simulation.

The full search block-matching algorithm is the simplest, but computationally very intensive approach. In recent years there was substantial progress in block motion estimation algorithms. Important milestones on this path were such algorithms as two-dimensional logarithmic search, three-step search, four step search, and diamond search. All these methods try to minimize the amount of search points applying sum of absolute differences (SAD) or equivalent metrics for each point. There were some works that studied partial SAD (PSAD) but mostly concentrated on a constant decimation factor. What we tried to study in this work is the performance of one of the best block matching search algorithms in combination with adaptive PSAD as a matching metric. The idea is that we use original motion estimation based on spatio-temporal correlation method, but instead of using SAD as a matching metric we use PSAD with adaptively chosen decimation factor. Our simulation results show that for high motion sequences PSNR degradation between full search and the proposed method was around 0.1-0.7 dB. The computational complexity reduction of 650-1700 times (compared with the full search) and 9 times (compared to the original method) is pretty big and may be well worth this video quality decrease. In the case of more static sequences PSNR degradation between full search and the original motion estimation method was around 0 dB. When we compare the original method with the proposed one the degradation increases to 0.1 dB. The computational complexity reduction was around 1600-1700 times (compared with the full search) and 5-7 times (compared to the original method).

H.264 adopts new coding tools such as variable block size, quarter-pixel-accuracy motion estimation/compensation, multiple reference frames, loop filter, etc. The adoption of these tools enables a macroblock to have more information compared with previous standards. In H.264 each macroblock can have up to sixteen motion vectors, four reference frames, and a mode of the macroblock, all of which can be used for temporal error concealment. The H.264 error concealment in the informative part, however, uses a similar method to prior coding standards that consider only motion vectors of macroblocks adjacent to the lost macroblock. In this paper, we propose an effective temporal error concealment algorithm for H.264-coded video. It uses not only motion vectors and reference frames but the modes of macroblocks adjacent to the lost macroblock as well. Depending on the modes of neighboring macroblocks, each lost macroblock is concealed on the basis of different block sizes. Simulation results show the proposed method yields better video quality than conventional approaches.

This paper presents a technique for high-accuracy correspondence search between two rectified images using 1D phase-only correlation (POC). The correspondence search between stereo images can be reduced to 1D search through image rectification. However, we usually employ block matching with 2D rectangular image blocks for finding the best matching point in the 1D search. We propose the use of 1D POC (instead of 2D block matching) for stereo correspondence search. The use of 1D POC makes possible significant reduction in computational cost without sacrificing reconstruction accuracy compared with the 2D POC-based approach. Also, the resulting reconstruction accuracy is much higher than those of conventional stereo matching techniques using SAD (sum of absolute differences) and SSD (sum of squared differences) combined with sub-pixel disparity estimation.

Motion estimation (ME) is the most critical component of a video coding system, and it also dominates the major part of computation complexity and memory bandwidth. For H.264/AVC integer motion estimation (IME), this paper presents a novel memory-access and computation efficient full-search block-matching hardware architecture. With the highest level of on-chip data reuse, one-access for off-chip reference pixels is achieved, and the off-chip memory bandwidth is thus minimized. By distributed data caching and virtual connection of reference picture boundaries, the data traffic scheduling is simple, regular and efficient. The computation engine employs a two-dimensional (2-D) systolic processor array to calculate the absolute differences in single-instruction multiple-data (SIMD) manner, and 2-D adder trees to sum up the absolute differences, all with 100% utilization. The proposed architecture fully supports variable block-size matching of H.264/AVC, and can produce 41 sums of absolute differences (SADs) for one search point every cycle without bubble. The architecture is described in parameterized design, and an implementation for standard-definition digital TV encoding applications is presented. Theoretical analysis and experimental results show that, the proposed architecture can achieve the minimum off-chip memory bandwidth and the maximum computational performance.

As the newest video coding standard, H.264/AVC adopts the high-efficiently predictive coding and variable length entropy coding to achieve high compression efficiency. On the other side, transmission errors become the major problem faced by video broadcasting service providers. Error concealment (EC) here is adopted to handle slices with huge conjunctive corrupted areas inside. Considering error propagation from corrupted slice to succeeding ones is the key factor affecting the video quality, this paper proposes a novel temporal EC scheme including the bi-direction motion vector (MV) retrieval method and an adaptive EC ordering basing on it. Background and motional steady shift part of slice will be given top and second priority, respectively. Combined with our proposed improved boundary matching algorithm (IBMA) which provides more accurate distortion function, experiments results show that our proposal achieves better performance under different error rate channel, compared with EC algorithm adopted in H.264 reference software.

A novel fast integer-pixel multi-pattern search algorithm is proposed in this paper. The cross search pattern (CSP) is used to get the coarse position of the best matching point, then small diamond search pattern (SDSP) or small square search pattern (SSSP) is chosen to locate the best matching point accurately according to the size of the partitioned block. In order to reduce the number of search points, the judging threshold is applied, which is determined by the compensation value and the sum of absolute difference (SAD) values of three adjacent blocks. Experimental results show that, compared to Unsymmetrical cross multi-hexagon grid search algorithm (UMHSA) and diamond search algorithm (DSA), the proposed algorithm reduces search time greatly, and keeps the generated bytes and the PSNR of luminance signals in similar levels.