RECENT TECHNICAL OF ADDED SAMPLE ADAPTIVE OFFSET FOR HEVC

As the upcoming video coding standard, Versatile Video Coding (i.e., VVC) achieves up to 30% Bjøntegaard delta bit-rate (BD-rate) reduction compared with High Efficiency Video Coding (H.265/HEVC). To eliminate or alleviate different kinds of compression artifacts like blocking, ringing, blurring and contouring effects, three in-loop filters, i.e. de-blocking filter (DBF), sample adaptive offset (SAO) and adaptive loop filter (ALF), have been involved in VVC. Recently, Convolutional Neural Network (CNN) has attracted tremendous attention and shows great potential in many tasks in image processing. In this work, we design a CNN-based in-loop filter as an integrated single-model solution which is adaptive to almost any scenarios in video coding. An architecture named as ADCNN (i.e., Attention based Dual-scale CNN) with an attention based processing block is proposed to reduce artifacts of I frames and B frames, which take advantage of informative priors such as the quantization parameter (QP) and partitioning information. Different from existing CNN-based filtering methods, which are mainly designed for the luma component and may need to train different models for different QPs, the proposed filter is adapted to different QPs and different frame types, and all the components (i.e., both luma and chroma) are processed simultaneously with feature exchange and fusion between components for information supplementary. Experimental results show that the proposed ADCNN filter can achieve 6.54%, 13.27%, 15.72% BD-rate savings for Y, U, V respectively under the all intra configuration and 2.81%, 7.86%, 8.60% BD-rate savings under the random access configuration. It can be used to replace all the conventional in-loop filters and also outperforms them without increase in encoding time.

Symmetry considerations play a key role in modern science, and any differentiable symmetry of the action of a physical system has a corresponding conservation law. Symmetry may be regarded as reduction of Entropy. This work focuses on reducing the computational complexity of modern video coding standards by using the maximum entropy principle. The high computational complexity of the coding unit (CU) size decision in modern video coding standards is a critical challenge for real-time applications. This problem is solved in a novel approach considering CU termination, skip, and normal decisions as three-class making problems. The maximum entropy model (MEM) is formulated to the CU size decision problem, which can optimize the conditional entropy; the improved iterative scaling (IIS) algorithm is used to solve this optimization problem. The classification features consist of the spatio-temporal information of the CU, including the rate–distortion (RD) cost, coded block flag (CBF), and depth. For the case analysis, the proposed method is based on High Efficiency Video Coding (H.265/HEVC) standards. The experimental results demonstrate that the proposed method can reduce the computational complexity of the H.265/HEVC encoder significantly. Compared with the H.265/HEVC reference model, the proposed method can reduce the average encoding time by 53.27% and 56.36% under low delay and random access configurations, while Bjontegaard Delta Bit Rates (BD-BRs) are 0.72% and 0.93% on average.

The high efficient video coding (HEVC) standard is crucial for video processing and communication. It is able to achieve about 50% bit-rate reduction with equivalent visual quality compared to H.264/AVC. In the in-loop filter part of HEVC, sample adaptive offset (SAO) is newly added in order to improve the coding efficiency as well as the subjective quality of encoded videos. In order to improve visual quality, we propose an approach by using convolutional neural networks (CNN) after de-blocking filter (DBF) to replace SAO, namely SAOCNN in this paper. Compared with SAO, the proposed SAOCNN is able to reduce the distortion of reconstructed pixels by learning residual feature. Further more, there will be no bit-rates by using the same SAOCNN model in both encoder and decoder sides. Experimental results show that our proposed approach achieves an average 2.04% BD-rate reduction as well as the average 0.12 BDPSNR increment for All intra configuration with sacrifice of encoding time rarely.

Fractional motion compensation (FMC) has been widely adopted in modern video coding standards such as high-efficiency video coding (HEVC). The fractional pixels in FMC are generated by the interpolation filters, and the coefficients of the filters are designed using a Fourier decomposition of the discrete cosine transform (DCT) in HEVC. To overcome the compression deficiency caused by these handcrafted filters, deep learning-based interpolation filter approaches have been introduced recently. In this paper, we analyze the drawbacks of the mean square error (MSE) only loss function which is widely adopted in existing learned-based approaches, and introduce the rate-distortion optimization (RDO) theory into network training to formulate a joint loss function to overcome this issue. Then, a new training scheme is proposed for the detailed implementation of the joint loss function. Besides the calculated MSE loss, the residual will be sent into a rate estimation model which includes three stages: transform/quantization/rate modeling, and the output rate cost will be used as the second part of the loss function. Finally, we propose a novel network structure that utilizes the deformable convolution layer and the residual dense block to enhance the feature extraction and transmission ability to adapt to complex video contents. The proposed RDO-based interpolation filter (RDOIF) has been integrated into HEVC reference software HM-16.7, and extensive experiments are conducted to verify the efficiency of the proposed method. Experimental results show that the proposed RDOIF method achieves 3.1%, 2.4%, and 1.4% BD-rate reduction on average under the low-delay-P, low-delay-B, and random access configurations, respectively. When implemented in the versatile video coding (VVC) test model VTM-10.0, the proposed method can achieve 2.3%, 1.2%, and 0.5% BD-rate savings on average under the low-delay-P, low-delay-B, and random access configurations, respectively.

The high compression efficiency that is provided by the high efficiency video coding (HEVC) standard comes at the cost of a significant increase of the computational load at the decoder. Such an increased burden is a limiting factor to accomplish real-time decoding, specially for high definition video sequences (e.g., Ultra HD 4K). In this scenario, a highly parallel HEVC decoder for the state-of-the-art graphics processor units (GPUs) is presented, i.e., GHEVC. Contrasting to our previous contributions, the data-parallel GHEVC decoder integrates the whole decompression pipeline (except for the entropy decoding), both for intra- and interframes. Furthermore, its processing efficiency was highly optimized by keeping the decompressed frames in the GPU memory for subsequent inter frame prediction. The proposed GHEVC decoder is fully compliant with the HEVC standard, where explicit synchronization points ensure the correct HEVC module execution order. Moreover, the GPU-based HEVC decoder is experimentally evaluated for different GPU devices, an extensive range of recommended HEVC configurations and video sequences, where an average frame rate of 145, 318, and 605 frames per second for Ultra HD 4K, WQXGA, and Full HD, respectively, was obtained in the Random Access configuration with the NVIDIA GeForce GTX TITAN X GPU.

Video surveillance and its applications have become increasingly ubiquitous in modern daily life. In video surveillance system, video coding as a critical enabling technology determines the effective transmission and storage of surveillance videos. In order to meet the real-time or time-critical transmission requirements of video surveillance systems, the low-delay (LD) configuration of the advanced high efficiency video coding (HEVC) standard is usually used to encode surveillance videos. The coding efficiency of the LD configuration is closely related to the quantization parameter (QP) cascading technique which selects or determines the QPs for encoding. However, the quantization parameter cascading (QPC) technique currently adopted for the LD configuration in HEVC test model (i.e., HM) is not optimized since it has not taken full account of the reference dependency in coding. In this paper, an efficient QPC technique for surveillance video coding, referred to as QPC-SV, is proposed, considering all inter reference frames under the LD configuration. Experimental results demonstrate the efficacy of the proposed QPC-SV. Compared with the default configuration of QPC in the HM, the QPC-SV achieves significant rate-distortion performance gain with average BD-rates of -9.35% and -9.76% for the LDP and LDB configurations, respectively.

Machine learning based fast H.264/AVC to HEVC transcoding exploiting block partition similarity

This paper is devoted to Sample Adaptive Offset (SAO). This technique was recently added into High Efficiency Video Coding (HEVC) standard. The concept of SAO is to reduce sample distortion of a region by classifying the region samples into multiple categories, obtaining an offset for each category, and then adding the offset to each sample, where the classifier index and the offsets are coded in the bit stream.

High Efficiency Video Coding (HEVC) has achieved significant coding efficiency improvement beyond existing video coding standard by employing several new coding tools. Deblocking Filter, Sample Adaptive Offset (SAO) and Adaptive Loop Filter (ALF) for in-loop filtering are currently introduced for the HEVC standard. However, these filters are implemented in spatial domain despite the fact of temporal correlation within video sequences. To reduce the artifacts and better align object boundaries in video, a proposed algorithm in in-loop filtering is proposed. The proposed algorithm is implemented in HM-11.0 software. This proposed algorithm allows an average bitrate reduction of about 0.7% and improves the PSNR of the decoded frame by 0.05%, 0.30% and 0.35% in luminance and chroma.

This paper seeks to provide performance analysis for High Efficiency Video Coding (HEVC) HM-16.6 reference coder for different resolution test sequences and picture structures. The HEVC coder is tested for the fixed Quantization Parameter (QP) value in lowdelay configurations, when Main profile with P and hierarchical B pictures, respectively, is used. Simulation results have shown differences in bit-rate and encoding time saving, as well as small difference in SNR values for luma component of picture. Beside objective results, subjective video assessments for all tested sequences are presented, too.

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

Encoding multiple videos in parallel and transmitting them as one joint stream over a limited bandwidth have become a popular strategy for broadcasting, which brings an opportunity to allocate different bitrate for each sequence to meet different demands. In this paper, considering visual experience for human beings, we propose a joint rate allocation scheme aims to reach an equal visual quality among all sequences by minimizing the distortion variance of all the sequences (denoted as minVAR problems). Existing methods assigned bits directly in proportion to their complexity measures and we named them as complexity based allocation scheme (CAS) methods. CAS methods rely on the accuracy of the complexity measures which can hardly be improved under limited computing resources. Also complexities may not be directly related to the distortions. To address these problems, we present a novel joint rate-distortion (R-D) based allocation scheme (RDAS) in this paper. Our proposed scheme can fit for different R-D models and in our method we model the R-D relationship with a hyperbolic function (RDAS-H). We also derive a closed-form solution of RDAS-H by a proposed joint R-D relationship. We integrated the RDAS-H method in high efficiency video coding reference software HM16.0. Experimental results demonstrate that our RDAS-H saves 75.29% variance on average over the related CAS-based method, where we apply both low delay and random access configurations with four different overall bandwidths for all classes recommended by the Joint Collaborative Team on Video Coding. Besides, RDAS-H also saves 36.62% variance on average over our previous method. The proposed RDAS-H method improves the performance significantly while requiring negligible computational cost.

Using curved angular intra-frame prediction to improve video coding efficiency

Recently, deep learning technology has made significant progresses in high efficiency video coding (HEVC), especially in in-loop filter. In this paper, we propose a dense inception attention network (DIA_Net) to delve into image information and model capacity. The DIA_Net contains multiple inception blocks which have different size kernels so as to dig out various scales information. Meanwhile, attention mechanism including spatial attention and channel attention is utilized to fully exploit feature information. Further we adopt a dense residual structure to deepen the network. We attach DIA_Net to the end of in-loop filter part in HEVC as a post-processor and apply it to luma components. The experimental results demonstrate the proposed DIA_Net has remarkable improvement over the standard HEVC. With all-intra(AI) and random access(RA) configurations, It achieves 8.2% bd-rate reduction in AI configuration and 5.6% bd-rate reduction in RA configuration.

High efficiency video coding (HEVC) is the newest video coding standard to greatly increase the coding efficiency of its ancestor H.264/AVC with the aids of its new features such as the quadtree-based coding unit (CU) partitioning, a simple deblocking filter, and other advanced coding techniques. However, HEVC delivers a highly increased computation complexity, which is mainly due to the exhaustive rate distortion optimization search of quadtree-based CU partitioning. Firstly, a feature reduction approach is proposed on a fuzzy support vector machine (SVM) based CU size decision method. The proposed feature reduction approach with rate control (RC) can reduce computational complexity by eliminating some correlated features of a fuzzy SVM-based CU size decision method under a similar coding efficiency. According to the empirical results, our approach can achieve up to 3% of complexity reduction under the same rate distortion (RD) performance over a fuzzy SVM-based approach. Secondly, instead of machine learning (ML) based fast algorithm approach, a CU partitioning pattern optimization method based on genetic algorithm (GA) is proposed to save the computational complexity of a hierarchical quadtree-based CU partitioning. The required coding unit partitioning pattern for an exhaustive partitioning and the rate distortion cost are efficiently considered as the chromosome and the fitness function of the genetic algorithm, respectively. To reduce the computational time, CU partitioning patterns of the key frame is searched and shared to other consecutive frames by taking into account the highly temporal correlation. Our evaluation results show that the proposed method can achieve 62.5% and 16.7% computational complexity reduction on average at 8 Mbps with a negligible average quality degradation compared with HM16.5 and state-of-the-art support vector machine-based fast algorithm, respectively, under low delay P configuration with rate control while 64.1% and 15.1% under low delay configuration with rate control.