SVM Based Fast CU Partitioning Algorithm for VVC Intra Coding

Quad-tree-based coding unit (CU) block partitioning structure in High Efficiency Video Coding (HEVC) intra prediction causes a significant coding complexity increase. Hence, a fast block partitioning algorithm based on Support Vector Machine (SVM) is proposed in this paper. Firstly, some effective features are extracted from CUs in each depth as the input vector of SVM. Secondly, three offline trained SVM CU splitting models are loaded in each CU depth, which predict the class label of the current CU according to the extracted features. Moreover, the parameters of SVM models are resolved by grid search method. Finally, based on the predicted class label, the encoder will decide whether to split the current CU or not. Experimental results show that the proposed algorithm reduces the computational complexity of HM13.0 to 30.1% and 53.9% in encoding time with and without RDO(rate distortion optimization), while the loss in coding efficiency is negligible.

H.266/Versatile Video Coding (VVC) is the most recent video coding standard developed by the Joint Video Experts Team (JVET). The quad-tree with nested multi-type tree (QTMT) architecture that improves the compression performance of H.266/VVC is introduced. Moreover, H.266/VVC contains a greater number of intra-prediction modes than H.265/High Efficiency Video Coding (HEVC), totalling 67. However, these lead to extremely the coding computational complexity. To cope with the above issues, a fast intra-coding unit (CU) size decision method and a fast intra-prediction mode decision method are proposed in this paper. Specifically, the trained Support Vector Machine (SVM) classifier models are utilized for determining CU partition mode in a fast CU size decision scheme. Furthermore, the quantity of intra-prediction modes added to the RDO mode set decreases in a fast intra-prediction mode decision scheme based on the improved search step. Simulation results illustrate that the proposed overall algorithm can decrease 55.24% encoding runtime with negligible BDBR.

Quadtree with nested multi-type tree (QTMT) partition structure in Versatile Video Coding (VVC) contributes to superior encoding performance compared to the basic quad-tree (QT) structure in High Efficiency Video Coding (HEVC). However, the improvement of performance leads to an un-avoidable increase of computational complexity. To achieve a balance between coding efficiency and compression quality, we propose a fast intra partition algorithm based on variance and gradient to solve the rectangular partition problem in VVC. First, further splitting of smooth areas is terminated. Then, QT partition is chosen depending on the gradient features extracted by Sobel operator. Finally, one partition from five possible QTMT partitions is directly chosen by computing the variance of variance of sub-CUs. The theoretical basis of our method is that a homogeneous area tends to be predicted with a larger coding unit (CU), and sub-parts of a split CU are prone to have different textures from each other. To our knowledge, this is the first attempt to apply traditional method to accelerating the rectangular partition problem in VVC intra prediction. Experimental results show that the proposed method can save averagely 53.17% encoding time with only 1.62% BDBR increase and 0.09dB BDPSNR loss compared to anchor VTM4.0.

Multi-Type Tree (MTT) block partition technology introduced by next-generation Versatile Video Coding (VVC) significantly improves Rate Distortion (RD) performance. Nevertheless, the complicated recursive search for the optimal partition structure significantly increases the computational complexity of VVC. To address this issue, previous fast intra-coding methods were intended to expedite the coding process by predicting the partition types of a Coding Unit (CU). However, these methods are time-consuming and few methods directly predict the coding depth to reduce complexity. Therefore, in this paper, we propose a two-stage Depth and Partition Convolutional Neural Network (DP-CNN) for VVC. The method can efficiently predict the Quad Tree with a nested Multi-type Tree (QTMT) structure and coding depths, and effectively reduce unnecessary partition processes under the condition of lower distortion. First, a Depth Convolutional Neural Network (D-CNN) and a depth post-processing algorithm are designed to predict the coding depth of 64× 64 blocks, facilitating early termination of the RD optimization (RDO) process. We then use a Partition Convolutional Neural Network (P-CNN) and a partition decision algorithm to extract boundary features of 4× 4 blocks for the 32× 32 CU and its sub-CUs through a single inference process. The coding performance and complexity are balanced by the synergistic integration of the above methods. Meanwhile, the CNN inference process has been optimized to eliminate unnecessary delays to minimize additional time overhead. Comprehensive experiments are performed to validate the efficacy of the proposed DP-CNN. The experimental results show that the proposed method reduces the VVC coding time by approximately 38.86% 59.69% while maintaining a negligible Bjontegaard Delta Bit-Rate increase of 0.47% 1.50%, superior to the existing state-of-the-art methods.

Since MPEG and VCEG jointly standardized the H.265/HEVC (High-Efficiency Video Coding) international standard in 2013, they have just completed H.266/VVC (Versatile Video Coding) [1] as the Final Draft International Standard (FDIS) in July 2020 through the Joint Video Experts Team (JVET) of ISO / IEC JTC1 / SC29 / WG11 MPEG and ITU-T SG16 / Q6 VCEG. VVC supports up to 87 intra coding modes including 65 general directional modes and 20 wide angular modes, which is increased more than twice compared to HEVC. VVC can accommodate not only more detailed intra prediction directions, but also so called wide angular modes which make its special sense in non-square coding blocks. The more detailed intra predictions naturally demand more computation in order to determine the most effective intra-prediction mode. In this sense, we investigate how to pre-prune the prediction candidate lists for a fast ISP intra mode decision and propose searching only a small number of prediction modes based on the shape of a block in the rate-distortion optimization (RDO)–based intra mode decision. The proposed method is verified to decrease the intra prediction processing time with only a little increase in the bit rate and a negligible reduction in PSNR values.

Versatile Video Coding (VVC) significantly improves the coding efficiency over the preceding high efficiency video coding (HEVC) standard, but at the expense of much higher computational complexity. Specifically for intra coding of VVC, the computational burden is mainly on the brute-force recursive rate-distortion optimization (RDO) search of quadtree with nested multi-type tree (QTMT) based coding unit (CU) partition structure. Consequently, we propose a random forest based algorithm to reduce the complexity of CU partition. The CUs are first divided into three categories, namely simple, fuzzy, and complex CUs. For simple and complex CUs, one random forest classifier is trained to directly predict the optimal partition mode. For fuzzy CUs, another random forest is trained to predict whether the partition process is terminated or not. The experimental results show that the complexity reduction of the proposed algorithm is up to 69% as compared to the VVC reference software (VTM 7.0), and averagely 57% encoding time saving is achieved with 1.21 % BDBR increase.

Versatile Video Coding (VVC) is the emerging video coding standard to be finalized by the Joint Video Experts Team in July 2020. Compared to its predecessor, the High Efficiency Video Coding (HEVC) standard, VVC provides 50% bit-rate reduction at comparable visual quality for natural video content in high-definition (HD) and ultra high-definition (UHD) resolution. To achieve this, the standard incorporates more advanced and generalized algorithms, leading to an increase in computational complexity. This includes for example additional in-loop filters, decoder-side motion refinement and search as well as an increased number of transforms, which creates a grand challenge for implementers to achieve live decoding on general-purpose CPUs. In this paper, the work on an efficient software decoder implementation for the upcoming VVC standard is described, including optimization of sample operations using single instruction multiple data (SIMD) instructions and parallelization approaches with multithreading. As a result, the presented decoder can perform live decoding of 10bit HD video at 60 frames per second (fps) and 10bit UHD video at 30fps on modern mobile consumer hardware, showcasing that VVC live decoding is possible already right before finalization of the standard.

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

The versatile video coding (VVC) standard has been finalized recently, but it did not take into account the different characteristics of depth videos. The purpose of a method presented in this paper was to improve intra-coding efficiency of depth video coding in versatile video coding (VVC). The proposed method enables to turn on/off intra-boundary filters used in VVC intra-prediction on the fly. Experimental results clearly demonstrate the superiority of the proposed method over the conventional VVC intra-prediction in terms of rate-distortion (RD) performance.

This paper presents a fast encoding method for versatile video coding (VVC) using an early determination scheme that skips redundant multi-type tree (MTT) pruning. MTT consists of binary and ternary trees (TTs) with traditional quadtrees and has recently attracted considerable VVC research interest due to coding efficiency beyond HEVC. However, the additional trees, particularly TTs, significantly increased VVC complexity, which has rarely been studied previously. Therefore, we identified TT characteristics in the VVC encoding context, and hence propose a simple early decision technique that can effectively reduce TT complexity. Experiments over various test sequences show that the proposed method substantially decreased the total encoding time for a VVC test model 4.0 (VTM 4.0) with only slight coding loss under the all-intra configuration. In particular, the proposed method reduced TT encoding time by up to 66% compared with VTM 4.0.

In spite of the recent advances in telecommunication standards, communication networks still have limited bandwidths and storage capacity. Therefore, video compression has drawn increasing importance since high-resolution video contents have become more and more used in various fields. These requirements raise the need for high-performance video-compression technologies able to reduce the amount of data to be transmitted or stored by compressing the input video signal into a bitstream file. Improving the coding efficiency was always one of the crucial issues of various compression standards that aim to get the most compact representation of the reconstructed video, with a high subjective quality. The high-efficiency video coding (HEVC) comes to respond to these requirements. However, the increased consumption of high-quality multimedia content has pushed the international communication companies to put much effort to better enhance video-coding techniques. In this perspective, an upcoming video-coding standard to be known as versatile video coding (VVC) has emerged aiming to improve the coding efficiency of the current HEVC codec. Improvements on rate distortion (RD) performance that came with both HEVC and VVC have brought an increased complexity in the majority of the coding modules, which makes it difficult to implement on hardware systems with real-time encoding. This chapter focuses on the transform coding stage as one of the most computationally demanding modules. For HEVC, efficient approximation algorithms in addition to reconfigurable and scalable architecture have been developed in order to decrease the computational complexity of the transform module. The main objectives are to meet low power and real-time processing constraints while maintaining a compression gain and a satisfying video quality. Field programmable gate array (FPGA) implementation results and comparisons with existing works confirm the efficiency of the proposed approximations since they contribute in reducing time and power consumption, optimizing the hardware resources and bringing peak signal-to noise ratio (PSNR) improvement as well. Similarly, for the adaptive multiple transform (AMT) introduced in the transform module of the VVC, approximations were done for discrete cosine transform (DCT)-II and discrete sine transform (DST)-VII transforms since they are statistically the most used ones among the five predefined types. Bitrate (BR) reduction with a slightly degradation of video quality and a less use of hardware resources are the main contributions of the proposed approximations.

Chaotic moving video quality enhancement based on deep in-loop filtering

In comparison with H.264/Advanced Video Coding, the newest video coding standard, High Efficiency Video Coding (HEVC), improves video coding rate-distortion (RD) performance, but at the price of significant increase in its encoding complexity, especially, in intra-mode decision due to the adoption of more complex block partitions and more candidate intra-prediction modes (IPMs). To reduce the mode decision complexity in HEVC intra-frame coding, while maintaining its RD performance, in this paper, we first formulate the mode decision problem in intra-frame coding as a Bayesian decision problem based on the newly proposed transparent composite model (TCM) for discrete cosine transform coefficients, and then present an outlier-based fast intra-mode decision (OIMD) algorithm. The proposed OIMD algorithm reduces the complexity using outliers identified by TCM to make a fast coding unit split/nonsplit decision and reduce the number of IPMs to be compared. To further take advantage of the outlier information furnished by TCM, we also refine entropy coding in HEVC by encoding the outlier information first, and then the actual mode decision conditionally given the outlier information. The proposed OIMD algorithm can work with and without the proposed entropy coding refinement. Experiments show that for the all-intra-main test configuration of HEVC: 1) when applied alone, the proposed OIMD algorithm reduces, on average, the encoding time (ET) by 50% with 0.7% Bjontegaard distortion (BD)-rate increase and 2) when applied in conjunction with the proposed entropy coding refinement, it reduces, on average, both the ET by 50% and BD-rate by 0.15%.

This paper presents an encoding time and encoding efficiency analysis of the Quadtree with nested Multi-type Tree (QTMT) structure in the Versatile Video Coding (VVC) intra-frame prediction. The QTMT structure enables VVC to improve the compression performance compared to its predecessor standard at the cost of a higher encoding complexity. The intra-frame prediction time raised about 26 times compared to the HEVC reference software, and most of this time is related to the new block partitioning structure. Thus, this paper provides a detailed description of the VVC block partitioning structure and an in-depth analysis of the QTMT structure regarding coding time and coding efficiency. Based on the presented analyses, this paper can guide outcoming works focusing on the block partitioning of the VVC intra-frame prediction.

Wavelet difference reduction (WDR) has recently been proposed as a method for efficient embedded image coding. In this paper, the WDR algorithm is analysed and four new techniques are proposed to either reduce its complexity or improve its rate distortion (RD) performance. The first technique, dubbed modified WDR-A (MWDR-A), focuses on improving the efficiency of the arithmetic coding (AC) stage of the WDR. Based on experiments with the statistics of the output symbol sequence, it is shown that the symbols can either be arithmetic coded under different contexts or output without AC. In the second technique, MWDR-B, the AC stage is dropped from the coder. By employing MWDR-B, up to 20% of coding time can be saved without sacrificing the RD performance, when compared to WDR. The third technique focuses on the improvement of RD performance using context modelling. A low-complexity context model is proposed to exploit the statistical dependency among the wavelet coefficients. This technique is termed context-modelled WDR (CM-WDR), and acts without the AC stage to improve the RD performance by up to 1.5 dB over WDR on a set of test images, at various bit rates. The fourth technique combines CM-WDR with AC and achieves a 0.2 dB improvement over CM-WDR in terms of PSNR. The proposed techniques retain all the features of WDR, including low complexity, region-of-interest capability, and embeddedness.