Fast CU size decision and intra-prediction mode decision method for H.266/VVC

A Fast CU Size Decision Optimal Algorithm Based on Neighborhood Prediction for HEVC

Intraprediction has been significantly improved in high efficiency video coding over H.264/AVC with quad-tree-based coding unit (CU) structure from size 64×64 to 8×8 and more prediction modes. However, these techniques cause a dramatic increase in computational complexity. An intracoding algorithm is proposed that consists of perceptual fast CU size decision algorithm and fast intraprediction mode decision algorithm. First, based on the visual saliency detection, an adaptive and fast CU size decision method is proposed to alleviate intraencoding complexity. Furthermore, a fast intraprediction mode decision algorithm with step halving rough mode decision method and early modes pruning algorithm is presented to selectively check the potential modes and effectively reduce the complexity of computation. Experimental results show that our proposed fast method reduces the computational complexity of the current HM to about 57% in encoding time with only 0.37% increases in BD rate. Meanwhile, the proposed fast algorithm has reasonable peak signal-to-noise ratio losses and nearly the same subjective perceptual quality.

High Efficiency Video Coding (HEVC) adopts a quad-tree structured coding unit (CU) and more prediction unit (PU) modes are used in all CU depth levels. These improvements bring better coding performance, but the exhaustive search process for optimal CU and PU selection causes higher computational complexity than earlier standards. To speed up the encoder, this paper proposes a fast intra coding algorithm. Firstly, a fast CU size decision method is introduced, which selects different depth decision methods for each largest coding unit. Secondly, a fast direction mode decision method is proposed. It first compares the direction modes of the parent unit and most probable modes (MPMs) list, and then the first direction mode of rate-distortion optimization (RDO) list is used to early terminate the RDO process. Experimental results show that compared with HM 10.0, the proposed algorithm achieves 47% time reduction, whereas the value of BDBR is only 0.7%.

3D-HEVC is developed by ISO/IEC MPEG and ITU-T Video Coding Experts Group (VCEG) as the highestprofile extension of HEVC for 3D video coding and Multi-View texture Videos plus Depth maps (MVD) compress. Concerning the current 3D-HEVC design, the uniform HEVC Intra prediction and Depth Modeling Modes (DMMs) are used in a sophisticated way to ameliorate the performance of the depth map video coding. Although, the enhancement of the coding efficiency is achieved at the cost of computational complexity load, which prevents 3D-HEVC from being used in real-world applications. Thus, we suggest a fast depth map Intra prediction mode decision based on the Self-organizing Map to resolve the aforementioned computational complexity increases. The experimental results demonstrate that the suggested algorithm can increase the encoding time savings up to 39.8%, with insignificant rate-distortion loss.

Since MPEG and VCEG jointly standardized the H.265/HEVC (High-Efficiency Video Coding) international standard in 2013, they have just completed H.266/VVC (Versatile Video Coding) [1] as the Final Draft International Standard (FDIS) in July 2020 through the Joint Video Experts Team (JVET) of ISO / IEC JTC1 / SC29 / WG11 MPEG and ITU-T SG16 / Q6 VCEG. VVC supports up to 87 intra coding modes including 65 general directional modes and 20 wide angular modes, which is increased more than twice compared to HEVC. VVC can accommodate not only more detailed intra prediction directions, but also so called wide angular modes which make its special sense in non-square coding blocks. The more detailed intra predictions naturally demand more computation in order to determine the most effective intra-prediction mode. In this sense, we investigate how to pre-prune the prediction candidate lists for a fast ISP intra mode decision and propose searching only a small number of prediction modes based on the shape of a block in the rate-distortion optimization (RDO)–based intra mode decision. The proposed method is verified to decrease the intra prediction processing time with only a little increase in the bit rate and a negligible reduction in PSNR values.

Recently, Joint Video Experts Team (JVET) has completed the new Versatile Video Coding (H.266/VVC) standard. VVC employs a new block partition structure named quad-tree with nested multi-type tree (QTMT) to improve coding efficiency. However, the new block partition structure increases huge encoding time compared with HEVC for brute-force ratedistortion (RD) optimization. To reduce encoding complexity, we propose a Support Vector Machine (SVM) based fast CU partitioning algorithm for VVC intra coding in this paper which terminates redundant partitions early by predicting the partition of CU using texture information. We trained classifiers for CUs of different sizes to improve accuracy and control the complexity of the classifiers themselves. Different thresholds are set for each classifier to achieve a trade-off between encoding complexity and RD performance. Experimental results show that the proposed method can save encoder time ranging from 30.78% to 63.16% with 1.10% to 2.71% BD-BR increase.

This paper proposes a content adaptive fast CU size selection algorithm for HEVC intra-prediction using weighted support vector machines. The proposed algorithm demonstrates an average encoding time reduction of 52.38% with 1.19% average BDBR increase compared to HM16.1 reference encoder.

In this paper, we present a simple but efficient algorithm for intra prediction mode decision in H.264/AVC. Based on our investigation, the DC mode appears to be the superior prediction mode among various candidates. We propose an intra-mode decision algorithm where the DC mode is chosen as a candidate for the best prediction mode. Our experimental results show that the proposed algorithm saves 80.82% of the encoding time, compared to the H.264 reference software on average, while reducing the negligible peak signal-to-noise ratio (PSNR) values and slightly increasing the bitrate. In order to reduce the bitrate increment of the above-mentioned algorithm and refine our DC mode condition, we propose another fast intra prediction mode decision algorithm where we use a condition of block boundary to select the DC mode. Our experimental results show that this proposed algorithm not only increases PSNR values and decreases bitrates, but also saves the encoding time.

Versatile Video Coding (VVC) is the emerging video coding standard to be finalized by the Joint Video Experts Team in July 2020. Compared to its predecessor, the High Efficiency Video Coding (HEVC) standard, VVC provides 50% bit-rate reduction at comparable visual quality for natural video content in high-definition (HD) and ultra high-definition (UHD) resolution. To achieve this, the standard incorporates more advanced and generalized algorithms, leading to an increase in computational complexity. This includes for example additional in-loop filters, decoder-side motion refinement and search as well as an increased number of transforms, which creates a grand challenge for implementers to achieve live decoding on general-purpose CPUs. In this paper, the work on an efficient software decoder implementation for the upcoming VVC standard is described, including optimization of sample operations using single instruction multiple data (SIMD) instructions and parallelization approaches with multithreading. As a result, the presented decoder can perform live decoding of 10bit HD video at 60 frames per second (fps) and 10bit UHD video at 30fps on modern mobile consumer hardware, showcasing that VVC live decoding is possible already right before finalization of the standard.

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

High efficiency video coding (HEVC) is the newest video coding standard to greatly increase the coding efficiency of its ancestor H.264/AVC with the aids of its new features such as the quadtree-based coding unit (CU) partitioning, a simple deblocking filter, and other advanced coding techniques. However, HEVC delivers a highly increased computation complexity, which is mainly due to the exhaustive rate distortion optimization search of quadtree-based CU partitioning. Firstly, a feature reduction approach is proposed on a fuzzy support vector machine (SVM) based CU size decision method. The proposed feature reduction approach with rate control (RC) can reduce computational complexity by eliminating some correlated features of a fuzzy SVM-based CU size decision method under a similar coding efficiency. According to the empirical results, our approach can achieve up to 3% of complexity reduction under the same rate distortion (RD) performance over a fuzzy SVM-based approach. Secondly, instead of machine learning (ML) based fast algorithm approach, a CU partitioning pattern optimization method based on genetic algorithm (GA) is proposed to save the computational complexity of a hierarchical quadtree-based CU partitioning. The required coding unit partitioning pattern for an exhaustive partitioning and the rate distortion cost are efficiently considered as the chromosome and the fitness function of the genetic algorithm, respectively. To reduce the computational time, CU partitioning patterns of the key frame is searched and shared to other consecutive frames by taking into account the highly temporal correlation. Our evaluation results show that the proposed method can achieve 62.5% and 16.7% computational complexity reduction on average at 8 Mbps with a negligible average quality degradation compared with HM16.5 and state-of-the-art support vector machine-based fast algorithm, respectively, under low delay P configuration with rate control while 64.1% and 15.1% under low delay configuration with rate control.

With the development of multimedia equipment and the increasing demand for high-quality video applications, the traditional video coding standard, H.265/High Efficiency Video Coding (HEVC), can no longer effectively satisfy the requirements. To promote the development of high-quality video, a new generation video coding standard, H.266/Versatile Video Coding (H.266/VVC), is established, and it is the inheritance and development of H.265/HEVC. It not only retains many mature technologies and methods in HEVC but also adds some new coding tools, such as wide-angle prediction and Multitype Tree (MTT) partition structure. The MTT partition structure brings a more flexible partition method of Coding Unit (CU), but the accompanying increase in computational complexity is unacceptable. In order to ensure an effective balance between coding efficiency and coding quality, a fast CU partition algorithm based on texture is proposed in this paper. First, the texture complexity of the neighboring CU is used as a threshold for evaluating the complexity of the current CU, so as to skip the unpromising depth. Then, the gradient features are extracted to determine whether the Quad-Tree (QT) partition is executed. Finally, the improved Canny operator is used to extract edge features, and the partition mode in the horizontal or vertical direction is excluded. The algorithm was embedded in VTM7.0, and the video sequences with different resolutions were tested under general experimental configuration. Simulation experiment results show that the average time saving of this method reached 50.56% compared with the anchor algorithm. At the same time, the average BDBR is increased by 1.31%.

The relevance of the spatial domain is mainly eliminated by intra-prediction encoding. First, the traditional intra-prediction algorithms are introduced. And then, according to the correlation between the most likely mode and the best mode, a mode matrix is arrived at, the use of this mode matrix for early termination. The experimental results show that when we use this matrix in the proposed algorithm, we can achieve about 9%-15% efficiency improvement with no any image quality degradation; If we use simple matrix, we can achieve about 33%-48% of the computing performance upgrade with the drop in PSNR 0.03db.

This paper proposes an elaborated approach for fast mode decision by using the probabilities estimated through context models. In H.264, the binary arithmetic coder is adopted to convert encoding modes into binary bins and assign each bin a context to estimate the probability of the bin as “1” or “0”. By collecting the probabilities of all bins, the probability of encoding modes can be calculated with few extra computations. The estimated probabilities can be applied to existed fast mode decision schemes to further improve the performance. It also provides probability information for scalable complexity encoding. From our experiment results, the proposed method can further improve 20% – 30% encoding speed of an optimized H.264 encoder, while keeping the same coding efficiency.

High-efficiency video coding is the most recent video compression standard. HEVC is designed to decrease the bit rate of video transmission without affecting the video quality. The intra prediction of HEVC features about 35 directional modes. The planar mode and DC modes are included into it. The decision about the most appropriate intra mode within the coding unit of high-efficiency video encoder is a vital component in video coding. The intra mode decision has been a crucial, computationally complex processing step and has a share of 85% in the overall video coding complexity. The optimal mode is selected by rough mode decision (RMD) process from all 35 modes and final decision of partitioning is taken through the rate distortion optimization (RDO) process. The brute force RD cost calculation process consumes a large portion of HEVC encoding complexity. This paper presents analysis of the spread of 35 directional modes over the video frame and the correlation between the homogeneous or non-homogeneous characteristics of video content and the spread of directional modes over the video frame. The proposed method is based on the sum of average gradient evaluated for each of the 35 directional modes which help to reduce the number of candidate modes for rough mode decision and RD cost calculation. The performance of the proposed algorithm is evaluated on three distinct classes of video sequences. The offline classification accuracy of the proposed scheme is measured to be 90%. The exhaustive analysis of mode decision carried out in the proposed method will be subsequently useful for training machine learning algorithm for early decision about coding unit depth and fast prediction of the appropriate intra mode. The early depth decision and reduction in the number of candidate coding units to be passed through iterative RD cost computation will drastically reduce the computation complexity and increase the encoding time of high-efficiency video encoder.