A VVC Video Steganography Based on Coding Units in Chroma Components with a Deep Learning Network

In the versatile video coding (VVC) proposed by the Joint Video Exploration Team (JVET), the quad-tree with the nested multi-type tree (QTMT) partition scheme has been adopted based on the quadtree structure in the high efficiency video coding (HEVC). The video coding quality of VVC is better than the HEVC, but the algorithm complexity has also increased greatly. In this work, we present an adaptive CU split decision for intra frame with the pooling-variable convolutional neural network (CNN), targeting at various coding unit (CU) shape. The shape-adaptive CNN is realized by the variable pooling layer size where we can make the most of the pooling layer in CNN and retain the original information. Based on the proposed CNN, the CU split or not will be decided by only one trained network, same architecture and parameters for the CUs with multiple sizes. Moreover, with the proposed shape-based CNN training scheme, the various training sample size can be processed successfully. The CUbased network can avoid the full rate-distortion optimization for the CU split and the CU-level rate control can also be enabled. The experiment results show that the proposed method can save 33% coding time with only 0.99% Bjontegaard Delta bitrate (BD-rate) increase.

With the rapid advancement in many multimedia applications, such as video gaming, computer vision applications, and video streaming and surveillance, video quality remains an open challenge. Despite the existence of the standardized video quality as well as high definition (HD) and ultrahigh definition (UHD), enhancing the quality for the video compression standard will improve the video streaming resolution and satisfy end user's quality of service (QoS). Versatile video coding (VVC) is the latest video coding standard that achieves significant coding efficiency. VVC will help spread high-quality video services and emerging applications, such as high dynamic range (HDR), high frame rate (HFR), and omnidirectional 360-degree multimedia compared to its predecessor high efficiency video coding (HEVC). Given its valuable results, the emerging field of deep learning is attracting the attention of scientists and prompts them to solve many contributions. In this study, we investigate the deep learning efficiency to the new VVC standard in order to improve video quality. However, in this work, we propose a wide-activated squeeze-and-excitation deep convolutional neural network (WSE-DCNN) technique-based video quality enhancement for VVC. Thus, the VVC conventional in-loop filtering will be replaced by the suggested WSE-DCNN technique that is expected to eliminate the compression artifacts in order to improve visual quality. Numerical results demonstrate the efficacy of the proposed model achieving approximately −2.85%, −8.89%, and −10.05% BD-rate reduction of the luma (Y) and both chroma (U, V) components, respectively, under random access profile.

The rapid proliferation of audiovisual services has surpassed the capabilities of contemporary communication infrastructures, necessitating the introduction of the Versatile Video Coding (VVC) standard. VVC is the latest video coding standard, which achieves approximately a 40 % reduction in bitrate relative to its predecessor, the High Efficiency Video Coding (HEVC) standard; however, this improvement comes with a substantial increase in encoding complexity. To alleviate this burden, we propose a Vision Transformer-driven strategy for fast Coding Unit (CU) partitioning. By leveraging the transformer's global context awareness, the model swiftly predicts optimal split patterns, allowing the encoder to bypass the exhaustive rate-distortion optimization (RDO) traditionally required by VVC. This approach significantly accelerates the CU decision process while preserving compression efficiency. Unlike existing methods, the transformer model effectively captures long-range dependencies and spatial structures across the entire frame, leading to more accurate partitioning. The self-attention mechanism enables the encoder to concentrate only on promising split candidates, thereby reducing unnecessary computations. Furthermore, the proposed framework is adaptable to various video resolutions and content types, ensuring robustness and generalizability. Experimental analysis demonstrates that the proposed method achieves up to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{7 0. 5 5 \%}$</tex> reduction in encoding time with negligible bitrate overhead and minimal degradation in visual quality, as evaluated by PSNR, SSIM, and VMAF metrics, when compared to the standalone VVC encoder.

Versatile Video Coding (VVC) is the latest generation of video coding standards released to meet the arrival of the ultra-high-definition (UHD) video era. Compared with the quad-tree (QT) structure in High Efficiency Video Coding (HEVC), VVC obtains excellent coding performance by introducing the division structure of binary-tree (BT) and ternary-tree (TT), but at the same time, it also greatly improves the computational complexity in the coding process. In order to reduce the coding unit (CU) division traversal time in the inter-frame coding process, this paper proposes an algorithm for fast CU division using random forest for the VVC inter-frame coding process. The algorithm first calculates the eigenvalues of the current CU block, and then inputs them into the random forest model to obtain the predicted classification. At the same time, it decides whether to adopt the predicted result based on the decision rule. If adopted, the stack in the encoder is modified to jump over-redundant traversal operations. Experimental results show that the proposed algorithm reduces the encoding time by about 37.30% on average in random access mode, while the video quality loss is only 1.41%.

Versatile Video Coding (VVC) significantly improves the coding efficiency over the preceding high efficiency video coding (HEVC) standard, but at the expense of much higher computational complexity. Specifically for intra coding of VVC, the computational burden is mainly on the brute-force recursive rate-distortion optimization (RDO) search of quadtree with nested multi-type tree (QTMT) based coding unit (CU) partition structure. Consequently, we propose a random forest based algorithm to reduce the complexity of CU partition. The CUs are first divided into three categories, namely simple, fuzzy, and complex CUs. For simple and complex CUs, one random forest classifier is trained to directly predict the optimal partition mode. For fuzzy CUs, another random forest is trained to predict whether the partition process is terminated or not. The experimental results show that the complexity reduction of the proposed algorithm is up to 69% as compared to the VVC reference software (VTM 7.0), and averagely 57% encoding time saving is achieved with 1.21 % BDBR increase.

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

Quadtree with nested multi-type tree (QTMT) partition structure in Versatile Video Coding (VVC) contributes to superior encoding performance compared to the basic quad-tree (QT) structure in High Efficiency Video Coding (HEVC). However, the improvement of performance leads to an un-avoidable increase of computational complexity. To achieve a balance between coding efficiency and compression quality, we propose a fast intra partition algorithm based on variance and gradient to solve the rectangular partition problem in VVC. First, further splitting of smooth areas is terminated. Then, QT partition is chosen depending on the gradient features extracted by Sobel operator. Finally, one partition from five possible QTMT partitions is directly chosen by computing the variance of variance of sub-CUs. The theoretical basis of our method is that a homogeneous area tends to be predicted with a larger coding unit (CU), and sub-parts of a split CU are prone to have different textures from each other. To our knowledge, this is the first attempt to apply traditional method to accelerating the rectangular partition problem in VVC intra prediction. Experimental results show that the proposed method can save averagely 53.17% encoding time with only 1.62% BDBR increase and 0.09dB BDPSNR loss compared to anchor VTM4.0.

This paper describes a video coding technology proposal submitted by MediaTek in April 2018 in response to a joint call for proposals (CfP) issued by ITU-T VCEG and ISO/IEC MPEG. The proposal uses the conventional block-based hybrid coding approach with a breakthrough design of block structure, substantial improvements of coding tools, and new inventions of loop filtering techniques. First, the block structure is designed using a highly flexible partitioning scheme, where each coding tree unit in a picture is split into multiple coding units (CUs) by a recursive quaternary tree followed by a recursive binary-ternary tree. The prediction unit (PU) and transform unit (TU) concepts are unified with the CU concept, i.e., PU and TU are basically the same as CU. Moreover, inter prediction, intra prediction, transform, quantization, in-loop filtering, and entropy coding are all improved. Finally, new loop filtering techniques are invented, where the proposed convolutional neural network loop filtering is of particular interest. The proposed video codec achieves 43.81% average bit rate savings based on objective measures compared to the High Efficiency Video Coding (HEVC) anchors and is among the best-performing CfP responses both objectively and subjectively. It has been partially adopted into the working draft of the Versatile Video Coding (VVC) standard.

Multi-Type Tree (MTT) block partition technology introduced by next-generation Versatile Video Coding (VVC) significantly improves Rate Distortion (RD) performance. Nevertheless, the complicated recursive search for the optimal partition structure significantly increases the computational complexity of VVC. To address this issue, previous fast intra-coding methods were intended to expedite the coding process by predicting the partition types of a Coding Unit (CU). However, these methods are time-consuming and few methods directly predict the coding depth to reduce complexity. Therefore, in this paper, we propose a two-stage Depth and Partition Convolutional Neural Network (DP-CNN) for VVC. The method can efficiently predict the Quad Tree with a nested Multi-type Tree (QTMT) structure and coding depths, and effectively reduce unnecessary partition processes under the condition of lower distortion. First, a Depth Convolutional Neural Network (D-CNN) and a depth post-processing algorithm are designed to predict the coding depth of 64× 64 blocks, facilitating early termination of the RD optimization (RDO) process. We then use a Partition Convolutional Neural Network (P-CNN) and a partition decision algorithm to extract boundary features of 4× 4 blocks for the 32× 32 CU and its sub-CUs through a single inference process. The coding performance and complexity are balanced by the synergistic integration of the above methods. Meanwhile, the CNN inference process has been optimized to eliminate unnecessary delays to minimize additional time overhead. Comprehensive experiments are performed to validate the efficacy of the proposed DP-CNN. The experimental results show that the proposed method reduces the VVC coding time by approximately 38.86% 59.69% while maintaining a negligible Bjontegaard Delta Bit-Rate increase of 0.47% 1.50%, superior to the existing state-of-the-art methods.

In spite of the recent advances in telecommunication standards, communication networks still have limited bandwidths and storage capacity. Therefore, video compression has drawn increasing importance since high-resolution video contents have become more and more used in various fields. These requirements raise the need for high-performance video-compression technologies able to reduce the amount of data to be transmitted or stored by compressing the input video signal into a bitstream file. Improving the coding efficiency was always one of the crucial issues of various compression standards that aim to get the most compact representation of the reconstructed video, with a high subjective quality. The high-efficiency video coding (HEVC) comes to respond to these requirements. However, the increased consumption of high-quality multimedia content has pushed the international communication companies to put much effort to better enhance video-coding techniques. In this perspective, an upcoming video-coding standard to be known as versatile video coding (VVC) has emerged aiming to improve the coding efficiency of the current HEVC codec. Improvements on rate distortion (RD) performance that came with both HEVC and VVC have brought an increased complexity in the majority of the coding modules, which makes it difficult to implement on hardware systems with real-time encoding. This chapter focuses on the transform coding stage as one of the most computationally demanding modules. For HEVC, efficient approximation algorithms in addition to reconfigurable and scalable architecture have been developed in order to decrease the computational complexity of the transform module. The main objectives are to meet low power and real-time processing constraints while maintaining a compression gain and a satisfying video quality. Field programmable gate array (FPGA) implementation results and comparisons with existing works confirm the efficiency of the proposed approximations since they contribute in reducing time and power consumption, optimizing the hardware resources and bringing peak signal-to noise ratio (PSNR) improvement as well. Similarly, for the adaptive multiple transform (AMT) introduced in the transform module of the VVC, approximations were done for discrete cosine transform (DCT)-II and discrete sine transform (DST)-VII transforms since they are statistically the most used ones among the five predefined types. Bitrate (BR) reduction with a slightly degradation of video quality and a less use of hardware resources are the main contributions of the proposed approximations.

The audiovisual entertainment industry has entered a race to find the video encoder offering the best Rate/Distortion (R/D) performance for high-quality high-definition video content. The challenge consists in providing a moderate to low computational/hardware complexity encoder able to run Ultra High-Definition (UHD) video formats of different flavours (360°, AR/VR, etc.) with state-of-the-art R/D performance results. It is necessary to evaluate not only R/D performance, a highly important feature, but also the complexity of future video encoders. New coding tools offering a small increase in R/D performance at the cost of greater complexity are being advanced with caution. We performed a detailed analysis of two evolutions of High Efficiency Video Coding (HEVC) video standards, Joint Exploration Model (JEM) and Versatile Video Coding (VVC), in terms of both R/D performance and complexity. The results show how VVC, which represents the new direction of future standards, has, for the time being, sacrificed R/D performance in order to significantly reduce overall coding/decoding complexity.

With the rapid development of 5G and the Internet of Things, video has gained broad application space as a carrier of information, and video codec technology can reduce video storage space, save transmission bandwidth, and provide the possibility for the promotion of video applications. Versatile Video Coding (VVC), the latest generation of video coding standard today, still uses block-based coding, which inevitably introduces coding artifacts, and deblocking filters can effectively reduce artifacts and improve the quality of compressed video. The deblocking filter in VVC still adopts the method based on empirical threshold when making the filtering decision, but the actual video scene changes variously, and it is obviously impossible to obtain the optimal filtering effect with a fixed threshold. The decision of the filtering mode is essentially a classification problem. Existing research shows that the convolutional neural network (CNN) has a strong ability in classification tasks, and its feature extraction and nonlinear fitting ability can greatly improve the learning ability of the target, so as to obtain better classification accuracy. Therefore, we improves the deblocking filter in VVC based on CNN. Experimental results show that, compared with the original filtering method of VVC, our method can better improve the quality of compressed video.

Versatile Video Coding (VVC) is the emerging video coding standard to be finalized by the Joint Video Experts Team in July 2020. Compared to its predecessor, the High Efficiency Video Coding (HEVC) standard, VVC provides 50% bit-rate reduction at comparable visual quality for natural video content in high-definition (HD) and ultra high-definition (UHD) resolution. To achieve this, the standard incorporates more advanced and generalized algorithms, leading to an increase in computational complexity. This includes for example additional in-loop filters, decoder-side motion refinement and search as well as an increased number of transforms, which creates a grand challenge for implementers to achieve live decoding on general-purpose CPUs. In this paper, the work on an efficient software decoder implementation for the upcoming VVC standard is described, including optimization of sample operations using single instruction multiple data (SIMD) instructions and parallelization approaches with multithreading. As a result, the presented decoder can perform live decoding of 10bit HD video at 60 frames per second (fps) and 10bit UHD video at 30fps on modern mobile consumer hardware, showcasing that VVC live decoding is possible already right before finalization of the standard.

The growing popularity of high resolution video and the continuously increasing demands for high quality video on mobile devices are producing stronger needs for more efficient video encoder. Concerning these desires, HEVC, a newest video coding standard, has been developed by a joint team formed by ISO/IEO MPEG and ITU/T VCEG. Its design goal is to achieve a 50% compression gain over its predecessor H.264 with an equal or even higher perceptual video quality. Motion Estimation (ME) being as one of the most critical module in video coding contributes almost 50%-70% of computational complexity in the video encoder. This high consumption of the computational resources puts a limit on the performance of encoders, especially for full HD or ultra HD videos, in terms of coding speed, bit-rate and video quality. Thus the major part of this work concentrates on the computational complexity reduction and improvement of timing performance of motion estimation algorithms for HEVC standard. First, a new strategy to calculate the SAD (Sum of Absolute Difference) for motion estimation is designed based on the statistics on property of pixel data of video sequences. This statistics demonstrates the size relationship between the sum of two sets of pixels has a determined connection with the distribution of the size relationship between individual pixels from the two sets. Taking the advantage of this observation, only a small proportion of pixels is necessary to be involved in the SAD calculation. Simulations show that the amount of computations required in the full search algorithm is reduced by about 58% on average and up to 70% in the best case. Secondly, from the scope of parallelization an enhanced TZ search for HEVC is proposed using novel schemes of multiple MVPs (motion vector predictor) and shared MVP. Specifically, resorting to multiple MVPs the initial search process is performed in parallel at multiple search centers, and the ME processing engine for PUs within one CU are parallelized based on the MVP sharing scheme on CU (coding unit) level. Moreover, the SAD module for ME engine is also parallelly implemented for PU size of 32×32. Experiments indicate it achieves an appreciable improvement on the throughput and coding efficiency of the HEVC video encoder. In addition, the other part of this thesis is contributed to the VLSI architecture design for finding the first W maximum/minimum values targeting towards high speed and low hardware cost. The architecture based on the novel bit-wise AND scheme has only half of the area of the best reference solution and its critical path delay is comparable with other implementations. While the FPCG (full parallel comparison grid) architecture, which utilizes the optimized comparator-based structure, achieves 3.6 times faster on average on the speed and even 5.2 times faster at best comparing with the reference architectures. Finally the architecture using the partial sorting strategy reaches a good balance on the timing performance and area, which has a slightly lower or comparable speed with FPCG architecture and a acceptable hardware cost

As the upcoming video coding standard, Versatile Video Coding (i.e., VVC) achieves up to 30% Bjøntegaard delta bit-rate (BD-rate) reduction compared with High Efficiency Video Coding (H.265/HEVC). To eliminate or alleviate different kinds of compression artifacts like blocking, ringing, blurring and contouring effects, three in-loop filters, i.e. de-blocking filter (DBF), sample adaptive offset (SAO) and adaptive loop filter (ALF), have been involved in VVC. Recently, Convolutional Neural Network (CNN) has attracted tremendous attention and shows great potential in many tasks in image processing. In this work, we design a CNN-based in-loop filter as an integrated single-model solution which is adaptive to almost any scenarios in video coding. An architecture named as ADCNN (i.e., Attention based Dual-scale CNN) with an attention based processing block is proposed to reduce artifacts of I frames and B frames, which take advantage of informative priors such as the quantization parameter (QP) and partitioning information. Different from existing CNN-based filtering methods, which are mainly designed for the luma component and may need to train different models for different QPs, the proposed filter is adapted to different QPs and different frame types, and all the components (i.e., both luma and chroma) are processed simultaneously with feature exchange and fusion between components for information supplementary. Experimental results show that the proposed ADCNN filter can achieve 6.54%, 13.27%, 15.72% BD-rate savings for Y, U, V respectively under the all intra configuration and 2.81%, 7.86%, 8.60% BD-rate savings under the random access configuration. It can be used to replace all the conventional in-loop filters and also outperforms them without increase in encoding time.