Efficient bit allocation and CTU level rate control for High Efficiency Video Coding

High-Efficiency Video Coding (HEVC) is the latest video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). An effective rate control mechanism plays a critical role in making the best possible use of the network resources. The coding performance can be maximised through the appropriate allocation of bits under the constraints of a total bit rate budget and the buffer size. Existing rate control strategies generally determine the rate control parameters according to previous encoding results. However, there is no priori-knowledge available when the first frame of a video sequence is encoded or scene changes occur. Consequently, it is challenging to choose appropriate rate control parameters in such cases. In order to address this issue, we investigated deep learning oriented methods and proposed a Convolutional Neural Network (CNN)-based rate control scheme for HEVC intra-coding. As the CNN has the advantage of local perception and the Coding Tree Unit's (CTU's) Quantisation Parameter (QP) value is determined during the CTU level rate control process, we applied the CNN at the CTU level as well to achieve an accurate QP prediction. An improved QP determination model (R-QP model) involving only two model coefficients (µ and v) was suggested to simplify the QP calculation process. A CNN was used to predict the model coefficients, namely µ and v. Our CNN has four convolution layers, two pooling layers and three fully connected layers. Each of the convolution layers contains a Rectified Linear Unit (ReLU). The last fully connected layer outputs the predicted values of µ and v. The images from the UCID dataset and the RAISE dataset were used to train the CNN. In order to obtain a universal model, none of the frames in the HEVC common test sequences were employed as training images. In order to achieve a reasonable bit budget allocation, a CNN-based framework was used to predict the bit consumption for each CTU according to the entire picture content that it contains. At the training stage, the network took the original images and the QP values as inputs, and the corresponding actual bit consumptions were used as the labels. Once the CNN coefficients have been obtained through training, the bit consumption for each CTU under different QP values can be directly predicted from the original images. The predicted bit consumption was subsequently used to readjust the bit budget for each CTU to achieve a more reasonable bit allocation. We implemented the proposed CNN-based rate control scheme in the HEVC reference software HM 16.9. As the priori-knowledge of the previous encoding results had not been considered in the proposed algorithm, only the first frame of each video sequence was encoded. The experimental results showed that our proposed scheme performs better than the default rate control implementation in the HEVC reference software, namely the BDBR is reduced by 1.33% on average while maintaining the same reconstructed picture quality. Compared with the state-of-the-art CNN-based rate control algorithm proposed by Li, our schemes achieved better coding efficiency. It can be concluded that the proposed R-QP model is effective in describing the rate-distortion relationship in HEVC, and the CNN-based bit allocation scheme is appropriate for ensuring that the best use of each bit is made.

S.1719-23

In this paper, a novel rate control scheme for low delay video coding of High Efficiency Video Coding (HEVC) is proposed. The proposed scheme is developed by considering a new temporal prediction structure of HEVC. In the proposed scheme, the relationship between bit rate and quantization step is exploited firstly to formulate an accurate quadratic rate-quantization (R-Q) model. Secondly, a method of determining the quantization parameters (QPs) for the first frames within a group of pictures is proposed. Thirdly, an accurate frame-level bit allocation method is proposed for HEVC. Finally, based on the proposed R-Q model and the target bit allocated for the frame, the QPs are predicted for coding tree units by using rate-distortion (R-D) optimization. We compare our scheme against that of three other state-of-the-art rate control schemes. Experimental results show that the proposed rate control scheme can increase the Bjøntegaard delta peak signal-to-noise ratio by 0.65 dB and 0.09 dB on average compared with the JCTVC-I0094 and JCTVC-M0036 schemes, respectively, both of which have been implemented in an HEVC test model encoder; furthermore, the proposed scheme achieves a similar R-D performance to Wang's scheme, as well as obtaining the smallest bit rate mismatch error of all the schemes.

The High Efficiency Video Coding (HEVC) standardization project mainly aims at improving the compression efficiency beyond the state-of-the-art H.264/AVC standard for high and ultra high definition video contents. The HEVC codec reference software, still under development, encodes video content using a fixed quantization step, thus leading to a variable bitrate stream which may not be suitable for the many multimedia applications where a constant bandwidth is required. Therefore, a rate control algorithm must be integrated in the HEVC codec taking into account that the novel coding tools further reduce the bitrate associated to texture data while increase the bitrate associated to the auxiliary information, e.g. motion vectors and coding modes. In this context, this paper studies the integration of a quadratic modeling rate control algorithm in the HEVC codec. Experimental results reveal that the rate control enabled HEVC codec provides a lower rate-distortion performance regarding a fixed quantization step HEVC codec, notably spending up to 20.12% more bitrate for the same objective quality. However, the integrated rate control algorithm offers lower bitrate fluctuations along the video contents than the HEVC codec with fixed quantization step.

This paper proposes an efficient binary representation of delta Quantization Parameter (QP) for High Efficiency Video Coding (HEVC). Video encoders adapt QPs of coding blocks for visual quality optimization and rate control. Although they send only delta QPs (dQPs) obtained by causal prediction, the side information overhead is expensive. Therefore the HEVC design necessitates an efficient dQP coding. The proposed scheme converts a dQP to a binary string in which the first and second bins indicate the significance and sign of the dQP respectively and the rest represents the magnitude minus 1. Furthermore, it detects and truncates redundant bins in the binary strings by using the sign and an admissible dQP range. Thus it reduces the length of dQP binary strings and improves dQP coding efficiency. Simulation results using HEVC reference software demonstrate that the proposed scheme improves the dQP coding efficiency by 6% while reducing its bin rates by 25%.

It has been shown that the quality of multimedia video and the need of real time video transmission become widely increased in these days and high efficiency video coding (HEVC) becomes the newest standard. Based on the theory of ratedistortion coding, the R-X model can describe the rate-distortion relationship more accurately in comparison with the existing traditional R-Q model. With the development of screen content coding, this paper proposes a new adaptive intra frame complexity measure method based on two complexity measure methods with a relative coefficient for intra bit rate control, respectively are the gradient based complexity measure and edge based complexity measure. Then the paper proposes an improved rate control method for the intra-frame rate control, which merges the adaptive intra frame complexity measure with R-X model based on HEVC. Along with this new adaptive rate control approach, it will show improved accuracy in rate control and enhanced performance in video coding which means better video quality. The proposed scheme can reduce the frame bit mismatch ratio by 2.8175% on average and the better B-D Rate which is 0.94% on average. Experimental results verify the effectiveness of the proposed intra frame complexity method based on X-domain rate control algorithm.

In this paper, a joint machine learning and game theory modeling (MLGT) framework is proposed for inter frame coding tree unit (CTU) level bit allocation and rate control (RC) optimization in high efficiency video coding (HEVC). First, a support vector machine-based multi-classification scheme is proposed to improve the prediction accuracy of CTU-level rate-distortion (R-D) model. The legacy "chicken-and-egg" dilemma in video coding is proposed to be overcome by the learning-based R-D model. Second, a mixed R-D model-based cooperative bargaining game theory is proposed for bit allocation optimization, where the convexity of the mixed R-D model-based utility function is proved, and Nash bargaining solution is achieved by the proposed iterative solution search method. The minimum utility is adjusted by the reference coding distortion and frame-level quantization parameter (QP) change. Finally, intra frame QP and inter frame adaptive bit ratios are adjusted to make inter frames have more bit resources to maintain smooth quality and bit consumption in the bargaining game optimization. Experimental results demonstrate that the proposed MLGT-based RC method can achieve much better R-D performances, quality smoothness, bit rate accuracy, buffer control results, and subjective visual quality than the other state-of-the-art one-pass RC methods, and the achieved R-D performances are very close to the performance limits from the FixedQP method.

Complexity reduction is a commonly used method to deal with complicated video coding standards, such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). But the unstable performance under different video contents and Quantization Parameters (QPs) makes it difficult to precisely specify the target encoding time of every single sequence, which limits the practical use of the encoder. Inspired by rate control, in this paper, encoding time budget is regarded as a resource. We incorporate the hierarchical Group of Picture (GOP) encoding structure, the non-accelerated proportion, and block content variation within the frame, and design a top-down allocation and bottom-up feedback scheme, to achieve precise control of encoding complexity by controlling the maximum depth of QuadTree with nested Multi-type Tree (QTMT). In the scheme, Temporal ID (Tid)-based, Sum of Absolute Transformed Difference (SATD)-based methods and Linear (L) Model are designed to facilitate weighted allocation and feedback. The relationship between Planar Cost and encoding time is exploited as a Time-Cost (T-C) model, which guides the selection of Largest Coding Unit (LCU) encoding strategies in I-frames. For B-frames, we investigate a switching suppression assisted status-based method, to efficiently decide the encoding strategy of each LCU. Through the collaboration of the proposed technologies in the scheme, given any encoding time target achievable, encoding strategies will be automatically switched to help the actual encoding time gradually approach the target. To the best of our knowledge, this work is the first work on VVC-based encoding complexity control. The proposed scheme supports directly specifying the target encoding time or target Frame Per Second (FPS), and accurately realizing it within one pass. According to experimental results, under the target encoding time ratio of 80%, 60% and 40%, the average encoding time error is kept under 0.24%, 0.03% and 0.02% by our method, outperforming state-of-the-art works base on HEVC. These results prove the advantage of the proposed scheme.

This paper presents a novel rate control framework for H.264/Advanced Video Coding-based video coding that improves the preservation of gradient-based features like scale-invariant feature transform or speeded up robust feature compared with the default rate control algorithm in the JM reference software. First, a criterion (matching score) for feature preservation on the basis of the bag-of-features concept is proposed. Then, the matching scores are collected as a function of the quantization parameters and analyzed for different feature types. With this analysis, macroblocks are categorized into different groups before encoding. Our rate control algorithm assigns different quantization parameters to each group according to the importance of the group for feature extraction. The experimental results show that our rate control algorithm achieves the desired target bit rate, and more features are preserved compared with videos encoded using the default rate control. The proposed approach not only improves feature preservation, but also leads to a noticeable performance improvement in a real image retrieval system. The rate control framework proposed in this paper is fully standard compatible.

Consistent video quality is important for video coding applications, which is also a popular optimization target for rate control. In this paper, a rate control scheme is proposed to reduce the fluctuation of video quality. First, we set up the optimization formulation of distortion and derive the distortion model by analyzing quadtree-based coding unit (CU) structure in high efficiency video coding (HEVC). Then the frame bit allocation algorithm is proposed by considering the inter-dependency. After the frame basic quantization parameter (QP) is obtained, the solution of optimization formulation finally regulates QP to maintain the consistent video quality. Experimental results show that the proposed rate control scheme can reduce the fluctuation of video quality up to 92.9% than benchmark, where the average reduction is 74.1%.

In this paper, we present a pixel-wise unified rate quantization (R-Q) model for a low-complexity rate control on configurable coding units of high efficiency video coding (HEVC). In the case of HEVC, which employs hierarchical coding block structure, multiple R-Q models can be employed for the various block sizes. However, we found that the ratios of distortions over bits for all the blocks are a nearly constant because of employment of the rate distortion optimization technique. Hence, one relationship model between rate and quantization can be derived from the characteristic of similar ratios of distortions over bits regardless of block sizes. Thus, we propose the pixel-wise unified R-Q model for HEVC rate control working on the multi-level for all block sizes. We employ a simple leaky bucket model for bit control. The rate control based on the proposed pixel-wise unified R-Q model is implemented on HEVC test model 6.1 (HM6.1). According to the evaluation for the proposed rate control, the average matching percentage to target bitrates is 99.47% and the average PSNR degradation is 0.76 dB. Based on the comparative study, we found that the proposed rate control shows low bit fluctuation and good RD performance, compared to R-lambda rate control for long sequences.

This paper presents an efficient rate control algorithm based on estimation theory. We measure a conditional mean by estimating a joint probability density function (PDF) using Parzen's window. The training data can estimate the nonlinear rate-distortion (R-D) relationship between the quantization parameter (QP) and the bits spent for each macroblock depending on the sum of absolute differences (SAD). We increase the accuracy of this joint PDF by clustering the training data depending on the QP values. This localization helps the estimation process to be very accurate. Next we apply a mean operator to simplify the conditional mean estimation of the rate given the SAD and QP values. This information is stored into three look-up tables depending on picture types. We use these tables to find the optimal QP values in least-mean-square (LMS) sense for a given bit budget of the current frame. Simulation results show that the proposed algorithm outperforms the informative MPEG-4 rate control algorithm in terms of reproduced image quality and coding efficiency while requiring much less implementation complexity. Most of all it keeps the bit rate very close to the required bit-rate due to the accuracy of the conditional mean estimator that solves the nonlinear R-D function.

Rate control has been regarded as an indispensable video coding tool for virtually any application involving video transmission. With the advent of many flexible tools introduced in the current state-of-the-art High Efficiency Video Coding (HEVC) standard, previous Rate-Distortion (RD) models used for rate control become insufficiently accurate. To overcome this issue, a new RD model has been recently proposed based on a robust correspondence between the rate and Lagrange multiplier λ. However, existing methods based on this model tend to perform sub-optimally after the scene change. In this paper, a two-pass rate control method is proposed, targeting Ultra High Definition Television (UHDTV) applications. In the first pass, a fast encoder with a reduced set of coding tools is used to obtain the data used for rate allocation and model parameter initialisation utilised during the second pass. Multiple encoding steps required to derive this information are avoided with the proposed variable quantization parameter framework. Experimental evaluation showed that the proposed two-pass rate control method achieves on average 4.4% BD-rate loss, compared with variable bit-rate encoding. That significantly outperforms the state-of-the-art HEVC rate control method with an average BD-rate loss of 8.8%.

We present a region of interest (ROI) based rate control for H.263 compatible video conferencing. A face detection and tracking scheme with very low complexity is proposed for segmentation. By analyzing quadratic rate models in frame layer, video object plane (VOP) layer, and macroblock (MB) layer extracted from the test data, a quadratic rate model at the MB layer with a modified physical meaning is proposed to improve the model accuracy. The basic idea is to use a group of uncoded MBs in the current VOP instead of individual MBs to update model parameters. A joint VOP layer and MB layer rate control algorithm is proposed. The VOP layer rate control assigns target bit rate for each VOP based on the coding complexity and visual importance, and determines an average quantization parameter (QP) for each VOP. Some new features of MB layer rate control are designed to utilize both average statistics of a VOP and individual statistics of MB together. The performance is compared with conventional TMN8 and object-based VM8 rate control, better peak SNR (PSNR) for ROI, and more accurate rate control can be achieved for various video sequences. The proposed rate control algorithm can be extended for H.264 ROI-based scalable video coding.

Due to the limited storage capacity and network bandwidth, an efficient rate control (RC) algorithm becomes more and more critical for multi-view video coding (MVC). Based on inter-view dependency and spatio-temporal correlation, a novel bit allocation method for multi-view texture video coding is proposed in this article. Firstly, considering that the distortion in the base view (BV) is directly transmitted to the dependent view (DV) by inter-view skip mode, a joint multi-view RD model is built based on the inter-view dependency. Based on the proposed joint multi-view RD model, a precise power model is derived to represent the target bitrates relationship between the BV and the DV. Secondly, since the P frame in the DV (P-DV) is mainly predicted from the corresponding I frame in the BV (I-BV) by disparity compensated prediction (DCP), the constant proportional relationship between the ratio of the average bitrates of the P-DV to the corresponding I-BV and the ratio of the total bitrates of the DV to the BV is discovered. Based on this discovery, a novel linear model is developed to assign the target bitrates of the P-DV. Finally, considering the spatio-temporal correlation, a new parameter prediction method is proposed for the R-$\lambda $ model in coding tree unit (CTU) level. Extensive experimental results show that the proposed overall method outperforms other state-of-the-art algorithms in terms of RD performance.