Decoder-Side Intra Mode Derivation Based on a Histogram of Gradients in Versatile Video Coding

Telehealth applications, such as remote diagnosis and examination, have become more and more popular nowadays. However, the generated large number of medical images and their impractical digital size when it comes to telehealth have brought pressure on communication infrastructures. More specifically, the image processing time has increased dramatically due to the size of the digitized images, while requiring substantial transmission bandwidth and storage space. For these reasons, medical image compression has become a hot research topic. While image compression techniques have evolved over several generations, it remains an open question how these standards will perform when it comes to sensitive oversized medical image content. In this paper, the emerging compression standard High Efficiency Video Coding (HEVC) and the next generation standard Versatile Video Coding (VVC) are evaluated on two different medical image datasets, covering the entire range of quality levels. The results of this study show that the VVC standard overperforms HEVC on raw medical image compression both visually and quantitatively, with the performance gap narrowing when dealing with pre-compressed medical images.

The growing popularity of high resolution video and the continuously increasing demands for high quality video on mobile devices are producing stronger needs for more efficient video encoder. Concerning these desires, HEVC, a newest video coding standard, has been developed by a joint team formed by ISO/IEO MPEG and ITU/T VCEG. Its design goal is to achieve a 50% compression gain over its predecessor H.264 with an equal or even higher perceptual video quality. Motion Estimation (ME) being as one of the most critical module in video coding contributes almost 50%-70% of computational complexity in the video encoder. This high consumption of the computational resources puts a limit on the performance of encoders, especially for full HD or ultra HD videos, in terms of coding speed, bit-rate and video quality. Thus the major part of this work concentrates on the computational complexity reduction and improvement of timing performance of motion estimation algorithms for HEVC standard. First, a new strategy to calculate the SAD (Sum of Absolute Difference) for motion estimation is designed based on the statistics on property of pixel data of video sequences. This statistics demonstrates the size relationship between the sum of two sets of pixels has a determined connection with the distribution of the size relationship between individual pixels from the two sets. Taking the advantage of this observation, only a small proportion of pixels is necessary to be involved in the SAD calculation. Simulations show that the amount of computations required in the full search algorithm is reduced by about 58% on average and up to 70% in the best case. Secondly, from the scope of parallelization an enhanced TZ search for HEVC is proposed using novel schemes of multiple MVPs (motion vector predictor) and shared MVP. Specifically, resorting to multiple MVPs the initial search process is performed in parallel at multiple search centers, and the ME processing engine for PUs within one CU are parallelized based on the MVP sharing scheme on CU (coding unit) level. Moreover, the SAD module for ME engine is also parallelly implemented for PU size of 32×32. Experiments indicate it achieves an appreciable improvement on the throughput and coding efficiency of the HEVC video encoder. In addition, the other part of this thesis is contributed to the VLSI architecture design for finding the first W maximum/minimum values targeting towards high speed and low hardware cost. The architecture based on the novel bit-wise AND scheme has only half of the area of the best reference solution and its critical path delay is comparable with other implementations. While the FPCG (full parallel comparison grid) architecture, which utilizes the optimized comparator-based structure, achieves 3.6 times faster on average on the speed and even 5.2 times faster at best comparing with the reference architectures. Finally the architecture using the partial sorting strategy reaches a good balance on the timing performance and area, which has a slightly lower or comparable speed with FPCG architecture and a acceptable hardware cost

In spite of the recent advances in telecommunication standards, communication networks still have limited bandwidths and storage capacity. Therefore, video compression has drawn increasing importance since high-resolution video contents have become more and more used in various fields. These requirements raise the need for high-performance video-compression technologies able to reduce the amount of data to be transmitted or stored by compressing the input video signal into a bitstream file. Improving the coding efficiency was always one of the crucial issues of various compression standards that aim to get the most compact representation of the reconstructed video, with a high subjective quality. The high-efficiency video coding (HEVC) comes to respond to these requirements. However, the increased consumption of high-quality multimedia content has pushed the international communication companies to put much effort to better enhance video-coding techniques. In this perspective, an upcoming video-coding standard to be known as versatile video coding (VVC) has emerged aiming to improve the coding efficiency of the current HEVC codec. Improvements on rate distortion (RD) performance that came with both HEVC and VVC have brought an increased complexity in the majority of the coding modules, which makes it difficult to implement on hardware systems with real-time encoding. This chapter focuses on the transform coding stage as one of the most computationally demanding modules. For HEVC, efficient approximation algorithms in addition to reconfigurable and scalable architecture have been developed in order to decrease the computational complexity of the transform module. The main objectives are to meet low power and real-time processing constraints while maintaining a compression gain and a satisfying video quality. Field programmable gate array (FPGA) implementation results and comparisons with existing works confirm the efficiency of the proposed approximations since they contribute in reducing time and power consumption, optimizing the hardware resources and bringing peak signal-to noise ratio (PSNR) improvement as well. Similarly, for the adaptive multiple transform (AMT) introduced in the transform module of the VVC, approximations were done for discrete cosine transform (DCT)-II and discrete sine transform (DST)-VII transforms since they are statistically the most used ones among the five predefined types. Bitrate (BR) reduction with a slightly degradation of video quality and a less use of hardware resources are the main contributions of the proposed approximations.

The latest video coding standard High Efficiency Video Coding (HEVC) is developed by Joint Collaborative Team on Video Coding(JCTVC). HEVC can achieve better compression performance than any other coding standards, which adopts the quadtree structured coding tree unit (CTU). In encoding process, the CTU is recursively split into four equally sized coding unit (CU)s. Along with the multiple sizes of the CUs, the number of prediction unit(PU) modes has been largely increased, which raises the computational complexity in HEVC encoder greatly. In this paper, a fast inter mode decision algorithm based on motion estimation (ME) and texture feature is proposed to speed up the PU mode decision in HEVC encoder. Experimental results demonstrate that the overall algorithm that is proposed can save 41% encoding time on average while the decrease of coding quality is negligible, when compared to that of the HM10.0 reference software.

Cross residual transform for lossless intra-coding for HEVC

The audiovisual entertainment industry has entered a race to find the video encoder offering the best Rate/Distortion (R/D) performance for high-quality high-definition video content. The challenge consists in providing a moderate to low computational/hardware complexity encoder able to run Ultra High-Definition (UHD) video formats of different flavours (360°, AR/VR, etc.) with state-of-the-art R/D performance results. It is necessary to evaluate not only R/D performance, a highly important feature, but also the complexity of future video encoders. New coding tools offering a small increase in R/D performance at the cost of greater complexity are being advanced with caution. We performed a detailed analysis of two evolutions of High Efficiency Video Coding (HEVC) video standards, Joint Exploration Model (JEM) and Versatile Video Coding (VVC), in terms of both R/D performance and complexity. The results show how VVC, which represents the new direction of future standards, has, for the time being, sacrificed R/D performance in order to significantly reduce overall coding/decoding complexity.

Scalable High efficiency Video Coding (SHVC) is the scalable extension of the latest video coding standard High Efficiency Video Coding (HEVC). One of the key novelties introduced by SHVC is that it enables hybrid codec scalability. This basically means that the video layers can be encoded with different video standards providing backward compatibility between codecs. In this paper, we propose a software parallel SHVC decoder in hybrid codec scalability configuration. The proposed design consists of an Advanced Video Coding (AVC) decoder for the Base Layer (BL) and a HEVC decoder for the Enhanced Layer (EL). In order to perform Inter Layer Prediction (ILP), a communication of decoding states and outputs is established between the two decoders. While the native frame based parallelism is still allowed within the two decoders, the proposed design also enables the use of frame based parallelism between the two decoders. The proposed software design enables a real time decoding of the HEVC EL at 2160p60 while the AVC base layer is decoded at 1080p60 for ×2 spatial scalability.

H.264/AVC is regarded as a popular video coding standard, and is widely used in multimedia applications. However, with an increasing demand for better quality videos, high efficiency video coding (HEVC) is all set to serve as the successor to H.264/AVC for higher resolution video applications. Since a majority of the multimedia devices have already been operating based on the H.264/AVC standard, it may not be worthwhile to completely replace the existing software and hardware components by different modules in order to adopt HEVC in such devices. Need is therefore felt to design a decoder for supporting H.264/AVC as well as HEVC, rather than attempting individual designs. This paper introduces a new dual‐standard deblocking filter architecture, which supports both H.264/AVC and HEVC standards. Algorithmic verification has been done in Matlab and then an appropriate VLSI architecture has been implemented on FPGA as well as in ASIC domain. The proposed architecture takes 26 clock cycles for H.264/AVC and 14 cycles for HEVC to complete the filtering of a 16 × 16 pixel block. It consumes 5.80 mW normalised power and occupies an area equivalent to 70.1k equivalent gate at frequency of 100 MHz. The proposed architecture takes 8.42 ms to filter the 4K ultra high definition (UHD) (3840 × 2160) frame in H.264 standard, and it takes 18 ms to filter the 8K UHD (7680 × 4320) frame in HEVC standard.

Versatile Video Coding (VVC) is the emerging video coding standard to be finalized by the Joint Video Experts Team in July 2020. Compared to its predecessor, the High Efficiency Video Coding (HEVC) standard, VVC provides 50% bit-rate reduction at comparable visual quality for natural video content in high-definition (HD) and ultra high-definition (UHD) resolution. To achieve this, the standard incorporates more advanced and generalized algorithms, leading to an increase in computational complexity. This includes for example additional in-loop filters, decoder-side motion refinement and search as well as an increased number of transforms, which creates a grand challenge for implementers to achieve live decoding on general-purpose CPUs. In this paper, the work on an efficient software decoder implementation for the upcoming VVC standard is described, including optimization of sample operations using single instruction multiple data (SIMD) instructions and parallelization approaches with multithreading. As a result, the presented decoder can perform live decoding of 10bit HD video at 60 frames per second (fps) and 10bit UHD video at 30fps on modern mobile consumer hardware, showcasing that VVC live decoding is possible already right before finalization of the standard.

Increasing the prediction modes and recursive quad-tree structure, can be the reasons for high compression efficiency in high-efficiency video coding (HEVC) standard. For the same reasons, the computational complexity of HEVC is increased compared with the previous standards. One of the parts that impose a high computational load is the intraprediction unit. In this paper, an improved intraprediction mode decision method is presented to accelerate intracoding of HEVC. A mass center direction-based approach is used to find correlation direction of pixels which can effectively eliminate the prediction modes that have low chance to be selected as the best intramode from the rate-distortion optimization computations. According to the founded correlation direction for 4 × 4 blocks, the depth range of coding units can also be reduced to accelerate the intracoding further. The performance of the proposed method evaluated with different test sequences proposed by the joint collaborative team on video coding. The coding results indicate that the proposed method reduces the encoding time significantly as compared with the HM-16.19 reference software with negligible loss of peak signal-to-noise ratio quality.

Since MPEG and VCEG jointly standardized the H.265/HEVC (High-Efficiency Video Coding) international standard in 2013, they have just completed H.266/VVC (Versatile Video Coding) [1] as the Final Draft International Standard (FDIS) in July 2020 through the Joint Video Experts Team (JVET) of ISO / IEC JTC1 / SC29 / WG11 MPEG and ITU-T SG16 / Q6 VCEG. VVC supports up to 87 intra coding modes including 65 general directional modes and 20 wide angular modes, which is increased more than twice compared to HEVC. VVC can accommodate not only more detailed intra prediction directions, but also so called wide angular modes which make its special sense in non-square coding blocks. The more detailed intra predictions naturally demand more computation in order to determine the most effective intra-prediction mode. In this sense, we investigate how to pre-prune the prediction candidate lists for a fast ISP intra mode decision and propose searching only a small number of prediction modes based on the shape of a block in the rate-distortion optimization (RDO)–based intra mode decision. The proposed method is verified to decrease the intra prediction processing time with only a little increase in the bit rate and a negligible reduction in PSNR values.

In the latest video coding standard high efficiency video coding (HEVC), a quadtree-based coding unit (CU) partitioning scheme is adopted to better adapt to the characteristics of the video contents. However, the flexible scheme significantly increases the coding complexity, because large amount of possible CU partitioning modes should be traversed. In this paper, we propose a two-stage fast inter CU decision method to reduce the coding complexity of the HEVC encoders. In Stage I, all the CUs are classified into three categories based on the Bayesian method after the prediction unit (PU) mode $merge\,\,2N\times 2N$ is checked. Early CU pruning and early CU skipping are then applied to two of the categories, respectively. For the remaining category which is difficult to differentiate by the rate-distortion cost of the PU mode $merge\,\,2N\times 2N$ , an early CU pruning scheme based on conditional random fields is performed in Stage II, which takes both the local characteristics of the current CU and the coding information of its neighboring CUs into consideration. Experimental results show that our method can reduce 54.93% and 45.84% of the coding complexity on average with only 1.19% and 1.03% Bjontegaard delta bitrate increment under the random access main and the low delay P configurations, respectively.

With the rapid advancement in many multimedia applications, such as video gaming, computer vision applications, and video streaming and surveillance, video quality remains an open challenge. Despite the existence of the standardized video quality as well as high definition (HD) and ultrahigh definition (UHD), enhancing the quality for the video compression standard will improve the video streaming resolution and satisfy end user's quality of service (QoS). Versatile video coding (VVC) is the latest video coding standard that achieves significant coding efficiency. VVC will help spread high-quality video services and emerging applications, such as high dynamic range (HDR), high frame rate (HFR), and omnidirectional 360-degree multimedia compared to its predecessor high efficiency video coding (HEVC). Given its valuable results, the emerging field of deep learning is attracting the attention of scientists and prompts them to solve many contributions. In this study, we investigate the deep learning efficiency to the new VVC standard in order to improve video quality. However, in this work, we propose a wide-activated squeeze-and-excitation deep convolutional neural network (WSE-DCNN) technique-based video quality enhancement for VVC. Thus, the VVC conventional in-loop filtering will be replaced by the suggested WSE-DCNN technique that is expected to eliminate the compression artifacts in order to improve visual quality. Numerical results demonstrate the efficacy of the proposed model achieving approximately −2.85%, −8.89%, and −10.05% BD-rate reduction of the luma (Y) and both chroma (U, V) components, respectively, under random access profile.

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

Context-based adaptive binary arithmetic coding (CABAC) is a single entropy coding mode of the newest video coding standard, High Efficiency Video Coding (HEVC). Despite the high coding efficiency of CABAC, its data dependencies prevent effective parallelization, which also means limited throughput. Accordingly, during the HEVC standardization activity of entropy coding, both coding efficiency and throughput were considered. This paper highlights the key techniques that were proposed and adopted up to HEVC Draft International Standard (DIS). In addition, a new method is proposed for high throughput entropy coding. This method reduces the worst-case complexity of entropy coding without significant coding efficiency loss. From the experimental results, in terms of throughput improvement, we confirm that the proposed method reduces the number of context-coded bins up to 42.1 and 15.9 % on average.