SR-ITM-GAN: Learning 4K UHD HDR With a Generative Adversarial Network

HDR video quality assessment: Perceptual evaluation of compressed HDR video

The migration from High Definition (HD) TV to Ultra High Definition (UHD) is already underway. In addition to an increase of picture spatial resolution, UHD will bring more color and higher contrast by introducing Wide Color Gamut (WCG) and High Dynamic Range (HDR) video. As both Standard Dynamic Range (SDR) and HDR devices will coexist in the ecosystem, the transition from Standard Dynamic Range (SDR) to HDR will require distribution solutions supporting some level of backward compatibility. This paper presents a new HDR content distribution scheme, named SL-HDR1, using a single layer codec design and providing SDR compatibility. The solution is based on a pre-encoding HDR-to-SDR conversion, generating a backward compatible SDR video, with side dynamic metadata. The resulting SDR video is then compressed, distributed and decoded using standard-compliant decoders (e.g. HEVC Main 10 compliant). The decoded SDR video can be directly rendered on SDR displays without adaptation. Dynamic metadata of limited size are generated by the pre-processing and used to reconstruct the HDR signal from the decoded SDR video, using a post-processing that is the functional inverse of the pre-processing. Both HDR quality and artistic intent are preserved. Pre- and post-processing are applied independently per picture, do not involve any inter-pixel dependency, and are codec agnostic. Compression performance, and SDR quality are shown to be solidly improved compared to the non-backward and backward-compatible approaches, respectively using the Perceptual Quantization (PQ) and Hybrid Log Gamma (HLG) Opto-Electronic Transfer Functions (OETF).

The new generation of organic light emitting diode display is designed to enable the high dynamic range (HDR), going beyond the standard dynamic range (SDR) supported by the traditional display devices. However, a large quantity of videos are still of SDR format. Further, most pre-existing videos are compressed at varying degrees for minimizing the storage and traffic flow demands. To enable movie-going experience on new generation devices, converting the compressed SDR videos to the HDR format (i.e., compressed-SDR to HDR conversion) is in great demands. The key challenge with this new problem is how to solve the intrinsic many-to-many mapping issue. However, without constraining the solution space or simply imitating the inverse camera imaging pipeline in stages, existing SDR-to-HDR methods can not formulate the HDR video generation process explicitly. Besides, they ignore the fact that videos are often compressed. To address these challenges, in this work we propose a novel imaging knowledge-inspired parallel networks (termed as KPNet) for compressed-SDR to HDR (CSDR-to-HDR) video reconstruction. KPNet has two key designs: Knowledge-Inspired Block (KIB) and Information Fusion Module (IFM). Concretely, mathematically formulated using some priors with compressed videos, our conversion from a CSDR-to-HDR video reconstruction is conceptually divided into four synergistic parts: reducing compression artifacts, recovering missing details, adjusting imaging parameters, and reducing image noise. We approximate this process by a compact KIB. To capture richer details, we learn HDR representations with a set of KIBs connected in parallel and fused with the IFM. Extensive evaluations show that our KPNet achieves superior performance over the state-of-the-art methods.

We conducted a large-scale study of human perceptual quality judgments of High Dynamic Range (HDR) and Standard Dynamic Range (SDR) videos subjected to scaling and compression levels and viewed on three different display devices. While conventional expectations are that HDR quality is better than SDR quality, we have found subject preference of HDR versus SDR depends heavily on the display device, as well as on resolution scaling and bitrate. To study this question, we collected more than 23,000 quality ratings from 67 volunteers who watched 356 videos on OLED, QLED, and LCD televisions, and among many other findings, observed that HDR videos were often rated as lower quality than SDR videos at lower bitrates, particularly when viewed on LCD and QLED displays. Since it is of interest to be able to measure the quality of videos under these scenarios, e.g. to inform decisions regarding scaling, compression, and SDR vs HDR, we tested several well-known full-reference and no-reference video quality models on the new database. Towards advancing progress on this problem, we also developed a novel no-reference model called HDRPatchMAX, that uses a contrast-based analysis of classical and bit-depth features to predict quality more accurately than existing metrics.

High dynamic range (HDR) television promises a qualitatively better viewing experience, but to fully realize its potential and to achieve consistent high quality images, the psychovisual aspects of human vision must be taken into account. This paper reviews the nonlinearities inherent in conventional standard dynamic range (SDR) television and discusses how these are required to accommodate psychovisual effects. The thresholds at which changes in brightness can be detected, embodied in Weber's and De Vries-Rose's laws, are discussed in the context of the need to minimize “banding” artifacts due to quantization. This effect also provides a simple psychovisual definition for the dynamic range of video. The second important psychovisual effect considered is that of surround brightness on the perception of emissive images in dim or dark surroundings. This effect is accommodated by implementing an overall system nonlinearity, the rendering intent, which is typically a gamma (power law) function. The appropriate gamma exponent is considered in special cases and interpolated to provide a general formula. Based on these psychovisual effects, an HDR television system is proposed that supports the highest quality HDR images with a simultaneous dynamic range substantially beyond the limits of the human visual system in a single adaptation state. It is defined by a single opto-electronic transfer function (OETF), with a variable display electro-optic transfer function (EOTF) to accommodate the eye's adaptation. It is also compatible with existing, SDR, displays and infrastructure and requires no metadata.

High dynamic range (HDR) video compression has until now been approached by using the high profile of existing state-of-the-art H.264/AVC (Advanced Video Coding) codec or by separately encoding low dynamic range (LDR) video and the residue resulted from the estimation of HDR video from LDR video. Although the latter approach has a distinctive advantage of providing backward compatibility to 8-bit LDR displays, the superiority of one approach to the other in terms of the rate distortion trade-off has not been verified yet. In this paper, we first give a detailed overview of the methods in these two approaches. Then, we experimentally compare two approaches with respect to different objective and perceptual metrics, such as HDR mean square error (HDR MSE), perceptually uniform peak signal to noise ratio (PU PSNR) and HDR visible difference predictor (HDR VDP). We first conclude that the optimized methods for backward compatibility to 8-bit LDR displays are superior to the method designed for high profile encoder both for 8-bit and 12-bit mappings in terms of all metrics. Second, using higher bit-depths with a high profile encoder is giving better rate-distortion performances than employing an 8-bit mapping with an 8-bit encoder for the same method, in particular when the dynamic range of the video sequence is high. Third, rather than encoding of the residue signal in backward compatible methods, changing the quantization step size of the LDR layer encoder would be sufficient to achieve a required quality. In other words, the quality of tone mapping is more important than residue encoding for the performance of HDR image and video coding.

Rate-distortion optimized layered coding of high dynamic range videos

High dynamic range (HDR) video compression technology, which is capable of delivering a wider range of luminance and a larger colour gamut than standard dynamic range (SDR) technology, has been widely used in recent years in many fields, including industrial image processing, digital entertainment, and machine vision. Rate control (RC) is of paramount importance to HDR compression and transmission; accordingly, an RC scheme for HDR in High Efficiency Video Coding (HEVC) is proposed in this paper. First, considering the HDR characteristics, we propose an HDR-Visual Difference Predictor (VDP)-2-based rate-distortion (R-D) model to improve the coding performance. Second, we directly utilize $\lambda $ rather than the bit rate in the optimization process to obtain the optimal solution. Finally, we propose a new model parameter estimation method to further reduce the RC errors. According to our experimental results, significant bit rate reductions in terms of HDR-VDP-2, the Video Quality Metric (VQM) and the mean peak-signal-to-noise ratio (mPSNR) can be achieved on average compared with the state-of-the-art algorithm used in HM16.19.

We introduce a novel feature set, which we call HDRMAX features, that when included into Video Quality Assessment (VQA) algorithms designed for Standard Dynamic Range (SDR) videos, sensitizes them to distortions of High Dynamic Range (HDR) videos that are inadequately accounted for by these algorithms. While these features are not specific to HDR, and also augment the equality prediction performances of VQA models on SDR content, they are especially effective on HDR. HDRMAX features modify powerful priors drawn from Natural Video Statistics (NVS) models by enhancing their measurability where they visually impact the brightest and darkest local portions of videos, thereby capturing distortions that are often poorly accounted for by existing VQA models. As a demonstration of the efficacy of our approach, we show that, while current state-of-the-art VQA models perform poorly on 10-bit HDR databases, their performances are greatly improved by the inclusion of HDRMAX features when tested on HDR and 10-bit distorted videos.

High Dynamic Range (HDR) image and video technology aims at conveying the full range of perceptible shadow and highlight details with sufficient precision. HDR is regarded by many experts as the next evolution in digital media. However, industrial broadcasters have concerns regarding the bandwidth overhead that this new technology entails. While many consider that broadcasting HDR content would increase bandwidth requirements by around 20%, this number is based on studies where, in addition to the SDR main stream, HDR-related side information is conveyed. A recent subjective evaluation reported that encoding HDR video content in a single layer might require less bandwidth than its associated SDR version. Similar results were discussed in the MPEG ad-hoc group on High Dynamic Range and Wide Color Gamut. In this article, we explain how having more information can result in lower bandwidth requirements. To this end, we describe several limitations of the human vision system that, when exploited, optimize the HDR distribution pipeline for a human observer. Our theoretical assumption about the higher efficiency of HDR is backed up by a statistical analysis of pixel distribution in real images. The Spatial Index objective metric also reconfirms our assumption.

Abstract. 3D reconstruction relies on accurate detection, extraction, description and matching of image features. This is even truer for complex architectural scenes that pose needs for 3D models of high quality, without any loss of detail in geometry or color. Illumination conditions influence the radiometric quality of images, as standard sensors cannot depict properly a wide range of intensities in the same scene. Indeed, overexposed or underexposed pixels cause irreplaceable information loss and degrade digital representation. Images taken under extreme lighting environments may be thus prohibitive for feature detection/extraction and consequently for matching and 3D reconstruction. High Dynamic Range (HDR) images could be helpful for these operators because they broaden the limits of illumination range that Standard or Low Dynamic Range (SDR/LDR) images can capture and increase in this way the amount of details contained in the image. Experimental results of this study prove this assumption as they examine state of the art feature detectors applied both on standard dynamic range and HDR images.

As high dynamic range video is gaining popularity, video coding solutions able to efficiently provide both low and high dynamic range video, notably with a single bitstream, are increasingly important. While simulcasting can provide both dynamic range videos at the cost of some compression efficiency penalty, bit-depth scalable video coding can provide a better trade-off between compression efficiency, adaptation flexibility and computational complexity. Considering the widespread use of H.264/AVC video, this paper proposes a H.264/AVC backward compatible bit-depth scalable video coding solution offering a low dynamic range base layer and two high dynamic range enhancement layers with different qualities, at low complexity. Experimental results show that the proposed solution has an acceptable rate-distortion performance penalty regarding the HDR H.264/AVC single-layer coding solution.

The final consumer of videos is mostly human. Therefore, if videos can be compressed by fully utilizing the perception characteristics of human visual systems (HVS), the bitrates of the compressed videos can be significantly reduced with subjective visual quality degradation as little as possible. Based on this, we newly propose a learning-based Just Noticeable Distortion (JND)-directed preprocessing scheme for perceptual video compression, especially for 10-bit High Dynamic Range (HDR) videos, which is called the HDR-JNDNet. Our HDR-JNDNet effectively suppresses the perceptual redundancy of 10-bit HDR video signals so that the compression efficiency can be significantly enhanced for the HEVC main10 profile encoder. To our best knowledge, our work is the first approach to training a CNN-based model to directly generate the JND-directed suppressed frames of 10-bit HDR video with the negligible perceptual quality difference between the decoded frames for the original HDR video input with and without the preprocessing by our HDR-JNDNet. Via intensive experiments, when the HDR-JNDNet is applied as preprocessing for the HDR video input before compression, it allows to remarkably save the required bitrates up to the maximum (average) 40.66% (18.37%) for 4K-UHD/HDR test videos, with little subjective video quality degradation without increasing the computational complexity.

There is a rapid increase in the amount of High Dynamic Range (HDR) display devices that are able to offer significantly better viewing experience with higher contrast and richer colors. However, there is a considerable shortage of content that is able to offer the much-sought-after HDR experience. One approach is to capture new content using specialized HDR-cameras, which are typically fewer and costlier than the conventional cameras. Another approach is to convert existing Standard Dynamic Range (SDR) content into HDR artificially, such that, it can reasonably mimic the viewing experience that a true HDR content would have offered. With the latter approach, we propose a Gaussian Process Regression (GPR) based machine learning method for estimating HDR content from their SDR counterparts. GPR is known as a powerful technique for estimating continuous real-valued functions. Given a set of training SDR-HDR image pairs, our proposed method is able to estimate the reverse tone mapping function that is used to convert SDR signal into its HDR equivalent. Preliminary experimental results indicate that our approach produces visually pleasing HDR images.

High dynamic range (HDR) imaging technology has been widely implemented in digital microscopes for taking still images of high contrast specimens. However, capturing HDR microscopic video is much more challenging. In this paper, we present a HDR microscopic video system based on GPU accelerated computing. We show that by combining CPU and GPU computing, it is possible to build a stable HDR video system using a single off the shelf camera. We show that capturing multiple frames of different exposure intervals, aligning consecutive neighboring frames, constructing HDR radiance map and tone mapping the radiance map for display, can be realized by using GPU computing to accelerate the processing speed. We present experimental results to show the effectiveness of our system and how HDR video can reveal much more details than conventional videos.