Compressed-SDR to HDR Video Reconstruction.

HDR video quality assessment: Perceptual evaluation of compressed HDR video

The migration from High Definition (HD) TV to Ultra High Definition (UHD) is already underway. In addition to an increase of picture spatial resolution, UHD will bring more color and higher contrast by introducing Wide Color Gamut (WCG) and High Dynamic Range (HDR) video. As both Standard Dynamic Range (SDR) and HDR devices will coexist in the ecosystem, the transition from Standard Dynamic Range (SDR) to HDR will require distribution solutions supporting some level of backward compatibility. This paper presents a new HDR content distribution scheme, named SL-HDR1, using a single layer codec design and providing SDR compatibility. The solution is based on a pre-encoding HDR-to-SDR conversion, generating a backward compatible SDR video, with side dynamic metadata. The resulting SDR video is then compressed, distributed and decoded using standard-compliant decoders (e.g. HEVC Main 10 compliant). The decoded SDR video can be directly rendered on SDR displays without adaptation. Dynamic metadata of limited size are generated by the pre-processing and used to reconstruct the HDR signal from the decoded SDR video, using a post-processing that is the functional inverse of the pre-processing. Both HDR quality and artistic intent are preserved. Pre- and post-processing are applied independently per picture, do not involve any inter-pixel dependency, and are codec agnostic. Compression performance, and SDR quality are shown to be solidly improved compared to the non-backward and backward-compatible approaches, respectively using the Perceptual Quantization (PQ) and Hybrid Log Gamma (HLG) Opto-Electronic Transfer Functions (OETF).

Currently, high dynamic range (HDR) videos with high resolution (HR) have become popular due to the display and the rendered technological advancements. However, making ultra-high definition (UHD) with HDR videos is expensive. The legacy low-resolution (LR) standard dynamic range (SDR) format is still largely used in practice. It is necessary to search for a solution to transform LR SDR videos into UHD HDR format. In this paper, we consider joint super resolution and learning inverse tone mapping an issue of high-frequency reconstruction and local contrast enhancement, and we propose an architecture based on a generative adversarial network to apply joint SR-ITM learning. Specifically, we include the residual ResNeXt block (RRXB) as a basic module to better capture high-frequency textures and adopt YUV interpolation to achieve local contrast enhancement. By adopting a generative adversarial network as a pivotal training mechanism, our designs show advantages in both integration and performance. Our code is now available on GitHub: SR-ITM-GAN.

We conducted a large-scale study of human perceptual quality judgments of High Dynamic Range (HDR) and Standard Dynamic Range (SDR) videos subjected to scaling and compression levels and viewed on three different display devices. While conventional expectations are that HDR quality is better than SDR quality, we have found subject preference of HDR versus SDR depends heavily on the display device, as well as on resolution scaling and bitrate. To study this question, we collected more than 23,000 quality ratings from 67 volunteers who watched 356 videos on OLED, QLED, and LCD televisions, and among many other findings, observed that HDR videos were often rated as lower quality than SDR videos at lower bitrates, particularly when viewed on LCD and QLED displays. Since it is of interest to be able to measure the quality of videos under these scenarios, e.g. to inform decisions regarding scaling, compression, and SDR vs HDR, we tested several well-known full-reference and no-reference video quality models on the new database. Towards advancing progress on this problem, we also developed a novel no-reference model called HDRPatchMAX, that uses a contrast-based analysis of classical and bit-depth features to predict quality more accurately than existing metrics.

High dynamic range (HDR) television promises a qualitatively better viewing experience, but to fully realize its potential and to achieve consistent high quality images, the psychovisual aspects of human vision must be taken into account. This paper reviews the nonlinearities inherent in conventional standard dynamic range (SDR) television and discusses how these are required to accommodate psychovisual effects. The thresholds at which changes in brightness can be detected, embodied in Weber's and De Vries-Rose's laws, are discussed in the context of the need to minimize “banding” artifacts due to quantization. This effect also provides a simple psychovisual definition for the dynamic range of video. The second important psychovisual effect considered is that of surround brightness on the perception of emissive images in dim or dark surroundings. This effect is accommodated by implementing an overall system nonlinearity, the rendering intent, which is typically a gamma (power law) function. The appropriate gamma exponent is considered in special cases and interpolated to provide a general formula. Based on these psychovisual effects, an HDR television system is proposed that supports the highest quality HDR images with a simultaneous dynamic range substantially beyond the limits of the human visual system in a single adaptation state. It is defined by a single opto-electronic transfer function (OETF), with a variable display electro-optic transfer function (EOTF) to accommodate the eye's adaptation. It is also compatible with existing, SDR, displays and infrastructure and requires no metadata.

High dynamic range (HDR) video compression has until now been approached by using the high profile of existing state-of-the-art H.264/AVC (Advanced Video Coding) codec or by separately encoding low dynamic range (LDR) video and the residue resulted from the estimation of HDR video from LDR video. Although the latter approach has a distinctive advantage of providing backward compatibility to 8-bit LDR displays, the superiority of one approach to the other in terms of the rate distortion trade-off has not been verified yet. In this paper, we first give a detailed overview of the methods in these two approaches. Then, we experimentally compare two approaches with respect to different objective and perceptual metrics, such as HDR mean square error (HDR MSE), perceptually uniform peak signal to noise ratio (PU PSNR) and HDR visible difference predictor (HDR VDP). We first conclude that the optimized methods for backward compatibility to 8-bit LDR displays are superior to the method designed for high profile encoder both for 8-bit and 12-bit mappings in terms of all metrics. Second, using higher bit-depths with a high profile encoder is giving better rate-distortion performances than employing an 8-bit mapping with an 8-bit encoder for the same method, in particular when the dynamic range of the video sequence is high. Third, rather than encoding of the residue signal in backward compatible methods, changing the quantization step size of the LDR layer encoder would be sufficient to achieve a required quality. In other words, the quality of tone mapping is more important than residue encoding for the performance of HDR image and video coding.

High dynamic range (HDR) video compression technology, which is capable of delivering a wider range of luminance and a larger colour gamut than standard dynamic range (SDR) technology, has been widely used in recent years in many fields, including industrial image processing, digital entertainment, and machine vision. Rate control (RC) is of paramount importance to HDR compression and transmission; accordingly, an RC scheme for HDR in High Efficiency Video Coding (HEVC) is proposed in this paper. First, considering the HDR characteristics, we propose an HDR-Visual Difference Predictor (VDP)-2-based rate-distortion (R-D) model to improve the coding performance. Second, we directly utilize $\lambda $ rather than the bit rate in the optimization process to obtain the optimal solution. Finally, we propose a new model parameter estimation method to further reduce the RC errors. According to our experimental results, significant bit rate reductions in terms of HDR-VDP-2, the Video Quality Metric (VQM) and the mean peak-signal-to-noise ratio (mPSNR) can be achieved on average compared with the state-of-the-art algorithm used in HM16.19.

The ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group issued a Call for Proposals (CfP) on video compression with capability beyond HEVC in October 2017. The CfP considered three categories of content – Standard Dynamic Range, High Dynamic Range and Wide Colour Gamut (HDR/WCG), and 360° Omni-directional video. As a result of the CfP process, the development of a new video coding standard, named Versatile Video Coding (VVC), was initiated. The goal of this paper is to provide an overview of the CfP responses for the HDR/WCG category. The paper includes a summary of work leading to the development of the CfP, a presentation of the CfP results for the HDR/WCG category, and a description of the specific HDR/WCG technologies submitted to the CfP.

We introduce a novel feature set, which we call HDRMAX features, that when included into Video Quality Assessment (VQA) algorithms designed for Standard Dynamic Range (SDR) videos, sensitizes them to distortions of High Dynamic Range (HDR) videos that are inadequately accounted for by these algorithms. While these features are not specific to HDR, and also augment the equality prediction performances of VQA models on SDR content, they are especially effective on HDR. HDRMAX features modify powerful priors drawn from Natural Video Statistics (NVS) models by enhancing their measurability where they visually impact the brightest and darkest local portions of videos, thereby capturing distortions that are often poorly accounted for by existing VQA models. As a demonstration of the efficacy of our approach, we show that, while current state-of-the-art VQA models perform poorly on 10-bit HDR databases, their performances are greatly improved by the inclusion of HDRMAX features when tested on HDR and 10-bit distorted videos.

Abstract. 3D reconstruction relies on accurate detection, extraction, description and matching of image features. This is even truer for complex architectural scenes that pose needs for 3D models of high quality, without any loss of detail in geometry or color. Illumination conditions influence the radiometric quality of images, as standard sensors cannot depict properly a wide range of intensities in the same scene. Indeed, overexposed or underexposed pixels cause irreplaceable information loss and degrade digital representation. Images taken under extreme lighting environments may be thus prohibitive for feature detection/extraction and consequently for matching and 3D reconstruction. High Dynamic Range (HDR) images could be helpful for these operators because they broaden the limits of illumination range that Standard or Low Dynamic Range (SDR/LDR) images can capture and increase in this way the amount of details contained in the image. Experimental results of this study prove this assumption as they examine state of the art feature detectors applied both on standard dynamic range and HDR images.

Rate-distortion optimized layered coding of high dynamic range videos

There are an increasing number of databases describing subjective quality responses for HDR (high dynamic range) imagery with various distortions. The dominant distortions across the databases are those that arise from video compression, which are primarily perceived as achromatic, but there are some chromatic distortions due to 422 and other chromatic sub-sampling. Tone mapping from the source HDR levels to various levels of reduced capability SDR (standard dynamic range) are also included in these databases. While most of these distortions are achromatic, tone-mapping can cause changes in saturation and hue angle when saturated colors are in the upper hull of the of the color space. In addition, there is one database that specifically looked at color distortions in an HDR-WCG (wide color gamut) space. From these databases we can test the improvements to well-known quality metrics if they are applied in the newly developed color perceptual spaces (i.e., representations) specifically designed for HDR and WCG. We present results from testing these subjective quality databases to computed quality using the new color spaces of Jzazbz and ICTCP, as well as the commonly used SDR color space of CIELAB.

High Dynamic Range (HDR) image and video technology aims at conveying the full range of perceptible shadow and highlight details with sufficient precision. HDR is regarded by many experts as the next evolution in digital media. However, industrial broadcasters have concerns regarding the bandwidth overhead that this new technology entails. While many consider that broadcasting HDR content would increase bandwidth requirements by around 20%, this number is based on studies where, in addition to the SDR main stream, HDR-related side information is conveyed. A recent subjective evaluation reported that encoding HDR video content in a single layer might require less bandwidth than its associated SDR version. Similar results were discussed in the MPEG ad-hoc group on High Dynamic Range and Wide Color Gamut. In this article, we explain how having more information can result in lower bandwidth requirements. To this end, we describe several limitations of the human vision system that, when exploited, optimize the HDR distribution pipeline for a human observer. Our theoretical assumption about the higher efficiency of HDR is backed up by a statistical analysis of pixel distribution in real images. The Spatial Index objective metric also reconfirms our assumption.

There is a rapid increase in the amount of High Dynamic Range (HDR) display devices that are able to offer significantly better viewing experience with higher contrast and richer colors. However, there is a considerable shortage of content that is able to offer the much-sought-after HDR experience. One approach is to capture new content using specialized HDR-cameras, which are typically fewer and costlier than the conventional cameras. Another approach is to convert existing Standard Dynamic Range (SDR) content into HDR artificially, such that, it can reasonably mimic the viewing experience that a true HDR content would have offered. With the latter approach, we propose a Gaussian Process Regression (GPR) based machine learning method for estimating HDR content from their SDR counterparts. GPR is known as a powerful technique for estimating continuous real-valued functions. Given a set of training SDR-HDR image pairs, our proposed method is able to estimate the reverse tone mapping function that is used to convert SDR signal into its HDR equivalent. Preliminary experimental results indicate that our approach produces visually pleasing HDR images.

Most Convolution Neural Network (CNN) based object detectors, to date, have been optimized for accuracy and/or detection performance on datasets typically comprised of well exposed 8-bits/pixel/channel Standard Dynamic Range (SDR) images. A major existing challenge in this area is to accurately detect objects under extreme/difficult lighting conditions as SDR image trained detectors fail to accurately detect objects under such challenging lighting conditions. In this paper, we address this issue for the first time by introducing High Dynamic Range (HDR) imaging to object detection. HDR imagery can capture and process ≈13 orders of magnitude of scene dynamic range similar to the human eye. HDR trained models are therefore able to extract more salient features from extreme lighting conditions leading to more accurate detections. However, introducing HDR also presents multiple new challenges such as the complete absence of resources and previous literature on such an approach. Here, we introduce a methodology to generate a large scale annotated HDR dataset from any existing SDR dataset and validate the quality of the generated dataset via a robust evaluation technique. We also discuss the challenges of training and validating HDR trained models using existing detectors. Finally, we provide a methodology to create an out of distribution (OOD) HDR dataset to test and compare the performance of HDR and SDR trained detectors under difficult lighting condition. Results suggest that using the proposed methodology, HDR trained models are able to achieve 10 – 12% more accuracy compared to SDR trained models on real-world OOD dataset consisting of high-contrast images under extreme lighting conditions.