Reverse Tone Mapping of High Dynamic Range Video Using Gaussian Process Regression

The migration from High Definition (HD) TV to Ultra High Definition (UHD) is already underway. In addition to an increase of picture spatial resolution, UHD will bring more color and higher contrast by introducing Wide Color Gamut (WCG) and High Dynamic Range (HDR) video. As both Standard Dynamic Range (SDR) and HDR devices will coexist in the ecosystem, the transition from Standard Dynamic Range (SDR) to HDR will require distribution solutions supporting some level of backward compatibility. This paper presents a new HDR content distribution scheme, named SL-HDR1, using a single layer codec design and providing SDR compatibility. The solution is based on a pre-encoding HDR-to-SDR conversion, generating a backward compatible SDR video, with side dynamic metadata. The resulting SDR video is then compressed, distributed and decoded using standard-compliant decoders (e.g. HEVC Main 10 compliant). The decoded SDR video can be directly rendered on SDR displays without adaptation. Dynamic metadata of limited size are generated by the pre-processing and used to reconstruct the HDR signal from the decoded SDR video, using a post-processing that is the functional inverse of the pre-processing. Both HDR quality and artistic intent are preserved. Pre- and post-processing are applied independently per picture, do not involve any inter-pixel dependency, and are codec agnostic. Compression performance, and SDR quality are shown to be solidly improved compared to the non-backward and backward-compatible approaches, respectively using the Perceptual Quantization (PQ) and Hybrid Log Gamma (HLG) Opto-Electronic Transfer Functions (OETF).

High Dynamic Range (HDR) imaging captures a wider range of luminance and color than Standard Dynamic Range (SDR), enabling more faithful representations of real-world scenes. Since most displays only support SDR, HDR content must be converted to SDR, often resulting in perceptual distortions like unnatural contrast, loss of detail or color artifacts. This paper evaluates No-Reference Image Quality Assessment (NR-IQA) methods on SDR images derived from HDR using the ESPL-LIVE HDR Image Database, which includes tone-mapped images and images created by multi-exposure fusion, together with associated mean opinion scores (MOS) and standard deviations, providing a reliable ground truth for evaluation. Seven NR-IQA methods, both traditional and Deep Learning (DL)-based, were tested and compared. DL algorithms performed best, with prediction errors below the standard deviation of MOS. Despite moderate correlation values (around 0.5), the results demonstrate the potential of these methods for practical image quality monitoring.

Abstract. 3D reconstruction relies on accurate detection, extraction, description and matching of image features. This is even truer for complex architectural scenes that pose needs for 3D models of high quality, without any loss of detail in geometry or color. Illumination conditions influence the radiometric quality of images, as standard sensors cannot depict properly a wide range of intensities in the same scene. Indeed, overexposed or underexposed pixels cause irreplaceable information loss and degrade digital representation. Images taken under extreme lighting environments may be thus prohibitive for feature detection/extraction and consequently for matching and 3D reconstruction. High Dynamic Range (HDR) images could be helpful for these operators because they broaden the limits of illumination range that Standard or Low Dynamic Range (SDR/LDR) images can capture and increase in this way the amount of details contained in the image. Experimental results of this study prove this assumption as they examine state of the art feature detectors applied both on standard dynamic range and HDR images.

We conducted a large-scale study of human perceptual quality judgments of High Dynamic Range (HDR) and Standard Dynamic Range (SDR) videos subjected to scaling and compression levels and viewed on three different display devices. While conventional expectations are that HDR quality is better than SDR quality, we have found subject preference of HDR versus SDR depends heavily on the display device, as well as on resolution scaling and bitrate. To study this question, we collected more than 23,000 quality ratings from 67 volunteers who watched 356 videos on OLED, QLED, and LCD televisions, and among many other findings, observed that HDR videos were often rated as lower quality than SDR videos at lower bitrates, particularly when viewed on LCD and QLED displays. Since it is of interest to be able to measure the quality of videos under these scenarios, e.g. to inform decisions regarding scaling, compression, and SDR vs HDR, we tested several well-known full-reference and no-reference video quality models on the new database. Towards advancing progress on this problem, we also developed a novel no-reference model called HDRPatchMAX, that uses a contrast-based analysis of classical and bit-depth features to predict quality more accurately than existing metrics.

High Dynamic Range (HDR) imaging is gaining increased attention due to its realistic content, for not only regular displays but also smartphones. Before sufficient HDR content is distributed, HDR visualization still relies mostly on converting Standard Dynamic Range (SDR) content. SDR images are often quantized, or bit depth reduced, before SDR-to-HDR conversion, e.g. for video transmission. Quantization can easily lead to banding artefacts. In some computing and/or memory I/O limited environment, the traditional solution using spatial neighborhood information is not feasible. Our method includes noise generation (offline) and noise injection (online), and operates on pixels of the quantized image. We vary the magnitude and structure of the noise pattern adaptively based on the luma of the quantized pixel and the slope of the inverse-tone mapping function. Subjective user evaluations confirm the superior performance of our technique.

Most Convolution Neural Network (CNN) based object detectors, to date, have been optimized for accuracy and/or detection performance on datasets typically comprised of well exposed 8-bits/pixel/channel Standard Dynamic Range (SDR) images. A major existing challenge in this area is to accurately detect objects under extreme/difficult lighting conditions as SDR image trained detectors fail to accurately detect objects under such challenging lighting conditions. In this paper, we address this issue for the first time by introducing High Dynamic Range (HDR) imaging to object detection. HDR imagery can capture and process ≈13 orders of magnitude of scene dynamic range similar to the human eye. HDR trained models are therefore able to extract more salient features from extreme lighting conditions leading to more accurate detections. However, introducing HDR also presents multiple new challenges such as the complete absence of resources and previous literature on such an approach. Here, we introduce a methodology to generate a large scale annotated HDR dataset from any existing SDR dataset and validate the quality of the generated dataset via a robust evaluation technique. We also discuss the challenges of training and validating HDR trained models using existing detectors. Finally, we provide a methodology to create an out of distribution (OOD) HDR dataset to test and compare the performance of HDR and SDR trained detectors under difficult lighting condition. Results suggest that using the proposed methodology, HDR trained models are able to achieve 10 – 12% more accuracy compared to SDR trained models on real-world OOD dataset consisting of high-contrast images under extreme lighting conditions.

Being able to automatically predict digital picture quality, as perceived by human observers, has become important in many applications where humans are the ultimate consumers of displayed visual information. Standard dynamic range (SDR) images provide 8 bits/color/pixel. High dynamic range (HDR) images which are usually created from multiple exposures of the same scene, can provide 16 or 32 bits/color/pixel, but must be tonemapped to SDR for display on standard monitors. Multi-exposure fusion (MEF) techniques bypass HDR creation, by fusing the exposure stack directly to SDR format while aiming for aesthetically pleasing luminance and color distributions. Here we describe a new no-reference image quality assessment (NR IQA) model for HDR pictures that is based on standard measurements of the bandpass and on newly-conceived differential natural scene statistics (NSS) of HDR pictures. We derive an algorithm from the model which we call the Gradient Image Quality Assessment algorithm (G-IQA). NSS models have previously been used to devise NR IQA models that effectively predict the subjective quality of SDR images, but they perform significantly worse on tonemapped HDR content. Towards ameliorating this we make here the following contributions: (1) We design a HDR picture NR IQA model and algorithm using both standard space-domain NSS features as well as novel HDR-specific gradient based features that significantly elevate prediction performance, (2) We validate the proposed models on a large-scale crowdsourced HDR image database, and (3) We demonstrate that the proposed model also perform well on legacy natural SDR images. The software is available at: http://signal.ece.utexas.edu/%7Edebarati/higradeRelease.zip.

In this chapter, depth estimation for stereo pair of High Dynamic Range (HDR) images is proposed. The proposed algorithm consists of two major techniques namely conversion of HDR images to Low Dynamic Range (LDR) images or Standard Dynamic Range (SDR) images and estimating the depth from the converted LDR / SDR stereo images. Local based tone mapping technique is used for the conversion of the HDR images to SDR images. And the depth estimation is done based on the corner features of the stereo pair images and block matching algorithm. Computationally much less expensive cost functions Mean Square Error (MSE) or Mean Absolute Difference (MAD) can be used for block matching algorithms. The proposed algorithm is explained with illustrations and results.

Tone expansion using lighting style aesthetics

Due to the standard dynamic range (SDR) feature of those widely used displays, high dynamic range (HDR) images need to be tone-mapped for its visualization. Tone-mapping operators (TMOs) convert HDR image to different SDR images, thus the perceptual information loss of these tone-mapped images during the process of tone-mapping become a natural question. In this work, an objective method based on measuring the entropy loss between HDR and SDR images is proposed. We tested several different approach of this method, and determined which is better by calculating their correlation with subjective mean opinion score (MOS) conducted in a tone-mapped images databased (TMID). This method is then compared and combined with other full-reference tone-mapping image assessment methods: TMQI, FSITM and CDMI. Only linear combinations of these methods are tested since this method is a practical way, and the combination with best correlation was selected as the final quality score.

High dynamic range (HDR) television promises a qualitatively better viewing experience, but to fully realize its potential and to achieve consistent high quality images, the psychovisual aspects of human vision must be taken into account. This paper reviews the nonlinearities inherent in conventional standard dynamic range (SDR) television and discusses how these are required to accommodate psychovisual effects. The thresholds at which changes in brightness can be detected, embodied in Weber's and De Vries-Rose's laws, are discussed in the context of the need to minimize “banding” artifacts due to quantization. This effect also provides a simple psychovisual definition for the dynamic range of video. The second important psychovisual effect considered is that of surround brightness on the perception of emissive images in dim or dark surroundings. This effect is accommodated by implementing an overall system nonlinearity, the rendering intent, which is typically a gamma (power law) function. The appropriate gamma exponent is considered in special cases and interpolated to provide a general formula. Based on these psychovisual effects, an HDR television system is proposed that supports the highest quality HDR images with a simultaneous dynamic range substantially beyond the limits of the human visual system in a single adaptation state. It is defined by a single opto-electronic transfer function (OETF), with a variable display electro-optic transfer function (EOTF) to accommodate the eye's adaptation. It is also compatible with existing, SDR, displays and infrastructure and requires no metadata.

In this paper, we propose a bit-depth scalable lossless coding method for high dynamic range (HDR) images based on a reversible logarithmic mapping. HDR images are generally expressed as floating-point data, such as in the OpenEXR or RGBE formats. Our bit-depth scalable coding approach outputs base layer data and enhancement layer data. It can reconstruct the low dynamic range (LDR) image from the base layer data and reconstructs the HDR image by adding the enhancement layer data. Most previous two-layer methods have focused on the lossy coding of HDR images. Unfortunately, the extension of previous lossy methods to lossless coding does not significantly compress the enhancement layer data. This is because the bit depth becomes very large, especially for HDR images in floating-point data format. To tackle this problem, we apply a reversible logarithmic mapping to the input HDR data. Moreover, we introduce a format conversion to avoid any degradation in the quality of the reconstructed LDR image. The proposed method is effective for both OpenEXR and RGBE formats. Through a series of experiments, we confirm that the proposed method decreases the volume of compressed data while maintaining the visual quality of the reconstructed LDR images.

There are an increasing number of databases describing subjective quality responses for HDR (high dynamic range) imagery with various distortions. The dominant distortions across the databases are those that arise from video compression, which are primarily perceived as achromatic, but there are some chromatic distortions due to 422 and other chromatic sub-sampling. Tone mapping from the source HDR levels to various levels of reduced capability SDR (standard dynamic range) are also included in these databases. While most of these distortions are achromatic, tone-mapping can cause changes in saturation and hue angle when saturated colors are in the upper hull of the of the color space. In addition, there is one database that specifically looked at color distortions in an HDR-WCG (wide color gamut) space. From these databases we can test the improvements to well-known quality metrics if they are applied in the newly developed color perceptual spaces (i.e., representations) specifically designed for HDR and WCG. We present results from testing these subjective quality databases to computed quality using the new color spaces of Jzazbz and ICTCP, as well as the commonly used SDR color space of CIELAB.

This paper presents an end-to-end workflow for high dynamic range (HDR) content acquisition and HDR representation that aims to reproduce in a perceptually realistic manner the appearance of the captured scene. The proposed workflow includes a camera-independent colour calibration method for accurate HDR content acquisition and a perceptually optimized HDR signal representation method that takes into account display and viewing conditions. Subjective evaluation of the proposed and other methods indicate that the proposed workflow offers statistically significant improvements in the quality of the HDR image/video displayed to the viewers relative to the anchor methods.

Tone mapping operators (TMO) are functions that map high dynamic range (HDR) images to a standard dynamic range (SDR), while aiming to preserve the perceptual cues of a scene that govern its visual quality. Despite the increasing number of studies on quality assessment of tone mapped images, current subjective quality datasets have relatively small numbers of images and subjective opinions. Moreover, existing challenges in transferring laboratory experiments to crowdsourcing platforms put a barrier for collecting large-scale datasets through crowdsourcing. In this work, we address these challenges and propose the RealVision-TMO (RV-TMO), a large-scale tone mapped image quality dataset. RV-TMO contains 250 unique HDR images, their tone mapped versions obtained using four TMOs and pairwise comparison results from seventy unique observers for each pair. To the best of our knowledge, this is the largest dataset available in the literature for quality evaluation of TMOs by the number of tone mapped images and number of annotations. Furthermore, we provide a content selection strategy to identify interesting and challenging HDR images. We also propose a novel methodology for observer screening in pairwise experiments. Our work does not only provide annotated data to benchmark existing objective quality metrics, but also paves the path to building new metrics for tone mapping quality evaluation.