Learning-Based JND-Directed HDR Video Preprocessing for Perceptually Lossless Compression With HEVC

HDR video quality assessment: Perceptual evaluation of compressed HDR video

The new generation of organic light emitting diode display is designed to enable the high dynamic range (HDR), going beyond the standard dynamic range (SDR) supported by the traditional display devices. However, a large quantity of videos are still of SDR format. Further, most pre-existing videos are compressed at varying degrees for minimizing the storage and traffic flow demands. To enable movie-going experience on new generation devices, converting the compressed SDR videos to the HDR format (i.e., compressed-SDR to HDR conversion) is in great demands. The key challenge with this new problem is how to solve the intrinsic many-to-many mapping issue. However, without constraining the solution space or simply imitating the inverse camera imaging pipeline in stages, existing SDR-to-HDR methods can not formulate the HDR video generation process explicitly. Besides, they ignore the fact that videos are often compressed. To address these challenges, in this work we propose a novel imaging knowledge-inspired parallel networks (termed as KPNet) for compressed-SDR to HDR (CSDR-to-HDR) video reconstruction. KPNet has two key designs: Knowledge-Inspired Block (KIB) and Information Fusion Module (IFM). Concretely, mathematically formulated using some priors with compressed videos, our conversion from a CSDR-to-HDR video reconstruction is conceptually divided into four synergistic parts: reducing compression artifacts, recovering missing details, adjusting imaging parameters, and reducing image noise. We approximate this process by a compact KIB. To capture richer details, we learn HDR representations with a set of KIBs connected in parallel and fused with the IFM. Extensive evaluations show that our KPNet achieves superior performance over the state-of-the-art methods.

Currently, high dynamic range (HDR) videos with high resolution (HR) have become popular due to the display and the rendered technological advancements. However, making ultra-high definition (UHD) with HDR videos is expensive. The legacy low-resolution (LR) standard dynamic range (SDR) format is still largely used in practice. It is necessary to search for a solution to transform LR SDR videos into UHD HDR format. In this paper, we consider joint super resolution and learning inverse tone mapping an issue of high-frequency reconstruction and local contrast enhancement, and we propose an architecture based on a generative adversarial network to apply joint SR-ITM learning. Specifically, we include the residual ResNeXt block (RRXB) as a basic module to better capture high-frequency textures and adopt YUV interpolation to achieve local contrast enhancement. By adopting a generative adversarial network as a pivotal training mechanism, our designs show advantages in both integration and performance. Our code is now available on GitHub: SR-ITM-GAN.

As high dynamic range video is gaining popularity, video coding solutions able to efficiently provide both low and high dynamic range video, notably with a single bitstream, are increasingly important. While simulcasting can provide both dynamic range videos at the cost of some compression efficiency penalty, bit-depth scalable video coding can provide a better trade-off between compression efficiency, adaptation flexibility and computational complexity. Considering the widespread use of H.264/AVC video, this paper proposes a H.264/AVC backward compatible bit-depth scalable video coding solution offering a low dynamic range base layer and two high dynamic range enhancement layers with different qualities, at low complexity. Experimental results show that the proposed solution has an acceptable rate-distortion performance penalty regarding the HDR H.264/AVC single-layer coding solution.

Rate-distortion optimized layered coding of high dynamic range videos

High dynamic range (HDR) video compression has until now been approached by using the high profile of existing state-of-the-art H.264/AVC (Advanced Video Coding) codec or by separately encoding low dynamic range (LDR) video and the residue resulted from the estimation of HDR video from LDR video. Although the latter approach has a distinctive advantage of providing backward compatibility to 8-bit LDR displays, the superiority of one approach to the other in terms of the rate distortion trade-off has not been verified yet. In this paper, we first give a detailed overview of the methods in these two approaches. Then, we experimentally compare two approaches with respect to different objective and perceptual metrics, such as HDR mean square error (HDR MSE), perceptually uniform peak signal to noise ratio (PU PSNR) and HDR visible difference predictor (HDR VDP). We first conclude that the optimized methods for backward compatibility to 8-bit LDR displays are superior to the method designed for high profile encoder both for 8-bit and 12-bit mappings in terms of all metrics. Second, using higher bit-depths with a high profile encoder is giving better rate-distortion performances than employing an 8-bit mapping with an 8-bit encoder for the same method, in particular when the dynamic range of the video sequence is high. Third, rather than encoding of the residue signal in backward compatible methods, changing the quantization step size of the LDR layer encoder would be sufficient to achieve a required quality. In other words, the quality of tone mapping is more important than residue encoding for the performance of HDR image and video coding.

Just Noticeable Distortion (JND) based on human visual system (HVS) is widely used in the transparent watermarking. But the computation of JND is very complex, which makes it difficult to embed it into integrated circuits. To solve this problem, Haar Wavelet based JND model is exploited in this paper. Furthermore, the fast pipeline architecture based on haar-wavelet for JND computation is developed. The architecture is modeled with hardware description language, and implemented on Altera EP2C35 FPGA device in order to evaluate the its performance. The hardware cost of JND core is 1460 Logic Cell combinations and 330 registers, which is significantly smaller than the Full Band JND based architecture. From the experiment results, the system goes on well with a 120MHz clock. Compared with the full band JND based architecture, ours achieves 75% time saving.

The genuine viewing quality of High Dynamic Range (HDR) data brought an evolution to content production and display manufacture markets. However, the efficiency of compression and transmission of HDR content still needs to be improved. In this paper we modify the H.264/AVC standard to better characterize HDR content. We propose a new Lagrangian multiplier that strikes a balance between the bit-rate and distortion of the HDR video. The updated Lagrange multiplier was implemented on the H.264/AVC reference software. Our experiment results show that the HDR-VDP2 quality scores of the videos encoded by the HDR-accustomed encoder are higher than the ones encoded with the reference encoder. Moreover, subjective tests confirmed that the visual quality of the compressed HDR videos using our proposed method is higher than the one encoded by the reference codec <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .

Medical images stored in a hospital system are generally confidential and integrated and require strict security. However, medical information stored on digital medical imaging systems, as well as picture archiving and communication systems (PACS), are vulnerable to attack when the data are transferred over wireless or wired communication networks. To solve this problem, a watermarking algorithm for medical images is proposed using a bit threshold map based on just noticeable distortion (JND) in the discrete cosine transform (DCT) method. The low-frequency component comprises a considerable amount of the signal energy for most images. As a result, it has a crucial effect on the image quality. Therefore, in this paper, the proposed algorithm embeds watermarks based on the low-frequency components of the image, such as the DC coefficient of the DCT. When watermarks are embedded in a low frequency area, the subjective image quality is often degraded. To compensate for the degradation of the imperceptibility of the watermarking system, which results from embedding watermarks in the low-frequency component, this research considers the human visual system. In addition, the embedding strength of the JND value is used to improve the watermarking imperceptibility. We applied the proposed watermarking algorithm to a variety of medical images using a computer simulation. The algorithm's performance was verified using a variety of attacks for eliminating watermarks. The simulation results show that the proposed algorithm robustly provides protection against a variety of possible attacks.

A distortion on an image can be invisible by the Human Visual System (HVS), if the distortion is under a threshold called Just Noticeable Distortion (JND), which is determined by several sensibility factors of the HVS, such as luminance and frequency sensibility. Until now, the JND is widely used for invisible watermarking schemes; however, in visible watermarking schemes this concept is not explored sufficiently. In this paper, a novel compressed domain visible watermarking method based on the JND profile is presented. The JND is estimated by texture and intensity of each 8×8 block of the Discrete Cosine Transform (DCT). Considering that the JPEG is the most widely used digital image format and almost all photographs captured by digital camera are stored in this format; the visible watermarking scheme in compressed domain without inverse DCT operation to obtain spatial image provides important advantage from security and complexity points of view.

High Dynamic Range (HDR) image and video technology aims at conveying the full range of perceptible shadow and highlight details with sufficient precision. HDR is regarded by many experts as the next evolution in digital media. However, industrial broadcasters have concerns regarding the bandwidth overhead that this new technology entails. While many consider that broadcasting HDR content would increase bandwidth requirements by around 20%, this number is based on studies where, in addition to the SDR main stream, HDR-related side information is conveyed. A recent subjective evaluation reported that encoding HDR video content in a single layer might require less bandwidth than its associated SDR version. Similar results were discussed in the MPEG ad-hoc group on High Dynamic Range and Wide Color Gamut. In this article, we explain how having more information can result in lower bandwidth requirements. To this end, we describe several limitations of the human vision system that, when exploited, optimize the HDR distribution pipeline for a human observer. Our theoretical assumption about the higher efficiency of HDR is backed up by a statistical analysis of pixel distribution in real images. The Spatial Index objective metric also reconfirms our assumption.

We propose a new spatial just noticeable distortion (JND) profile for color image processing. The JND threshold depends on various masking effects underlying existing in the human vision system (HVS). How to efficiently integrate different masking effects together is the key issue of modelling the JND profile. Based on recent vision research results, we model the masking effects in different stimulus dimensions as a nonlinear additivity model for masking (NAMM). It applies to all color components and accounts for the compound impact of luminance masking and texture masking to estimate the JND threshold in images. In our PSNR and subjective comparison to the related work, the proposed NAMM scheme provides a more accurate JND profile towards the actual JND bound in the HVS.

Fast synthesized and predicted just noticeable distortion maps for perceptual multiview video coding

Just Noticeable Distortion (JND) finds the minimum distortion level perceivable by humans. This can be a natural solution for setting the compression for each video region in perceptual video coding. However, existing JND-based solutions estimate JND levels for each video frame and ignore the fact that different video regions have different perceptual importance. To address this issue, we propose a Block-Level Just Noticeable Distortion-based Perceptual (BL-JUNIPER) framework for video coding. The proposed four-stage framework combines different perceptual information to further improve the prediction accuracy. The JND mapping in the first stage derives block-level JNDs from frame-level information without the need to collect a new bock-level JND dataset. In the second stage, an efficient CNN-based model is proposed to predict JND levels for each block according to spatial and temporal characteristics. Unlike existing methods, BL-JUNIPER works on raw video frames and avoids re-encoding each frame several times, making it computationally practical. Third, the visual importance of each block is measured using a visual attention model. Finally, a proposed quantization control algorithm uses both JND levels and visual importance to adjust the Quantization Parameter (QP) for each block. The specific algorithm for each stage of the proposed framework can be changed, as long as the input and output formats of each block are followed, without the need to change other stages, based on any current or future methods, providing a flexible and robust solution. Extensive experimental results demonstrate that BL-JUNIPER achieves a mean bitrate reduction of 27.75% with a Delta Mean Opinion Score (DMOS) close to zero and BD-Rate gains of 25.44% based on MOS, compared to the baseline encoding, and also gains a better performance compared to competing methods.

High dynamic range (HDR) imaging technology has been widely implemented in digital microscopes for taking still images of high contrast specimens. However, capturing HDR microscopic video is much more challenging. In this paper, we present a HDR microscopic video system based on GPU accelerated computing. We show that by combining CPU and GPU computing, it is possible to build a stable HDR video system using a single off the shelf camera. We show that capturing multiple frames of different exposure intervals, aligning consecutive neighboring frames, constructing HDR radiance map and tone mapping the radiance map for display, can be realized by using GPU computing to accelerate the processing speed. We present experimental results to show the effectiveness of our system and how HDR video can reveal much more details than conventional videos.