Formal Visual Evaluation and Study of Objective Metrics for MPEG Dynamic Mesh Coding

Scalable video coding is emerging as an efficient alternative to simulcast encoding to distribute the same video content simultaneously to many users having different terminals and network conditions. In order to select the best combination of video scalability options for a given network condition and content, the availability of objective metrics that can reliably predict the video quality of scalable video sequence is crucial. In this paper, we propose a performance evaluation study of a set of state of the art Full-Reference and No-Reference metrics, considering a public database of test video sequences and related subjective quality annotations. The results demonstrate that most of the considered metrics show lack of robustness as predictors of subjective quality when both spatial and temporal quality distortions occur.

The ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group issued a Call for Proposals (CfP) on video compression with capability beyond HEVC in October 2017. The CfP considered three categories of content – Standard Dynamic Range, High Dynamic Range and Wide Colour Gamut (HDR/WCG), and 360° Omni-directional video. As a result of the CfP process, the development of a new video coding standard, named Versatile Video Coding (VVC), was initiated. The goal of this paper is to provide an overview of the CfP responses for the HDR/WCG category. The paper includes a summary of work leading to the development of the CfP, a presentation of the CfP results for the HDR/WCG category, and a description of the specific HDR/WCG technologies submitted to the CfP.

Deep learning methods have achieved impressive performance in compressed video quality enhancement tasks. However, these methods rely excessively on practical experience by manually designing the network structure and do not fully exploit the potential of the feature information contained in the video sequences, i.e., not taking full advantage of the multiscale similarity of the compressed artifact information and not seriously considering the impact of the partition boundaries in the compressed video on the overall video quality. In this article, we propose a novel Mixed Difference Equation inspired Transformer (MDEformer) for compressed video quality enhancement, which provides a relatively reliable principle to guide the network design and yields a new insight into the interpretable transformer. Specifically, drawing on the graphical concept of the mixed difference equation (MDE), we utilize multiple cross-layer cross-attention aggregation (CCA) modules to establish long-range dependencies between encoders and decoders of the transformer, where partition boundary smoothing (PBS) modules are inserted as feedforward networks. The CCA module can make full use of the multiscale similarity of compression artifacts to effectively remove compression artifacts, and recover the texture and detail information of the frame. The PBS module leverages the sensitivity of smoothing convolution to partition boundaries to eliminate the impact of partition boundaries on the quality of compressed video and improve its overall quality, while not having too much impacts on non-boundary pixels. Extensive experiments on the MFQE 2.0 dataset demonstrate that the proposed MDEformer can eliminate compression artifacts for improving the quality of the compressed video, and surpasses the state-of-the-arts (SOTAs) in terms of both objective metrics and visual quality.

In the last few years, Internet Protocol Television (IPTV) has emerged as one of the major distribution technologies for broadband multimedia services. However, the delivery of High Definition Television (HDTV) services over IP networks still faces many challenges. This paper presents a study on quality assessment of HD video content for IPTV applications. Video sequences with different content types are delivered through an IPTV testbed. Subject to packet loss, their quality is evaluated with common objective video quality measurement tools in the emulated error-prone environment with respect to their content complexity. Objective criticality based on the calculation of spatio-temporal activities is examined to validate its suitability as video content complexity indicator in quality assessment. Subjective assessment is further carried out to validate the objective test results. With a good correlation between subjective Mean Opinion Score (MOS) and objective metrics, the prediction of viewer satisfaction in terms of HD video content quality in IPTV networks becomes feasible. Suggestions are also made regarding the choice of suitable measurements for more meaningful HD content classifications.

This paper analyzes the coding efficiency of recent compression formats H.264, H.265 and AV1 for the visual weather prediction purpose. It compares the bitrate saving of these codecs using objective metrics. The analysis is done on 8 video sequences at Full HD and Ultra HD resolutions. We discovered, that bitstream saving of AV1 is approximately 50% higher than H.264 for both resolutions, 30% higher than H.265 encoded videos at Full HD and 20% higher than those at Ultra HD resolution. From the experimental results, we can also state that objective video quality is content dependent.

The paper presents a new objective metrics of blocking artifacts visibility in MPEG compressed video sequences. The metric estimates a local visibility of the block grid by analyzing the discontinuity of a pixel intensity trend across the block edge and by comparing it against pixel activities within blocks. Application of the local blockiness metrics for adaptive control of low-pass filtering provides significant reduction of coding artifacts without blurring of image details. The results of our experiments show the high efficiency of the proposed approach.

Although there are very few metrics which correlate well with the Human Visual System (HVS), most of them do not. We extensively studied about the Video Quality Assessment (VQA) of 2D svideos for enhancing the Quality of Experience (QoE) using better Quality of Service (QoS). We propose a solution which helps us to find a high correlation between HVS and the objective metrics using Perceptual Quality Metrics (PQM). The motive behind this work is to introduce an objective metric that is adequate to predict the Mean Opinion Score (MOS) of distorted video sequences based on the Full Reference (FR) method.

When capturing distant targets, video sequence images are subject to atmospheric turbulence, resulting in distortion and blurring. To restore degraded images caused by atmospheric turbulence in video sequences, we propose an algorithm that combines lucky imaging with generative adversarial networks. The algorithm utilizes spatial lucky imaging to select fortunate regions from limited video sequence images, which are then stitched and sorted to eliminate geometric distortion induced by atmospheric turbulence. Additionally, the DeblurGAN-v2 model is introduced to further enhance image quality. Real turbulent degradation images captured by a high-speed camera are employed for experimentation using the proposed method. Comparative analyses are conducted with methods such as image resampling, grayscale transformation, Butterworth high-pass filtering, MPRNet model, and DeblurGAN model. Objective evaluation metrics are employed to assess the results of different algorithms. Experimental results indicate that the proposed method yields significant improvements in Brenner gradient function, Laplacian gradient function, SMD, entropy function, energy gradient function, PIQE, and Brisque indicators, showing enhancements of 194%, 58%, 84%, 7%, 55%, 74%, and 163%, respectively, compared to other methods. From a subjective perspective, the algorithm combining lucky imaging with generative adversarial networks significantly enhances the visual quality of images and effectively reduces the degree of blurring and geometric distortion.

Review of objective video quality metrics and performance comparison using different databases

Modelling the non-stationary videos for performance assessment of frame reconstruction

A multi-stage temporal pooling mechanism is proposed in this paper for improving the prediction capability of an objective quality metric for video quality assessment. The performance of the proposed pooling mechanism is evaluated along with that of traditional pooling mechanisms in terms of linear correlation coefficient and Spearman rank order correlation coefficient on four publicly available video quality databases. The experimental results demonstrate that the proposed pooling mechanism can significantly improve the prediction capability of an objective quality metric and can be adopted for predicting the video level quality scores of longer duration video sequences with time varying video qualities.

We evaluate the performance of three state of the art video codecs on synthetic videos. The evaluation is based on both subjective and objective quality metrics. The subjective quality of the compressed video sequences is evaluated using the Double Stimulus Impairment Scale (DSIS) assessment metric while the Peak Signal-to-Noise Ratio (PSNR) is used for the objective evaluation. An extensive number of experiments are conducted to study the effect of frame rate and resolution on codecs’ performance for synthetic videos. The evaluation results show that video codecs respond in different ways to frame rate and frame resolution change. H.264 shows superior capabilities compared to other codecs. Mean Opinion Score (MOS) results are shown for various bitrates, frame rates and frame resolutions.

We present a scene understanding strategy for video sequences based on clustering object trajectories. In this chapter, we discuss a set of relevant feature spaces for trajectory representation and we critically analyze their relative merits. Next, we examine various trajectory clustering methods that can be employed to learn activity models, based on their classification into hierarchical and partitional algorithms. In particular, we focus on parametric and non-parametric partitional algorithms and discuss the limitations of existing approaches. To overcome the limitations of state-of-the-art approaches we present a soft partitional algorithm based on non-parametric Mean-shift clustering. The proposed algorithm is validated on real datasets and compared with state-of-the-art approaches, based on objective evaluation metrics.

High dynamic range (HDR) video compression has until now been approached by using the high profile of existing state-of-the-art H.264/AVC (Advanced Video Coding) codec or by separately encoding low dynamic range (LDR) video and the residue resulted from the estimation of HDR video from LDR video. Although the latter approach has a distinctive advantage of providing backward compatibility to 8-bit LDR displays, the superiority of one approach to the other in terms of the rate distortion trade-off has not been verified yet. In this paper, we first give a detailed overview of the methods in these two approaches. Then, we experimentally compare two approaches with respect to different objective and perceptual metrics, such as HDR mean square error (HDR MSE), perceptually uniform peak signal to noise ratio (PU PSNR) and HDR visible difference predictor (HDR VDP). We first conclude that the optimized methods for backward compatibility to 8-bit LDR displays are superior to the method designed for high profile encoder both for 8-bit and 12-bit mappings in terms of all metrics. Second, using higher bit-depths with a high profile encoder is giving better rate-distortion performances than employing an 8-bit mapping with an 8-bit encoder for the same method, in particular when the dynamic range of the video sequence is high. Third, rather than encoding of the residue signal in backward compatible methods, changing the quantization step size of the LDR layer encoder would be sufficient to achieve a required quality. In other words, the quality of tone mapping is more important than residue encoding for the performance of HDR image and video coding.

A systematic comparison between visual cues for boundary detection