Designing Video Quality Metrics

In the conventional design approach to video quality metrics, the temporal nature of video is often considered only inadequately and also knowledge about the human visual system is required that is not readily available. In this thesis, I therefore propose a data-driven design methodology using multi-way data analysis for the design of video quality metrics that not only allows for the appropriate consideration of the temporal nature of video, but also leads to an increased prediction performance.

Objective video quality metrics are an essential part of modern video processing pipelines, guiding video encoding decisions and encoding recipes, helping adaptive bitrate streaming algorithms make smart decisions and providing system-level monitoring capabilities. We will offer a breakdown of an end-to-end such pipeline, highlighting which types of video quality metrics are deployed in each system component and then focus on the single aspect that makes social videos so much different - and one can argue more difficult - to process: their wildly varying and typically inferior source quality. We will then discuss how no-reference video quality metrics have been typically used to measure user-generated video content quality with limited success and make a case for how the video industry can unite and solve this problem at its root.

Online monitoring of multimedia networks is required to ensure seamless and ubiquitous delivery of services to the end users. Quality of multimedia content, such as video streams, often gets degraded due to network losses such as packet loss. Frame freezing artifacts are introduced in a video stream when packet loss or packet delay takes place. Estimating the perceptual impact of these artifacts on quality of experience of end users helps service providers to maintain quality of service. In this paper, we have presented a novel no-reference video quality metric, which measures the impact of frame freezing due to packet loss and delay in video streaming networks. The proposed metric is based on several features that directly impact the quality of experience of end users. These features, including motion characteristics of videos, are calculated using the temporal information between video frames and then combined mathematically to form a video quality metric. Different weights are assigned to different features for better performance of the proposed metric. With detailed experiments, we have shown that our method outperforms other contemporary methods in terms of high accuracy and low computation time in frame freeze detection, low root mean square values, high coefficient of determination, and high correlation between subjective and objective measurements. We have used five video databases for our model's evaluation and validation. Furthermore, we have shown that our method is statistically superior to the other models in comparison.

Objective video quality metrics reported in the literature have so far been focused on TV signals with large frame sizes, full TV frame rates, and high compressed bit rates. The aim of this paper is to provide an analysis of 4 objective video quality metrics tested on low bit rate videos with small frame sizes, various frame rates, and low bit rates. The video quality metrics used were the NTIA video quality metric, modified Watson's digital video quality metric, video structural similarity metric, and the perceptual video quality metric. The test videos consist of H.264 compressed videos with CIF and QCIF frame sizes, at various bit rates (24 kbps - 384 kbps) and frame rates (7.5 fps - 30 fps). Here, results and analysis derived from the comparison of the metrics are provided.

Although a lot of progress has been made in the development of 2D objective video quality metrics, the area of 3D video quality metrics is still in its infancy. Many of the proposed metrics are simply adaptations of 2D quality metrics that consider the depth channel as an extra color channel. In this paper, we propose a 3D no-reference objective quality metric that estimates 3D quality taking into account spatial distortions, excessive disparity, depth representation and temporal information of the video. The metric is resolution and frame-rate independent. To estimate the amount of spatial distortion in the video, the proposed metric uses a blockiness metric. The contribution of motion and excessive disparity to 3D quality is calculated using a non-linear relative disparity measure and a frame-rate proportional motion measure. The metric's performance is verified against the COSPAD1 database. The MOS predicted using the proposed metric obtained good correlation values with the subjective scores. The performance was on average better than the performance of two simple 2D full reference metrics: SIMM and PSNR.

It is beneficial to have adaptive resolution selection during encoding for the viewer’s optimal experience in a video delivery scenario. For instance, per-title content-adaptive techniques have been exploited for OTT/VOD delivery. An established solution is to build the selection of better resolution on typical video quality metrics. In this context, this paper first introduces measuring the performance of video quality metrics to accurately predict which encoding resolution is subjectively better suited to a particular scene of interest in a video. Then, with a Random Forest (RF) classifier, a novel, subjectively accurate classifier called RF-based fusion metric is proposed to decide which encoding resolution is best suited to compensate for individual metrics’ disparate performance over different quality ranges. The proposed RF classifier encompasses classical video quality metric scores as features and is trained to be closer to subjective experimental results. Performance comparison of this proposed RF-based fusion metric is discussed in comparison to other metrics and subjective experiments.

The dominant image and video quality metrics rarely consider the physical specification of video content (luminance, frame rate, resolution) or incorporate models of human vision. In this talk, I will show how a robust video quality metric can be built from fundamental models of low-level human vision and how such a metric can generalize to any display specification (size, resolution, dynamic range) and viewing conditions (ambient light, viewing distance). I will explain how we built a model of contrast sensitivity for achromatic and chromatic spatiotemporal patterns (castleCSF) and combined it with a contrast masking model to create our new video quality metric - ColorVideoVDP. ColorVideoVDP brings many advantages, such as explainability, robustness to unseen distortions, differentiable formulation, and the ability to adapt to the physical characteristics of the content (frame rate, HDR/SDR colour spaces).

Currently, three-dimensional (3D) video is gaining increasing popularity by providing immersive user experience. Compared against conventional 2D video, 3D video excels at bringing a “live” scene closer to the users, and/or trying to place the users in the environment of the displayed content. However, streaming 3D video sequences over the IP networks is challenging due to the impact of dynamic network conditions on user quality. Accurate objective 3D video quality assessment is critical for advanced real-time video streaming adaptation solutions. Most state-of-the-art objective 3D video quality metrics are reference-based and require access to the original 3D video sequences, which is not possible for the real-time applications. This paper proposes the extended No-reference objective 3D Video Quality Metric (eNVQM) for real time 3D video quality assessment. eNVQM establishes a correlation between network packet loss and stereoscopic 3D video quality and was tuned according to extensive subjective testing results. Performance of eNVQM is studied in comparison with two state-of-the-art objective video quality metrics: structural similarity index (SSIM) and video quality metric (VQM).

Purpose The Video Quality Metric (VQM) is one of the most used objective methods to assess video quality, because of its high correlation with the human visual system (HVS). VQM is, however, not viable in real-time deployments such as mobile streaming, not only due to its high computational demands but also because, as a Full Reference (FR) metric, it requires both the original video and its impaired counterpart. In contrast, No Reference (NR) objective algorithms operate directly on the impaired video and are considerably faster but loose out in accuracy. The purpose of this paper is to study how differently NR metrics perform in the presence of network impairments. Design/methodology/approach The authors assess eight NR metrics, alongside a lightweight FR metric, using VQM as benchmark in a self-developed network-impaired video data set. This paper covers a range of methods, a diverse set of video types and encoding conditions and a variety of network impairment test-cases. Findings The authors show the extent by which packet loss affects different video types, correlating the accuracy of NR metrics to the FR benchmark. This paper helps identifying the conditions under which simple metrics may be used effectively and indicates an avenue to control the quality of streaming systems. Originality/value Most studies in literature have focused on assessing streams that are either unaffected by the network (e.g. looking at the effects of video compression algorithms) or are affected by synthetic network impairments (i.e. via simulated network conditions). The authors show that when streams are affected by real network conditions, assessing Quality of Experience becomes even harder, as the existing metrics perform poorly.

While existing video and image quality datasets have extensively studied natural videos and traditional distortions, the perception of synthetic content and modern rendering artifacts remains underexplored. We present a novel video quality dataset focused on distortions introduced by advanced rendering techniques, including neural supersampling, novel‐view synthesis, path tracing, neural denoising, frame interpolation, and variable rate shading. Our evaluations show that existing full‐reference quality metrics perform sub‐optimally on these distortions, with a maximum Pearson correlation of 0.78. Additionally, we find that the feature space of pre‐trained 3D CNNs aligns strongly with human perception of visual quality. We propose CGVQM, a full‐reference video quality metric that significantly outperforms existing metrics while generating both per‐pixel error maps and global quality scores. Our dataset and metric implementation is available at https://github.com/IntelLabs/CGVQM.

The video quality degradation in wireless network is much changeable for complex packet loss scenes of wireless channels. However, few Video Quality Metrics (VQM) analyze the quality degradation under different wireless packet loss conditions. This paper proposes a novel Universal Video Quality Metric (UVQM), which is suitable for both distributed loss situation and burst loss situation. The major contributions of this paper can be concluded as follows. Firstly, this metric is an end-to-end measurement which considers the impacts of encoding parameter (quantization parameter, q-scale) and transmission parameters (Packet Loss Rate, PLR and Maximum Transmission Unit, MTU) to make a complete VQM. Then, parameters used in UVQM are extracted from bitstreams which reduces the use of computation resource for decoding at the receiver. Finally, UVQM coalesces the impact of content-aware features by introducing a developed Just Noticeable Distortion (JND) which makes the prediction much accuracy. The verification results show that the predicted video quality measured by Mean Opinion Score (MOS) has a high Pearson correlation (higher than 0.74 in distributed loss situation and 0.65 in burst loss situation).

We present a review on human visual system (HVS) based digital video quality metrics. Particularly, three objective video quality metrics are discussed and analyzed in detail because they represent the state-of-the-art of HVS based quality metric research and have been proposed to and verified by VQEG (Video Quality Expert Group) as the candidates of a possible ITU standard. The purpose of the paper is to provide an up-to-date knowledge of the HVS modeling, how the characteristics of the HVS have been incorporated into quality metrics, and the implementation issues of the metrics as well as the directions of future research.

We present a low complexity approach for the estimation of the temporal and spatial activity parameters of videos which are captured by a front-facing camera of a vehicle based on context information of the vehicle. The estimated parameters are integrated into an objective video quality metric, which can be used to determine the perceptual quality of a compressed video stream. Our proposed video quality metric has very low computational complexity, which makes it suitable for live video streaming applications. It shows a high Pearson correlation of 0.98 with an average root-mean-square error of 6%, as verified by statistical analysis with data from subjective tests.

In adaptive video streaming, optimizing the selection of representations for the encoding bitrate ladder has a significant impact on the quality and economics of media delivery. An efficient way to select representations for the bitrate ladder of a given clip is to consider the Satisfied User Ratio (SUR) of the perceived quality of consecutive representations. This ensures that only representations with one Just Noticeable Difference (JND) are encoded and streamed by avoiding encoding similar-quality representations. VMAF (Video Multi-method Assessment Fusion) presently stands as the most commonly utilized quality metric for constructing bitrate ladders. Hence, the precise determination of JND-optimal encoding step-sizes for the VMAF proxy holds paramount importance; nevertheless, this task is intricate and can present considerable challenges.In this paper, we evaluate the effectiveness of different Video Quality Metrics (VQMs) in predicting SUR for the VMAF proxy to better capture content-specific characteristics. Our experimental results provide evidence that incorporating VQMs can improve the precision of the SUR prediction for the VMAF proxy. Compared to a state-of-the-art approach that utilizes video complexity metrics, our proposed approach, which incorporates two quality metrics—specifically, VMAF and SSIM calculated at an optimized quantization parameter (QP)—achieves a substantially reduced Mean Absolute Error (MAE) of 1.67. In contrast, the state-of-the-art approach yields an MAE of 2.01. Hence, we recommend using the above quality metrics to improve the accuracy of the SUR prediction for the VMAF proxy.

This paper describes a no-reference temporal qualitymetric to model the impact of frame freezing impairments on perceived video quality. The proposed metric shows a high correlation with subjective data and is equally applicable to any type of objective video quality assessment model: full-reference, reduced-reference or no-reference. This temporal metric is a parameter of the video quality assessment algorithmin ITU-T Recommendation J.247 Annex C for the objective perceptual quality measurement of multimedia video.