Chapter 4 - Subjective and objective quality assessment for omnidirectional video

Objective video quality metrics reported in the literature have so far been focused on TV signals with large frame sizes, full TV frame rates, and high compressed bit rates. The aim of this paper is to provide an analysis of 4 objective video quality metrics tested on low bit rate videos with small frame sizes, various frame rates, and low bit rates. The video quality metrics used were the NTIA video quality metric, modified Watson's digital video quality metric, video structural similarity metric, and the perceptual video quality metric. The test videos consist of H.264 compressed videos with CIF and QCIF frame sizes, at various bit rates (24 kbps - 384 kbps) and frame rates (7.5 fps - 30 fps). Here, results and analysis derived from the comparison of the metrics are provided.

Currently, three-dimensional (3D) video is gaining increasing popularity by providing immersive user experience. Compared against conventional 2D video, 3D video excels at bringing a “live” scene closer to the users, and/or trying to place the users in the environment of the displayed content. However, streaming 3D video sequences over the IP networks is challenging due to the impact of dynamic network conditions on user quality. Accurate objective 3D video quality assessment is critical for advanced real-time video streaming adaptation solutions. Most state-of-the-art objective 3D video quality metrics are reference-based and require access to the original 3D video sequences, which is not possible for the real-time applications. This paper proposes the extended No-reference objective 3D Video Quality Metric (eNVQM) for real time 3D video quality assessment. eNVQM establishes a correlation between network packet loss and stereoscopic 3D video quality and was tuned according to extensive subjective testing results. Performance of eNVQM is studied in comparison with two state-of-the-art objective video quality metrics: structural similarity index (SSIM) and video quality metric (VQM).

We present a low complexity approach for the estimation of the temporal and spatial activity parameters of videos which are captured by a front-facing camera of a vehicle based on context information of the vehicle. The estimated parameters are integrated into an objective video quality metric, which can be used to determine the perceptual quality of a compressed video stream. Our proposed video quality metric has very low computational complexity, which makes it suitable for live video streaming applications. It shows a high Pearson correlation of 0.98 with an average root-mean-square error of 6%, as verified by statistical analysis with data from subjective tests.

We present a review on human visual system (HVS) based digital video quality metrics. Particularly, three objective video quality metrics are discussed and analyzed in detail because they represent the state-of-the-art of HVS based quality metric research and have been proposed to and verified by VQEG (Video Quality Expert Group) as the candidates of a possible ITU standard. The purpose of the paper is to provide an up-to-date knowledge of the HVS modeling, how the characteristics of the HVS have been incorporated into quality metrics, and the implementation issues of the metrics as well as the directions of future research.

Review of objective video quality metrics and performance comparison using different databases

Objective video quality metrics are an essential part of modern video processing pipelines, guiding video encoding decisions and encoding recipes, helping adaptive bitrate streaming algorithms make smart decisions and providing system-level monitoring capabilities. We will offer a breakdown of an end-to-end such pipeline, highlighting which types of video quality metrics are deployed in each system component and then focus on the single aspect that makes social videos so much different - and one can argue more difficult - to process: their wildly varying and typically inferior source quality. We will then discuss how no-reference video quality metrics have been typically used to measure user-generated video content quality with limited success and make a case for how the video industry can unite and solve this problem at its root.

In image and video compression and transmission, it is important to rely on an objective image/video quality metric which accurately represents the subjective quality of processed images and video sequences. In some scenarios, it is also important to evaluate the quality of the received video sequence with minimal reference to the transmitted one. For instance, for quality improvement of video transmission through closed-loop optimisation, the video quality measure can be evaluated at the receiver and provided as feedback information to the system controller. The original image/video sequence--prior to compression and transmission--is not usually available at the receiver side, and it is important to rely at the receiver side on an objective video quality metric that does not need reference or needs minimal reference to the original video sequence. The observation that the human eye is very sensitive to edge and contour information of an image underpins the proposal of our reduced reference (RR) quality metric, which compares edge information between the distorted and the original image. Results highlight that the metric correlates well with subjective observations, also in comparison with commonly used full-reference metrics and with a state-of-the-art RR metric.

Objective video quality metrics can be viewed as myopic expert systems that focus on particular aspects of visual information in video, such as image edges or motion parameters. We conjecture that the combination of many such high-level metrics leads to statistically-significant improvement in the prediction of reference-based perceptual video quality in comparison to each individual metric. To examine this hypothesis in a systematic and rigorous manner, we use: (i) the LIVE and the EPFL/PoliMi databases that provide the difference mean opinion scores (DMOS) for several video sequences under encoding and packet-loss errors; (ii) ten well-known metrics that range from mean-squared error based criteria to sophisticated visual quality estimators; (iii) five variants of regression-based supervised learning. For 400 experimental trials with random (non-overlapping) estimation and prediction subsets taken from both databases, we show that the best of our regression methods: (i) leads to statistically-significant improvement against the best individual metrics for DMOS prediction for more than 97% of the experimental trials; (ii) is statistically-equivalent to the performance of humans rating the video quality for 36.75% of the experiments with the EPFL/PoliMi database. On the contrary, no single metric achieves such statistical equivalence to human raters in any of the experimental trials.

A method for deriving an objective video quality metric that emulates the human video quality metric is described. The objective metric comprises a family of measurements that quantify spatial, temporal, luminance, and chrominance distortions, followed by a prediction algorithm. The intent is that the derived metric will work well over a wide range of digital and analog video transmission and storage technologies. After describing the need for these measures, a detailed description of the derivation process and some preliminary results are given. >

Welcome to the third column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG). The last VQEG plenary meeting took place online from 14 to 18 December. Given the current circumstances, it was organized all online for the second time, with multiple sessions distributed over five to six hours each day allowing remote participation of people from different time zones. About 130 participants from 24 different countries registered to the meeting and could attend the several presentations and discussions that took place in all working groups. This column provides an overview of this meeting, while all the information, minutes, files (including the presented slides), and video recordings from the meeting are available online in the VQEG meeting website. As highlights of interest for the SIGMM community, apart from several interesting presentations of state-of-the-art works, relevant contributions to ITU recommendations related to multimedia quality assessment were reported from various groups (e.g., on adaptive bitrate streaming services, on subjective quality assessment of 360-degree videos, on statistical analysis of quality assessments, on gaming applications, etc.), the new group on quality assessment for health applications was launched, and an interesting session on 5G use cases took place, as well as a workshop dedicated to user testing during Covid-19. In addition, new efforts have been launched related to the research on quality metrics for live media streaming applications, and to provide guidelines on implementing objective video quality metrics (ahead of PSNR) to the video compression community. We encourage those readers interested in any of the activities going on in the working groups to check their websites and subscribe to the corresponding reflectors, to follow them and get involved.

Uploading videos to platforms for sharing has been a core feature of the internet for nearly two decades. Advances in video encoding algorithms, their standardization, and increased computational power have significantly improved the visual quality of online videos. This paper evaluates the video quality across several popular platforms, including YouTube, Facebook, and Vimeo. Using three test sequences with varying content characteristics, the study employs full-reference objective video quality metrics—VMAF, PSNR-HVS-M, and MS-SSIM—to assess video quality at resolutions supported by all platforms. The results provide insights into the comparative performance of each platform in delivering high-quality video

Since 1997, the Video Quality Experts Group (VQEG) has been active in the field of subjective and objective video quality assessment. The group has validated competitive quality metrics throughout several projects. Each of these projects requires mandatory actions such as creating a testplan and obtaining databases consisting of degraded video sequences with corresponding subjective quality ratings. Recently, VQEG started a new open initiative, the Joint Effort Group (JEG), for encouraging joint collaboration on all mandatory actions needed to validate video quality metrics. Within the JEG, effort is made to advance the field of both subjective and objective video quality measurement by providing proper software tools and subjective databases to the community. One of the subprojects of the JEG is the joint development of a hybrid H.264/AVC objective quality metric. In this paper, we introduce the JEG and provide an overview of the different ongoing activities within this newly started group.

With the increasing demand in video traffic, delivering appropriate Quality of Experience may appear as a challenge for highly popular services such as Twitch.tv and YouTube-Gaming. This paper presents the evaluation of three widely used encoders, namely H.264, H.265 and AV1, for the case of gaming videos, assuming streaming scenario. The performance of the codecs has been assessed in terms of objective video quality metrics (PSNR, SSIM and VMAF) and subjective video quality assessment. The encoding settings were kept similar for all the three aforementioned encoders. In the considered setting, AV1 achieves a better level of video quality both in terms of objective and subjective VQA, for almost all bitrates and content considered. The improvement is remarkable in particular for the lower range of bitrates considered.

In this study, the perceptual video quality (PVQ) for the vogue 360-degree video with compression artefacts viewed on the visual reality (VR) head-mounted display (HMD) is obtained for the first time by an elaborately designed subjective experiment. The characteristic of the PVQ of omnidirectional (360-degree) video on the VR HMD and PC monitor is then investigated. The PVQ on the HMD is found to be linearly related to the video coding quality (VCQ) on the PC monitor. A quality evaluation model is then proposed based on the mapping formula and a new assessment parameter, where the impact of the video resolution and display device is involved. In this way, the traditional video quality assessment metrics for two-dimensional video can be extended to assess the PVQ of the 360-degree video viewed on the HMD. At last, the video structural similarity metric is taken as an example to predict the input of the proposed model, i.e. the VCQ of the 360-degree video. Experimental results demonstrate that the proposed model can effectively and conveniently solve the challenge of PVQ assessment for the 360-degree video with compression distortions on the HMD.

The focus of this book is on the design of video quality metrics with multi-way data analysis in general and not on the development of a particular metric itself. It is therefore not sensible to design a video quality metric highly optimised for a certain application. Yet in order to assess if multi-way data analysis provides an advantage in the design of video quality, two simple examples for no-reference video quality metrics are introduced in this chapter. They represent the main categories in the engineering approach to video quality metrics: one bitstream-based and one pixel-based video quality metric. The first example describes how a video quality metric can be designed with H.264/AVC bitstream-based no-reference features, suitable for the visual quality estimation of H.264/AVC encoded video sequences. The second example describes a more universal approach, utilizing only pixel-based no-reference features and is thus also suitable for other coding technologies, as the features are selected with respect to generally occurring coding artefacts.