Objective quality metrics for video scalability

With the increasing complexity of system-on-chip (SoC) design, intellectual property (IP)-based design flow is becoming an inevitable trend to achieve high performance and short time-to-market. However the transfer of IP core remains a complex and time-consuming process. Different IP vendors usually use different design rules and tool flows, which make it difficult for IP integrators to select suitable IP cores and integrate them rapidly. In this paper, a common platform for IP qualification (CPIPQ) is presented to provide and establish a set of common standards to qualify Soft/Hard IP. There are two kinds of qualification methods: subjective quality metric and objective quality metric. We use IEEE QIP Metric to evaluate the subjective quality metrics of IP cores. Objective metrics are measured automatically by using commercial EDA tools, as well as tools that developed by ASTRI IC Design Group.

Omnidirectional image and video have gained popularity thanks to availability of capture and display devices for this type of content. Recent studies have assessed performance of objective metrics in predicting visual quality of omnidirectional content. These metrics, however, have not been rigorously validated by comparing their prediction results with ground-truth subjective scores. In this paper, we present a set of 360-degree images along with their subjective quality ratings. The set is composed of four contents represented in two geometric projections and compressed with three different codecs at four different bitrates. A range of objective quality metrics for each stimulus is then computed and compared to subjective scores. Statistical analysis is performed in order to assess performance of each objective quality metric in predicting subjective visual quality as perceived by human observers. Results show the estimated performance of the state-of-the-art objective metrics for omnidirectional visual content. Objective metrics specifically designed for 360-degree content do not outperform conventional methods designed for 2D images.

Video communication requires sufficient resources to obtain the desired image quality. Availability of bandwidth and screen resolution from different devices on the client end, as well as compression techniques used for transmission also significantly influence to the video quality. Even though the H.264/AVC video compression standard and its scalable extension is known as Scalable Video Coding (SVC) was introduced and successfully addressed the challenge, some weaknesses persist especially for high-resolution video transmissions. The size of the bitstream generated by the SVC standard for ultra-high resolution video compression can only be met if transmitted on a network with a large bandwidth. The Scalable High-Efficiency Video Coding (SHVC) were introduced to overcome problems in the SVC standards. The new video compression standard is proven to reduce bandwidth requirements by up to 46.41 % compared to the SVC standards for the same video quality. This study introduces the performance analysis of SHVC standard video transmission as compared to SVC standard. Both compression standards are transmitted and emulated through network emulator called Mininet-Wifi to get the real transmission picture. The client-server video transmission scheme with different bandwidth and various client position are set in the experiment scenario. It is found that transmission time, packet loss, and video reconstruction quality on scalable video transmission with SHVC standard is better than scalable video transmissions with SVC standard.

The main goal of hiding data is to conceal the very existence of the hidden information, therefore there is a significant demand for steganographic approaches that can ensure imperceptibility of such infromation. However, there is a limited corresponding evaluation parameters available. Most of the studies use the Peak Signal to Noise Ratio (PSNR) as a metric for imperceptibility evaluation, although it could provide less accurate results than the Human Visual System (HVS) evaluation. This paper provides a review of the existent evaluation metrics that are used to assess the quality of steganography. The examination of the correlation between the existing objective and subjective metrics is also conducted. Pixel differences metrics have a poor correlation with the subjective metrics, hence the HSV based metrics have better correlation than pixel metrics.

Recent advances in high dynamic range (HDR) capture and display technologies have attracted a lot of interest from scientific, professional, and artistic communities. As in any technology, the evaluation of HDR systems in terms of quality of experience is essential. Subjective evaluations are time consuming and expensive, and thus objective quality assessment tools are needed as well. In this paper, we report and analyze the results of an extensive benchmarking of objective quality metrics for HDR image quality assessment. In total, 35 objective metrics were benchmarked on a database of 20 HDR contents encoded with 3 compression algorithms at 4 bit rates, leading to a total of 240 compressed HDR images, using subjective quality scores as ground truth. Performance indexes were computed to assess the accuracy, monotonicity, and consistency of the metric estimation of subjective scores. Statistical analysis was performed on the performance indexes to discriminate small differences between metrics. Results demonstrated that metrics designed for HDR content, i.e., HDR-VDP-2 and HDR-VQM, are the most reliable predictors of perceived quality. Finally, our findings suggested that the performance of most full-reference metrics can be improved by considering non-linearities of the human visual system, while further efforts are necessary to improve performance of no-reference quality metrics for HDR content.

Since 1997, the Video Quality Experts Group (VQEG) has been active in the field of subjective and objective video quality assessment. The group has validated competitive quality metrics throughout several projects. Each of these projects requires mandatory actions such as creating a testplan and obtaining databases consisting of degraded video sequences with corresponding subjective quality ratings. Recently, VQEG started a new open initiative, the Joint Effort Group (JEG), for encouraging joint collaboration on all mandatory actions needed to validate video quality metrics. Within the JEG, effort is made to advance the field of both subjective and objective video quality measurement by providing proper software tools and subjective databases to the community. One of the subprojects of the JEG is the joint development of a hybrid H.264/AVC objective quality metric. In this paper, we introduce the JEG and provide an overview of the different ongoing activities within this newly started group.

Evaluation of objective quality metrics for multidimensional video scalability

A multi-stage temporal pooling mechanism is proposed in this paper for improving the prediction capability of an objective quality metric for video quality assessment. The performance of the proposed pooling mechanism is evaluated along with that of traditional pooling mechanisms in terms of linear correlation coefficient and Spearman rank order correlation coefficient on four publicly available video quality databases. The experimental results demonstrate that the proposed pooling mechanism can significantly improve the prediction capability of an objective quality metric and can be adopted for predicting the video level quality scores of longer duration video sequences with time varying video qualities.

The Video Quality Experts Group (VQEG) is preparing a programme of subjective multimedia quality tests. The results from these tests will be used to evaluate the performance of competing objective multimedia quality metrics. The reliability of the subjective test data is of great importance for VQEG's task. This paper provides an overview of VQEG's multimedia ad-hoc group. The work of this group will require subjective tests to be performed by laboratories located in Europe, Asia and North America. For VQEG's multimedia work to be successful, the subjective assessment methodology must be precisely defined and produce reliable and repeatable subjective quality data. Although international standards covering multimedia quality assessment methods are in force, there remains some uncertainty regarding the most effective approach to assessing the subjective quality of multimedia. A review of existing methods is provided. Two experiments are presented investigating the suitability of alternative subjective assessment methods (single-stimulus ACR and SAMVIQ). The results of these experiments are discussed within the context of the VQEG multimedia testing programme.

We evaluate the performance of three state of the art video codecs on synthetic videos. The evaluation is based on both subjective and objective quality metrics. The subjective quality of the compressed video sequences is evaluated using the Double Stimulus Impairment Scale (DSIS) assessment metric while the Peak Signal-to-Noise Ratio (PSNR) is used for the objective evaluation. An extensive number of experiments are conducted to study the effect of frame rate and resolution on codecs’ performance for synthetic videos. The evaluation results show that video codecs respond in different ways to frame rate and frame resolution change. H.264 shows superior capabilities compared to other codecs. Mean Opinion Score (MOS) results are shown for various bitrates, frame rates and frame resolutions.

Textures are being extensively used in next-generation video coding, image restoration and computer graphics wherein the stochastic and perceptual properties of textures are exploited. Given the prominence of texture content in image and video processing applications, texture quality assessment and objective quality metrics geared specifically towards texture are of particular importance. This paper outlines an extensive subjective study in which a total of 340 distorted images extracted from 10 reference images were evaluated by a group of 20 human subjects. The subjective Mean Opinion Scores (MOS) thus obtained from the study were used to evaluate several existing mainstream full reference quality metrics with the help of a recently developed quality assessment framework (IVQUEST). Based on the results obtained, it can be concluded that traditional objective metrics do not perform well on texture images and there is a need for specialized texture quality metrics.

Simple quality metrics such as PSNR are known to not correlate well with subjective quality when tested across a wide spectrum of video content or quality regime. Recently, efforts have been made in designing objective quality metrics trained on subjective data, demonstrating better correlation with video quality perceived by human. Clearly, the accuracy of such a metric heavily depends on the quality of the subjective data that it is trained on. In this paper, we propose a new approach to recover subjective quality scores from noisy raw measurements, by jointly estimating the subjective quality of impaired videos, the bias and consistency of test subjects, and the ambiguity of video contents all together. Compared to previous methods which partially exploit the subjective information, our approach is able to exploit the information in full, yielding better handling of outliers without the need for z-scoring or subject rejection. It also handles missing data more gracefully. Lastly, as side information, it provides interesting insights on the test subjects and video contents.

Current digital video applications require delivery of digital video content to different clients over heterogeneous networks. These clients may have different system resources and bandwidth capabilities. Thus, delivering of digital video data compressed at specific spatial resolution, quality levels, frame rates and encoding those video data at specific bit rates according to the client resources and the available network is essential. The characteristics of current video transmission system pose many problems for video delivery and scalable video coding is a highly attractive solution. This thesis investigates the existing video coding standards, especially the newest H.264/advanced video coding and its scalable extension: scalable video coding. In this thesis, we address two issues related to the state-of-the-art scalable video coding: 1) computational complexity reduction; and 2) video coding efficiency enhancement. In the first part of this thesis, we address the complexity issue of spatial, signal-to-noise ratio (SNR), temporal and combined scalable video coding. Our goal is to find a good tradeoff between computational complexity and video coding efficiency. We first establish the need to reduce computational complexity for current H.264/advanced video coding and scalable video coding framework. We then evaluate the mode distribution correlations between the base layer and its enhancement layers for different video scalabilities. After the exhaustive search over all possible block partitions is performed at the base layer, the number of candidate modes for luma and chroma blocks in a macroblock that take part in rate distortion optimization calculation at enhancement layers could be greatly reduced based on the correlations. Finally, adaptive fast mode decision schemes for spatial, SNR and temporal scalable video coding are presented. Our schemes could achieve consistent and significant computational complexity reduction with negligible loss in objective and subjective video quality and insignificant increments in bit rate consumption. The second part of this thesis is to provide a solution for the problem of high computational complexity of scalable video decoders. An adaptive encoding algorithm is proposed to reduce decoder complexity in coarse grain SNR scalable video coding.

The use of point clouds to digitally represent three-dimensional objects with both geometry and color attributes is rapidly increasing in several applications. Since storage and transmission of uncompressed point cloud data are often impractical, several lossy compression algorithms have been proposed, each exhibiting specific types of visual distortion. This creates a challenging environment for objective quality metrics, which might be effective only on a restricted number of distortion types and contexts. To evaluate the performance of the most recent objective quality metrics in predicting distortions as perceived by humans, a benchmarking study is conducted using subjective scores from observers examining models distorted with a conventional as well as a learning-based compression method, while rendering them on both a traditional flat monitor and an eye-sensing light field display. The results are then analyzed and conclusions are drawn on the correlation between recent state-of-the-art objective quality metrics and human perception on two different types of visualization devices.

To deal with congestion control in internet protocol (IP) networks, different active queue management schemes have been successfully proposed. However, many of these schemes do not take into consideration the particular Voice over Internet Protocol (VoIP) requirements, thereby potentially penalizing this traffic. In this paper, we propose and evaluate a dropping packet selection algorithm for queue management schemes. This algorithm, called Drop-SwD , protects VoIP traffic without affecting other User Datagram Protocol (UDP) and Transmission Control Protocol (TCP) traffic. Additionally, to preserve the quality of the VoIP service, the proposed algorithm selects the lowest quality packets (with less impact on the subjective quality) to be dropped. The application of the algorithm in different active queue management schemes is evaluated using objective and subjective quality metrics and intelligibility metrics as well. As a result, we show that Drop-SwD improves the final quality of the voice packets’ flow, without degrading the performance of other traffic class, with respect to the packet dropping schemes of reference.