Saliency and texture information based full-reference quality metrics for video QoE assessment

This paper studies the problem of full reference visual quality assessment of denoised images with a special emphasis on images with low contrast and noise-like texture. Denoising of such images together with noise removal often results in image details loss or smoothing. A new test image database, FLT, containing 75 noise-free "reference" images and 300 filtered ("distorted") images is developed. Each reference image, corrupted by an additive white Gaussian noise, is denoised by the BM3D filter with four different values of threshold parameter (four levels of noise suppression). After carrying out a perceptual quality assessment of distorted images, the mean opinion scores (MOS) are obtained and compared with the values of known full reference quality metrics. As a result, the Spearman Rank Order Correlation Coefficient (SROCC) between PSNR values and MOS has a value close to zero, and SROCC between values of known full-reference image visual quality metrics and MOS does not exceed 0.82 (which is reached by a new visual quality metric proposed in this paper). The FLT dataset is more complex than earlier datasets used for assessment of visual quality for image denoising. Thus, it can be effectively used to design new image visual quality metrics for image denoising.

Noise is one of the main factors that degrade image visual quality. Assessment of perceptual quality of noisy images is of critical importance for imaging systems and image processing application. In this paper, we propose a framework that employs multi-layer neural networks to predict visual quality for noisy images. In particular, the proposed method utilizes quite simple features extracted in discrete cosine transform (DCT) domain for predicting values of full-reference visual quality metrics for noisy images. The neural networks are trained and tested on noisy images corrupted by additive white Gaussian noise (AWGN). The experimental results show that prediction is quite accurate and fast. Source codes of the proposed method and datasets will be available at https://github.com/ViA-RiVaL/Blind-Noisy-Image-Visual-Quality-Prediction.

Textures are being extensively used in next-generation video coding, image restoration and computer graphics wherein the stochastic and perceptual properties of textures are exploited. Given the prominence of texture content in image and video processing applications, texture quality assessment and objective quality metrics geared specifically towards texture are of particular importance. This paper outlines an extensive subjective study in which a total of 340 distorted images extracted from 10 reference images were evaluated by a group of 20 human subjects. The subjective Mean Opinion Scores (MOS) thus obtained from the study were used to evaluate several existing mainstream full reference quality metrics with the help of a recently developed quality assessment framework (IVQUEST). Based on the results obtained, it can be concluded that traditional objective metrics do not perform well on texture images and there is a need for specialized texture quality metrics.

The increasing popularity of video (i.e., audio-visual) applications or services over both wired and wireless links has prompted recent growing interests in the investigations of quality of experience (QoE) in online video transmission. Conventional video quality metrics, such as peak-signal-to-noise-ratio and quality of service, only focus on the reception quality from the systematic perspective. As a result, they cannot represent the true visual experience of an individual user. Instead, the QoE introduces a user experience-driven strategy which puts special emphasis on the contextual and human factors in addition to the transmission system. This advantage has raised the popularity and widespread usage of QoE in video transmission. In this paper, we present an overview of selected issues pertaining to QoE and its recent applications in video transmission, with consideration of the compelling features of QoE (i.e., context and human factors). The selected issues include QoE modeling with influence factors in the end-to-end chain of video transmission, QoE assessment (including subjective test and objective QoE monitoring) and QoE management of video transmission over different types of networks. Through the literature review, we observe that the context and human factors in QoE-aware video transmission have attracted significant attentions since the past two to three years. A vast number of high quality works were published in this area, and will be highlighted in this survey. In addition to a thorough summary of recent progresses, we also present an outlook of future developments on QoE assessment and management in video transmission, especially focusing on the context and human factors that have not been addressed yet and the technical challenges that have not been completely solved so far. We believe that our overview and findings can provide a timely perspective on the related issues and the future research directions in QoE-oriented services over video communications.

Remote sensing images are subject to complex noise and distortions, which reduce their quality and can lead to loss of information. Visual quality metrics can be applied to automate and optimize remote sensing image processing. The problem of taking into account distortions typical for remote sensing data in existing test image databases is considered in this paper. The efficiency of 50 full-reference quality metrics for typical remote sensing distortions is estimated. The paper proposes a robust combined metric based on alpha-trimmed mean. The influence of linearization and the use of various correlation coefficients on the formation of a robust metric are studied. The effectiveness of the solution is tested using the cross-database approach based on the test image databases including TID2013, KADID10k, MDID and LIVE Multiply Distorted Image Quality Database.

Super-multiview content with high angular resolution: 3D quality assessment on horizontal-parallax lightfield display

The quality of experience (QoE) is known to be subjective and context-dependent. Identifying and calculating the factors that affect QoE is indeed a difficult task. Recently, a lot of effort has been devoted to estimate the users' QoE in order to improve video delivery. In the literature, most of the QoE-driven optimization schemes that realize trade-offs among different quality metrics have been addressed under the assumption of homogenous populations. Nevertheless, people perceptions on a given video quality may not be the same, which makes the QoE optimization a hard task. This paper aims at taking a step further in order to address this limitation and meet users' profiles. Specifically, we propose a closed-loop control framework based on the users' (subjective) feedbacks to learn the QoE function and optimize it at the same time. Extensive simulation results show that the proposed scheme converges to a steady state, where the resulting QoE function noticeably improves the users' feedbacks.

Most distortion/artifacts in low bit-rate compressed videos are strong and obvious to human eyes, and may even change the spatio-temporal structure of visual objects. Therefore it is important to evaluate the compound perceptual impact of spatial and temporal changes. Recent neurophysiological researches have found that the high-level spatial information (shape and depth, etc.) and temporal information (motion, etc.) are processed separately in two visual pathways in human brains. In this paper, spatial and temporal distortions are evaluated separately in two modules, and are then combined in an integration module. Two subjective viewing experiments are performed. One experiment is to evaluate the subjective quality on temporal distortion only, and second experiment is to evaluate the low bit-rate videos with both spatial and temporal distortions. Experimental results and the associated analysis show that the proposed visual quality metric is in good alignment with human perception

Omnidirectional video (ODV) streaming applications are becoming increasingly popular. They enable a highly immersive experience as the user can freely choose her/his field of view within the 360-degree environment. Current deployments are fairly simple but viewport-agnostic which inevitably results in high storage/bandwidth requirements and low Quality of Experience (QoE). A promising solution is referred to as tile-based streaming which allows to have higher quality within the user’s viewport while quality outside the user’s viewport could be lower. However, empirical QoE assessment studies in this domain are still rare. Thus, this paper investigates the impact of different tile-based streaming approaches and configurations on the QoE of ODV. We present the results of a lab-based subjective evaluation in which participants evaluated 8K omnidirectional video QoE as influenced by different (i) tile-based streaming approaches (full vs. partial delivery), (ii) content types (static vs. moving camera), and (iii) tile encoding quality levels determined by different quantization parameters. Our experimental setup is characterized by high reproducibility since relevant media delivery aspects (including the user’s head movements and dynamic tile quality adaptation) are already rendered into the respective processed video sequences. Additionally, we performed a complementary objective evaluation of the different test sequences focusing on bandwidth efficiency and objective quality metrics. The results are presented in this paper and discussed in detail which confirm that tile-based streaming of ODV improves visual quality while reducing bandwidth requirements.

Quality of Experience (QoE) is defined as the overall acceptability of an application or service, as perceived subjectively by the end user including the complete end-to-end system effects. That means QoE may be influenced by the user expectations and context, adding a subjective component to measurements. Nevertheless, it has been studied that some metrics of Quality of Service (QoS), more related to the application and thus closer to the user' side, can be correlated with users evaluations for multimedia transmissions. In this article, we build a modular framework to study the relation between QoS and QoE metrics for multimedia transmissions following the same separation of concerns as the Internet protocol suite. This framework, called BoxingExperience, relies on open-source softwares (NS3 and VLC) and Linux containers. We tested our framework in the context of video streaming from a network camera to several clients. Our simulations show that BoxingExperience can easily simulate, on a typical desktop computer, a scenario where multiple clients are connected to one streaming server.

The use of video streaming services are increasing in the cellular networks, inferring a need to monitor video quality to meet users’ Quality of Experience (QoE). The so-called no-reference (NR) models for estimating video quality metrics mainly rely on packet-header and bitstream information. However, there are situations where the availability of such information is limited due to tighten security and encryption, which necessitates exploration of alternative parameters for conducting video QoE assessment. In this study we collect real-live in-smartphone measurements describing the radio link of the LTE connection while streaming reference videos in uplink. The radio measurements include metrics such as RSSI, RSRP, RSRQ, and CINR. We then use these radio metrics to train a Random Forrest machine learning model against calculated video quality metrics from the reference videos. The aim is to estimate the Mean Opinion Score (MOS), PSNR, Frame delay, Frame skips, and Blurriness. Our result show 94% classification accuracy, and 85% model accuracy (R2 value) when predicting the MOS using regression. Correspondingly, we achieve 89%, 84%, 85%, and 82% classification accuracy when predicting PSNR, Frame delay, Frame Skips, and Blurriness respectively. Further, we achieve 81%, 77%, 79%, and 75% model accuracy (R2 value) regarding the same parameters using regression.

With the exponential increase of mobile users and internet subscribers, the utilization of voice over internet protocol (VoIP) application is increasing dramatically. People exploit different VoIP applications for effective communication, for example, Google Meet, Microsoft Skype, Zoom video conferencing applications, etc. The single-ended speech quality metrics are employed for measuring and monitoring the quality of speech. However, different types of degradations present in the surroundings distort the quality of speech. In order to meet the desired quality of experience (QoE) level of end-user while using VoIP applications, it is necessary to reduce VoIP degradations and obtain the optimized speech quality. Along that line, this paper investigates the conjunction of filtering of silence and noise as a pre-processing block with the single-ended speech quality metric under various common occurring degradations encountered during VoIP communication. This can help the internet service providers in understanding the potential root cause of decrement in quality of speech and then applying the QoE management service to fulfill desired human QoE level. Results demonstrate that the deployment of joint pre-processing on speech samples under various VoIP degradations improves the quality of speech to a great extent.

Interest in playing/streaming multimedia content in Full High Definition (Full HD) and the upcoming Ultra High Definition (UHD) is rapidly increasing. However, TVs with such resolutions has higher hardware and bandwidth requirements. Hence, advanced compression tools are essential to provide TV services in Full HD and UHD quality. The aim of this paper is to explore coding efficiency performance of High Efficiency Video Coding (HEVC) and VP9 compression tools for video content in Full HD and UHD resolutions. Performance of both video compression tools is evaluated and compared with objective and subjective video quality metrics. The results show that the HEVC and VP9 codecs for Full HD and UHD videos have comparable coding efficiency. Moreover, in several cases, Quality of Experience (QoE) for Full HD, depending on the video content and target bitrate, can outperform QoE for UHD.

Although a lot of progress has been made in the development of 2D objective video quality metrics, the area of 3D video quality metrics is still in its infancy. Many of the proposed metrics are simply adaptations of 2D quality metrics that consider the depth channel as an extra color channel. In this paper, we propose a 3D no-reference objective quality metric that estimates 3D quality taking into account spatial distortions, excessive disparity, depth representation and temporal information of the video. The metric is resolution and frame-rate independent. To estimate the amount of spatial distortion in the video, the proposed metric uses a blockiness metric. The contribution of motion and excessive disparity to 3D quality is calculated using a non-linear relative disparity measure and a frame-rate proportional motion measure. The metric's performance is verified against the COSPAD1 database. The MOS predicted using the proposed metric obtained good correlation values with the subjective scores. The performance was on average better than the performance of two simple 2D full reference metrics: SIMM and PSNR.

Executing an accurate full-reference metric such as VQM can take minutes in an average computer for just one user. Therefore, it can be unfeasible to analyze all the videos received by users in an IPTV network for example consisting of 10.000 users using a single computer running the VQM metric. One solution can be to use a lightweight no-reference metrics in addition to the full-reference metric mentioned. Lightweight no-reference metrics can be used for discarding potential situations to evaluate because they are accurate enough for that task, and then the full-reference metric VQM can be used when more accuracy is needed. The work in this paper is focused on determining the maximum number of situations/users that can be analyzed simultaneously using the VQM metric in a computer with good performance. The full-reference metric is applied on the transmitter using a method specified in the recommendation ITU BT.1789. The best performance achieved was 112.8 seconds per process.