Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework

The growing demand on video streaming services is the main motive power for video quality assessment. Due to the rapid development of video streaming services, customers are more enjoying higher quality videos, and their perceptions on video service will directly influence the service provider's performance, which makes it significant to study end-users' subjective perception, named as Quality of Experience (QoE), on video streaming for both service providers and end-users. In this paper, a QoE evaluation model is proposed to predict the end-users' perception on video streaming service considering different video content types. This QoE model, named as Video-Mean Opinion Score (VMOS) model, directly focuses on end-users' feeling. Thus, the Key Performance Indicators (KPIs), which can be straightly felt by end-users, are mapped to QoE score without considering the network parameters. The excellent performance of VMOS model for QoE evaluation on video quality has been verified via plenty subjective Mean Opinion Score (MOS) test, which includes 180 video samples with 1280 valid votes. The Pearson Correlation Coefficient between VMOS score and MOS is as high as 0.925, which indicates that this model can evaluate users' perception on video quality with almost the same accuracy as subjective test.

Multipath diversity leads to a possible higher performance for real-time high definition video, especially for medical video transmission, which would improve the stability of multiple transmission paths in the symmetrical state, and avoid the potential losses of imaging information in the communication process. Most of the previous works are always based on the single-path end-to-end transmission, although the service had been demonstrated that it is unable to meet the rigorous demand for the RT-HDMV. In the paper, a multipath relay service based on the QoE (quality of experience) evaluation method is proposed for the RT-HDMV (real-time high definition medical video). The method eliminates several of the limitations in the existing methods for some conventional single-path transmission. It can fully utilize the finite network resources and transmission bandwidth to meet the users’ demands of the RT-HDMV to get a better score of the QoE. We use a four-stage framework to evaluate the QoE, which consists of constructing the multipath relay transmission for the RT-HDMV, calculating the weights of diversified QoS parameters in the multipath, designing the load distribution strategy by the mapping between the QoS (quality of service) and QoE, and redefining the rule of the QoE evaluation. Many experiments show that the proposed design scheme achieves weighting of the transmission sub-paths and computes the QoE score. Compared with the state-of-art methods in the single path transmission scene, our framework mainly gains the excellent performance for the RT-HDMV.

Betas: Deriving quantiles from MOS-QoS relations of IQX models for QoE management

Introduction and penetration of high-speed and innovative mobile data services including social media and video services have been increasing significantly across the globe. This growth is driven by deployment and availability of capable and affordable mobile networks and smartphones. These trends have also been observed in Ethiopia as the country’s sole incumbent operator ethio telecom has expanded coverage and capacity of its mobile data network, especially for universal mobile telecommunications system (UMTS) network. The successful provision of these innovative services requires mobile operators to have continuous monitoring of the quality of their networks and customer satisfaction. To that end, this monitoring allows the operator to make timely network optimization, marketing and other relevant decisions that enhance service quality. Traditionally measurement and analysis of network quality of service (QoS) is performed using data from network management system (NMS) and drive/walk test (DT/WT). In recent times, the usage QoS data from crowdsourcing (CS) is also emerging as useful monitoring tool. However, all these methods do not fully capture users’ quality of experience (QoE), which is a function of not only network QoS but also other factors including subjective users’ technical literacy and opinion. As a result, operators have recently provided attention not only for QoS but also QoE. Yet, most quality related analyses are performed using data from only subjective survey, NMS, DT/WT, or CS. It is not straightforward to find quality investigation based on data from all the methods. More importantly, such quality investigation has not been made in the context of Ethiopia’s mobile market. In this paper, we perform QoS and QoE evaluation for UMTS mobile data service based on quality data from all NMS, WT, CS and subjective survey for a business area in Addis Ababa. The evaluation is made using downlink user throughput and mean opinion score (MOS) metrics. The obtained 50th percentile throughput results from NMS, WT and CS are good enough for web browsing and offline video streaming but not for high definition (HD) video calls and live HD video streaming. Furthermore, CS provides optimistic results compared to NMS and WT and its 90th percentile throughput is very good even for live HD video streaming. Moreover, the obtained MOS results from subjective survey almost fit to the QoS results, thus indicating that the network QoS is a significant contributor to users’ perceived QoE.

무선 접속을 통한 인터넷 서비스가 보편화된 최근 사람이 서비스를 이용하는 과정에서 실제로 느끼는 품질인 사용자 체감 품질(QoE; Quality of Experience)의 중요성이 더욱 강조되고 있는데 사용자 체감 품질은 서비스 품질 (QoS; Quality of Service)와 같이 객관적인 수치화가 불가능하다는 특징이 있다. 유선과는 달리 무선 접속을 통해서 제공되는 IP 서비스는 매우 많은 요인에 의해서 사용자 체감 품질이 영향을 받을 수 있다. 따라서 본 논문에서는 대표적인 무선 접속 서비스인 HSPA (High Speed Packet Access)를 통해서 음성 서비스가 제공될 때 측정 가능한 품질지표를 선정하고 이들에 대한 실측값을 통계적으로 분석하여 서비스 품질과 사용자 체감 품질 지표와의 상관관계를 밝히고자 한다. 분석 결과 RSSI (Received Signal Strength Indicator)와 전송 지연의 상관관계가 매우 높고 그에 이어 전송 지연과 MOS (Mean Opinion Score)와 매우 높은 상관관계를 가짐을 알 수 있었다. As internet services are being provided through a wireless access, the importance of quality of experience (QoE) is stressed that is defined as the quality that indicates user's actual feeling when a service is provided. Unlike quality of service (QoS) that can be expressed as a numerical value, it is difficult to represent QoE in an objective way. If an internet service is serviced over a wireless channel, its QoE can be affected by a number of factors such as fading, mobility and so on. This paper, therefore, attempts to specify the relationship between QoE and QoS by conducting practical measurements for the voice service through 3G high speed packet access (HSPA) access network. Analysing the measured results, it has been shown that received signal strength indicator (RSSI) has a great influence on mean opinion score (MOS) through transmission delay.

Massively Multiplayer Online Cloud Game (MMOCG) is a combination of cloud gaming and Massively Multiplayer Online Game (MMOG). Running MMOGs on the cloud can reduce the cost for the players, attract potential players, and meet the nowadays preference of mobile devices. However, the MMOCGs introduce new challenges that need to be addressed. MMOGs usually consume more computational resources because of their sophisticated graphics. Hosting MMOCGs can be more expensive for game operators than hosting other genres of games on the cloud. In order to earn a decent profit, the cost of hosting MMOCGs must be reduced to an acceptable cost. Moreover, MMOCG introduce extra cloud servers to MMOG, which increased the burden of network transmission. So, the MMOCGs must have a good quality of service (QoS) and quality of experience (QoE) to retain the players. In this thesis, we study the server allocation problem to improve MMOCGs in many aspects, including money cost, QoS, and QoE. We proposed three different server allocation strategies to address the challenges of MMOCGs in different situations. We first consider the MMOCG system with several cloud gaming platforms that lie in different locations in the world, each can provide a type of server that can host MMOG on them. The cloud gaming system only obtains the information necessary for gaming. With only the log-in information, a server allocation problem is defined to reduce the money cost of hosting MMOCGs and provide an acceptable network latency for the players. The problem is formulated into an optimization problem that aims to minimize the weighted combination of server rental cost, data transfer cost, and network latency. A genetic algorithm is developed to solve the minimization problem. Extensive experiments are conducted to evaluate the proposed method and compare it with the state-of-the-art as well. The experimental results show that the method gives a lower money cost and a shorter network latency most of the time. We then consider the MMOCG system that can collect the past logging in and out data of the players. We proposed a server allocation problem that aims to reduce the cost of cloud server rental and improve the QoE of the system. A genetic algorithm with gaming time prediction is proposed to solve the problem. The gaming time of the players logging into the system is predicted by the model trained by a neural network. We conduct experiments to evaluate our method and the proposed gaming time prediction model. The results show that our prediction model can achieve almost highly accurate prediction and the server allocation strategy we proposed outperforms the prior art in both server rental and QoE. At last, we consider the influence of QoE on the movement of players during gaming time and propose a server allocation problem aiming to improve the QoE of players during gaming time. We propose a genetic algorithm with trace prediction to address this problem. A trace prediction that predicts the future positions of players for a few time steps when they log into the MMOCG system. Experiments are conducted to evaluate our method by measuring the network latency, mean opinion score, and the time consuming computation. The results show that our method outperforms the genetic algorithm without trace prediction in over 50% of the cases.

The volume of Internet video traffic which consists of downloaded or streamed video from the Internet is projected to increase from 42,029PB monthly in 2016 to 159,161PB monthly, in 2021, representing a 31% increase in the Compound Annual Growth Rate (CAGR). The market for mobile network operators is unpredictable, fast-paced and very competitive. End users now have more options when choosing service providers. With superior network Quality of Experience (QoE), service providers can increase margins by charging more for better quality. Packet corruption occurs when the receiver cannot correctly decode transmitted bits. This study identified the threshold at which the QoE of video streaming services became unacceptable due to the effect of packet corruption. In this paper, several experiments were carried out on video streaming services, creating disturbances to evaluate the user satisfaction level using the mean opinion scores. Network Emulator (NetEm) tool was used to create the packet corruption experienced during the video sessions and the QoE for different packet corruption percentages was established. From the experiments conducted, we found that user QoE decreased as the Packet Corruption Ratio (PCR) increased. With knowledge of the effect of the PCR, service providers can ensure that the PCR is kept within acceptable limits from end-to-end and this will ultimately lead to superior QoE from end users, which will in turn translate to improved subscriber base and profitability.

With the development of streaming media service, Quality of Experience (QoE) has become the key competency for those content providers to attract customers. Many researches on QoE assessment have been conducted, but most of them had problems that those subjective experiments are difficult to be regulated and the data is not collected uniformly because of long tail effect. The purpose of this paper is to give a review of QoE assessment, which includes the concept of QoE, the influential factors, the approaches of experiment and QoE analysis and evaluation. Through the survey which shows the development of the QoE assessment, this paper indicates the future trends and challenges. Based on such current challenges and future analysis, an experiment-oriented methodology of QoE assessment is raised by conceptual model description. Also, those technologies which are based on different subjects including communication study and psychology study for supporting the methodology are described. The problems of small data volume, long tail phenomenon in QoE experiments and influence of malicious feedback will be solved by implementation of the personal QoE assessment methodology.

For research on quality of experience (QoE), mean opinion scores (MOS) are widely chosen as the results of subjective tests and the ground-truth reference for further research on objective quality modeling. Furthermore, the results of objective quality modeling are used for QoE management subsequently. Therefore, the performance of QoE management process actually depends heavily on MOS. However, the rationality of MOS for QoE management is not yet technically proven in the literature. In this paper, we first prove that subject homogeneity is implicitly assumed for obtaining MOS by modeling the arithmetic averaging process from a systematic viewpoint. However, we point out that actually subjects exhibit variability in terms of quality assessment. Then we elaborate that this mismatch may results in failures if we conduct QoE management based on MOS. Finally we propose a utility-based averaging method (uMOS) which improves the performance of QoE management.

Voice over IP (VoIP) services are now offered on many mobile networks. However, delay spikes, which are sudden large increases in packet delay, affect the quality of VoIP on mobile networks. This paper proposes a quality of experience (QoE) indicator considering delay spikes. The QoE indicator combines a voice quality indicator such as ITU-T P.862 and the delay part of ITU-T G.107, which is a quality indicator for VoIP. This paper also proposes a transmission control method that selects the codec type and the sending interval of voice packets so as to maximize the QoE indicator. The method improves the mean opinion score (MOS), which indicates QoE as a value between 1 and 5, by up to 0.4.

Quality of experience (QoE) serves as a direct evaluation of users' experience in mobile video transmission and is critical to ensure good network service. Although many efforts have been made to predict QoE based on network parameters of the user terminal equipment, it is difficult to predict QoE based on Quality of Service (QoS) offered by the network servers. In this paper, a machine learning based QoE evaluation method is proposed to evaluate user QoE in real-time by analyzing the QoS characteristics for mobile video transmission. For this purpose, we construct a large-scale dataset by collecting more than 300 thousand pieces of metrics data with two kinds of key quality indicators (KQIs) describing the QoE and 91 key performance indicators (KPIs) describing the QoS. A two-process feature subset selection (FSS) method consisting of single parameter pre-FSS and multi-parameter FSS is then proposed to find the KPIs related to KQIs. An Extra-Trees model is finally developed to learn the relationships between the KPIs and KQIs. By employing machine learning and data analytics on network data with the data-driven framework, the proposed method can predict the user QoE according to the QoS of network servers. The results prove that our proposed method can outperform other state-of-the-art approaches.

The development of Quality of Experience (QoE) models using Machine Learning (ML) is challenging, since it can be difficult to share datasets between research entities to protect the intellectual property of the ML model and the confidentiality of user studies in compliance with data protection regulations such as General Data Protection Regulation (GDPR). This makes distributed machine learning techniques that do not necessitate sharing of data or attribute names appealing. One suitable use case in the scope of QoE can be the task of mapping QoE indicators for the perception of quality such as Mean Opinion Scores (MOS), in a distributed manner. In this article, we present Distributed Ensemble Learning (DEL), and Vertical Federated Learning (vFL) to address this context. Both approaches can be applied to datasets that have different feature sets, i.e., split features. The DEL approach is ML model-agnostic and achieves up to 12% accuracy improvement of ensembling various generic and specific models. The vFL approach is based on neural networks and achieves on-par accuracy with a conventional Fully Centralized machine learning model, while exhibiting statistically significant performance that is superior to that of the Isolated local models with an average accuracy improvement of 26%. Moreover, energy-efficient vFL with reduced network footprint and training time is obtained by further tuning the model hyper-parameters.

Recently user quality of experience (QoE) is employed in evaluating end user satisfaction in communications systems. Generally, current approaches for QoE assessment are obtrusive, laboratory based and offline. Estimation of user satisfaction in static manner based on mean opinion score is not directly related to instantaneous individual end user contentment. In this paper, based on correlations between user's physiological signals and her/his feelings about the service quality, a non-intrusive and user centric QoE assessment system for voice communications is developed. The findings of this study indicate that the emotional patterns in response to the changes in channel quality can be adapted to estimate the level of satisfaction in a QoE assessment system in a live manner. Based on experimental results, two categories of users are identified: sensitive and insensitive towards quality degradations. The results indicate that for the sensitive users, our non-intrusive subjective quality assessment method outperforms ITU-T P.563 standard with respect to root mean square error; while, the results are much better among the insensitive users.

The MPEG-DASH protocol has been rapidly adopted by most major network content providers and enables clients to make informed decisions in the context of HTTP streaming, based on network and device conditions using the available media representations. A review of the literature on adaptive streaming over mobile shows that most emphasis has been on adapting the video quality whereas this work examines the trade-off between video and audio quality. In particular, subjective tests were undertaken for live music streaming over emulated mobile networks with MPEG-DASH. A group of audio/video sequences was designed to emulate varying bandwidth arising from network congestion, with varying trade-off between audio and video bit rates. Absolute Category Rating was used to evaluate the relative impact of both audio and video quality in the overall Quality of Experience (QoE). One key finding from the statistical analysis of Mean Opinion Scores (MOS) results using Analysis of Variance indicates that providing reduced audio quality has a much lower impact on QoE than reducing video quality at similar total bandwidth situations. This paper also describes an objective model for audiovisual quality estimation that combines the outcomes from audio and video metrics into a joint parametric model. The correlation between predicted and subjective MOS was computed using several outcomes (Pearson and Spearman correlation coefficients, Root Mean Square Error (RMSE) and epsilon-insensitive RMSE). The obtained results indicate that the proposed approach is a viable solution for objective audiovisual quality assessment in the context of live music streaming over mobile network.

Currently, the satisfaction of user is becoming one of the most important topics concerned by the service provider. Existing methods used for evaluating the quality of experience (QoE) mostly rely on user survey and scores from the user, which are too subjective and need much processing time and cost. Pervasive computing environment brings the method of QoE evaluation a new viewpoint through kinds of evidence context related to QoE. In the paper, the QoE evaluation method in pervasive computing environment is studied, QoE evaluation parameter model is enhanced, a rough-set based algorithm is proposed to reduce context attributes and determine the weight of each attribute, the algorithm has been validated on video streaming service, and the architecture of QoE evaluation system is described.