QoXcloud: A Cloud Platform for QoE Evaluation

Received: 9 December 2014; Accepted: 9 March 2015

Eduardo Saiz, Eva Ibarrola, Leire Cristobo and Ianire Taboada

Faculty of Engineering of Bilbao. University of the Basque Country UPV/EHU, Spain
E-mail: eduardo.saiz, eva.ibarrola, leire.cristobo, ianire.taboada@ehu.eus

Abstract

The current financial situation, together with the new market conditions, has led to major changes in the ICT sector over the last few years. Many services that in the past were only offered by operators are now held by third parties through the cloud, which has caused a shift towards new business models at the expense of a more traditional market. Furthermore, these economic changes have conducted to socio-cultural transformations with great impact on the user’s behaviors.

In this paper a cloud platform for the measurement and evaluation of the Quality of Experience (QoE) is presented. The platform is based in a model (QoXphere) that ensures the user satisfaction in terms of Quality of Service (QoS) and the provider economic benefits, as demanded in current market situation. In addition, the proposed cloud architecture intends to help in the advance of the work item recently opened in ITU-T to establish an ITU recognition procedure of testing laboratories with competence in ITU-T Recommendations by providing a unified cloud environment in which to validate them.

Keywords

QoS
QoP
QoE
QoBiz
QoX
Cloud

1 Introduction

The continuous economic growth in the telecommunications sector over the last decades has slowed down in recent years. The financial crisis has caused a drastic reduction in economic activities and investments in this area and this situation has promoted new business models. Many of the services that had been only offered by operators over the last years are now being provided by third parties through the cloud, driving the rise of this model as compared to the traditional sort. (Figure 1).

Figure 1 Traditional service model vs. Cloud services model

These new models have also led to a socio-cultural transformation with great repercussion on the users of this sector. The user’s behaviour has evolved as a result of this globalization progress. Nowadays, full time connectivity and ubiquity are highly required by users, who are becoming more demanding in terms of capacity and quality of service (QoS). These changes are significantly responsible in the boosting of the cloud service model. Nevertheless, the proliferation of the cloud services can lead to the existence of a ‘nebula', from both the point of view of the user and the provider, when implementing policies to manage the demanded QoS.

Since user-centric viewpoint in QoS evaluation is becoming of greater importance nowadays, a new approach to what should be addressed for an optimal QoS management is being embraced. This approach implies an increasing complexity in terms of QoS measurement that must be taken into account. In this regard, new QoS-related terms have emerged in the last years, in both the standardization bodies and the scientific environment. Terms such as the quality experienced by the client (QoE, Quality of Experience) [1], the quality perceived by the user (QoP, Perceived QoS) [2] or the quality of business (QoBiz) [3] have helped to transmit in a more accurate way, the new dimension of what QoS should include. Recently, all these concepts have lead to a new term, which encompasses them all under the name of QoX [4].

In this scenario, the new cloud services model can complicate even more the QoS management, although it can also provide significant benefits in its evaluation. Namely, the cloud itself can be used as a mean for the globalization of the QoS measuring tools and results, which might be of great benefit to the different stakeholders of the sector.

In this paper we present a cloud platform for the measurement and evaluation of the QoS and QoE in the telecommunication services. This platform aims to aid in the new ITU proposal to establish testing laboratories all over the world with the ITU recognition so the measurement platforms and their results can be shared by all the telecommunications community.

The remainder of the paper is organized as follows: Section 2 summarizes related work in both standardization bodies and scientific area. Section 3 describes the new QoXcloud architecture and the QoS model that is adopted as framework [5]. In Section 4, two of the prototypes developed to be included in the cloud QoE platform are presented. Section 5 describes in detail the experiments carried out to validate the prototypes presented in Section 4 and, finally, Sections 6 and 7 contain some conclusions and final remarks.

2 Background

2.1 Standardization Bodies and the Cloud

The cloud services model that is being deployed these days has been defined for the provision and use of any kind of service, application, information and infrastructure with particular emphasis on speed and scalability, according to the needs of each organization, company or user (Figure 2). Given the nature of this model, it’s reasonable to see its suitability for organizations with a large number of users, companies or providers. That is the case, for example, of a standardization body. Since this kind of organization has to deal with the provision of multiple services, within its specific coverage ambit, it seems rather appropriate to provide those services through the cloud.

Figure 2 Service provisioning in the cloud

However, and despite the growing activity around this model, there are still very few references coming from the standardization bodies about the management and provision of QoS in the cloud. In particular, the ITU-T SG-13 Study Group, “Future networks including cloud computing, mobile and next-generation networks” [6], has raised some cloud-related questions (Q.17/13, Q.18/13 and Q.19/13) but not with any particular QoS approach. The ITU-T Study Group 11 (SG-11), “Protocols and Specifications” [7, 8], has received proposals showing interest in developing a global architecture (as seen in Figure 3) in which to perform interoperability tests, the validation of recommendations or even the development of measurement procedures for technologies and services that might be of interest for the ITU itself, the Conformity & Interoperability Group [9] or any other collaborating members of the ITU-T study groups such as regulators, operators, providers and clients. This initiative seems to fit very well with the cloud services model. The ITU-T Study Group 12 (SG-12), “Performance, QoS and QoE”, has contributed as well to this matter with several proposals for the evaluation of QoS and QoE through the cloud [10, 11]. These proposals will be later discussed since they have been considered in the development of the platform presented in this paper.

Figure 3 ITU proposed cloud architecture

The European Telecommunications Standards Institute (ETSI) has also published some reports related to the QoS management in the cloud. Specifically, the TR 103 125 report, “CLOUD: SLAs for Cloud services” [12], deals with the complex problematic of the management of service level agreements in an environment of cloud-based services provisioning.

2.2 Research on QoS Management in Cloud Services

QoS management in the telecommunications environment is a complex issue. Ever since the appearance of newer QoS-related terms that have moved the focus from the objective viewpoint of the Network Performance (NP) parameters to the subjective evaluation of the user’s perception (QoP) and experience (QoE), the network resilience (QoR) [13] or the Quality of Business (QoBiz), there is the need of an effective QoS management that does not study each of these terms isolatedly, but as a whole (QoX). The aim is to improve the customer’s loyalty and satisfaction, thus ensuring the interests of the different stakeholders (providers, regulators and users).

These factors are also needed in the cloud services model, where resource allocation is quite vital for a proper QoS management. Works by Sharkh [14] in this matter, reveal that QoS provisioning is a rather important challenge. When users execute sensible tasks on a cloud environment, they need a networking service with adequate QoS standards to ensure the successful delivery of their application data. Furthermore, users might require extra guarantees that their information is securely deleted or properly encrypted on the cloud, which implies an extra pressure on the performance and an additional difficulty to comply with QoS requirements.

For this challenge, many authors have made proposals to achieve an effective QoS management in the cloud. According to Cao et al [15], for example, the essence of the cloud services model is to provide network services. As to the user, resources in the cloud should be acquirable anywhere and accessible anytime ‘on demand’ or by ‘pay-per-use'. In combination with Multi-Agent technology and SOA, they propose a cloud architecture that includes physical devices, a cloud services provision layer, cloud services management and a Multi-Agent layer to guarantee QoS in the cloud.

Another interesting work is the one developed by Ferretti [16], in which an architecture specifically designed to respond effectively to the QoS requirements of the cloud applications is proposed (Figure 4). These application requests are handed to a virtual execution environment manager, responsible for optimizing and scheduling the availability of the cloud resources.

Figure 4 QoS management in the cloud (Ferretti)

Nevertheless, and despite all the recent research, QoS management in the cloud is still a relatively unexplored topic with a great importance that should be taken into account. This, along with the aforementioned ITU proposal of a cloud platform for the validation of recommendations, interoperability tests and other services and technologies, is the primary motivation for the development of QoXcloud, a cloud platform for the measurement and evaluation of QoS and QoE, that can also help the ITU to establish testing laboratories all over the world, with ITU recognition, for their results and developed platforms to be shared amongst the telecommunications community.

3 QoXcloud Platform

In this section, the QoXcloud platform architecture is presented. Figure 5 shows the three main entities in which the platform is divided.

Figure 5 QoXcloud architecture

Services Entity
IT Infrastructure
Information Entity

It must be remarked that the QoXcloud architecture has been designed in order to satisfy the related ITU-T recommendations and standards (Rec. G.1000 [2], Rec. G.1011 [17], Rec. M.3050 [3] as well as Recs. Y.1541, Y. 1542 [18] and Y.1543 [19]).

3.1 Services Entity: QoXphere Framework

QoXcloud services entity is based upon the design of the QoXphere framework [5] (Figure 6), which establishes relationships amongst all the aspects of the QoS (QoX), organized in the following four layers that comply, amongst others, these respective ITU-T recommendations:

Figure 6 QoXphere framework within the QoXcloud architecture

Intrinsic QoS Layer: This layer analyzes the objective QoS parameters evaluation at the Network Performance level, as defined in ITU-T Y.1540 and Y.1541 Recs. [20, 21].

For this layer implementation, two platforms has been designed: QoSmeter [22] and LabQoS [23]. The first one is a neutral QoS measurement infrastructure meant to measure objective network parameters through a wide variety of tests that help determine the degree of compliance to the SLA. The second one, LabQoS, is a further development of QoSmeter focused on testing and simulating experimental scenarios.
Perceived QoS Layer: Based on the four viewpoints of ITU-T G.1000 recommendation [2], this layer is considered the ‘core’ of the QoXphere model, since the whole framework is defined in accordance with it. In fact, QoXphere has been proposed as the reference framework for future updating of the ITU-T G.1000 recommendation [24].

For this layer implementation, the OBservatory for the Analysis and Validation of QoS (ObavaQoS) has been defined. This subsystem helps determining the required quality indicators (KQI) and performance indicators (KPI) to estimate the QoS perceived in terms of the referred ITU-T G.1000 recommendation.
Assessed QoS Layer: Based on the Xiao’s CSAT model [25], this layer evaluates the user’s satisfaction based on the user’s experience, which is based in its turn on QoP, as evaluated in the previous layer. For this layer’s development, a general-purpose on-demand survey management and configuration subsystem is used. This subsystem, named ENQoS, is based upon the ITU-T G.1030 Recommendation [26].
QoS Business Layer: The top layer of the framework is oriented to guarantee the provider’s profitability based on the feedback from the other three layers.

In particular, this layer has been defined to comply the ITU-T E.419 Recommendation [27]. Also, since the Telemanagement Forum has also shown interest in the ITU cloud platform, this layer also accomplishes the references of the TMF GB935 document on business metrics [28].

On a final note to the services entity, it must be said that all the referred subsystems (QoSmeter, LabQoS, ObavaQoS and ENQoS) required being adapted to contemplate their usage in a cloud scenario.

3.2 IT Infrastructure

The IT Infrastructure in the QoXcloud platform is defined as the set of facilities (hardware, software, network devices and the required interconnections) that is capable of running any kind of measurement test within its boundaries. Since different measurement tests require different test scenarios or layouts, the IT Infrastructure has been defined as a modular and scalable distributed architecture, capable of accommodating the widest diversity of tests possible by adding additional test servers or designing different test applications. The aforementioned architecture of QoSmeter [22] (Figure 7) has been used defining the interaction of four distributed entities:

Figure 7 QoXcloud IT Infrastructure: QoSmeter Infrastructure

Parameter Measurement Services (PMS): A PMS is the joint of the required test measurement modules and the auxiliary servers towards the tests are held. For example, different endpoint locations for a particular measurement service.

Any measurement test can be designed for its integration in the QoXcloud IT Infrastructure, by complying with the requirements of the PMS entity and additional authorization policies, as discussed later in the paper.
Storage Server (SS): All results obtained by the PMS modules are persisted in the databases of this server, which is also the main element of the Information entity, as seen in Figure 5.
Central Server (CS): It is the core of the QoSmeter infrastructure and it’s in charge of four main actions:
- User management: In charge of the management of user rights according to the roles or groups of belonging.
- System management: Test configurations, reports definition and system status monitoring are managed in this module of the CS.
- Scheduling: This module is in charge of the scheduled execution of tests and their configuration, if applicable.
- Report Generation: Finally, this module is capable of generating a specific report from the required data obtained from the Storage Server.
Test App Container (AC): The fourth entity of the IT infrastructure is an application container from which the measurements are launched (Figure 8). Using a Single Sign On (SSO) procedure, the AC authenticates the user within the infrastructure domains and obtains the list of PMS available for execution.

All the entities interact within a specific authentication and authorization policy. This A & A policy allows the system to distinguish different user types and the specific measurement tests to which they have credentials.

In this way, a client can execute a particular test related to his contracted service, a provider can use the infrastructure as a neutral reference of its users’ results, and a regulator or standardization body can access specific test reports for a particular recommendation to validate.

Figure 8 Sample captures of the Test App Container

3.3 Information Entity

As a result of the adoption of the QoSmeter infrastructure as the IT infrastructure, the Information entity of the QoXcloud platform is consequently affected, being the Storage Server of QoSmeter the most adequate implementation for this entity.

The definition of the SS server should be made compliant to the ITU databases in order to get the whole QoXcloud platform validated for its integration.

4 Validation

In this section the two prototypes developed for the validation of the QoXCloud platform are described. These prototypes consist of two measurement platforms for the evaluation of the QoE in two of nowadays most used services:

The Internet access service, as defined in the ETSI EG 202 057-4 guide [29].
The web service, as defined in the P.STMWeb drafts [10, 11], which were under active development at the time this prototype was proposed.

The definition of the test scenario for these services, as well as the methodological approach to their respective evaluation is explained in the following lines, prior to the presentation of the validation results in Section 5.

4.1 Internet Access Service Prototype

The scope of this prototype (Figure 9) is to measure those parameters that help determine the QoS delivered to users in the Internet access service, as defined in the aforementioned ETSI guide.

Figure 9 QoS measurement in Internet access service (prototype)

As a side note, this same prototype is intended to embrace a new recommendation that is being promoted from the ITU SG-11 in the last couple of months: the ITU-T Q.Int_speed_test draft [30]. Working on the basis of existing regulation and recommendations as the ETSI guide [29], this new recommendation aims to unify the methodology for the measurement of Internet speed by end-users.

Since it is still in an early stage of development, it has been considered appropriate to observe its evolution in the next couple of months in order to adapt the prototype for its requirements. In the meantime, the current implementation of the prototype has been designed for the measurement of the following parameters:

Data transmission speed achieved: Mean value and standard deviation of the transmission rate achieved (in kbit/s), as well as the maximum and minimum values calculated, respectively, as the highest 95% and the lowest 5% of the data transmission rate achieved (in kbit/s).
Delay (one way transmission time): Mean value and standard deviation of the delay (in milliseconds).

For these parameters to be measured, a test file has been defined, as specified in Annex D of the guide [29]. That file should consist of incompressible data, so the file remains the same no matter how many distorting/compressing effects the network might introduce.

Also it should be at least twice the size (in kbit) of the theoretically maximum data transmission rate per second (in kbit/s) of the Internet access under consideration. In the particular case of our prototype, that multiplier factor has been set to 4x, to also comply local regulations [31].

To accomplish the measurement of the aforementioned parameters, two subtests have been defined:

Speed Test: This test consists of measuring the elapsed time in transmitting the correspondent test file for both the downlink and uplink transmission rates of the contracted Internet access service, as defined in the previous paragraphs.

Given the size of the test file for each link, the respective transmission speed achieved is calculated as the size of the test file (in kbit) between the measured time (in seconds) in each link subtest.

The test scenario consists of the app container installed in the user device and three geographically dispersed servers, located in Madrid (Spain), Dallas (TX, USA) and Amsterdam (The Netherlands). Those auxiliary servers have the required logic to process the download request from the client and response back with the specific file. The container is responsible for measuring the elapsed times and save them in the SS.
Latency Test: This other test consists of the same test scenario presented in the speed test specification, but, in this particular case, the app container launches a poll of 20 ping interactions to each destination host. Times are measured at the beginning and end of each interaction. Please note that, as a soft adoption of the ETSI guidelines, the two-way latency instead of the one-way delay is measured, since time measurements are done on the client side (from the app container).

After the polling is done, the maximum, minimum and average latency values are stored, as well as the jitter (standard deviation), the percentage of losses and an estimation of the Mean Opinion Score (MOS) value based on the E-model, as defined in the ITU-T G.107 Recommendation [32].

4.2 Web Service Prototype

The second prototype designed for the validation of the QoXcloud platform is based on the P.STMWeb Draft [10, 11], which defines a subjective test methodology for the web browsing service. As a complement to this draft, the jointly developed G.QoEWeb Draft [33] enumerates the QoE influence factors in the web browsing service. Both drafts are also intended to update and supplement the ITU-T G.1030 Recommendation [26].

Therefore, the Web Service Prototype takes into account those influence factors, including network bottlenecks and other hardware and software specifications in the test facilities disposed for the validation of the prototype, and adopts the test methodology of the P.STMWeb draft, in accordance to its requirements.

Concretely, the aim of this draft is to define a test to evaluate the QoE of the participants by introducing alterations on the network level, such as in the round trip time (RTT) or in the bandwidth; as well as on the application level (for example, page load times). The draft defines 6 bandwidth conditions (ranging from 64 Kbps to 2048 Kbps) with a fixed RTT of 20 ms. Those conditions are to be applied when navigating two kinds of contents: A news site and a photo gallery site.

The facilities for this test consist of a controlled LAN with a User Device (UD) with a basic web browser, a Network Emulator (EMU) capable of inducing the conditions in the link UD-EMU, a local Content Server (CS) that hosts well constructed news and photo gallery sites, a Page Health Monitor (PHM) to analyze de availability of web contents that might be hosted outside the controlled LAN, and a router, required for LAN interconnection (Figure 10).

Figure 10 QoE measurement in web service (prototype)

The implemented methodology consists of a warm-up phase where users are guided throughout 3 sessions (condition periods of 150 seconds), followed by two rounds of 6 × 2 session iterations per round. This is, each round consists of the 6 bandwidth conditions randomly presented to the user for each of the 2 referred content types. After each session, users are prompted a form in which to evaluate their navigational experience to the just finished condition in a MOS scale from very bad (1) to very good (5).

In the particular implementation of our prototype, no external contents to the controlled LAN environment were to be accessed as everything was served from the CS. For that reason, no PHM was used.

5 Results

5.1 Internet Access Service Results

According to the Ministerial Order ITC/912/2006 [31] of the Spanish Government, any Internet Service Provider (ISP) that operates in the Spanish territory and fulfils certain specifications is required to publish results on the levels of QoS they offer and deliver on a quarterly basis.

Therefore, it might be of great interest for the validation of the prototype to present the results of the Internet Access service for a specific ISP obtained from the prototype implementation as compared to the correspondent results published by that same ISP in the Spanish territory, according to the specifications of the Spanish Ministerial Order.

It can be seen in Table 1 that, for the volume of tests performed by users of the QoSmeter platform (integrated in the QoXcloud IT infrastructure) whose Internet access service has been contracted with the ISP to which results are being compared, the obtained results are relatively stable over time but its mean value is below the correspondent results published by the ISP.

Table 1 Sample comparison of official ISP bandwidth results vs.

This does not mean that the results of the prototype are not valid. Actually, what explains that difference is that users who took part in the Internet access service tests are distributed throughout the Spanish geography and, even for the same contracted bandwidth, they have very different access conditions amongst them, such as different distances to the ISP facilities, different last mile network technologies or even geographical difficulties that might require a high investment by the ISP for a better access provision; while probes that are carried out by the ISP itself are emulated in a controlled environment that ensures the best compliance with the standards required by the Ministerial Order, thus not contemplating real case scenarios.

For this reason, the importance of using a third-party neutral measurement infrastructure can be crucial for a real compliance of current recommendations and regulations. The results obtained in this prototype validation are a good account of it.

5.2 Web Service Results

The Web Service Prototype was tested with a group of 46 participants divided in two groups: experts and non-experts. The non-expert group was composed of 23 undergraduate students of the Telecommunication Engineering Degree at the University of the Basque Country in Spain, whilst the remaining 23 participants were professors and researchers of this university.

The results presented in this section were obtained during the two months the tests were held (October 2013–November 2013). MOS value is calculated within the 95% confidence interval and results are disaggregated by content type and user category.

Figure 11 presents the MOS results for the totality of the participants without group disaggregation. The graphic presents quite linearity for both content categories. However, a lower tolerance to contents with higher page weights can be observed, as photos are rated lower. This effect can be better appreciated in faster conditions, where news text is loaded rather quickly in comparison to the heavy images of the photo contents. When in worse conditions, since loading times are not good even for news contents, both contents’ graphics are quite similar.

Figure 11 MOS vs. BW (results for 46 participants)

In Figure 12 the comparison between the results of experts and non-experts is shown. It can be perceived that, in general, experts seem more demanding than non-experts, as well as more accurate when deciding the MOS value. This effect is easily seen in the news category. However, there is no clear tendency for photo contents, as this kind of content is more difficult to evaluate since the final page size depends mainly of the varying sizes of the photos to load.

Figure 12 Experts vs. Non-experts MOS results

Another interesting effect observed during the two-month testing period, was derived from the test duration itself: After a few conditions evaluation, the panel of experts seemed to have a very clear decision on what to answer in the subsequent forms within the first seconds of the conditions under way.

For this reason, half of the experts were conducted a short version of the test in which each condition period was reduced from 150 to 100 seconds, in order to compare whether the results showed significant variations or not. Given the similarity of the graphs for both versions of the test, it was concluded that the test durations defined in the P.STMWeb Drafts [10, 11] could be revised downwards for the lesser fatigue of the participants.

Additionally, the fixed range of bandwidth values defined in the draft should be adapted to the existing offer of Internet access speeds in the region or country in which to conduct the tests, in order to obtain more consistent results from the participants with their real web browsing experience.

These proposals, as well as the obtained results and graphics, were included in the contribution to P.STMWeb Draft enhancement, presented for that purpose [34].

6 Conclusions

In this paper a cloud platform (QoXcloud) for the measurement and evaluation of the QoS and QoE in the telecommunication services has been presented.

The proposed platform has been designed on the basis of the QoXphere framework [5], which fully accomplished the most important ITU-T related recommendations. As a consequence of this, the resulting architecture has been designed to fit within the proposal of an ITU global architecture for the performance of interoperability tests and the validation of existing or new ITU-T Recommendations.

The proposed cloud platform has also been conceived with the aim of helping to advance in the work item opened recently in ITU-T to establish an ITU recognition procedure of testing laboratories with competence in ITU-T Recommendations.

The results of the test carried out in both the developed prototypes demonstrate the validity of the platform in terms of a suitable architecture for the evaluation of QoE and the validation of the aforementioned recommendations.

In fact, the contribution to the enhancement of P.STMWeb Draft [34], that was presented following the validation of the Web Service Prototype, was well considered in the final version of the recommendation. The newly published ITU-T P.1501 Recommendation [35] defines lower condition periods and a not fixed but orientative range of bandwidth values for the subjective testing methodology in web browsing, as suggested in the contribution.

Indirectly, the presented contribution also helped in the validation of the G.QoEWeb Draft [33], which resulted with the publication of the new G.1031 Recommendation [36] and the update of the G.1030 Recommendation [26].

7 Future work

As previously mentioned, the ITU-T Study Group 11, with the support of the Conformity & Interoperability Group [37], has promoted in the last couple of months the development of a new recommendation (ITU-T Q.Int_speed_test Draft [30]), intended to describe a “unified methodology of Internet speed quality measurement usable by end-users”. The main objective of this draft is to obtain comparable results from the many Internet testing tools already developed that are accessible from a web browser or desktop application.

Despite being in an early stage, the draft shows some indicators of the suitability of the QoXcloud platform to help serve the ambitious undertaking of the recommendation.

One of those indicators is the test facilities architecture that the Q.Int_speed_test defines. This architecture shares certain similarities with the QoXcloud IT Infrastructure shown in Figure 7, so minor modifications would be needed. As for the methodology itself, the draft defines two measurement tests (an Internet access speed test and an Internet resources access speed test), to which the Internet Access Service prototype presented in this paper could be useful for, if modified adequately to fit the draft requirements.

As a consequence of this, not only studying the evolution of the Q.Int_speed_test Draft in the next couple of months is mandatory, but also a deep analysis of the methodology is required, in order to contribute to the enhancement and validation of the recommendation. Pursuing this objective is one of the main essences of the QoXcloud platform.

8 Acknowledgment

This work has been partially funded by the Spanish Ministerio de Economía y Competitividad (MINECO) under grant TEC2013-46766-R: QoEverage - “QoE-aware optimization mechanisms for next generation networks and services”.

References

[1] ITU-T, “P.10/G.100 (2006) Amendment 2 (07/08): New definitions for inclusion in Recommendation ITU-T P.10/G.100”, 2008.

[2] ITU-T, “G.1000: Communications quality of service: A framework and definitions”, 2001.

[3] ITU-T, “M.3050.1: Enhanced Telecom Operations Map (eTOM) – The business process framework”, 2007.

[4] R. Stankiewicz et al, “QoX: What is it really?”. Communications Magazine, IEEE, vol. 49, pp. 148–158, 2010.

[5] E. Ibarrola, E. Saiz, J. Xiao, L. Zabala, and L. Cristobo, “QOXPHERE: A new QoS framework for future networks,” in ITU Kaleidoscope 2013: Building Sustainable Communities (K-2013), 2013 Proceedings of, 2013, pp. 1–7.

[6] ITU-T, Study Group 13: “Future networks including cloud computing, mobile and next-generation networks”, http://www.itu.int/en/ITU-T/studygroups/2013-2016/13/Pages/default.aspx

[7] ITU-T, TD 048 Rev.1 (GEN/11): “Living list of key technologies which are under study in SG11 and are suitable for conformance and/or interoperability testing”.

[8] ITU-T, SG-11. COM 11 – C 24 – E: “Proposals for the collaborative QoE/QoS testing platform to advance in “Q.QMS” work item”, University of the Basque Country. February 2013.

[9] ITU, ITU Conformity and Interoperability (ITU C&I), from: http:// www.itu.int/en/ITU-T/C-I/Pages/default.aspx

[10] ITU-T, SG-12. COM 12 – C 337 – E: “Draft P.STMWeb”, A1 Telekom Austria AG. March 2013.

[11] ITU-T, SG-12. COM 12 – C 46 – E: “Draft Test Plan for P.STMWeb”. A1 Telekom Austria AG. March 2013.

[12] ETSI, TR 103 125 V1.1.1: “CLOUD; SLAs for Cloud services”, 2012.

[13] G. Ghinea et al, “Quality of perception to quality of service mapping using a dynamically reconfigurable communication system”, in Global Telecommunications Conference, 1999. GLOBECOM '99, Rio de Janeiro, Brazil, 1999, pp. 2061–2065.

[14] Sharkh, M.A. et al, “Resource allocation in a network-based cloud computer environment: design challenges”, Communications Magazine, IEEE, vol. 51, no. 11, pp. 46–52, November 2013.

[15] Cao et al (2009). “A Service-Oriented Qos-Assured and Multi-Agent Cloud Computing Architecture”. In M. Jaatun, G. Zhao & C. Rong (Eds.), Cloud Computing (Vol. 5931, pp. 644–649): Springer Berlin Heidelberg.

[16] Ferretti et al., “QoS-Aware Clouds”. Paper presented at Cloud Computing (CLOUD), 2010 IEEE 3rd International Conference on, Miami, USA, 5–10 July 2010.

[17] ITU-T. “G.1011: Reference guide to quality of experience assessment methodologies”, 2013.

[18] ITU-T. “Y.1542: Framework for achieving end-to-end IP performance objectives”, 2010.

[19] ITU-T. “Y.1543: Measurements in IP networks for inter-domain performance assessment”, 2007.

[20] ITU-T. “Y.1541: Network performance objectives for IP-based services”, 2006.

[21] ITU-T. “Y.1540: Internet protocol data communication service – IP packet transfer and availability performance parameters”, 2007.

[22] R. Partearroyo et al, “QoSmeter: Generic quality of service measurement infrastructure,” in IFIP Networking 2006, workshop ‘Towards the QoS Internet’ (To-QoS'2006), Coimbra, Portugal, 2006.

[23] L. Zabala et al, “LabQoS: A platform for network test environments,” in ITU-T Kaleidoscope 2011. The fully networked human? – Innovations for future networks and services, Cape Town, South Africa, 2011.

[24] ITU-T, SG-12. COM 12 – C 30 – E: “Proposal on enhancement of G.1000 framework”. March 2013.

[25] J. Xiao and R. Boutaba, “Assesing network service profitability: modeling from market science perspective”, Networking, IEEE/ACM Transactions on, vol. 15, pp. 1307–1320, 2007.

[26] ITU-T, “G.1030: Estimating end-to-end performance in IP networks for data applications”, 2005.

[27] ITU-T, “E.419: Business oriented Key Performance Indicators for management of networks and services”, 2006.

[28] TMF GB935, “Business Metrics Concepts and Principles”, Release 7.1.1.

[29] ETSI, EG 202 057-4 V1.2.1: “Speech processing, transmission and Quality aspects (STQ); User related QoS parameter definitions and measurements; Part 4”, 2008.

[30] ITU-T, SG-11. TD 631-GEN: “Draft Recommendation ITU-T Q.Int_speed_test: Unified methodology of Internet speed quality measurement usable by end-users on the fixed and mobile networks”, December 2014.

[31] OM ITC/912/2006: “Por la que se regulan las condiciones relativas a la calidad de servicio en la prestación de los servicios de comunicaciones electrónicas”, BOE 31-03-2006, Spanish Gov. 2006.

[32] ITU-T, “G.107: The E model: A computational model for use in transmission planning”, 2014.

[33] ITU-T, SG-12. COM 12 – C 34 – E: “G.QoE-Web: Relevant factors and use cases for QoE web”. A1 Telekom Austria AG. March 2013.

[34] ITU-T, SG-12. COM 12 – C 0124 – E: “Test results and new proposals for enhancement of Draft P.STMWeb”, University of the Basque Country. December 2013.

[35] ITU-T, “P.1501: Subjective testing methodology for web browsing”, 2014.

[36] ITU-T, “G.1031: QoE factors in web-browsing”, 2014.

[37] ITU, “Measurements of Internet speed”, from: http://www.itu.int/en/ ITU-T/C-I/Pages/IM/Internet-speed.aspx.

Biographies

E. Saiz received B.S. and M.S. degrees in telecommunications engineering from the University of the Basque Country in December 2009. Since January 2010 he has worked as a research fellow for the Networking Quality and Security (NQaS) Research Group of the University of the Basque Country. In this period he has cooperated in the development of a neutral infrastructure for the deployment of different QoS measurement services. He is currently working toward his Ph.D. degree. His research interests focus mainly on QoS network performance and parameters and QoS/QoE performance measurements.

Dr. E. Ibarrola received the Ph.D. degree in Telecommunications Engineering in 2010 from the University of the Basque Country for her work in the area of user-centric Quality of Service (QoS) management models. She was honored with the Best Thesis Award in Management, Economy and Telecommunications Regulation from the Telecommunication Engineering Official College in Spain. Prior to joining University of the Basque Country, in January 2000, she worked in the National Network Supervision and Operation Center of Telefónica. She has been participating in different R&D projects and cooperating with different standardization bodies. Her research interests focus mainly on users’ oriented (QoS) management models and frameworks.

L. Cristobo received B.S. and M.S. degrees in telecommunications engineering from the University of the Basque Country, Spain, in 2004. She has nine years of communications engineering experience and is currently head of the IT Department at ST3 Elkartea. Since 2007 she has been an associate professor at the University of the Basque Country, where she is currently a Ph.D. candidate. She conducts research in the areas of QoE and quality of business (QoBiz) management processes.

Dr. I. Taboada is a member of NQaS – Networking Quality and Security Research Group from University of the Basque Country (UPV/EHU). She received her degree and her Ph.D in Telecommunications Engineering from UPV/EHU, in 2008 and 2013 respectively. After being working in themes related to PQoS/QoE/QoS assessment in converged networks, for the last three years, her research interests mainly include dynamic scheduling of network resources, especially for QoE optimization in wireless networks. She is familiar with Markov Decision Processes, Gittins index approach and Marginal Productivity indices (Whittle index theory).