Evaluation and Improvement of QoE and QoS Parameters in Commercial 5G Networks: 5G-TOURS Approach

Cellular communication systems have become an urgent part not only of our everyday life, but also play a big role in the process of building a fundamentally new intelligent concept of the digital world. We are currently in the development phase of 5G cellular networks on the way to their transformation to 6G. 5G has become a truly revolutionary technology, able to use all available advantages, technologies and create the digital world of the future. Therefore, 5GPPP within the framework of the Horizon 2020 program was granted funding for a large number of research projects, the purpose of which was to test the proposed innovative solutions, to combine efforts with cellular operators, vendors, etc., in order to test the real capabilities of 5G and demonstrate all the advantages to vertical industries. One of these flagship projects was 5G-TOURS. The 5G-TOURS project was aimed at improving mobility systems in the city, the e-Health industry and the tourism sector. Thus, within the framework of 5G-TOURS project was developed the special methodology for assessment of QoE and QoS and mapping between these parameters. To apply developed evaluation methodology for all the use cases of the project is very difficult, as it needs a lot of additional efforts by the partners, additional testing tools, testing procedures, etc. That is why it was decided to apply the developed approach only to one use cases in which it was possible to collect all the required data. Thus, UC 4 “High quality video service distribution” was under study to ensure the applicability of the developed approach. All the experiments were conducted in Turin, wherein a movable car, specially equipped, collected QoE and QoS data for further analysis. Thus, the methodological approach was confirmed by the results of the conducted experiments. And in the future, similar models can be developed to study the correlations of service quality parameters and user experience for any developed use cases


I. INTRODUCTION
ELLULAR communication systems have become an urgent part not only of our everyday life, but also play a big role in the process of building a fundamentally new intelligent concept of the digital world.It is impossible to even imagine what our life would be like, the process of universal digitalization was going on, if modern wireless communication systems had not been developed.These systems currently combine a huge number of connected devices and have truly become the connecting link between smart devices, automation systems, robots, and various applications.In Fig. 1 shows the approved plan for the development and research of cellular communication networks.
We are currently in the development phase of 5G cellular networks on the way to their transformation to 6G. 5G has become a truly revolutionary technology, able to use all available advantages, technologies and create the digital world of the future.Today, the 5G network supports Enhanced Mobile Broadband (eMBB) to provide maximum data rates of up to 10 Gbps.In addition, ultra-reliable low-latency link (uRLLC) minimizes delays down to 1ms, and massive machine-type link (mMTC) supports more than 100 times more devices per unit area compared to 4G.The expected reliability and availability of the network is more than 99.999% [2].Network software is an integral component of 5G technology, which provides dynamism, programmability and abstraction of networks [3].The capabilities of 5G have opened up the possibility of fundamentally new use cases, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), autonomous vehicles, the Internet of Things (IoT), and Industry 4.0 [4,5].
Figure 1.Possible ITU and 3GPP timelines [1] These concepts are evolving to become full-fledged technologies that can be used for future generations of communication networks.On the other hand, applications such as holographic telepresence (HT), unmanned aerial vehicles (UAVs), augmented reality (XR), smart grid 2.0, industry 5.0, space, and deep-sea tourism are expected to become major applications of future communication.However, the requirements for these use cases, such as ultra-high data rates, real-time access to powerful computing resources, extremely low latency, accurate localization and detection, as well as extremely high reliability and availability, exceed the network capabilities promised by 5G so far [8,9].
In 2010, the world community carried out a huge amount of work aimed at developing new technological solutions in the field of communication.They were indeed able to close some unresolved issues, but for further development it is necessary, in addition to the developers themselves, to actively involve vertical industries that will take advantage of 5G and 6G in the future.Therefore, 5GPPP within the framework of the Horizon 2020 program was granted funding for a large number of research projects, the purpose of which was to test the proposed innovative solutions, to combine efforts with cellular operators, vendors, etc., in order to test the real capabilities of 5G and demonstrate all the advantages to vertical industries [10].One of these flagship projects was 5G-TOURS [11].Within the framework of this project, it was decided to develop a certain framework for KPIs, and their combination in order to evaluate the QoE characteristic, which is more in demand from the point of view of vertical industries and end users.

II. RELATED WORKS
A large number research papers [12 -14] and 5GPPP projects were devoted to researching the possibilities of using 5th generation cellular networks for the needs of vertical industries [10].They were mostly focused on advanced 5G validation trials across multiple vertical industries.For example, 5G Solutions for European Citizens [15] aimed to prove and validate that 5G provides prominent industry verticals with ubiquitous access to a wide range of forward-looking services with orders of magnitude of improvement over 4G, thus bringing the 5G vision closer to realisation.5G!Drones [16] aimed to trial several UAV use-cases covering eMBB, URLLC, and mMTC 5G services, and to validate 5G KPIs for supporting such challenging use-cases.5Growth [17] aimed to empower verticals industries such as Industry 4.0, Transportation, and Energy with an AI-driven Automated and Sharable 5G End-to-End Solution.A common feature of these and other projects of the third phase of the 5GPPP was the need to conduct experimental studies, measure and validate the effectiveness of 5G cellular networks.Each of the projects developed its own approach to solving these problems.5G requirements and key performance indicators were considered and analyzed in [18,19].These papers discuss basic requirements that are new for 5G and provides 5G performance requirements and contribute to the understanding of vertical services' needs, by offering a thorough and concise vertical requirements analysis methodology.In [20] tools for measuring the most important KPIs, which are most suitable for the use cases of the 5GENESIS project, were proposed.The same has been done in other research projects, such as 5G-EVE [21] or 5G-HEART [22].[23] gives a review of various approaches in the measurement process of QoS and QoE that grouped into objects, methods, and data acquisition.In addition, this paper provides suggestions on how to prioritize a measurement process related to the location and conditions where 5G technology will be implemented.The identification of servicerelevant QoE metrics and modeling of how these are affected by the different 5G QoS metrics is discussed in [24].The assessment model of Quality of Experience (QoE) for 5G mobile technology was proposed in [25].[26] was devoted to the feasibility of Machine Learning (ML) techniques for QoE estimation and evaluates their performance for a mobile video streaming use-case.A special developed simulator has been implemented and described in the paper.The other papers considered similar approaches for different applications [27].VOLUME 22(4), 2023 But the vast majority of the analyzed literary sources and projects did not provide a single unified methodology for evaluating subjective QoE level based on measured objective QoS parameters.Since the networks have already become customer-oriented, there is a need for constant monitoring of QoE, and in the case of applying the methodology, there is no need to conduct user surveys, and the measurement of user experience indicators will only be based on the assessment of QoS indicators.This was the goal pursued by the 5G-TOURS project.To achieve this goal, it was necessary to develop a new model that would take into account the relationship between QoE and QoS parameters for completely different groups of use cases.After that, it was necessary to select the most suitable regression models that would allow us to establish correlations between the measured values.Based on these models, it was necessary to develop a generalized methodology and apply it in practice.This has been implemented for some of the use cases considered in the 5G-TOURS project.

A. 5G-TOURS APPROACH
The 5G-TOURS project was aimed at improving mobility systems in the city, the e-Health industry and the tourism sector.Thus, 5G-TOURS aimed to create a pan-European ecosystem that would connect these sectors, giving them access to advanced technological solutions in the field of 5G cellular systems.For this, the project used the experience of previous successful 5GPPP projects: 5G-Xcast, 5G-Monarch, etc.We also developed our own innovative solutions that were applied to 13 highly complex use cases in three European cities, each of which combined a group of these use cases: Turin -a tourist city; Rennes is a safe city; Athens is a mobile efficient city.The 5G-TOURS approach outlined above is reflected in Fig. 2.
Figure 2. 5G-TOURS methodology [28] Work on infrastructure development under the project was focused on the interaction of commercial cellular operators with the platform developed under the 5G-EVE project [29].This combined and improved architecture became the basis for conducting 5G network user satisfaction studies for the groups of use cases outlined above.For each UC, specific KPIs were determined, and indicators of overall user satisfaction were formed due to the QoE assessment.In the future, we will consider the QoE assessment methodology that was developed within the framework of the project.

B. QoE AND QoS ASSESSING METHODOLOGY
In recent years, the technical community has shifted some of its focus from one related metric, quality of service (QoS), to a more consumer-focused metric, quality of experience (QoE).While QoS stands between the network and the application, QoE focuses on the subscriber.In particular, QoE focuses on the human as the user interacting with the application and the human as the customer dealing with the service provider.The difference between QoE and QoS is highlighted below [30]: QoS -quality of service:  characteristics/behavior of the network;  performance guarantees provided by the network provider based on measurements; QoE -quality of experience:  the impact of network behavior on the end user;  some flaws may remain unnoticed;  some flaws can make the application useless;  is not fixed by network measurements. QoE takes into account user expectations, QoS is more rational based on technical measurements (Fig. 3).On the basis of the above, a model for the analysis of the overall QoE evaluation using QoS indicators, which can be evaluated in a more objective way, is proposed.
of elements of the quality assurance system.Subsets of QoE metrics ij i S S  can be represented as:  At the second stage, QoS and QoE indicators are selected using multivariate correlation-regression analysis.To build a multivariate regression model, the following steps must be performed: 1.All possible QoS factors affecting the studied QoE indicator (or process) are selected.For each factor, its numerical characteristics are determined.
2. The form of the regression or multivariate model is selected, i.e. finding the analytical expression that most fully reflects the relationship of the factor characteristics with the uniform one, i.e. the appropriate functions are selected (Table 1).
where: At the next stage, it is necessary to calculate subsets of QoS indicators using appropriate algorithms and formulas for their calculation [31].QoE must be calculated using, for example, MOS, DSCQR, ACR [32] or other appropriate methods/techniques.
At the last stage, the obtained values are compared with the maximum permissible, possible to ensure the normal functioning of the network and the achieved KPIs.
To compare the values obtained as a result of the calculations with the maximum permissible values, a logical equivalence function is introduced: 1, , ( , ) 0, .
QoE is perhaps the most important parameter that can be used to determine user experience and compare it with the expectations of other users.
That is why a QoE evaluation method was developed to assess the level of satisfaction of end users and vertical players with deployed use cases.This includes user QoE as well as feedback from vertical players on how the technology provided can improve their business operations [31].
In addition to verifying QoS results, which mainly illustrate the performance of key network performance indicators, it is extremely important to confirm the actual satisfaction of end users and vertical players.(either as service providers or users of secondary services).
In this direction, a QoE assessment method was developed, the high-level architecture of which is presented in Fig. 4 [31].The method is implemented in two stages.The first stage is implemented by testing both the QoS metrics automatically collected from the infrastructure and the QoE metrics (and vertical satisfaction) collected using the respective questionnaires.The second stage is implemented after testing and using correlation-regression analysis, from which a model for QoS-QoE correlation is created.
Each approach used in previous studies to measure QoS and QoE has its own potential.This potential is expected to a balance between QoS and QoE, a simple, fast and accurate process, and flexibility between ideal conditions and reality.
Measuring traffic and user feedback in a group of objectbased approaches has potential due to the balance of QoS and QoE, simplicity and flexibility, as in the sub-material that becomes the measurement target.This is because the traffic has different sub-materials and sub-materials in user responses that are very flexible.User traffic and response also meet the value requirements of high-tech and systematic QoS and QoE, which are highly attached to users.
The step of choosing a regression model should be highlighted separately.This choice can be quite difficult.Trying to model it with just a sample won't make this choice any easier.
Therefore, there is a need to mathematically describe the relationship between some predictors and the response variable.In practice, usually many variables are measured, but only some of them are included in the model.Then, they try to exclude variables that are not related to each other, and include only those that have a real relationship.In this case, a number of possible models are considered.At the same time, they strive to achieve a balance regarding the number of predictors they include:  Not enough: an underspecified model tends to give biased estimates. Too much: An over inflated model usually produces less accurate estimates. Optimal: A model with the correct conditions has no bias and produces the most accurate estimates.To select the most appropriate regression model, it is best to include the variables specifically being tested along with other variables that affect the response to avoid biased results.There are statistical measures and procedures that help specify the regression model [32].
Adjusted R-squared and Predicted R-squared: Models with higher adjusted and predicted R-squared values.These statistics are designed to avoid a key problem with the regular R-squared -it increases every time when is added a predictor and can trick into specifying an overly complex model:  Adjusted R-squared only increases if the new term improves the model more than would be expected by chance, and it can also decrease with poor quality predictors. The predicted R-squared is a form of cross-validation, and it can also decrease.Cross-validation determines how well model generalizes to other data sets by partitioning of data.P-values for predictors: In regression, low p-values indicate terms that are statistically significant."Model reduction" refers to the practice of including all candidate predictors in the model and then systematically removing the term with the highest p value one by one until only significant predictors remain.
Stepwise Regression and Best Subset Regression: These are two automated procedures that can identify useful predictors VOLUME 22(4), 2023 during the model building stages.In Fig. 5 below shows the scheme of method selection and subsequent model creation.
Figure 5. Regression method selection and subsequent QoE-to-QoS model generation [33] Feature selection already occurs before model construction and determines the input attributes of the later regression model.Datasets have already been structured during creation to contain only the relevant attributes.
Regression methods are good for different problems.For evaluation, the data set is split into a training and test data set before building the model.This step is automatically performed iteratively during cross-validation.

Hyperparameter optimization by iterative cross-validation with different hyperparameter settings
The result of the cross-validation is a list with the values of the selected scoring parameters.Since the evaluation is performed after each run, if the data set is divided into five partitions, there is also a list of five evaluation values.Averaging these values makes it possible to estimate the efficiency of the regression procedure.Since most regression methods allow one or more hyperparameters to adjust the complexity of the model, hyperparameter adjustment is necessary for a meaningful comparison of regression methods.Finding these optimal hyperparameter settings is done by iteratively building the model.Cross-validation is performed repeatedly for different hyperparameter settings.Finally, the parameter settings that showed the best model accuracy during evaluation are selected.This process is performed using loops that automatically change the hyperparameters within certain limits and store the estimated values.The selection of optimal settings is then performed by manually or automatically searching for the best evaluation results.

IV. 5G-TOURS TRIALS RESULTS EVALUATION
To apply developed QoE evaluation methodology for all the use cases of the project is very difficult, as it needs a lot of additional efforts by the partners, additional testing tools, testing procedures, etc.That is why it was decided to apply the developed approach only to one use cases in which it was possible to collect all the required data.Thus, UC 4 "High quality video service distribution" was under study to ensure the applicability of the developed approach.All the experiments were conducted in Turin, wherein a movable car, specially equipped, collected QoE and QoS data for further analysis.The routes of the vehicle are shown in Fig. 6.There were 18 different locations for the data collection.For each location were collected the next values: Video OK level; Location, Frequency band; Speed of the vehicle; Date; Time; Signal Power.The Video OK level was considered as a binary QoE parameter and is the result of the observations of just one person:    As it is obvious from the Table 2 and Fig. 10 accordingly the most powerful dependence is for the location Zara Baldissera.That is why, was decided to apply the data analysis especially for this location, parameters for which is presented below.

V. CONCLUSION
This paper showed that cellular communication systems have long been an urgent part of the present.At the same time, their use is currently provided for advanced vertical industries.Unfortunately, not all advantages are fully used yet, and therefore it is necessary to spread the positive experience of using cellular networks for the most complex use cases.It was for this purpose that the 5G-TOURS project was organized.
Its purpose was to test the use of 5G for vertical industries such as smart cities, the healthcare system, and the tourism sector.Since the networks have already become customeroriented, there is a need for constant monitoring of QoE, and in the case of applying the 5G-TOURS methodology, there is no need to conduct user surveys, and the measurement of user experience indicators are only based on the assessment of QoS indicators.Therefore, in order to fully validate the main indicators of user experience, the project developed a methodology for evaluation of QoE and QoS indicators.This methodology has been applied to many use cases and fully tested.The results of using this methodology were reflected in the use case related to the generation of multimedia content in a tourist city.
S p r  QoE indicators characterizing -QoE for ij S ; ij r -the number of such indicators.


variable; Y -dependent variable; b -slope of the straight line; c -interception;  -model error.-Y-intercept (the value of y when all other parameters are set to 0); p  -regression coefficient of the independent variable; p x -number of independent variables;  -model error.VOLUME 22(4


i y -predicted value; j  -independent regression coefficient of variable x;  -denotes the amount of shrinkage.-is the Y-intersection point; p  -regression coefficient of the independent variable x; P -degree of variable x.

Figure 10 .
Figure 10.Graphical representation of correlation coefficients for different locations During the data analysis the next KPIs were considered: Power [dBm]; Approx.speed [Km/h].Graphical representation of conducted measurements of Signal Power [dBm], Video Ok level [0;1], Speed [Km/h] is shown in Fig. 11.The correlation coefficients were estimated for each the considered parameter:  -For Signal Power [dBm]: Correl[FS] = 0,65 (0,84 -for first 250 values in the structured dataset)  -For Approx.speed [Km/h]: Correl[S] = -0,19 According to the received values was made the conclusion that the strength of the relationship by the correlation coefficient is quite high.Fig. 11 proves this.There is no any visible correlation between studied variables.Thus, for this case, data analysis allowed to determine that there is a dependency between QoE (Video Ok level) and Signal power [dBm], but not with Speed [Km/h].For this case can be used a threshold of Signal Power [dBm] -Psignal_tresh= -73 dBm.That is why QoE function can be represented as a binary function in the next way:In Gaussian channel for MCS 12 we can expect a threshold of about -84 dBm.On mobile channel there is a loss of about 10-12 dB.Hence, in the condition we carried out the test an estimated threshold of -73 dBm sounds correct.

Table 1 .
Set of studied models[30] Name of the model / Graphical model Mathematical model (expression) Linear regression

Table 2 .
Correlation coefficients for different locations