[ippm] Re: AD Review of draft-ietf-ippm-qoo

[[e-mail]]

Hi Werner, hi Magnus and Bjørn,

Werner (and Alex), thanks for your review.

as I'm in charge of Liaison management between IPPM and ITU-T SG12 (and the other way around), some (in)formal clarifications on the process:
- There's no liaison statement in the making on either side yet
- After reading the current draft version, I thought it would be best to give SG12 experts the opportunity to comment before publication
- And do so following IETF process (individual submissions)
- An official liaison process between IETF and ITU would have delayed all that
- Med (Mohamed Boucadair) and the ITU-T Counsellor, Martin Adolph, have been informed by me prior to starting this informal exchange

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Werner Robitza <[e-mail]>
Gesendet: Donnerstag, 18. Dezember 2025 08:55
An: [e-mail]; Bjørn Teigen <[e-mail]>
Cc: [e-mail]; [e-mail]
Betreff: [ippm] Re: AD Review of draft-ietf-ippm-qoo

Dear Bjørn, Magnus, and ippm group,

We have been recently made aware of the outgoing Liaison from IETF to ask for comments on the draft of the QoO document.

We have already read Rüdiger Geib's comments
(https://mailarchive.ietf.org/arch/msg/ippm/4WFtvc4X2CRGrqlvStEnSXrsdpo/) and would like to express that we support his viewpoint.

Furthermore, we would like to provide additional comments and suggestions. Our perspective is from our background at ITU-T where we have been contributing to various work items in Study Group 12 related to the assessment of QoE of video streaming, with resulting standards including the series ITU-T Rec. P.1203 and P.1204.

Our main viewpoint is that the QoO framework provides a useful tool for network operators to gauge QoS-related aspects of various applications on the network (such as likelihood of a particular application performance) by applying heuristics on top of objective network-based measurements. It is evident from the industry reception that such a framework is considered useful.

However, one concern is that the resulting metrics could be interpreted as a "QoE metric" or "QoE score", when in fact, the actual application performance cannot be measured in-situ, and the translation from (inferred) application performance metrics to a final QoE score is not performed in a rigorous manner.
Hence, it is important to state that it is not QoE that is being measured, but a QoS indicator that may be related to, but cannot be considered the same as, the actual QoE of end-users.

As you are aware - and it is hinted at in the discussion sections in the document - a real QoE measurement requires a more direct approach of measuring different services as they are. Hence, some changes in wording may help clarify this aspect.

Below, we would like to make concrete suggestions as to how the draft could be enhanced to avoid potential confusion from the market, i.e.
implementers and users.

Note that these comments reflect our personal views and may not represent the official position of ITU-T or any other standardization organization.

If there are any questions or concerns, please feel free to let us know.

Best regards from:

Werner Robitza (AVEQ GmbH)
Prof. Alexander Raake (RWTH Aachen, Co-Rapporteur ITU-T Study Group 12 Question 14/12)

----

    This document introduces the Quality of Outcome network score.
    Quality of Outcome is a network quality score designed to be easy to

[Comment] Both "network score" and "network quality score" are used, please consider harmonizing these terms.

----

   along with a way to create a simple user-facing metric based on comparing
   application requirements

[Comment] Is that metric the same as the "QoO network score"? If so, the same term should be used all across the document.

----

   This document proposes representing network quality as a minimum required
   throughput and set of latency and loss percentiles.

[Comment] This would be, as the authors themselves note, a strong simplification. This could read, for example: "This document assumes that the quality of the network can be represented by a minimum required throughput and a set of latency and loss percentiles."

----

    This
    document defines a distance measure between perfect and unusable.

[Comment] Is that distance measure the same as the "QoO network (quality) score"? If so, please consider using the same term, or define it as a "distance measure" earlier on.

----

    With some assumptions, we can use this distance measure to calculate
    something that can be simplified into statements such as "A Video

[Comment] Please consider defining what "something that can be simplified" is.
Is that the "QoO network quality score" (or other term, see previous comments)?
It may be unclear to the reader at this stage what the output(s) of this document really are.

----

    ... results with a description of the measurement approach that allow for
    analysis of the precision.

[Comment] As a reader one might wonder: how shall the precision be measured? It would be useful if the authors gave recommendations on how the precision could be assessed, and what levels of precision are considered acceptable.

----

   For example: "Video conferencing has a 94% chance of being lag-free on this
   network."

[Comment] Please consider clarifying already at this point who or what standard defines what "works well". This is an inherently subjective definition, as it is written right now. E.g., "represent the probability an application will work well on a network, based on predetermined network metric thresholds" could clarify this aspect.

----

   *  Perfect performance (NRP): Network conditions where the app works
      optimally

[Comment] As the authors note towards the end of the document, this should mean that beyond that point, the overall quality will not be (subjectively) improved.
Please consider already defining this meaning here.

----

   *  Unusable performance (NRPoU): Network conditions where the app becomes
      unusable

[Comment] Who defines what "unusable" means in this context? In general, the term "unusableness" is not as well defined in the literature as, say, "unacceptability". There may be cases where the app is somewhat usable, but still unacceptably so. Hence, the term may be ambiguous if not more clearly defined at this stage.

----

   unified approach to measuring both latency and loss characteristics of
   network performance [TR-452.1].

[Comment] Here, "Quality" would better be specified in terms of whether it is "Quality of Service" or "Quality of Experience". QoO is a part of QoS, unless it is mapped to "acceptability", as mentioned in other comments, or "network quality attenuation", which would also be clearer.

Please also add the following definitions:

  Quality of Experience (QoE):The degree of delight or annoyance of the user of
  an application or service [ITU-T Rec. P.10/G.100]

  Quality of Service (QoS): The totality of characteristics of a
  telecommunications service that bear on its ability to satisfy stated and
  implied needs of the user of the service. [ITU-T Rec. P.10/G.100]

----

    unified approach that serves the needs of end-users, application
    developers, and network operators.

[Comment] Here it is unclear what "successful outcomes" means. Consider
rewriting: "A network quality framework and metric that estimates the probability of the application performance being between unusable and optimal, based on the underlying network conditions."

Also, we suggest removing this sentence: "QoO provides a unified approach that serves the needs of end-users, application  developers, and network operators."
- this reads more like a benefit description rather than an objective definition.

----

    QoO Score: A numerical value derived from the QoO framework that
    represents the likelihood of application success on a given network,
    typically expressed as a percentage.

[Comment] See above. "Application success" is not properly defined in the scope of this document; if it relates to the acceptability or "usableness", (NB: not a real
word!) then the word "success" should not be used any further to avoid confusion.

----

   Network Requirements for Perfection (NRP): The network performance
   characteristics at which an application achieves optimal performance and user
   experience.

[Comment] Optimal performance and user experience do not necessarily occur at the same time; both are not directly dependent. Hence this definition is slightly vague. For instance, user experience for video streaming may already peak even though the application may still be able to present higher resolutions at higher bandwidths (optimal UX, suboptimal performance), but these resolutions could not be seen by the users due to viewing distance. Further, there are technical aspects that may affect user experience which are not captured here.

----

    Network Requirements Point of Unusableness (NRPoU): The network
    performance threshold below which an application becomes unusable or
    fails to provide acceptable user experience.

[Comment] Consider that "acceptability" has been used more in the standardization and research area, and that the relationship to "unusable" is not easy  to measure. As mentioned before, an application may be usable (e.g., one could technically use Zoom for a meeting with significant lag, i.e. it is
usable) but unacceptably so. Hence, these two terms cannot be interchanged. This is not suggesting to replace the word here; it is more of a comment for reflection.

----

    In general, all stakeholders ultimately care about the success of
    applications running over the network.  Application success depends
    not just on bandwidth but also on the delay of network links and
    computational steps involved in making the application function.

See above comments on "success", please consider revising.

----

    1.  *Capture a set of network performance metrics which provably
        correlate to the application quality of a set of different
        applications as perceived by users.* (Useful for end-users and
        application developers.)

[Comment] How is "application quality" defined?

----

   The ideal framework should be objective, like QoS metrics, and
   understandable, like QoE metrics.

[Comment] There might be a misunderstanding about the "subjectiveness" of QoE metrics and whether this is an intrinsic weakness. QoE models developed e.g. by ITU-T that output metrics indeed predict subjective ratings, typically on a MOS scale, but based on objective input data. The ground truth for training these models is of course subjective, and hence, the MOS score can be considered "subjective" too, but it is valid, reliable, and deterministic.

Note that QoO as a framework, adds a layer of conversion from objectively measured parameters, via an arbitrarily chosen threshold (as explained later in the requirements section), to a QoO score, and thus is more subjective than when using standardized ITU-T algorithms that prescribe the coefficients and thresholds to be used. It may be noted that any suboptimally (and subjectively) chosen "objective" approach will per se be subjective, too.

----

    Examples are how quickly a web page
    loads, the smoothness of a video conference, or whether or not a
    video game has any lag.

[Comment] Rather: "any perceptible lag"?

----

    However it may not be feasible to capture and
    represent these tolerances _per user_ as the user group scales.  A
    compromise is for the quality of experience framework to place the

[Comment] As far as we have understood this draft, QoO is not a "quality of experience framework". QoE in the typical definition is not measured. Please consider revising, perhaps simply replacing with "the QoO framework". To help the scope of the document to improve the landscape of terms and measurement concepts, it should be kept clear across the document that QoO is a specific aggregation of QoS indicators, not a QoE measure / metric. In some places higher up in the draft, this was more clearly outlined.

----

   range of users, terminals and network conditions to determine the terminal
   and network requirements that will meet the end-user quality threshold for an
   acceptable subset of their end-users.

[Comment] Rather: "meet the acceptability thresholds for a representative subset of their end-users".

----

    Therefore, the network requirements should include a minimum
    throughput requirement.  A fully specified requirement can be thought

[Comment] Above, it says that a complete framework "must" take capacity into account, here it says it "should include" - please reconsider using "must" here.

----

    Whether the requirements are one-way or two-way must be specified.
    Where the requirement is one-way, the direction (uplink or downlink)
    must be specified.  If two-way, a decomposition into uplink and
    downlink measurements may be specified.

[Comment] Should this be "requirement" or rather "application", or maybe "intended measurement"?

----

   To do that it is necessary to make articulating the network requirements a
   little bit more complicated.  A key design goal was to

[Comment] "a little bit more complicated" seems vague in the context of such a document. Perhaps rephrase it to: "it is necessary to add more requirements".

----

    The requirements specification is extended to include the quality
    required for perfection and a quality threshold beyond which the
    application is considered unusable.

[Comment] Rather than "quality required for perfection", this could read: "the network performance required for achieving optimal application performance", and instead of "quality threshold ...": "lower network performance threshold below which the application performance is considered unusable".

----

    As an
    example: At 4 Mbps, 99% of packets need to arrive within 100ms, 99.9%
    within 200ms (implying that 0.1% packet loss is acceptable) for the
    outcome to be perfect.  Network Requirements Point of Unusability
    (NRPoU): If 99% of the packets have not arrived after 200ms, or 99.9%
    within 300ms, the outcome will be unusable.

[Comment] Here the question is how this exact threshold has been determined?

----

    Without such
    standardization, the overall accuracy and precision of QoO metrics
    may be reduced due to variations in testing approaches across
    different applications and developers.

[Comment] It is good that there is a call to use standardized testing conditions to ensure consistency and accuracy, however links to relevant standards would be very helpful.

----

    Continue adding
    network Quality Attenuation until the application fails completely.
    The corresponding network quality levels are the points of perfection
    and unusability.

[Comment] This section contradicts the previous section. The described approach is not standardized, and leads to arbitrarily chosen thresholds, because they are based on a single person's subjective expectations of an application and what they consider to be "perfect."

----

   A key advantage of a measurement that spans the range from perfect to
   unusable, rather than relying on binary (Good/Bad) or other low- resolution
   (Terrible/Bad/OK/Great/Excellent) metrics, is the

[Comment] We are not aware of any metric that is specified in such a range (Terrible-Excellent). Did you mean to refer to the ITU-T Rec. P.800 or P.910 ACR MOS scale (Bad, Poor, Fair, Good, Excellent)? If that is meant, consider that these are the five points of a continuous scale, hence, a MOS score is expressed as a continuous number between 1 and 5, so, the above argument about "low resolution" is invalid and should be removed.

There even is proof that the approach works similarly well as more fine-grained scales. Also, there does not seem to be any statistical evidence provided that the QoO approach may be advantageous with regard to resolution, also considering that no direct link to QoE is being established. See, for reference: Huynh-Thu, Q., Garcia, M. N., Speranza, F., Corriveau, P., & Raake, A. (2010). Study of rating scales for subjective quality assessment of high-definition video. IEEE Transactions on Broadcasting, 57(1), 1-14.

----

    For example, a chance of lag-free
    experience below 20% is intuitively undesirable, while a chance above
    90% is intuitively favorable-demonstrating that absolute perfection
    is not required for the QoO metric to be meaningful.

[Comment] We don't understand how a QoO score of 20% would mean that this is the "chance of the session being lag-free." - the way we understand the metric is that the session is 20% above the threshold where the application is deemed unusable due to lag, based on the measured latency distribution. Whether an individual session is lag-free or not cannot be determined. Further, the fact that the metrics refer to "intuition" rather than quantitative evidence makes the approach little quantitative, in contrast to its claims.

----

    To assist in establishing requirements, it is
    recommended that the Network Requirements for Perfection (NRP) be set
    at the point where further reductions in latency do not result in a
    perceivable improvement in end-user experience.

[Comment] The latter sentence is important and should be mentioned much earlier (e.g. in the definition of NRP). Same for the first sentence in that paragraph, which also should be stated earlier, as it counters some of the criticism mentioned in previous parts of this document.

----

    For adaptive applications, there are typically different levels of
    "perfection" rather than a single absolute threshold.  A video
    streaming application might provide:

    *  Excellent quality: 4K resolution, low latency

    *  Good quality: 1080p resolution, moderate latency

    *  Acceptable quality: 720p resolution, higher latency

    *  Minimum quality: 480p resolution, high latency

[Comment] It is not clear how latencies come into play here. Why would a larger resolution be related to low latency? Note that normally, lower streaming latencies are achieved by transmitting lower resolutions (because they are faster to encode).

Also, the labels "Excellent" and "Good" stem from the ACR scale, whereas "Acceptable" is on a different dimension. It may be possible to rewrite this without the subjective labels, by simply removing them, and instead writing about the "available video representations" (an industry standard term).

Moreover, it should be clarified more specifically that the indicated values are examples and to not represent proven parameter-pairs. It can be assumed that this is how it was meant, but it could well be read as if these are considered proven parameter values for that application.

----

    Another, less complex approach
    at the cost of reduced fidelity in the QoO score, is to set the
    threshold for perfection at where Excellent quality can be delivered,
    and the threshold for unusability where not even minimum quality can
    be delivered.  This approach assumes that the app can perform

[Comment] It would be more meaningful to set these thresholds to whichever representation is the highest and lowest offered by the VoD provider, respectively. Also, even with bad network conditions, a minimum quality stream can be delivered. The question is "how well" can it be delivered, for instance with the absence of stalling. E.g. it is suggested to reword this: "... is to set the threshold for perfection at the highest rendition available for the video stream, and the threshold for unusability where the lowest rendition cannot be delivered without resulting in stalling events"

   adaptations without frustrating the users, for instance that quality level
   can be lowered without significant video or audio interruptions.

[Comment] This is not a property of the application; sometimes there is no way to avoid frustrating the users. At this point it is the network's fault.

----

    These results provide supporting evidence for QoO's value as a user-
    focused tool, bridging technical metrics with real-world application
    performance to enhance end-user satisfaction.

[Comment] While the study presents evidence for the usefulness of the framework, the framework has not been compared to other QoS-type approaches or QoE ratings provided from applications themselves. Also, while agreement was found, the study appears to have been designed in such a way that biases may have been introduced. Participants may have been biased regarding demand characteristics and may have tried to correspond to a "good-participant role", as the specific QoO metric and the intent to standardize it are specifically introduced in the test. Hence subjects may have thought that a good rating of the approach is a wanted outcome of the study.

----

    Taking
    these scenarios into account would add another magnitude of
    complexity to determining network requirements and finding a distance
    measure (between requirement and actual measured capability).

[Comment] Beyond the above comment - and in our opinion more important than thresholds - it must be considered that user-experienced quality for video streaming, assuming a linear change in network factors such as bandwidth, is not linearly increasing (see e.g., the sigmoid-like function in Robitza, W., Kittur, D. G., Dethof, A. M., Göring, S., Feiten, B., & Raake, A. (2018). Measuring YouTube QoE with ITU-T P.1203 under Constrained Bandwidth Conditions. In 10th International Conference on Quality of Multimedia Experience (QoMEX 2018). Santa Margherita di Pula.). There may be saturation effects where between the point of "unusableness" and being able to play the highest representation without interruption, improvements on the network may have diminishing returns, or compound at the lower end of the scale.

_______________________________________________
ippm mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]