[ippm] Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)

[Bjorn Ivar Teigen Monclair <[e-mail]>]

Hi Jason, Michael, Ike, Guiseppe, Greg, Luis, Ruediger, IPPM chairs and participants,

Thank you very much for the detailed feedback! Greatly appreciate the time you've all invested in this.

On restructuring:
I see how the document would benefit from restructuring in the way Ike suggests. If restructuring can be done without causing a lot of delay (and work!), I am happy to submit a new version along the suggested lines. If restructuring would require significant delays and reviews (or, if others disagree about the need for restructuring), then perhaps adding a new sub-section to the introduction is the better option? Chairs and seasoned IETF'ers, I hope you can advice.

 I will take all of your feedback into account and produce a new version of the document before the deadline on the 7th.

Cheers,
Bjørn

________________________________
From: [e-mail] <[e-mail]>
Sent: Wednesday, July 2, 2025 5:20 PM
To: Magnus Olden <[e-mail]>; Bjorn Ivar Teigen Monclair <[e-mail]>
Cc: [e-mail] <[e-mail]>
Subject: AW: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


Hi Bjørn, hi Magnus,



Thanks for your framework for multi-dimensional metric linking delay and packet loss. Please find some [RG] marked comments below.



Please note, that I’m off for vacations and likely won’t respond to replies before early August. I’m sorry for that.



Regards, Ruediger







[RG] The document frequently refers to “network” as to be characterized. Yet I failed to find a definition or a reference to a definition. Please clarify, what you mean by “network”.

I note, that “network operators” have little incentive…in one statement. Does this express, that a consumer will reliably buy the best performing Home Gateway to optimize his QoE by minimizing the effect of the home network on their app?



Another general comment, without being a native speaker, maybe “comprehensible” rather than “understandable”? Could some native speaker confirm/reject that?



----



Section 2:

Existing network quality metrics and frameworks typically address the needs of one or two of these stakeholders,



[RG] Please provide references for existing quality metrics & frameworks addressing the needs of one or two of these stakeholders only.



----



Section 2:

While solutions exist for many of the problems causing high and unstable latency in the Internet, the incentives to deploy them have remained relatively weak.



[RG] Please provide references for both, “problems” and “incentives”.



----



Section 2

operators have little incentive to improve network quality beyond increasing throughput. Despite the availability of open-source solutions, vendors rarely implement them. The root cause lies in the absence of a universally accepted network quality framework that captures how well applications are likely to perform.



[RG] Do you mean throughput or bandwidth? Consumers may receive offers of access types with higher bandwidth. Above a minimum access bandwidth, further increase of bandwidth doesn’t increase per app throughput. A suitable location of content and service servers would.



[RG] I’m not sure, whether the root cause for the claimed unwillingness of vendors to implement open source solutions is a missing network quality framework. Can you provide some proof or a reference for your claim? Further, your proposal is to characterize two thresholds (perfect and stop of app) and interpolate linearly between these, based on a subset of performance metrics. You apply a set of simplification, rendering the diagnosis less reliable. The dependency on app developer publications isn’t increasing the quality of diagnosis too, unless a standardized testing environment & conditions is the basis for these publications.



Please reword:
“A universally accepted network quality framework that successfully captures how well applications are likely to perform may help to increase the willingness of vendors to….”




----



Section 2

Therefore, one critical requirement of a meaningful framework is its ability to answer the question, "Will an application work properly?"



[RG] To me a framework is a starting point and a standard an end. To say, the standard would offer the “ability”. Please reword:

… is to enable work on a standard answering the question, "Will networking conditions stop an application from working properly?"



----



Section 2

Answering this question requires several considerations. First, the Internet is inherently stochastic from the perspective of any given client, so certainty i unattainable.



[RG] Please correct/reword (would that be an “is”?)



----



2.2. Requirements

Flexibility in describing application requirements and the ability to capture the delay characteristics of the network in sufficient detail are necessary to compute application success with satisfactory accuracy and precision.



[RG] Please specify “satisfactory accuracy and precision” or provide a reference. Please reword:
necessary to improve accuracy and / or precision of…



[RG] Please define what you mean by “to compute application success” and how that is linked to the QoO network score (the framework set out to describe the latter).

----



2.2. Requirements

To summarize, the framework and "meaningful metric" should have the following properties:



Capture the information necessary to compute the probability that applications will work well. (Useful for end-users and application developers.)



[RG] I’d prefer rewording to

To summarize, the framework and "meaningful QoO metric" should have the following properties:



Capture a set of network performance metrics which provably correlate to the application quality of a set of different applications as perceived by users.


----



2.2. Requirements



Compare meaningfully to different application requirements.



[RG] Please reword to be more specific on what you mean. Mathematically, and if you’d like to compute something meaningful, QoO score and app QoE must be correlated, even if you don’t go for MOS. If this is captured by my reworded version of the first requirement, then that’s it. If the above second requirement is that the function individual_app_quality (QoE score) needs be known, then clearly say that. As far as I understand your draft, this function is out of scope.



----



2.2. Requirements



Compose.



[RG] Should that be de-compose and isolate? So far the text didn’t explain that the QoO score is calculated network-section-wise and finally composed.



----



2.2.1. Requirements for end-users



The quality framework should facilitate a metric that is objective, relatable, and relatively understandable



[RG] please reword to
…. metric that is based on objective QoS measurements, correlated to application quality  and relatively understandable



----



2.2.1. Requirements for end-users



If these requirements are met, the end-user can understand if a network can reliably deliver what they care about: the outcomes of applications.



[RG] I’d prefer a more defensive way of putting it, but the statement doesn’t say, the network is the only cause of bad app outcome.

My preference:

… can understand if a network can is a likely source of impairment for what they care about: the outcomes of applications.



----



2.2.1. Requirements for end-users

A compromise is for the quality of experience framework to place the responsibility for sourcing and representing end-user requirements onto the application developer.



[RG] If I understand this statement correctly, you put crating input for the second requirement of section 2.2 above out of scope of this framework. Please do clearly state that in the first instance where his is mentioned. Suggestion:



Compare meaningfully to different application requirements. Note that providing the relevant information depends on application developers publications or standards, where available.



[RG] Further, this statement and the remaining content define requirements from app developers. Please move to section 2.2.2



----



2.2.1. Requirements for end-users



Some real world examples where 'acceptable levels' have been derived by application developers include (note: developers of similar applications may have arrived at different figures):



Remote music collaboration: 28ms latency note-to-ear for direct monitoring, <2ms jitter



Online gaming: 6Mb/s downlink throughput and 30ms RTT to join a multiplayer game



Virtual reality: <20ms RTT from head motion to rendered update in VR



Performing this UAT helps the developer understand what likelihood a new end-user has of an acceptable Quality of Experience based on the application's existing requirements towards the network.



[RG] I’d prefer references for the performance figures.



----



2.2.2. Requirements from Application and Platform Developers



[RG] From my point of view asking for useful UAT requirement publication from App developers requires standard testing conditions (testbed, config and path, forwarding & load characteristics). This might be read into the paragraph as is, but it should be explicitely expressed, whether or not such standard conditions for app characterization are required by this section, or not. I don’t know the authors preference, so I don’t propose text.
As a thought, if app developers derive data under different networking conditions, the overall accuracy and precision of the QoO metric will be reduced.



----



3. Background



[RG] Please move the text of “3.1.8. Quality Attenuation” to “Background”, as this seems to indicate latency and loss as two metrics required. Please be also more precise, as to what Quality Attenuation captures regarding latency distribution. I’m wooried, as you address Average and 99th percentile latency by separate sections.





3. Background



The novelty of the Quality Attenuation metric is to view packet loss as infinite (or too late to be of use e.g. > 3 seconds) latency [TR-452.1].



[RG] Please add a ref to https://www.rfc-editor.org/rfc/rfc7680.html#section-2.8.2 and discuss the novelty of TR-452.1 as compared to the RFC 7680 loss specification. You may also reword to:

“Similar, as the One-Way Loss Metric for IP Performance Metrics (IPPM) [RFC 7680], the Quality Attenuation metric is to view…



----



3. Background



All applications require a minumum level of throughput and a maximum packet loss rate.



[RG] Please reword to  ..and work acceptable up to a maximum…



----



3.1. Discussion of other performance metrics



[RG] Which metrics have been discussed prior to the “other performance metrics”. If there’s no clear text to that, please



-----



3.1.3. Variance of latency



[RG] Adding the variance of random processes is producing correct results is producing correct results, if these are not correlated. Please add a statement. Please provide a reference to a metric definition for Latency Variation.



-----



5. Describing Network Requirements



Applications do of course have throughput requirements, and thus a complete framework for application-level network quality must also take capacity into account.



[RG] Please clarify, whether this statement refers to throughput or capacity. Capacity impacts throughput in the case of congestion, but they aren’t the same. Throughput may also depend on the bandwidth limitation. A policer is a worst case for TCP, a port with a WRED drop-profile might result in a different throughput, even if both offer the same nominal bandwidth.



----



5. Describing Network Requirements



[RG] Please reword: To do that it is necessary to make articulating the network requirements a little bit more complicated.



----



5. Describing Network Requirements



Where the NRPoU percentiles and NRP



[RG] Please inform the reader, where the new per App threshold values NRPoU percentiles and NRP are originating or determined.



----



8. How to find network requirements



[RG] Your discussion is comprehensible, but lacks some important aspects, which I think need to be clear to a reader: Who is expected to execute this characterization? Are results to be published? Are the networking conditions to derive these parameters to be published?



----



11.1. Missing Temporal Information in Distributions.



These two latency series: … will have identical distributions, but may have different application performance. Ignoring this information is a tradeoff between simplicity and precision.



[RG] I think (exactly) this problem is solved if QoO incorporates more and suitable QoS metrics, like IPDV.



-----



11.3. Assuming Linear Relationship between Perfect and Unusable (and that it is not really a probability)



[RG] Assuming this linearity is introducing an error. From what I know from ITU-T SG12, QoE correlations to performance measurements can be mathematically demanding. Please read into ITU-T G.1051 , Modelling approach for perceived interactivity of interactive applications to learn more about some math applied there.



I quote:
“The basic assumption of modelling perceived interactivity is its monotonous dependency on data latency. The shorter the data transport time is, the shorter the response time in an interactive application is and the more interactive the use of the application is perceived to be.

However, this dependency is not a simple linear function; rather there are saturation areas at both tails of the function, where no further change in perception happens even if the latency changes.”



-----



11.4. Binary Bandwidth threshold



[RG] Your judgement is correct. You move closer to ITU-T SG12… Codec, FPS, version of the streaming software, transport used, kind of content, error concealment and so on have an impact on QoE, not just the network or capacity.



----







Von: Marcus Ihlar <marcus.ihlar=[e-mail]>
Gesendet: Donnerstag, 12. Juni 2025 15:42
An: IETF IPPM WG ([e-mail]) <[e-mail]>
Betreff: [ippm] Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)



Hello IPPM,



This email initiates the working group last call for Quality of Outcome (draft-ietf-ippm-qoo).



The current version of the document can be found here:

https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/

https://www.ietf.org/archive/id/draft-ietf-ippm-qoo-03.html



Please review the document and reply to this email with any comments and indicate whether you believe it is ready for publication.

This WGLC will last for three weeks and end on Thursday July 3, 2025.



BR,

Marcus and Thomas



This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.