Re: [ippm] Adoption call for draft-olden-ippm-qoo-02

[Bjørn Ivar Teigen <[e-mail]>]

Thanks Dave, appreciate the support!

QoO can be used with any network requirement, and the oft-cited 100ms is
used in a few examples simply because it's a nice round number. I can add
some examples.

Cheers,
Bjørn

On Wed, 31 Jan 2024 at 12:02, Dave Taht <[e-mail]> wrote:

> Aside from the commonly cited 100ms goal ignoring a century's worth of
> human factors research, I support adoption, if this method
> could be extended down to numbers closer to 20ms (voip), 16ms (a frame),
> or 4ms (VR).
>
> On Wed, Jan 24, 2024 at 8:13 AM Bjørn Ivar Teigen <[e-mail]> wrote:
>
>> Hi all,
>>
>> Thanks for the feedback Sukru,
>>
>> Please find my comments inline.
>>
>> On Mon, 22 Jan 2024 at 08:51, Mehmet Sukru Kuran <[e-mail]>
>> wrote:
>>
>>> Hi all,
>>>
>>>  I thank all the contributors who have worked on "Quality of Outcome".
>>> Below are my comments,
>>>
>>>  1. First of all, I really liked the idea and I think QoO is a very neat
>>> metric for evaluation QoE without going into the pains of conducting a lot
>>> of manual testing and try to rely on MOS values. Getting the required NRP
>>> and NRPoU values for different applications considering different codecs,
>>> buffering mechanisms, etc... might be tricky for mass deployment scenarios
>>> but all in all I believe this is a very nice addition for measuring QoE of
>>> QoS sensitive traffic.
>>>
>>>  2. Assume for an application, the NRP P99 is 100 ms, and NRPoU P99 is
>>> 200 ms; when a particular "application experience" has a P99 is 110 ms, how
>>> come can you say there will be 10% of lagging?  I mean, how can you show
>>> that lagging is a linear function?
>>>
>>>    - I'm aware that showing this is a linear function (or any other
>>>    type of function) is not easy at all. If there is a proof showing that such
>>>    a linear function holds I believe it will increase the value of the IETF
>>>    document to either explain it or refer to the proof in another document
>>>    (etc. a paper).
>>>    - In case there is no such proof at the moment, for the
>>>    documentation it might be better to say that "There can be different
>>>    functions and one classical implementation can be using a linear function."
>>>    - In case there is no such proof at the moment, conducting a lot of
>>>    tests with empirical data and back the claim with these empirical data or
>>>    trying to prove that function F is the correct one can be very interesting
>>>    and quite valuable.
>>>
>>> This is an excellent point and something I think merits more
>> investigation. The assumption that the transition from perfect to useless
>> is modeled well by a straight line has not been tested thoroughly.
>> I've seen other work (such as the ITU-T model for voice quality) that
>> model this transition with a more complex function. A straight line is a
>> significant simplification from the point of view of designing network
>> requirements, and simplicity has a value of it's own here. That said, model
>> simplicity must of course be balanced with model accuracy.
>>
>>
>>>
>>> 3. Is there a systematic way of converting packet loss, max. delay, max.
>>> jitter into P90, P99, P99,9 NRP and NRPoU values? Also, how to combine
>>> throughput requirements into this QoO?
>>>
>>>    - You have an example in the document but the details are not
>>>    clearly laid out. I believe such a conversion method, formulation would be
>>>    pretty useful.
>>>    - In the long run, if application vendors define P90, P99, P99,9 NRP
>>>    and NRPoU values; this will stop being a problem. However, for a
>>>    wide-spread use of QoO I believe such a "conversion formulations" will be
>>>    very, very valuable.
>>>
>>> That's a very good point. I think we should aim to include a section in
>> the document describing in some detail how to create network requirements.
>> I've made an issue on the github page, here:
>> https://github.com/domoslabs/QoOID/issues/7
>>
>>
>>>
>>>    -
>>>
>>> 4. Regarding the measurements to be used for building the CDF, I see
>>> that the only limitation is "having at least 10 samples".
>>>
>>>    - I think just this requirement is just is too loose. On Page 4-5
>>>    you explained this not being ideal and in any report on QoO 3 variables
>>>    must be available. However, even then I believe this requirement can be
>>>    tightened after some test runs and trials.
>>>    - There can be several suggested measurement patterns (e.g., a)
>>>    1/sec; the measurement should be E2E including encoding/decoding; b) 1/10
>>>    sec; the measurement should be from the home router to the server, etc...)
>>>
>>> The idea of making the requirement as loose as possible is to keep the
>> barrier to adoption as low as possible. I think that is an important
>> principle, but I get your point that we need to maintain a certain standard
>> in order for the QoO results to be meaningful.
>>
>>
>>>
>>>    -
>>>
>>> 5. On page 7, there is a formulation as
>>>
>>>     QoO = min(ML, NRP, NRPoU) = (1-(ML-NRP)/(NRPoU-NRP))*100
>>>
>>>         - The document says mathematically these two formulations are
>>> equal. However, I see that in many occurrences the two formulations give
>>> different results.
>>>         - Am I missing something here? or making an error?
>>>
>> It's possible we've caused some confusion with the notation here. The
>> line min(ML, NRP, NRPoU) is indended to mean "find the point where the
>> measured latency crosses each of the NRP - NRPoU lines, and choose the
>> "worst one" as your QoO value". Sorry if that is a little obscurely worded
>> - I've made an issue to clarify the explanation in the text (
>> https://github.com/domoslabs/QoOID/issues/8)
>>
>>
>>>
>>> 6. On page 7, there is a formulation as
>>>
>>>     QoO = (1-(ML-NRP)/(NRPoU-NRP))*100
>>>
>>>         - This formulation "I thought" should give values between 100
>>> (perfect) and 0 (unacceptable). However, if ML < NRP the result is more
>>> than 100.
>>>         - In case I'm not making a mistake and the intention is the QoO
>>> to have a value of [0,100], the formulation can be changed into
>>>
>>>     MIN( 100, (1-(ML-NRP)/(NRPoU-NRP))*100 )
>>>
>>
>> You are correct, the value should be limited to the [0, 100] range. (
>> https://github.com/domoslabs/QoOID/issues/9)
>>
>> Thank you for the valuable feedback and insightful comments!
>>
>> Best regards,
>> Bjørn
>>
>>
>>>
>>> Regards,
>>> ------------------------------
>>> *From:* ippm <[e-mail]> on behalf of Tommy Pauly <tpauly=
>>> [e-mail]>
>>> *Sent:* 16 January 2024 20:13
>>> *To:* IETF IPPM WG ([e-mail]) <[e-mail]>
>>> *Subject:* [ippm] Adoption call for draft-olden-ippm-qoo-02
>>>
>>> Hello IPPM,
>>>
>>> This email starts a working group adoption call for "Quality of Outcome”
>>> (draft-olden-ippm-qoo).
>>>
>>> https://datatracker.ietf.org/doc/draft-olden-ippm-qoo/
>>> https://www.ietf.org/archive/id/draft-olden-ippm-qoo-02.html
>>>
>>> The call will last for 3 weeks, and end on *Tuesday, February 6*.
>>> Please reply to this email with your review comments and indicate if you
>>> support adopting this work.
>>>
>>> Please note that we did a previous adoption call that did not receive
>>> sufficient feedback. At the last meeting at IETF 118, we did have a good
>>> amount of comments and questions, so please do reply to this email if you
>>> have reviewed the document.
>>>
>>> Thanks,
>>> Tommy & Marcus
>>>
>>> Information in this email including any attachments may be privileged,
>>> confidential and is intended exclusively for the addressee. The views
>>> expressed may not be official policy, but the personal views of the
>>> originator. If you have received it in error, please notify the sender by
>>> return e-mail and delete it from your system. You should not reproduce,
>>> distribute, store, retransmit, use or disclose its contents to anyone.
>>> _______________________________________________
>>> ippm mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/ippm
>>>
>>
>>
>> --
>> Bjørn Ivar Teigen, Ph.D.
>> Head of Research
>> [phone] | [e-mail] | www.domos.ai
>> [image: https://www.understandinglatency.com/]
>> <https://www.understandinglatency.com/>
>> _______________________________________________
>> ippm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ippm
>>
>
>
> --
> 40 years of net history, a couple songs:
> https://www.youtube.com/watch?v=D9RGX6QFm5E
> Dave Täht CSO, LibreQos
>


-- 
Bjørn Ivar Teigen, Ph.D.
Head of Research
[phone] | [e-mail] | www.domos.ai
[image: https://www.understandinglatency.com/]
<https://www.understandinglatency.com/>