Re: [ippm] Adoption call for draft-olden-ippm-qoo-02

[Bjørn Ivar Teigen <[e-mail]>]

Hi all,

Thanks for the feedback Sukru,

Please find my comments inline.

On Mon, 22 Jan 2024 at 08:51, Mehmet Sukru Kuran <[e-mail]>
wrote:

> Hi all,
>
>  I thank all the contributors who have worked on "Quality of Outcome".
> Below are my comments,
>
>  1. First of all, I really liked the idea and I think QoO is a very neat
> metric for evaluation QoE without going into the pains of conducting a lot
> of manual testing and try to rely on MOS values. Getting the required NRP
> and NRPoU values for different applications considering different codecs,
> buffering mechanisms, etc... might be tricky for mass deployment scenarios
> but all in all I believe this is a very nice addition for measuring QoE of
> QoS sensitive traffic.
>
>  2. Assume for an application, the NRP P99 is 100 ms, and NRPoU P99 is 200
> ms; when a particular "application experience" has a P99 is 110 ms, how
> come can you say there will be 10% of lagging?  I mean, how can you show
> that lagging is a linear function?
>
>    - I'm aware that showing this is a linear function (or any other type
>    of function) is not easy at all. If there is a proof showing that such a
>    linear function holds I believe it will increase the value of the IETF
>    document to either explain it or refer to the proof in another document
>    (etc. a paper).
>    - In case there is no such proof at the moment, for the documentation
>    it might be better to say that "There can be different functions and one
>    classical implementation can be using a linear function."
>    - In case there is no such proof at the moment, conducting a lot of
>    tests with empirical data and back the claim with these empirical data or
>    trying to prove that function F is the correct one can be very interesting
>    and quite valuable.
>
> This is an excellent point and something I think merits more
investigation. The assumption that the transition from perfect to useless
is modeled well by a straight line has not been tested thoroughly.
I've seen other work (such as the ITU-T model for voice quality) that model
this transition with a more complex function. A straight line is a
significant simplification from the point of view of designing network
requirements, and simplicity has a value of it's own here. That said, model
simplicity must of course be balanced with model accuracy.


>
> 3. Is there a systematic way of converting packet loss, max. delay, max.
> jitter into P90, P99, P99,9 NRP and NRPoU values? Also, how to combine
> throughput requirements into this QoO?
>
>    - You have an example in the document but the details are not clearly
>    laid out. I believe such a conversion method, formulation would be pretty
>    useful.
>    - In the long run, if application vendors define P90, P99, P99,9 NRP
>    and NRPoU values; this will stop being a problem. However, for a
>    wide-spread use of QoO I believe such a "conversion formulations" will be
>    very, very valuable.
>
> That's a very good point. I think we should aim to include a section in
the document describing in some detail how to create network requirements.
I've made an issue on the github page, here:
https://github.com/domoslabs/QoOID/issues/7


>
>    -
>
> 4. Regarding the measurements to be used for building the CDF, I see that
> the only limitation is "having at least 10 samples".
>
>    - I think just this requirement is just is too loose. On Page 4-5 you
>    explained this not being ideal and in any report on QoO 3 variables must be
>    available. However, even then I believe this requirement can be tightened
>    after some test runs and trials.
>    - There can be several suggested measurement patterns (e.g., a) 1/sec;
>    the measurement should be E2E including encoding/decoding; b) 1/10 sec; the
>    measurement should be from the home router to the server, etc...)
>
> The idea of making the requirement as loose as possible is to keep the
barrier to adoption as low as possible. I think that is an important
principle, but I get your point that we need to maintain a certain standard
in order for the QoO results to be meaningful.


>
>    -
>
> 5. On page 7, there is a formulation as
>
>     QoO = min(ML, NRP, NRPoU) = (1-(ML-NRP)/(NRPoU-NRP))*100
>
>         - The document says mathematically these two formulations are
> equal. However, I see that in many occurrences the two formulations give
> different results.
>         - Am I missing something here? or making an error?
>
It's possible we've caused some confusion with the notation here. The line
min(ML, NRP, NRPoU) is indended to mean "find the point where the measured
latency crosses each of the NRP - NRPoU lines, and choose the "worst one"
as your QoO value". Sorry if that is a little obscurely worded - I've made
an issue to clarify the explanation in the text (
https://github.com/domoslabs/QoOID/issues/8)


>
> 6. On page 7, there is a formulation as
>
>     QoO = (1-(ML-NRP)/(NRPoU-NRP))*100
>
>         - This formulation "I thought" should give values between 100
> (perfect) and 0 (unacceptable). However, if ML < NRP the result is more
> than 100.
>         - In case I'm not making a mistake and the intention is the QoO to
> have a value of [0,100], the formulation can be changed into
>
>     MIN( 100, (1-(ML-NRP)/(NRPoU-NRP))*100 )
>

You are correct, the value should be limited to the [0, 100] range. (
https://github.com/domoslabs/QoOID/issues/9)

Thank you for the valuable feedback and insightful comments!

Best regards,
Bjørn


>
> Regards,
> ------------------------------
> *From:* ippm <[e-mail]> on behalf of Tommy Pauly <tpauly=
> [e-mail]>
> *Sent:* 16 January 2024 20:13
> *To:* IETF IPPM WG ([e-mail]) <[e-mail]>
> *Subject:* [ippm] Adoption call for draft-olden-ippm-qoo-02
>
> Hello IPPM,
>
> This email starts a working group adoption call for "Quality of Outcome”
> (draft-olden-ippm-qoo).
>
> https://datatracker.ietf.org/doc/draft-olden-ippm-qoo/
> https://www.ietf.org/archive/id/draft-olden-ippm-qoo-02.html
>
> The call will last for 3 weeks, and end on *Tuesday, February 6*. Please
> reply to this email with your review comments and indicate if you support
> adopting this work.
>
> Please note that we did a previous adoption call that did not receive
> sufficient feedback. At the last meeting at IETF 118, we did have a good
> amount of comments and questions, so please do reply to this email if you
> have reviewed the document.
>
> Thanks,
> Tommy & Marcus
>
> Information in this email including any attachments may be privileged,
> confidential and is intended exclusively for the addressee. The views
> expressed may not be official policy, but the personal views of the
> originator. If you have received it in error, please notify the sender by
> return e-mail and delete it from your system. You should not reproduce,
> distribute, store, retransmit, use or disclose its contents to anyone.
> _______________________________________________
> ippm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ippm
>


-- 
Bjørn Ivar Teigen, Ph.D.
Head of Research
[phone] | [e-mail] | www.domos.ai
[image: https://www.understandinglatency.com/]
<https://www.understandinglatency.com/>