Re: [ippm] Adoption call for draft-olden-ippm-qoo-02

[Will Hawkins <[e-mail]>]

On Wed, Mar 13, 2024 at 11:10 AM Bjørn Ivar Teigen <[e-mail]> wrote:

> IPPM members,
>
> We have not been able to complete the WG -00 draft in time for the IETF
> 119 deadline, but many of the comments have now been resolved and pushed to
> Github.
> The up-to-date document can be found here:
> https://domoslabs.github.io/QoOID/draft-olden-ippm-qoo.html
>
> There are a few questions we would like to discuss with the group:
> * We're suggesting a new method for taking loss into account. The details
> of this will be presented at the meeting, and slides will soon be
> available. It is not yet described in the draft.
> * We've included the contents of draft-teigen-ippm-app-quality-metric-reqs
> as Motivation and Background. Do you agree with the way we've done this?
> * Do you have anything to add to the way comments have been addressed?
>
> There's a number of comments we still need to address. Please weigh in if
> you have opinions/contributions related to any of these:
> * Remove frequent use of "we", and instead use a more formal tone.
> * The throughput aspect of both the measurements and the requirements
> needs more work.
> * Clarify that latency can be measured in different ways (i.e. not
> TR-452.1 exclusive)
> * Make a section on how to create a network requirement
> * Add an example of the passive measurement method that supports the
> measurement of latency distribution
>

Not that the world needs to hear my opinion, but ...

I think that the work that you have done on this draft is excellent and I
am very pleased that the WG is moving forward with it. I have provided
feedback to you directly and I appreciate you taking it into consideration.

At several points in the discussion you have mentioned that you are taking
a pragmatic approach and want to make this proposal as widely applicable as
possible. For that reason, I think that it would be incredibly helpful to
have both a section on how to create a network requirement and an example
of how to embed a measurement system into an app/network to assess that
requirement.

In meetings where I have seen you present this work, you make great points
about how QoO meets the criteria of "Requirements for a Network Quality
Framework Useful for Applications, Users and Operators" (with the exception
of the additions you present in this document). However, as a naive user of
probability and statistics, it seems possible that an application developer
like me could make mistakes in setting the requirements and, as a result,
come up with a result that is less than "Useful". A set of "best practices"
and an example would help people who want to apply this work see any
potential pitfalls from certain choices made about how to set application
requirements.

Again, I am really excited by this work and I look forward to hearing your
update at the meeting next week!

Sincerely,
Will



> * Add additional information on whether the list of the measurement
> parameters in Section 3 is sufficient to "ensure network measurements can
> be analyzed for precision and confidence".
>
> Thanks again to the chairs and all commentators for contributing to
> improving the draft!
>
> Cheers,
> Bjørn
>
> On Wed, 14 Feb 2024 at 11:25, Bjørn Ivar Teigen <[e-mail]> wrote:
>
>> Hi Luis,
>>
>> Thanks for your comments on the draft.
>>
>> On Sun, 11 Feb 2024 at 20:03, LUIS MIGUEL CONTRERAS MURILLO <
>> [e-mail]> wrote:
>>
>>> Hi Charis, all,
>>>
>>>
>>>
>>> Apologies for answering late to this adoption call. I support the
>>> adoption of the draft since it seems to be a useful mechanism for deriving
>>> probabilistic insight on the expected performance of applications.
>>>
>>>
>>>
>>> From my review, there are some comments that I would like to be
>>> addressed by the authors:
>>>
>>>
>>>
>>>    - The draft is positioned as a kind of extension or complementary
>>>    work of BBF TR-452.1. However, the development of the proposed ideas in the
>>>    draft refer to distribution of percentile values of latency. Thus, in my
>>>    understanding, whatever technique that could generate such latency
>>>    distribution could work (e.g., any monitoring system). In other words, it
>>>    is not clear to me if the solution can be claimed as generic and
>>>    independent of BBF TR-452.1 (now, for instance, are claims such as “The
>>>    foundation of the framework is Quality Attenuation”).
>>>
>>>  Good point. I think we should clarify that latency can be measured in
>> different ways. It does not have to be exactly in accordance with TR-452.1
>> (although that's a very good way to do it!).
>>
>>>
>>>    - Not clear to me the use and need of measuring the throughput. The
>>>    examples are also unclear in that respect.
>>>
>>> Agreed. The throughput aspect of both the measurements and the
>> requirements need more work.
>>
>>>
>>>    - The lowest threshold of the quality boundaries is called Network
>>>    Requirement for Perfection (NRP). This marks the baseline on the
>>>    performance, in other words, it refers to the behavior of the application
>>>    on “ideal conditions”. That ideal conditions essentially mean the behavior
>>>    observed without any impairment from the network. That is, no latency nor
>>>    packet loss coming from the network, in a kind of performance observed in a
>>>    back-to-back running of the application between its endpoints. Once
>>>    impairments are introduced, the application performance will begin to
>>>    degrade. In summary, such “perfection” correspond in fact to “ideal
>>>    conditions”. The performance under ideal conditions can be maintained to
>>>    some extent even when some impairments (latency, packet loss) are
>>>    introduced. The NRP as described on the examples includes some latency and
>>>    some packet loss, so it would be interesting to have as reference how far
>>>    NRP is from ideal conditions. This is because NRP does not actually
>>>    represents the 100% reference, but something lower than that. This is
>>>    important to fix the expectations of the application user’s against how
>>>    robust are the applications respect to the network effects.
>>>
>>> The idea of network requirement for perfection is to specify the level
>> of network impairments the application "knows how to deal with". In other
>> words, if the network is as responsive and stable as the requirement
>> specifies, then the application will work well. If I understand correctly,
>> that is what you are saying also?
>>
>>>
>>>    - How compatible is this approach with adaptative applications? That
>>>    applications are able to react to network conditions changing the NRP and
>>>    NRPoU references dynamically.
>>>
>>>  Good question. Answering this question properly will require some more
>> work, I think, but these are my thoughts on the subject at the moment:
>> If the application adapts by lowering resolution, frame rate, or some
>> other user-affecting aspect in a gradual way, then we might use the network
>> requirement for the highest-fidelity level as the requirement for
>> perfection and the lowest-fidelity level as the "useless" end of the
>> requirement. Then the QoO score would correlate with the delivered
>> fidelity. This might not work for all application, and in some cases it
>> might be more appropriate to have one QoO score for each level of
>> application fidelity, for instance.
>>
>>
>>>    - The paragraph about Volatile Networks of section 7 seems to be a
>>>    subsection, so probably requires to be numbered as 7.1
>>>
>>> Agreed
>>
>>>
>>>    - Section 9 is probably not needed.
>>>
>>> I think this section is required since it's part of the template, but if
>> I'm wrong about that then I agree with you.
>>
>> Thanks again for the detailed feedback.
>>
>> Cheers,
>> Bjørn
>>
>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> Best regards
>>>
>>>
>>>
>>> Luis
>>>
>>>
>>>
>>> *De:* ippm <[e-mail]> *En nombre de * Tommy Pauly
>>> *Enviado el:* martes, 16 de enero de 2024 18:13
>>> *Para:* IETF IPPM WG ([e-mail]) <[e-mail]>
>>> *Asunto:* [ippm] Adoption call for draft-olden-ippm-qoo-02
>>>
>>>
>>>
>>> Hello IPPM,
>>>
>>>
>>>
>>> This email starts a working group adoption call for "Quality of Outcome”
>>> (draft-olden-ippm-qoo).
>>>
>>>
>>>
>>> https://datatracker.ietf.org/doc/draft-olden-ippm-qoo/
>>>
>>> https://www.ietf.org/archive/id/draft-olden-ippm-qoo-02.html
>>>
>>>
>>>
>>> The call will last for 3 weeks, and end on *Tuesday, February 6*.
>>> Please reply to this email with your review comments and indicate if you
>>> support adopting this work.
>>>
>>>
>>>
>>> Please note that we did a previous adoption call that did not receive
>>> sufficient feedback. At the last meeting at IETF 118, we did have a good
>>> amount of comments and questions, so please do reply to this email if you
>>> have reviewed the document.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Tommy & Marcus
>>>
>>> ------------------------------
>>>
>>> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
>>> puede contener información privilegiada o confidencial y es para uso
>>> exclusivo de la persona o entidad de destino. Si no es usted. el
>>> destinatario indicado, queda notificado de que la lectura, utilización,
>>> divulgación y/o copia sin autorización puede estar prohibida en virtud de
>>> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
>>> que nos lo comunique inmediatamente por esta misma vía y proceda a su
>>> destrucción.
>>>
>>> The information contained in this transmission is confidential and
>>> privileged information intended only for the use of the individual or
>>> entity named above. If the reader of this message is not the intended
>>> recipient, you are hereby notified that any dissemination, distribution or
>>> copying of this communication is strictly prohibited. If you have received
>>> this transmission in error, do not read it. Please immediately reply to the
>>> sender that you have received this communication in error and then delete
>>> it.
>>>
>>> Esta mensagem e seus anexos se dirigem exclusivamente ao seu
>>> destinatário, pode conter informação privilegiada ou confidencial e é para
>>> uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o
>>> destinatário indicado, fica notificado de que a leitura, utilização,
>>> divulgação e/ou cópia sem autorização pode estar proibida em virtude da
>>> legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos
>>> o comunique imediatamente por esta mesma via e proceda a sua destruição
>>> _______________________________________________
>>> ippm mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/ippm
>>>
>>
>>
>> --
>> Bjørn Ivar Teigen, Ph.D.
>> Head of Research
>> [phone] | [e-mail] | www.domos.ai
>> [image: https://www.understandinglatency.com/]
>> <https://www.understandinglatency.com/>
>>
>
>
> --
> Bjørn Ivar Teigen, Ph.D.
> Head of Research
> [phone] | [e-mail] | www.domos.ai
> [image: https://www.understandinglatency.com/]
> <https://www.understandinglatency.com/>
> _______________________________________________
> ippm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ippm
>