Re: [ippm] Adoption call for draft-olden-ippm-qoo-02

[Bjørn Ivar Teigen <[e-mail]>]

IPPM members,

We have not been able to complete the WG -00 draft in time for the IETF 119
deadline, but many of the comments have now been resolved and pushed to
Github.
The up-to-date document can be found here:
https://domoslabs.github.io/QoOID/draft-olden-ippm-qoo.html

There are a few questions we would like to discuss with the group:
* We're suggesting a new method for taking loss into account. The details
of this will be presented at the meeting, and slides will soon be
available. It is not yet described in the draft.
* We've included the contents of draft-teigen-ippm-app-quality-metric-reqs
as Motivation and Background. Do you agree with the way we've done this?
* Do you have anything to add to the way comments have been addressed?

There's a number of comments we still need to address. Please weigh in if
you have opinions/contributions related to any of these:
* Remove frequent use of "we", and instead use a more formal tone.
* The throughput aspect of both the measurements and the requirements needs
more work.
* Clarify that latency can be measured in different ways (i.e. not TR-452.1
exclusive)
* Make a section on how to create a network requirement
* Add an example of the passive measurement method that supports the
measurement of latency distribution
* Add additional information on whether the list of the measurement
parameters in Section 3 is sufficient to "ensure network measurements can
be analyzed for precision and confidence".

Thanks again to the chairs and all commentators for contributing to
improving the draft!

Cheers,
Bjørn

On Wed, 14 Feb 2024 at 11:25, Bjørn Ivar Teigen <[e-mail]> wrote:

> Hi Luis,
>
> Thanks for your comments on the draft.
>
> On Sun, 11 Feb 2024 at 20:03, LUIS MIGUEL CONTRERAS MURILLO <
> [e-mail]> wrote:
>
>> Hi Charis, all,
>>
>>
>>
>> Apologies for answering late to this adoption call. I support the
>> adoption of the draft since it seems to be a useful mechanism for deriving
>> probabilistic insight on the expected performance of applications.
>>
>>
>>
>> From my review, there are some comments that I would like to be addressed
>> by the authors:
>>
>>
>>
>>    - The draft is positioned as a kind of extension or complementary
>>    work of BBF TR-452.1. However, the development of the proposed ideas in the
>>    draft refer to distribution of percentile values of latency. Thus, in my
>>    understanding, whatever technique that could generate such latency
>>    distribution could work (e.g., any monitoring system). In other words, it
>>    is not clear to me if the solution can be claimed as generic and
>>    independent of BBF TR-452.1 (now, for instance, are claims such as “The
>>    foundation of the framework is Quality Attenuation”).
>>
>>  Good point. I think we should clarify that latency can be measured in
> different ways. It does not have to be exactly in accordance with TR-452.1
> (although that's a very good way to do it!).
>
>>
>>    - Not clear to me the use and need of measuring the throughput. The
>>    examples are also unclear in that respect.
>>
>> Agreed. The throughput aspect of both the measurements and the
> requirements need more work.
>
>>
>>    - The lowest threshold of the quality boundaries is called Network
>>    Requirement for Perfection (NRP). This marks the baseline on the
>>    performance, in other words, it refers to the behavior of the application
>>    on “ideal conditions”. That ideal conditions essentially mean the behavior
>>    observed without any impairment from the network. That is, no latency nor
>>    packet loss coming from the network, in a kind of performance observed in a
>>    back-to-back running of the application between its endpoints. Once
>>    impairments are introduced, the application performance will begin to
>>    degrade. In summary, such “perfection” correspond in fact to “ideal
>>    conditions”. The performance under ideal conditions can be maintained to
>>    some extent even when some impairments (latency, packet loss) are
>>    introduced. The NRP as described on the examples includes some latency and
>>    some packet loss, so it would be interesting to have as reference how far
>>    NRP is from ideal conditions. This is because NRP does not actually
>>    represents the 100% reference, but something lower than that. This is
>>    important to fix the expectations of the application user’s against how
>>    robust are the applications respect to the network effects.
>>
>> The idea of network requirement for perfection is to specify the level of
> network impairments the application "knows how to deal with". In other
> words, if the network is as responsive and stable as the requirement
> specifies, then the application will work well. If I understand correctly,
> that is what you are saying also?
>
>>
>>    - How compatible is this approach with adaptative applications? That
>>    applications are able to react to network conditions changing the NRP and
>>    NRPoU references dynamically.
>>
>>  Good question. Answering this question properly will require some more
> work, I think, but these are my thoughts on the subject at the moment:
> If the application adapts by lowering resolution, frame rate, or some
> other user-affecting aspect in a gradual way, then we might use the network
> requirement for the highest-fidelity level as the requirement for
> perfection and the lowest-fidelity level as the "useless" end of the
> requirement. Then the QoO score would correlate with the delivered
> fidelity. This might not work for all application, and in some cases it
> might be more appropriate to have one QoO score for each level of
> application fidelity, for instance.
>
>
>>    - The paragraph about Volatile Networks of section 7 seems to be a
>>    subsection, so probably requires to be numbered as 7.1
>>
>> Agreed
>
>>
>>    - Section 9 is probably not needed.
>>
>> I think this section is required since it's part of the template, but if
> I'm wrong about that then I agree with you.
>
> Thanks again for the detailed feedback.
>
> Cheers,
> Bjørn
>
>
>>
>>
>> Thanks,
>>
>>
>>
>> Best regards
>>
>>
>>
>> Luis
>>
>>
>>
>> *De:* ippm <[e-mail]> *En nombre de * Tommy Pauly
>> *Enviado el:* martes, 16 de enero de 2024 18:13
>> *Para:* IETF IPPM WG ([e-mail]) <[e-mail]>
>> *Asunto:* [ippm] Adoption call for draft-olden-ippm-qoo-02
>>
>>
>>
>> Hello IPPM,
>>
>>
>>
>> This email starts a working group adoption call for "Quality of Outcome”
>> (draft-olden-ippm-qoo).
>>
>>
>>
>> https://datatracker.ietf.org/doc/draft-olden-ippm-qoo/
>>
>> https://www.ietf.org/archive/id/draft-olden-ippm-qoo-02.html
>>
>>
>>
>> The call will last for 3 weeks, and end on *Tuesday, February 6*. Please
>> reply to this email with your review comments and indicate if you support
>> adopting this work.
>>
>>
>>
>> Please note that we did a previous adoption call that did not receive
>> sufficient feedback. At the last meeting at IETF 118, we did have a good
>> amount of comments and questions, so please do reply to this email if you
>> have reviewed the document.
>>
>>
>>
>> Thanks,
>>
>> Tommy & Marcus
>>
>> ------------------------------
>>
>> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
>> puede contener información privilegiada o confidencial y es para uso
>> exclusivo de la persona o entidad de destino. Si no es usted. el
>> destinatario indicado, queda notificado de que la lectura, utilización,
>> divulgación y/o copia sin autorización puede estar prohibida en virtud de
>> la legislación vigente. Si ha recibido este mensaje por error, le rogamos
>> que nos lo comunique inmediatamente por esta misma vía y proceda a su
>> destrucción.
>>
>> The information contained in this transmission is confidential and
>> privileged information intended only for the use of the individual or
>> entity named above. If the reader of this message is not the intended
>> recipient, you are hereby notified that any dissemination, distribution or
>> copying of this communication is strictly prohibited. If you have received
>> this transmission in error, do not read it. Please immediately reply to the
>> sender that you have received this communication in error and then delete
>> it.
>>
>> Esta mensagem e seus anexos se dirigem exclusivamente ao seu
>> destinatário, pode conter informação privilegiada ou confidencial e é para
>> uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o
>> destinatário indicado, fica notificado de que a leitura, utilização,
>> divulgação e/ou cópia sem autorização pode estar proibida em virtude da
>> legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos
>> o comunique imediatamente por esta mesma via e proceda a sua destruição
>> _______________________________________________
>> ippm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ippm
>>
>
>
> --
> Bjørn Ivar Teigen, Ph.D.
> Head of Research
> [phone] | [e-mail] | www.domos.ai
> [image: https://www.understandinglatency.com/]
> <https://www.understandinglatency.com/>
>


-- 
Bjørn Ivar Teigen, Ph.D.
Head of Research
[phone] | [e-mail] | www.domos.ai
[image: https://www.understandinglatency.com/]
<https://www.understandinglatency.com/>