From bjorn@domos.no Wed Feb 14 02:25:58 2024 Return-Path: X-Original-To: ippm@ietfa.amsl.com Delivered-To: ippm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C42DFC14F6A9 for ; Wed, 14 Feb 2024 02:25:58 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.895 X-Spam-Level: X-Spam-Status: No, score=-6.895 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=domos-no.20230601.gappssmtp.com Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bumQmBqZMlo9 for ; Wed, 14 Feb 2024 02:25:54 -0800 (PST) Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 85D75C14F61F for ; Wed, 14 Feb 2024 02:25:54 -0800 (PST) Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-511831801f3so4208544e87.2 for ; Wed, 14 Feb 2024 02:25:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=domos-no.20230601.gappssmtp.com; s=20230601; t=1707906352; x=1708511152; darn=ietf.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=PFVQScUSJ6Qn0sdWLa70UUKIl+4q0fllutuw9Xa3+MA=; b=kledNR+UeoJa0rgCiTCdtzlNYQYQa44oPCdYA31G5YsE8ctWhI84l9ZCqf//i9sGXb 1BJviFBvXOVj1V0KdOe+kN+uvZ22aXWCjGn4Wy8jHXElI1n3fjfyM8eu5gzKuAbimjTz 5wm9kZMq7FVbaG/riifa/T6NcQUVaqZV+PK0ZXuDDWLlQvfUq/noBeCaBjoDomuLlriH zLq21n4aWojJL8YatDs0D5p2rLZ46p9OVnrb9bWFu5OIxc2u6qBnPuJTDvZ4izdAk9H/ 6il7aL31B4pdK3f6smtyVBJOG6b+rnWCj/0h5LLpojzaHRDysPy3acCtWdnsku1uDZ6j dImw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707906352; x=1708511152; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PFVQScUSJ6Qn0sdWLa70UUKIl+4q0fllutuw9Xa3+MA=; b=e52XOiqY0e5X/fpm9CJygM3cOvKbwD6pmp1Hfn5qAa3jVx6exmgva4y8WCmkmb4P6O +6BIPYUKcDWNxrqc0AZlC8rP+I4rsnXPOo2QyvXNjF2IgySZ6BzI+Pgu/3Ov4XjvHsfA k0V4oZH7VylrpZ/uG55x3sg68IX6Ux0gLwyvHubhvmmOZV04Eyw+R11/oHrOSHnaaURJ cJ55AsPE3eu8fpZA0tf/eNVgBRd1aOjRrebKVQSYIrZqaDZhcc+dXHTGdv1FJ+wAfAO4 mdJoXm+BQ04ceaTp9jEIJJNO6S50eYCGd5Zo3P9OCcktOtCmnfsztoLt7csVAXYtKjgn aDFw== X-Forwarded-Encrypted: i=1; AJvYcCXkyqSnXjMcl0w0OTe3mxVGTkYwNdQ1Q/mNcM1bAZ8q28ZEncUKBw8dWnkSdeb9efJac2zM/dn1wWNigPoI X-Gm-Message-State: AOJu0YzYT8CZHu8J4lipjzcgq12sWmTfhgKFQlBD0j7RP5LB2DY8WUsD R76PuwuAQstJQtKJzYWc8ndFykiTc55Yw9ZPdQLN4nwDvSY2VmuSM/n0QuyyFZ+/AL1+bHPpjIM fctVtUeZCSOpayidUjXFF2745G9qZRjSkFh0JU1A68TtK/8sC05M= X-Google-Smtp-Source: AGHT+IH7hf/KryzXOmVpli4Gumf+YtMqwVbraJFAHzsssHZreeLhO5RUIK6KHzZP8nyMpURuhcN1qo2oQFZ0yZJX59g= X-Received: by 2002:a19:f615:0:b0:511:4f77:c4a6 with SMTP id x21-20020a19f615000000b005114f77c4a6mr1566262lfe.42.1707906352092; Wed, 14 Feb 2024 02:25:52 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Bj=C3=B8rn_Ivar_Teigen?= Date: Wed, 14 Feb 2024 11:25:40 +0100 Message-ID: To: LUIS MIGUEL CONTRERAS MURILLO Cc: Tommy Pauly , "IETF IPPM WG (ippm@ietf.org)" Content-Type: multipart/alternative; boundary="0000000000006d2c2c061154efba" Archived-At: Subject: Re: [ippm] Adoption call for draft-olden-ippm-qoo-02 X-BeenThere: ippm@ietf.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: IETF IP Performance Metrics Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Feb 2024 10:25:58 -0000 --0000000000006d2c2c061154efba Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Luis, Thanks for your comments on the draft. On Sun, 11 Feb 2024 at 20:03, LUIS MIGUEL CONTRERAS MURILLO < luismiguel.contrerasmurillo@telefonica.com> wrote: > Hi Charis, all, > > > > Apologies for answering late to this adoption call. I support the adoptio= n > of the draft since it seems to be a useful mechanism for deriving > probabilistic insight on the expected performance of applications. > > > > From my review, there are some comments that I would like to be addressed > by the authors: > > > > - The draft is positioned as a kind of extension or complementary work > of BBF TR-452.1. However, the development of the proposed ideas in the > draft refer to distribution of percentile values of latency. Thus, in = my > understanding, whatever technique that could generate such latency > distribution could work (e.g., any monitoring system). In other words,= it > is not clear to me if the solution can be claimed as generic and > independent of BBF TR-452.1 (now, for instance, are claims such as =E2= =80=9CThe > foundation of the framework is Quality Attenuation=E2=80=9D). > > Good point. I think we should clarify that latency can be measured in different ways. It does not have to be exactly in accordance with TR-452.1 (although that's a very good way to do it!). > > - Not clear to me the use and need of measuring the throughput. The > examples are also unclear in that respect. > > Agreed. The throughput aspect of both the measurements and the requirements need more work. > > - The lowest threshold of the quality boundaries is called Network > Requirement for Perfection (NRP). This marks the baseline on the > performance, in other words, it refers to the behavior of the applicat= ion > on =E2=80=9Cideal conditions=E2=80=9D. That ideal conditions essential= ly mean the behavior > observed without any impairment from the network. That is, no latency = nor > packet loss coming from the network, in a kind of performance observed= in a > back-to-back running of the application between its endpoints. Once > impairments are introduced, the application performance will begin to > degrade. In summary, such =E2=80=9Cperfection=E2=80=9D correspond in f= act to =E2=80=9Cideal > conditions=E2=80=9D. The performance under ideal conditions can be mai= ntained to > some extent even when some impairments (latency, packet loss) are > introduced. The NRP as described on the examples includes some latency= and > some packet loss, so it would be interesting to have as reference how = far > NRP is from ideal conditions. This is because NRP does not actually > represents the 100% reference, but something lower than that. This is > important to fix the expectations of the application user=E2=80=99s ag= ainst how > robust are the applications respect to the network effects. > > The idea of network requirement for perfection is to specify the level of network impairments the application "knows how to deal with". In other words, if the network is as responsive and stable as the requirement specifies, then the application will work well. If I understand correctly, that is what you are saying also? > > - How compatible is this approach with adaptative applications? That > applications are able to react to network conditions changing the NRP = and > NRPoU references dynamically. > > Good question. Answering this question properly will require some more work, I think, but these are my thoughts on the subject at the moment: If the application adapts by lowering resolution, frame rate, or some other user-affecting aspect in a gradual way, then we might use the network requirement for the highest-fidelity level as the requirement for perfection and the lowest-fidelity level as the "useless" end of the requirement. Then the QoO score would correlate with the delivered fidelity. This might not work for all application, and in some cases it might be more appropriate to have one QoO score for each level of application fidelity, for instance. > - The paragraph about Volatile Networks of section 7 seems to be a > subsection, so probably requires to be numbered as 7.1 > > Agreed > > - Section 9 is probably not needed. > > I think this section is required since it's part of the template, but if I'm wrong about that then I agree with you. Thanks again for the detailed feedback. Cheers, Bj=C3=B8rn > > > Thanks, > > > > Best regards > > > > Luis > > > > *De:* ippm *En nombre de * Tommy Pauly > *Enviado el:* martes, 16 de enero de 2024 18:13 > *Para:* IETF IPPM WG (ippm@ietf.org) > *Asunto:* [ippm] Adoption call for draft-olden-ippm-qoo-02 > > > > Hello IPPM, > > > > This email starts a working group adoption call for "Quality of Outcome= =E2=80=9D > (draft-olden-ippm-qoo). > > > > https://datatracker.ietf.org/doc/draft-olden-ippm-qoo/ > > https://www.ietf.org/archive/id/draft-olden-ippm-qoo-02.html > > > > The call will last for 3 weeks, and end on *Tuesday, February 6*. Please > reply to this email with your review comments and indicate if you support > adopting this work. > > > > Please note that we did a previous adoption call that did not receive > sufficient feedback. At the last meeting at IETF 118, we did have a good > amount of comments and questions, so please do reply to this email if you > have reviewed the document. > > > > Thanks, > > Tommy & Marcus > > ------------------------------ > > Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, > puede contener informaci=C3=B3n privilegiada o confidencial y es para uso > exclusivo de la persona o entidad de destino. Si no es usted. el > destinatario indicado, queda notificado de que la lectura, utilizaci=C3= =B3n, > divulgaci=C3=B3n y/o copia sin autorizaci=C3=B3n puede estar prohibida en= virtud de > la legislaci=C3=B3n vigente. Si ha recibido este mensaje por error, le ro= gamos > que nos lo comunique inmediatamente por esta misma v=C3=ADa y proceda a s= u > destrucci=C3=B3n. > > The information contained in this transmission is confidential and > privileged information intended only for the use of the individual or > entity named above. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, distribution o= r > copying of this communication is strictly prohibited. If you have receive= d > this transmission in error, do not read it. Please immediately reply to t= he > sender that you have received this communication in error and then delete > it. > > Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinat=C3= =A1rio, > pode conter informa=C3=A7=C3=A3o privilegiada ou confidencial e =C3=A9 pa= ra uso exclusivo > da pessoa ou entidade de destino. Se n=C3=A3o =C3=A9 vossa senhoria o des= tinat=C3=A1rio > indicado, fica notificado de que a leitura, utiliza=C3=A7=C3=A3o, divulga= =C3=A7=C3=A3o e/ou > c=C3=B3pia sem autoriza=C3=A7=C3=A3o pode estar proibida em virtude da le= gisla=C3=A7=C3=A3o vigente. > Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique > imediatamente por esta mesma via e proceda a sua destrui=C3=A7=C3=A3o > _______________________________________________ > ippm mailing list > ippm@ietf.org > https://www.ietf.org/mailman/listinfo/ippm > --=20 Bj=C3=B8rn Ivar Teigen, Ph.D. Head of Research +47 47335952 | bjorn@domos.ai | www.domos.ai [image: https://www.understandinglatency.com/] --0000000000006d2c2c061154efba Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Luis,

Thanks fo= r your comments on the draft.

On Sun, 11 Feb 2024 at 20:03, LUIS MIGUE= L CONTRERAS MURILLO <luismiguel.contrerasmurillo@telefonica.com> wrote:

Hi Charis, all,

=C2=A0

Apologies for answering late to= this adoption call. I support the adoption of the draft since it seems to = be a useful mechanism for deriving probabilistic insight on the expected pe= rformance of applications.

=C2=A0

From my review, there are some = comments that I would like to be addressed by the authors:

=C2=A0

  • The draft is positioned as a kind of extension or c= omplementary work of BBF TR-452.1. However, the development of the proposed ideas in the draft refer to distribution of percentile values of latency. = Thus, in my understanding, whatever technique that could generate such late= ncy distribution could work (e.g., any monitoring system). In other words, = it is not clear to me if the solution can be claimed as generic and independent of BBF TR-452.1 (now, for instan= ce, are claims such as =E2=80=9CThe foundation of the framework is Quality = Attenuation=E2=80=9D).
= =C2=A0Good point. I think we should clarify that latency can be measured in= different ways. It does not have to be exactly in accordance with TR-452.1= (although that's a very good way to do it!).
  • Not clear to me the use and need of measuring the throughput= . The examples are also unclear in that respect. =C2=A0=C2=A0
Agreed. The throughput aspect of bot= h the measurements and the requirements need more work.=C2=A0
  • The lowest threshold of the quality boundaries is = called Network Requirement for Perfection (NRP). This marks the baseline on the performance, in other words, it refers to the behavior of the applicat= ion on =E2=80=9Cideal conditions=E2=80=9D. That ideal conditions essentiall= y mean the behavior observed without any impairment from the network. That = is, no latency nor packet loss coming from the network, in a kind of performance observed in a back-to-back running of the applica= tion between its endpoints. Once impairments are introduced, the applicatio= n performance will begin to degrade. In summary, such =E2=80=9Cperfection= =E2=80=9D correspond in fact to =E2=80=9Cideal conditions=E2=80=9D. The performance under ideal conditions can be maintained to some extent ev= en when some impairments (latency, packet loss) are introduced. The NRP as = described on the examples includes some latency and some packet loss, so it= would be interesting to have as reference how far NRP is from ideal conditions. This is because NRP does n= ot actually represents the 100% reference, but something lower than that. T= his is important to fix the expectations of the application user=E2=80=99s = against how robust are the applications respect to the network effects.
The idea of network requirement for perfection is to specify the = level of network impairments the application "knows how to deal with&q= uot;. In other words, if the network is as responsive and stable as the req= uirement specifies, then the application will work well. If I understand co= rrectly, that is what you are saying also?=C2=A0
  • How compatible is this approach with adaptative applications? T= hat applications are able to react to network conditions changing the NRP and NRPoU references dynamically.
=C2=A0Good question. Answering this question properly w= ill require some more work, I think, but these are my thoughts on the subje= ct at the moment:=C2=A0
If the application adapts by lowering res= olution, frame rate, or some other user-affecting aspect in a gradual way, = then we might use the network requirement for the highest-fidelity level as= the requirement for perfection and the lowest-fidelity level as the "= useless" end of the requirement. Then the QoO score would correlate wi= th the delivered fidelity. This might not work for all application, and in = some cases it might be more appropriate to have one QoO score for each leve= l of application fidelity, for instance.

<= div lang=3D"ES" style=3D"overflow-wrap: break-word;">
  • The paragraph about Volatile Networks of section 7 seems= to be a subsection, so probably requires to be numbered as 7.1 =C2=A0
Agreed
  • Section 9 is probably not needed.
<= /div>
I think this section is required since it'= s part of the template, but if I'm wrong about that then I agree with y= ou.

Thanks again for the detailed feedback.
<= div>
Cheers,
Bj=C3=B8rn
=C2=A0

=C2=A0

Thanks,

=C2=A0

Best regards

=C2=A0

Luis =C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0

=C2=A0

De: ippm <ippm-bounces@ietf.org> En nombre de Tommy Pauly
Enviado el: martes, 16 de enero de 2024 18:13
Para: IETF IPPM WG (ippm@ietf.org) <ippm@ietf.org>
Asunto: [ippm] Adoption call for draft-olden-ippm-qoo-02

=C2=A0

Hello IPPM,

=C2=A0

This email starts a working group adoption call for = "Quality of Outcome=E2=80=9D (draft-olden-ippm-qoo).

=C2=A0

=C2=A0

The call will last for 3 weeks, and end on Tuesda= y, February 6. Please reply to this email with your review comments and= indicate if you support adopting this work.

=C2=A0

Please note that we did a previous adoption call tha= t did not receive sufficient feedback. At the last meeting at IETF 118, we = did have a good amount of comments and questions, so please do reply to thi= s email if you have reviewed the document.

=C2=A0

Thanks,

Tommy & Marcus



Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, pu= ede contener informaci=C3=B3n privilegiada o confidencial y es para uso exc= lusivo de la persona o entidad de destino. Si no es usted. el destinatario = indicado, queda notificado de que la lectura, utilizaci=C3=B3n, divulgaci=C3=B3n y/o copia sin autorizaci=C3=B3= n puede estar prohibida en virtud de la legislaci=C3=B3n vigente. Si ha rec= ibido este mensaje por error, le rogamos que nos lo comunique inmediatament= e por esta misma v=C3=ADa y proceda a su destrucci=C3=B3n.

The information contained in this transmission is confidential and privileg= ed information intended only for the use of the individual or entity named = above. If the reader of this message is not the intended recipient, you are= hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If y= ou have received this transmission in error, do not read it. Please immedia= tely reply to the sender that you have received this communication in error= and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinat=C3=A1= rio, pode conter informa=C3=A7=C3=A3o privilegiada ou confidencial e =C3=A9= para uso exclusivo da pessoa ou entidade de destino. Se n=C3=A3o =C3=A9 vo= ssa senhoria o destinat=C3=A1rio indicado, fica notificado de que a leitura, utiliza=C3=A7=C3=A3o, divulga=C3=A7=C3=A3o e/ou c=C3=B3pia sem au= toriza=C3=A7=C3=A3o pode estar proibida em virtude da legisla=C3=A7=C3=A3o = vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique= imediatamente por esta mesma via e proceda a sua destrui=C3=A7=C3=A3o
_______________________________________________
ippm mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm


--
Bj=C3=B8rn Ivar Teigen, Ph.D.
Head of Research
+47 47335952 | bjorn@domos.ai=C2=A0|=C2=A0www.domos.ai
<= span style=3D"background-color:rgb(255,255,255)">