[ippm] Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)
"Kunze, Ike" <ike.kunze@comsys.rwth-aachen.de> Wed, 25 June 2025 15:03 UTC
Return-Path: <ike.kunze@comsys.rwth-aachen.de>
X-Original-To: ippm@mail2.ietf.org
Delivered-To: ippm@mail2.ietf.org
Received: from localhost (localhost [127.0.0.1]) by mail2.ietf.org (Postfix) with ESMTP id 26EA6395428A for <ippm@mail2.ietf.org>; Wed, 25 Jun 2025 08:03:06 -0700 (PDT)
X-Virus-Scanned: amavisd-new at ietf.org
X-Spam-Flag: NO
X-Spam-Score: -4.298
X-Spam-Level:
X-Spam-Status: No, score=-4.298 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: mail2.ietf.org (amavisd-new); dkim=pass (2048-bit key) header.d=rwth-aachen.de
Received: from mail2.ietf.org ([166.84.6.31]) by localhost (mail2.ietf.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YNOZKmbfZGjy for <ippm@mail2.ietf.org>; Wed, 25 Jun 2025 08:03:03 -0700 (PDT)
Received: from mail-out-2a.itc.rwth-aachen.de (mail-out-2a.itc.rwth-aachen.de [IPv6:2a00:8a60:1:e501::5:45]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail2.ietf.org (Postfix) with ESMTPS id 04D443954278 for <ippm@ietf.org>; Wed, 25 Jun 2025 08:03:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=rwth-aachen.de; i=@rwth-aachen.de; q=dns/txt; s=20240516-RWTH; t=1750863783; x=1782399783; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; bh=K6I37CLrxzw71Lj6AKQuhb37snIEGvikpVLvjmoGz2c=; b=pYgPK/Mgct6RTfgdQDpxbpTrUM9SCDRtTvoPtgvfc1tcwg9JH6KAiZUg 3CX+VVGNxKGOIrE+IzLj3agBtudakQSlEHjfjmTofigauHihjy/bGjSEa o6HJ+nTeZnmGFV/OBI+aQf6CQZOTprKRu4RC1qvCY4UX/noSVUtdS2fkA felCj2UT/GUzsSAEHT5sO33SkeqBA4HodfHqaF84+hNaz7EXQxik8aApV k82VpFrDzDhIG47zmi+u8I7ETSQo7O/AmdPZhNjgAYfyphxQeLOSO8Y4r zwiJCU9x6cA4gV6/cUmkHxHDv+STg2rKEsaAFbe7ef/Va/ikHRILDvAFU w==;
X-CSE-ConnectionGUID: 9P6eN6NdQOKhTdA1TLoJsg==
X-CSE-MsgGUID: LjWmV2dzT36UHu2MBwl2Cg==
X-IPAS-Result: A2BEAABqDlxo/6YagoZaGQEBAQEBAQEBAQEBAQEBAQEBARIBAQEBAQEBAQEBAQGCE4FBggMugTeEVJFxA5FIgjyGfi+CeYFrDwEBAQEBAQEBAQQEAT0UBAEBhQcCH4tOKDgTAQIEAQEBAQMCAwEBAQEBAQEBAQEBCwEBBgEBAQEBAQYFgSGFNUYBDIZaAQEBAQMaAwYEBiEdAwQXAgEGAhEDAQEBIQEJAgICLgEUCQgBAQQICw4NAgSCYQGCXBIUBj+UE5tLen8zgQGDbEFP2xkKBoFJAYFXgyuDNBoBKkhpAgIOgwyBK4Q/ggxDgRUnCwMNgWZRMT6CYQIBAQGBIwUBEQIBLgkJGAKDITqCLwSCCQQVWigKChILD36BGG2BZAIBBDw+BQ4oNxCBWoEtgk8mTwZTLgNAcgRqfj4CBAGGT1KBcCwBVRMXCwcFgSBDAyo0FRwjSwUtHYEngQCBThyEPIRFK0+CKAF7gX5MGUECg1weBm0PBoEhLUADC209NxQbkG4SIU9jGSgagiwJARAVRkQCASADBCIQCQgOAgkLPAcBDjItCBEBBRkFAQEPAQ8PCwsCLbESk1iBPgcDgjWBZoZdgzOCDYtkiWwEL4QEjQ2GSDmPAIJpZpcWcX0ijWWVLiwEHIUOAgQCBAUCGIE0S4EOWg8HcS4hKgGCPFIXAg+OLRYcg0KFE7UqeAIBOQIHAQoBAQMJkU4BAQ
IronPort-Data: A9a23:0GLC2ahgIDEQ9pgBduitLB2XX161NBEKZh0ujC45NGQN5FlHY01je htvWziHaa6CMGH0c9F3adm290oO7cCHm9RnGQJsri03QStjpJueD7x1DKtR0wC6c5efFhI3t 63yTvGacajYm1eF/k/F3oDJ9Cc6jefRAOKlWYYoAwgpLSd8UiAtlBl/rOAwh49skLCRDhiE0 T/Ii5S31GSNhXgtbAr414rZ8Eky5Kqo5WtC1rADTakjUGH2xiF94K03ePnZw0vQGuF8AuO8T uDf+7C1lkux1wstEN6sjoHgeUQMRLPIVSDW4paBc/H/6vTqjnVaPpcTbJLwW28O49m6t4wZJ OF2iHCFYVxB0pvkxb5BCUkIS0mSCoUdkFPPCSDXXcV+VCQqeVO0qxllJBle0YH1Zo+bDEkWn cH0JgzhYTjSv7y75uydRtBLpcQfIOCyAKcYmGpvmGSx4fYOGfgvQo3Q+sNY0Sd1mZwLFLDEe NYZLDNjKhjNC/FNEg5MVNRnx7/u3Ce5KmMGwL6WjfNfD2z7zQ98lrvwNsjYUteNQN5EhQCCu X7Gum3wCREXMpqTxFJp91r13baex3mlBd16+LuQ0vxVhWPJyn4vU0MrdUbhhNOmjhSVVIcKQ 6AT0m90xUQoz2SqVsH4dxy1vHDCuQQTM+e8CMUg9x2Nx7qR/FrcDy4eUSJBLdUq8sM7LdA36 mK0cxrSLWQHmNWopbi1rN94cRva1fApEFI/
IronPort-HdrOrdr: A9a23:RiRcnq5iVs96LqxUuQPXwHHXdLJyesId70hD6qkRc20tTiX8ra 2TdZsguiMc5Ax8ZJhCo6HnBED/exLhHPdOiOF6AZ6YUAPnoWulK5xL4Ib4zyf7OiXl94dmpM NdWpNzAt3oDVR8kO727RK1CcwAzNGL/LvAv5a4854Kd2FXgspbgTuRZDzrcXGf7GR9dOEEPa vZys1Zqz+nYHhSc8LTPBI4dtmGg+eOuYLgaRoHDQRiyA6VkDuu4rKSKWn84j4uFxd0hZoz9m DAkwbiooGuqeuyxBOZ90K71eU1pPLRjuBODsCNlcQZbg/UpW+TFfxcZ4E=
X-Talos-CUID: 9a23:nheHuWCfiV8GZPT6E3hfqkMyROYpS1r+zEvcChaELWFLU7LAHA==
X-Talos-MUID: 9a23:i4NmwgzXf/9bLoAih2xzh+U5aZCaqICsN0Uzy5kUh5LaCAxtYRK3kzDtaYByfw==
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-AV: E=Sophos;i="6.16,265,1744063200"; d="p7s'346?scan'346,208,217,346";a="131922657"
Received: from rwthex-w4-a.rwth-ad.de ([134.130.26.166]) by mail-in-2a.itc.rwth-aachen.de with ESMTP; 25 Jun 2025 17:03:02 +0200
Received: from rwthex-s1-a.rwth-ad.de (2a00:8a60:1:e500::26:152) by rwthex-w4-a.rwth-ad.de (2a00:8a60:1:e500::26:166) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1748.26; Wed, 25 Jun 2025 17:03:01 +0200
Received: from rwthex-s1-a.rwth-ad.de ([fe80::4e4b:1686:d8fa:8990]) by rwthex-s1-a.rwth-ad.de ([fe80::4e4b:1686:d8fa:8990%3]) with mapi id 15.02.1748.026; Wed, 25 Jun 2025 17:03:01 +0200
From: "Kunze, Ike" <ike.kunze@comsys.rwth-aachen.de>
To: "ippm@ietf.org" <ippm@ietf.org>
Thread-Topic: [ippm] Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)
Thread-Index: Advbn6X/VqbW5OwTT0OdZ8Dt/vRBWwHLJZ3wAF/2ITAAAQ1LgABkfMWA
Date: Wed, 25 Jun 2025 15:03:01 +0000
Message-ID: <D7B5AE74-0862-4B31-873A-806592D664BA@comsys.rwth-aachen.de>
References: <AS2PR07MB8978E53617A8E596106482A0E274A@AS2PR07MB8978.eurprd07.prod.outlook.com> <ZR1P278MB11703D6CBE27D29DE9577E13897FA@ZR1P278MB1170.CHEP278.PROD.OUTLOOK.COM> <PH7PR11MB7662B0CFD88FF7DD802DA4ABC779A@PH7PR11MB7662.namprd11.prod.outlook.com> <51CD422E-F881-4B1B-BC94-B0431F0EB528@ifi.uio.no>
In-Reply-To: <51CD422E-F881-4B1B-BC94-B0431F0EB528@ifi.uio.no>
Accept-Language: en-GB, de-DE, en-US
Content-Language: en-GB
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
x-originating-ip: [2a00:8a60:1014:10:94d:684d:35b5:c402]
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg="sha256"; boundary="B_3833715780_2803490514"
MIME-Version: 1.0
Message-ID-Hash: TDNWYMZ3M6YOVCZNC5CBWY3R6EDZ6Q55
X-Message-ID-Hash: TDNWYMZ3M6YOVCZNC5CBWY3R6EDZ6Q55
X-MailFrom: ike.kunze@comsys.rwth-aachen.de
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-ippm.ietf.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
X-Mailman-Version: 3.3.9rc6
Precedence: list
Subject: [ippm] Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/R0Nbg5MLwtnh-QhfYwQtW5OYUG4>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Owner: <mailto:ippm-owner@ietf.org>
List-Post: <mailto:ippm@ietf.org>
List-Subscribe: <mailto:ippm-join@ietf.org>
List-Unsubscribe: <mailto:ippm-leave@ietf.org>
Hi all, Having last read the document while it was still an individual draft, I have now reviewed the latest version and think that it is generally ready for publication in the current form. Thank you to the authors for this good work. I do have some questions and editorial comments, but all except one are non-blocking and should be easy fixes within the scope of this WGLC. I will list those at the end of this mail, starting with general comments and then going through the document in chronological order, ending with some nits. Pick the suggestions that you think make sense. I have one larger comment regarding the general structure of the document, though, which might require restructuring (parts of) the document if others agree that this is indeed important to address. I do not have a strong opinion on this, but would slightly favor this restructuring. In short, while the draft has lots of helpful (additional) information, I am not sure that the current structure is the best solution for helping readers to easily find the relevant information when only skimming or jumping through the document or, more generally, get a broad feeling for the conceptual components of the framework. Having read the document in full for the first time since the 01-version of the original individual draft, I can at least report that, personally, I only got the full QoO picture, i.e., how the framework as a whole is supposed to work, after really reading almost the entire document. I do not think that this is a problem per se, but easing access to the concepts might be a worthwhile endeavor. One way to approach this, that I could think of, is to have a dedicated section tackling the “overall design” of the framework. There, we could specify the different required components (latency/loss measurements, performance metrics, user-/operator-/developer-defined requirements,QoO calculation,..) and link them to the different corresponding (sub-)sections. Maybe it might make sense to then also move the “component” sections to subsections of this general design. >From a structural perspective, a position after the “Background” section could be viable and the current Sec. 4, 5, 6, 7, and 8 could then become subsections, maybe also Sec 9 and 10. Doing something like this proposal or similar would at least have helped me get better access to the content. Maybe, not doing such larger changes but simply adding a short additional paragraph at the end of the introduction could also do the trick. I guess it really depends on the amount of work we would like to spend on this vs. the expected benefit. If I am the only one seeing this potential improvement, then let’s leave it as it is. As I said above, I do not have a strong opinion on this and would be fine with the current form. I am mainly sharing my thoughts because I really like the content and would like to see this work get broad traction. @Bjørn: If you would like to have a chat about this, please feel free to reach out. I hope my comments help. Cheers, Ike — General comment: - Sometimes, it is not 100% clear if the document refers to “the QoO framework/metric” or a “general” metric/framework that would, e.g., satisfy some of the listed requirements. Would it make sense to always use specific modifiers, such as “the QoO framework/metric” when refering to QoO to remove any ambiguities? Section 1: - “Quality attenuation is a network quality metric that meets most of the criteria set out in the requirements” >From this text alone, it is not 100% clear which requirements are referred to here as “requirements” is used quite often in the first paragraph. Maybe refer to “design goals” to achieve a link to the “QoO is designed to” at the start of the first paragraph? - “The challenge lies in specifying how to simplify enough without losing too much in terms of precision and accuracy.” The crucial piece of information, that is only contained implicitly in the preceding paragraph, is that raw quality attenuation values are hard to interpret for end-users and app developers. Maybe make it explicit? For example: “However, interpreting raw quality attenuation values is difficult, raising the general challenge of how to simplify the entailed information without losing too much in terms of precision and accuracy.” - “This distance measure can, with some assumptions, calculate” Is it not more common to say something like “we can calculate the distance measure” than “the distance measure can calculate”? Maybe rephrase this? Section 2: - “First, the Internet is inherently stochastic from the perspective of any given client, so certainty is unattainable.” -> ’s/certainty/absolute certainty/‘? - ’s/accommodate diverse application requirements/such diverse application requirements’? - ’s/The overall goal/The overall goal of this document’? - “How can operators take action when measurements show that applications fail too often?” Having a question at that position in the document in this way feels a bit “weird” (for the lack of a better, more concise description of what I mean). Maybe rephrase it without a question? - “To summarize, the framework and "meaningful metric" should have the following properties:” The properties described hereafter seem a bit unbalanced regarding the form of presentation and information depth. Maybe it might be worth aligning them a bit more. Maybe explicitly add framework/metric to the bullet points to signify to which it applies. I do not have concrete alternatives to suggest, though, sorry. - “Requirements for end-users / from developers / for operators” Is the mixed used of for/from intended? Maybe make this consistent. - “Not all developers have network expertise, so to make it easy for developers to use the framework, developers must be able to specify network requirements approximately.” I am not sure I parse this sentence correctly. How would you specify requirements approximately? Or are you alluding to the “performance range” from NRP to NRPoU that QoO provides? -“Why?” As above, having a question here feels a bit “weird”. - ’s/to be maximally useful for operators/to maximize utility for operators’ - “s/OSI layer/layer in the protocol stack” - “The active testing [..] is OSI Layer and network technology independent” Is it really layer independent? I mean, the measurements that can be performed depend on the layer, do they not? And the mentioned examples are all layer 4+. Section 3: - I am wondering if “Background” is really the most suitable/descriptive title for this section. Maybe something like “Measuring Network Quality”? - Do you have references for the application requirements toward the end of the introduction paragraph? Those would be helpful starting points for people looking into this domain. - “Each metric is evaluated against the three criteria established in the requirements section.” -> “s/three criteria/three properties” This would link to the wording in Sec 2.2 more explicitly. - In the sections about IPDV and PDV, you talk about “one-delay”. Do you mean “one-way delay”? If not, what is one-delay? - 3.1 is titled “Discussion of other performance metrics” However, you also discuss Quality Attenuation, so in relation to what else are these the “other” metrics? Maybe just drop the “other”? I cannot come up with a better title right now, sorry. - The short paragraph in front of Table 1 as well as the explanations after Table 1 have some redundancies. Maybe try to better align these two parts or even incorporate the bullet points into the preceding paragraph? Section 4: - s/“a distribution—though such cases”/a distribution, though such cases” - “Description of the measurement path” What exactly does this description entail? Only the involved links? Section 7: - The formatting of the start of the paragraphs starting with “Latency Component”, “Packet Loss Component”, and “Final QoO Calculation” looks a bit bumpy. More specifically, having this short “bullet point”-style introductory part is not in the flow of the sentence. Maybe rephrase or restructure this a bit, so that there is a more “natural” flow. Section 8: - s/intuitively favorable—demonstrating that/intuitively favorable, demonstrating that/ - There are “step-by-step” guides at various points in the document which I like. However, they vary in the concrete format, e.g, sometimes they are formatted in bullet points and sometimes they are in full text. Maybe format all “step-by-step guides” in the same way? If I were to decide, I would probably choose the bullet-point style. - “Observe the application and note the threshold where the application stops being useful at all” Previously, you correctly discuss that there are different “thresholds” for different users. How would you deal with this aspect in this process? Section 9: - “In practice, these findings mean” Maybe rephrase this to: “From thesee findings, we deduce the following guidlines for the practical application:” to emphasize that you deduce guidelines? Section 11: - s/A method has been described/A method has been described in this document/ - “These two latency series: 1,200,1,200,1,200,1,200,1,200 and 1,1,1,1,1,200,200,200,200,200 will have identical distributions” -> The two latency series (1,200,1,200,1,200,1,200,1,200) and (1,1,1,1,1,200,200,200,200,200) have identical distributions Nits: - The use of acronyms is sometimes inconsistent, e.g., mixing the use of the acronym and the long form. Ideally, just introduce the acronym once and then use it thereafter. - Inconsistent use of end users vs. end-users - Sometimes inconsistent upper/lower case use, e.g., for Quality Attenuation - I have fixed additional nits in a merge request (see: https://github.com/getCUJO/QoOID/pull/17) From: Michael Welzl <michawe@ifi.uio.no> Date: Monday, 23. June 2025 at 19:06 To: "ippm@ietf.org" <ippm@ietf.org> Subject: [ippm] Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo) Dear all, I also support this draft - I have been following this document with great interest and think it could be very useful. So, I’m excited about the results Jason mentions below! Cheers, Michael Welzl On Jun 23, 2025, at 4:41 PM, Livingood, Jason <Jason_Livingood=40comcast.com@dmarc.ietf.org> wrote: I support this QoO draft. Via my employer, Comcast, we did a research study with Bjorn with some customers. The results, which he presented at a prior IETF meeting, were extremely positive. In particular: · 100% of participants said they liked Quality of Outcome better than speedtests · 86% felt Quality of Outcome matched their usage much better than their looking at bandwidth utilization over time · 96% said Quality of Outcome reports increased their trust in the ISP · 100% said Quality of Outcome info will increase their satisfaction · 66% said Quality of Outcome info will reduce their likelihood of calling customer support · 83% said getting recommendations in an application context will increase the trust in the recommendation (e.g., to get better Zoom quality to Jane’s iPad, do X or Y) We support this and, once standardized, we hope to deploy it. Jason From: Thomas.Graf@swisscom.com <Thomas.Graf@swisscom.com> Date: Saturday, June 21, 2025 at 12:51 To: ippm@ietf.org <ippm@ietf.org> Cc: bmwg@ietf.org <bmwg@ietf.org>, marcus.ihlar=40ericsson.com@dmarc.ietf.org <marcus.ihlar=40ericsson.com@dmarc.ietf.org> Subject: [ippm] Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo) Dear IPPM, We are almost at the half time of the WGLC and haven't seen feedback. Please consider reviewing the document and feedback on the mailing list. Best wishes Marcus and Thomas From: Marcus Ihlar <marcus.ihlar=40ericsson.com@dmarc.ietf.org> Sent: Thursday, June 12, 2025 3:42 PM To: IETF IPPM WG (ippm@ietf.org) <ippm@ietf.org> Subject: [ippm] Working group last call for Quality of Outcome (draft-ietf-ippm-qoo) Be aware: This is an external email. Hello IPPM, This email initiates the working group last call for Quality of Outcome (draft-ietf-ippm-qoo). The current version of the document can be found here: https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/ https://www.ietf.org/archive/id/draft-ietf-ippm-qoo-03.html Please review the document and reply to this email with any comments and indicate whether you believe it is ready for publication. This WGLC will last for three weeks and end on Thursday July 3, 2025. BR, Marcus and Thomas _______________________________________________ ippm mailing list -- ippm@ietf.org To unsubscribe send an email to ippm-leave@ietf.org
- [ippm] Working group last call for Quality of Out… Marcus Ihlar
- [ippm] Re: Working group last call for Quality of… Thomas.Graf
- [ippm] Re: Working group last call for Quality of… Livingood, Jason
- [ippm] Re: Working group last call for Quality of… Michael Welzl
- [ippm] Re: Working group last call for Quality of… Kunze, Ike
- [ippm] Re: Working group last call for Quality of… Greg Mirsky
- [ippm] Re: Working group last call for Quality of… Greg Mirsky
- [ippm] Re: Working group last call for Quality of… Giuseppe Fioccola
- [ippm] Re: Working group last call for Quality of… LUIS MIGUEL CONTRERAS MURILLO
- [ippm] Re: Working group last call for Quality of… Ruediger.Geib
- [ippm] Re: Working group last call for Quality of… Bjorn Ivar Teigen Monclair
- [ippm] Re: Working group last call for Quality of… Bjorn Ivar Teigen Monclair
- [ippm] Re: Working group last call for Quality of… Bjorn Ivar Teigen Monclair
- [ippm] Re: Working group last call for Quality of… Kunze, Ike
- [ippm] Re: Working group last call for Quality of… Giuseppe Fioccola
- [ippm] Re: Working group last call for Quality of… Marcus Ihlar
- [ippm] Re: Working group last call for Quality of… Marcus Ihlar
- [ippm] Re: Working group last call for Quality of… Kevin Smith, Vodafone