[ippm] Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)

"Kunze, Ike" <ike.kunze@comsys.rwth-aachen.de> Mon, 07 July 2025 13:32 UTC

Return-Path: <ike.kunze@comsys.rwth-aachen.de>
X-Original-To: ippm@mail2.ietf.org
Delivered-To: ippm@mail2.ietf.org
Received: from localhost (localhost [127.0.0.1]) by mail2.ietf.org (Postfix) with ESMTP id 0B3D13FD615B for <ippm@mail2.ietf.org>; Mon, 7 Jul 2025 06:32:36 -0700 (PDT)
X-Virus-Scanned: amavisd-new at ietf.org
X-Spam-Flag: NO
X-Spam-Score: -4.288
X-Spam-Level:
X-Spam-Status: No, score=-4.288 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=ham autolearn_force=no
Authentication-Results: mail2.ietf.org (amavisd-new); dkim=pass (2048-bit key) header.d=rwth-aachen.de
Received: from mail2.ietf.org ([166.84.6.31]) by localhost (mail2.ietf.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 03uAvtRCtNt6 for <ippm@mail2.ietf.org>; Mon, 7 Jul 2025 06:32:32 -0700 (PDT)
Received: from mail-out-2a.itc.rwth-aachen.de (mail-out-2a.itc.rwth-aachen.de [IPv6:2a00:8a60:1:e501::5:45]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail2.ietf.org (Postfix) with ESMTPS id E3DC03FD60DD for <ippm@ietf.org>; Mon, 7 Jul 2025 06:32:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=rwth-aachen.de; i=@rwth-aachen.de; q=dns/txt; s=20240516-RWTH; t=1751895152; x=1783431152; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; bh=MgaG7goH5vEPb0Gw9LCM+kBQIsQ4g8wFOkFgNdeJaAc=; b=JhfOKNz4YXcrpc0o55UL458gE3mYgjZeXU0Sx6FuMFmkT6glW7jy+2pN MIqGc0hh9UadjujhZOzqInK5ae8iztbQkzjsILVr993JHMgCiNXPmc9r0 KWNdAthYbXGBitHbKZ0hP+ysd8rQfrksqkGm0cbv8yBVITP4Y/yWHwea+ JC7m1/sMtZd+4MZjEZRrixCWOrMTRLSsz1deFXrrDVOxtQB6aWXaIzi15 j665ZtIudb2MNmCYDwDJ+/XDZYPv18Cc5NDho3SSHxN/j0DGZkOpGbl57 gPyQjJQbb3IjKYp1S3NI+Y+P1Mk5DQgNlce3KurfvVN3gzBeBD7XUNmTY Q==;
X-CSE-ConnectionGUID: 90cIbe1BSKOTuj2J1gHWFg==
X-CSE-MsgGUID: nPZk3/QgT0OH9i8W8e1syw==
X-IPAS-Result: A2AMAADEy2to/6YagoZaFgQBAQEBAQEBAQEBAwEBAQESAQEBAQICAQEBAYIBAwEBAQELAYFAW4EoLoE3hFSRcQOBE4pRhWSJO4MUFIFrDwEBAQEBAQEBAQQEAT0UBAEBghOCdAIfi1ooNgcOAQIEAQEBAQMCAwEBAQEBAQEBAQEBCwEBBgEBAQEBAQYFgSGFNUYBDIZaAQEBAQIBGgECBgpFDAsCAQYCEQMBAQEJDQsBCQICAh4FDB0IAQEEEgEODQSCYwGCOQMOEhIUBj+SYJsUN3qBMoEBg2xBT9hBDYJMEIFJAYFXgyuDMAQaASpIaQICDoMMgQ4dg0R7ggxDJm4BJw4NgWZKBzE+gUpVQgIBAQGBFggBBAUBCwcBByEHCQgBBgEFDAIFgxw6gi8EgQ5/FUQ+FB2BDTQEEEoDKDaBXDYTgiCBKkOCHiYCJoEOhDsshi1SgXAsAVUTFwsHBYEgQwMqNBUcI0sFLR2BJ36BTRqEGoQqK0+CInWBeUEZPwKDUx4GbQ8GgR0bTAICAgUCGkADC209NxQbkGwSIU5sGSkagjEKBSBGBgEHDwwPDBsIAwQUBw0KEQgGAgkLDg0eAggBIwgCCwE0CwoEAgQEEQkBAQEECgIBJwsCHg+STRQbk1yKQZNYTXEHA4I1gWeGXYMzgg6CWIxkhhYEL4QEgVeLOAOGRjmPAIJpZpcTdn0igjaLL4QIkSksBA+FGwIEAgQFAhiBNDsHgRdwcU8qAYI8CUkXAg9XjVYWgRIBAgeHVbxyeAIBATgCBwEKAQEDCYZIBYZfgV8eAQE
IronPort-Data: A9a23:lhFOvK+M6ZeRUwfugnOoDrUD1H+TJUtcMsCJ2f8bNWPcYEJGY0x3z 2ofDWiDO6zcM2v8L4x1bYS29klXupXSm9Q1QVdo+HtEQiMRo6IpJzg4wmTYYnnOdJ2TFCqLy +1EN7Es+ehtFie0Si+Fa+an9T8lk/nRF9IQMcacUghpXwhoVSw9vhxqnu89k+ZAjMOwa++3k YqaT/b3Zhn8hVaYDkpOs/je8Ew346yu0N8llgVWic5j7Qe2e0Y9Ucp3yZGZdxPQXoRSF+imc OfPpJnR1n/Z5RokFuS+mb/9dEAQKpaKVeRZoiMLM0QKqkEqSh0ais7XBtJFAatko2nhc+RK9 Tl4ncfYpTEBY/eQwrRNC3G0JAklVUFO0OevzXFSKqV/xWWeG5fn660G4E3boeT0984vaVyi+ 8D0JxhcNhuije/v6omYbdVxhJ4BdcbLA60Q7yQIITHxVZ7KQLjZXLnK6M8dw2t1j4ZUAureI sMVLzZiBPjCS0QUZhFOU8p4xrnu3yehG9FbgAv9Sa4f4mveig9s1qrgGNHSf8ebXoNPgVqY4 2vP9GT0BFcWObRzzBLcqSn227SXxH2TtIQ6EoWK9aUtuVKvnkszWRxGagu2+OKolRvrMz5YA wlOksY0loAz7FSuZtjwQxP+p2SL1iPwQPJKDPE65RHI1faR6kCDGXQECzdNLtAr3CMreQEXO payt4uBLVRSXHe9EhpxKp/8QeuOBBUo
IronPort-HdrOrdr: A9a23:EaKoJaul8wyJNTXDDyeYT5iK7skDFtV00zEX/kB9WHVpm6Oj+v xG8M526faXslcssRgb8LjqBEDqex3hHPBOjrX5cY3DYOHX0lHDEL1f
X-Talos-CUID: 9a23:O6y4xWh8so/qaFbrcVmg0KzehjJuIyXW7X3demWDKkFVVb/Pawaf2JxKnJ87
X-Talos-MUID: 9a23:j1PiwgZbRG6o4+BTsQfeh2olDchUu4OOWUFdrsUL4tKVKnkl
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-AV: E=Sophos;i="6.16,294,1744063200"; d="p7s'346?scan'346,208,217,346";a="133253016"
Received: from rwthex-w4-a.rwth-ad.de ([134.130.26.166]) by mail-in-2a.itc.rwth-aachen.de with ESMTP; 07 Jul 2025 15:32:30 +0200
Received: from rwthex-s1-a.rwth-ad.de (2a00:8a60:1:e500::26:152) by rwthex-w4-a.rwth-ad.de (2a00:8a60:1:e500::26:166) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1748.26; Mon, 7 Jul 2025 15:32:29 +0200
Received: from rwthex-s1-a.rwth-ad.de ([fe80::4e4b:1686:d8fa:8990]) by rwthex-s1-a.rwth-ad.de ([fe80::4e4b:1686:d8fa:8990%3]) with mapi id 15.02.1748.026; Mon, 7 Jul 2025 15:32:29 +0200
From: "Kunze, Ike" <ike.kunze@comsys.rwth-aachen.de>
To: "ippm@ietf.org" <ippm@ietf.org>
Thread-Topic: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)
Thread-Index: Advbn6X/VqbW5OwTT0OdZ8Dt/vRBWwO74zMwAF92NUIAPCnE7wCFBmbwAAxyMQA=
Date: Mon, 07 Jul 2025 13:32:29 +0000
Message-ID: <5C0C556B-E03C-43D0-9F7E-C3406287936D@comsys.rwth-aachen.de>
References: <AS2PR07MB8978E53617A8E596106482A0E274A@AS2PR07MB8978.eurprd07.prod.outlook.com> <BEZP281MB2007F0F810967A3EAA5450D89C40A@BEZP281MB2007.DEUP281.PROD.OUTLOOK.COM> <GVXPR02MB10560D120CBB675B3CD0164A5EA43A@GVXPR02MB10560.eurprd02.prod.outlook.com> <GVXPR02MB105600730C69B06F3BE8FE7DDEA42A@GVXPR02MB10560.eurprd02.prod.outlook.com> <GVXPR02MB105608D29859C95E32F8EEE78EA4FA@GVXPR02MB10560.eurprd02.prod.outlook.com>
In-Reply-To: <GVXPR02MB105608D29859C95E32F8EEE78EA4FA@GVXPR02MB10560.eurprd02.prod.outlook.com>
Accept-Language: en-GB, de-DE, en-US
Content-Language: en-GB
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
x-originating-ip: [2a00:8a60:1014:10:5f8:5960:4491:af]
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg="sha256"; boundary="B_3834747148_4094410352"
MIME-Version: 1.0
Message-ID-Hash: MHHAPAEMNY2KK56UQD3JMNUJKNBRVS5F
X-Message-ID-Hash: MHHAPAEMNY2KK56UQD3JMNUJKNBRVS5F
X-MailFrom: ike.kunze@comsys.rwth-aachen.de
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-ippm.ietf.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
X-Mailman-Version: 3.3.9rc6
Precedence: list
Subject: [ippm] Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/t2eaOGKZm_UG_e6fnV8IYy7XiTc>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Owner: <mailto:ippm-owner@ietf.org>
List-Post: <mailto:ippm@ietf.org>
List-Subscribe: <mailto:ippm-join@ietf.org>
List-Unsubscribe: <mailto:ippm-leave@ietf.org>

Thanks, Bjørn!

The changes look good to me and address my comments.

 

 

From: Bjorn Ivar Teigen Monclair <bjorn.monclair@cujo.com>
Date: Monday, 7. July 2025 at 14:58
To: "ippm@ietf.org" <ippm@ietf.org>, IPPM Chairs <ippm-chairs@ietf.org>
Cc: Magnus Olden <magnus.olden@cujo.com>, "Ruediger.Geib@telekom.de" <Ruediger.Geib@telekom.de>, "Jason Livingood (Comcast)" <Jason_Livingood@comcast.com>, Michael Welzl <michawe@ifi.uio.no>, LUIS MIGUEL CONTRERAS MURILLO <luismiguel.contrerasmurillo@telefonica.com>, Greg Mirsky <gregimirsky@gmail.com>, "giuseppe.fioccola=40huawei.com@dmarc.ietf.org" <giuseppe.fioccola=40huawei.com@dmarc.ietf.org>, "Kunze, Ike" <ike.kunze@comsys.rwth-aachen.de>
Subject: Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)

 

Dear IPPM Working Group,

 

On Friday I did not have time to detail the changes made. Sorry about that. Here is the complete breakdown:

 

## Giuseppe's Comments

 

> * In the document there are new terms introduced (e.g. Quality Attenuation, QoO score, etc). I think they can be all defined in a terminology section.

 

[Bjørn] Added a  Terminology section (Section 3) with definitions for Network, Quality Attenuation, Quality of Outcome (QoO), QoO Score, NRP, NRPoU, Composability, and Accuracy/Precision.

 

> * In section 3 on Background, several techniques are listed and I suggest to also add other IPPM hybrid techniques, such as the On-Path Telemetry methods (i.e. IOAM, AltMark).

 

[Bjørn] Added "On-Path Telemetry methods (IOAM, AltMark)" to the list of relevant techniques in the Background section.

 

> * The overall organization of the document should be revised and improved. For example section 4, section 5 and section 6 can become subsections of a new section on Sampling and Network Requirements and the same applies to section 9 and section 10, that can be put under a new section on Insights.

 

[Bjørn] Restructured document with consolidated "Sampling and Network Requirements" section and "Insights" section as suggested.

 

> * Note that the current section 13 on "Conventions and Definitions" should be moved earlier in the document since it is usually placed after the Introduction.

 

[Bjørn] Removed "Conventions and Definitions" since no reference to RFC2119 is needed.

 

## Ruediger's Comments

 

> [RG] The document frequently refers to "network" as to be characterized. Yet I failed to find a definition or a reference to a definition. Please clarify, what you mean by "network".

 

[Bjørn] Added definition of "Network" in the Terminology section: "the communication infrastructure that connects endpoints, including all intermediate devices, links, and protocols that affect the transmission of data between a source and destination."

 

> ‎[RG] I note, that “network operators” have little incentive…in one statement. Does this express, that a consumer will reliably buy the best performing Home Gateway to optimize his QoE by minimizing the effect of the home network on their app?

 

[Bjørn] I assume some well-informed consumers will act in that way, yes, but certainly not all of them. I believe there are two mechanisms at work  here - one is existing consumer demand for a home network that "just works", and the other is ISPs strategically choosing to optimize for application performance in anticipation of a consumer shift away from a pure focus on capacity.

 

> Section 2:

> 

> Existing network quality metrics and frameworks typically address the needs of one or two of these stakeholders,

> 

> [RG] Please provide references for existing quality metrics & frameworks addressing the needs of one or two of these stakeholders only.

 

[Bjørn] Added context and references discussing how existing metrics (throughput for regulatory compliance, QoE metrics for users) typically serve limited stakeholder groups.

 

> Section 2:

> 

> While solutions exist for many of the problems causing high and unstable latency in the Internet, the incentives to deploy them have remained relatively weak.

> [RG] Please provide references for both, "problems" and "incentives".

 

[Bjørn] Added references to bufferbloat mitigation techniques (RFC8290), congestion control algorithms (RFC8033), and BITAG latency study to support claims about deployment incentives.

 

> Section 2

> 

> operators have little incentive to improve network quality beyond increasing throughput. Despite the availability of open-source solutions, vendors rarely implement them. The root cause lies in the absence of a > universally accepted network quality framework that captures how well applications are likely to perform.

 

> [RG] Do you mean throughput or bandwidth?

 

[Bjørn] Changed to "have little incentive to improve network quality beyond increasing bandwidth."

 

> ‎[RG] I’m not sure, whether the root cause for the claimed unwillingness of vendors to implement open source solutions is a missing network quality framework. Can you provide some proof or a reference for your claim? Further, your proposal is to characterize two thresholds (perfect and stop of app) and interpolate linearly between these, based on a subset of performance metrics. You apply a set of simplification, rendering the diagnosis less reliable. The dependency on app developer publications isn’t increasing the quality of diagnosis too, unless a standardized testing environment & conditions is the basis for these publications.

 

[Bjørn] It's hard to really prove that statement, but I do think it's pointing in the direction of truth. I've added references and softened the language a bit. 

 

> [RG] Please reword:

> “A universally accepted network quality framework that successfully captures how well applications are likely to perform may help to increase the willingness of vendors to….”

 

[Bjørn] Clarified terminology and softened claims with "may help to increase the willingness of vendors to implement such solutions".

 

> Section 2

> 

> Therefore, one critical requirement of a meaningful framework is its ability to answer the question, "Will an application work properly?"

> 

> 

> [RG] To me a framework is a starting point and a standard an end. To say, the standard would offer the "ability". Please reword: … is to enable work on a standard answering the question, "Will networking conditions stop an application from working properly?"

 

[Bjørn] Reworded to focus on the framework's ability to answer the question about networking conditions negatively affecting applications.

 

>Section 2

> 

> Answering this question requires several considerations. First, the Internet is inherently stochastic from the perspective of any given client, so certainty i unattainable.

> 

> 

> [RG] Please correct/reword (would that be an "is"?)

 

[Bjørn] Fixed "certainty i unattainable" to "absolute certainty is unattainable."

 

> [RG] Please reword: To summarize, the framework and "meaningful QoO metric" should have the following properties: Capture a set of network performance metrics which provably correlate to the application quality of a set of different applications as perceived by users.

 

[Bjørn] Updated requirements section with the suggested more precise wording.

 

> [RG] Please add a ref to https://www.rfc-editor.org/rfc/rfc7680.html#section-2.8.2 and discuss the novelty of TR-452.1 as compared to the RFC 7680 loss specification.

 

[Bjørn] Added RFC 7680 reference and discussion: "Similar to the One-Way Loss Metric for IP Performance Metrics (IPPM) [RFC7680], which defines packet loss in terms of packets that fail to arrive within a specified time threshold, the Quality Attenuation approach extends this concept..."

 

> [RG] Please inform the reader, where the new per App threshold values NRPoU percentiles and NRP are originating or determined.

 

[Bjørn] Added section on standardized testing conditions and requirement for application developers to publish their testing methodologies.

 

> [RG] I'd prefer references for the performance figures.

 

[Bjørn] Added JamKazam reference for remote music collaboration latency requirements and updated the example to be more specific.

 

 

## Luis's Comment

 

> One question / comment form my side. It is not clear to me what could be the behavior of the QoO with adaptive applications that can accommodate e.g. the bit rate to the network conditions.

 

[Bjørn] Added  "Adaptive Applications" section explaining how QoO can accommodate applications with multiple quality levels through multiple NRP thresholds or simplified single-threshold approaches.

 

## Greg's Comments

 

> "Mean" and "average" are used in the document almost equal number of times. However, RFC 6049 uses only "mean" in reference to composable spatial performance metric.

 

[Bjørn] Standardized terminology to use "mean" consistently throughout the document.

 

> "Accuracy" and "precision" are used in the document extensively, sometimes even in the same sentence. In general, accuracy can be interpreted as precision.

 

[Bjørn] Added definitions to Terminology section clarifying that "accuracy" refers to how close measurements are to the true value, while "precision" refers to consistency and repeatability, noting they are not interchangeable.

 

> A nit. I think that we use "Quality Attenuation" as a term. In that case, it must be uppercased throughout the document.

 

[Bjørn] Standardized capitalization of "Quality Attenuation" throughout the document.

 

> In the Motivation s/i unattainable/is unattainable/

 

[Bjørn] Fixed typo.

 

## Ike's Comments

 

> "Quality attenuation is a network quality metric that meets most of the criteria set out in the requirements" - From this text alone, it is not 100% clear which requirements are referred to here

 

[Bjørn] Changed to "meets most of the design goals set out in the requirements section of this document" for clarity.

 

> Maybe make it explicit? For example: "However, interpreting raw quality attenuation values is difficult, raising the general challenge of how to simplify the entailed information without losing too much in terms of precision and accuracy."

 

[Bjørn] Added explicit statement about the difficulty of interpreting raw Quality Attenuation values.

 

> 's/certainty/absolute certainty/'?

 

[Bjørn] Changed to "absolute certainty is unattainable."

 

> 's/accommodate diverse application requirements/such diverse application requirements'?

 

[Bjørn] Updated

 

> 's/The overall goal/The overall goal of this document'?

 

[Bjørn] Updated.

 

> "s/OSI layer/layer in the protocol stack"

 

[Bjørn] Changed to "applied across different layers of the protocol stack and is network technology independent."

 

> The formatting of the start of the paragraphs starting with "Latency Component", "Packet Loss Component", and "Final QoO Calculation" looks a bit bumpy.

 

[Bjørn] Improved formatting of the calculation examples for better readability.

 

> There are "step-by-step" guides at various points in the document which I like. However, they vary in the concrete format

 

[Bjørn] Standardized formatting of step-by-step guides throughout the document.

 

> "These two latency series: 1,200,1,200,1,200,1,200,1,200 and 1,1,1,1,1,200,200,200,200,200 will have identical distributions"

 

[Bjørn] Reformatted as: "The two latency series (1,200,1,200,1,200,1,200,1,200) and (1,1,1,1,1,200,200,200,200,200) have identical distributions"

 

> Inconsistent use of end users vs. end-users

 

[Bjørn] Standardized to "end-users" throughout the document.

 

[Bjørn] I also added an Overview section (Section 2) to help readers quickly understand the key concepts before diving into detailed specifications, addressing the general concern about document accessibility.

 

The updated draft is available at: https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/

draft-ietf-ippm-qoo-04 - Quality of Outcome
This document introduces the Quality of Outcome (QoO) framework, a novel approach to network quality assessment designed to align with the needs of application developers, users, and operators. By leveraging the Quality Attenuation metric, QoO provides a unified method for defining and evaluating application-specific network requirements while ensuring actionable insights for network ...
datatracker.ietf.org
 

We believe these changes significantly improve the document's clarity, completeness, and technical accuracy while addressing all substantive comments from the WGLC review.

 

Best regards,

Bjørn Ivar Teigen Monclair

From: Bjorn Ivar Teigen Monclair <bjorn.monclair@cujo.com>
Sent: Friday, July 4, 2025 6:11 PM
To: ippm@ietf.org <ippm@ietf.org>; IPPM Chairs <ippm-chairs@ietf.org>
Cc: Magnus Olden <magnus.olden@cujo.com>; Ruediger.Geib@telekom.de <Ruediger.Geib@telekom.de>; Jason Livingood (Comcast) <Jason_Livingood@comcast.com>; Michael Welzl <michawe@ifi.uio.no>; LUIS MIGUEL CONTRERAS MURILLO <luismiguel.contrerasmurillo@telefonica.com>; Greg Mirsky <gregimirsky@gmail.com>; giuseppe.fioccola=40huawei.com@dmarc.ietf.org <giuseppe.fioccola=40huawei.com@dmarc.ietf.org>; ike.kunze@comsys.rwth-aachen.de <ike.kunze@comsys.rwth-aachen.de>
Subject: Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo) 

 

Hi IPPM,

 

A new version of the draft, draft-ietf-ippm-qoo-04, has now been uploaded to the data tracker: https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/

 

It addresses, to the extent we have been capable of doing so, all of the suggestions and issues raised since the WGLC was announced.

 

Thanks again to everyone who has contributed to the document, your time and effort is greatly appreciated.

 

Cheers,

Bjørn

From: Bjorn Ivar Teigen Monclair <bjorn.monclair@cujo.com>
Sent: Thursday, July 3, 2025 1:34 PM
To: ippm@ietf.org <ippm@ietf.org>; IPPM Chairs <ippm-chairs@ietf.org>
Cc: Magnus Olden <magnus.olden@cujo.com>; Ruediger.Geib@telekom.de <ruediger.geib@telekom.de>; Jason Livingood (Comcast) <Jason_Livingood@comcast.com>; Michael Welzl <michawe@ifi.uio.no>; Ruediger.Geib@telekom.de <ruediger.geib@telekom.de>; LUIS MIGUEL CONTRERAS MURILLO <luismiguel.contrerasmurillo@telefonica.com>; Greg Mirsky <gregimirsky@gmail.com>; giuseppe.fioccola=40huawei.com@dmarc.ietf.org <giuseppe.fioccola=40huawei.com@dmarc.ietf.org>; ike.kunze@comsys.rwth-aachen.de <ike.kunze@comsys.rwth-aachen.de>
Subject: Re: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo) 

 

Hi Jason, Michael, Ike, Guiseppe, Greg, Luis, Ruediger, IPPM chairs and participants,

 

Thank you very much for the detailed feedback! Greatly appreciate the time you've all invested in this.

 

On restructuring:

I see how the document would benefit from restructuring in the way Ike suggests. If restructuring can be done without causing a lot of delay (and work!), I am happy to submit a new version along the suggested lines. If restructuring would require significant delays and reviews (or, if others disagree about the need for restructuring), then perhaps adding a new sub-section to the introduction is the better option? Chairs and seasoned IETF'ers, I hope you can advice.

 

 I will take all of your feedback into account and produce a new version of the document before the deadline on the 7th.

 

Cheers,

Bjørn

 

From: Ruediger.Geib@telekom.de <Ruediger.Geib@telekom.de>
Sent: Wednesday, July 2, 2025 5:20 PM
To: Magnus Olden <magnus.olden@cujo.com>; Bjorn Ivar Teigen Monclair <bjorn.monclair@cujo.com>
Cc: ippm@ietf.org <ippm@ietf.org>
Subject: AW: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo) 

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

 

Hi Bjørn, hi Magnus,

 

Thanks for your framework for multi-dimensional metric linking delay and packet loss. Please find some [RG] marked comments below.

 

Please note, that I’m off for vacations and likely won’t respond to replies before early August. I’m sorry for that.

 

Regards, Ruediger

 

 

 

[RG] The document frequently refers to “network” as to be characterized. Yet I failed to find a definition or a reference to a definition. Please clarify, what you mean by “network”.

I note, that “network operators” have little incentive…in one statement. Does this express, that a consumer will reliably buy the best performing Home Gateway to optimize his QoE by minimizing the effect of the home network on their app? 

 

Another general comment, without being a native speaker, maybe “comprehensible” rather than “understandable”? Could some native speaker confirm/reject that?

 

----

 

Section 2:

Existing network quality metrics and frameworks typically address the needs of one or two of these stakeholders,

 

[RG] Please provide references for existing quality metrics & frameworks addressing the needs of one or two of these stakeholders only.

 

----

 

Section 2:

While solutions exist for many of the problems causing high and unstable latency in the Internet, the incentives to deploy them have remained relatively weak.

 

[RG] Please provide references for both, “problems” and “incentives”.

 

----

 

Section 2

operators have little incentive to improve network quality beyond increasing throughput. Despite the availability of open-source solutions, vendors rarely implement them. The root cause lies in the absence of a universally accepted network quality framework that captures how well applications are likely to perform.

 

[RG] Do you mean throughput or bandwidth? Consumers may receive offers of access types with higher bandwidth. Above a minimum access bandwidth, further increase of bandwidth doesn’t increase per app throughput. A suitable location of content and service servers would. 

 

[RG] I’m not sure, whether the root cause for the claimed unwillingness of vendors to implement open source solutions is a missing network quality framework. Can you provide some proof or a reference for your claim? Further, your proposal is to characterize two thresholds (perfect and stop of app) and interpolate linearly between these, based on a subset of performance metrics. You apply a set of simplification, rendering the diagnosis less reliable. The dependency on app developer publications isn’t increasing the quality of diagnosis too, unless a standardized testing environment & conditions is the basis for these publications.

 

Please reword:
“A universally accepted network quality framework that successfully captures how well applications are likely to perform may help to increase the willingness of vendors to….”

 

----

 

Section 2

Therefore, one critical requirement of a meaningful framework is its ability to answer the question, "Will an application work properly?"

 

[RG] To me a framework is a starting point and a standard an end. To say, the standard would offer the “ability”. Please reword:

… is to enable work on a standard answering the question, "Will networking conditions stop an application from working properly?"

 

----

 

Section 2

Answering this question requires several considerations. First, the Internet is inherently stochastic from the perspective of any given client, so certainty i unattainable.

 

[RG] Please correct/reword (would that be an “is”?)

 

----

 

2.2. Requirements

Flexibility in describing application requirements and the ability to capture the delay characteristics of the network in sufficient detail are necessary to compute application success with satisfactory accuracy and precision.

 

[RG] Please specify “satisfactory accuracy and precision” or provide a reference. Please reword:
necessary to improve accuracy and / or precision of…

 

[RG] Please define what you mean by “to compute application success” and how that is linked to the QoO network score (the framework set out to describe the latter). 

----

 

2.2. Requirements

To summarize, the framework and "meaningful metric" should have the following properties:

 

Capture the information necessary to compute the probability that applications will work well. (Useful for end-users and application developers.)

 

[RG] I’d prefer rewording to 

To summarize, the framework and "meaningful QoO metric" should have the following properties:

 

Capture a set of network performance metrics which provably correlate to the application quality of a set of different applications as perceived by users.

----

 

2.2. Requirements

 

Compare meaningfully to different application requirements.

 

[RG] Please reword to be more specific on what you mean. Mathematically, and if you’d like to compute something meaningful, QoO score and app QoE must be correlated, even if you don’t go for MOS. If this is captured by my reworded version of the first requirement, then that’s it. If the above second requirement is that the function individual_app_quality (QoE score) needs be known, then clearly say that. As far as I understand your draft, this function is out of scope.

 

----

 

2.2. Requirements

 

Compose.

 

[RG] Should that be de-compose and isolate? So far the text didn’t explain that the QoO score is calculated network-section-wise and finally composed.

 

----

 

2.2.1. Requirements for end-users

 

The quality framework should facilitate a metric that is objective, relatable, and relatively understandable

 

[RG] please reword to
…. metric that is based on objective QoS measurements, correlated to application quality  and relatively understandable

 

----

 

2.2.1. Requirements for end-users

 

If these requirements are met, the end-user can understand if a network can reliably deliver what they care about: the outcomes of applications.

 

[RG] I’d prefer a more defensive way of putting it, but the statement doesn’t say, the network is the only cause of bad app outcome.

My preference: 

… can understand if a network can is a likely source of impairment for what they care about: the outcomes of applications.

 

----

 

2.2.1. Requirements for end-users

A compromise is for the quality of experience framework to place the responsibility for sourcing and representing end-user requirements onto the application developer.

 

[RG] If I understand this statement correctly, you put crating input for the second requirement of section 2.2 above out of scope of this framework. Please do clearly state that in the first instance where his is mentioned. Suggestion:

 

Compare meaningfully to different application requirements. Note that providing the relevant information depends on application developers publications or standards, where available.  

 

[RG] Further, this statement and the remaining content define requirements from app developers. Please move to section 2.2.2

 

----

 

2.2.1. Requirements for end-users

 

Some real world examples where 'acceptable levels' have been derived by application developers include (note: developers of similar applications may have arrived at different figures):

 

Remote music collaboration: 28ms latency note-to-ear for direct monitoring, <2ms jitter

 

Online gaming: 6Mb/s downlink throughput and 30ms RTT to join a multiplayer game

 

Virtual reality: <20ms RTT from head motion to rendered update in VR

 

Performing this UAT helps the developer understand what likelihood a new end-user has of an acceptable Quality of Experience based on the application's existing requirements towards the network.

 

[RG] I’d prefer references for the performance figures. 

 

----

 

2.2.2. Requirements from Application and Platform Developers

 

[RG] From my point of view asking for useful UAT requirement publication from App developers requires standard testing conditions (testbed, config and path, forwarding & load characteristics). This might be read into the paragraph as is, but it should be explicitely expressed, whether or not such standard conditions for app characterization are required by this section, or not. I don’t know the authors preference, so I don’t propose text. 
As a thought, if app developers derive data under different networking conditions, the overall accuracy and precision of the QoO metric will be reduced.

 

----

 

3. Background

 

[RG] Please move the text of “3.1.8. Quality Attenuation” to “Background”, as this seems to indicate latency and loss as two metrics required. Please be also more precise, as to what Quality Attenuation captures regarding latency distribution. I’m wooried, as you address Average and 99th percentile latency by separate sections.

 

 

3. Background

 

The novelty of the Quality Attenuation metric is to view packet loss as infinite (or too late to be of use e.g. > 3 seconds) latency [TR-452.1].

 

[RG] Please add a ref to https://www.rfc-editor.org/rfc/rfc7680.html#section-2.8.2 and discuss the novelty of TR-452.1 as compared to the RFC 7680 loss specification. You may also reword to:

“Similar, as the One-Way Loss Metric for IP Performance Metrics (IPPM) [RFC 7680], the Quality Attenuation metric is to view…

 

----

 

3. Background

 

All applications require a minumum level of throughput and a maximum packet loss rate.

 

[RG] Please reword to  ..and work acceptable up to a maximum…

 

----

 

3.1. Discussion of other performance metrics

 

[RG] Which metrics have been discussed prior to the “other performance metrics”. If there’s no clear text to that, please

 

-----

 

3.1.3. Variance of latency

 

[RG] Adding the variance of random processes is producing correct results is producing correct results, if these are not correlated. Please add a statement. Please provide a reference to a metric definition for Latency Variation.

 

-----

 

5. Describing Network Requirements

 

Applications do of course have throughput requirements, and thus a complete framework for application-level network quality must also take capacity into account.

 

[RG] Please clarify, whether this statement refers to throughput or capacity. Capacity impacts throughput in the case of congestion, but they aren’t the same. Throughput may also depend on the bandwidth limitation. A policer is a worst case for TCP, a port with a WRED drop-profile might result in a different throughput, even if both offer the same nominal bandwidth.

 

----

 

5. Describing Network Requirements

 

[RG] Please reword: To do that it is necessary to make articulating the network requirements a little bit more complicated.

 

----

 

5. Describing Network Requirements

 

Where the NRPoU percentiles and NRP

 

[RG] Please inform the reader, where the new per App threshold values NRPoU percentiles and NRP are originating or determined.

 

----

 

8. How to find network requirements

 

[RG] Your discussion is comprehensible, but lacks some important aspects, which I think need to be clear to a reader: Who is expected to execute this characterization? Are results to be published? Are the networking conditions to derive these parameters to be published?

 

----

 

11.1. Missing Temporal Information in Distributions.

 

These two latency series: … will have identical distributions, but may have different application performance. Ignoring this information is a tradeoff between simplicity and precision. 

 

[RG] I think (exactly) this problem is solved if QoO incorporates more and suitable QoS metrics, like IPDV.

 

-----

 

11.3. Assuming Linear Relationship between Perfect and Unusable (and that it is not really a probability)

 

[RG] Assuming this linearity is introducing an error. From what I know from ITU-T SG12, QoE correlations to performance measurements can be mathematically demanding. Please read into ITU-T G.1051 , Modelling approach for perceived interactivity of interactive applications to learn more about some math applied there.

 

I quote:
“The basic assumption of modelling perceived interactivity is its monotonous dependency on data latency. The shorter the data transport time is, the shorter the response time in an interactive application is and the more interactive the use of the application is perceived to be. 

However, this dependency is not a simple linear function; rather there are saturation areas at both tails of the function, where no further change in perception happens even if the latency changes.”

 

-----

 

11.4. Binary Bandwidth threshold

 

[RG] Your judgement is correct. You move closer to ITU-T SG12… Codec, FPS, version of the streaming software, transport used, kind of content, error concealment and so on have an impact on QoE, not just the network or capacity.

 

----

 

 

 

Von: Marcus Ihlar <marcus.ihlar=40ericsson.com@dmarc.ietf.org> 
Gesendet: Donnerstag, 12. Juni 2025 15:42
An: IETF IPPM WG (ippm@ietf.org) <ippm@ietf.org>
Betreff: [ippm] Working group last call for Quality of Outcome (draft-ietf-ippm-qoo)

 

Hello IPPM,

 

This email initiates the working group last call for Quality of Outcome (draft-ietf-ippm-qoo).

 

The current version of the document can be found here:

https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/

https://www.ietf.org/archive/id/draft-ietf-ippm-qoo-03.html

 

Please review the document and reply to this email with any comments and indicate whether you believe it is ready for publication. 

This WGLC will last for three weeks and end on Thursday July 3, 2025.

 

BR,

Marcus and Thomas

 

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.