Return-Path: X-Original-To: ippm@mail2.ietf.org Delivered-To: ippm@mail2.ietf.org Received: from localhost (localhost [127.0.0.1]) by mail2.ietf.org (Postfix) with ESMTP id EB14E3FCC372; Mon, 7 Jul 2025 05:58:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at ietf.org X-Spam-Flag: NO X-Spam-Score: -1.846 X-Spam-Level: X-Spam-Status: No, score=-1.846 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.232, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, T_REMOTE_IMAGE=0.01] autolearn=unavailable autolearn_force=no Authentication-Results: mail2.ietf.org (amavisd-new); dkim=pass (1024-bit key) header.d=cujo.com Received: from mail2.ietf.org ([166.84.6.31]) by localhost (mail2.ietf.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MrZ3GgwyE-jt; Mon, 7 Jul 2025 05:58:30 -0700 (PDT) Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazon11021116.outbound.protection.outlook.com [52.101.70.116]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail2.ietf.org (Postfix) with ESMTPS id 99C4E3FCC335; Mon, 7 Jul 2025 05:58:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=W5zkqbVyDJyulmDxTQvqUfukF0hESK5N0WcHoXZz+Vn3tjtza6Gp/5qyiLsn045eVoKHy86/dJgbpQQxGlbWl/UWPfEui5o1U11a/kdYASo3aWg8NJv+YPsYMk9s5AjY1KmAq5bEtx31oEmd+B2knCqY7SoR9EFVyvSSDVgdNrfKhNf3G3ynb4+mJfYlxqOH6zzht/BfnkMv98KUv8FuH8kLTEFccnWji2pHN+sSM5AR17mf+68S/W30Rdc7ZgQZx1PEsBBoh/z3jidtcivlKE08HxYRPSg+gKbIcPDg3LwCM71WTDi7hEHJRWTBKNnn5hx8rDW0utLRUYvLB6/+pQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E6eh8PO5rdPinz/2eYQqh7d4rWw+HZnK7Jloi+KjZCw=; b=bTUIOMHr3cW1LCMxSDPui8oi8XD+2Xw6NNwJOUv05QSVsUTQwDjJsyqy2sG+PgrOrDvGWcZ3KSR0PyHMxPQZ9ffki5APq3JZWs48RTw1QAzsZHgtYgOgJpuD4aw+K1NYriCdfcKql+UW/tY2tvs0S4ZAqhk+q6hvz1FgPN7V3WOelCPKMiFU0wCdXmsl+P8+wAjgY047pbi9W3IwFImq/xExuzTe0IebLRwzppGDMDn/KEc1QhPuP4WRaIcUBMt+DnnqjKItAxcIu1SLXBQjsPfDwnGIo8uY9IHftc3EosIpqZoRHCgM+5F3erbXJuSqmkvo2Nx6AAsq36SsfsCjBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cujo.com; dmarc=pass action=none header.from=cujo.com; dkim=pass header.d=cujo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cujo.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E6eh8PO5rdPinz/2eYQqh7d4rWw+HZnK7Jloi+KjZCw=; b=rOqW4Gu7tsUXfL0QRv4aQwBwIN33N5none8TyNQ9eF/GH6Q9BwMtN6krLBuXWYIuE1WuX2CSrqNvjJE9QnzZPMGQOGNToTxgOUHQfstm5LIkxlju3PV3PVAdB0K2zmjAEplP/nZXUcezIT3TpSIr4M2ZmybmSUz6BFp4EaQ4TJs= Received: from GVXPR02MB10560.eurprd02.prod.outlook.com (2603:10a6:150:155::10) by DBAPR02MB6136.eurprd02.prod.outlook.com (2603:10a6:10:184::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8901.26; Mon, 7 Jul 2025 12:58:24 +0000 Received: from GVXPR02MB10560.eurprd02.prod.outlook.com ([fe80::5362:be3f:3397:38eb]) by GVXPR02MB10560.eurprd02.prod.outlook.com ([fe80::5362:be3f:3397:38eb%3]) with mapi id 15.20.8901.021; Mon, 7 Jul 2025 12:58:24 +0000 From: Bjorn Ivar Teigen Monclair To: "ippm@ietf.org" , IPPM Chairs Thread-Topic: Working group last call for Quality of Outcome (draft-ietf-ippm-qoo) Thread-Index: Advbn6X/VqbW5OwTT0OdZ8Dt/vRBWwO74zMwAF92NUIAPCnE7wCFBmbw Date: Mon, 7 Jul 2025 12:58:24 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=cujo.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: GVXPR02MB10560:EE_|DBAPR02MB6136:EE_ x-ms-office365-filtering-correlation-id: 30dad41b-f619-4f6c-0de3-08ddbd55f55e x-ld-processed: a5d2f1ce-8ca6-49ce-893a-b4f922d7fc6b,ExtAddr x-ms-exchange-atpmessageproperties: SA x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|366016|69100299015|39142699007|1800799024|31052699007|376014|13003099007|8096899003|38070700018; x-microsoft-antispam-message-info: =?Windows-1252?Q?6gRrijNzLr+ufJ9CU6W9OTjM2dtZEH/voyIwiWrKD+RmagkNjR9370i2?= =?Windows-1252?Q?QfeWPjO9VuhP3whUxWk2KDqHgbk7EzR2Rn3OdI8APt12pf7wwiegVlF9?= =?Windows-1252?Q?X60S3DRSsjwdOdSTeUIJmFhC05OrmKBL41MIwc3i5+KvTWkGJDTV+L93?= =?Windows-1252?Q?hIF6GwlEpDXtzHmtI1XnTPW1Iuc8jlDRJxeLfRlOjgzTapE3VdzglOYG?= =?Windows-1252?Q?J9BXMb5NRvjuI9/bVy2Cs3JvgS5aLWyg23IZ/uBHnBOb4av8P3fNtcdM?= =?Windows-1252?Q?hXac4rGjgDAMo4ugcxWOjVpw/8FFA5NZpi7UNUSz+YWsd5o88Z/03xgv?= =?Windows-1252?Q?PnwrHASqf6sBMhKZ9GyNyZxP7Cq2HJgVmo2GEI2KghKKnirIeRSMTLRp?= =?Windows-1252?Q?eqtqvXtcnoTF/Jz16cd8TyFddyoY9AhSsYJ0fBzucpwZ+J8z9rdsBiL4?= =?Windows-1252?Q?Na7+zzCouGCda2q35v7h5zjhKRz9p5Ju+itF/hyAUASi5W0EbeT85o1P?= =?Windows-1252?Q?YXv+NvrkEwpP7DuMxvLXoJw4X1bLsjHzDpEKUu/HBMC0vsxax+45ZnYy?= =?Windows-1252?Q?b2QJDuSycMrGrG5bymlymGGJi1FMH51eRYHOkxDGNeSdBBFlE4QzCLDj?= =?Windows-1252?Q?E/tFu6RrkbXe0CMMu/ua4cWb/BOykre8THMx7TRoG9RmQMLOjeW9W9Hg?= =?Windows-1252?Q?iQdc0f/vN3jeMsFou9A7Vx4/0PiiYcEg6+HNQ2REAWMY/h557n9qn4PG?= =?Windows-1252?Q?i7L4iNnDbo+rJRp4M+ghq06vWSBYxKAIBc8KdOHr35BNxbl9pakVXscZ?= =?Windows-1252?Q?IFm2y28zNyArYsXtEIvNmpHOQ3V2a29uzT5F6m5Bb8B3UJgUdAmSKJRT?= =?Windows-1252?Q?Cc4J5ZcVrYG5qAxLsj7YjJIHwoQO0A7fGx5N/KAIchrNI7uOfdV5lG6J?= =?Windows-1252?Q?GuDBmBfSKv6rOgSV4nR+MPkkrV0f3fuS0EfUmNDRYhhEbcRURDktJujW?= =?Windows-1252?Q?2wBO4325nmWscXvOLQaYWCq3Jv9Avv0idTs/IrM5o0k6gq7wAEXN0yvs?= =?Windows-1252?Q?E+g7yo8Y3zs3VLIlCVqjyLNFuBmu7O6hKEwR3II+sS44YpRMi4EZ9Nv0?= =?Windows-1252?Q?R3SxkVpPMTOnINUFNIxAXNL3No0WfBfjxR6Yy2D/wridx/wURQcL9L9r?= =?Windows-1252?Q?EztpMb2iQIpC/n6d7ItUEM/sv4b7cpKH3+yAy2B2Ge/zexszI65IlJKe?= =?Windows-1252?Q?JElD+rWoP8fhBPxYYe3Z/NZmnc2o2HJjFx8E7mTDKCsuCAdsDghc9pWT?= =?Windows-1252?Q?MI3OrduZYTe9OQE7aMPJbLRswjRCZ3U+ikA6H/D5dn/sO2XrynTNX+Cq?= =?Windows-1252?Q?T6oCHf+FdARqtd96y93Cc6U/aRUCCI4oQzXTW82mPoZGN/pRQ2lGhXrt?= =?Windows-1252?Q?pbfzGNlch3KTFEf+tpEu3g2fr1OCbyXNO7/jSfonE6qcnMC2RYc8Jf/W?= =?Windows-1252?Q?Okcs6tb1j+LR+MPsauDgx4L4szKJr2gor3W+avSKY7ssFJ6N4u8=3D?= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:GVXPR02MB10560.eurprd02.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(69100299015)(39142699007)(1800799024)(31052699007)(376014)(13003099007)(8096899003)(38070700018);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?cPxztMU8452bcApIO6hQWBjAohydjgT0N3eoSGOc+H64aEdrxccLKUod?= =?Windows-1252?Q?7YEyKCoBa0vSIzsdFhb2qPvdl5BAqutYmMSVc1WmVZvHqAt1uL+d9zAn?= =?Windows-1252?Q?Ef/qCu+VdyRz+wwmcBG0AnQQZ2/INkx8eqPVfHgXakXsoNSbpVGYyQRq?= =?Windows-1252?Q?Vkuyz1hhUB2kxkeplRcV96lmzPnm8q1bOONgpQLBERvBLBsyPqKN4SWi?= =?Windows-1252?Q?PV87MWBd+FGR+Xk98IAjQOCSPKNLyrM7v1jrnkgJQ+omILzUjurfGJJM?= =?Windows-1252?Q?XUAm3wdNYfnKzblFqjz5zltvrpsbuewGQfDVASqWBOgLntjXb4MQS3Jh?= =?Windows-1252?Q?5iYv2zTJochxWxj/EuMc0AKglFRbPFW1BwxVv/DsJzkPH80Y8XLLVH9X?= =?Windows-1252?Q?awy4hXkIg8xXfxRYpiMUen39a+Hz+z9jLbuVNsCcumuD2aeqhWzz+QAE?= =?Windows-1252?Q?pVGkKz64dH+qPLnHdfAYh3fNTXjXLOQIvM0zvV57wpz5Jf1L675WdcR+?= =?Windows-1252?Q?NLc1aDKwA0RgjcCN/u1sdvRlEpLGTa3EgPeK6oDz9bzWb2cwqR/E9D+I?= =?Windows-1252?Q?788J/1MHCrEYsCi8XZL4cw2vr7af7HSQJ3RDvVJRn9mWhviQhBjGLUFn?= =?Windows-1252?Q?SlAJGO0d/wD7ogw+tjHyABwqUmTC20zZDdf1odWeEXhKyaNdNOIl1gBq?= =?Windows-1252?Q?CRGcp+NUNFaIxewRQy3/Y7cdCUZyMoeF4C0iNumeAayZkl+iJjch46WQ?= =?Windows-1252?Q?J4B7VSWbZkr3hGg892cdOG7TD9HnLVtZ8KOcYv3LYW83q/ejCSE3agP9?= =?Windows-1252?Q?WKZGcFkrYnQJKrC42/ZuW7ulfI4zmCaW6R6ldE9nJFBGt5YZ8xoXFTAT?= =?Windows-1252?Q?MwNRv1wB8vp8Jh1oMF+q2+YQpiYnYLerpq7BxQV92A7QlO7qsN/qAANE?= =?Windows-1252?Q?BOrEejaUGVzVoBgX1Jr7ITJ5rONLI3+K5NXvMmsteizYP4d6McGnEHct?= =?Windows-1252?Q?rv0OjhV6pmLUk3yQLiphDtexBQ2iqQvPI5i91mv/3dl8LwnJL7oFA+1X?= =?Windows-1252?Q?GLvYZyI7J8rtVXm+RxoIzPdoJmT6A/5zOcCS5AnZf8UCfQLwwJL1w24e?= =?Windows-1252?Q?G8Ld+00XP3F0yskQNZdbsIL4TsplmWlzNMLD28EEExl+lH55SO2tt5Hb?= =?Windows-1252?Q?CKHaKLFWS5n3rwdSXRlsm3S6KOa4iMaR+v9Ffs0MJNysDuN9UtNa/vbM?= =?Windows-1252?Q?6MuJ2RP5nfpGMQPyJ/L5zefrXdYLIpp0orQhRZzWRBl6AwKAIvV5ulLv?= =?Windows-1252?Q?hs8Az06yectlXPeLpRT5PUlITh82QDzX3T+x0ekMoclCSDBttounES2q?= =?Windows-1252?Q?HVw5bcOQvvGn7sb1fYrTg4ajTA2hu/6sISsKMEl0Y+oySoz41klex+Od?= =?Windows-1252?Q?wIkyxjWbe7JfH1p8t6VQLsdLQnz36URb9aER4ewNPnkMf8O286OrLJeB?= =?Windows-1252?Q?1aG2IXJkQM/pk2ISIoUBJkXMuST/WFusGhPgrteF+D6tVy/K5OR0b1Rf?= =?Windows-1252?Q?IxmWC70BPVA6yPwNYCU66b8l9T2jBq+19oEyMmg4BvPsluzY9uKfTcmT?= =?Windows-1252?Q?p6TU9Otz5az7gL7rufsaxl89QaUw8PCfolnnwbn5cjC48eYnaKzeVQLH?= =?Windows-1252?Q?2nw2/SdVx6sPJZMRCl6C77EDVYL43PND?= Content-Type: multipart/alternative; boundary="_000_GVXPR02MB105608D29859C95E32F8EEE78EA4FAGVXPR02MB10560eu_" MIME-Version: 1.0 X-OriginatorOrg: cujo.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: GVXPR02MB10560.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 30dad41b-f619-4f6c-0de3-08ddbd55f55e X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Jul 2025 12:58:24.2455 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a5d2f1ce-8ca6-49ce-893a-b4f922d7fc6b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: rFQACjNBCMZygXmDYsdH5DkMdSGfTn9xRUu5JHbD0pXI5OcKutIBBz+o1qIDIWDyoGhTHHPX/DKLPEzP8GKplw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBAPR02MB6136 Message-ID-Hash: AJQHOIDHDKCL7UDCJLFQ5Q4Q6DHVBLF4 X-Message-ID-Hash: AJQHOIDHDKCL7UDCJLFQ5Q4Q6DHVBLF4 X-MailFrom: bjorn.monclair@cujo.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-ippm.ietf.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Magnus Olden , "Ruediger.Geib@telekom.de" , "Jason Livingood (Comcast)" , "giuseppe.fioccola=40huawei.com@dmarc.ietf.org" , "ike.kunze@comsys.rwth-aachen.de" X-Mailman-Version: 3.3.9rc6 Precedence: list Subject: =?utf-8?q?=5Bippm=5D_Re=3A_Working_group_last_call_for_Quality_of_Outcome_?= =?utf-8?q?=28draft-ietf-ippm-qoo=29?= List-Id: IETF IP Performance Metrics Working Group Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --_000_GVXPR02MB105608D29859C95E32F8EEE78EA4FAGVXPR02MB10560eu_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Dear IPPM Working Group, On Friday I did not have time to detail the changes made. Sorry about that.= Here is the complete breakdown: ## Giuseppe's Comments > * In the document there are new terms introduced (e.g. Quality Attenuatio= n, QoO score, etc). I think they can be all defined in a terminology sectio= n. [Bj=F8rn] Added a Terminology section (Section 3) with definitions for Net= work, Quality Attenuation, Quality of Outcome (QoO), QoO Score, NRP, NRPoU,= Composability, and Accuracy/Precision. > * In section 3 on Background, several techniques are listed and I suggest= to also add other IPPM hybrid techniques, such as the On-Path Telemetry me= thods (i.e. IOAM, AltMark). [Bj=F8rn] Added "On-Path Telemetry methods (IOAM, AltMark)" to the list of = relevant techniques in the Background section. > * The overall organization of the document should be revised and improved= . For example section 4, section 5 and section 6 can become subsections of = a new section on Sampling and Network Requirements and the same applies to = section 9 and section 10, that can be put under a new section on Insights. [Bj=F8rn] Restructured document with consolidated "Sampling and Network Req= uirements" section and "Insights" section as suggested. > * Note that the current section 13 on "Conventions and Definitions" shoul= d be moved earlier in the document since it is usually placed after the Int= roduction. [Bj=F8rn] Removed "Conventions and Definitions" since no reference to RFC21= 19 is needed. ## Ruediger's Comments > [RG] The document frequently refers to "network" as to be characterized. = Yet I failed to find a definition or a reference to a definition. Please cl= arify, what you mean by "network". [Bj=F8rn] Added definition of "Network" in the Terminology section: "the co= mmunication infrastructure that connects endpoints, including all intermedi= ate devices, links, and protocols that affect the transmission of data betw= een a source and destination." > ?[RG] I note, that =93network operators=94 have little incentive=85in one= statement. Does this express, that a consumer will reliably buy the best p= erforming Home Gateway to optimize his QoE by minimizing the effect of the = home network on their app? [Bj=F8rn] I assume some well-informed consumers will act in that way, yes, = but certainly not all of them. I believe there are two mechanisms at work = here - one is existing consumer demand for a home network that "just works"= , and the other is ISPs strategically choosing to optimize for application = performance in anticipation of a consumer shift away from a pure focus on c= apacity. > Section 2: > > Existing network quality metrics and frameworks typically address the nee= ds of one or two of these stakeholders, > > [RG] Please provide references for existing quality metrics & frameworks = addressing the needs of one or two of these stakeholders only. [Bj=F8rn] Added context and references discussing how existing metrics (thr= oughput for regulatory compliance, QoE metrics for users) typically serve l= imited stakeholder groups. > Section 2: > > While solutions exist for many of the problems causing high and unstable = latency in the Internet, the incentives to deploy them have remained relati= vely weak. > [RG] Please provide references for both, "problems" and "incentives". [Bj=F8rn] Added references to bufferbloat mitigation techniques (RFC8290), = congestion control algorithms (RFC8033), and BITAG latency study to support= claims about deployment incentives. > Section 2 > > operators have little incentive to improve network quality beyond increas= ing throughput. Despite the availability of open-source solutions, vendors = rarely implement them. The root cause lies in the absence of a > universall= y accepted network quality framework that captures how well applications ar= e likely to perform. > [RG] Do you mean throughput or bandwidth? [Bj=F8rn] Changed to "have little incentive to improve network quality beyo= nd increasing bandwidth." > ?[RG] I=92m not sure, whether the root cause for the claimed unwillingnes= s of vendors to implement open source solutions is a missing network qualit= y framework. Can you provide some proof or a reference for your claim? Furt= her, your proposal is to characterize two thresholds (perfect and stop of a= pp) and interpolate linearly between these, based on a subset of performanc= e metrics. You apply a set of simplification, rendering the diagnosis less = reliable. The dependency on app developer publications isn=92t increasing t= he quality of diagnosis too, unless a standardized testing environment & co= nditions is the basis for these publications. [Bj=F8rn] It's hard to really prove that statement, but I do think it's poi= nting in the direction of truth. I've added references and softened the lan= guage a bit. > [RG] Please reword: > =93A universally accepted network quality framework that successfully cap= tures how well applications are likely to perform may help to increase the = willingness of vendors to=85.=94 [Bj=F8rn] Clarified terminology and softened claims with "may help to incre= ase the willingness of vendors to implement such solutions". > Section 2 > > Therefore, one critical requirement of a meaningful framework is its abil= ity to answer the question, "Will an application work properly?" > > > [RG] To me a framework is a starting point and a standard an end. To say,= the standard would offer the "ability". Please reword: =85 is to enable wo= rk on a standard answering the question, "Will networking conditions stop a= n application from working properly?" [Bj=F8rn] Reworded to focus on the framework's ability to answer the questi= on about networking conditions negatively affecting applications. >Section 2 > > Answering this question requires several considerations. First, the Inter= net is inherently stochastic from the perspective of any given client, so c= ertainty i unattainable. > > > [RG] Please correct/reword (would that be an "is"?) [Bj=F8rn] Fixed "certainty i unattainable" to "absolute certainty is unatta= inable." > [RG] Please reword: To summarize, the framework and "meaningful QoO metri= c" should have the following properties: Capture a set of network performan= ce metrics which provably correlate to the application quality of a set of = different applications as perceived by users. [Bj=F8rn] Updated requirements section with the suggested more precise word= ing. > [RG] Please add a ref to https://www.rfc-editor.org/rfc/rfc7680.html#sect= ion-2.8.2 and discuss the novelty of TR-452.1 as compared to the RFC 7680 l= oss specification. [Bj=F8rn] Added RFC 7680 reference and discussion: "Similar to the One-Way = Loss Metric for IP Performance Metrics (IPPM) [RFC7680], which defines pack= et loss in terms of packets that fail to arrive within a specified time thr= eshold, the Quality Attenuation approach extends this concept..." > [RG] Please inform the reader, where the new per App threshold values NRP= oU percentiles and NRP are originating or determined. [Bj=F8rn] Added section on standardized testing conditions and requirement = for application developers to publish their testing methodologies. > [RG] I'd prefer references for the performance figures. [Bj=F8rn] Added JamKazam reference for remote music collaboration latency r= equirements and updated the example to be more specific. ## Luis's Comment > One question / comment form my side. It is not clear to me what could be = the behavior of the QoO with adaptive applications that can accommodate e.g= . the bit rate to the network conditions. [Bj=F8rn] Added "Adaptive Applications" section explaining how QoO can acc= ommodate applications with multiple quality levels through multiple NRP thr= esholds or simplified single-threshold approaches. ## Greg's Comments > "Mean" and "average" are used in the document almost equal number of time= s. However, RFC 6049 uses only "mean" in reference to composable spatial pe= rformance metric. [Bj=F8rn] Standardized terminology to use "mean" consistently throughout th= e document. > "Accuracy" and "precision" are used in the document extensively, sometime= s even in the same sentence. In general, accuracy can be interpreted as pre= cision. [Bj=F8rn] Added definitions to Terminology section clarifying that "accurac= y" refers to how close measurements are to the true value, while "precision= " refers to consistency and repeatability, noting they are not interchangea= ble. > A nit. I think that we use "Quality Attenuation" as a term. In that case,= it must be uppercased throughout the document. [Bj=F8rn] Standardized capitalization of "Quality Attenuation" throughout t= he document. > In the Motivation s/i unattainable/is unattainable/ [Bj=F8rn] Fixed typo. ## Ike's Comments > "Quality attenuation is a network quality metric that meets most of the c= riteria set out in the requirements" - From this text alone, it is not 100%= clear which requirements are referred to here [Bj=F8rn] Changed to "meets most of the design goals set out in the require= ments section of this document" for clarity. > Maybe make it explicit? For example: "However, interpreting raw quality a= ttenuation values is difficult, raising the general challenge of how to sim= plify the entailed information without losing too much in terms of precisio= n and accuracy." [Bj=F8rn] Added explicit statement about the difficulty of interpreting raw= Quality Attenuation values. > 's/certainty/absolute certainty/'? [Bj=F8rn] Changed to "absolute certainty is unattainable." > 's/accommodate diverse application requirements/such diverse application = requirements'? [Bj=F8rn] Updated > 's/The overall goal/The overall goal of this document'? [Bj=F8rn] Updated. > "s/OSI layer/layer in the protocol stack" [Bj=F8rn] Changed to "applied across different layers of the protocol stack= and is network technology independent." > The formatting of the start of the paragraphs starting with "Latency Comp= onent", "Packet Loss Component", and "Final QoO Calculation" looks a bit bu= mpy. [Bj=F8rn] Improved formatting of the calculation examples for better readab= ility. > There are "step-by-step" guides at various points in the document which I= like. However, they vary in the concrete format [Bj=F8rn] Standardized formatting of step-by-step guides throughout the doc= ument. > "These two latency series: 1,200,1,200,1,200,1,200,1,200 and 1,1,1,1,1,20= 0,200,200,200,200 will have identical distributions" [Bj=F8rn] Reformatted as: "The two latency series (1,200,1,200,1,200,1,200,= 1,200) and (1,1,1,1,1,200,200,200,200,200) have identical distributions" > Inconsistent use of end users vs. end-users [Bj=F8rn] Standardized to "end-users" throughout the document. [Bj=F8rn] I also added an Overview section (Section 2) to help readers quic= kly understand the key concepts before diving into detailed specifications,= addressing the general concern about document accessibility. The updated draft is available at: https://datatracker.ietf.org/doc/draft-i= etf-ippm-qoo/ [https://static.ietf.org/dt/12.43.0/ietf/images/ietf-logo-card.png] draft-ietf-ippm-qoo-04 - Quality of Outcome This document introduces the Quality of Outcome (QoO) framework, a novel ap= proach to network quality assessment designed to align with the needs of ap= plication developers, users, and operators. By leveraging the Quality Atten= uation metric, QoO provides a unified method for defining and evaluating ap= plication-specific network requirements while ensuring actionable insights = for network ... datatracker.ietf.org We believe these changes significantly improve the document's clarity, comp= leteness, and technical accuracy while addressing all substantive comments = from the WGLC review. Best regards, Bj=F8rn Ivar Teigen Monclair ________________________________ From: Bjorn Ivar Teigen Monclair Sent: Friday, July 4, 2025 6:11 PM To: ippm@ietf.org ; IPPM Chairs Cc: Magnus Olden ; Ruediger.Geib@telekom.de ; Jason Livingood (Comcast) = ; Michael Welzl ; LUIS MIGUEL CONTRERAS MURILLO ; Greg Mirsky ;= giuseppe.fioccola=3D40huawei.com@dmarc.ietf.org ; ike.kunze@comsys.rwth-aachen.de Subject: Re: Working group last call for Quality of Outcome (draft-ietf-ipp= m-qoo) Hi IPPM, A new version of the draft, draft-ietf-ippm-qoo-04, has now been uploaded t= o the data tracker: https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/ It addresses, to the extent we have been capable of doing so, all of the su= ggestions and issues raised since the WGLC was announced. Thanks again to everyone who has contributed to the document, your time and= effort is greatly appreciated. Cheers, Bj=F8rn ________________________________ From: Bjorn Ivar Teigen Monclair Sent: Thursday, July 3, 2025 1:34 PM To: ippm@ietf.org ; IPPM Chairs Cc: Magnus Olden ; Ruediger.Geib@telekom.de ; Jason Livingood (Comcast) = ; Michael Welzl ; Ruediger.Geib@telekom.de ; LUIS MIGUEL CONTRERAS MURILLO ; Greg Mirsky ; giuseppe.fioccola=3D4= 0huawei.com@dmarc.ietf.org ; ike.kunze@comsys.rwth-aachen.de Subject: Re: Working group last call for Quality of Outcome (draft-ietf-ipp= m-qoo) Hi Jason, Michael, Ike, Guiseppe, Greg, Luis, Ruediger, IPPM chairs and par= ticipants, Thank you very much for the detailed feedback! Greatly appreciate the time = you've all invested in this. On restructuring: I see how the document would benefit from restructuring in the way Ike sugg= ests. If restructuring can be done without causing a lot of delay (and work= !), I am happy to submit a new version along the suggested lines. If restru= cturing would require significant delays and reviews (or, if others disagre= e about the need for restructuring), then perhaps adding a new sub-section = to the introduction is the better option? Chairs and seasoned IETF'ers, I h= ope you can advice. I will take all of your feedback into account and produce a new version of= the document before the deadline on the 7th. Cheers, Bj=F8rn ________________________________ From: Ruediger.Geib@telekom.de Sent: Wednesday, July 2, 2025 5:20 PM To: Magnus Olden ; Bjorn Ivar Teigen Monclair Cc: ippm@ietf.org Subject: AW: Working group last call for Quality of Outcome (draft-ietf-ipp= m-qoo) CAUTION: This email originated from outside of the organization. Do not cli= ck links or open attachments unless you recognize the sender and know the c= ontent is safe. Hi Bj=F8rn, hi Magnus, Thanks for your framework for multi-dimensional metric linking delay and pa= cket loss. Please find some [RG] marked comments below. Please note, that I=92m off for vacations and likely won=92t respond to rep= lies before early August. I=92m sorry for that. Regards, Ruediger [RG] The document frequently refers to =93network=94 as to be characterized= . Yet I failed to find a definition or a reference to a definition. Please = clarify, what you mean by =93network=94. I note, that =93network operators=94 have little incentive=85in one stateme= nt. Does this express, that a consumer will reliably buy the best performin= g Home Gateway to optimize his QoE by minimizing the effect of the home net= work on their app? Another general comment, without being a native speaker, maybe =93comprehen= sible=94 rather than =93understandable=94? Could some native speaker confir= m/reject that? ---- Section 2: Existing network quality metrics and frameworks typically address the needs= of one or two of these stakeholders, [RG] Please provide references for existing quality metrics & frameworks ad= dressing the needs of one or two of these stakeholders only. ---- Section 2: While solutions exist for many of the problems causing high and unstable la= tency in the Internet, the incentives to deploy them have remained relative= ly weak. [RG] Please provide references for both, =93problems=94 and =93incentives= =94. ---- Section 2 operators have little incentive to improve network quality beyond increasin= g throughput. Despite the availability of open-source solutions, vendors ra= rely implement them. The root cause lies in the absence of a universally ac= cepted network quality framework that captures how well applications are li= kely to perform. [RG] Do you mean throughput or bandwidth? Consumers may receive offers of a= ccess types with higher bandwidth. Above a minimum access bandwidth, furthe= r increase of bandwidth doesn=92t increase per app throughput. A suitable l= ocation of content and service servers would. [RG] I=92m not sure, whether the root cause for the claimed unwillingness o= f vendors to implement open source solutions is a missing network quality f= ramework. Can you provide some proof or a reference for your claim? Further= , your proposal is to characterize two thresholds (perfect and stop of app)= and interpolate linearly between these, based on a subset of performance m= etrics. You apply a set of simplification, rendering the diagnosis less rel= iable. The dependency on app developer publications isn=92t increasing the = quality of diagnosis too, unless a standardized testing environment & condi= tions is the basis for these publications. Please reword: =93A universally accepted network quality framework that successfully captu= res how well applications are likely to perform may help to increase the wi= llingness of vendors to=85.=94 ---- Section 2 Therefore, one critical requirement of a meaningful framework is its abilit= y to answer the question, "Will an application work properly?" [RG] To me a framework is a starting point and a standard an end. To say, t= he standard would offer the =93ability=94. Please reword: =85 is to enable work on a standard answering the question, "Will networkin= g conditions stop an application from working properly?" ---- Section 2 Answering this question requires several considerations. First, the Interne= t is inherently stochastic from the perspective of any given client, so cer= tainty i unattainable. [RG] Please correct/reword (would that be an =93is=94?) ---- 2.2. Requirements Flexibility in describing application requirements and the ability to captu= re the delay characteristics of the network in sufficient detail are necess= ary to compute application success with satisfactory accuracy and precision= . [RG] Please specify =93satisfactory accuracy and precision=94 or provide a = reference. Please reword: necessary to improve accuracy and / or precision of=85 [RG] Please define what you mean by =93to compute application success=94 an= d how that is linked to the QoO network score (the framework set out to des= cribe the latter). ---- 2.2. Requirements To summarize, the framework and "meaningful metric" should have the followi= ng properties: Capture the information necessary to compute the probability that applicati= ons will work well. (Useful for end-users and application developers.) [RG] I=92d prefer rewording to To summarize, the framework and "meaningful QoO metric" should have the fol= lowing properties: Capture a set of network performance metrics which provably correlate to th= e application quality of a set of different applications as perceived by us= ers. ---- 2.2. Requirements Compare meaningfully to different application requirements. [RG] Please reword to be more specific on what you mean. Mathematically, an= d if you=92d like to compute something meaningful, QoO score and app QoE mu= st be correlated, even if you don=92t go for MOS. If this is captured by my= reworded version of the first requirement, then that=92s it. If the above = second requirement is that the function individual_app_quality (QoE score) = needs be known, then clearly say that. As far as I understand your draft, t= his function is out of scope. ---- 2.2. Requirements Compose. [RG] Should that be de-compose and isolate? So far the text didn=92t explai= n that the QoO score is calculated network-section-wise and finally compose= d. ---- 2.2.1. Requirements for end-users The quality framework should facilitate a metric that is objective, relatab= le, and relatively understandable [RG] please reword to =85. metric that is based on objective QoS measurements, correlated to appl= ication quality and relatively understandable ---- 2.2.1. Requirements for end-users If these requirements are met, the end-user can understand if a network can= reliably deliver what they care about: the outcomes of applications. [RG] I=92d prefer a more defensive way of putting it, but the statement doe= sn=92t say, the network is the only cause of bad app outcome. My preference: =85 can understand if a network can is a likely source of impairment for wh= at they care about: the outcomes of applications. ---- 2.2.1. Requirements for end-users A compromise is for the quality of experience framework to place the respon= sibility for sourcing and representing end-user requirements onto the appli= cation developer. [RG] If I understand this statement correctly, you put crating input for th= e second requirement of section 2.2 above out of scope of this framework. P= lease do clearly state that in the first instance where his is mentioned. S= uggestion: Compare meaningfully to different application requirements. Note that provi= ding the relevant information depends on application developers publication= s or standards, where available. [RG] Further, this statement and the remaining content define requirements = from app developers. Please move to section 2.2.2 ---- 2.2.1. Requirements for end-users Some real world examples where 'acceptable levels' have been derived by app= lication developers include (note: developers of similar applications may h= ave arrived at different figures): Remote music collaboration: 28ms latency note-to-ear for direct monitoring,= <2ms jitter Online gaming: 6Mb/s downlink throughput and 30ms RTT to join a multiplayer= game Virtual reality: <20ms RTT from head motion to rendered update in VR Performing this UAT helps the developer understand what likelihood a new en= d-user has of an acceptable Quality of Experience based on the application'= s existing requirements towards the network. [RG] I=92d prefer references for the performance figures. ---- 2.2.2. Requirements from Application and Platform Developers [RG] From my point of view asking for useful UAT requirement publication fr= om App developers requires standard testing conditions (testbed, config and= path, forwarding & load characteristics). This might be read into the para= graph as is, but it should be explicitely expressed, whether or not such st= andard conditions for app characterization are required by this section, or= not. I don=92t know the authors preference, so I don=92t propose text. As a thought, if app developers derive data under different networking cond= itions, the overall accuracy and precision of the QoO metric will be reduce= d. ---- 3. Background [RG] Please move the text of =933.1.8. Quality Attenuation=94 to =93Backgro= und=94, as this seems to indicate latency and loss as two metrics required.= Please be also more precise, as to what Quality Attenuation captures regar= ding latency distribution. I=92m wooried, as you address Average and 99th p= ercentile latency by separate sections. 3. Background The novelty of the Quality Attenuation metric is to view packet loss as inf= inite (or too late to be of use e.g. > 3 seconds) latency [TR-452.1]. [RG] Please add a ref to https://www.rfc-editor.org/rfc/rfc7680.html#sectio= n-2.8.2 and discuss the novelty of TR-452.1 as compared to the RFC 7680 los= s specification. You may also reword to: =93Similar, as the One-Way Loss Metric for IP Performance Metrics (IPPM) [R= FC 7680], the Quality Attenuation metric is to view=85 ---- 3. Background All applications require a minumum level of throughput and a maximum packet= loss rate. [RG] Please reword to ..and work acceptable up to a maximum=85 ---- 3.1. Discussion of other performance metrics [RG] Which metrics have been discussed prior to the =93other performance me= trics=94. If there=92s no clear text to that, please ----- 3.1.3. Variance of latency [RG] Adding the variance of random processes is producing correct results i= s producing correct results, if these are not correlated. Please add a stat= ement. Please provide a reference to a metric definition for Latency Variat= ion. ----- 5. Describing Network Requirements Applications do of course have throughput requirements, and thus a complete= framework for application-level network quality must also take capacity in= to account. [RG] Please clarify, whether this statement refers to throughput or capacit= y. Capacity impacts throughput in the case of congestion, but they aren=92t= the same. Throughput may also depend on the bandwidth limitation. A police= r is a worst case for TCP, a port with a WRED drop-profile might result in = a different throughput, even if both offer the same nominal bandwidth. ---- 5. Describing Network Requirements [RG] Please reword: To do that it is necessary to make articulating the net= work requirements a little bit more complicated. ---- 5. Describing Network Requirements Where the NRPoU percentiles and NRP [RG] Please inform the reader, where the new per App threshold values NRPoU= percentiles and NRP are originating or determined. ---- 8. How to find network requirements [RG] Your discussion is comprehensible, but lacks some important aspects, w= hich I think need to be clear to a reader: Who is expected to execute this = characterization? Are results to be published? Are the networking condition= s to derive these parameters to be published? ---- 11.1. Missing Temporal Information in Distributions. These two latency series: =85 will have identical distributions, but may ha= ve different application performance. Ignoring this information is a tradeo= ff between simplicity and precision. [RG] I think (exactly) this problem is solved if QoO incorporates more and = suitable QoS metrics, like IPDV. ----- 11.3. Assuming Linear Relationship between Perfect and Unusable (and that i= t is not really a probability) [RG] Assuming this linearity is introducing an error. From what I know from= ITU-T SG12, QoE correlations to performance measurements can be mathematic= ally demanding. Please read into ITU-T G.1051 , Modelling approach for perc= eived interactivity of interactive applications to learn more about some ma= th applied there. I quote: =93The basic assumption of modelling perceived interactivity is its monoton= ous dependency on data latency. The shorter the data transport time is, the= shorter the response time in an interactive application is and the more in= teractive the use of the application is perceived to be. However, this dependency is not a simple linear function; rather there are = saturation areas at both tails of the function, where no further change in = perception happens even if the latency changes.=94 ----- 11.4. Binary Bandwidth threshold [RG] Your judgement is correct. You move closer to ITU-T SG12=85 Codec, FPS= , version of the streaming software, transport used, kind of content, error= concealment and so on have an impact on QoE, not just the network or capac= ity. ---- Von: Marcus Ihlar Gesendet: Donnerstag, 12. Juni 2025 15:42 An: IETF IPPM WG (ippm@ietf.org) Betreff: [ippm] Working group last call for Quality of Outcome (draft-ietf-= ippm-qoo) Hello IPPM, This email initiates the working group last call for Quality of Outcome (dr= aft-ietf-ippm-qoo). The current version of the document can be found here: https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/ https://www.ietf.org/archive/id/draft-ietf-ippm-qoo-03.html Please review the document and reply to this email with any comments and in= dicate whether you believe it is ready for publication. This WGLC will last for three weeks and end on Thursday July 3, 2025. BR, Marcus and Thomas This email and any files transmitted with it are confidential and intended = solely for the use of the individual or entity to whom they are addressed. = If you have received this email in error please notify the system manager. = This message contains confidential information and is intended only for the= individual named. If you are not the named addressee you should not dissem= inate, distribute or copy this e-mail. Please notify the sender immediately= by e-mail if you have received this e-mail by mistake and delete this e-ma= il from your system. If you are not the intended recipient you are notified= that disclosing, copying, distributing or taking any action in reliance on= the contents of this information is strictly prohibited. --_000_GVXPR02MB105608D29859C95E32F8EEE78EA4FAGVXPR02MB10560eu_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
Dear IPPM Working Group,

On Friday I did not have time to detail the changes made. Sorry about that.= Here is the complete breakdown:

## Giuseppe's Comments

> * In the document there are new terms introduced (e.g. Quality Attenua= tion, QoO score, etc). I think they can be all defined in a terminology sec= tion.

[Bj=F8rn] Added a  Terminology section (Section 3) with definitio= ns for Network, Quality Attenuation, Quality of Outcome (QoO), QoO Score, N= RP, NRPoU, Composability, and Accuracy/Precision.

> * In section 3 on Background, several techniques are listed and I sugg= est to also add other IPPM hybrid techniques, such as the On-Path Telemetry= methods (i.e. IOAM, AltMark).

[Bj=F8rn] Added "On-Path Telemetry methods (IOAM, AltMark)" = to the list of relevant techniques in the Background section.

> * The overall organization of the document should be revised and impro= ved. For example section 4, section 5 and section 6 can become subsections = of a new section on Sampling and Network Requirements and the same applies = to section 9 and section 10, that can be put under a new section on Insights.

[Bj=F8rn] Restructured document with consolidated "Sampling and N= etwork Requirements" section and "Insights" section as sugge= sted.

> * Note that the current section 13 on "Conventions and Definition= s" should be moved earlier in the document since it is usually placed = after the Introduction.

[Bj=F8rn] Removed "Conventions and Definitions" since no ref= erence to RFC2119 is needed.

## Ruediger's Comments

> [RG] The document frequently refers to "network" as to be ch= aracterized. Yet I failed to find a definition or a reference to a definiti= on. Please clarify, what you mean by "network".

[Bj=F8rn] Added definition of "Network" in the Terminology s= ection: "the communication infrastructure that connects endpoints, inc= luding all intermediate devices, links, and protocols that affect the trans= mission of data between a source and destination."

> ‎[RG] I note, that =93network operators=94 have little incentive= =85in one statement. Does this express, that a consumer will reliably buy t= he best performing Home Gateway to optimize his QoE by minimizing the effec= t of the home network on their app?

[Bj=F8rn] I assume some well-informed consumers will act in that way, yes, = but certainly not all of them. I believe there are two mechanisms at work&n= bsp; here - one is existing consumer demand for a home network that "j= ust works", and the other is ISPs strategically choosing to optimize for application performance in anticipation of a= consumer shift away from a pure focus on capacity.

> Section 2:
>
> Existing network quality metrics and frameworks typically address the = needs of one or two of these stakeholders,
>
> [RG] Please provide references for existing quality metrics & fram= eworks addressing the needs of one or two of these stakeholders only.

[Bj=F8rn] Added context and references discussing how existing metrics= (throughput for regulatory compliance, QoE metrics for users) typically se= rve limited stakeholder groups.

> Section 2:
>
> While solutions exist for many of the problems causing high and unstab= le latency in the Internet, the incentives to deploy them have remained rel= atively weak.
> [RG] Please provide references for both, "problems" and &quo= t;incentives".

[Bj=F8rn] Added references to bufferbloat mitigation techniques (RFC82= 90), congestion control algorithms (RFC8033), and BITAG latency study to su= pport claims about deployment incentives.

> Section 2
>
> operators have little incentive to improve network quality beyond incr= easing throughput. Despite the availability of open-source solutions, vendo= rs rarely implement them. The root cause lies in the absence of a > univ= ersally accepted network quality framework that captures how well applications are likely to perform.

> [RG] Do you mean throughput or bandwidth?

[Bj=F8rn] Changed to "have little incentive to improve network quality= beyond increasing bandwidth."

> ‎[RG] I=92m not sure, whether the root cause for the claimed unw= illingness of vendors to implement open source solutions is a missing netwo= rk quality framework. Can you provide some proof or a reference for your cl= aim? Further, your proposal is to characterize two thresholds (perfect and stop of app) and interpolate linearly between = these, based on a subset of performance metrics. You apply a set of simplif= ication, rendering the diagnosis less reliable. The dependency on app devel= oper publications isn=92t increasing the quality of diagnosis too, unless a standardized testing environment &a= mp; conditions is the basis for these publications.

[Bj=F8rn] It's hard to really prove that statement, but I do think it's poi= nting in the direction of truth. I've added references and softened the lan= guage a bit. 

> [RG] Please reword:
> =93A universally accepted network quality framework that successfully = captures how well applications are likely to perform may help to increase t= he willingness of vendors to=85.=94

[Bj=F8rn] Clarified terminology and softened claims with "may help to = increase the willingness of vendors to implement such solutions".

> Section 2
>
> Therefore, one critical requirement of a meaningful framework is its a= bility to answer the question, "Will an application work properly?&quo= t;
>
>
> [RG] To me a framework is a starting point and a standard an end. To s= ay, the standard would offer the "ability". Please reword: =85 is= to enable work on a standard answering the question, "Will netwo= rking conditions stop an application from working properly?"

[Bj=F8rn] Reworded to focus on the framework's ability to answer the q= uestion about networking conditions negatively affecting applications.

>Section 2
>
> Answering this question requires several considerations. First, the In= ternet is inherently stochastic from the perspective of any given client, s= o certainty i unattainable.
>
>
> [RG] Please correct/reword (would that be an "is"?)

[Bj=F8rn] Fixed "certainty i unattainable" to "absolute= certainty is unattainable."

> [RG] Please reword: To summarize, the framework and "meaningful Q= oO metric" should have the following properties: Capture a set of netw= ork performance metrics which provably correlate to the application quality= of a set of different applications as perceived by users.

[Bj=F8rn] Updated requirements section with the suggested more precise= wording.

> [RG] Please add a ref to https://www.rfc-editor.org/rfc/rfc7680.html#s= ection-2.8.2 and discuss the novelty of TR-452.1 as compared to the RFC 768= 0 loss specification.

[Bj=F8rn] Added RFC 7680 reference and discussion: "Similar to the One= -Way Loss Metric for IP Performance Metrics (IPPM) [RFC7680], which defines= packet loss in terms of packets that fail to arrive within a specified tim= e threshold, the Quality Attenuation approach extends this concept..."

> [RG] Please inform the reader, where the new per App threshold values = NRPoU percentiles and NRP are originating or determined.

[Bj=F8rn] Added section on standardized testing conditions and require= ment for application developers to publish their testing methodologies.

> [RG] I'd prefer references for the performance figures.

[Bj=F8rn] Added JamKazam reference for remote music collaboration latency r= equirements and updated the example to be more specific.


## Luis's Comment

> One question / comment form my side. It is not clear to me what could = be the behavior of the QoO with adaptive applications that can accommodate = e.g. the bit rate to the network conditions.

[Bj=F8rn] Added  "Adaptive Applications" section explai= ning how QoO can accommodate applications with multiple quality levels thro= ugh multiple NRP thresholds or simplified single-threshold approaches.

## Greg's Comments

> "Mean" and "average" are used in the document almo= st equal number of times. However, RFC 6049 uses only "mean" in r= eference to composable spatial performance metric.

[Bj=F8rn] Standardized terminology to use "mean" consistentl= y throughout the document.

> "Accuracy" and "precision" are used in the documen= t extensively, sometimes even in the same sentence. In general, accuracy ca= n be interpreted as precision.

[Bj=F8rn] Added definitions to Terminology section clarifying that &qu= ot;accuracy" refers to how close measurements are to the true value, w= hile "precision" refers to consistency and repeatability, noting = they are not interchangeable.

> A nit. I think that we use "Quality Attenuation" as a term. = In that case, it must be uppercased throughout the document.

[Bj=F8rn] Standardized capitalization of "Quality Attenuation&quo= t; throughout the document.

> In the Motivation s/i unattainable/is unattainable/

[Bj=F8rn] Fixed typo.

## Ike's Comments

> "Quality attenuation is a network quality metric that meets most = of the criteria set out in the requirements" - From this text alone, i= t is not 100% clear which requirements are referred to here

[Bj=F8rn] Changed to "meets most of the design goals set out in t= he requirements section of this document" for clarity.

> Maybe make it explicit? For example: "However, interpreting raw q= uality attenuation values is difficult, raising the general challenge of ho= w to simplify the entailed information without losing too much in terms of = precision and accuracy."

[Bj=F8rn] Added explicit statement about the difficulty of interpretin= g raw Quality Attenuation values.

> 's/certainty/absolute certainty/'?

[Bj=F8rn] Changed to "absolute certainty is unattainable."

> 's/accommodate diverse application requirements/such diverse applicati= on requirements'?

[Bj=F8rn] Updated

> 's/The overall goal/The overall goal of this document'?

[Bj=F8rn] Updated.

> "s/OSI layer/layer in the protocol stack"

[Bj=F8rn] Changed to "applied across different layers of the prot= ocol stack and is network technology independent."

> The formatting of the start of the paragraphs starting with "Late= ncy Component", "Packet Loss Component", and "Final QoO= Calculation" looks a bit bumpy.

[Bj=F8rn] Improved formatting of the calculation examples for better readab= ility.

> There are "step-by-step" guides at various points in the doc= ument which I like. However, they vary in the concrete format

[Bj=F8rn] Standardized formatting of step-by-step guides throughout th= e document.

> "These two latency series: 1,200,1,200,1,200,1,200,1,200 and 1,1,= 1,1,1,200,200,200,200,200 will have identical distributions"

[Bj=F8rn] Reformatted as: "The two latency series (1,200,1,200,1,200,1= ,200,1,200) and (1,1,1,1,1,200,200,200,200,200) have identical distribution= s"

> Inconsistent use of end users vs. end-users

[Bj=F8rn] Standardized to "end-users" throughout the document.

[Bj=F8rn] I also added an Overview section (Section 2) to help readers quic= kly understand the key concepts before diving into detailed specifications,= addressing the general concern about document accessibility.

The updated draft is available at: https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/
This document introduces the Quality of Outcome (QoO) framework, a novel ap= proach to network quality assessment designed to align with the needs of ap= plication developers, users, and operators. By leveraging the Quality Atten= uation metric, QoO provides a unified method for defining and evaluating application-specific network requiremen= ts while ensuring actionable insights for network ...
datatracker.ietf.org

We believe these changes significantly improve the document's clarity, comp= leteness, and technical accuracy while addressing all substantive comments = from the WGLC review.

Best regards,
Bj=F8rn Ivar Teigen Monclair
From: Bjorn Ivar Teigen Mon= clair <bjorn.monclair@cujo.com>
Sent: Friday, July 4, 2025 6:11 PM
To: ippm@ietf.org <ippm@ietf.org>; IPPM Chairs <ippm-chairs= @ietf.org>
Cc: Magnus Olden <magnus.olden@cujo.com>; Ruediger.Geib@teleko= m.de <Ruediger.Geib@telekom.de>; Jason Livingood (Comcast) <Jason_= Livingood@comcast.com>; Michael Welzl <michawe@ifi.uio.no>; LUIS M= IGUEL CONTRERAS MURILLO <luismiguel.contrerasmurillo@telefonica.com>; Greg Mirsky <gregimirsky@gmail.com>; giuseppe.fioccola=3D40huawei.co= m@dmarc.ietf.org <giuseppe.fioccola=3D40huawei.com@dmarc.ietf.org>; i= ke.kunze@comsys.rwth-aachen.de <ike.kunze@comsys.rwth-aachen.de>
Subject: Re: Working group last call for Quality of Outcome (draft-i= etf-ippm-qoo)
 
Hi IPPM,

A new version of the draft, draft-ietf-ippm-qoo-04, has now been uploaded t= o the data tracker: https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/

It addresses, to the extent we have been capable of doing so, all of the su= ggestions and issues raised since the WGLC was announced.

Thanks again to everyone who has contributed to the document, your time and= effort is greatly appreciated.

Cheers,
Bj=F8rn
From: Bjorn Ivar Teigen M= onclair <bjorn.monclair@cujo.com>
Sent: Thursday, July 3, 2025 1:34 PM
To: ippm@ietf.org <ippm@ietf.org>; IPPM Chairs <ippm-chairs= @ietf.org>
Cc: Magnus Olden <magnus.olden@cujo.com>; Ruediger.Geib@teleko= m.de <ruediger.geib@telekom.de>; Jason Livingood (Comcast) <Jason_= Livingood@comcast.com>; Michael Welzl <michawe@ifi.uio.no>; Ruedig= er.Geib@telekom.de <ruediger.geib@telekom.de>; LUIS MIGUEL CONTRERAS MURILLO <luismiguel.contrerasmurillo@telefonica.com>; Greg= Mirsky <gregimirsky@gmail.com>; giuseppe.fioccola=3D40huawei.com@dma= rc.ietf.org <giuseppe.fioccola=3D40huawei.com@dmarc.ietf.org>; ike.ku= nze@comsys.rwth-aachen.de <ike.kunze@comsys.rwth-aachen.de>
Subject: Re: Working group last call for Quality of Outcome (draft-i= etf-ippm-qoo)
 
Hi Jason, Michael, Ike, Guiseppe, Greg, Luis, Ruediger, IPPM chairs and par= ticipants,

Thank you very much for the detailed feedback! Greatly appreciate the time&= nbsp;you've all invested in this.

On restructuring:
I see how the document would benefit from restructuring in the way Ike sugg= ests. If restructuring can be done without causing a lot of delay (and work= !), I am happy to submit a new version along the suggested lines. If restru= cturing would require significant delays and reviews (or, if others disagree about the need for restructurin= g), then perhaps adding a new sub-section to the introduction is the better= option? Chairs and seasoned IETF'ers, I hope you can advice.

 I will take all of your feedback into account and produce a new versi= on of the document before the deadline on the 7th.

Cheers,
Bj=F8rn

From: Ruediger.Geib@tel= ekom.de <Ruediger.Geib@telekom.de>
Sent: Wednesday, July 2, 2025 5:20 PM
To: Magnus Olden <magnus.olden@cujo.com>; Bjorn Ivar Teigen Mo= nclair <bjorn.monclair@cujo.com>
Cc: ippm@ietf.org <ippm@ietf.org>
Subject: AW: Working group last call for Quality of Outcome (draft-i= etf-ippm-qoo)
 

CAUTION: This email originated from ou= tside of the organization. Do not click links or open attachments unless yo= u recognize the sender and know the content is safe.

Hi Bj=F8rn, hi Magnus,

 

Thanks for your framework for m= ulti-dimensional metric linking delay and packet loss. Please find some [RG= ] marked comments below.

 

Please note, that I=92m off for= vacations and likely won=92t respond to replies before early August. I=92m= sorry for that.

 

Regards, Ruediger

 

 

 

[RG] The document frequently re= fers to =93network=94 as to be characterized. Yet I failed to find a defini= tion or a reference to a definition. Please clarify, what you mean by =93network=94.

I note, that =93network operato= rs=94 have little incentive=85in one statement. Does this express, that a c= onsumer will reliably buy the best performing Home Gateway to optimize his QoE by minimizing the effect of the home network on their = app?

 

Another general comment, withou= t being a native speaker, maybe =93comprehensible=94 rather than =93underst= andable=94? Could some native speaker confirm/reject that?

 

----

 

Section 2:

Existing network quality metric= s and frameworks typically address the needs of one or two of these stakeho= lders,

 

[RG] Please provide references = for existing quality metrics & frameworks addressing the needs of one o= r two of these stakeholders only.

 

----

 

Section 2:

While solutions exist for many = of the problems causing high and unstable latency in the Internet, the ince= ntives to deploy them have remained relatively weak.

 

[RG] Please provide references = for both, =93problems=94 and =93incentives=94.

 

----

 

Section 2

operators have little incentive= to improve network quality beyond increasing throughput. Despite the avail= ability of open-source solutions, vendors rarely implement them. The root cause lies in the absence of a universally accept= ed network quality framework that captures how well applications are likely= to perform.

 

[RG] Do you mean throughput or = bandwidth? Consumers may receive offers of access types with higher bandwid= th. Above a minimum access bandwidth, further increase of bandwidth doesn=92t increase per app throughput. A suitable location of= content and service servers would.

 

[RG] I=92m not sure, whether th= e root cause for the claimed unwillingness of vendors to implement open sou= rce solutions is a missing network quality framework. Can you provide some proof or a reference for your claim? Further, your pr= oposal is to characterize two thresholds (perfect and stop of app) and inte= rpolate linearly between these, based on a subset of performance metrics. Y= ou apply a set of simplification, rendering the diagnosis less reliable. The dependency on app developer pub= lications isn=92t increasing the quality of diagnosis too, unless a standar= dized testing environment & conditions is the basis for these publicati= ons.

 

Please reword:
=93A universally accepted network quality framework that successfully captu= res how well applications are likely to perform may help to increase the wi= llingness of vendors to=85.=94

 

----

 

Section 2

Therefore, one critical require= ment of a meaningful framework is its ability to answer the question, "= ;Will an application work properly?"

 

[RG] To me a framework is a sta= rting point and a standard an end. To say, the standard would offer the =93= ability=94. Please reword:

=85 is to enable work on a stan= dard answering the question, "Will networking conditions stop an appli= cation from working properly?"

 

----

 

Section 2

Answering this question require= s several considerations. First, the Internet is inherently stochastic from= the perspective of any given client, so certainty i unattainable.

 

[RG] Please correct/reword (wou= ld that be an =93is=94?)

 

----

 

2.2. Requirements

Flexibility in describing appli= cation requirements and the ability to capture the delay characteristics of= the network in sufficient detail are necessary to compute application success with satisfactory accuracy and precision.

 

[RG] Please specify =93satisfac= tory accuracy and precision=94 or provide a reference. Please reword:
necessary to improve accuracy and / or precision of=85

 

[RG] Please define what you mean by =93to compute application= success=94 and how that is linked to the QoO network score (the framework set out to describe the latter).

----

 

2.2. Requirements

To summarize, the framework and= "meaningful metric" should have the following properties:=

 

Capture the information necessa= ry to compute the probability that applications will work well. (Useful for= end-users and application developers.)

 

[RG] I=92d prefer rewording to

To summarize, the framework and= "meaningful QoO metric" should have the following properties:

 

Capture a set of network perfor= mance metrics which provably correlate to the application quality of a set = of different applications as perceived by users.

----

 

2.2. Requirements

 

Compare meaningfully to differe= nt application requirements.

 

[RG] Please reword to be more s= pecific on what you mean. Mathematically, and if you=92d like to compute so= mething meaningful, QoO score and app QoE must be correlated, even if you don=92t go for MOS. If this is captured by my rewo= rded version of the first requirement, then that=92s it. If the above secon= d requirement is that the function individual_app_quality (QoE score) needs= be known, then clearly say that. As far as I understand your draft, this function is out of scope.

 

----

 

2.2. Requirements

 

Compose.

 

[RG] Should that be de-compose = and isolate? So far the text didn=92t explain that the QoO score is calcula= ted network-section-wise and finally composed.

 

----

 

2.2.1. Requirements for end-use= rs

 

The quality framework should fa= cilitate a metric that is objective, relatable, and relatively understandab= le

 

[RG] please reword to
=85. metric that is based on objective QoS measurements, correlated to appl= ication quality  and relatively understandable

 

----

 

2.2.1. Requirements for end-use= rs

 

If these requirements are met, = the end-user can understand if a network can reliably deliver what they car= e about: the outcomes of applications.

 

[RG] I=92d prefer a more defens= ive way of putting it, but the statement doesn=92t say, the network is the = only cause of bad app outcome.

My preference:

=85 can understand if a network= can is a likely source of impairment for what they care about: the outcome= s of applications.

 

----

 

2.2.1. Requirements for end-use= rs

A compromise is for the quality= of experience framework to place the responsibility for sourcing and repre= senting end-user requirements onto the application developer.

 

[RG] If I understand this state= ment correctly, you put crating input for the second requirement of section= 2.2 above out of scope of this framework. Please do clearly state that in the first instance where his is mentioned. Sugges= tion:

 

Compare meaningfully to differe= nt application requirements. Note that providing the relevant information d= epends on application developers publications or standards, where available.  

 

[RG] Further, this statement an= d the remaining content define requirements from app developers. Please mov= e to section 2.2.2

 

----

 

2.2.1. Requirements for end-use= rs

 

Some real world examples where = 'acceptable levels' have been derived by application developers include (no= te: developers of similar applications may have arrived at different figures):

 

Remote music collaboration: 28m= s latency note-to-ear for direct monitoring, <2ms jitter

 

Online gaming: 6Mb/s downlink t= hroughput and 30ms RTT to join a multiplayer game

 

Virtual reality: <20ms RTT f= rom head motion to rendered update in VR

 

Performing this UAT helps the d= eveloper understand what likelihood a new end-user has of an acceptable Qua= lity of Experience based on the application's existing requirements towards the network.

 

[RG] I=92d prefer references fo= r the performance figures.

 

----

 

2.2.2. Requirements from Applic= ation and Platform Developers

 

[RG] From my point of view aski= ng for useful UAT requirement publication from App developers requires stan= dard testing conditions (testbed, config and path, forwarding & load characteristics). This might be read into the paragr= aph as is, but it should be explicitely expressed, whether or not such stan= dard conditions for app characterization are required by this section, or n= ot. I don=92t know the authors preference, so I don=92t propose text.
As a thought, if app developers derive data under different networking cond= itions, the overall accuracy and precision of the QoO metric will be reduce= d.

 

----

 

3. Background

 

[RG] Please move the text of = =933.1.8. Quality Attenuation=94 to =93Background=94, as this seems to indi= cate latency and loss as two metrics required. Please be also more precise, as to what Quality Attenuation captures regarding latency di= stribution. I=92m wooried, as you address Average and 99th perce= ntile latency by separate sections.

 

 

3. Background

 

The novelty of the Quality Atte= nuation metric is to view packet loss as infinite (or too late to be of use= e.g. > 3 seconds) latency [TR-452.1].

 

[RG] Please add a ref to https://www.rfc-editor.org/rfc/rfc7680.html#section-2.8.2 and discuss t= he novelty of TR-452.1 as compared to the RFC 7680 loss specification. You = may also reword to:

=93Similar, as the One-Way Loss= Metric for IP Performance Metrics (IPPM) [RFC 7680], the Quality Attenuati= on metric is to view=85

 

----

 

3. Background

 

All applications require a minu= mum level of throughput and a maximum packet loss rate.

 

[RG] Please reword to  ..a= nd work acceptable up to a maximum=85

 

----

 

3.1. Discussion of other perfor= mance metrics

 

[RG] Which metrics have been di= scussed prior to the =93other performance metrics=94. If there=92s no clear= text to that, please

 

-----

 

3.1.3. Variance of latency

 

[RG] Adding the variance of ran= dom processes is producing correct results is producing correct results, if= these are not correlated. Please add a statement. Please provide a reference to a metric definition for Latency Variation.

 

-----

 

5. Describing Network Requireme= nts

 

Applications do of course have = throughput requirements, and thus a complete framework for application-leve= l network quality must also take capacity into account.

 

[RG] Please clarify, whether th= is statement refers to throughput or capacity. Capacity impacts throughput = in the case of congestion, but they aren=92t the same. Throughput may also depend on the bandwidth limitation. A policer is a wor= st case for TCP, a port with a WRED drop-profile might result in a differen= t throughput, even if both offer the same nominal bandwidth.

 

----

 

5. Describing Network Requireme= nts

 

[RG] Please reword: To do that = it is necessary to make articulating the network requirements a little bit = more complicated.

 

----

 

5. Describing Network Requireme= nts

 

Where the NRPoU percentiles and= NRP

 

[RG] Please inform the reader, = where the new per App threshold values NRPoU percentiles and NRP are origin= ating or determined.

 

----

 

8. How to find network requirem= ents

 

[RG] Your discussion is compreh= ensible, but lacks some important aspects, which I think need to be clear t= o a reader: Who is expected to execute this characterization? Are results to be published? Are the networking conditions to derive these= parameters to be published?

 

----

 

11.1. Missing Temporal Informat= ion in Distributions.

 

These two latency series: =85 w= ill have identical distributions, but may have different application perfor= mance. Ignoring this information is a tradeoff between simplicity and precision.

 

[RG] I think (exactly) this pro= blem is solved if QoO incorporates more and suitable QoS metrics, like IPDV= .

 

-----

 

11.3. Assuming Linear Relations= hip between Perfect and Unusable (and that it is not really a probability)<= /span>

 

[RG] Assuming this linearity is= introducing an error. From what I know from ITU-T SG12, QoE correlations t= o performance measurements can be mathematically demanding. Please read into ITU-T G.1051 , Modelling approach for perceive= d interactivity of interactive applications to learn more about some math a= pplied there.

 

I quote:
=93The basic assumption of modelling perceived interactivity is its monoton= ous dependency on data latency. The shorter the data transport time is, the= shorter the response time in an interactive application is and the more in= teractive the use of the application is perceived to be.

However, this dependency is not= a simple linear function; rather there are saturation areas at both tails = of the function, where no further change in perception happens even if the latency changes.=94

 

-----

 <= /p>

11.4. Binary B= andwidth threshold

 <= /p>

[RG] Your judg= ement is correct. You move closer to ITU-T SG12=85 Codec, FPS, version of t= he streaming software, transport used, kind of content, error concealment a= nd so on have an impact on QoE, not just the network or capacity.

 <= /p>

----

 <= /p>

 <= /p>

 <= /p>

Von: Marcus Ihlar <marcus.ihlar=3D40= ericsson.com@dmarc.ietf.org>
Gesendet: Donnerstag, 12. Juni 2025 15:42
An: IETF IPPM WG (ippm@ietf.org) <ippm@ietf.org>
Betreff: [ippm] Working group last call for Quality of Outcome (draf= t-ietf-ippm-qoo)

 

Hello IPPM,

 

This email initiates the working group last ca= ll for Quality of Outcome (draft-ietf-ippm-qoo).

 

The current version of the document can be fou= nd here:

https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/

https://www.ietf.org/archive/id/draft-ietf-ippm-= qoo-03.html

 

Please review the document and reply to this e= mail with any comments and indicate whether you believe it is ready for pub= lication.

This WGLC will last for three weeks and end on= Thursday July 3, 2025.

 

BR,

Marcus and Thomas

 

This email and any files transmitted with it are confidential and in= tended solely for the use of the individual or entity to whom they are addr= essed. If you have received this email in error please notify the system manager. This message contains confident= ial information and is intended only for the individual named. If you are n= ot the named addressee you should not disseminate, distribute or copy this = e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and dele= te this e-mail from your system. If you are not the intended recipient you = are notified that disclosing, copying, distributing or taking any action in= reliance on the contents of this information is strictly prohibited.


--_000_GVXPR02MB105608D29859C95E32F8EEE78EA4FAGVXPR02MB10560eu_--