From sukru.kuran@airties.com Sun Jan 21 23:50:11 2024 Return-Path: X-Original-To: ippm@ietfa.amsl.com Delivered-To: ippm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 72249C14F5E6 for ; Sun, 21 Jan 2024 23:50:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.905 X-Spam-Level: X-Spam-Status: No, score=-6.905 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=airties.onmicrosoft.com Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6YhR4iHhK1Z2 for ; Sun, 21 Jan 2024 23:50:07 -0800 (PST) Received: from repost01.tmes.trendmicro.eu (repost01.tmes.trendmicro.eu [18.185.115.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 65F5FC14F683 for ; Sun, 21 Jan 2024 23:50:07 -0800 (PST) Received: from 104.47.51.232_.trendmicro.com (unknown [172.21.193.99]) by repost01.tmes.trendmicro.eu (Postfix) with SMTP id 5FF0E10001FA0; Mon, 22 Jan 2024 07:50:05 +0000 (UTC) X-TM-MAIL-RECEIVED-TIME: 1705909804.516000 X-TM-MAIL-UUID: f119d261-d656-4c66-b461-9c9fcfdfa9de Received: from EUR03-AM7-obe.outbound.protection.outlook.com (unknown [104.47.51.232]) by repre01.tmes.trendmicro.eu (Trend Micro Email Security) with ESMTPS id 7E47310005E0B; Mon, 22 Jan 2024 07:50:04 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LD4kMeUH6ygj38l4h18HZ0BOdNBi8YOgmJNLg5bUYBWPc1RyNShqsDg6vfJCJLiEoEZwyPcbQBEF6L/Q7u3YoKdlPKdOf1+ON45mcdK0onCgnGr/trKVZ4MxKuZgTjeEis9KXun5iBXEa0tEIIstxpEDdJVzVugh2BzByDycZIYJrXiTzCV68GgHe7qZj1GvvQg0U7yp5qsqckqBA0ErJGjBYZFVXEWGG2S1W/D6gD4CIDJPahFVvWeC6qNPusgiQHzagPoe/qdgepV/21AjlfUXxAMMILPniSxSf2Alie/Qr/7NB9wk7M00VikCUeSaMMoA4eu6ueLSn86BROqiOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JwZbmLFDq1+uwNjmXZ6mZoMBVH1hKcEW/Z7g88fLfCg=; b=MmDvikQK2MeRkzXrRkuB/D8wqR8vj9RUDzabrIHZrMC5GZV+wP6gkYGQQJx/hJZ18xJF/tuv/OPvrjMZO2LZ2lmSmLa0DPdSbx7SlGwGs2I3WPdlttBvL/OjYZuaPuwv/bBoDumT03tKzFzXtSUM+iO8mh9Udz3qGHhWerAcjjUjQMdB3Ji24afzTaVgxbLTMD0knP+EvVtx4oEAwoDgskJCSfycUYPSAynpy8G5j5Y2t/qT9qiKj+xjtPSDCegeXZG1ZlMeelfAAuSm9t4DL6yqILu7RkgDaeXd9dH6JK2JezKMfV+UwUmxA1qPNodshb9DGBWcTcRmtwbYJwYuSA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=airties.com; dmarc=pass action=none header.from=airties.com; dkim=pass header.d=airties.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=airties.onmicrosoft.com; s=selector2-airties-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JwZbmLFDq1+uwNjmXZ6mZoMBVH1hKcEW/Z7g88fLfCg=; b=Zd17rltIqN/fA4bGsjFnifCwsu513Kg4BUNKTvN2TaBJlqHspvSnbF8i6niRUkiHlxcY/SdTkyW0OTnNTsbaV3WuRBUC2nmF0hG9RL12gkMJ/dfy6bAmDkh+ZwG2sFML5SFfB4yeTk9ngaXSZ3D4dfPW9YGRj4c48eQDeKs2bzU= Received: from VI1PR10MB1695.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:802:39::13) by AS4PR10MB6136.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:587::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7202.31; Mon, 22 Jan 2024 07:50:01 +0000 Received: from VI1PR10MB1695.EURPRD10.PROD.OUTLOOK.COM ([fe80::e302:4e5e:c051:8425]) by VI1PR10MB1695.EURPRD10.PROD.OUTLOOK.COM ([fe80::e302:4e5e:c051:8425%6]) with mapi id 15.20.7202.031; Mon, 22 Jan 2024 07:50:01 +0000 From: Mehmet Sukru Kuran To: Tommy Pauly , "IETF IPPM WG (ippm@ietf.org)" Thread-Topic: [ippm] Adoption call for draft-olden-ippm-qoo-02 Thread-Index: AQHaSJ9gXVzQ3bEtlkiJlgciO+UrSrDlfHfd Date: Mon, 22 Jan 2024 07:50:00 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, tr-TR, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=airties.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR10MB1695:EE_|AS4PR10MB6136:EE_ x-ms-office365-filtering-correlation-id: 999c6e56-82f7-4b74-cce3-08dc1b1ebcb4 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: M0QCqttRLShCTknIL+Wwu4cFpN4v0ptSqLQYuuJq+gkT289SBWOvsSaGSRyk67wtDczrOpAE6DZF/4hFw2SNQMBHIbFsP7oA7nE252KuaI/78yCn2puFcl2m3An8IgU/WPa4OdECiWjoZtX0GWWFlTs8hVmaGfilgEAWzGbHyTiK+crfIVQ8bqJoHhBuUQMmxpLRK03etu312rpDnZgArlixpWvEsvbb87l6wFyvlWbjXZWRmdAK4MGVQ2mQDd7Nrpx9HhStNzQxrKUxhWTbddNGAdQ5T/kfRrD1Ue4rsHUb0tV3kw/ly/CL91X1Os9mFWCn0IRVoWA69crY3CQOtIK1sjUs0XCYvmDB7vk78w2rsxUSocPnRamshKAOz3XVsW/QdGwvyqvL2KqvnhH0ulnFuUdBIp/2Pbl8F2TQCDDgFrKG0rmQOO/px6roHFTjIqTVSszU5UKcMlRp/eQ30FhfN7j6owrcXXXuAyzrpccYulp6NjpbMSmZlL1LWFFZUBU1mNrmcSBY9Y7lgDBHNvuw5j7zvcMDVZP6cyTPtUxItYuR1AlyC3FNyDpNiK6Lm3+++L82XobmdZ3Xlo9clC3N31zBk2mY6fSirqQhSgU= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR10MB1695.EURPRD10.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(13230031)(346002)(366004)(136003)(396003)(376002)(39840400004)(230922051799003)(186009)(1800799012)(64100799003)(451199024)(55016003)(19627405001)(38070700009)(66899024)(2906002)(66556008)(66476007)(966005)(76116006)(64756008)(66446008)(66946007)(53546011)(26005)(66574015)(71200400001)(6506007)(7696005)(478600001)(316002)(8676002)(8936002)(110136005)(5660300002)(52536014)(21615005)(9686003)(122000001)(38100700002)(83380400001)(166002)(86362001)(33656002)(41300700001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?BNwDwSSPN++CScyO/I6NxoFiI9U35L7Xk4hLZBASvQzrkhM6dpTjw1o/?= =?Windows-1252?Q?aYcORtTGXKTpoQeZSHcu72QMmS2EIzjykOiKXKzAA6dnNCyXU5VixIJg?= =?Windows-1252?Q?wYV2WpdY1nCNoEL1xbQ6kFU/DxDB6i25y65a1Fha7Jv395cuXYKx19Sw?= =?Windows-1252?Q?PTdXDue3rLwcBA85r4DNOVsA17Iqa0CA4JWamMG2cZJTQWjJarGoLY9i?= =?Windows-1252?Q?hvBbME0vuqqIgiEIh5l7dH4HqybTCW9uBlq1oLBcaE/kAhe9GjZSWARc?= =?Windows-1252?Q?06JGsrwEPFt31o8X3ZcHGmQ4mfK1dBB6GPSdoF/fz2b9ZA+iMX/LX/TP?= =?Windows-1252?Q?WA5wQGhW140HDXzStF+TXp6BIKMeajpKL+A59j/KaiU9Wghw5QmYpKvY?= =?Windows-1252?Q?b2oCtnwYNNJPAOXyWhO94aPGdagci8CTBJX+aKJMIMSM3F9za0OhTcQn?= =?Windows-1252?Q?bxykAhNB/0Db7VeLZeX5zg1B3TeVofEI97tXmnkSrEiaL05u59qzWrSM?= =?Windows-1252?Q?lTdHeckSSGSMWekaKZRznwBnkpZS7aCnDPORIHj8FozA4thxQMirkkQ8?= =?Windows-1252?Q?2HFOGxKS7cInh1jUUKC2Wem1U+i62CRRBvJhc1RaoUM2POvhoKcxTBfz?= =?Windows-1252?Q?S/5AB73lMlDEzE7tqfT+/9s5nyBUzHv/2tEtAd2JE/DZ9GW7s7g5/+Ro?= =?Windows-1252?Q?5WZCXS6h/UOYRvrDNORkZ5zyeu5Qj2YuYS2ZV+XliYn9sFGKtKHSb507?= =?Windows-1252?Q?Re/ZRITxz88fN3JjXyfYc53YQ+H2LhOu4P03Z/HchMShsJlC4NoMgQ9O?= =?Windows-1252?Q?/o88DAqUJ49b75HjZps1TekU/5HmGfvq1OcmvLCD0hhqAEHQUKovSWkA?= =?Windows-1252?Q?m1FQqP4vMLJmOLuri52N/z9XHmxDYGlBuCg2LucTEgMUqjZrAe5cWNRy?= =?Windows-1252?Q?PXfpn/Jfi2NdLCXwPf9kWJbs/MQZRMePueXllQPQL3UH5zDHlmgn8pHa?= =?Windows-1252?Q?Z9Q8IflliV2QbE53jZW9T87mVSaNm/fCDdvlXNwixZHYXze27MjkKQKz?= =?Windows-1252?Q?2ji6T0V1GmtP0R45N8ahASvlEh9fmk2zANtu9uWuROCEGbOTTBiQQzqC?= =?Windows-1252?Q?R1Jgr+3UM6vsqWQtYE2WZDHD/Rxvi68ON7GB6/OrLy3t7c3EymEyDToM?= =?Windows-1252?Q?k7cMBL/1QSvR/aM46qHfC026+Npd0XljLHP8Rgds3FUSiPIeXooAY9KS?= =?Windows-1252?Q?f7SEH9JbsxNJ3My6GNByfVroY4ZXeKCXM7QLZn2/ygnqvfptEa0afj9g?= =?Windows-1252?Q?NLbEhjwALhihd3B9Z1V1UibIhRvKX1ztUtmmWnuW+vGSXVhqZNUYzQUT?= =?Windows-1252?Q?lmP+YyVobj0VukCN9rBPHZKk+NEQ3tSUGOXRD3E/EHKnBy2arTKV3G1h?= =?Windows-1252?Q?AO8nnm6JnuUVrXVLTInNZXKBk+hspzFwkW/hHZk5c148DCVi/32B1Q+1?= =?Windows-1252?Q?skeRr5ZO3/DF+wJDa7v96JD+YJu2rWr2TK1c8ybytVDuo/9hHmWg2xAD?= =?Windows-1252?Q?VtUG5DvCNISM8M6uqs4cnsQ9bYaGFcSWXKCg/oK+XAth0Z3Tol8p+zCD?= =?Windows-1252?Q?5fFHVQx9FC0X1o/2X3dYeqB8hAeStXYHmxVTlGJ9Jr7zLH0vo7fkSYFV?= =?Windows-1252?Q?M7Ky/ipQUiTRtFtJnMVPIL4oSeYcghF5?= Content-Type: multipart/alternative; boundary="_000_VI1PR10MB16958312D2C72D737557042BE3752VI1PR10MB1695EURP_" MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: OwaU6dV4nmbkrQuqoLQkJyY+BuzP6H1+zEPsq4RKo7eHlzdTLi381K6fBJKf4HOl0DZGXbVqh7xVtyLRUyqbJVXTmZWUI2Og1U0+OtKdihYvgg+axCd9w2S4hLamCEglZ3YGVXqheOpGaB10DC3ePzL/zujGpPjPKoPkbiby7WbezToAWZOUEWU7k8RuIifQg5GFiDu77l0MCYyZiduGmWXKAn+x5Fc4cDkqez9t+Eyk0y2yvBc3/1zUtfrEuEXi1gNmgWIgwqZ0O18fozoaku6/6bIgNRzTnqeHefHERcNuUqRUhRuDeZ5F3X/W1p0wEuoT5fnVxRBgUiWohfWU5ZTqzhjZF1e0bssJDAJ3X6XrxHF4keYlOjURfM+vRCfTjQiNaym2x24JM6U0p7hv/WU6QJMHfSjjL5qlCjUZbhXG0gy1R7G+T0IwCQc3oD+Gvn9ATy+HT481dp6S8evn4un+OVEwVmT1IDJkJQ6NmEKqqArUFB1n0r3RDQUA/oVAMi+jCS3Z3Zxhb2OJjaY1dSyEOCKiGkzLet9G7b5xzPGq23R61QpkwZlnbkX2wqKsp4n+oo7TJdVkEhep3UYXV3s5aS2z/5qVvE2pzmpEVeUL8KURDBR8TI273Cgb9BFgKBeDPcmfCiNbEcENtoATJQ954lAjDBeXcfSBMGHzsxBSogS0Rv/O1b4N80+Z9BVA6zgSOLVvHQnXbwHcw4JQFw== X-OriginatorOrg: airties.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR10MB1695.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 999c6e56-82f7-4b74-cce3-08dc1b1ebcb4 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jan 2024 07:50:00.8109 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 316fd558-a5a6-42fd-9da7-d0f38e0864f6 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: vgEgGEAHO/2EkpLlJpo/+UZK30Curzkp7EVGncB8RYTKYgseWvvz2skN+FJj/hy8N18grZOol3F7BdUq8A72NQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR10MB6136 X-TM-AS-ERS: 104.47.51.232-0.0.0.0 X-TMASE-Version: StarCloud-1.3-9.1.1015-28134.005 X-TMASE-Result: 10--23.267800-4.000000 X-TMASE-MatchedRID: 7u3eoxEoplC3ewN2uw1RPt+PuzKH4QXrzxHg3Mb1A4y3ud8LAieio1yE o5yPzsdRSjwx3Bij8vlfi/qCjJOfw5rXfVDpxQVufKkqkkoPV3evj1V06s9OqWTbCITnGjLQl2K BmLXt0epch0W2aUveS6RpGvK1xmVRciAaUUJ5e6jS51Sxx29ASTnP3JZqsdOaB4BMfq4SzeJCdj TzwvXemonNMAq6FuOtOZ2vZscuAC+lc+z1WGQe1RuIUYqOqjYXXAeS+w5YgSXuAT6T68zFpXKJ+ HXXpf9uJu83Ba1vsFH0ZvN6DzIUD0o3b4k+QmYuTbSs3l71c3X4PV3mEBaU4q9JSkXsXWchelJS +dKIGoew/JXCfBXMTMOLB/tryGA3WilyBa7fsvHnzGhnjWUN/d2crNR8miTUYnktlVUqr6Ho5VV s9R1QNucXRfpFx7tcSpLC5gJFdjBgYsm/Puy7amcJ5T2AC1fhbImvEGrHk0I4+taAy0d3a6IzQ5 kSF2A4+EzyWOS9eUXHlJWhYfvKXMy3ob409MRIqIpjxEoeciFkRDVifPlAcR2OWvOeeOFgkj4YN hB8WcLVx4LDgWk+nP2uFRflH/2DBnpcDcXfJC/6O0dbizLJYbDQN+uXSQvrlgNa7tJb8yaIcOmd 382XOz+z0S2PWR7I5MrTKS84v0JS+gaCeXpFI98CIqxAH44m0XB/qaiHbt0tzM8wdlz5DlIhOZR rGf3IhaEadNgEBzexzvcAvY5dsQWqg6oT8oAaJTqhCibs88qpSoJx15avkR+HTcaJMBgqc6R0ga 12srXYI7fLwCbgXM4ueQycaIDwSTmyKIUQoMPg7ZbVT1KkpUI9AEPfED/dn8cUfSHIwCASCDdzA MeX/e2bIS/HALlkg6tVyaF73JX56Vci9CWQWGLK5A5md0r9ulxMqwyzVlEhtlJZdlnnbQ== X-TMASE-XGENCLOUD: 7e5c8c67-c298-49e2-b635-35155484c3b8-0-0-200-0 X-TM-Addin-Auth: 4MeBvtGorzthiCzOyjH7NuPZt+i+7GOHFeGsBnh79nBrUcG37wpSe3Tu0nh SIxWswhT55ptYUa1NLd42jdy4FUcGGc9qi3PMlPAUoNSB2GhLrmeGpu/cW+0H9zpr2suQ/w/eQo Grvd3Itc/OzWEwdjQXm3wJIpJyfFRfAUhFSYhJjq8+EzDUW8qe6flacH4KR47uJbknl8uydjIKy EiChBycfhzOQLiI2aewn5R4FGMRAJIpwejkufu0IYbuxZvGlRzAVxHVUKrxYwx8mieWDY8jXmaR YcW/KOjZF9DTUv2iBhIiilQ7QEpMMFSwXHQY9bDdDX3CgEmyZvY25qpIhQ==.nSCtIeSboz5ZuK R+RjMHuzJ3cIBhr9gU8eBbtV452kDxxQI3CDz3MytVe0UEOWXDI6w+36rI6fvf6GKJFfoZwt04o qT8fxz1DjnnDPh0WQv39nGvFMP/GzR1F3TEp9I/6o4DBHIVRlBZ4nBnl58zr6nEebTf6+sSo2Ys xmqTUKIz7YRSqqvYmzvvrMuJUmGdKAZrkzCTvFuZGyA+I24CURzsBVaU1/oEklcVVsj5zVg+DLx Gt/SWqvrj0shAzmJQ0lB5XYRxPt8xB2lgPVIxwcOJ6J/Oo3rGF//ofgd382oWgtm/MsxpscRCog SjB/0gPz4HIQKIMkc4PvY+KIJeSg== X-TM-Addin-ProductCode: EMS X-TM-Deliver-Signature: B4FCF7C4AB4D03650E5805B9095EA4CD Archived-At: Subject: Re: [ippm] Adoption call for draft-olden-ippm-qoo-02 X-BeenThere: ippm@ietf.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: IETF IP Performance Metrics Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2024 07:51:33 -0000 --_000_VI1PR10MB16958312D2C72D737557042BE3752VI1PR10MB1695EURP_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi all, I thank all the contributors who have worked on "Quality of Outcome". Belo= w are my comments, 1. First of all, I really liked the idea and I think QoO is a very neat me= tric for evaluation QoE without going into the pains of conducting a lot of= manual testing and try to rely on MOS values. Getting the required NRP and= NRPoU values for different applications considering different codecs, buff= ering mechanisms, etc... might be tricky for mass deployment scenarios but = all in all I believe this is a very nice addition for measuring QoE of QoS = sensitive traffic. 2. Assume for an application, the NRP P99 is 100 ms, and NRPoU P99 is 200 = ms; when a particular "application experience" has a P99 is 110 ms, how com= e can you say there will be 10% of lagging? I mean, how can you show that = lagging is a linear function? * I'm aware that showing this is a linear function (or any other type o= f function) is not easy at all. If there is a proof showing that such a lin= ear function holds I believe it will increase the value of the IETF documen= t to either explain it or refer to the proof in another document (etc. a pa= per). * In case there is no such proof at the moment, for the documentation i= t might be better to say that "There can be different functions and one cla= ssical implementation can be using a linear function." * In case there is no such proof at the moment, conducting a lot of tes= ts with empirical data and back the claim with these empirical data or tryi= ng to prove that function F is the correct one can be very interesting and = quite valuable. 3. Is there a systematic way of converting packet loss, max. delay, max. ji= tter into P90, P99, P99,9 NRP and NRPoU values? Also, how to combine throug= hput requirements into this QoO? * You have an example in the document but the details are not clearly l= aid out. I believe such a conversion method, formulation would be pretty us= eful. * In the long run, if application vendors define P90, P99, P99,9 NRP an= d NRPoU values; this will stop being a problem. However, for a wide-spread = use of QoO I believe such a "conversion formulations" will be very, very va= luable. 4. Regarding the measurements to be used for building the CDF, I see that t= he only limitation is "having at least 10 samples". * I think just this requirement is just is too loose. On Page 4-5 you e= xplained this not being ideal and in any report on QoO 3 variables must be = available. However, even then I believe this requirement can be tightened a= fter some test runs and trials. * There can be several suggested measurement patterns (e.g., a) 1/sec; = the measurement should be E2E including encoding/decoding; b) 1/10 sec; the= measurement should be from the home router to the server, etc...) 5. On page 7, there is a formulation as QoO =3D min(ML, NRP, NRPoU) =3D (1-(ML-NRP)/(NRPoU-NRP))*100 - The document says mathematically these two formulations are equal= . However, I see that in many occurrences the two formulations give differe= nt results. - Am I missing something here? or making an error? 6. On page 7, there is a formulation as QoO =3D (1-(ML-NRP)/(NRPoU-NRP))*100 - This formulation "I thought" should give values between 100 (perf= ect) and 0 (unacceptable). However, if ML < NRP the result is more than 100= . - In case I'm not making a mistake and the intention is the QoO to = have a value of [0,100], the formulation can be changed into MIN( 100, (1-(ML-NRP)/(NRPoU-NRP))*100 ) Regards, ________________________________ From: ippm on behalf of Tommy Pauly Sent: 16 January 2024 20:13 To: IETF IPPM WG (ippm@ietf.org) Subject: [ippm] Adoption call for draft-olden-ippm-qoo-02 Hello IPPM, This email starts a working group adoption call for "Quality of Outcome=94 = (draft-olden-ippm-qoo). https://datatracker.ietf.org/doc/draft-olden-ippm-qoo/ https://www.ietf.org/archive/id/draft-olden-ippm-qoo-02.html The call will last for 3 weeks, and end on Tuesday, February 6. Please repl= y to this email with your review comments and indicate if you support adopt= ing this work. Please note that we did a previous adoption call that did not receive suffi= cient feedback. At the last meeting at IETF 118, we did have a good amount = of comments and questions, so please do reply to this email if you have rev= iewed the document. Thanks, Tommy & Marcus Information in this email including any attachments may be privileged, conf= idential and is intended exclusively for the addressee. The views expressed= may not be official policy, but the personal views of the originator. If y= ou have received it in error, please notify the sender by return e-mail and= delete it from your system. You should not reproduce, distribute, store, r= etransmit, use or disclose its contents to anyone. --_000_VI1PR10MB16958312D2C72D737557042BE3752VI1PR10MB1695EURP_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
Hi all,

 I thank all the contributors who have worked on "Quality of Outc= ome". Below are my comments,

 1. First of all= , I really liked the idea and I think QoO is a very neat metric for evaluat= ion QoE without going into the pains of conducting a lot of manual testing and try to rely on MOS values. Getting the require= d NRP and NRPoU values for different applications considering different cod= ecs, buffering mechanisms, etc... might be tricky for mass deployment scena= rios but all in all I believe this is a very nice addition for measuring QoE of QoS sensitive traffic.=

 2. Assume for a= n application, the NRP P99 is 100 ms, and NRPoU P99 is 200 ms; when a parti= cular "application experience" has a P99 is 110 ms, how come can you say there will be 10% of lagging?  I mean= , how can you show that lagging is a linear function?
  • I'm aware that showing this is a linear function (o= r any other type of function) is not easy at all. If there is a proof showi= ng that such a linear function holds I believe it will increase the value of the IETF document to either explai= n it or refer to the proof in another document (etc. a paper).
    • <= li style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; c= olor: rgb(0, 0, 0); list-style-type: "- ";"> In case there is no such proof at the moment, for t= he documentation it might be better to say that "There can be differen= t functions and one classical implementation can be using a linear function."
  • In case there is no such proof at the moment, condu= cting a lot of tests with empirical data and back the claim with these empi= rical data or trying to prove that function F is the correct one can be very interesting and quite valuable.<= /span>
    &n= bsp;   
3. Is there a systema= tic way of converting packet loss, max. delay, max. jitter into P90, P99, P= 99,9 NRP and NRPoU values? Also, how to combine throughput requirements into this QoO?
  • You have an example in the document but the details= are not clearly laid out. I believe such a conversion method, formulation = would be pretty useful.
  • In the long run, if application vendors define P90,= P99, P99,9 NRP and NRPoU values; this will stop being a problem. However, = for a wide-spread use of QoO I believe such a "conversion formulations" will be very, very valuable.

4. Regarding the meas= urements to be used for building the CDF, I see that the only limitation is= "having at least 10 samples".
  • I think just this requirement is just is too loose.= On Page 4-5 you explained this not being ideal and in any report on QoO 3 = variables must be available. However, even then I believe this requirement can be tightened after some test runs= and trials.
  • There can be several suggested measurement patterns= (e.g., a) 1/sec; the measurement should be E2E including encoding/decoding= ; b) 1/10 sec; the measurement should be from the home router to the server, etc...)

5. On page 7, there i= s a formulation as

    QoO =3D= min(ML, NRP, NRPoU) =3D (1-(ML-NRP)/(NRPoU-NRP))*100

   &nb= sp;    - The document says mathematically these two formulat= ions are equal. However, I see that in many occurrences the two formulation= s give different results.
   &nb= sp;    - Am I missing something here? or making an error?

6. On page 7, there i= s a formulation as

    QoO =3D= (1-(ML-NRP)/(NRPoU-NRP))*100

   &nb= sp;    - This formulation "I thought" should give = values between 100 (perfect) and 0 (unacceptable). However, if ML < NRP = the result is more than 100.
      =   - In case I'm not making a mistake and the intention is the QoO to h= ave a value of [0,100], the formulation can be changed into

    MIN( 10= 0, (1-(ML-NRP)/(NRPoU-NRP))*100 )

Regards,
From: ippm <ippm-bounces= @ietf.org> on behalf of Tommy Pauly <tpauly=3D40apple.com@dmarc.ietf.= org>
Sent: 16 January 2024 20:13
To: IETF IPPM WG (ippm@ietf.org) <ippm@ietf.org>
Subject: [ippm] Adoption call for draft-olden-ippm-qoo-02
 
Hello IPPM,

This email starts a working group adoption call for "Quality of O= utcome=94 (draft-olden-ippm-qoo).


The call will last for 3 weeks, and end on Tuesday, February 6.= Please reply to this email with your review comments and indicate if you s= upport adopting this work.

Please note that we did a previous adoption call that did not receive = sufficient feedback. At the last meeting at IETF 118, we did have a good am= ount of comments and questions, so please do reply to this email if you hav= e reviewed the document.

Thanks,
Tommy & Marcus

Information in this email= including any attachments may be privileged, confidential and is intended = exclusively for the addressee. The views expressed may not be official poli= cy, but the personal views of the originator. If you have received it in error, please notify the sender by = return e-mail and delete it from your system. You should not reproduce, dis= tribute, store, retransmit, use or disclose its contents to anyone.

--_000_VI1PR10MB16958312D2C72D737557042BE3752VI1PR10MB1695EURP_--