Re: [ippm] Adoption call for draft-olden-ippm-qoo-02
Dave Taht <dave.taht@gmail.com> Wed, 31 January 2024 11:02 UTC
Return-Path: <dave.taht@gmail.com>
X-Original-To: ippm@ietfa.amsl.com
Delivered-To: ippm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4DFFFC18DB97 for <ippm@ietfa.amsl.com>; Wed, 31 Jan 2024 03:02:54 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.095
X-Spam-Level:
X-Spam-Status: No, score=-2.095 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z6YjIj3YnNH2 for <ippm@ietfa.amsl.com>; Wed, 31 Jan 2024 03:02:47 -0800 (PST)
Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C384BC15108E for <ippm@ietf.org>; Wed, 31 Jan 2024 03:02:47 -0800 (PST)
Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-5d8bc1caf00so3152722a12.1 for <ippm@ietf.org>; Wed, 31 Jan 2024 03:02:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706698967; x=1707303767; darn=ietf.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=mxB3tBC1zEmnxOQWpOLfsxqlCCO51KOK02j3OYXcDxo=; b=Mt4t8MUA2pFTMWaIrbALaQSCka/aiXcq6EeXODr8acYMH8OypDsE8gAb/rG8KBxP3x Sxpzl3tKiga3dUtP1CyTOCC1VI2VHSHg8Pb1vgFhIMx28FxIbfCcgGQJsr5oWoES9XCQ HyVR97Vl2U/8XdfTnDtl6vG/DLJSvYcrzieuznwOa8eZJoT903LDXc8tkaLvZVI/nJXG ETJwduQloSELWr29gmbdKjeqaJVznIE8LaNUA0Zt6v2NhECn2U6mSTqsOp4ODEmUPexF xqyVX+21/Q/tzSIznAqSFIeM+PYkm8CbYUBLzZjsD76Pl84TV4nEjxeqNBJu9pNQE61b QLfQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706698967; x=1707303767; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mxB3tBC1zEmnxOQWpOLfsxqlCCO51KOK02j3OYXcDxo=; b=e9Yp6qy3qE+PGhoSZWDNm6PsTpIGk89/a81JXaD8Kg4YdqHGpEOQcLrhSA3HL2HA6H YVPCJK9FbAjt5CcFexrhM1d8o4qe/lSetVDY9XtrmVn8RceXWiykczAmvtzRavt5Z8NW UtJHazuvP1aaF+gj8Mo0n+q5yGH4/yXKRtPJcoTE8n6T/onynZ6zm2Ra/MICz4PLJDWl R+zw31ACzb6PoDpcw1gmZ93AGXCaX6Y7SVmy4AlV04MQ1aK48mGDZgLmNBrnU4+jZf39 qHk8NiNimbHmhlIAhOI4qZvHVh/DOrG4P4yrUYCWwElcsUnLIcNePNnWAHzNKA7xvL+Q jLnA==
X-Gm-Message-State: AOJu0YzcZQFv9CxsLtmG4lsN4OtEBpsA+X5/0fgfPKjGDRwktPQ+ZJOF vu077fU+Bf+hNWPJ0pD0xhTKuqgDBR5gyYalTSbgtXWdjc6ktT/JUF+vd8jG72Q3iAvY+zhBUqu gBQLHZHbZJVzPAlmeqd/ezH7KYXU0rhIN
X-Google-Smtp-Source: AGHT+IElmP9Gwgc+WYaxGwmJKqoBBVRaEPewwFcF9X56YNvl+6PpzJBamweaChFOVq6NjUANWBALDBbqjwvyDOAGLPs=
X-Received: by 2002:a05:6a21:38c5:b0:19c:7e9f:44c1 with SMTP id yk5-20020a056a2138c500b0019c7e9f44c1mr1096995pzb.30.1706698966988; Wed, 31 Jan 2024 03:02:46 -0800 (PST)
MIME-Version: 1.0
References: <DCD67FE3-AFC5-4689-89EF-66387949214C@apple.com> <VI1PR10MB16958312D2C72D737557042BE3752@VI1PR10MB1695.EURPRD10.PROD.OUTLOOK.COM> <CAKf5G6KxskTc6GAEkezRjP9YmbeJ2HLcUvgmX+hMktg201nTdw@mail.gmail.com>
In-Reply-To: <CAKf5G6KxskTc6GAEkezRjP9YmbeJ2HLcUvgmX+hMktg201nTdw@mail.gmail.com>
From: Dave Taht <dave.taht@gmail.com>
Date: Wed, 31 Jan 2024 06:02:34 -0500
Message-ID: <CAA93jw7i8tH4Zj9OAdUM6RHx6wmu7G98i6vsrD5WOEAhsPQsiA@mail.gmail.com>
To: Bjørn Ivar Teigen <bjorn@domos.no>
Cc: Mehmet Sukru Kuran <sukru.kuran@airties.com>, Tommy Pauly <tpauly=40apple.com@dmarc.ietf.org>, "IETF IPPM WG (ippm@ietf.org)" <ippm@ietf.org>
Content-Type: multipart/alternative; boundary="000000000000aa7c6706103bd1d6"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/mzvSSBEk9SdRhemxPO5Dqyxo2Yw>
Subject: Re: [ippm] Adoption call for draft-olden-ippm-qoo-02
X-BeenThere: ippm@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ippm>, <mailto:ippm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm/>
List-Post: <mailto:ippm@ietf.org>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ippm>, <mailto:ippm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 31 Jan 2024 11:02:54 -0000
Aside from the commonly cited 100ms goal ignoring a century's worth of human factors research, I support adoption, if this method could be extended down to numbers closer to 20ms (voip), 16ms (a frame), or 4ms (VR). On Wed, Jan 24, 2024 at 8:13 AM Bjørn Ivar Teigen <bjorn@domos.no> wrote: > Hi all, > > Thanks for the feedback Sukru, > > Please find my comments inline. > > On Mon, 22 Jan 2024 at 08:51, Mehmet Sukru Kuran <sukru.kuran@airties.com> > wrote: > >> Hi all, >> >> I thank all the contributors who have worked on "Quality of Outcome". >> Below are my comments, >> >> 1. First of all, I really liked the idea and I think QoO is a very neat >> metric for evaluation QoE without going into the pains of conducting a lot >> of manual testing and try to rely on MOS values. Getting the required NRP >> and NRPoU values for different applications considering different codecs, >> buffering mechanisms, etc... might be tricky for mass deployment scenarios >> but all in all I believe this is a very nice addition for measuring QoE of >> QoS sensitive traffic. >> >> 2. Assume for an application, the NRP P99 is 100 ms, and NRPoU P99 is >> 200 ms; when a particular "application experience" has a P99 is 110 ms, how >> come can you say there will be 10% of lagging? I mean, how can you show >> that lagging is a linear function? >> >> - I'm aware that showing this is a linear function (or any other type >> of function) is not easy at all. If there is a proof showing that such a >> linear function holds I believe it will increase the value of the IETF >> document to either explain it or refer to the proof in another document >> (etc. a paper). >> - In case there is no such proof at the moment, for the documentation >> it might be better to say that "There can be different functions and one >> classical implementation can be using a linear function." >> - In case there is no such proof at the moment, conducting a lot of >> tests with empirical data and back the claim with these empirical data or >> trying to prove that function F is the correct one can be very interesting >> and quite valuable. >> >> This is an excellent point and something I think merits more > investigation. The assumption that the transition from perfect to useless > is modeled well by a straight line has not been tested thoroughly. > I've seen other work (such as the ITU-T model for voice quality) that > model this transition with a more complex function. A straight line is a > significant simplification from the point of view of designing network > requirements, and simplicity has a value of it's own here. That said, model > simplicity must of course be balanced with model accuracy. > > >> >> 3. Is there a systematic way of converting packet loss, max. delay, max. >> jitter into P90, P99, P99,9 NRP and NRPoU values? Also, how to combine >> throughput requirements into this QoO? >> >> - You have an example in the document but the details are not clearly >> laid out. I believe such a conversion method, formulation would be pretty >> useful. >> - In the long run, if application vendors define P90, P99, P99,9 NRP >> and NRPoU values; this will stop being a problem. However, for a >> wide-spread use of QoO I believe such a "conversion formulations" will be >> very, very valuable. >> >> That's a very good point. I think we should aim to include a section in > the document describing in some detail how to create network requirements. > I've made an issue on the github page, here: > https://github.com/domoslabs/QoOID/issues/7 > > >> >> - >> >> 4. Regarding the measurements to be used for building the CDF, I see that >> the only limitation is "having at least 10 samples". >> >> - I think just this requirement is just is too loose. On Page 4-5 you >> explained this not being ideal and in any report on QoO 3 variables must be >> available. However, even then I believe this requirement can be tightened >> after some test runs and trials. >> - There can be several suggested measurement patterns (e.g., a) >> 1/sec; the measurement should be E2E including encoding/decoding; b) 1/10 >> sec; the measurement should be from the home router to the server, etc...) >> >> The idea of making the requirement as loose as possible is to keep the > barrier to adoption as low as possible. I think that is an important > principle, but I get your point that we need to maintain a certain standard > in order for the QoO results to be meaningful. > > >> >> - >> >> 5. On page 7, there is a formulation as >> >> QoO = min(ML, NRP, NRPoU) = (1-(ML-NRP)/(NRPoU-NRP))*100 >> >> - The document says mathematically these two formulations are >> equal. However, I see that in many occurrences the two formulations give >> different results. >> - Am I missing something here? or making an error? >> > It's possible we've caused some confusion with the notation here. The line > min(ML, NRP, NRPoU) is indended to mean "find the point where the measured > latency crosses each of the NRP - NRPoU lines, and choose the "worst one" > as your QoO value". Sorry if that is a little obscurely worded - I've made > an issue to clarify the explanation in the text ( > https://github.com/domoslabs/QoOID/issues/8) > > >> >> 6. On page 7, there is a formulation as >> >> QoO = (1-(ML-NRP)/(NRPoU-NRP))*100 >> >> - This formulation "I thought" should give values between 100 >> (perfect) and 0 (unacceptable). However, if ML < NRP the result is more >> than 100. >> - In case I'm not making a mistake and the intention is the QoO >> to have a value of [0,100], the formulation can be changed into >> >> MIN( 100, (1-(ML-NRP)/(NRPoU-NRP))*100 ) >> > > You are correct, the value should be limited to the [0, 100] range. ( > https://github.com/domoslabs/QoOID/issues/9) > > Thank you for the valuable feedback and insightful comments! > > Best regards, > Bjørn > > >> >> Regards, >> ------------------------------ >> *From:* ippm <ippm-bounces@ietf.org> on behalf of Tommy Pauly <tpauly= >> 40apple.com@dmarc.ietf.org> >> *Sent:* 16 January 2024 20:13 >> *To:* IETF IPPM WG (ippm@ietf.org) <ippm@ietf.org> >> *Subject:* [ippm] Adoption call for draft-olden-ippm-qoo-02 >> >> Hello IPPM, >> >> This email starts a working group adoption call for "Quality of Outcome” >> (draft-olden-ippm-qoo). >> >> https://datatracker.ietf.org/doc/draft-olden-ippm-qoo/ >> https://www.ietf.org/archive/id/draft-olden-ippm-qoo-02.html >> >> The call will last for 3 weeks, and end on *Tuesday, February 6*. Please >> reply to this email with your review comments and indicate if you support >> adopting this work. >> >> Please note that we did a previous adoption call that did not receive >> sufficient feedback. At the last meeting at IETF 118, we did have a good >> amount of comments and questions, so please do reply to this email if you >> have reviewed the document. >> >> Thanks, >> Tommy & Marcus >> >> Information in this email including any attachments may be privileged, >> confidential and is intended exclusively for the addressee. The views >> expressed may not be official policy, but the personal views of the >> originator. If you have received it in error, please notify the sender by >> return e-mail and delete it from your system. You should not reproduce, >> distribute, store, retransmit, use or disclose its contents to anyone. >> _______________________________________________ >> ippm mailing list >> ippm@ietf.org >> https://www.ietf.org/mailman/listinfo/ippm >> > > > -- > Bjørn Ivar Teigen, Ph.D. > Head of Research > +47 47335952 | bjorn@domos.ai | www.domos.ai > [image: https://www.understandinglatency.com/] > <https://www.understandinglatency.com/> > _______________________________________________ > ippm mailing list > ippm@ietf.org > https://www.ietf.org/mailman/listinfo/ippm > -- 40 years of net history, a couple songs: https://www.youtube.com/watch?v=D9RGX6QFm5E Dave Täht CSO, LibreQos
- [ippm] Adoption call for draft-olden-ippm-qoo-02 Tommy Pauly
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Kevin Smith, Vodafone
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Mehmet Sukru Kuran
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Dave Taht
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Michael Welzl
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Greg Mirsky
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Greg Mirsky
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Tommy Pauly
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… LUIS MIGUEL CONTRERAS MURILLO
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Will Hawkins
- Re: [ippm] Adoption call for draft-olden-ippm-qoo… Bjørn Ivar Teigen