Re: [ippm] Adoption call for draft-olden-ippm-qoo-02

Dave Taht <dave.taht@gmail.com> Wed, 31 January 2024 11:02 UTC

Return-Path: <dave.taht@gmail.com>
X-Original-To: ippm@ietfa.amsl.com
Delivered-To: ippm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4DFFFC18DB97 for <ippm@ietfa.amsl.com>; Wed, 31 Jan 2024 03:02:54 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.095
X-Spam-Level:
X-Spam-Status: No, score=-2.095 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z6YjIj3YnNH2 for <ippm@ietfa.amsl.com>; Wed, 31 Jan 2024 03:02:47 -0800 (PST)
Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C384BC15108E for <ippm@ietf.org>; Wed, 31 Jan 2024 03:02:47 -0800 (PST)
Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-5d8bc1caf00so3152722a12.1 for <ippm@ietf.org>; Wed, 31 Jan 2024 03:02:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706698967; x=1707303767; darn=ietf.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=mxB3tBC1zEmnxOQWpOLfsxqlCCO51KOK02j3OYXcDxo=; b=Mt4t8MUA2pFTMWaIrbALaQSCka/aiXcq6EeXODr8acYMH8OypDsE8gAb/rG8KBxP3x Sxpzl3tKiga3dUtP1CyTOCC1VI2VHSHg8Pb1vgFhIMx28FxIbfCcgGQJsr5oWoES9XCQ HyVR97Vl2U/8XdfTnDtl6vG/DLJSvYcrzieuznwOa8eZJoT903LDXc8tkaLvZVI/nJXG ETJwduQloSELWr29gmbdKjeqaJVznIE8LaNUA0Zt6v2NhECn2U6mSTqsOp4ODEmUPexF xqyVX+21/Q/tzSIznAqSFIeM+PYkm8CbYUBLzZjsD76Pl84TV4nEjxeqNBJu9pNQE61b QLfQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706698967; x=1707303767; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mxB3tBC1zEmnxOQWpOLfsxqlCCO51KOK02j3OYXcDxo=; b=e9Yp6qy3qE+PGhoSZWDNm6PsTpIGk89/a81JXaD8Kg4YdqHGpEOQcLrhSA3HL2HA6H YVPCJK9FbAjt5CcFexrhM1d8o4qe/lSetVDY9XtrmVn8RceXWiykczAmvtzRavt5Z8NW UtJHazuvP1aaF+gj8Mo0n+q5yGH4/yXKRtPJcoTE8n6T/onynZ6zm2Ra/MICz4PLJDWl R+zw31ACzb6PoDpcw1gmZ93AGXCaX6Y7SVmy4AlV04MQ1aK48mGDZgLmNBrnU4+jZf39 qHk8NiNimbHmhlIAhOI4qZvHVh/DOrG4P4yrUYCWwElcsUnLIcNePNnWAHzNKA7xvL+Q jLnA==
X-Gm-Message-State: AOJu0YzcZQFv9CxsLtmG4lsN4OtEBpsA+X5/0fgfPKjGDRwktPQ+ZJOF vu077fU+Bf+hNWPJ0pD0xhTKuqgDBR5gyYalTSbgtXWdjc6ktT/JUF+vd8jG72Q3iAvY+zhBUqu gBQLHZHbZJVzPAlmeqd/ezH7KYXU0rhIN
X-Google-Smtp-Source: AGHT+IElmP9Gwgc+WYaxGwmJKqoBBVRaEPewwFcF9X56YNvl+6PpzJBamweaChFOVq6NjUANWBALDBbqjwvyDOAGLPs=
X-Received: by 2002:a05:6a21:38c5:b0:19c:7e9f:44c1 with SMTP id yk5-20020a056a2138c500b0019c7e9f44c1mr1096995pzb.30.1706698966988; Wed, 31 Jan 2024 03:02:46 -0800 (PST)
MIME-Version: 1.0
References: <DCD67FE3-AFC5-4689-89EF-66387949214C@apple.com> <VI1PR10MB16958312D2C72D737557042BE3752@VI1PR10MB1695.EURPRD10.PROD.OUTLOOK.COM> <CAKf5G6KxskTc6GAEkezRjP9YmbeJ2HLcUvgmX+hMktg201nTdw@mail.gmail.com>
In-Reply-To: <CAKf5G6KxskTc6GAEkezRjP9YmbeJ2HLcUvgmX+hMktg201nTdw@mail.gmail.com>
From: Dave Taht <dave.taht@gmail.com>
Date: Wed, 31 Jan 2024 06:02:34 -0500
Message-ID: <CAA93jw7i8tH4Zj9OAdUM6RHx6wmu7G98i6vsrD5WOEAhsPQsiA@mail.gmail.com>
To: Bjørn Ivar Teigen <bjorn@domos.no>
Cc: Mehmet Sukru Kuran <sukru.kuran@airties.com>, Tommy Pauly <tpauly=40apple.com@dmarc.ietf.org>, "IETF IPPM WG (ippm@ietf.org)" <ippm@ietf.org>
Content-Type: multipart/alternative; boundary="000000000000aa7c6706103bd1d6"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/mzvSSBEk9SdRhemxPO5Dqyxo2Yw>
Subject: Re: [ippm] Adoption call for draft-olden-ippm-qoo-02
X-BeenThere: ippm@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ippm>, <mailto:ippm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm/>
List-Post: <mailto:ippm@ietf.org>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ippm>, <mailto:ippm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 31 Jan 2024 11:02:54 -0000

Aside from the commonly cited 100ms goal ignoring a century's worth of
human factors research, I support adoption, if this method
could be extended down to numbers closer to 20ms (voip), 16ms (a frame), or
4ms (VR).

On Wed, Jan 24, 2024 at 8:13 AM Bjørn Ivar Teigen <bjorn@domos.no> wrote:

> Hi all,
>
> Thanks for the feedback Sukru,
>
> Please find my comments inline.
>
> On Mon, 22 Jan 2024 at 08:51, Mehmet Sukru Kuran <sukru.kuran@airties.com>
> wrote:
>
>> Hi all,
>>
>>  I thank all the contributors who have worked on "Quality of Outcome".
>> Below are my comments,
>>
>>  1. First of all, I really liked the idea and I think QoO is a very neat
>> metric for evaluation QoE without going into the pains of conducting a lot
>> of manual testing and try to rely on MOS values. Getting the required NRP
>> and NRPoU values for different applications considering different codecs,
>> buffering mechanisms, etc... might be tricky for mass deployment scenarios
>> but all in all I believe this is a very nice addition for measuring QoE of
>> QoS sensitive traffic.
>>
>>  2. Assume for an application, the NRP P99 is 100 ms, and NRPoU P99 is
>> 200 ms; when a particular "application experience" has a P99 is 110 ms, how
>> come can you say there will be 10% of lagging?  I mean, how can you show
>> that lagging is a linear function?
>>
>>    - I'm aware that showing this is a linear function (or any other type
>>    of function) is not easy at all. If there is a proof showing that such a
>>    linear function holds I believe it will increase the value of the IETF
>>    document to either explain it or refer to the proof in another document
>>    (etc. a paper).
>>    - In case there is no such proof at the moment, for the documentation
>>    it might be better to say that "There can be different functions and one
>>    classical implementation can be using a linear function."
>>    - In case there is no such proof at the moment, conducting a lot of
>>    tests with empirical data and back the claim with these empirical data or
>>    trying to prove that function F is the correct one can be very interesting
>>    and quite valuable.
>>
>> This is an excellent point and something I think merits more
> investigation. The assumption that the transition from perfect to useless
> is modeled well by a straight line has not been tested thoroughly.
> I've seen other work (such as the ITU-T model for voice quality) that
> model this transition with a more complex function. A straight line is a
> significant simplification from the point of view of designing network
> requirements, and simplicity has a value of it's own here. That said, model
> simplicity must of course be balanced with model accuracy.
>
>
>>
>> 3. Is there a systematic way of converting packet loss, max. delay, max.
>> jitter into P90, P99, P99,9 NRP and NRPoU values? Also, how to combine
>> throughput requirements into this QoO?
>>
>>    - You have an example in the document but the details are not clearly
>>    laid out. I believe such a conversion method, formulation would be pretty
>>    useful.
>>    - In the long run, if application vendors define P90, P99, P99,9 NRP
>>    and NRPoU values; this will stop being a problem. However, for a
>>    wide-spread use of QoO I believe such a "conversion formulations" will be
>>    very, very valuable.
>>
>> That's a very good point. I think we should aim to include a section in
> the document describing in some detail how to create network requirements.
> I've made an issue on the github page, here:
> https://github.com/domoslabs/QoOID/issues/7
>
>
>>
>>    -
>>
>> 4. Regarding the measurements to be used for building the CDF, I see that
>> the only limitation is "having at least 10 samples".
>>
>>    - I think just this requirement is just is too loose. On Page 4-5 you
>>    explained this not being ideal and in any report on QoO 3 variables must be
>>    available. However, even then I believe this requirement can be tightened
>>    after some test runs and trials.
>>    - There can be several suggested measurement patterns (e.g., a)
>>    1/sec; the measurement should be E2E including encoding/decoding; b) 1/10
>>    sec; the measurement should be from the home router to the server, etc...)
>>
>> The idea of making the requirement as loose as possible is to keep the
> barrier to adoption as low as possible. I think that is an important
> principle, but I get your point that we need to maintain a certain standard
> in order for the QoO results to be meaningful.
>
>
>>
>>    -
>>
>> 5. On page 7, there is a formulation as
>>
>>     QoO = min(ML, NRP, NRPoU) = (1-(ML-NRP)/(NRPoU-NRP))*100
>>
>>         - The document says mathematically these two formulations are
>> equal. However, I see that in many occurrences the two formulations give
>> different results.
>>         - Am I missing something here? or making an error?
>>
> It's possible we've caused some confusion with the notation here. The line
> min(ML, NRP, NRPoU) is indended to mean "find the point where the measured
> latency crosses each of the NRP - NRPoU lines, and choose the "worst one"
> as your QoO value". Sorry if that is a little obscurely worded - I've made
> an issue to clarify the explanation in the text (
> https://github.com/domoslabs/QoOID/issues/8)
>
>
>>
>> 6. On page 7, there is a formulation as
>>
>>     QoO = (1-(ML-NRP)/(NRPoU-NRP))*100
>>
>>         - This formulation "I thought" should give values between 100
>> (perfect) and 0 (unacceptable). However, if ML < NRP the result is more
>> than 100.
>>         - In case I'm not making a mistake and the intention is the QoO
>> to have a value of [0,100], the formulation can be changed into
>>
>>     MIN( 100, (1-(ML-NRP)/(NRPoU-NRP))*100 )
>>
>
> You are correct, the value should be limited to the [0, 100] range. (
> https://github.com/domoslabs/QoOID/issues/9)
>
> Thank you for the valuable feedback and insightful comments!
>
> Best regards,
> Bjørn
>
>
>>
>> Regards,
>> ------------------------------
>> *From:* ippm <ippm-bounces@ietf.org> on behalf of Tommy Pauly <tpauly=
>> 40apple.com@dmarc.ietf.org>
>> *Sent:* 16 January 2024 20:13
>> *To:* IETF IPPM WG (ippm@ietf.org) <ippm@ietf.org>
>> *Subject:* [ippm] Adoption call for draft-olden-ippm-qoo-02
>>
>> Hello IPPM,
>>
>> This email starts a working group adoption call for "Quality of Outcome”
>> (draft-olden-ippm-qoo).
>>
>> https://datatracker.ietf.org/doc/draft-olden-ippm-qoo/
>> https://www.ietf.org/archive/id/draft-olden-ippm-qoo-02.html
>>
>> The call will last for 3 weeks, and end on *Tuesday, February 6*. Please
>> reply to this email with your review comments and indicate if you support
>> adopting this work.
>>
>> Please note that we did a previous adoption call that did not receive
>> sufficient feedback. At the last meeting at IETF 118, we did have a good
>> amount of comments and questions, so please do reply to this email if you
>> have reviewed the document.
>>
>> Thanks,
>> Tommy & Marcus
>>
>> Information in this email including any attachments may be privileged,
>> confidential and is intended exclusively for the addressee. The views
>> expressed may not be official policy, but the personal views of the
>> originator. If you have received it in error, please notify the sender by
>> return e-mail and delete it from your system. You should not reproduce,
>> distribute, store, retransmit, use or disclose its contents to anyone.
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm
>>
>
>
> --
> Bjørn Ivar Teigen, Ph.D.
> Head of Research
> +47 47335952 | bjorn@domos.ai | www.domos.ai
> [image: https://www.understandinglatency.com/]
> <https://www.understandinglatency.com/>
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm
>


-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos