Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

Greg Mirsky <gregimirsky@gmail.com> Thu, 06 January 2022 23:00 UTC

Return-Path: <gregimirsky@gmail.com>
X-Original-To: ippm@ietfa.amsl.com
Delivered-To: ippm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7DCD03A089F for <ippm@ietfa.amsl.com>; Thu, 6 Jan 2022 15:00:25 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level:
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nPeVSsSLOfBW for <ippm@ietfa.amsl.com>; Thu, 6 Jan 2022 15:00:21 -0800 (PST)
Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BBA643A087D for <ippm@ietf.org>; Thu, 6 Jan 2022 15:00:20 -0800 (PST)
Received: by mail-ed1-x530.google.com with SMTP id z9so15135833edm.10 for <ippm@ietf.org>; Thu, 06 Jan 2022 15:00:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HyFHTeMHlsP1WYKPW1vodcoI18kcGyZ1XNliIhiYrz4=; b=YYgSP2ByKrgM5oiFY9vYQ9nEybQK6a5HkPBK2pBbsbTNPKoSoU7jp8mpOonL2XIk5+ aqj1x+6QpgNXvGw8ppDrKEAgiA+A/vOVceuHH/AYU5VW8EGRODZ8/FzTWGymVMUOJc1W XpUV6d2LEMj2jG1br+uE84noq73eHiKlRzeo7iCbYnBHuAAlxQY0DWfgqoFI/QtNDJUr pkhd9PVaRpem2cmBn61IXWUoapG0RoZ41SI4aDgwKEMAZ6Xdzd24oa2J0vUlH6v/CiTq b29ySV7yHb6jog9jmsq9KEiEAHua6iiO8QCaZDuvQsmCvkRsZWRp86siiHRYWoa4uCDZ 2XzQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HyFHTeMHlsP1WYKPW1vodcoI18kcGyZ1XNliIhiYrz4=; b=IXtDPH1KcM1CdngTCATL2f+Ue/ugP6GD4eTSUSmzYUJgIsNRr892ySTK7m0y72QiVv K+b9h9P7PMVUDut1RGRsZKggnS0er/k2sWzHDO/BUzhKBsEZItxlGoi8veSJQoecUNri RUnryHZkn9sn32dPuBCF3Xj460rtUlpeNS7cj+CkbE9Yz7KTIhLb7qWHe1JzZ5J8ORuz cmV5NqdzzSszgDoe728sYbJKAwBR5mINV1B2ft8S2cPxI1aNqVv3sFB9E14wvtcemZnr OdZaaNSF3b0fowy2shlWD/izqmNYGsQDJAnfdSOu8BPdpQv15rI6NiBZtVEaxHCEtLLS NPfA==
X-Gm-Message-State: AOAM533s9ZHcUyoXNoYxg75Rwnf7jhLObNFthQ4JKvjxcgeNue9zgLqA 63I4ZCQ4S8CeWXyVg3sPZSxImtoy6lfnvNEEHIa3eeiHm1o=
X-Google-Smtp-Source: ABdhPJwqXgD1h8vwOq3sKhOOLYpmb/kvh0ICZ62ZheCMxJHcuOBhvAu1slV5a3vrowC4gm6vWMU5ryaLP60oZlw61No=
X-Received: by 2002:a05:6402:1d81:: with SMTP id dk1mr60583007edb.328.1641510018179; Thu, 06 Jan 2022 15:00:18 -0800 (PST)
MIME-Version: 1.0
References: <AM0PR07MB4131542BCD0A6DE3F82F1E19E26D9@AM0PR07MB4131.eurprd07.prod.outlook.com> <CA+RyBmU_j9-vR+BnjvhKCDuaWYPZ_Ym96yUJPX0LhGihfsp1ng@mail.gmail.com> <3DC3F6B6-229E-46C6-BD84-2A6A7FE6DD48@apple.com>
In-Reply-To: <3DC3F6B6-229E-46C6-BD84-2A6A7FE6DD48@apple.com>
From: Greg Mirsky <gregimirsky@gmail.com>
Date: Thu, 06 Jan 2022 15:00:06 -0800
Message-ID: <CA+RyBmV_+yysquiZ=2PwB=oaqeJmfKV39c3=GE9sxWkb4qTM=Q@mail.gmail.com>
To: Christoph Paasch <cpaasch@apple.com>
Cc: Marcus Ihlar <marcus.ihlar=40ericsson.com@dmarc.ietf.org>, IETF IPPM WG <ippm@ietf.org>
Content-Type: multipart/alternative; boundary="0000000000008766bc05d4f1d5eb"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/WaBKz_iNyGqF_ytlp_wA3jL2fiM>
Subject: Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness
X-BeenThere: ippm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ippm>, <mailto:ippm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm/>
List-Post: <mailto:ippm@ietf.org>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ippm>, <mailto:ippm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 06 Jan 2022 23:00:26 -0000

Hi Christoph,
a happy and healthy New Year to you and All!
Thank you for your kind consideration of my notes and detailed responses.
Please find my follow-up notes in-line below under the GIM>> tag.

Regards,
Greg

On Wed, Jan 5, 2022 at 10:52 AM Christoph Paasch <cpaasch@apple.com> wrote:

> Hello Greg,
>
> thanks for your comments. Please see inline:
>
> On Dec 22, 2021, at 11:43 PM, Greg Mirsky <gregimirsky@gmail.com> wrote:
>
> Dear Marcus, Authors, et al,
> apologies for the belated response.
> I've read the draft and have some comments to share with you:
>
>    - as I understand it, the proposed new responsiveness metric is viewed
>    as the single indicator of a bufferbloat condition in a network. As I
>    recall, the discussion at Measuring Network Quality for End-Users workshop
>    and on the mailing list
>    <https://mailarchive.ietf.org/arch/browse/network-quality-workshop/?gbt=1&index=cuW_1lh4DD22V28EvlPFB_NjjZY>
>    indicated, that there’s no consensus on what behaviors, symptoms can
>    reliably signal the bufferbloat.
>
> We are not trying for this responsiveness metric to be "the single
> indicator of bufferbloat". Bufferbloat can be measured in many different
> number of ways. And each of these will produce a correct, but a different
> result. Thus, "bufferbloat" is whatever the methodology tries to detect.
>
> Let me give an example of two methodologies that are both correct but both
> will produce entirely different numbers :
>
> If we would decide to generate the load by flooding the network with UDP
> traffic from a specific 4-tuple and measure latency with parallel ICMP
> pings. Then, on a over-buffered FIFO queue we would measure huge latencies
> (thus correctly expose bufferbloat), while on a FQ-codel queue we would not
> measure any bufferbloat.
>
> If on the other hand, the load-generating traffic is changing the
> source-port for every single UDP-packet, then in both the FIFO-queue and
> the FQ-codel queue we will measure huge amounts of bufferbloat.
>
> Thus, these two methods both produced correct results but with hugely
> different numbers in the FQ-codel case. [1]
>
> Now, while both methods measure some variant of bufferbloat, they both
> don't measure a realistic usage of the network.
>
GIM>> Thank you for the insights. It seems to me that what the method can
demonstrate is rather the level of efficiency of the AQM in the network for
a particular class of applications.

>
>
> That is why the "Responsiveness under working conditions" tries to clearly
> specify how the load is generated and how the latency is being measured.
> And it does not measure "bufferbloat" but it measures "responsiveness under
> working conditions" based on the methodology that is being used (using
> HTTP/2 or HTTP/3, multiple flows, ...). It does expose bufferbloat which
> can happen in the network. It also exposes certain server-side behaviors
> that can cause (huge amounts of) additional latency - those behaviors are
> typically not called "bufferbloat".
>
GIM>> Thank you for pointing out that the result of the RTT measurement has
two contributing factors - network and server. It seems worth enhancing the
method to localize each contribution and measure them separately.

>
>
>
>
>    - It seems that it would be reasonable to first define what is being
>    measured, characterized by the responsiveness metric. Having a document
>    that discusses and defines the bufferbloat would be great.
>
> I agree that there is a lack of definition for what "bufferbloat" really
> is.
>
> The way we look at "responsiveness under working conditions" is that it
> measures the latency in conditions that may realistically happen in
> worst-case scenarios with end-users/implementations that are non-malicious
> (non-malicious to exclude the UDP-flooding scenario).
>
> Thus, I assume we should make a better job at explaining this. The lack of
> a formal definition of "bufferbloat" doesn't help and thus we are indeed
> using this term a bit freely in the current draft. We will improve the
> Introduction to better set the stage (
> https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/31
> ).
>
>
>    - It seems like in the foundation of the methodology described in the
>    draft lies the assumption that without adding new flows the
>    available bandwidth is constant, does not change. While that is mostly the
>    case, there are technologies that behave differently and may change
>    bandwidth because of the outside conditions. Some of these behaviors of
>    links with variable discrete bandwidth are discussed in, for example, RFC
>    8330 <https://datatracker.ietf.org/doc/rfc8330/> and RFC 8625
>    <https://datatracker.ietf.org/doc/rfc8625/>.
>
> I'm not sure I entirely understand your comment. But let me explain why we
> are gradually adding new flows:
>
> 1. TCP-implementations have usually a fixed limit for the upper bound of
> the receive window. In some networks that upper bound is lower than the BDP
> of the network. Thus, the only way to reach full capacity is by having
> multiple flows.
> 2. Having multiple connections allows to quicker achieve full capacity in
> high-RTT networks and thus speeds up the test-duration.
> 3. In some networks with "random" packet-loss, congestion-control may come
> in the way of achieving full capacity. Again, multiple flows will work
> around that.
>
GIM>> I might have asked several questions at once. Let me clarify what I
am looking for:

   - As I understand the method of creating the "working conditions in a
   network" is based on certain assumptions. First, seems is that the
   bandwidth is symmetrical between the measurement points. Second, that the
   bandwidth doesn't change for the duration of the measurement session.
   AFAIK, in the access networks, both are not necessarily always the case.
   - On the other hand, I might have missed how the method of creating the
   "working conditions" guarantees a symmetrical load between the measurement
   points.


>
>    - Then, I find the motivation not to use time units to express the
>    responsiveness metric not convincing:
>
>    "Latency" is a poor measure of responsiveness, since it can be hard
>    for the general public to understand.  The units are unfamiliar
>    ("what is a millisecond?") and counterintuitive ("100 msec - that
>    sounds good - it's only a tenth of a second!").
>
>
> Can you expand on what exactly is not convincing to you? Do you think that
> people will mis-understand the metric or that milli-seconds is the right
> way to communicate responsiveness to the general public?
>
GIM>> Let me try. We know packet delay requirements for AR, VR
applications. I believe that gamers are familiar with these numbers too.
The same is likely the case for the industrial automation use cases served,
for example, by Deterministic Networking.

>
>
> Thanks a lot,
> Christoph
>
> [1] And there are many networks that prioritize ICMP pings, thus we could
> observe even more different results based on what protocol is used to
> measure the latency.
>
>
> On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar <marcus.ihlar=
> 40ericsson.com@dmarc.ietf.org> wrote:
>
>> Hi IPPM,
>>
>>
>>
>> This email starts an adoption call for draft-cpaasch-ippm-responsiveness,
>> "Responsiveness under Working Conditions”. This document specifies the “RPM
>> Test” for measuring user experience when the network is fully loaded. The
>> intended status of the document is Experimental.
>>
>>
>>
>> https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/
>>
>> https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01
>>
>>
>>
>> This adoption call will last until *Monday, December 20*. Please review
>> the document, and reply to this email thread to indicate if you think IPPM
>> should adopt this document.
>>
>>
>>
>> BR,
>>
>> Marcus
>>
>>
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm
>>
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm
>
>
>