Re: [E!] : Re: Draft: QUIC Throughput Testing

Ian Swett <ianswett@google.com> Wed, 22 September 2021 14:13 UTC

Return-Path: <ianswett@google.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 10AF63A2350 for <quic@ietfa.amsl.com>; Wed, 22 Sep 2021 07:13:36 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.997
X-Spam-Level:
X-Spam-Status: No, score=-17.997 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4HuDfTSe1Z6C for <quic@ietfa.amsl.com>; Wed, 22 Sep 2021 07:13:30 -0700 (PDT)
Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0017D3A2352 for <quic@ietf.org>; Wed, 22 Sep 2021 07:13:29 -0700 (PDT)
Received: by mail-qk1-x732.google.com with SMTP id bk29so10013992qkb.8 for <quic@ietf.org>; Wed, 22 Sep 2021 07:13:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5ZpvAWQM59zsjMLv7PeSuPg4xuQba+5jdh3VSWoQnaw=; b=GZTp2aF0X48RAMh5ZPBdXs3Ie3Ew90Q2bEBE+STpjGVqucT+zlfM3LtsMrZnt/BZlv 7ou/Epw6a8A9dM8ND9Z8avB6boswA0ZbZknC93W+cG9RvQHdCMZvdGwdiizueQIH6zfO tGiQUgR9rJOiT/1uT+hCFogsp/AkHfLp71lYBSHdLom8+OWKzXVKp5tyosJOgNQTmnSX tnW2EcKGmsO5kw2eNs7GbP1Ne5ZPYzZXpl9MGvAh9btXvL14A0pZ3c5Wq9lu9ZZeKJSG m9yqLei1NE1+Z/My+TJ2kr+L5GZ/wPyXovQUDYauX4lkZrBX1xIVsxXVIQfhoO6pRMvJ 9HGA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5ZpvAWQM59zsjMLv7PeSuPg4xuQba+5jdh3VSWoQnaw=; b=ACNj0Gq9tkzivSz3pYSNWYUKgT5G/6V785picT5Ssxi0DbWjzpIFxbtJpkiUiBoXUM pCcWWMx9vjJgjUF5EOratogAujt2U7220YbIQPyLyhP9nSPi+zbqdCaHRTJEcIE3l2Wd T3votOf8JzZAklu+pbFg7wOgFt5dX1pzVkEYD0+NFOPBFVBFwcaDN/gtzy09peXNk0iJ v5qiBqS1P+fZP6yozCSb4Y294AjEoEBYQFohM0A6M7qCa0ozCExTtbR1mHGfFS/pxW4K gn+t/0/i7jzA2ynxcxuEgM4GubSA4QXdfd7PclENCIj5l3RJnV+U0yj0oP3sXx+pkoId 5RWQ==
X-Gm-Message-State: AOAM533OxS8BjMoQMetgHZ4EcGzZFKk5eQPrnpR4CWosNQhP6Xjs1VXa Zfvwtp7QMSbM7bhSagwnmnSDHEhf3buiHLy819qAuQ==
X-Google-Smtp-Source: ABdhPJwfbJNXI+JZ/3ioWPLsGeCvzCFDJNJRebTwc+g0zW3bsXOpTKZmpcOJTjHAxlaE/ulOtGiXtvnRXw8oTwl0TWU=
X-Received: by 2002:a25:d2c8:: with SMTP id j191mr36912589ybg.429.1632320006777; Wed, 22 Sep 2021 07:13:26 -0700 (PDT)
MIME-Version: 1.0
References: <DM4PR11MB53417177BDF29E29E7FA57F299A29@DM4PR11MB5341.namprd11.prod.outlook.com> <20210922100925.GA16294@tom-desk.erg.abdn.ac.uk> <CAKcm_gNyDVG7jd7pcAxvBeO0bOTAiotD6SB7SbjvD66ZT2Q5zA@mail.gmail.com> <DM4PR11MB5341FFE3C382122CC25088D299A29@DM4PR11MB5341.namprd11.prod.outlook.com>
In-Reply-To: <DM4PR11MB5341FFE3C382122CC25088D299A29@DM4PR11MB5341.namprd11.prod.outlook.com>
From: Ian Swett <ianswett@google.com>
Date: Wed, 22 Sep 2021 10:13:15 -0400
Message-ID: <CAKcm_gMGGaieg7rWawZ+nmL4t2RheYA+TbKCwDCi+qG6joh9cQ@mail.gmail.com>
Subject: Re: [E!] : Re: Draft: QUIC Throughput Testing
To: =?UTF-8?B?S8OpdmluIENvcnJl?= <kevin.corre@exfo.com>
Cc: Tom Jones <tom@erg.abdn.ac.uk>, "quic@ietf.org" <quic@ietf.org>
Content-Type: multipart/alternative; boundary="0000000000002e2bfc05cc961ee5"
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/kFob1ZEaZ5080bMYUYrLY0x_WEA>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 22 Sep 2021 14:13:36 -0000

On Wed, Sep 22, 2021 at 9:57 AM Kévin Corre <kevin.corre@exfo.com> wrote:

> @Tom Jones <tom@erg.abdn.ac.uk>
> > Have you seen draft-banks-quic-performance? I don't see a reference to
> > it in your document. This protocol has implementations using a number of
> > quic different libraries.
> @Ian Swett <ianswett@google.com>
> > I would suggest looking at the ACK frequency draf
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SymOgANn9Tt737DFff%2FSXqpeqpHHpDpSoZq2AkDMOkk%3D&reserved=0>t,
> or some similar approaches.  Otherwise it can be very difficult to achieve
> comparable speeds as TCP.
>
> Hi Tom, Ian,
> Thanks for your comments.
>
> I've read draft Banks before starting my work. Actually, just to cite
> myself discussing that with Lucas Pardue: " Regarding RFC6349 and
> draft-banks, it's true that there is a similarity in the fact they stress a
> QUIC implementation and work similarly with clients opening multiple
> connections to a server. The major difference between the two is that their
> intent and metrics differ.
>
> Banks states in section 3 that "Generally, the goal of all these
> performance tests is to measure the maximum load that can be achieved with
> the given QUIC implementation and hardware configuration. To that end, the
> network is not expected to be the bottleneck in any of these tests. To
> achieve that, the appropriate network hardware must be used so as to not
> limit throughput". Conversely, the "primary focus of [RFC6349] methodology
> is managed business-class IP networks, e.g. those Ethernet-terminated
> services for wich organizations are provided an SLA from the network
> provider. Because of the SLA, the expectation is that the TCP Throughput
> should achieve the guaranteed bandwidth". The RFC6349 methodology is to 1.
> measure the path MTU, 2. baseline RTT and bottleneck bandwidth, 3. do a TCP
> Throughput test "to baseline network performance".
> The metrics captured also show this difference in intent. RFC6349 measures
> transfer time ratio over an ideal transfer time depending on the network,
> TCP efficiency depending on the loss rate of the network, and buffer delay
> which depends on the baselined RTT. Whereas Banks performance scenarios
> measure metrics related to the server performance (e.g. requests per
> second)."
>
> Now that you both mention performances, I can see how a QUIC Throughput
> test software intended for testing network should be first validated using
> recommendations in draft-banks and other recommended optimizations in the
> goal of demonstrating performance rather than interoperability. My draft
> states that parameters are specified to optimize the QUIC Throughput. At
> the moment it's more a declaration of intent taken from RFC6349,  but I
> guess I could be more specific and more reading/experimentation is probably
> required to get a better picture of the impact parameters have on actual
> network performances.
>
> @Ian Swett <ianswett@google.com>
> Thanks for your suggestions. I came across fastly's post but now better
> realise I should take a look at it again 😉.
> While reading the ACK frequency draft, I noted a difference in exponents
> limiting min_ack_delay and max_ack_delay.
> min_ack_delay (0xff02de1a):
>
> A variable-length integer representing the minimum amount of time in
> microseconds by which the endpoint can delay an acknowledgement. *Values
> of 2^24 or greater are invalid,*
> *An endpoint's min_ack_delay MUST NOT be greater than its max_ack_delay. *
> At the same time, RFC9000 states that
>
>    - max_ack_delay (0x0b):  [...] *Values of 214* or greater are invalid
>
>
> Thanks for noticing.  That was issue 44
<https://github.com/quicwg/ack-frequency/issues/44> which has been fixed in
the editors copy, but we haven't published an updated version yet.  We'll
push an update quite soon I expect, ideally in the next week.


> Regards,
> Kevin
>
>
> ------------------------------
> *From:* Ian Swett <ianswett@google.com>
> *Sent:* Wednesday, September 22, 2021 1:05 PM
> *To:* Tom Jones <tom@erg.abdn.ac.uk>
> *Cc:* Kévin Corre <kevin.corre@exfo.com>om>; quic@ietf.org <quic@ietf.org>
> *Subject:* [E!] : Re: Draft: QUIC Throughput Testing
>
> I would suggest looking at the ACK frequency draf
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SymOgANn9Tt737DFff%2FSXqpeqpHHpDpSoZq2AkDMOkk%3D&reserved=0>t,
> or some similar approaches.  Otherwise it can be very difficult to achieve
> comparable speeds as TCP.
>
> See Fastly's nice post
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.fastly.com%2Fblog%2Fmeasuring-quic-vs-tcp-computational-efficiency&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bMAieBa9ZkZFyJbobiCFRyy54TiUTcXqeCKPCO279uA%3D&reserved=0>
> about what it took to get comparable performance in QUIC as TCP.
>
> On Wed, Sep 22, 2021 at 6:06 AM Tom Jones <tom@erg.abdn.ac.uk> wrote:
>
> On Wed, Sep 22, 2021 at 09:15:58AM +0000, Kévin Corre wrote:
> > Hi everyone,
> >
> > Our team at EXFO is working on adapting the TCP Throughput Testing
> methodology described in RFC6349 to QUIC. We just published a draft that
> can be found at
> https://datatracker.ietf.org/doc/html/draft-corre-quic-throughput-testing
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-corre-quic-throughput-testing&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450428506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5aJ79pllQs2zckYUPyvj1ccNNg5kQE8tsSSSTwpluX8%3D&reserved=0>
> > The draft follows the same organization as the RFC6349 and discuss what
> should be changed to support QUIC Throughput Testing.
> > The methodology is interested in measuring the achievable QUIC
> throughput and efficiency of a managed network when the connection is at an
> equilibrium. Although, it could also be used for un-managed networks.
> >
> > We are particularly looking for comments on the way we use the protocol
> to measure the network's QUIC Throughput but also feedback from potential
> users of such a test.
> > For instance, our draft closely follows the RFC6349 methodology, but
> since QUIC introduces new features, it would be interesting to gauge
> interest in extending the methodology to test features such as 0-RTT effect.
> >
> > Comments would be gladly appreciated, so let us know what you think.
> >
>
> Hi Kévin,
>
> Have you seen draft-banks-quic-performance? I don't see a reference to
> it in your document. This protocol has implementations using a number of
> quic different libraries.
>
> There is also a perf channel on the quic slack, please ask off list if
> you require an invite.
>
> https://datatracker.ietf.org/doc/html/draft-banks-quic-performance-00.txt
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-banks-quic-performance-00.txt&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450428506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BXdEofehdUsCMCixoA%2BD1QmAnrhfizVsCrnrk1o4mkY%3D&reserved=0>
>
> - Tom
>
>