Re: [E!] : Re: Draft: QUIC Throughput Testing

Ian Swett <> Wed, 22 September 2021 14:13 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 10AF63A2350 for <>; Wed, 22 Sep 2021 07:13:36 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -17.997
X-Spam-Status: No, score=-17.997 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 4HuDfTSe1Z6C for <>; Wed, 22 Sep 2021 07:13:30 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::732]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 0017D3A2352 for <>; Wed, 22 Sep 2021 07:13:29 -0700 (PDT)
Received: by with SMTP id bk29so10013992qkb.8 for <>; Wed, 22 Sep 2021 07:13:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5ZpvAWQM59zsjMLv7PeSuPg4xuQba+5jdh3VSWoQnaw=; b=GZTp2aF0X48RAMh5ZPBdXs3Ie3Ew90Q2bEBE+STpjGVqucT+zlfM3LtsMrZnt/BZlv 7ou/Epw6a8A9dM8ND9Z8avB6boswA0ZbZknC93W+cG9RvQHdCMZvdGwdiizueQIH6zfO tGiQUgR9rJOiT/1uT+hCFogsp/AkHfLp71lYBSHdLom8+OWKzXVKp5tyosJOgNQTmnSX tnW2EcKGmsO5kw2eNs7GbP1Ne5ZPYzZXpl9MGvAh9btXvL14A0pZ3c5Wq9lu9ZZeKJSG m9yqLei1NE1+Z/My+TJ2kr+L5GZ/wPyXovQUDYauX4lkZrBX1xIVsxXVIQfhoO6pRMvJ 9HGA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5ZpvAWQM59zsjMLv7PeSuPg4xuQba+5jdh3VSWoQnaw=; b=ACNj0Gq9tkzivSz3pYSNWYUKgT5G/6V785picT5Ssxi0DbWjzpIFxbtJpkiUiBoXUM pCcWWMx9vjJgjUF5EOratogAujt2U7220YbIQPyLyhP9nSPi+zbqdCaHRTJEcIE3l2Wd T3votOf8JzZAklu+pbFg7wOgFt5dX1pzVkEYD0+NFOPBFVBFwcaDN/gtzy09peXNk0iJ v5qiBqS1P+fZP6yozCSb4Y294AjEoEBYQFohM0A6M7qCa0ozCExTtbR1mHGfFS/pxW4K gn+t/0/i7jzA2ynxcxuEgM4GubSA4QXdfd7PclENCIj5l3RJnV+U0yj0oP3sXx+pkoId 5RWQ==
X-Gm-Message-State: AOAM533OxS8BjMoQMetgHZ4EcGzZFKk5eQPrnpR4CWosNQhP6Xjs1VXa Zfvwtp7QMSbM7bhSagwnmnSDHEhf3buiHLy819qAuQ==
X-Google-Smtp-Source: ABdhPJwfbJNXI+JZ/3ioWPLsGeCvzCFDJNJRebTwc+g0zW3bsXOpTKZmpcOJTjHAxlaE/ulOtGiXtvnRXw8oTwl0TWU=
X-Received: by 2002:a25:d2c8:: with SMTP id j191mr36912589ybg.429.1632320006777; Wed, 22 Sep 2021 07:13:26 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <> <>
In-Reply-To: <>
From: Ian Swett <>
Date: Wed, 22 Sep 2021 10:13:15 -0400
Message-ID: <>
Subject: Re: [E!] : Re: Draft: QUIC Throughput Testing
To: Kévin Corre <>
Cc: Tom Jones <>, "" <>
Content-Type: multipart/alternative; boundary="0000000000002e2bfc05cc961ee5"
Archived-At: <>
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 22 Sep 2021 14:13:36 -0000

On Wed, Sep 22, 2021 at 9:57 AM Kévin Corre <> wrote:

> @Tom Jones <>
> > Have you seen draft-banks-quic-performance? I don't see a reference to
> > it in your document. This protocol has implementations using a number of
> > quic different libraries.
> @Ian Swett <>
> > I would suggest looking at the ACK frequency draf
> <>t,
> or some similar approaches.  Otherwise it can be very difficult to achieve
> comparable speeds as TCP.
> Hi Tom, Ian,
> Thanks for your comments.
> I've read draft Banks before starting my work. Actually, just to cite
> myself discussing that with Lucas Pardue: " Regarding RFC6349 and
> draft-banks, it's true that there is a similarity in the fact they stress a
> QUIC implementation and work similarly with clients opening multiple
> connections to a server. The major difference between the two is that their
> intent and metrics differ.
> Banks states in section 3 that "Generally, the goal of all these
> performance tests is to measure the maximum load that can be achieved with
> the given QUIC implementation and hardware configuration. To that end, the
> network is not expected to be the bottleneck in any of these tests. To
> achieve that, the appropriate network hardware must be used so as to not
> limit throughput". Conversely, the "primary focus of [RFC6349] methodology
> is managed business-class IP networks, e.g. those Ethernet-terminated
> services for wich organizations are provided an SLA from the network
> provider. Because of the SLA, the expectation is that the TCP Throughput
> should achieve the guaranteed bandwidth". The RFC6349 methodology is to 1.
> measure the path MTU, 2. baseline RTT and bottleneck bandwidth, 3. do a TCP
> Throughput test "to baseline network performance".
> The metrics captured also show this difference in intent. RFC6349 measures
> transfer time ratio over an ideal transfer time depending on the network,
> TCP efficiency depending on the loss rate of the network, and buffer delay
> which depends on the baselined RTT. Whereas Banks performance scenarios
> measure metrics related to the server performance (e.g. requests per
> second)."
> Now that you both mention performances, I can see how a QUIC Throughput
> test software intended for testing network should be first validated using
> recommendations in draft-banks and other recommended optimizations in the
> goal of demonstrating performance rather than interoperability. My draft
> states that parameters are specified to optimize the QUIC Throughput. At
> the moment it's more a declaration of intent taken from RFC6349,  but I
> guess I could be more specific and more reading/experimentation is probably
> required to get a better picture of the impact parameters have on actual
> network performances.
> @Ian Swett <>
> Thanks for your suggestions. I came across fastly's post but now better
> realise I should take a look at it again 😉.
> While reading the ACK frequency draft, I noted a difference in exponents
> limiting min_ack_delay and max_ack_delay.
> min_ack_delay (0xff02de1a):
> A variable-length integer representing the minimum amount of time in
> microseconds by which the endpoint can delay an acknowledgement. *Values
> of 2^24 or greater are invalid,*
> *An endpoint's min_ack_delay MUST NOT be greater than its max_ack_delay. *
> At the same time, RFC9000 states that
>    - max_ack_delay (0x0b):  [...] *Values of 214* or greater are invalid
> Thanks for noticing.  That was issue 44
<> which has been fixed in
the editors copy, but we haven't published an updated version yet.  We'll
push an update quite soon I expect, ideally in the next week.

> Regards,
> Kevin
> ------------------------------
> *From:* Ian Swett <>
> *Sent:* Wednesday, September 22, 2021 1:05 PM
> *To:* Tom Jones <>
> *Cc:* Kévin Corre <>; <>
> *Subject:* [E!] : Re: Draft: QUIC Throughput Testing
> I would suggest looking at the ACK frequency draf
> <>t,
> or some similar approaches.  Otherwise it can be very difficult to achieve
> comparable speeds as TCP.
> See Fastly's nice post
> <>
> about what it took to get comparable performance in QUIC as TCP.
> On Wed, Sep 22, 2021 at 6:06 AM Tom Jones <> wrote:
> On Wed, Sep 22, 2021 at 09:15:58AM +0000, Kévin Corre wrote:
> > Hi everyone,
> >
> > Our team at EXFO is working on adapting the TCP Throughput Testing
> methodology described in RFC6349 to QUIC. We just published a draft that
> can be found at
> <>
> > The draft follows the same organization as the RFC6349 and discuss what
> should be changed to support QUIC Throughput Testing.
> > The methodology is interested in measuring the achievable QUIC
> throughput and efficiency of a managed network when the connection is at an
> equilibrium. Although, it could also be used for un-managed networks.
> >
> > We are particularly looking for comments on the way we use the protocol
> to measure the network's QUIC Throughput but also feedback from potential
> users of such a test.
> > For instance, our draft closely follows the RFC6349 methodology, but
> since QUIC introduces new features, it would be interesting to gauge
> interest in extending the methodology to test features such as 0-RTT effect.
> >
> > Comments would be gladly appreciated, so let us know what you think.
> >
> Hi Kévin,
> Have you seen draft-banks-quic-performance? I don't see a reference to
> it in your document. This protocol has implementations using a number of
> quic different libraries.
> There is also a perf channel on the quic slack, please ask off list if
> you require an invite.
> <>
> - Tom