Re: [E!] : Re: Draft: QUIC Throughput Testing

[Ian Swett <[e-mail]>]

On Wed, Sep 22, 2021 at 9:57 AM Kévin Corre <[e-mail]> wrote:

> @Tom Jones <[e-mail]>
> > Have you seen draft-banks-quic-performance? I don't see a reference to
> > it in your document. This protocol has implementations using a number of
> > quic different libraries.
> @Ian Swett <[e-mail]>
> > I would suggest looking at the ACK frequency draf
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SymOgANn9Tt737DFff%2FSXqpeqpHHpDpSoZq2AkDMOkk%3D&reserved=0>t,
> or some similar approaches.  Otherwise it can be very difficult to achieve
> comparable speeds as TCP.
>
> Hi Tom, Ian,
> Thanks for your comments.
>
> I've read draft Banks before starting my work. Actually, just to cite
> myself discussing that with Lucas Pardue: " Regarding RFC6349 and
> draft-banks, it's true that there is a similarity in the fact they stress a
> QUIC implementation and work similarly with clients opening multiple
> connections to a server. The major difference between the two is that their
> intent and metrics differ.
>
> Banks states in section 3 that "Generally, the goal of all these
> performance tests is to measure the maximum load that can be achieved with
> the given QUIC implementation and hardware configuration. To that end, the
> network is not expected to be the bottleneck in any of these tests. To
> achieve that, the appropriate network hardware must be used so as to not
> limit throughput". Conversely, the "primary focus of [RFC6349] methodology
> is managed business-class IP networks, e.g. those Ethernet-terminated
> services for wich organizations are provided an SLA from the network
> provider. Because of the SLA, the expectation is that the TCP Throughput
> should achieve the guaranteed bandwidth". The RFC6349 methodology is to 1.
> measure the path MTU, 2. baseline RTT and bottleneck bandwidth, 3. do a TCP
> Throughput test "to baseline network performance".
> The metrics captured also show this difference in intent. RFC6349 measures
> transfer time ratio over an ideal transfer time depending on the network,
> TCP efficiency depending on the loss rate of the network, and buffer delay
> which depends on the baselined RTT. Whereas Banks performance scenarios
> measure metrics related to the server performance (e.g. requests per
> second)."
>
> Now that you both mention performances, I can see how a QUIC Throughput
> test software intended for testing network should be first validated using
> recommendations in draft-banks and other recommended optimizations in the
> goal of demonstrating performance rather than interoperability. My draft
> states that parameters are specified to optimize the QUIC Throughput. At
> the moment it's more a declaration of intent taken from RFC6349,  but I
> guess I could be more specific and more reading/experimentation is probably
> required to get a better picture of the impact parameters have on actual
> network performances.
>
> @Ian Swett <[e-mail]>
> Thanks for your suggestions. I came across fastly's post but now better
> realise I should take a look at it again 😉.
> While reading the ACK frequency draft, I noted a difference in exponents
> limiting min_ack_delay and max_ack_delay.
> min_ack_delay (0xff02de1a):
>
> A variable-length integer representing the minimum amount of time in
> microseconds by which the endpoint can delay an acknowledgement. *Values
> of 2^24 or greater are invalid,*
> *An endpoint's min_ack_delay MUST NOT be greater than its max_ack_delay. *
> At the same time, RFC9000 states that
>
>    - max_ack_delay (0x0b):  [...] *Values of 214* or greater are invalid
>
>
> Thanks for noticing.  That was issue 44
<https://github.com/quicwg/ack-frequency/issues/44> which has been fixed in
the editors copy, but we haven't published an updated version yet.  We'll
push an update quite soon I expect, ideally in the next week.


> Regards,
> Kevin
>
>
> ------------------------------
> *From:* Ian Swett <[e-mail]>
> *Sent:* Wednesday, September 22, 2021 1:05 PM
> *To:* Tom Jones <[e-mail]>
> *Cc:* Kévin Corre <[e-mail]>; [e-mail] <[e-mail]>
> *Subject:* [E!] : Re: Draft: QUIC Throughput Testing
>
> I would suggest looking at the ACK frequency draf
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SymOgANn9Tt737DFff%2FSXqpeqpHHpDpSoZq2AkDMOkk%3D&reserved=0>t,
> or some similar approaches.  Otherwise it can be very difficult to achieve
> comparable speeds as TCP.
>
> See Fastly's nice post
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.fastly.com%2Fblog%2Fmeasuring-quic-vs-tcp-computational-efficiency&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bMAieBa9ZkZFyJbobiCFRyy54TiUTcXqeCKPCO279uA%3D&reserved=0>
> about what it took to get comparable performance in QUIC as TCP.
>
> On Wed, Sep 22, 2021 at 6:06 AM Tom Jones <[e-mail]> wrote:
>
> On Wed, Sep 22, 2021 at 09:15:58AM +0000, Kévin Corre wrote:
> > Hi everyone,
> >
> > Our team at EXFO is working on adapting the TCP Throughput Testing
> methodology described in RFC6349 to QUIC. We just published a draft that
> can be found at
> https://datatracker.ietf.org/doc/html/draft-corre-quic-throughput-testing
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-corre-quic-throughput-testing&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450428506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5aJ79pllQs2zckYUPyvj1ccNNg5kQE8tsSSSTwpluX8%3D&reserved=0>
> > The draft follows the same organization as the RFC6349 and discuss what
> should be changed to support QUIC Throughput Testing.
> > The methodology is interested in measuring the achievable QUIC
> throughput and efficiency of a managed network when the connection is at an
> equilibrium. Although, it could also be used for un-managed networks.
> >
> > We are particularly looking for comments on the way we use the protocol
> to measure the network's QUIC Throughput but also feedback from potential
> users of such a test.
> > For instance, our draft closely follows the RFC6349 methodology, but
> since QUIC introduces new features, it would be interesting to gauge
> interest in extending the methodology to test features such as 0-RTT effect.
> >
> > Comments would be gladly appreciated, so let us know what you think.
> >
>
> Hi Kévin,
>
> Have you seen draft-banks-quic-performance? I don't see a reference to
> it in your document. This protocol has implementations using a number of
> quic different libraries.
>
> There is also a perf channel on the quic slack, please ask off list if
> you require an invite.
>
> https://datatracker.ietf.org/doc/html/draft-banks-quic-performance-00.txt
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-banks-quic-performance-00.txt&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450428506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BXdEofehdUsCMCixoA%2BD1QmAnrhfizVsCrnrk1o4mkY%3D&reserved=0>
>
> - Tom
>
>