Re: [E!] : Re: Draft: QUIC Throughput Testing

[Kévin Corre <[e-mail]>]

@Tom Jones<mailto:[e-mail]>
> Have you seen draft-banks-quic-performance? I don't see a reference to
> it in your document. This protocol has implementations using a number of
> quic different libraries.
@Ian Swett<mailto:[e-mail]>
> I would suggest looking at the ACK frequency draf<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SymOgANn9Tt737DFff%2FSXqpeqpHHpDpSoZq2AkDMOkk%3D&reserved=0>t, or some similar approaches.  Otherwise it can be very difficult to achieve comparable speeds as TCP.

Hi Tom, Ian,
Thanks for your comments.

I've read draft Banks before starting my work. Actually, just to cite myself discussing that with Lucas Pardue: " Regarding RFC6349 and draft-banks, it's true that there is a similarity in the fact they stress a QUIC implementation and work similarly with clients opening multiple connections to a server. The major difference between the two is that their intent and metrics differ.

Banks states in section 3 that "Generally, the goal of all these performance tests is to measure the maximum load that can be achieved with the given QUIC implementation and hardware configuration. To that end, the network is not expected to be the bottleneck in any of these tests. To achieve that, the appropriate network hardware must be used so as to not limit throughput". Conversely, the "primary focus of [RFC6349] methodology is managed business-class IP networks, e.g. those Ethernet-terminated services for wich organizations are provided an SLA from the network provider. Because of the SLA, the expectation is that the TCP Throughput should achieve the guaranteed bandwidth". The RFC6349 methodology is to 1. measure the path MTU, 2. baseline RTT and bottleneck bandwidth, 3. do a TCP Throughput test "to baseline network performance".
The metrics captured also show this difference in intent. RFC6349 measures transfer time ratio over an ideal transfer time depending on the network, TCP efficiency depending on the loss rate of the network, and buffer delay which depends on the baselined RTT. Whereas Banks performance scenarios measure metrics related to the server performance (e.g. requests per second)."

Now that you both mention performances, I can see how a QUIC Throughput test software intended for testing network should be first validated using recommendations in draft-banks and other recommended optimizations in the goal of demonstrating performance rather than interoperability. My draft states that parameters are specified to optimize the QUIC Throughput. At the moment it's more a declaration of intent taken from RFC6349,  but I guess I could be more specific and more reading/experimentation is probably required to get a better picture of the impact parameters have on actual network performances.

@Ian Swett<mailto:[e-mail]>
Thanks for your suggestions. I came across fastly's post but now better realise I should take a look at it again 😉.
While reading the ACK frequency draft, I noted a difference in exponents limiting min_ack_delay and max_ack_delay.
min_ack_delay (0xff02de1a):

A variable-length integer representing the minimum amount of time in microseconds by which the endpoint can delay an acknowledgement. Values of 2^24 or greater are invalid,

An endpoint's min_ack_delay MUST NOT be greater than its max_ack_delay.
At the same time, RFC9000 states that
  *   max_ack_delay (0x0b):  [...] Values of 214 or greater are invalid

Regards,
Kevin


________________________________
From: Ian Swett <[e-mail]>
Sent: Wednesday, September 22, 2021 1:05 PM
To: Tom Jones <[e-mail]>
Cc: Kévin Corre <[e-mail]>; [e-mail] <[e-mail]>
Subject: [E!] : Re: Draft: QUIC Throughput Testing

I would suggest looking at the ACK frequency draf<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SymOgANn9Tt737DFff%2FSXqpeqpHHpDpSoZq2AkDMOkk%3D&reserved=0>t, or some similar approaches.  Otherwise it can be very difficult to achieve comparable speeds as TCP.

See Fastly's nice post<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.fastly.com%2Fblog%2Fmeasuring-quic-vs-tcp-computational-efficiency&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bMAieBa9ZkZFyJbobiCFRyy54TiUTcXqeCKPCO279uA%3D&reserved=0> about what it took to get comparable performance in QUIC as TCP.

On Wed, Sep 22, 2021 at 6:06 AM Tom Jones <[e-mail]<mailto:[e-mail]>> wrote:
On Wed, Sep 22, 2021 at 09:15:58AM +0000, Kévin Corre wrote:
> Hi everyone,
>
> Our team at EXFO is working on adapting the TCP Throughput Testing methodology described in RFC6349 to QUIC. We just published a draft that can be found at https://datatracker.ietf.org/doc/html/draft-corre-quic-throughput-testing<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-corre-quic-throughput-testing&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450428506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5aJ79pllQs2zckYUPyvj1ccNNg5kQE8tsSSSTwpluX8%3D&reserved=0>
> The draft follows the same organization as the RFC6349 and discuss what should be changed to support QUIC Throughput Testing.
> The methodology is interested in measuring the achievable QUIC throughput and efficiency of a managed network when the connection is at an equilibrium. Although, it could also be used for un-managed networks.
>
> We are particularly looking for comments on the way we use the protocol to measure the network's QUIC Throughput but also feedback from potential users of such a test.
> For instance, our draft closely follows the RFC6349 methodology, but since QUIC introduces new features, it would be interesting to gauge interest in extending the methodology to test features such as 0-RTT effect.
>
> Comments would be gladly appreciated, so let us know what you think.
>

Hi Kévin,

Have you seen draft-banks-quic-performance? I don't see a reference to
it in your document. This protocol has implementations using a number of
quic different libraries.

There is also a perf channel on the quic slack, please ask off list if
you require an invite.

https://datatracker.ietf.org/doc/html/draft-banks-quic-performance-00.txt<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-banks-quic-performance-00.txt&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450428506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BXdEofehdUsCMCixoA%2BD1QmAnrhfizVsCrnrk1o4mkY%3D&reserved=0>

- Tom