RE: AEAD and header encryption overhead in QUIC

[Nick Banks <[e-mail]>]

Hi Kazuho,

Thanks for the information! It’s very interesting. You might try this too:

We’ve found that an easy way to minimize the CPU cost of the header protection (on send and receive) is batching. If you copy 8 packet headers into a single contiguous block of memory, you can do a single crypto operation on it. The cost of doing one batch of 8 is essentially the same as doing just a single header. This effectively cuts the cost of header protection to 1/8th the original cost. Obviously this only works if you have enough packets to batch process, but for high throughput tests, you usually do.

Feel free to take a look at the MsQuic code (receive path<https://github.com/microsoft/msquic/blob/master/src/core/connection.c#L4784>, send path<https://github.com/microsoft/msquic/blob/master/src/core/packet_builder.c#L728>) The nice thing about this design (for those of us who don’t have as much special crypto expertise) is that it doesn’t require any special crypto changes. I’d be interested to see if our two approaches could be combined for even better performance!

Thanks,
- Nick

From: Kazuho Oku<mailto:[e-mail]>
Sent: Thursday, June 18, 2020 1:24 AM
To: IETF QUIC WG<mailto:[e-mail]>
Subject: AEAD and header encryption overhead in QUIC

Hello folks,

TL;DR: When existing crypto libraries are used, the AEAD cost of QUIC can become somewhat greater than that of TLS over TCP, due to those crypto libraries not optimized for handling short AEAD blocks. Crypto libraries optimized for short AEAD blocks can tighten the gap. Also, crypto libraries optimized for QUIC can combine AEAD operation and header protection vector generation, effectively eliminating the cost of header protection.

A bit longer story:

Recently, when testing quicly [1] using a NIC with *hardware support* for UDP GSO offload, we noticed that it uses +30% CPU cycles compared to TLS over TCP. This was due to our underlying crypto not being optimized for short AEAD blocks, and due to the call overhead of doing header protection one packet at a time. [2]

We went on to implement our own AES-GCM library optimized for short AEAD blocks (i.e. packet-sized), that also generates the header protection vector while doing AEAD computation. By switching to the new crypto library, the performance drawback of the crypto itself compared to large AEAD blocks when down from 30% to 10% [3].

The overhead of header protection (relative to all the crypto cost) on the send side is now between 0.3% to 1% when using packets sized around 1400 bytes to 1500 bytes. The ratio against the total transport cost is going to be less than that, and in practice, I think that they are negligible. The story might be a bit different on the receive-side due to the dependency between header protection and AEAD, though I am sure there would also be ways to optimize.

In terms of CPU cycles spent by the QUIC stack, the overhead compared to TLS-over-TCP came down to 5%. quicly can now sustain the send rate of 5Gbps while spending about 30% of CPU cycles of one modern Intel CPU core [2].

These numbers are observations from one TLS / QUIC stack doing micro-benchmarks, though I tend to think that sharing performance numbers lead to people having better expectations regarding the CPU cost of QUIC. Some of us were also concerned about the cost of header protection (a.k.a. packet number encryption) [4]. Hence posting here.

[1] https://github.com/h2o/quicly<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fh2o%2Fquicly&data=02%7C01%7Cnibanks%40microsoft.com%7Cce44384090cd41b9874f08d81360f9dc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637280654511448872&sdata=hjeugkkQwFtQNx7UriHYA9eQ3z1z9omSxB6XRPiUF1Y%3D&reserved=0>
[2] https://github.com/h2o/quicly/pull/359<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fh2o%2Fquicly%2Fpull%2F359&data=02%7C01%7Cnibanks%40microsoft.com%7Cce44384090cd41b9874f08d81360f9dc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637280654511458858&sdata=VSe%2Br2rp81U7RIFhRaJ%2FAh4YGXobO4F90v7lUcxqMoA%3D&reserved=0>
[3] https://github.com/h2o/picotls/pull/310<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fh2o%2Fpicotls%2Fpull%2F310&data=02%7C01%7Cnibanks%40microsoft.com%7Cce44384090cd41b9874f08d81360f9dc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637280654511458858&sdata=xbHdVd9cewW%2BlwSlIQtG4f3sTGYFxqM6c60qnVFkVbI%3D&reserved=0>
[4] https://mailarchive.ietf.org/arch/msg/quic/Wy8JXwKyeYgWgRWizWDbyOLg0xc/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailarchive.ietf.org%2Farch%2Fmsg%2Fquic%2FWy8JXwKyeYgWgRWizWDbyOLg0xc%2F&data=02%7C01%7Cnibanks%40microsoft.com%7Cce44384090cd41b9874f08d81360f9dc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637280654511468853&sdata=ZT3%2FAtl8HpbN5KKYdI0vY8cmVnoeSJaS1Jys%2F6QiQ9A%3D&reserved=0>

--
Kazuho Oku