RE: Impact of hardware offloads on network stack performance

[Praveen Balasubramanian <[e-mail]>]

>>.  There is one caveat there, which is that (IIUC, on current hardware) you can’t do packet pacing with LSO, and so my understanding is that even for TCP many providers disable this in pursuit of better egress management
Some do some don’t. Windows hasn’t done packet pacing for TCP so far and is quite widely deployed across servers and clients. Even with pacing you can come up with proper sizing so you don’t lose the advantages of LSO. As you can see in the numbers LSO has a pretty sizeable impact on perf.

>> TCP will dramatically gain performance once hardware offload catches up and allows paced LSO
TCP as deployed today on Windows uses hardware offload (LSO, LRO, checksum) by default on all systems when available and support is almost universal except for some client NICs. There is no strict requirement for pacing to do LSO since we have been successfully deployed this way for 10+ years. TCP already has dramatically better perf than UDP on all public clouds across all VM sizes.

As to overhead due to PNE, we did an experiment for that right when the PNE discussion stated. With our current QUIC implementation over the (unoptimized) UDP stack and there was 1 to 1.25% overhead. Once the UDP and QUIC implementations are optimized, this number will increase. Happy to do the measurement again.

As to what extent hardware is impacted, hardware vendors will have to chime in. Several folks had chimed in on the original thread. Since we have a lack of participation from hardware vendors, I am trying to reach out to more folks to join the group and participate.

From: Mike Bishop [mailto:[e-mail]]
Sent: Thursday, April 5, 2018 2:57 PM
To: Mikkel Fahnøe Jørgensen <[e-mail]>; Praveen Balasubramanian <[e-mail]>; IETF QUIC WG <[e-mail]>
Subject: RE: Impact of hardware offloads on network stack performance

That’s interesting data, Praveen – thanks.  There is one caveat there, which is that (IIUC, on current hardware) you can’t do packet pacing with LSO, and so my understanding is that even for TCP many providers disable this in pursuit of better egress management.  So what we’re really saying is that:

  *   On current hardware and OS setups, UDP runs less than half as much throughput as TCP; that needs some optimization, as we’ve already discussed
  *   😊

However, as Mikkel points out, the crypto costs are likely to overwhelm the OS throughput / scheduling issues in real deployments.  So I think the other relevant piece to understanding the cost here is this:

  *   What is the throughput and relative cost of TLS/TCP versus QUIC (i.e. how much are the smaller units of encryption hurting us versus, say, 16KB TLS records)?
     *   TLS implementations already vary here:  Some implementations choose large record sizes, some vary record sizes to reduce delay / HoLB, so this probably isn’t a single number.
  *   How much additional load does PNE add to this difference?
  *   To what extent would PNE make future crypto offloads impossible versus requires more R&D to develop?

From: QUIC [mailto:[e-mail]] On Behalf Of Mikkel Fahnøe Jørgensen
Sent: Wednesday, April 4, 2018 1:34 PM
To: Praveen Balasubramanian <pravb=[e-mail]<mailto:pravb=[e-mail]>>; IETF QUIC WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: Impact of hardware offloads on network stack performance

Thanks for sharing these numbers.

My guess is that these offloads deal with kernel scheduling, memory cache issues, and interrupt scheduling.

It probably has very little to do with crypto, TCP headers, and any other cpu sensitive processing.

This is where netmap enters: you can transparently feed the data as it becomes available with very little sync work and you can also efficiently pipeline so you pass data to decryptor, then to app, with as little bus traffic as possible. No need to copy data or synchronize kernel space.

For 1K packets you would use about 1000ns (3 cycles/byte) on crypto and this would happen in L1 cache. It would of course consume a core which a crypto offload would not, but that can be debated because with 18 core systems, your problem is memory and network more than CPU.

My concern with PNE is for small packets and low latency where
1) an estimated 24ns for PN encryption and decryption becomes measurable if your network is fast enough.
2) any damage to the packet buffer cause all sorts of memory and cpu bus traffic issues.

1) is annoying. 2) is bad, completely avoidable but not as PR 1079 is formulated currently.

2) is also likely bad for hardware offload units as well.

As to SRV-IO - I’m not familiar with it, but obviously there is some IO abstraction layer - the question is how you make it accessible to apps as opposed to device drivers that does nor work with you QUIC custom stack, and netmap is one option here.

Kind Regards,
Mikkel Fahnøe Jørgensen


On 4 April 2018 at 22.15.52, Praveen Balasubramanian (pravb=[e-mail]<mailto:pravb=[e-mail]>) wrote:
Some comparative numbers from an out of box default settings Windows Server 2016 (released version) for a single connection with a microbenchmark tool:

Offloads enabled

TCP gbps

UDP gbps

LSO + LRO + checksum

24

3.6

Checksum only

7.6

3.6

None

5.6

2.3


This is for a fully bottlenecked CPU core -- if you run lower data rates there is still a significant difference in Cycles/byte cost. Same increased CPU cost applies for client systems going over high data rate Wifi and cellular.

This is without any crypto. Once you add crypto the numbers become much worse with crypto cost becoming dominant. Adding another crypto step further exacerbates the problem. Hence crypto offload gains in importance followed by these batch offloads.

If folks need any more numbers I’d be happy to provide them.

Thanks