Re: Impact of hardware offloads on network stack performance

[Ian Swett <[e-mail]>]

PNE doesn't preclude UDP LRO or pacing offload.

However, in a highly optimized stack(ie: The Disk|Crypt|Net paper from
SIGCOMM 2017) not touching(ie: bringing into memory or cache) the content
to be sent until the last possible moment is critical.  PNE likely means
touching that content much earlier.

But personally I consider the actual crypto we're talking about(2 AES
instructions?) adding basically free.

On Fri, Apr 6, 2018 at 4:37 PM Mike Bishop <[e-mail]> wrote:

> Thanks for the clarification.  I guess what I’m really getting down to is
> how do we quantify the actual *cost* of PNE?  Are we substantially
> increasing the crypto costs?  If Ian is saying that crypto is comparatively
> cheap and the cost is that it’s harder to offload something that’s
> comparatively cheap, what have we lost?  I’d think we want to offload the
> most intensive piece we can, and it seems like we’re talking about crypto
> offloads....  Or are we saying instead that the crypto makes it harder to
> offload other things in the future, like a QUIC equivalent to LRO/LSO?
>
>
>
> *From:* Mikkel Fahnøe Jørgensen [mailto:[e-mail]]
> *Sent:* Thursday, April 5, 2018 9:45 PM
> *To:* Ian Swett <[e-mail]>; Mike Bishop <[e-mail]>
> *Cc:* Praveen Balasubramanian <[e-mail]>; IETF QUIC WG <
> [e-mail]>
> *Subject:* Re: Impact of hardware offloads on network stack performance
>
>
>
> To be clear - I don’t think crypto is overshadowing other issues as Mike
> read my post. It certainly comes at a cost, but either multiple cores or
> co-processors will deal with this. 1000ns is 10Gbps crypto speed on one
> core and it is higly parallelisable and cache friendly.
>
>
>
> But if you have to drip your packets through a traditional send interface
> that is copying, buffering or blocking, and certainly sync’ing with the
> kernel - it is going to be tough.
>
>
>
> For receive you risk high latency or top much scheduling in receive
> buffers.
>
>
> ------------------------------
>
> *From:* Ian Swett <[e-mail]>
> *Sent:* Friday, April 6, 2018 4:06:01 AM
> *To:* Mike Bishop
> *Cc:* Mikkel Fahnøe Jørgensen; Praveen Balasubramanian; IETF QUIC WG
> *Subject:* Re: Impact of hardware offloads on network stack performance
>
>
>
>
>
> On Thu, Apr 5, 2018 at 5:57 PM Mike Bishop <[e-mail]> wrote:
>
> That’s interesting data, Praveen – thanks.  There is one caveat there,
> which is that (IIUC, on current hardware) you can’t do packet pacing with
> LSO, and so my understanding is that even for TCP many providers disable
> this in pursuit of better egress management.  So what we’re really saying
> is that:
>
>    - On current hardware and OS setups, UDP runs less than half as much
>    throughput as TCP; that needs some optimization, as we’ve already discussed
>    - TCP will dramatically gain performance once hardware offload catches
>    up and allows *paced* LSO 😊
>
>
>
> However, as Mikkel points out, the crypto costs are likely to overwhelm
> the OS throughput / scheduling issues in real deployments.  So I think the
> other relevant piece to understanding the cost here is this:
>
>
>
> I have a few cases(both client and server) where my UDP send costs are
> more than 30%(in some cases 50%) of CPU consumption, and crypto is less
> than 10%.  So currently, I assume crypto is cheap(especially AES-GCM when
> hardware acceleration is available) and egress is expensive.  UDP ingres is
> not that expensive, but could use a bit of optimization as well.
>
>
>
> The only 'fancy' thing our current code is doing for crypto is encrypting
> in place, which was a ~2% win.  Nice, but not transformative.  See
> EncryptInPlace
> <https://cs.chromium.org/chromium/src/net/quic/core/quic_framer.cc?sq=package:chromium&l=1893>
>
>
>
> I haven't done much work benchmarking Windows, so possibly the Windows UDP
> stack is really fast, and so crypto seems really slow in comparison?
>
>
>
>
>    - What is the throughput and relative cost of TLS/TCP versus QUIC
>    (i.e. how much are the smaller units of encryption hurting us versus, say,
>    16KB TLS records)?
>
>
>    - TLS implementations already vary here:  Some implementations choose
>       large record sizes, some vary record sizes to reduce delay / HoLB, so this
>       probably isn’t a single number.
>
>
>    - How much additional load does PNE add to this difference?
>    - To what extent would PNE make future crypto offloads *impossible*
>    versus * requires more R&D to develop*?
>
>
>
> *From:* QUIC [mailto:[e-mail]] *On Behalf Of *Mikkel Fahnøe
> Jørgensen
> *Sent:* Wednesday, April 4, 2018 1:34 PM
> *To:* Praveen Balasubramanian <pravb=[e-mail]>;
> IETF QUIC WG <[e-mail]>
> *Subject:* Re: Impact of hardware offloads on network stack performance
>
>
>
> Thanks for sharing these numbers.
>
>
>
> My guess is that these offloads deal with kernel scheduling, memory cache
> issues, and interrupt scheduling.
>
>
>
> It probably has very little to do with crypto, TCP headers, and any other
> cpu sensitive processing.
>
>
>
> This is where netmap enters: you can transparently feed the data as it
> becomes available with very little sync work and you can also efficiently
> pipeline so you pass data to decryptor, then to app, with as little bus
> traffic as possible. No need to copy data or synchronize kernel space.
>
>
>
> For 1K packets you would use about 1000ns (3 cycles/byte) on crypto and
> this would happen in L1 cache. It would of course consume a core which a
> crypto offload would not, but that can be debated because with 18 core
> systems, your problem is memory and network more than CPU.
>
>
>
> My concern with PNE is for small packets and low latency where
>
> 1) an estimated 24ns for PN encryption and decryption becomes measurable
> if your network is fast enough.
>
> 2) any damage to the packet buffer cause all sorts of memory and cpu bus
> traffic issues.
>
>
>
> 1) is annoying. 2) is bad, completely avoidable but not as PR 1079 is
> formulated currently.
>
>
>
> 2) is also likely bad for hardware offload units as well.
>
>
>
> As to SRV-IO - I’m not familiar with it, but obviously there is some IO
> abstraction layer - the question is how you make it accessible to apps as
> opposed to device drivers that does nor work with you QUIC custom stack,
> and netmap is one option here.
>
>
>
> Kind Regards,
>
> Mikkel Fahnøe Jørgensen
>
>
>
> On 4 April 2018 at 22.15.52, Praveen Balasubramanian (
> pravb=[e-mail]) wrote:
>
> Some comparative numbers from an out of box default settings Windows
> Server 2016 (released version) for a single connection with a
> microbenchmark tool:
>
>
>
> *Offloads enabled*
>
> *TCP gbps*
>
> *UDP gbps*
>
> *LSO + LRO + checksum*
>
> 24
>
> 3.6
>
> *Checksum only*
>
> 7.6
>
> 3.6
>
> *None*
>
> 5.6
>
> 2.3
>
>
>
> This is for a fully bottlenecked CPU core -- *if you run lower data rates
> there is still a significant difference in Cycles/byte cost*. Same
> increased CPU cost applies for client systems going over high data rate
> Wifi and cellular.
>
>
>
> This is without any crypto. Once you add crypto the numbers become much
> worse *with crypto cost becoming dominant*. Adding another crypto step
> further exacerbates the problem. Hence crypto offload gains in importance
> followed by these batch offloads.
>
>
>
> If folks need any more numbers I’d be happy to provide them.
>
>
>
> Thanks
>
>
>
>