Re: [tsvwg] A review of draft-ietf-tsvwg-udp-options-12

[Tom Herbert <[e-mail]>]

On Sun, Jun 13, 2021 at 9:31 PM Joseph Touch <[e-mail]> wrote:
>
>
>
> On Jun 13, 2021, at 7:20 PM, C. M. Heard <[e-mail]> wrote:
>
>> If we DO support zero-copy and thus want to allow non-terminal fragments to have post-fragoption options that operate on each fragment, then we would add THISFRAGLEN to the nonterminal format and issue different KIND numbers to nonterminal/terminal fragment.
>
>
>
> I for one would appreciate further discussion of these last points. I admit that I have failed to grasp Joe's message on the RDMA thread, and I would appreciate some time to think about it.
>
>
> Sure - here’s how it all works. Note that this is relevant mostly for long transfers with persistent UDP fragmentation; if that is assumed to be ‘adjusted’ at the app layer (as QUIC does), then we don’t need zero-copy support...
>
> - right now, UDP data can be zero-copied when received into user space, starting with the user data

Only if the device supports header/data split where the headers are in
one buffer and UDP data is in aligned buffer.

> - if we add options, UDP data can still be zero-copied because it hasn’t moved (it still begins the payload
> - however, fragments are different because (esp given the merging of frag and lite) they don’t start at the beginning of data
> - they always start after OCS (which I think we should make fit the uniform KIND/LEN/OCS format of 4 bytes)
> - if the FRAG comes next, then we can move the frag content around a little and still support zero-copy
>
> notably, we move the first 10 bytes of the fragment to the end
> 4 for OCS
> 6 for FRAG (assuming FRAG includes KIND/OPTLEN/FRAGOFFSET/ID/FRAGLEN)
> that way we can zero-copy the frag packet into place, then just copy those last 8 bytes over OCS and the FRAG header
>

An obvious feature we'd want is NIC hardware to do UDP options
fragementation and reassembly, analogous to existing UDP Fragmentation
Offload (UFO) which performs IP fragmentation of UDP packets. The
impediment with supporting this is that hardware devices would need to
perform protocol processing on trailers as opposed to headers. Nearly
all hardware devices, including switches and NICs, are optimized to
process protocol headers and in modern devices they are quite
programmable in that regard. However, they typically rely on a parsing
buffer that holds the first N bytes of the packet and assume that all
the protocol headers lie within that. They wouldn't process data after
that header in the fast path at least, and almost certainly would have
capability to process protocol headers at that end of a large packet.
I am doubtful we'll ever see hardware support for trailer protocols,
and hence it's unlikely we'd see accelerations for UDP options like we
have for TCP.

Tom

> This method assumes that we try to keep FRAG early in the packet - preferably right after OCS. The later it comes, the more additional bytes we need to move to “fix” the copy (beyond the 8 bytes noted above).
>
> —
>
> This method is the only reason we would want to allow options after non-terminal fragments - basically to keep the fragment toward the front of the packet, using the rule that post-noninitial frag options still operate on the fragment, rather than waiting for reassembly. The exception is the terminal fragment, where post-terminal fragment options operate on the reassembled packet.
>
> Joe
>
>
>
>