Re: [tsvwg] A review of draft-ietf-tsvwg-udp-options-12

[Joseph Touch <[e-mail]>]

Note that this also means that non-frags might not have offload support unless offload engines evolve to support trailers.

(And, as noted earlier, we need an option-length equivalent to TCP’s data offset in a sense - do we add that to OCS or as an independent option?)

Note, though, that offload engines already do lots of things that are in direct violation of specs, so we shouldn’t assume they set the tone for where we go either.

Joe

> On Jun 14, 2021, at 10:33 AM, Gorry Fairhurst <[e-mail]> wrote:
> 
> On 14/06/2021 18:15, Joe Touch wrote:
>> FYI that’s what fragments look like. We can’t do this for non fragments.
> 
> OK, we might be on the same page here!
> 
> If Tom is saying that might allow Frags to be "easily" reassembled in offload, I'm interested.
> 
> Gorry
> 
>> 
>>> On Jun 14, 2021, at 10:03 AM, Tom Herbert <[e-mail]> wrote:
>>> 
>>> On Mon, Jun 14, 2021 at 9:31 AM Gorry Fairhurst <[e-mail]> wrote:
>>>>> On 14/06/2021 17:17, Tom Herbert wrote:
>>>>> On Sun, Jun 13, 2021 at 9:31 PM Joseph Touch <[e-mail]> wrote:
>>>>>> 
>>>>>> On Jun 13, 2021, at 7:20 PM, C. M. Heard <[e-mail]> wrote:
>>>>>> 
>>>>>>> If we DO support zero-copy and thus want to allow non-terminal fragments to have post-fragoption options that operate on each fragment, then we would add THISFRAGLEN to the nonterminal format and issue different KIND numbers to nonterminal/terminal fragment.
>>>>>> 
>>>>>> I for one would appreciate further discussion of these last points. I admit that I have failed to grasp Joe's message on the RDMA thread, and I would appreciate some time to think about it.
>>>>>> 
>>>>>> 
>>>>>> Sure - here’s how it all works. Note that this is relevant mostly for long transfers with persistent UDP fragmentation; if that is assumed to be ‘adjusted’ at the app layer (as QUIC does), then we don’t need zero-copy support...
>>>>>> 
>>>>>> - right now, UDP data can be zero-copied when received into user space, starting with the user data
>>>>> Only if the device supports header/data split where the headers are in
>>>>> one buffer and UDP data is in aligned buffer.
>>>>> 
>>>>>> - if we add options, UDP data can still be zero-copied because it hasn’t moved (it still begins the payload
>>>>>> - however, fragments are different because (esp given the merging of frag and lite) they don’t start at the beginning of data
>>>>>> - they always start after OCS (which I think we should make fit the uniform KIND/LEN/OCS format of 4 bytes)
>>>>>> - if the FRAG comes next, then we can move the frag content around a little and still support zero-copy
>>>>>> 
>>>>>> notably, we move the first 10 bytes of the fragment to the end
>>>>>> 4 for OCS
>>>>>> 6 for FRAG (assuming FRAG includes KIND/OPTLEN/FRAGOFFSET/ID/FRAGLEN)
>>>>>> that way we can zero-copy the frag packet into place, then just copy those last 8 bytes over OCS and the FRAG header
>>>>>> 
>>>>> An obvious feature we'd want is NIC hardware to do UDP options
>>>>> fragementation and reassembly, analogous to existing UDP Fragmentation
>>>>> Offload (UFO) which performs IP fragmentation of UDP packets. The
>>>>> impediment with supporting this is that hardware devices would need to
>>>>> perform protocol processing on trailers as opposed to headers. Nearly
>>>>> all hardware devices, including switches and NICs, are optimized to
>>>>> process protocol headers and in modern devices they are quite
>>>>> programmable in that regard. However, they typically rely on a parsing
>>>>> buffer that holds the first N bytes of the packet and assume that all
>>>>> the protocol headers lie within that. They wouldn't process data after
>>>>> that header in the fast path at least, and almost certainly would have
>>>>> capability to process protocol headers at that end of a large packet.
>>>>> I am doubtful we'll ever see hardware support for trailer protocols,
>>>>> and hence it's unlikely we'd see accelerations for UDP options like we
>>>>> have for TCP.
>>>>> 
>>>>> Tom
>>>> OK.... Is there any way that we could design to enable this?
>>>> 
>>>> I'm "fishing" for ideas because I know you've talked about the various
>>>> offload methods.
>>>> 
>>> Gorry,
>>> 
>>> My suggestion was to place UDP options after the UDP header. Instead
>>> of just placing fragment header after the UDP header, place all the
>>> UDP options there and then follow that by the Payload. So packet looks
>>> like:
>>> 
>>> +-------------------+
>>> |   UDP header  |
>>> +-------------------+
>>> |  UDP options  |
>>> +-------------------+
>>> |     Payload      |
>>> +-------------------+
>>> 
>>> Now this looks a lot like a TCP packet and other variable length
>>> headers which we know how to handle. For zero copy we can do
>>> header/split by programming emerging smart devices to split through
>>> UDP options in one buffer and payload in another thereby also
>>> eliminating any need to move headers or data around.
>>> 
>>> Tom
>>> 
>>>> So for options in the trailer, this is clearly an impediment.
>>>> 
>>>> For UDP-Opt fragmentation, I understand there is no standard UDP payload,
>>>> 
>>>> .... only an option containing a fragment, so the Fragment information
>>>> would actually be in the" first N bytes of the packet".
>>>> 
>>>> So, what do you think  could be most likely helpful to enable fastpath
>>>> accelleration for the fragments?
>>>> 
>>>> Gorry
>>>> 
>>>>>> This method assumes that we try to keep FRAG early in the packet - preferably right after OCS. The later it comes, the more additional bytes we need to move to “fix” the copy (beyond the 8 bytes noted above).
>>>>>> 
>>>>>> —
>>>>>> 
>>>>>> This method is the only reason we would want to allow options after non-terminal fragments - basically to keep the fragment toward the front of the packet, using the rule that post-noninitial frag options still operate on the fragment, rather than waiting for reassembly. The exception is the terminal fragment, where post-terminal fragment options operate on the reassembled packet.
>>>>>> 
>>>>>> Joe
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>