Re: [tsvwg] A review of draft-ietf-tsvwg-udp-options-12

[Joseph Touch <[e-mail]>]

Hi, Tom,

OCS has been required since -08 (Sept. 2019). Here’s the relevant text:
   >> The OCS MUST be included when the UDP checksum is nonzero and UDP
   options are present.
Joe

> On Jun 14, 2021, at 10:32 AM, Tom Herbert <[e-mail]> wrote:
> 
> Joe,
> 
> I suggest that the UDP options should be preceded by a four byte
> header consisting of one byte type, one byte length, and two byte
> checksum. As I've mentioned previously, making the checksum optional
> is inherently problematic because it cannot protect against a
> corrupted type field for the optional checksum. e.g. a single bit flip
> in the type field for the checksum could turn the checksum option into
> some other type and there is no way to detect that.
> 
> Tom
> 
> Tom
> 
> 
> On Mon, Jun 14, 2021 at 10:20 AM Joe Touch <[e-mail]> wrote:
>> 
>> Ps - we need an option length field to make fragments look like tcp. I can put that in - do we want that in OCS? Or independent?
>> 
>>> On Jun 14, 2021, at 10:16 AM, Joe Touch <[e-mail]> wrote:
>>> 
>>> FYI that’s what fragments look like. We can’t do this for non fragments.
>>> 
>>>>> On Jun 14, 2021, at 10:03 AM, Tom Herbert <[e-mail]> wrote:
>>>>> 
>>>>> On Mon, Jun 14, 2021 at 9:31 AM Gorry Fairhurst <[e-mail]> wrote:
>>>>> 
>>>>>> On 14/06/2021 17:17, Tom Herbert wrote:
>>>>>> On Sun, Jun 13, 2021 at 9:31 PM Joseph Touch <[e-mail]> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On Jun 13, 2021, at 7:20 PM, C. M. Heard <[e-mail]> wrote:
>>>>>>> 
>>>>>>>> If we DO support zero-copy and thus want to allow non-terminal fragments to have post-fragoption options that operate on each fragment, then we would add THISFRAGLEN to the nonterminal format and issue different KIND numbers to nonterminal/terminal fragment.
>>>>>>> 
>>>>>>> 
>>>>>>> I for one would appreciate further discussion of these last points. I admit that I have failed to grasp Joe's message on the RDMA thread, and I would appreciate some time to think about it.
>>>>>>> 
>>>>>>> 
>>>>>>> Sure - here’s how it all works. Note that this is relevant mostly for long transfers with persistent UDP fragmentation; if that is assumed to be ‘adjusted’ at the app layer (as QUIC does), then we don’t need zero-copy support...
>>>>>>> 
>>>>>>> - right now, UDP data can be zero-copied when received into user space, starting with the user data
>>>>>> Only if the device supports header/data split where the headers are in
>>>>>> one buffer and UDP data is in aligned buffer.
>>>>>> 
>>>>>>> - if we add options, UDP data can still be zero-copied because it hasn’t moved (it still begins the payload
>>>>>>> - however, fragments are different because (esp given the merging of frag and lite) they don’t start at the beginning of data
>>>>>>> - they always start after OCS (which I think we should make fit the uniform KIND/LEN/OCS format of 4 bytes)
>>>>>>> - if the FRAG comes next, then we can move the frag content around a little and still support zero-copy
>>>>>>> 
>>>>>>> notably, we move the first 10 bytes of the fragment to the end
>>>>>>> 4 for OCS
>>>>>>> 6 for FRAG (assuming FRAG includes KIND/OPTLEN/FRAGOFFSET/ID/FRAGLEN)
>>>>>>> that way we can zero-copy the frag packet into place, then just copy those last 8 bytes over OCS and the FRAG header
>>>>>>> 
>>>>>> An obvious feature we'd want is NIC hardware to do UDP options
>>>>>> fragementation and reassembly, analogous to existing UDP Fragmentation
>>>>>> Offload (UFO) which performs IP fragmentation of UDP packets. The
>>>>>> impediment with supporting this is that hardware devices would need to
>>>>>> perform protocol processing on trailers as opposed to headers. Nearly
>>>>>> all hardware devices, including switches and NICs, are optimized to
>>>>>> process protocol headers and in modern devices they are quite
>>>>>> programmable in that regard. However, they typically rely on a parsing
>>>>>> buffer that holds the first N bytes of the packet and assume that all
>>>>>> the protocol headers lie within that. They wouldn't process data after
>>>>>> that header in the fast path at least, and almost certainly would have
>>>>>> capability to process protocol headers at that end of a large packet.
>>>>>> I am doubtful we'll ever see hardware support for trailer protocols,
>>>>>> and hence it's unlikely we'd see accelerations for UDP options like we
>>>>>> have for TCP.
>>>>>> 
>>>>>> Tom
>>>>> 
>>>>> OK.... Is there any way that we could design to enable this?
>>>>> 
>>>>> I'm "fishing" for ideas because I know you've talked about the various
>>>>> offload methods.
>>>>> 
>>>> 
>>>> Gorry,
>>>> 
>>>> My suggestion was to place UDP options after the UDP header. Instead
>>>> of just placing fragment header after the UDP header, place all the
>>>> UDP options there and then follow that by the Payload. So packet looks
>>>> like:
>>>> 
>>>> +-------------------+
>>>> |   UDP header  |
>>>> +-------------------+
>>>> |  UDP options  |
>>>> +-------------------+
>>>> |     Payload      |
>>>> +-------------------+
>>>> 
>>>> Now this looks a lot like a TCP packet and other variable length
>>>> headers which we know how to handle. For zero copy we can do
>>>> header/split by programming emerging smart devices to split through
>>>> UDP options in one buffer and payload in another thereby also
>>>> eliminating any need to move headers or data around.
>>>> 
>>>> Tom
>>>> 
>>>>> So for options in the trailer, this is clearly an impediment.
>>>>> 
>>>>> For UDP-Opt fragmentation, I understand there is no standard UDP payload,
>>>>> 
>>>>> .... only an option containing a fragment, so the Fragment information
>>>>> would actually be in the" first N bytes of the packet".
>>>>> 
>>>>> So, what do you think  could be most likely helpful to enable fastpath
>>>>> accelleration for the fragments?
>>>>> 
>>>>> Gorry
>>>>> 
>>>>>>> This method assumes that we try to keep FRAG early in the packet - preferably right after OCS. The later it comes, the more additional bytes we need to move to “fix” the copy (beyond the 8 bytes noted above).
>>>>>>> 
>>>>>>> —
>>>>>>> 
>>>>>>> This method is the only reason we would want to allow options after non-terminal fragments - basically to keep the fragment toward the front of the packet, using the rule that post-noninitial frag options still operate on the fragment, rather than waiting for reassembly. The exception is the terminal fragment, where post-terminal fragment options operate on the reassembled packet.
>>>>>>> 
>>>>>>> Joe
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>> 
>