Re: [tsvwg] RDMA Support by UDP FRAG Option

[Tom Herbert <[e-mail]>]

On Sun, Jun 13, 2021 at 5:01 PM Joseph Touch <[e-mail]> wrote:
>
>
> On Jun 13, 2021, at 4:37 PM, C. M. Heard <[e-mail]> wrote:
>
> On Thu, Jun 10, 2021 at 9:43 PM Joseph Touch wrote:
>>
>> - do we want/need RDMA support?
>>         if not, we ditch the frag length in non-terminal frags
>
>
>
> I am going to reveal my ignorance by asking what is meant by RDMA (Remote Direct Memory Access). But I'm fine with that as long as we end up with an unambiguous problem statement.
>
>
> I may have conflated two issues.
>
> 1. Whether UDP supports zero-copy in the presence of reassembly.
> 2. Whether FRAG needs a frag length per se
>
> I think #2 might strictly depend only on whether non-terminal fragments can have post-fragment options. There doesn’t appear to be reason for doing this; options that apply to the whole reassembled payload would occur after the last fragment and any options that apply to individual fragments appear before FRAG in each non-terminal fragment.
>
> #1 determines whether the data occurs in normal order or whether data in each fragment (including the final one) are moved around. Here’s what that looks like:
>
> - in normal order, each fragment’s data just occurs in-order in each fragment, e.g.,:
>
> abcde fghijkl mnopqrst
>
> - to support zero-copy, we previously had a technique that swapped bytes. Showing this for a full option is a bit complex; a simple version here would assume that each fragment had 3 bytes of options from the beginning of the option space to the start of the FRAG data (it’s always longer, but let’s make it 3 just to show it more simply). In that case we do the following:
>
> 123deabc 123ijklfgh 123parstmno
>
> The idea is that the data can be placed directly in memory, with the last 3 bytes overwriting the first three. By moving only the space covered by the options, direct data placement can be used:
>
> 123de - write up to the last 3 bytes
> a23de - overwrite the last 3 bytes starting at the beginning (next 2 lines too)

That presumes that the memory holding the payload is writeable which
is not always true. For instance, that would not be the case if the
device DMAs the data into GPU memory.

Tom

> ab3de
> abcde
>
> So this doesn’t affect the option format, but does affect whether FRAG data is in-place or wraps-around to support zero-copy.
>
> Joe