Re: [tsvwg] RDMA Support by UDP FRAG Option

[Joe Touch <[e-mail]>]

Tom, 

This thread is OBE because we are no longer proposing individual byte moves at all. I posted the new frag option earlier this week.

Joe

> On Jun 18, 2021, at 9:44 AM, Tom Herbert <[e-mail]> wrote:
> 
> On Wed, Jun 16, 2021 at 5:16 PM Joseph Touch <[e-mail]> wrote:
>> 
>> 
>> 
>>>> On Jun 16, 2021, at 10:16 AM, Tom Herbert <[e-mail]> wrote:
>>> 
>>>> ...
>>>> The idea is that the data can be placed directly in memory, with the last 3 bytes overwriting the first three. By moving only the space covered by the options, direct data placement can be used:
>>>> 
>>>> 123de - write up to the last 3 bytes
>>>> a23de - overwrite the last 3 bytes starting at the beginning (next 2 lines too)
>>> 
>>> That presumes that the memory holding the payload is writeable which
>>> is not always true. For instance, that would not be the case if the
>>> device DMAs the data into GPU memory.
>> 
>> This approach is already OBE and thus not relevant.
>> 
> If you don't like the receive problems, consider those on transmit.
> The most common use case of zero copy is sendfile where an application
> sends data directly from files, specifically this is done by DMA'ing
> buffers in the page cache directly to the NIC without data going
> through userspace application. This technique is extremely common in
> video servers so there is no question that it's relevant. The
> algorithm requires reading and writing three bytes of the application
> payload.
> 
> If the algorithm is performed in the userspace application then we
> need to read the first three bytes of the zero copy data into
> userspace, perform a sendmsg consisting of at least three parts: the
> three byte FRAG option, the zero copy data offset by three bytes,
> followed by the trailer containing the originally value. So any time
> we want to invoke sendfile we'd need to read part of the data that we
> wanted to do zero copy on (expensive syscall). But there is another
> subtle problem, between the time we read the first three bytes of data
> and invoke the sendmsg, the page cache data may be changed by a writer
> of the file. For instance, the first 128 bytes of the page cache may
> have been modified such that the data actually sent is now corrupted.
> 
> If the algorithm is performed in the kernel, one might think that we
> can just modify the page cache directly, however page cache cache is
> shared so modifying it, even temporarily, risks data corruption for
> another reader-- so this is a non-starter. We would need to perform
> the same steps that I described above in the userspace case where
> bytes are copied and we need to offset the zero copy data. The problem
> of the page cache being modified still exists, although since we're in
> the kernel we may be able to lock the pages during the operation.
> 
> In any case, to make zero copy work correctly for both TX and RX with
> an algorithm requires manipulating application payload is going to be
> much harder to implement and get right than zero copy in plain UDP or
> TCP.
> 
> Tom
> 
>> However, the approach above remains an expectation of any DMA transfer - there are always edge aspects that need to be adjusted, whether by a separate DMA or some other mechanism. It doesn’t have to be one byte at a time, as shown above.
>> 
>> Joe
>> 
>