Re: [tsvwg] RDMA Support by UDP FRAG Option

[Tom Herbert <[e-mail]>]

On Wed, Jun 16, 2021 at 5:16 PM Joseph Touch <[e-mail]> wrote:
>
>
>
> > On Jun 16, 2021, at 10:16 AM, Tom Herbert <[e-mail]> wrote:
> >
> >> ...
> >> The idea is that the data can be placed directly in memory, with the last 3 bytes overwriting the first three. By moving only the space covered by the options, direct data placement can be used:
> >>
> >> 123de - write up to the last 3 bytes
> >> a23de - overwrite the last 3 bytes starting at the beginning (next 2 lines too)
> >
> > That presumes that the memory holding the payload is writeable which
> > is not always true. For instance, that would not be the case if the
> > device DMAs the data into GPU memory.
>
> This approach is already OBE and thus not relevant.
>
If you don't like the receive problems, consider those on transmit.
The most common use case of zero copy is sendfile where an application
sends data directly from files, specifically this is done by DMA'ing
buffers in the page cache directly to the NIC without data going
through userspace application. This technique is extremely common in
video servers so there is no question that it's relevant. The
algorithm requires reading and writing three bytes of the application
payload.

If the algorithm is performed in the userspace application then we
need to read the first three bytes of the zero copy data into
userspace, perform a sendmsg consisting of at least three parts: the
three byte FRAG option, the zero copy data offset by three bytes,
followed by the trailer containing the originally value. So any time
we want to invoke sendfile we'd need to read part of the data that we
wanted to do zero copy on (expensive syscall). But there is another
subtle problem, between the time we read the first three bytes of data
and invoke the sendmsg, the page cache data may be changed by a writer
of the file. For instance, the first 128 bytes of the page cache may
have been modified such that the data actually sent is now corrupted.

If the algorithm is performed in the kernel, one might think that we
can just modify the page cache directly, however page cache cache is
shared so modifying it, even temporarily, risks data corruption for
another reader-- so this is a non-starter. We would need to perform
the same steps that I described above in the userspace case where
bytes are copied and we need to offset the zero copy data. The problem
of the page cache being modified still exists, although since we're in
the kernel we may be able to lock the pages during the operation.

In any case, to make zero copy work correctly for both TX and RX with
an algorithm requires manipulating application payload is going to be
much harder to implement and get right than zero copy in plain UDP or
TCP.

Tom

> However, the approach above remains an expectation of any DMA transfer - there are always edge aspects that need to be adjusted, whether by a separate DMA or some other mechanism. It doesn’t have to be one byte at a time, as shown above.
>
> Joe
>