Re: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt

[Christoph Hellwig <[e-mail]>]

On Tue, Jun 06, 2017 at 10:32:50AM -0400, David Noveck wrote:
> I'm clear that copies will kill performance, but I think you are wrong
> about this
> requiring copies.  The idea here is that the client will post as read
> buffers, aligned
> cache buffers not recently used.

What cache buffer?  The destination of a READ I/O might be any piece
of memory in the system, including user memory.  And even for the page
cache most modern OSes need to allocate the page first before doing
I/O as it's the synchronization object.

Your idea might work in the context of a system that only uses buffered
I/O, uses a fixed buffer cache like a Unix system designed in the 70s,
and lets you heavily hack^H^H^H^Hoptimize the core code for your
super specialized use case.  It ain't gonna happen in practice.

> assigned to receiving block-aligned data.   This is basically the same as
> the case in
> wich the client picks less-recently used buffer to use when reading from
> disk.

It's nothing at all like a page cache in the sense how most OSes use
it - we usually don't directly recycle 'buffers' but pages from the
general system allocator.  And they are page size (usually 4k or 64k(
and might not fit your I/O size at all.

If you want send based data placement to work you need tagged read
buffers so that the HCA uses exactly the read buffer your want, something
that is not supported by the current Verbs API, and not by any HCA
I know off.