Re: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt

[David Noveck <[e-mail]>]

> For transfers
> from the server to the client (e.g. NFSv4 READ) the client has to place
> the data into arbitrary buffers not controller by the NFS client,
> so send-based data placement will lead to data copies on the client,
> which will kill performance.

I'm clear that copies will kill performance, but I think you are wrong
about this
requiring copies.  The idea here is that the client will post as read
buffers, aligned
cache buffers not recently used.  The RNIC is free to pick one of those
buffers
when a send comes in.  so the buffers are not precisely controlled, as they
are
with DDP, but they are not arbitrary either.  They are buffers that the
client has
assigned to receiving block-aligned data.   This is basically the same as
the case in
wich the client picks less-recently used buffer to use when reading from
disk.  The
only difference is that, to accommodate the RNIC, the client is not in
control of which
read goes with which buffer (among the set of pending reads).   The client
can find
out which buffer got which block and map them to user addresses the same
way it does
with data read from block devices.

> That being said I've started drafting and prototyping a Verbs extension
> for a tagged RECV WR.  Right now it's just a software implementation,
> and there is conѕiderable hardware complexity if we want to implement
> it in HCA.  I could talk about this idea a bit in Prague, but I'm not
> sure about the venue -

Ultimately, you might wind up discussing this in a blue-sheet-less
environment.  For example, over a beer.

If you can send out the draft, I'd be interested in seeing it.  Others
probably
would as well.

> it's clearly not NFSv4 WG material,

Probably not, and given that the schedule is tight, there isn;t much point
trying
that.  Still, I think some people in the working group would be interested.

> and there
> doesn't seem to be a follow on to the STORM WG.  Ccing Dave Black if
> he has any idea.

I there should be some follow-on.  Maybe A BOF could be thought about for a
future meeting.


On Tue, Jun 6, 2017 at 2:42 AM, Christoph Hellwig <[e-mail]> wrote:

> Hi David,
>
> send based data placement in it's current form only makes sense for
> transfers from the client to the server (e.g. NFSv4 WRITE).  For transfers
> from the server to the client (e.g. NFSv4 READ) the client has to place
> the data into arbitrary buffers not controller by the NFS client,
> so send-based data placement will lead to data copies on the client,
> which will kill performance.  I'm speaking from experience because we
> actually implemented this to verify the performance overhead for copies
> in early NVMe over Fabrics prototypes.
>
> That being said I've started drafting and prototyping a Verbs extension
> for a tagged RECV WR.  Right now it's just a software implementation,
> and there is conѕiderable hardware complexity if we want to implement
> it in HCA.  I could talk about this idea a bit in Prague, but I'm not
> sure about the venue - it's clearly not NFSv4 WG material, and there
> doesn't seem to be a follow on to the STORM WG.  Ccing Dave Black if
> he has any idea.
>
> Also I find the idea to split requests into multiple sends just to
> avoid RDMA ops very questionable - the latency advantage of SEND
> from the client to the server vs RDMA read from the server mostly
> matters to reduce the latency for small I/Os.  For large overheads
> the registration (or in fact usually unregistration) overhead does
> not matter in the grand scheme.  Last but not least modern RDMA
> protocols have show us that we should simplify credit management
> instead of making it even more complex as that has been one of the
> major sources of bugs.
>
> On Mon, Jun 05, 2017 at 11:15:04AM -0400, David Noveck wrote:
> > Submitted with a small set of revisions to avoid expiration:
> >
> >    - New contact information
> >    - Updated references
> >    - Changed some terminology.  "Send-based DDP" has become "send-based
> >    data placement", while "DDP" is limited to data placement using
> explicit
> >    RDMA operations.  Since there are corresponding XDR changes, this
> makes the
> >    diff large, but the actual extensions is pretty much unchanged
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: <[e-mail]>
> > Date: Mon, Jun 5, 2017 at 10:58 AM
> > Subject: New Version Notification for
> > draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt
> > To: David Noveck <[e-mail]>
> >
> >
> >
> > A new version of I-D, draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt
> > has been successfully submitted by David Noveck and posted to the
> > IETF repository.
> >
> > Name:           draft-dnoveck-nfsv4-rpcrdma-rtrext
> > Revision:       02
> > Title:          RPC-over-RDMA Extensions to Reduce Internode Round-trips
> > Document date:  2017-06-05
> > Group:          Individual Submission
> > Pages:          44
> > URL:            https://www.ietf.org/internet-
> drafts/draft-dnoveck-nfsv4-
> > rpcrdma-rtrext-02.txt
> > Status:         https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-
> > rpcrdma-rtrext/
> > Htmlized:       https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-
> > rtrext-02
> > Htmlized:       https://datatracker.ietf.org/
> doc/html/draft-dnoveck-nfsv4-
> > rpcrdma-rtrext-02
> > Diff:           https://www.ietf.org/rfcdiff?url2=draft-dnoveck-nfsv4-
> > rpcrdma-rtrext-02
> >
> > Abstract:
> >    It is expected that a future version of the RPC-over-RDMA transport
> >    will allow protocol extensions to be defined.  This would provide for
> >    the specification of OPTIONAL features allowing participants who
> >    implement such features to cooperate as specified by that extension,
> >    while still interoperating with participants who do not support that
> >    extension.
> >
> >    A particular extension is described herein, whose purpose is to
> >    reduce the latency due to inter-node round-trips needed to effect
> >    operations which involve direct data placement or which transfer RPC
> >    messages longer than the fixed inline buffer size limit.
> >
> >
> >
> >
> > Please note that it may take a couple of minutes from the time of
> submission
> > until the htmlized version and diff are available at tools.ietf.org.
> >
> > The IETF Secretariat
>
> > _______________________________________________
> > nfsv4 mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/nfsv4
>
> ---end quoted text---
>