Re: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt

Hi David,

send based data placement in it's current form only makes sense for
transfers from the client to the server (e.g. NFSv4 WRITE).  For transfers
from the server to the client (e.g. NFSv4 READ) the client has to place
the data into arbitrary buffers not controller by the NFS client,
so send-based data placement will lead to data copies on the client,
which will kill performance.  I'm speaking from experience because we
actually implemented this to verify the performance overhead for copies
in early NVMe over Fabrics prototypes.

That being said I've started drafting and prototyping a Verbs extension
for a tagged RECV WR.  Right now it's just a software implementation,
and there is conѕiderable hardware complexity if we want to implement
it in HCA.  I could talk about this idea a bit in Prague, but I'm not
sure about the venue - it's clearly not NFSv4 WG material, and there
doesn't seem to be a follow on to the STORM WG.  Ccing Dave Black if
he has any idea.

Also I find the idea to split requests into multiple sends just to
avoid RDMA ops very questionable - the latency advantage of SEND
from the client to the server vs RDMA read from the server mostly
matters to reduce the latency for small I/Os.  For large overheads
the registration (or in fact usually unregistration) overhead does
not matter in the grand scheme.  Last but not least modern RDMA
protocols have show us that we should simplify credit management
instead of making it even more complex as that has been one of the
major sources of bugs.

On Mon, Jun 05, 2017 at 11:15:04AM -0400, David Noveck wrote:
> Submitted with a small set of revisions to avoid expiration:
> 
>    - New contact information
>    - Updated references
>    - Changed some terminology.  "Send-based DDP" has become "send-based
>    data placement", while "DDP" is limited to data placement using explicit
>    RDMA operations.  Since there are corresponding XDR changes, this makes the
>    diff large, but the actual extensions is pretty much unchanged
> 
> 
> 
> ---------- Forwarded message ----------
> From: <internet-drafts@ietf.org>
> Date: Mon, Jun 5, 2017 at 10:58 AM
> Subject: New Version Notification for
> draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt
> To: David Noveck <davenoveck@gmail.com>
> 
> 
> 
> A new version of I-D, draft-dnoveck-nfsv4-rpcrdma-rtrext-02.txt
> has been successfully submitted by David Noveck and posted to the
> IETF repository.
> 
> Name:           draft-dnoveck-nfsv4-rpcrdma-rtrext
> Revision:       02
> Title:          RPC-over-RDMA Extensions to Reduce Internode Round-trips
> Document date:  2017-06-05
> Group:          Individual Submission
> Pages:          44
> URL:            https://www.ietf.org/internet-drafts/draft-dnoveck-nfsv4-
> rpcrdma-rtrext-02.txt
> Status:         https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-
> rpcrdma-rtrext/
> Htmlized:       https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-
> rtrext-02
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-dnoveck-nfsv4-
> rpcrdma-rtrext-02
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-dnoveck-nfsv4-
> rpcrdma-rtrext-02
> 
> Abstract:
>    It is expected that a future version of the RPC-over-RDMA transport
>    will allow protocol extensions to be defined.  This would provide for
>    the specification of OPTIONAL features allowing participants who
>    implement such features to cooperate as specified by that extension,
>    while still interoperating with participants who do not support that
>    extension.
> 
>    A particular extension is described herein, whose purpose is to
>    reduce the latency due to inter-node round-trips needed to effect
>    operations which involve direct data placement or which transfer RPC
>    messages longer than the fixed inline buffer size limit.
> 
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat

> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4

---end quoted text---