Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

[Chuck Lever <[e-mail]>]

> On Jun 4, 2016, at 6:35 AM, David Noveck <[e-mail]> wrote:
> 
> > Alright, then, "unreliable" might be closer to my
> > intended meaning.
> 
> I don't see any unreliability either.  I think we have 
> something which is difficult to generalize, because,
> as you point out, it has an unfortunate layering issue,
> which we are unable to fix in Version One because it
> is embedded in the Version One XDR.  We can fix it in subsequent versions where we can change the XDR but we have to take extra care because a lot of the layering confusion is located in the four words that we have defined as not subject to change :-(
> 
> > I disagree with that. Having a single credit flow, with
> > fixed roles of which side sends a request, and which
> > sends a grant, solves the problem simply and fully, for
> > all message types.
> 
> It solves the ambiguity problem but it creates a bunch of
> new ones.  I image that the first is interoperability with existing implementations, who I assume are using the model in rpcrdma-bidirection as it is today.
> 
> I don't see how one can implement bidirectional operation with a single credit pool.  The reason for credits is to ensure that there are sufficient preposted receives to correspond to sends that the peer makes.  Since there is a set of receives on each peer node then there needs to be a corresponding credit pool for each direction.  I don't see how you can have a situation in which both client and server may be requesters
> and have them share a single credit pool.

Fair enough. One set of credits for each responder on
a connection seems to align with what's already in the
documents, and discourages the addition of credit
flows for new message types.


> > It is also possible that this issue could be resolved
> > with language in rpcrdma-version-two, and allow the
> > one-credit-per-RPC concept to linger in V1 only.
> 
> Or alternatively, it could apply to both V1 and those
> parts of V2 that use the existing V1 message types.
> 
> > I think that would mean rpcrdma-bidirection is turning
> > into a V1-only enhancement. Bidirection would have
> > to be partially or fully specified again in rpcrdma-
> > version-two.
> 
> I don't think you need to do that, if you structure/layer things properly (see below for an approach).  Then any 
> further specification of bidirection for version two would  only address how the new classes of features we want to enable in version two would interact with the existing bidirectional operation.  In doing that we could use our 
> freedom to define the needed new XDR to avoid propagating the layering issue that you have identified.

I think you're suggesting that we have a problem only
for RDMA2_OPTIONAL, and that can be dealt with in
rpcrdma-version-two?


> > Having multiple different ways to do backward 
> > direction operation is onerous for implementations.
> 
> Agree.
> 
> > So I'd prefer to have this addressed in rfc5666bis and
> > rpcrdma-bidirection if we can muster it.
> 
> So here's the proposed division of responsibility:
> 	• The responsibility for defining the basic structure of bidirectional operation belongs to rpcrdma-bidirection.  This includes the use of two separate credit pools and the fact each message may contain either a request or a grant applying to the credit pool owned by the message receiver.

OK: the receiver has to decide whether each message
contains a credit request (the receiver is acting as
a responder) or a credit grant (the receiver is
acting as a requester).

I might refine the text of rpcrdma-bidirection in this
area, but otherwise I don't think there's a significant
change needed.


> 	• The responsibility for defining how it is to be determined, whether any particular message has a grant or a request in the credit field is the responsibility of the specific RPC-over-RDMA version being used.

I might add this suggestion to rpcrdma-bidrection.


> 	• The responsibility of deciding whether bidirectional operation is OPTIONAL, REQUIRED, or not allowed is the responsibility of the specific RPC-over-RDMA version being used.

I'm uncertain how that aligns with rfc5666bis, which
does not mention backwards direction operation at all.
This implies that it is allowed, and not REQUIRED. Is
that enough?


> 	• The text that now defines how the credit field for specfic Version One messages is interpreted as illustrative of version one only.

Is a text change needed for this, or are you simply
stating an operating assumption for spec writers?


> 	• draft-ietf-nfsv4-rpcrdma-bidirection explicitly takes on the responsibility for defining this for Version Two.  We have some issues to resolve which I'll address in a later message.

Not sure what "this" is, and not sure we want version
two anything in rpcrdma-bidirection. Awaiting your
later message!


Thanks for helping me with this. Let's try to close on
what is needed for rpcrdma-bidirection and try to get
a fresh revision by the end of this week.

--
Chuck Lever