Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

[Chuck Lever <[e-mail]>]

> On Jun 3, 2016, at 2:31 PM, David Noveck <[e-mail]> wrote:
> 
> > I concur that rfc5666bis and rfc5666-implementation-experience
> > are ready to move forward.
> 
> I guess the implication is that bidirection is not ready and I'm unclear about the reason for that.
> 
> > However, I've been thinking about an earlier comment Dave made
> > in this thread:
> 
> > > My only concern is with regard to a possible future extension in which multiple RPC messages are carried in a single SEND (i.e., the precise opposite of message continuation).  
> > > When the language for the new field is drafted, we should make sure that it doesn't assume all the messages in a SEND are all related to a single direction of operation.
> 
> > There seem to be two related issues, and one of them has
> > ramifications for rpcrdma-bidirection.
> 
> A few things I'd like to note here:
> 	• As far as I'm concerned, I only raised a single issue.
> 	• My issue is only with regard to how to deal with a potential Version Two extension issue.
> 	• When I said it was my "only concern" (emphasis added), I hoped to indicate that, as a potential future extension, this should not hold us up now.
> 	• Although it may relate to bidirection in general, I don't think it affects use of it in the Version one case. 

I. relates to the Version One case because we may want
to rewrite parts of the rpcrdma-bidirection I-D, which
has a direct impact on NFSv4.1 on RPC-over-RDMA V1.

And recall that we stated earlier that rpcrdma-bidirection
was not meant to apply solely to V1. The current V2
specification requires support for RPC bidirection as
specified in rpcrdma-bidirection.


> > I. Ambiguity of the meaning of the credit field
> >
> > In the rfc5666bis world, the credit field in all messages
> > flowing on a connection had an unambiguous meaning. If
> > the message was going from requester to responder, the
> > credit field was a credit request. In the other direction,
> > it was a credit grant.
> 
> That remains the case, even with bidirection.  Your last
> two sentences are still true.

> The problem is that, when bidirection is in effect, it is not
> always clear who is the requester and who is the responder:
> The sendr always knows but the receiver might not know and,
> in case where a message is not decipherable, the question 
> might not be answerable.
> 
> So,
> 	• With only forward direction operation, If the message was going from client to server, the credit field is a credit request. In the other direction, it is a credit grant.
> 	• With bidirectional operation, the corresponding statement with requester/responder is still true, but the code in the client and server whenthey receive a message do not know if thety are action as requester or responder :-(
> > rpcrdma-bidirection introduced the idea of two separate
> > credit flows, which depend on whether the message was
> > part of forward RPC operation or backward RPC operation
> > on that connection.

There may be problematic language in rfc5666bis as well.

RFC 5666 Section 3.3 attaches credit requests to RPC Call
messages and credit grants to RPC Reply messages.

The updated language in rfc5666bis has this: credit
request and grant is described in Sections 4.3.1 and 5.2.3
as being tied to requester and responder.

So, this is a problem not only for bidirectional operation
but also for any type of message where there are multiple
or no RPC messages associated with the RPC-over-RDMA
message.

My proposal is this:

The roles of client (active connector) and server (passive
accepter) are always the same for a given connection, and
are independent of direction of RPC operation.

Thus there is never any ambiguity about what the credit
field means: the active side always makes requests, and
the passive side always makes grants.


> > This is the source of the problem.
> 
> It solved one problem, but it created another.  The question we have is
> what to do about it.

Agreed.


> > The meaning of the credit field depends on a field in
> > the upper layer, so it's a layering violation, to say
> > the least.
> 
> I agree that it is layering violation and it may even be a layering misdemeanor.
> I don't think it is a layering felony, and I hope we can avoid any
> Draconian penalty.
> 
> > Also, the original concept of credit was to manage RDMA
> > receive buffers, but rpcrdma-bidirection interpretation
> > of the credit value is one-credit-per-RPC.
> 
> If it switched to that, I think of it as a purely verbal mistake, which is understandable 
> given the context, i.e. that we are in a one-message-per-RPC environment.  In any case,
> I don't see how that shift is related to bidirectional operation.

Given that the one-credit-per-RPC language appears as
early as RFC 5666, perhaps you are correct.


> > Now if we want to add support for partial RPC message
> > per credit (message continuation) 
> 
> I think we do.  I discussed doing that in draft-dnoveck-nfsv4-rprcrdma-rtissuues-00.
> 
> I will discuss the credit management issues in draft-dnoveck-nfsv4-rocrdma-rtrext-00,
> but I'm not sure if I'll be able to submit that before IETF96. 
> 
> > or multiple RPC
> > messages per credit (as proposed above), 
> 
> I haven't actually proposed that but I agree that we don't want to foreclose this option.
> 
> > or NO RPC
> > message per credit (for RDMA2_OPTIONAL control messages)
> 
> I actually have proposed messages that don't carry RPC in draft-dnoveck-nfsv4-rpcrdma-xcharext-00.

Sorry, the above "if" was a rhetorical "if". I do
presume that all three proposals are interesting to
pursue.


> I assumed that they required a credit because receiving the message requires a buffer and there
> is no way to avoid the fact that a buffer (and this a credit) is used up.  This is true whether the credit 
> field is a request or a grant or is ignored.
> 
> > the credit field is unusable.
> 
> My formulation would be "problematic for some classes of future extensions to the next RPC-over-RDMA
> version".  That's light-years (or even megaparsecs) away from "unusable".

Alright, then, "unreliable" might be closer to my
intended meaning.


> > Would we add an independent pool of credits for each of
> > these transmission mechanisms? Probably not.
> 
> I agree that that is a REAL BAD IDEA. 
> 
> > A logical course of action, then, would be to alter the
> > rpcrdma-bidirection I-D so that forward and backward
> > direction operation use the same pool of credits.
> 
> I think this is a possible course of action.  I don't think it is 
> logical because:
> 	• The potential ambiguity you are worried about is between credit and grant, and having a single pool wouldn't solve that.
> 	• There is no ambiguity about which credit pool you are dealing with, so this, not being broken, doesn't require a fix.

I disagree with that. Having a single credit flow, with
fixed roles of which side sends a request, and which
sends a grant, solves the problem simply and fully, for
all message types.


> 	• There are design/supplementation issues that you note below that may make this infeasible
> 
> > The question is whether it is feasible for the two
> > directions to share the credit pools without deadlocking.
> > Some prototyping and/or careful thought would be needed
> > to answer it.

It is also possible that this issue could be resolved
with language in rpcrdma-version-two, and allow the
one-credit-per-RPC concept to linger in V1 only.

I think that would mean rpcrdma-bidirection is turning
into a V1-only enhancement. Bidirection would have to
be partially or fully specified again in
rpcrdma-version-two.

Having multiple different ways to do backward direction
operation is onerous for implementations.

So I'd prefer to have this addressed in rfc5666bis and
rpcrdma-bidirection if we can muster it.


> > II. Ambiguity of the meaning of the XID field
> 
> I think this issue has no implications for rpcrdma-bidirection.

I agree. We can set this one aside for the moment.


--
Chuck Lever