Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

[David Noveck <[e-mail]>]

> I think you're suggesting that we have a problem only
> for RDMA2_OPTIONAL, and that can be dealt with in
> rpcrdma-version-two?

That was what I was suggesting, but I just read rpcdama-version-two-01
and so I think "can be dealt with in rpcrdma-version-two" can be replaced by
"has been dealt with in rpcrdma-version-two" :-)

>> The responsibility for defining the basic structure of bidirectional
operation belongs to rpcrdma-bidirection.
>> This includes the use of two separate credit pools and the fact each
message may contain either a request or a grant applying to the credit pool
owned by the message receiver.

> OK: the receiver has to decide whether each message
> contains a credit request (the receiver is acting as
> a responder) or a credit grant (the receiver is
> acting as a requester).

Yes but I'd boil this down to the receiver deciding whether the credit
field is a request or a grant (and the
RPC-over-RDMA version telling him how to decide.

The place where this gets "interesting" is a subcase of the feature you
brought up.  If a transmission can
contain multiple rpc messages, it can be the case the sender would want to
send both requests and replies
and thus want/need to send both a request and a grant.

I think we want to be careful that, where we specify how the credit field
is to be used, we don't foreclose
passing credits or grants in other fields.    If an XMOPT_GROUP were
defined, that allowed a group of
RPC messages to be sent, the version two sender would make the direction
field reflect the majority, but
I would want it to have the option to have an other-credit field so it
could convey a request and a grant in
the same transmission.

> I might refine the text of rpcrdma-bidirection in this
> area, but otherwise I don't think there's a significant
> change needed.

there might be minor text tweaking required.


> >       • The responsibility for defining how it is to be determined,
whether any particular message has a grant or a request in the credit field
is the responsibility of the specific RPC-over-RDMA version being used.

> I might add this suggestion to rpcrdma-bidrection.

OK, bu so far I hven't seen any obvious place to put it.


> >       • The responsibility of deciding whether bidirectional operation
is OPTIONAL, REQUIRED, or not allowed is the responsibility of the specific
RPC-over-RDMA version being used.

> I'm uncertain how that aligns with rfc5666bis, which
> does not mention backwards direction operation at all.
> This implies that it is allowed, and not REQUIRED. Is
> that enough?

I think what is in 5666bis is fine.  Section 8.1 introduces it as a
potential conventional extension and references rpcrdma-bidirection.


>>       • The text that now defines how the credit field for specfic
Version One messages is interpreted as illustrative of version one only.

> Is a text change needed for this, or are you simply
> stating an operating assumption for spec writers?

I was thinking of rewriting the last paragraph of section 4.1 of
rpcrdma-bidirection to read as follows:

When message direction is not fully determined by context or by an
accompanying RPC message with a call direction field, it may not be
clear whether the header credit value is a request or grant.  In such
cases the definition of the RPC-over-RDMA version used may provide
guidance as to how the credit value is to be interpreted. In such
cases in which such guidance is not provided or is adequate, the
receiver MUST NOT use the header's credit value.

I didn't see anything else that need to change.

>>       • draft-ietf-nfsv4-rpcrdma-bidirection explicitly takes on the
responsibility for defining this for Version Two.  We have some issues to
resolve which I'll address in a later message.

> Not sure what "this" is, and not sure we want version
> two anything in rpcrdma-bidirection. Awaiting your
> later message!

The first point is that I meant to type
draft-ietf-nfsv4-rpcrdma-version-two. :-(

The issue I was referring to was dealing with cases in which deciding
whether the credit was a request or a grant was hard either:

   - The message in question is, strictly speaking, neither an RPC request
   or a reply.  I think that is now dealt with in -01

You now allow the sender to decide whether the credit is a request or a
grant and the extension is free to tell him

how to choose.


   - The transmission might contain both requests and replies as discussed
   above.

I think that can be resolved in the next iteration of
draft-cel-nfsv4-rpcrdma-version-two.



On Mon, Jun 6, 2016 at 11:17 AM, Chuck Lever <[e-mail]> wrote:

>
> > On Jun 4, 2016, at 6:35 AM, David Noveck <[e-mail]> wrote:
> >
> > > Alright, then, "unreliable" might be closer to my
> > > intended meaning.
> >
> > I don't see any unreliability either.  I think we have
> > something which is difficult to generalize, because,
> > as you point out, it has an unfortunate layering issue,
> > which we are unable to fix in Version One because it
> > is embedded in the Version One XDR.  We can fix it in subsequent
> versions where we can change the XDR but we have to take extra care because
> a lot of the layering confusion is located in the four words that we have
> defined as not subject to change :-(
> >
> > > I disagree with that. Having a single credit flow, with
> > > fixed roles of which side sends a request, and which
> > > sends a grant, solves the problem simply and fully, for
> > > all message types.
> >
> > It solves the ambiguity problem but it creates a bunch of
> > new ones.  I image that the first is interoperability with existing
> implementations, who I assume are using the model in rpcrdma-bidirection as
> it is today.
> >
> > I don't see how one can implement bidirectional operation with a single
> credit pool.  The reason for credits is to ensure that there are sufficient
> preposted receives to correspond to sends that the peer makes.  Since there
> is a set of receives on each peer node then there needs to be a
> corresponding credit pool for each direction.  I don't see how you can have
> a situation in which both client and server may be requesters
> > and have them share a single credit pool.
>
> Fair enough. One set of credits for each responder on
> a connection seems to align with what's already in the
> documents, and discourages the addition of credit
> flows for new message types.
>
>
> > > It is also possible that this issue could be resolved
> > > with language in rpcrdma-version-two, and allow the
> > > one-credit-per-RPC concept to linger in V1 only.
> >
> > Or alternatively, it could apply to both V1 and those
> > parts of V2 that use the existing V1 message types.
> >
> > > I think that would mean rpcrdma-bidirection is turning
> > > into a V1-only enhancement. Bidirection would have
> > > to be partially or fully specified again in rpcrdma-
> > > version-two.
> >
> > I don't think you need to do that, if you structure/layer things
> properly (see below for an approach).  Then any
> > further specification of bidirection for version two would  only address
> how the new classes of features we want to enable in version two would
> interact with the existing bidirectional operation.  In doing that we could
> use our
> > freedom to define the needed new XDR to avoid propagating the layering
> issue that you have identified.
>
> I think you're suggesting that we have a problem only
> for RDMA2_OPTIONAL, and that can be dealt with in
> rpcrdma-version-two?
>
>
> > > Having multiple different ways to do backward
> > > direction operation is onerous for implementations.
> >
> > Agree.
> >
> > > So I'd prefer to have this addressed in rfc5666bis and
> > > rpcrdma-bidirection if we can muster it.
> >
> > So here's the proposed division of responsibility:
> >       • The responsibility for defining the basic structure of
> bidirectional operation belongs to rpcrdma-bidirection.  This includes the
> use of two separate credit pools and the fact each message may contain
> either a request or a grant applying to the credit pool owned by the
> message receiver.
>
> OK: the receiver has to decide whether each message
> contains a credit request (the receiver is acting as
> a responder) or a credit grant (the receiver is
> acting as a requester).
>
> I might refine the text of rpcrdma-bidirection in this
> area, but otherwise I don't think there's a significant
> change needed.
>
>
> >       • The responsibility for defining how it is to be determined,
> whether any particular message has a grant or a request in the credit field
> is the responsibility of the specific RPC-over-RDMA version being used.
>
> I might add this suggestion to rpcrdma-bidrection.
>
>
> >       • The responsibility of deciding whether bidirectional operation
> is OPTIONAL, REQUIRED, or not allowed is the responsibility of the specific
> RPC-over-RDMA version being used.
>
> I'm uncertain how that aligns with rfc5666bis, which
> does not mention backwards direction operation at all.
> This implies that it is allowed, and not REQUIRED. Is
> that enough?
>
>
> >       • The text that now defines how the credit field for specfic
> Version One messages is interpreted as illustrative of version one only.
>
> Is a text change needed for this, or are you simply
> stating an operating assumption for spec writers?
>
>
> >       • draft-ietf-nfsv4-rpcrdma-bidirection explicitly takes on the
> responsibility for defining this for Version Two.  We have some issues to
> resolve which I'll address in a later message.
>
> Not sure what "this" is, and not sure we want version
> two anything in rpcrdma-bidirection. Awaiting your
> later message!
>
>
> Thanks for helping me with this. Let's try to close on
> what is needed for rpcrdma-bidirection and try to get
> a fresh revision by the end of this week.
>
> --
> Chuck Lever
>
>
>
>