Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

[David Noveck <[e-mail]>]

> Alright, then, "unreliable" might be closer to my
> intended meaning.

I don't see any unreliability either.  I think we have
something which is difficult to generalize, because,
as you point out, it has an unfortunate layering issue,
which we are unable to fix in Version One because it
is embedded in the Version One XDR.  We can fix it in subsequent versions
where we can change the XDR but we have to take extra care because a lot of
the layering confusion is located in the four words that we have defined as
not subject to change :-(

> I disagree with that. Having a single credit flow, with
> fixed roles of which side sends a request, and which
> sends a grant, solves the problem simply and fully, for
> all message types.

It solves the ambiguity problem but it creates a bunch of
new ones.  I image that the first is interoperability with existing
implementations, who I assume are using the model in rpcrdma-bidirection as
it is today.

I don't see how one can implement bidirectional operation with a single
credit pool.  The reason for credits is to ensure that there are sufficient
preposted receives to correspond to sends that the peer makes.  Since there
is a set of receives on each peer node then there needs to be a
corresponding credit pool for each direction.  I don't see how you can have
a situation in which both client and server may be requesters
and have them share a single credit pool.

> It is also possible that this issue could be resolved
> with language in rpcrdma-version-two, and allow the
> one-credit-per-RPC concept to linger in V1 only.

Or alternatively, it could apply to both V1 and those
parts of V2 that use the existing V1 message types.

> I think that would mean rpcrdma-bidirection is turning
> into a V1-only enhancement. Bidirection would have
> to be partially or fully specified again in rpcrdma-
> version-two.

I don't think you need to do that, if you structure/layer things properly
(see below for an approach).  Then any
further specification of bidirection for version two would  only address
how the new classes of features we want to enable in version two would
interact with the existing bidirectional operation.  In doing that we could
use our
freedom to define the needed new XDR to avoid propagating the layering
issue that you have identified.

> Having multiple different ways to do backward
> direction operation is onerous for implementations.

Agree.

> So I'd prefer to have this addressed in rfc5666bis and
> rpcrdma-bidirection if we can muster it.

So here's the proposed division of responsibility:

   - The responsibility for defining the basic structure of bidirectional
   operation belongs to rpcrdma-bidirection.  This includes the use of two
   separate credit pools and the fact each message may contain either a
   request or a grant applying to the credit pool owned by the message
   receiver.
   - The responsibility for defining how it is to be determined, whether
   any particular message has a grant or a request in the credit field is the
   responsibility of the specific RPC-over-RDMA version being used.
   - The responsibility of deciding whether bidirectional operation is
   OPTIONAL, REQUIRED, or not allowed  is the responsibility of the specific
   RPC-over-RDMA version being used.
   - The text that now defines how the credit field for specfic Version One
   messages is interpreted as illustrative of version one only.
   - draft-ietf-nfsv4-rpcrdma-bidirection explicitly takes on the
   responsibility for defining this for Version Two.  We have some  issues to
   resolve which I'll address in a later message.




On Fri, Jun 3, 2016 at 3:34 PM, Chuck Lever <[e-mail]> wrote:

>
> > On Jun 3, 2016, at 2:31 PM, David Noveck <[e-mail]> wrote:
> >
> > > I concur that rfc5666bis and rfc5666-implementation-experience
> > > are ready to move forward.
> >
> > I guess the implication is that bidirection is not ready and I'm unclear
> about the reason for that.
> >
> > > However, I've been thinking about an earlier comment Dave made
> > > in this thread:
> >
> > > > My only concern is with regard to a possible future extension in
> which multiple RPC messages are carried in a single SEND (i.e., the precise
> opposite of message continuation).
> > > > When the language for the new field is drafted, we should make sure
> that it doesn't assume all the messages in a SEND are all related to a
> single direction of operation.
> >
> > > There seem to be two related issues, and one of them has
> > > ramifications for rpcrdma-bidirection.
> >
> > A few things I'd like to note here:
> >       • As far as I'm concerned, I only raised a single issue.
> >       • My issue is only with regard to how to deal with a potential
> Version Two extension issue.
> >       • When I said it was my "only concern" (emphasis added), I hoped
> to indicate that, as a potential future extension, this should not hold us
> up now.
> >       • Although it may relate to bidirection in general, I don't think
> it affects use of it in the Version one case.
>
> I. relates to the Version One case because we may want
> to rewrite parts of the rpcrdma-bidirection I-D, which
> has a direct impact on NFSv4.1 on RPC-over-RDMA V1.
>
> And recall that we stated earlier that rpcrdma-bidirection
> was not meant to apply solely to V1. The current V2
> specification requires support for RPC bidirection as
> specified in rpcrdma-bidirection.
>
>
> > > I. Ambiguity of the meaning of the credit field
> > >
> > > In the rfc5666bis world, the credit field in all messages
> > > flowing on a connection had an unambiguous meaning. If
> > > the message was going from requester to responder, the
> > > credit field was a credit request. In the other direction,
> > > it was a credit grant.
> >
> > That remains the case, even with bidirection.  Your last
> > two sentences are still true.
>
> > The problem is that, when bidirection is in effect, it is not
> > always clear who is the requester and who is the responder:
> > The sendr always knows but the receiver might not know and,
> > in case where a message is not decipherable, the question
> > might not be answerable.
> >
> > So,
> >       • With only forward direction operation, If the message was going
> from client to server, the credit field is a credit request. In the other
> direction, it is a credit grant.
> >       • With bidirectional operation, the corresponding statement with
> requester/responder is still true, but the code in the client and server
> whenthey receive a message do not know if thety are action as requester or
> responder :-(
> > > rpcrdma-bidirection introduced the idea of two separate
> > > credit flows, which depend on whether the message was
> > > part of forward RPC operation or backward RPC operation
> > > on that connection.
>
> There may be problematic language in rfc5666bis as well.
>
> RFC 5666 Section 3.3 attaches credit requests to RPC Call
> messages and credit grants to RPC Reply messages.
>
> The updated language in rfc5666bis has this: credit
> request and grant is described in Sections 4.3.1 and 5.2.3
> as being tied to requester and responder.
>
> So, this is a problem not only for bidirectional operation
> but also for any type of message where there are multiple
> or no RPC messages associated with the RPC-over-RDMA
> message.
>
> My proposal is this:
>
> The roles of client (active connector) and server (passive
> accepter) are always the same for a given connection, and
> are independent of direction of RPC operation.
>
> Thus there is never any ambiguity about what the credit
> field means: the active side always makes requests, and
> the passive side always makes grants.
>
>
> > > This is the source of the problem.
> >
> > It solved one problem, but it created another.  The question we have is
> > what to do about it.
>
> Agreed.
>
>
> > > The meaning of the credit field depends on a field in
> > > the upper layer, so it's a layering violation, to say
> > > the least.
> >
> > I agree that it is layering violation and it may even be a layering
> misdemeanor.
> > I don't think it is a layering felony, and I hope we can avoid any
> > Draconian penalty.
> >
> > > Also, the original concept of credit was to manage RDMA
> > > receive buffers, but rpcrdma-bidirection interpretation
> > > of the credit value is one-credit-per-RPC.
> >
> > If it switched to that, I think of it as a purely verbal mistake, which
> is understandable
> > given the context, i.e. that we are in a one-message-per-RPC
> environment.  In any case,
> > I don't see how that shift is related to bidirectional operation.
>
> Given that the one-credit-per-RPC language appears as
> early as RFC 5666, perhaps you are correct.
>
>
> > > Now if we want to add support for partial RPC message
> > > per credit (message continuation)
> >
> > I think we do.  I discussed doing that in
> draft-dnoveck-nfsv4-rprcrdma-rtissuues-00.
> >
> > I will discuss the credit management issues in
> draft-dnoveck-nfsv4-rocrdma-rtrext-00,
> > but I'm not sure if I'll be able to submit that before IETF96.
> >
> > > or multiple RPC
> > > messages per credit (as proposed above),
> >
> > I haven't actually proposed that but I agree that we don't want to
> foreclose this option.
> >
> > > or NO RPC
> > > message per credit (for RDMA2_OPTIONAL control messages)
> >
> > I actually have proposed messages that don't carry RPC in
> draft-dnoveck-nfsv4-rpcrdma-xcharext-00.
>
> Sorry, the above "if" was a rhetorical "if". I do
> presume that all three proposals are interesting to
> pursue.
>
>
> > I assumed that they required a credit because receiving the message
> requires a buffer and there
> > is no way to avoid the fact that a buffer (and this a credit) is used
> up.  This is true whether the credit
> > field is a request or a grant or is ignored.
> >
> > > the credit field is unusable.
> >
> > My formulation would be "problematic for some classes of future
> extensions to the next RPC-over-RDMA
> > version".  That's light-years (or even megaparsecs) away from "unusable".
>
> Alright, then, "unreliable" might be closer to my
> intended meaning.
>
>
> > > Would we add an independent pool of credits for each of
> > > these transmission mechanisms? Probably not.
> >
> > I agree that that is a REAL BAD IDEA.
> >
> > > A logical course of action, then, would be to alter the
> > > rpcrdma-bidirection I-D so that forward and backward
> > > direction operation use the same pool of credits.
> >
> > I think this is a possible course of action.  I don't think it is
> > logical because:
> >       • The potential ambiguity you are worried about is between credit
> and grant, and having a single pool wouldn't solve that.
> >       • There is no ambiguity about which credit pool you are dealing
> with, so this, not being broken, doesn't require a fix.
>
> I disagree with that. Having a single credit flow, with
> fixed roles of which side sends a request, and which
> sends a grant, solves the problem simply and fully, for
> all message types.
>
>
> >       • There are design/supplementation issues that you note below that
> may make this infeasible
> >
> > > The question is whether it is feasible for the two
> > > directions to share the credit pools without deadlocking.
> > > Some prototyping and/or careful thought would be needed
> > > to answer it.
>
> It is also possible that this issue could be resolved
> with language in rpcrdma-version-two, and allow the
> one-credit-per-RPC concept to linger in V1 only.
>
> I think that would mean rpcrdma-bidirection is turning
> into a V1-only enhancement. Bidirection would have to
> be partially or fully specified again in
> rpcrdma-version-two.
>
> Having multiple different ways to do backward direction
> operation is onerous for implementations.
>
> So I'd prefer to have this addressed in rfc5666bis and
> rpcrdma-bidirection if we can muster it.
>
>
> > > II. Ambiguity of the meaning of the XID field
> >
> > I think this issue has no implications for rpcrdma-bidirection.
>
> I agree. We can set this one aside for the moment.
>
>
> --
> Chuck Lever
>
>
>
>