Re: [nfsv4] review of draft-ietf-nfsv4-rpcrdma-bidirection-03

[Spencer Shepler <[e-mail]>]

Thanks for your opinion, Dave.  We will wait to ensure that no one else in the WG has comments.

Spencer

From: nfsv4 [mailto:[e-mail]] On Behalf Of David Noveck
Sent: Friday, May 20, 2016 9:52 AM
To: Chuck Lever <[e-mail]>
Cc: [e-mail]
Subject: Re: [nfsv4] review of draft-ietf-nfsv4-rpcrdma-bidirection-03

Works for me.

I think we can now (or at midnight) proceed to move these documents into WG Consensus state.

On Fri, May 20, 2016 at 11:34 AM, Chuck Lever <[e-mail]<mailto:[e-mail]>> wrote:

> On May 19, 2016, at 11:52 AM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:
>
> > It seems natural to me because, logically, you
> > can't have bi-directional operation without
> > backward direction operation.
>
> That's likely to be the case, but note that, you can't have bi-directional operation
> without forward direction operation either, and it somehow doesn't seem natural
> to identify forward and bi-directional operation.  I think logic does not have as
> much to do with this as you assume.
>
> > > If this is the case, then bidirectional protocols  do not currently exist but might be defined in the future.
>
> > I agree with that.
>
> Yes, but it seems we disagree about whether they currently exist because we use the word "protocol" in
> different ways.  This is not a big problem right now but I'm worried about a situation in which two of the
> documents that NFSv4 implementers refer to use such a basic term in such different ways.


The language in Section 7 of -03 was indeed problematic.
It contradicted, for example, our long-standing claim
that RFC 5667 is missing DDP-eligibility statements for
callback operations. (I think part of the reason for
the problematic language was the desire to grandfather
NFSv4 callback operations into existing RFC 5667 language,
and perhaps that is not consistent with what has been
written and discussed so far).

I've rewritten Section 7:


7.  Considerations For Upper Layer Bindings

   An Upper Layer Protocol that operates on RPC-over-RDMA transports may
   have procedures that include DDP-eligible data items.  DDP-
   eligibility is specified in an Upper Layer Binding.  Direction of
   operation does not obviate the need for DDP-eligibility statements.

   Backward-only operation requires the client endpoint to establish a
   fresh connection.  The Upper Layer Binding can specify appropriate
   RPC binding parameters for such connections.

   Bi-directional operation occurs on an already-established connection.
   Specification of RPC binding parameters is usually not necessary in
   this case.

   For bi-directional operation, other considerations about sharing an
   RPC-over-RDMA transport with another ULP may apply.  Consult
   Section 7 of [I-D.ietf-nfsv4-rfc5666bis] for details about what else
   may be contained in an Upper Layer Binding.


Hopefully that addresses your concerns about conflating
backward direction and bidirection, and issues with the
use of the term 'Upper Layer Protocol.'


> On Thu, May 19, 2016 at 10:13 AM, Chuck Lever <[e-mail]<mailto:[e-mail]>> wrote:
> Hi Dave-
>
>
> > On May 18, 2016, at 10:21 PM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:
> >
> > > How do we define Upper Layer Protocol? An Upper Layer
> > > Binding applies to an Upper Layer Protocol, which is
> > > defined as a distinct RPC Program and Version. See
> > > Section 7 rfc5666bis:
> >
> > I think it would be helpful if the defnition of "Upper Layer Protocol"
> > matched the way the word "protocol" is normally used.  And NFSv4.1
> > is, quite reasonably, described as a protocol, and not as a set of
> > two protocols described alongside one another.
>
> But we do make a distinction between NLM, NFSACL,
> MOUNT, NSM, and NFS protocols, referring to NFS
> as the "main protocol" and the others as "side-band
> protocols."
>
> In NFSv4.0, forward operation and callback are
> always used on distinct connections, and they have
> distinct RPC Program numbers and separate XDR
> definitions. Forward operation can occur without
> a callback channel. Are those considered one
> protocol, or two?
>
> Is NFSv3 the same ULP as NFSv4.0 or NFSv4.1? If so,
> why are there separate discussions in RFC 5667 for
> v3 and v4.0, and why are we not attempting to claim
> that RFC 5667 or rfc5667bis are normative for all
> future versions of NFS?
>
> Thus I don't consider the colloquial usage of the
> term 'protocol' to be adequate in this context.
>
>
> > > RFC 5661/2 define two separate RPC Programs, thus
> > > separate ULBs are needed, according to rfc5666bis.
> >
> > > This is the way the I-Ds are written now. I am
> > > willing to entertain alternatives.
> >
> > OK.  First of all it might be stated that when two separate RPC
> > programs are defined together and used together as a  single protocol capable of bidirectional operation, they can be
> > considered as a single protocol with a ULB,covering both
> > directions of operation.
>
> Except that last time I included a mention of call
> direction in rfc5666bis, you objected to that. :-)
>
> I still prefer the way rfc5666bis is written now.
> It is unambiguous about what needs to have a
> Binding. The boundaries defined by Program and
> Version number are historical, simple to
> understand, and well-defined.
>
> (And, this is a valuable conversation, since we
> as a WG should be striving to get this right).
>
>
> > > I don't see how backward-only and bidirectional operation
> > > are addressed separately, except for the separate definitions
> > > in Section 2.
> >
> > The introduction to the subject defines forward operation,
> > backward operation, and bi-directional operation, all on the
> > same level, as the three classes of RPC directionality.
> >
> > > Can you give other examples of where they are
> > > separately treated?
> >
> > No I can't and part of the reason is that,  in most contexts, you are talking about a request or response that part of a forward or backward RPC.  there are no bidirectional RPC's even though there are (or may be) bidirectional protocols.
> >
> > It is quite natural in the context to treat backward an bidirectional
> > together.   The issue is that if you introduce these three classes and then choose to treat two of them together, you need to tell the reader
> > that you are gong to do so.  You may think it is natural and obvious to treat these together but you can't assume the reader will
> > naturally share that view, given he has been told that these are separate things.
>
> It seems natural to me because, logically, you
> can't have bi-directional operation without
> backward direction operation. I can review the
> text with your comment in mind, however.
>
> Besides the sentence you proposed in Section 7,
> are there other places in the document where I
> should add "and bi-directional" or its equivalent?
> I'm open to suggestion.
>
>
> > Of course,, we now have to deal with the irony that arises from the fact that you originally began this work to enable the NFSv4.1 protocol to operate bidirectionally, and having done that, you have (magically :-) made the NFSv4.1 protocol disappear  and with it bidirectional operation.  If I understand your position correctly, we now no longer have a single protocol operating bidirectionally.
>
> When it is broken down, the RPC transport, not the
> protocol, operates bi-directionally. For example:
>
> A "uni-directional" protocol such as NFSv3 or NLM
> can be used bi-directionally, if two connection
> endpoints act as both clients and servers.
>
> NFSv4.1 implementations can set up distinct and
> separate connections for forward and backward
> operation. In that case, no bi-directional
> operation is occurring on either connection (after
> the BC2S operation on the callback connection).
>
>
> > Instead, we have one ULP operating in the forward direction and another ULP, defined alongside it, operating in the backward direction.
>
> In addition, the set of procedures that may be used
> in the forward direction and the set of procedures
> that may be used in the backward direction are
> disjoint.
>
>
> > If this is the case, then bidirectional protocols  do not currently exist but might be defined in the future.
>
> I agree with that.
>
> What the I-D creates is the ability to send RPC
> requests in both directions on a transport. How
> ULPs use this capability is rather up to them.
>
>
> > > > > How about: Before any operation can flow in either
> > > > > direction, it MUST be covered by a DDP-eligibility
> > > > > statement.
> >
> > > > OK, except for the "MUST".
> >
> > > It's the same MUST that applies to other ULPs, stated
> > > in rfc5666bis Section 7.
> >
> > It is the same word but it is not used in the same way:
> >
> > There are the only three uses of "MUST" in section 7:
> >
> > Senders MUST NOT reduce data items that are not DDP-eligible.
> > In the first case, a responder MUST NOT process the Call message.
> > Both types of violations MUST be reported as described in Section 5.5.2.
> > These are all straightforward directions to implementers.
> >
> > I know there is uncertainty/dispute about whether RFC2119 terms are appropriately directed to spec authors.
> >
> > I found your statement above troubling in that it tied an implementation-level event together with a specification level event.
>
> Oh, I see. "MUST be covered by a DDP-eligibility
> statement" is directed to spec authors, not to
> implementers, thus an RFC 2119 term is not
> appropriate. Got it.
>
>
> > On Wed, May 18, 2016 at 5:25 PM, Chuck Lever <[e-mail]<mailto:[e-mail]>> wrote:
> >
> > > On May 18, 2016, at 5:13 PM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:
> > >
> > > > Where is "bi-directional operation declared out of scope?"
> > >
> > > I agree that it doesn't say that.
> > >
> > > > Section 1 appropriately says that an NFSv4.x Upper Layer
> > > > Binding is out of scope. It says nothing about
> > > > bi-directional operation of RPC-over-RDMA
> > >
> > > Note that this comment was made in the context of section 7 which concerns itself with ULBs and section one does say that the ULB for the only protocol that supports bidirectional operation is out of the scope of this document.  I agree that that is a valid statement, BTW.
> > >
> > > > If needed I
> > > > can clarify Section 1.
> > >
> > > I think the clarification I am looking for is one that says where the ULB will be provided, rather than where it is not provided.
> > > This could be an informative reference to rfc5667bis or a simple statement that the issue is addressed in section 7.
> > >
> > > > >> I'm going to propose changes assuming that the necessary material will be added to this section, although a separate section is possible.
> > >
> > > > A separate section for what?
> > >
> > > A separate section for ULBs for bidirectional operation, as opposed to the existing section for backlward operation.
> > >
> > > We both agree that this is undesirable, but I need to answer your question.
> > >
> > > > I guess I don't understand why backward operation by
> > > > itself is any different than backward operation on
> > > > the same transport as forward operation.
> > >
> > > The question is whether backward operation of a single protocol is different from bidirectional operation of a single protocol.  I don't think it is any different but given that this distinction is made, it needs to be made clear that the same ULB rules apply to both.
> > >
> > > >> Suggest changing the section title to "Upper Layer Binding for Backward Direction and Bi-directional Operation"
> > > >> Suggest adding the following new sentence to the end of the first paragraph:
> > > >> This applies to both backward direction and bi-directional operation.
> > >
> > > > I don't understand why you believe this is necessary to
> > > > state here.
> > >
> > > I don't know that it is necessary but I do think it would helpful.
> > >
> > > In many places you make the distinction between backward and bi-directional operation, so it would helpful, if this section is to cover both, to state that it does cover both.
> > >
> > > > OK but: don't NFSv4.1 callback operations also use a
> > > > different RPC program than NFSv4.1 forward channel
> > > > operations?
> > >
> > > Yes they do.
> > >
> > > > Since there is a separate XDR definition,
> > > > and a unique RPC Program number, that makes these two
> > > > separate and distinct RPC Programs
> > >
> > > They are distinct but they  are not separate.  They function as a
> > > single protocol.
> > >
> > > RFC5661 does not say it describes two protocols.  It says it describes a single protocol, NFSv4.1, which operates in both directions.
> > >
> > > >  (and thus are
> > > > different ULPs), IMO.
> > >
> > > I don't think it is actually your opinion that forward and backward direction NFSv4 operations are part of two separate protocols.  If it is, this is a very new opinion.
> >
> > How do we define Upper Layer Protocol? An Upper Layer
> > Binding applies to an Upper Layer Protocol, which is
> > defined as a distinct RPC Program and Version. See
> > Section 7 rfc5666bis:
> >
> > > Each RPC Program and Version tuple that utilizes RPC-over-RDMA
> > > Version One needs to have an Upper Layer Binding specification.
> >
> >
> > RFC 5661/2 define two separate RPC Programs, thus
> > separate ULBs are needed, according to rfc5666bis.
> >
> > This is the way the I-Ds are written now. I am
> > willing to entertain alternatives.
> >
> >
> > > > So what is the essential difference we want to call out
> > > > here? Thinking aloud:
> > >
> > > > If a backward-only ULP is defined by itself, it has to
> > > > have a statement, even a simple one, about DDP-eligible
> > > > data items.
> > >
> > > Right, but my concern about section 7 was that it seemed that was the only case that was addressed there and, as I read it, bidirectional operation was not addressed.
> >
> > I don't see how backward-only and bidirectional operation
> > are addressed separately, except for the separate definitions
> > in Section 2. Can you give other examples of where they are
> > separately treated?
> >
> >
> > > > If a backward-only ULP is defined along side a forward
> > > > only ULP, it can fall back on the default DDP-eligibility
> > > > stated in rfc5666bis (or the equivalent normative
> > > > standard that specifies the RPC-over-RDMA version that
> > > > is being used).
> > >
> > > Yes, but as the bidirection docment is now, you also have to make provision for the non-default case, in which there might be backward DDP-eligible items.
> > >
> > > So I guess you are saying, and it might even be your opinion that forward and backward direction are two protocols, described "alongside" one another.  If that's your opinion and you want to say it that way OK, but it might be confusing to people.  If you are going to adopt that usage, you should make it clear by citing the case of NFSv4.1, that that is what you mean.
> > >
> > > > If a ULP defines operations that can flow in either
> > > > direction, then what? Note: we don't have one of these
> > > > yet, AFAIK.
> > >
> > > In that case, the ULB is responsible for defining DDP-eligibility for transmission in both directions.
> > >
> > > there isn't a case yet, but neither is there a case of backward direction chunks.   This sort of spec needs to address these kinds of future possibilities the best that it can.
> > >
> > > Fortunately, in this case, that isn't all that hard.
> > >
> > > > I don't see why these distinctions are necessary.
> > >
> > > They aren't necessary, but they have been made and given that they have been made, you have to deal with the consequences of having made them.
> > >
> > > In this case, if you want to treat bidirectional and backward direction operation the same, from the ULB viewpoint, you have to state that that is what you are doing.
> > >
> > > > How about: Before any operation can flow in either
> > > > direction, it MUST be covered by a DDP-eligibility
> > > > statement.
> > >
> > > OK, except for the "MUST".
> >
> > It's the same MUST that applies to other ULPs, stated
> > in rfc5666bis Section 7.
> >
> >
> > > > If there is an associated forward-only ULP that already
> > > > has a binding, the backward operations are covered by
> > > > that binding (meaning: if the backward operations are
> > > > not mentioned, they have the default DDP eligibility).
> > >
> > > I guess that works but I think that you are adding confusion
> > > by trying to treat NFSv4.1 as two protocols.  If you want to do
> > > that, you need to make clear, in the introduction if necessary,
> > > that that is what you are doing
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, May 18, 2016 at 3:44 PM, Chuck Lever <[e-mail]<mailto:[e-mail]>> wrote:
> > > Hi Dave-
> > >
> > >
> > > > On May 5, 2016, at 10:07 AM, Chuck Lever <[e-mail]<mailto:[e-mail]>> wrote:
> > > >
> > > >
> > > >> On May 4, 2016, at 1:10 PM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:
> > > >>
> > > >> 7.  Backward Direction Upper Layer Binding
> > > >> The basic problem in this section is that it concerns itself with ULBs for the backward direction case while the more important case of bidirectional operation is not explicitly addressed (except in 1.  Introduction) where it is declared out-of-scope).
> > >
> > > Maybe mark it up to my progressing years, but as I'm
> > > looking back at these comments, I find that I don't
> > > understand them at all.
> > >
> > > Where is "bi-directional operation declared out of scope?"
> > > Section 1 appropriately says that an NFSv4.x Upper Layer
> > > Binding is out of scope. It says nothing about
> > > bi-directional operation of RPC-over-RDMA. If needed I
> > > can clarify Section 1.
> > >
> > >
> > > >> I'm going to propose changes assuming that the necessary material will be added to this section, although a separate section is possible.
> > >
> > > A separate section for what?
> > >
> > > I guess I don't understand why backward operation by
> > > itself is any different than backward operation on
> > > the same transport as forward operation.
> > >
> > >
> > > >> Suggest changing the section title to "Upper Layer Binding for Backward Direction and Bi-directional Operation"
> > > >> Suggest adding the following new sentence to the end of the first paragraph:
> > > >> This applies to both backward direction and bi-directional operation.
> > >
> > > I don't understand why you believe this is necessary to
> > > state here.
> > >
> > >
> > > >> Given Chuck's choice regarding how to deal with this issue, I suggest rewriting the third paragraph as follows:
> > > >> By default, no data items in a ULP are DDP-eligible.  If there are no DDP-eligible data items to document, the specification of DDP-eligible items need not be part of the Upper Layer Binding for an Upper Layer Protocol that operates only in the backward direction.
> > >
> > > That's my recollection as well.
> > >
> > >
> > > >> I suggest adding a new paragraph after the existing third paragraph, reading as follows:
> > > >> An Upper Layer Protocol that operates on RPC-over-RDMA transports and incorporates operation in both  directions may have DDP-eligible data items for transfers in either or both directions.  All of these are specified in an Upper Layer Binding document, as they would be for a protocol which only operates in a single direction. This applies even when, as in the case of NFSv4. the XDR is constructed so that each direction of operation is defined as part of a separate XDR program.
> > >
> > > OK, but: don't NFSv4.1 callback operations also use a
> > > different RPC program than NFSv4.1 forward channel
> > > operations? Since there is a separate XDR definition,
> > > and a unique RPC Program number, that makes these two
> > > separate and distinct RPC Programs (and thus are
> > > different ULPs), IMO.
> > >
> > > So what is the essential difference we want to call out
> > > here? Thinking aloud:
> > >
> > > If a backward-only ULP is defined by itself, it has to
> > > have a statement, even a simple one, about DDP-eligible
> > > data items.
> > >
> > > If a backward-only ULP is defined along side a forward
> > > only ULP, it can fall back on the default DDP-eligibility
> > > stated in rfc5666bis (or the equivalent normative
> > > standard that specifies the RPC-over-RDMA version that
> > > is being used).
> > >
> > > If a ULP defines operations that can flow in either
> > > direction, then what? Note: we don't have one of these
> > > yet, AFAIK.
> > >
> > > I don't see why these distinctions are necessary.
> > >
> > > How about: Before any operation can flow in either
> > > direction, it MUST be covered by a DDP-eligibility
> > > statement.
> > >
> > > If there is an associated forward-only ULP that already
> > > has a binding, the backward operations are covered by
> > > that binding (meaning: if the backward operations are
> > > not mentioned, they have the default DDP eligibility).
> > >
> > >
> > > --
> > > Chuck Lever
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [e-mail]<mailto:[e-mail]>
> > > https://www.ietf.org/mailman/listinfo/nfsv4<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.ietf.org%2fmailman%2flistinfo%2fnfsv4&data=01%7c01%7csshepler%40microsoft.com%7cd0d8d162cc3746acfd4908d380cf139d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=%2fonEjqwWqYZEnE9eaRUZ1bCYi8fwuj7WURzT9fDDSxQ%3d>
> >
> > --
> > Chuck Lever
> >
> >
> >
> >
>
> --
> Chuck Lever
>
>
>
>
--
Chuck Lever