Re: [nfsv4] review of draft-ietf-nfsv4-rpcrdma-bidirection-03

[David Noveck <[e-mail]>]

Works for me.

I think we can now (or at midnight) proceed to move these documents into WG
Consensus state.

On Fri, May 20, 2016 at 11:34 AM, Chuck Lever <[e-mail]>
wrote:

>
> > On May 19, 2016, at 11:52 AM, David Noveck <[e-mail]> wrote:
> >
> > > It seems natural to me because, logically, you
> > > can't have bi-directional operation without
> > > backward direction operation.
> >
> > That's likely to be the case, but note that, you can't have
> bi-directional operation
> > without forward direction operation either, and it somehow doesn't seem
> natural
> > to identify forward and bi-directional operation.  I think logic does
> not have as
> > much to do with this as you assume.
> >
> > > > If this is the case, then bidirectional protocols  do not currently
> exist but might be defined in the future.
> >
> > > I agree with that.
> >
> > Yes, but it seems we disagree about whether they currently exist because
> we use the word "protocol" in
> > different ways.  This is not a big problem right now but I'm worried
> about a situation in which two of the
> > documents that NFSv4 implementers refer to use such a basic term in such
> different ways.
>
>
> The language in Section 7 of -03 was indeed problematic.
> It contradicted, for example, our long-standing claim
> that RFC 5667 is missing DDP-eligibility statements for
> callback operations. (I think part of the reason for
> the problematic language was the desire to grandfather
> NFSv4 callback operations into existing RFC 5667 language,
> and perhaps that is not consistent with what has been
> written and discussed so far).
>
> I've rewritten Section 7:
>
>
> 7.  Considerations For Upper Layer Bindings
>
>    An Upper Layer Protocol that operates on RPC-over-RDMA transports may
>    have procedures that include DDP-eligible data items.  DDP-
>    eligibility is specified in an Upper Layer Binding.  Direction of
>    operation does not obviate the need for DDP-eligibility statements.
>
>    Backward-only operation requires the client endpoint to establish a
>    fresh connection.  The Upper Layer Binding can specify appropriate
>    RPC binding parameters for such connections.
>
>    Bi-directional operation occurs on an already-established connection.
>    Specification of RPC binding parameters is usually not necessary in
>    this case.
>
>    For bi-directional operation, other considerations about sharing an
>    RPC-over-RDMA transport with another ULP may apply.  Consult
>    Section 7 of [I-D.ietf-nfsv4-rfc5666bis] for details about what else
>    may be contained in an Upper Layer Binding.
>
>
> Hopefully that addresses your concerns about conflating
> backward direction and bidirection, and issues with the
> use of the term 'Upper Layer Protocol.'
>
>
> > On Thu, May 19, 2016 at 10:13 AM, Chuck Lever <[e-mail]>
> wrote:
> > Hi Dave-
> >
> >
> > > On May 18, 2016, at 10:21 PM, David Noveck <[e-mail]>
> wrote:
> > >
> > > > How do we define Upper Layer Protocol? An Upper Layer
> > > > Binding applies to an Upper Layer Protocol, which is
> > > > defined as a distinct RPC Program and Version. See
> > > > Section 7 rfc5666bis:
> > >
> > > I think it would be helpful if the defnition of "Upper Layer Protocol"
> > > matched the way the word "protocol" is normally used.  And NFSv4.1
> > > is, quite reasonably, described as a protocol, and not as a set of
> > > two protocols described alongside one another.
> >
> > But we do make a distinction between NLM, NFSACL,
> > MOUNT, NSM, and NFS protocols, referring to NFS
> > as the "main protocol" and the others as "side-band
> > protocols."
> >
> > In NFSv4.0, forward operation and callback are
> > always used on distinct connections, and they have
> > distinct RPC Program numbers and separate XDR
> > definitions. Forward operation can occur without
> > a callback channel. Are those considered one
> > protocol, or two?
> >
> > Is NFSv3 the same ULP as NFSv4.0 or NFSv4.1? If so,
> > why are there separate discussions in RFC 5667 for
> > v3 and v4.0, and why are we not attempting to claim
> > that RFC 5667 or rfc5667bis are normative for all
> > future versions of NFS?
> >
> > Thus I don't consider the colloquial usage of the
> > term 'protocol' to be adequate in this context.
> >
> >
> > > > RFC 5661/2 define two separate RPC Programs, thus
> > > > separate ULBs are needed, according to rfc5666bis.
> > >
> > > > This is the way the I-Ds are written now. I am
> > > > willing to entertain alternatives.
> > >
> > > OK.  First of all it might be stated that when two separate RPC
> > > programs are defined together and used together as a  single protocol
> capable of bidirectional operation, they can be
> > > considered as a single protocol with a ULB,covering both
> > > directions of operation.
> >
> > Except that last time I included a mention of call
> > direction in rfc5666bis, you objected to that. :-)
> >
> > I still prefer the way rfc5666bis is written now.
> > It is unambiguous about what needs to have a
> > Binding. The boundaries defined by Program and
> > Version number are historical, simple to
> > understand, and well-defined.
> >
> > (And, this is a valuable conversation, since we
> > as a WG should be striving to get this right).
> >
> >
> > > > I don't see how backward-only and bidirectional operation
> > > > are addressed separately, except for the separate definitions
> > > > in Section 2.
> > >
> > > The introduction to the subject defines forward operation,
> > > backward operation, and bi-directional operation, all on the
> > > same level, as the three classes of RPC directionality.
> > >
> > > > Can you give other examples of where they are
> > > > separately treated?
> > >
> > > No I can't and part of the reason is that,  in most contexts, you are
> talking about a request or response that part of a forward or backward
> RPC.  there are no bidirectional RPC's even though there are (or may be)
> bidirectional protocols.
> > >
> > > It is quite natural in the context to treat backward an bidirectional
> > > together.   The issue is that if you introduce these three classes and
> then choose to treat two of them together, you need to tell the reader
> > > that you are gong to do so.  You may think it is natural and obvious
> to treat these together but you can't assume the reader will
> > > naturally share that view, given he has been told that these are
> separate things.
> >
> > It seems natural to me because, logically, you
> > can't have bi-directional operation without
> > backward direction operation. I can review the
> > text with your comment in mind, however.
> >
> > Besides the sentence you proposed in Section 7,
> > are there other places in the document where I
> > should add "and bi-directional" or its equivalent?
> > I'm open to suggestion.
> >
> >
> > > Of course,, we now have to deal with the irony that arises from the
> fact that you originally began this work to enable the NFSv4.1 protocol to
> operate bidirectionally, and having done that, you have (magically :-) made
> the NFSv4.1 protocol disappear  and with it bidirectional operation.  If I
> understand your position correctly, we now no longer have a single protocol
> operating bidirectionally.
> >
> > When it is broken down, the RPC transport, not the
> > protocol, operates bi-directionally. For example:
> >
> > A "uni-directional" protocol such as NFSv3 or NLM
> > can be used bi-directionally, if two connection
> > endpoints act as both clients and servers.
> >
> > NFSv4.1 implementations can set up distinct and
> > separate connections for forward and backward
> > operation. In that case, no bi-directional
> > operation is occurring on either connection (after
> > the BC2S operation on the callback connection).
> >
> >
> > > Instead, we have one ULP operating in the forward direction and
> another ULP, defined alongside it, operating in the backward direction.
> >
> > In addition, the set of procedures that may be used
> > in the forward direction and the set of procedures
> > that may be used in the backward direction are
> > disjoint.
> >
> >
> > > If this is the case, then bidirectional protocols  do not currently
> exist but might be defined in the future.
> >
> > I agree with that.
> >
> > What the I-D creates is the ability to send RPC
> > requests in both directions on a transport. How
> > ULPs use this capability is rather up to them.
> >
> >
> > > > > > How about: Before any operation can flow in either
> > > > > > direction, it MUST be covered by a DDP-eligibility
> > > > > > statement.
> > >
> > > > > OK, except for the "MUST".
> > >
> > > > It's the same MUST that applies to other ULPs, stated
> > > > in rfc5666bis Section 7.
> > >
> > > It is the same word but it is not used in the same way:
> > >
> > > There are the only three uses of "MUST" in section 7:
> > >
> > > Senders MUST NOT reduce data items that are not DDP-eligible.
> > > In the first case, a responder MUST NOT process the Call message.
> > > Both types of violations MUST be reported as described in Section
> 5.5.2.
> > > These are all straightforward directions to implementers.
> > >
> > > I know there is uncertainty/dispute about whether RFC2119 terms are
> appropriately directed to spec authors.
> > >
> > > I found your statement above troubling in that it tied an
> implementation-level event together with a specification level event.
> >
> > Oh, I see. "MUST be covered by a DDP-eligibility
> > statement" is directed to spec authors, not to
> > implementers, thus an RFC 2119 term is not
> > appropriate. Got it.
> >
> >
> > > On Wed, May 18, 2016 at 5:25 PM, Chuck Lever <[e-mail]>
> wrote:
> > >
> > > > On May 18, 2016, at 5:13 PM, David Noveck <[e-mail]>
> wrote:
> > > >
> > > > > Where is "bi-directional operation declared out of scope?"
> > > >
> > > > I agree that it doesn't say that.
> > > >
> > > > > Section 1 appropriately says that an NFSv4.x Upper Layer
> > > > > Binding is out of scope. It says nothing about
> > > > > bi-directional operation of RPC-over-RDMA
> > > >
> > > > Note that this comment was made in the context of section 7 which
> concerns itself with ULBs and section one does say that the ULB for the
> only protocol that supports bidirectional operation is out of the scope of
> this document.  I agree that that is a valid statement, BTW.
> > > >
> > > > > If needed I
> > > > > can clarify Section 1.
> > > >
> > > > I think the clarification I am looking for is one that says where
> the ULB will be provided, rather than where it is not provided.
> > > > This could be an informative reference to rfc5667bis or a simple
> statement that the issue is addressed in section 7.
> > > >
> > > > > >> I'm going to propose changes assuming that the necessary
> material will be added to this section, although a separate section is
> possible.
> > > >
> > > > > A separate section for what?
> > > >
> > > > A separate section for ULBs for bidirectional operation, as opposed
> to the existing section for backlward operation.
> > > >
> > > > We both agree that this is undesirable, but I need to answer your
> question.
> > > >
> > > > > I guess I don't understand why backward operation by
> > > > > itself is any different than backward operation on
> > > > > the same transport as forward operation.
> > > >
> > > > The question is whether backward operation of a single protocol is
> different from bidirectional operation of a single protocol.  I don't think
> it is any different but given that this distinction is made, it needs to be
> made clear that the same ULB rules apply to both.
> > > >
> > > > >> Suggest changing the section title to "Upper Layer Binding for
> Backward Direction and Bi-directional Operation"
> > > > >> Suggest adding the following new sentence to the end of the first
> paragraph:
> > > > >> This applies to both backward direction and bi-directional
> operation.
> > > >
> > > > > I don't understand why you believe this is necessary to
> > > > > state here.
> > > >
> > > > I don't know that it is necessary but I do think it would helpful.
> > > >
> > > > In many places you make the distinction between backward and
> bi-directional operation, so it would helpful, if this section is to cover
> both, to state that it does cover both.
> > > >
> > > > > OK but: don't NFSv4.1 callback operations also use a
> > > > > different RPC program than NFSv4.1 forward channel
> > > > > operations?
> > > >
> > > > Yes they do.
> > > >
> > > > > Since there is a separate XDR definition,
> > > > > and a unique RPC Program number, that makes these two
> > > > > separate and distinct RPC Programs
> > > >
> > > > They are distinct but they  are not separate.  They function as a
> > > > single protocol.
> > > >
> > > > RFC5661 does not say it describes two protocols.  It says it
> describes a single protocol, NFSv4.1, which operates in both directions.
> > > >
> > > > >  (and thus are
> > > > > different ULPs), IMO.
> > > >
> > > > I don't think it is actually your opinion that forward and backward
> direction NFSv4 operations are part of two separate protocols.  If it is,
> this is a very new opinion.
> > >
> > > How do we define Upper Layer Protocol? An Upper Layer
> > > Binding applies to an Upper Layer Protocol, which is
> > > defined as a distinct RPC Program and Version. See
> > > Section 7 rfc5666bis:
> > >
> > > > Each RPC Program and Version tuple that utilizes RPC-over-RDMA
> > > > Version One needs to have an Upper Layer Binding specification.
> > >
> > >
> > > RFC 5661/2 define two separate RPC Programs, thus
> > > separate ULBs are needed, according to rfc5666bis.
> > >
> > > This is the way the I-Ds are written now. I am
> > > willing to entertain alternatives.
> > >
> > >
> > > > > So what is the essential difference we want to call out
> > > > > here? Thinking aloud:
> > > >
> > > > > If a backward-only ULP is defined by itself, it has to
> > > > > have a statement, even a simple one, about DDP-eligible
> > > > > data items.
> > > >
> > > > Right, but my concern about section 7 was that it seemed that was
> the only case that was addressed there and, as I read it, bidirectional
> operation was not addressed.
> > >
> > > I don't see how backward-only and bidirectional operation
> > > are addressed separately, except for the separate definitions
> > > in Section 2. Can you give other examples of where they are
> > > separately treated?
> > >
> > >
> > > > > If a backward-only ULP is defined along side a forward
> > > > > only ULP, it can fall back on the default DDP-eligibility
> > > > > stated in rfc5666bis (or the equivalent normative
> > > > > standard that specifies the RPC-over-RDMA version that
> > > > > is being used).
> > > >
> > > > Yes, but as the bidirection docment is now, you also have to make
> provision for the non-default case, in which there might be backward
> DDP-eligible items.
> > > >
> > > > So I guess you are saying, and it might even be your opinion that
> forward and backward direction are two protocols, described "alongside" one
> another.  If that's your opinion and you want to say it that way OK, but it
> might be confusing to people.  If you are going to adopt that usage, you
> should make it clear by citing the case of NFSv4.1, that that is what you
> mean.
> > > >
> > > > > If a ULP defines operations that can flow in either
> > > > > direction, then what? Note: we don't have one of these
> > > > > yet, AFAIK.
> > > >
> > > > In that case, the ULB is responsible for defining DDP-eligibility
> for transmission in both directions.
> > > >
> > > > there isn't a case yet, but neither is there a case of backward
> direction chunks.   This sort of spec needs to address these kinds of
> future possibilities the best that it can.
> > > >
> > > > Fortunately, in this case, that isn't all that hard.
> > > >
> > > > > I don't see why these distinctions are necessary.
> > > >
> > > > They aren't necessary, but they have been made and given that they
> have been made, you have to deal with the consequences of having made them.
> > > >
> > > > In this case, if you want to treat bidirectional and backward
> direction operation the same, from the ULB viewpoint, you have to state
> that that is what you are doing.
> > > >
> > > > > How about: Before any operation can flow in either
> > > > > direction, it MUST be covered by a DDP-eligibility
> > > > > statement.
> > > >
> > > > OK, except for the "MUST".
> > >
> > > It's the same MUST that applies to other ULPs, stated
> > > in rfc5666bis Section 7.
> > >
> > >
> > > > > If there is an associated forward-only ULP that already
> > > > > has a binding, the backward operations are covered by
> > > > > that binding (meaning: if the backward operations are
> > > > > not mentioned, they have the default DDP eligibility).
> > > >
> > > > I guess that works but I think that you are adding confusion
> > > > by trying to treat NFSv4.1 as two protocols.  If you want to do
> > > > that, you need to make clear, in the introduction if necessary,
> > > > that that is what you are doing
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 18, 2016 at 3:44 PM, Chuck Lever <[e-mail]>
> wrote:
> > > > Hi Dave-
> > > >
> > > >
> > > > > On May 5, 2016, at 10:07 AM, Chuck Lever <[e-mail]>
> wrote:
> > > > >
> > > > >
> > > > >> On May 4, 2016, at 1:10 PM, David Noveck <[e-mail]>
> wrote:
> > > > >>
> > > > >> 7.  Backward Direction Upper Layer Binding
> > > > >> The basic problem in this section is that it concerns itself with
> ULBs for the backward direction case while the more important case of
> bidirectional operation is not explicitly addressed (except in 1.
> Introduction) where it is declared out-of-scope).
> > > >
> > > > Maybe mark it up to my progressing years, but as I'm
> > > > looking back at these comments, I find that I don't
> > > > understand them at all.
> > > >
> > > > Where is "bi-directional operation declared out of scope?"
> > > > Section 1 appropriately says that an NFSv4.x Upper Layer
> > > > Binding is out of scope. It says nothing about
> > > > bi-directional operation of RPC-over-RDMA. If needed I
> > > > can clarify Section 1.
> > > >
> > > >
> > > > >> I'm going to propose changes assuming that the necessary material
> will be added to this section, although a separate section is possible.
> > > >
> > > > A separate section for what?
> > > >
> > > > I guess I don't understand why backward operation by
> > > > itself is any different than backward operation on
> > > > the same transport as forward operation.
> > > >
> > > >
> > > > >> Suggest changing the section title to "Upper Layer Binding for
> Backward Direction and Bi-directional Operation"
> > > > >> Suggest adding the following new sentence to the end of the first
> paragraph:
> > > > >> This applies to both backward direction and bi-directional
> operation.
> > > >
> > > > I don't understand why you believe this is necessary to
> > > > state here.
> > > >
> > > >
> > > > >> Given Chuck's choice regarding how to deal with this issue, I
> suggest rewriting the third paragraph as follows:
> > > > >> By default, no data items in a ULP are DDP-eligible.  If there
> are no DDP-eligible data items to document, the specification of
> DDP-eligible items need not be part of the Upper Layer Binding for an Upper
> Layer Protocol that operates only in the backward direction.
> > > >
> > > > That's my recollection as well.
> > > >
> > > >
> > > > >> I suggest adding a new paragraph after the existing third
> paragraph, reading as follows:
> > > > >> An Upper Layer Protocol that operates on RPC-over-RDMA transports
> and incorporates operation in both  directions may have DDP-eligible data
> items for transfers in either or both directions.  All of these are
> specified in an Upper Layer Binding document, as they would be for a
> protocol which only operates in a single direction. This applies even when,
> as in the case of NFSv4. the XDR is constructed so that each direction of
> operation is defined as part of a separate XDR program.
> > > >
> > > > OK, but: don't NFSv4.1 callback operations also use a
> > > > different RPC program than NFSv4.1 forward channel
> > > > operations? Since there is a separate XDR definition,
> > > > and a unique RPC Program number, that makes these two
> > > > separate and distinct RPC Programs (and thus are
> > > > different ULPs), IMO.
> > > >
> > > > So what is the essential difference we want to call out
> > > > here? Thinking aloud:
> > > >
> > > > If a backward-only ULP is defined by itself, it has to
> > > > have a statement, even a simple one, about DDP-eligible
> > > > data items.
> > > >
> > > > If a backward-only ULP is defined along side a forward
> > > > only ULP, it can fall back on the default DDP-eligibility
> > > > stated in rfc5666bis (or the equivalent normative
> > > > standard that specifies the RPC-over-RDMA version that
> > > > is being used).
> > > >
> > > > If a ULP defines operations that can flow in either
> > > > direction, then what? Note: we don't have one of these
> > > > yet, AFAIK.
> > > >
> > > > I don't see why these distinctions are necessary.
> > > >
> > > > How about: Before any operation can flow in either
> > > > direction, it MUST be covered by a DDP-eligibility
> > > > statement.
> > > >
> > > > If there is an associated forward-only ULP that already
> > > > has a binding, the backward operations are covered by
> > > > that binding (meaning: if the backward operations are
> > > > not mentioned, they have the default DDP eligibility).
> > > >
> > > >
> > > > --
> > > > Chuck Lever
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > nfsv4 mailing list
> > > > [e-mail]
> > > > https://www.ietf.org/mailman/listinfo/nfsv4
> > >
> > > --
> > > Chuck Lever
> > >
> > >
> > >
> > >
> >
> > --
> > Chuck Lever
> >
> >
> >
> >
>
> --
> Chuck Lever
>
>
>
>