Re: [nfsv4] New Version Notification for draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt

[David Noveck <[e-mail]>]

> So, there is a protocol mechanism for reporting partial or full
> completion of a property change request, yet no mechanism for
> indicating which change request it is the receiver is completing.
> But multiple property change requests are allowed to be in flight
> concurrently.

The protocol allows it, but particular implementations, may well
restrict things so that they have only one in flight for each property
at any particular time.

> Without an XID, the only way this might work is by serializing
> updates to each property.

It is one way.  I don't think it is the only way.  The requester, if it
is interested in knowing when there are no outstanding requests
for a property can increment on requests and decrement on messages
that indicate completion.

> And, if the sender can send an "I'm done" update followed by any
> number of "Oops, I changed this again" updates,

The updates indicate that there has been an unsolicited change in
the property:  no "Oops" and no "again".

Whether such changes happen or not, depends on the
implementation.  In any case, I would expect them to be
rare.

The proposed protocol provides a way for the peer to be
informed of such changes.   I don't see how that undercuts
the facilities that allow the completion of requested changes
to be reported asynchronously.

> I don't see the
> point of a receiver ever remembering that it is waiting for a
> pending change, or the protocol distinguishing between unsolicited
> and pendclr.

You seem to be saying the potential existence of unsolicited changes
some time after the completion of a requested change somehow
makes the notion of completion of the requested change useless.

I'm not sure why you believe that.  The protocol allows unsolicited changes
but that doesn't mean they are going to be happening all that frequently.

> That makes pendclr largely a hint, and a temporary one at that.

I don't see things that way.


>In other words "rejection" doesn't matter.

Yes, but the indication that the request is done can matter, even if it
might not
matter to you.  See below.

> There's no recourse for the receiver;

I disagree.

> it just needs to know the current value of the property.

It certainly needs to know that.  Your implementaition might
not be interested in anything else but the protocol provides it
because it is easy for the sender to provide and useful in some
cases.

> If two or more ULPs are using the same connection, they might
> send a sequence of possibly contradictory property change
> requests.

I don't understand why one might assume that ULP's
are sending property change requests.  ULPs are RPC
protocols.  They send rpc requests and receive replies.

> Also, there's no co-ordination between the two senders on the
> same connection; or an administrator might adjust these
> properties.

Any co-ordination requirement that a transport implementation
might impose to deal with the case of multiple ULPs would be
out of scope in this document.

I don't see how you decide there is "no co-ordination".  If you build
an implementation and give multiple ULPs the ability to request and
make such changes independently, you are asking for (or begging for)
trouble.  As an implementer, the choice regarding co-ordination (or not)
is up to you.

I think the entities making such changes are most likely to reflect
administrative requests or internal transport optimization functions.

I can't see why one would allow the ULP to do this itself.


> I posit that the receiver cares only about the current effective
> value of these properties.

That's one of the things he cares about, but I believe it is not
the only one.

pendclr is completely unreliable.

If you were to always set it to false, as you suggest below that you
might, you can make it unreliable, but there is no good reason to do that.

> Can you provide a real world example of how pendclr might be used?
> Because it appears to me to convey no actionable information, and
> I don't see why I shouldn't always set it to false.


Let's consider that a client implementation encounters a server
which has a very large receive buffer size and feels that it would prefer
more
smaller buffers.

It then requests a smaller receiver buffer size.  as part of doing that it
decides
to not send requests or responses that are larger than this anticipated size
limit.  I think the spec already discusses this but the basic point is that
by making
this request the sender is licensing the peer to reduce the buffer size.
If this
request is completed, the sender will make that new size permanent (or at
least until
it hears about a change). On the other hand, if the peer rejects the
request
(either fairly soon via RESPXCH, or  after a while via UPDXCH), the sender
would
then adopt the current value, since the possibility of the
peer eventually
satisfying the orginal request is now gone. On the other hand, an update
not including pendclr
should not have that effect.  As long as the request is pending, he has to
be prepared
for its peer to adopt the lower value that it has requested.



> Think of this as an NFS attribute

The point is that it isn't an NFS attribute, even though there are some
similarities.

Let's look at the differences and the reasons for them.

   - SETATTR does not have UPDXCH since there is no way to have a
   potentially long-running change requiring asychrnous notification.  SETATTR
   generally cannot take significant time.
   - Although there are attributes subject to unrequested change, you are
   not informed about them.  for example space-available



> SETATTR doesn't return "Oh, Ichanged this just a little"

> "Someone adjusted this value, but it may change again"

It doesn't tell you that but you have no guarantee that someone else will
not change
after you.  In the model we are talking about, it also may change but you
receive a
notification.  That sort of notification would make not sense for an NFSv4
attribute
but it does make sense about a transport property because it is a
reasonable assumption that
the sender wants to know about this ASAP since it might affect his next
transfer

> Does the document's language forbid a receiver from simply discarding
> a property change request? (I'm thinking of similar language as RFC
> 7350 which requires that NFS servers never discard requests).

I can add something like that.

> It's very easy to imagine an implementer deciding that UPDXCH is
> better for his or her client or server than FIRSTPROP, and noting
> stridently that the document does not prohibit or even warn against
> this design.

> IMO the document needs to make an explicit requirement not to
> behave this way, or state that it is acceptable though not preferred.

Are you sure that it is "requirement' and not a "REQUIREMENT" that you
are looking for?

I'm OK with saying you shouldn't do that, but I have problem saying it is
"acceptable".  I think it is noxious, although there is no "NOXIOUS".

Maybe the right thing is to state the obvious, that the function of UPDXCH
is to
convey changes, making its use to present values at connection time dubious,
while the function of ROPT_FIRSTPROP is to present the value at initial
connection.

> The closest we have is SHOULD NOT (which I always read as "don't do
> that, you muppet").

I know that is a common usage/interpretation but it is not consistent with
RFC2119.
Lately the IESG has started to notice that.

> For a client to reduce its receive buffers in mid-flight, all it
> needs to do is send Reply chunks more often. That will work OK
> if the server uses the Reply chunk whenever one is provided.
> (Not all servers behave nicely in this regard, though).

> For a server to reduce its receive buffers in mid-flight, the
> server needs to notify the client of the change in receive
> buffer size, the client would need to acknowledge the change,
> and only then can the server reduce its receive buffer size.
> How might that be done with the protocol proposed in
> rpcrdma-xcharext ?

I don't think unsolicited reduction in receive buffer size (as opposed to
the
requested reduction discussed above) can be made to work.
Maybe I should add an explicit statement
to that effect.

> OK imagine some less harmful changes, like:

> The administrator on the client has turned a knob that enables
> Remote Invalidation. Yet it is perfectly safe for the server to
> continue not using RI. Is UPDXCH required in that case?

No.

> If you'd rather the document does not commit, then maybe a
> sentence or two could be added that suggests the implementer
> needs to consider carefully the safety of not sending an
> unsolicited property change.

OK.

On Tue, Aug 23, 2016 at 12:57 PM, Chuck Lever <[e-mail]>
wrote:

>
> > On Aug 23, 2016, at 10:45 AM, David Noveck <[e-mail]> wrote:
> >
> > Thanks. I'll address these as part of draft-dnoveck-nfsv4-rpcrdma-xcharext-03,
> which will be out in the first half of September.
> >
> > Unless there are objections, that draft will adopt Karen's suggestion to
> change "characteristics" to "properties".  The title of
> > the document will change but the file name will stay the the same.
> >
> > > There's no discussion of RPC-over-RDMA credit accounting in this
> > > document. There needs to be some discussion of credit consumption.
> >
> > Right.
> >
> > > In particular:
> >
> > > Requesters will have to have posted extra receive buffers (over
> > > and above credits for forward channel replies and backchannel
> > > requests) to deal with XCHAR messages.
> >
> > I think you mean that clients will.
> >
> > > Likewise, responders will
> > > need to post similar extra receives for this purpose.
> >
> > Similarly, I think servers are being referred to.
> >
> > > Perhaps both peers should reserve one credit, and the specification
> > > should insist that these operations are always single-threaded on a
> > > connection.
> >
> > I think single-threading might be an implementation choice but I'm
> > reluctant to make it part of the protocol.
> >
> > > Alternately, when xcharext is merged into rpcrdma-version-two,
> > > there might be a generic discussion of non-RPC-payload-bearing
> > > messages that could cover this issue.
> >
> > I think there will need to be generic text regarding the issues, but I
> think it should
> > be consistent with the following approach regarding messages related to
> > transport.
> >
> > In the case of messages that do not serve effectively as the response
> > to a previous message (i.e. ROPT_FIRSTPROP, ROPT_REQPROP,
> > ROPT_UPDPROP), it is the responsibility of the sender to ensure that
> > there is a credit available to enable sending the message, just as would
> > be the case if it were sending an RPC request.
> >
> > In the case of ROPT_RESPPROP meesages, it is the responsibility of
> > the sender of the original ROPT_REQPROP to post a receive to
> > receive the response, which is to be sent by the receiver of the
> ROPT_REQPROP,
> > without consuming a credit.   This is similar to the case of ending an
> RPC
> > response.
> >
> >
> > > Overloading the term "initial"
> >
> > Your'e right.  It is overloaded.
> >
> > > Section 3 introduces an "initial set of transport characteristics"
> > > and Section 4.1 defines "ROPT_INITXCH: Specify Initial
> > > Characteristics". I think the use of "initial" means different
> > > things in these two cases?
> >
> >
> > It does.
> >
> > > Instead:
> >
> > > Section 3 could propose "initially-defined" characteristics.
> >
> >
> > OK.
> >
> > > Section 4.1 could define ROPT_STARTXCH (like STARTTLS). I'm not
> > > attached to that name, but it probably shouldn't be ROPT_INITXCH
> > > if Section 3 defines "initial characteristics".
> >
> > > ROPT_CONNXCH
> >
> > > ROPT_FIRSTXCH
> >
> > > ROPT_EARLYXCH
> >
> > My current choice is ROPT_FIRSTPROP.
> >
> > > Section 4.4
> >
> > > The text in this section has been clarified to address previous
> > > reviewer comments; thanks! There are a number of syntax and
> > > grammatical errors that still need to be addressed (most often,
> > > a few words are repeated in some sentences).
> >
> > I'll address those in -03.
> >
> > > The "argument structure" of ROPT_UPDXCH is:
> >
> > > struct optinfo_updxch {
> > >    xcharval        optupdxch_now;
> > >    bool            optupdxch_pendclr;
> > > };
> >
> > > I prefer optupdxch_new instead of optupdxch_now; "_now" suggests this
> > > field records a time and/or date stamp.
> >
> > OK.
> >
> > > It would help me to understand why a receiver needs to distinguish
> > > between these types of notification.
> >
> > When I specified the possible situations in which these messages could
> arise, I did not mean to imply that the receiver necessarily would need to
> distinguish these notifications.  My focus was on the sender.
>
> > > For instance, if pendclr is false, this could be either a rejection
> > >of a pending change request,
> >
> > If it it is a *rejection* of a pending change request pendclr would true
> >
> > > or it could be an unsolicited change
> > > notification.
> >
> > > How does the receiver make use of the difference?
> >
> > > Instead of a boolean, an enumeration of update event types would be
> > > a little friendlier, and could be expressed in the same amount of
> > > space (since an XDR boolean consumes 4 octets on the wire).
> >
> > I'm OK in principle, but it seems we are both uncertain if this is
> > indeed "friendlier"
> >
> > > Based
> > > on the discussion in Section 4.4, we have:
> >
> > > enum optupdxch_event_type {
> > >    OPTUPD_UNSOL  = 1,
> >
> > pendclr = false as there is nothing pending to clear.
> >
> > >   OPTUPD_MORE   = 2
> >
> > pemdclr=false but there is still a pending request.
> > ,
> > >   OPTUPD_DONE   = 3,
> >
> > pendclr = true and the request has completed successfully.
> >
> >    OPTUPD_REJECT = 4,
> >
> > This sound to me pendclr= true but the word "REJECT" suggests no change
> at all was made.  In this case you might also have a partial change
> > };
> >
> > > But since the rdma_xid field is not used to tie change requests to
> > > these change update notifications, I'm not sure why the receiver
> > > needs to know that a pending request has been completed.
> >
> > If the receiver keeps track of pending requests, it needs to know when
> one is no longer pending.
> >
> > The receiver is not required to do this and some might not choose to do
> so, but the protocol should provide
> > an implmentation that does the means to keep track.
> >
> > This does not require the xid, but peer that requested the change can
> keep track of the properites for
> > which it has a pending requested change.
>
> So, there is a protocol mechanism for reporting partial or full
> completion of a property change request, yet no mechanism for
> indicating which change request it is the receiver is completing.
> But multiple property change requests are allowed to be in flight
> concurrently.
>
> Without an XID, the only way this might work is by serializing
> updates to each property.
>
> And, if the sender can send an "I'm done" update followed by any
> number of "Oops, I changed this again" updates, I don't see the
> point of a receiver ever remembering that it is waiting for a
> pending change, or the protocol distinguishing between unsolicited
> and pendclr.
>
> That makes pendclr largely a hint, and a temporary one at that.
>
>
> > > I think
> > > REJECT might be more interesting than the difference between UNSOL,
> > > DONE, and MORE.
> >
> > How about an enum that distinguished:
> >       • Unsolicited
> >       • Clear pending
> >       • Still Pending
>
> > The additional distinction among degrees of request satisfaction in the
> last two
> > cases could be the responsibility of the receiver to determine. Since he
> made the request
> > and has access to the current value, he could determine this himself.
> Alternatively,
> > we could have an int with two distinct bit fields:
> >       • One distinguishing unsolicited/clear-pending/still-pending
> >       • Another regarding degrees of request satisfaction.
>
> In other words "rejection" doesn't matter. There's no recourse for
> the receiver; it just needs to know the current value of the
> property.
>
>
> > > There is a bit of a race here: a sender could send an unsolicited
> > > update notification at the same time the receiver requests a change
> > > of the same xchar. Could that result in a non-deterministic outcome?
> >
> > It shouldn't.  The point is that the property changes happen in a
> sequence which
> > is the same for all observers (no weird relativistic effects!) and that
> the updates
> > to the peer should happen that same sequence.  The important point is a
> > RESPXCH indicating a succesful change request be in the proper place in
> that
> > sequence.
> >
> > > Would it ever be reasonable to send two or more updates
> > > simultaneously for the same XCHAR?
> >
> > Since this is a single connection with sequenced delivery, there is
> > no way to send updates simultaneously.  You can queue them for
> > sending at the same time, but the delivery will reflect the order in
> > which were queued.
>
> If two or more ULPs are using the same connection, they might
> send a sequence of possibly contradictory property change
> requests.
>
> Also, there's no co-ordination between the two senders on the
> same connection; or an administrator might adjust these
> properties.
>
> I posit that the receiver cares only about the current effective
> value of these properties. pendclr is completely unreliable.
>
> Can you provide a real world example of how pendclr might be used?
> Because it appears to me to convey no actionable information, and
> I don't see why I shouldn't always set it to false.
>
> Think of this as an NFS attribute: SETATTR doesn't return "Oh, I
> changed this just a little" or "Someone adjusted this value, but
> it may change again".
>
>
> > > (Requiring single-threading here would prevent that from occurring).
> >
> > Given that there is no response to UPDXCH, not sure how you
> > could specify single-threading.  There is no way to define when
> > it would be OK to send the next.
> >
> > > What if the sender emits two optinfo_updxch messages: both with
> > > pendclr set to false, but one with an intermediate value, and one
> > > with the original value. The result on the receiver could depend on
> > > the order in which these messages arrive.
> >
> > It would.
> >
> > > Possibly some text
> > > regarding the ordering of these messages is needed.
> >
> > I can add something.
> >
> > > What happens if the receiver of ROPT_REQXCH drops the request?
> >
> > >Is there a timeout after which ROPT_REQXCH may be sent again?
> >
> > There is no timeout in the protocol.
> >
> > An implementation may choose to do so, but since this is sent
> > on a reliable connection, it is hard to imagine it being worth doing.
>
> Does the document's language forbid a receiver from simply discarding
> a property change request? (I'm thinking of similar language as RFC
> 7350 which requires that NFS servers never discard requests).
>
>
> > >What happens if an ROPT_RESPXCH is dropped? If ROPT_REQXCH is sent
> > > again, the reply is :
> >
> > >  ROPT_RESPXCH with the requested value marked done ?
> >
> > This should result.
> >
> > >   ROPT_RESPXCH with a rejection (no change was done) ?
> >
> > It is true no change was done but requested value was achieved,
> >
> > >  ROPT_UPDXCH with the requested value and pendclr set to true ?
> >
> > ROPT_UPPDXCH is not an alternative to ROPT_RESPXCH.  It is
> > a possible additional message.
> >
> > > I don't see language that disallows any of these responses. Which
> > > one means "I already set this value" ? Sorry if I missed that.
> >
> > I can add some clarification.
> >
> > > Assuming that both sides support ROPT_UPDXCH, can an implementation
> > > use ROPT_UPDXCH exclusively instead of ROPT_INITXCH?
> >
> > Yes but it  is kind of bogus.  You would be relying on the default
> initial values
> > and then changing them, which would be good in a test but, in real life,
> it is
> > asking for trouble.
>
> It's very easy to imagine an implementer deciding that UPDXCH is
> better for his or her client or server than FIRSTPROP, and noting
> stridently that the document does not prohibit or even warn against
> this design.
>
> IMO the document needs to make an explicit requirement not to
> behave this way, or state that it is acceptable though not preferred.
>
>
> > BTW, I've always wished there would be an RFC2119bis defining "BOGUS" and
> > "BRAIN-DEAD" :-)
>
> The closest we have is SHOULD NOT (which I always read as "don't do
> that, you muppet").
>
>
> > > Assuming that both sides support ROPT_UPDXCH, may a peer change an
> > > XCHAR and not send an unsolicited ROPT_UPDXCH?
> >
> > It may but in most cases it would be BOGUS (or BRAIN-DEAD).
> >
> > Suppose you raise the receive buffer size and don't tell you peer that
> it is raised,  In
> > that case. raising it is pretty pointless since the peer can't take
> advantage of the bigger
> > buffer.
> >
> > If you lower the receive buffer size and don't tell the peer it has been
> lowered, then he
> > is going to continue to assume a larger size and break things.
>
> For a client to reduce its receive buffers in mid-flight, all it
> needs to do is send Reply chunks more often. That will work OK
> if the server uses the Reply chunk whenever one is provided.
> (Not all servers behave nicely in this regard, though).
>
> For a server to reduce its receive buffers in mid-flight, the
> server needs to notify the client of the change in receive
> buffer size, the client would need to acknowledge the change,
> and only then can the server reduce its receive buffer size.
> How might that be done with the protocol proposed in
> rpcrdma-xcharext ?
>
> OK imagine some less harmful changes, like:
>
> The administrator on the client has turned a knob that enables
> Remote Invalidation. Yet it is perfectly safe for the server to
> continue not using RI. Is UPDXCH required in that case?
>
> If you'd rather the document does not commit, then maybe a
> sentence or two could be added that suggests the implementer
> needs to consider carefully the safety of not sending an
> unsolicited property change.
>
>
> --
> Chuck Lever
>
>
>
>