Re: [nfsv4] New Version Notification for draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt

[David Noveck <[e-mail]>]

> Does the receiver have any input on the unsolicited xchar
> update?

No.  It is an unsolicited one-way message.  Reporting to
the receiver what has occurred.

> Lets say the sender (i.e. NFS server for example), wants to
> increase its recv buffer size, but the receiver (NFS Client) is
> low on memory.

I don't see how the fact that the client is low on memory
affects the server.  The client does not have to use the server's
larger buffer size.  He only has to make sure that he doesn't send anything
bigger than the receive buffer size.  If he is
aleady doing that for receive buffer size X, then there should be no
problem doing it for  receive buffer size Y, with Y > X.

> Is there any way for it to say I don't like that value, not now,
> rejected, etc?

No.

> Also, not sure if this was discussed, but if a sender or
> receiver support changes to properties, seems like they must
> at least have 1 additional recv buffer posted to accept the
> reqxch, is one enough?

Chuck did raise this issue.  My plan for -03 is basically to say
that a REQXCH_RESPXCH pair, is, for purposes of credit management, to be
treated as if it were

> If a reqxch were sent to make a change, and the receiver
> believes the property can be changed, just maybe not to the > value the
sender wants, it needs to return the respxch with
> rejected, and then follow up with an updxch to increase the
> value possible just a little bit more, just not to the value the
> sender wanted.

Actually the requested is not rejected.  The requested charcteristic is
included in optrespxch_pend nd there is a UPDXCH to follow up with the
selected value.

> Seems like maybe to avoid two messages, the receiver
> could have returned a respxch but indicated that new value. > But I don't
see a value in the return in respxch.

There isn't one now but I am considering a change to UPDXCH in -03 that
would include this.  The goal would not be to reduce the message count,
although it would do that.  Instead, the idea would to separate solicited
and unsolicited
changes and get rid of confusion/controversy regarding pendclr.

> If reqxch can only request 1 xcharspec, then why does
> respxch have the ability to indicate more than one property
> were changed, or rejected,etc?

one xcharspec --> multiple xcharvals so changes in multiple properties can
be requested.

> Also, why would an updxch be used at all, wouldn't the peer > just send
its own reqxch?  Thats whats confusing to me.  Is it > more like I'm not
asking to change this propery, I'm telling
> you its changed?

It is exactly the latter.

Clearly the existing text isn't clear enough.  Do you have any suggestions
to make this clearer?

> Also, what guarentees that the xid of the updxch is unique
> among the xid's the peer is using?

There aren't any and I don't believe any are needed.  There is some text
that says xid for updxch is not tied to the rqxch, but I think it need to
be more clearly stated that there is no use for this.  I could either tell
the receiver to ignore this or tell the sender to always make this zero.

> What if a reqxch and an updxch pass at the same time for
> the same property?

Do you have a specific issue in mind? What would hppen depens on the
particulars of the type of change, whether
the change

> The updxch pendclr maybe true cause it never saw the
> reqxch, no?

> Would it make sense with the optinfo_initxch to also have a > supported
max/min value that would possible be accepted if > a change were to be made?

It's possible but this is much bigger than the sort of changes I can
consider for -03.  Let's leave this for -04 or after this is merge into
version-twp

One issue with this is it assume all properties can be ordered linearly
with a min and max.  There are propertes in -rtrext that don't follow that
model

> Also seems that the updxch's optupdxch_now (new)
> should be xcharspec.

Is see that xcharspec is typedef'ed to xcharval and its really the same
thing, but it makes for easier reading when the reqxch is an xcharspec.



On Fri, Aug 26, 2016 at 7:57 PM, karen deitke <[e-mail]>
wrote:

> Does the receiver have any input on the unsolicited xchar update?  Lets
> say the sender (i.e. NFS server for example), wants to increase its recv
> buffer size, but the receiver (NFS Client) is low on memory.  Is there any
> way for it to say I don't like that value, not now, rejected, etc?
>
> Also, not sure if this was discussed, but if a sender or receiver support
> changes to properties, seems like they must at least have 1 additional recv
> buffer posted to accept the reqxch, is one enough?
>
> If a reqxch were sent to make a change, and the receiver believes the
> property can be changed, just maybe not to the value the sender wants, it
> needs to return the respxch with rejected, and then follow up with an
> updxch to increase the value possible just a little bit more, just not to
> the value the sender wanted.  Seems like maybe to avoid two messages, the
> receiver could have returned a respxch but indicated that new value.  But I
> don't see a value in the return in respxch.
>
> If reqxch can only request 1 xcharspec, then why does respxch have the
> ability to indicate more than one property were changed, or rejected,etc?
> Is this to allow multiple reqxch requests to be notified of completion with
> 1 respxch?
> If the respxch were one for one with the reqxch, seems we could get rid of
> the concept of pending reqxch, especially if the respxch had the value
> which could eliminate the updxch?  Or can you put more than one reqxch in
> one messages?  That might be nice.
>
> Also, why would an updxch be used at all, wouldn't the peer just send its
> own reqxch?  Thats whats confusing to me.  Is it more like I'm not asking
> to change this propery, I'm telling you its changed?
>
>
> Also, what guarentees that the xid of the updxch is unique among the xid's
> the peer is using?
>
> What if a reqxch and an updxch pass at the same time for the same
> property?  The updxch pendclr maybe true cause it never saw the reqxch,
> no?
>
> Would it make sense with the optinfo_initxch to also have a supported
> max/min value that would possible be accepted if a change were to be made?
>
> Also seems that the updxch's optupdxch_now (new) should be xcharspec.  Is
> see that xcharspec is typedef'ed to xcharval and its really the same thing,
> but it makes for easier reading when the reqxch is an xcharspec.
>
>
>
> Karen
>
>
>
>
> On 8/26/2016 11:57 AM, David Noveck wrote:
>
> > No, I'm saying that pendclr adds no value; the receiver knows
> > everything it needs to know by looking only at the new value.
>
> That seems to be because, you believe that the status of a
> previously pending request (completed or not) is not something
> that the receiver needs to know.  In some cases it might
> not but given that there is mechanism to make requests, it
> seems to me that the one making the request
>
> > I don't have a problem with UPDXCH as a mechanism for unsolicited
> > change notifications, though I'm not sure we have a use case
> > where the same kind of notification cannot be done with REQXCH.
>
> REQXCH cannot be used for unsolicited change notifications.  It
> is intended for  requested/solicited changes.
>
>
> > It's especially perplexing to me to see a facility for asynchronous
> > notification of completion when there is no real call-reply transaction
> > relationship between REQXCH and REPXCH, and there is no timeout when
> > sending REQXCH messages. A key question for me is "How is [UPDXCH
> > with pendclr set] different than a delayed REPXCH response?" If they
> > are semantically the same, then why have both?
>
> It is possible to limit UPDXCH to unsolicited changes.  The issue I
> have to consider is how to deal with partial satisfaction (e.g.  you ask
> for 32K and get 16K).
>
> > If the document moves forward with pendclr, I think implementers need
> > more guidance about how to use this field.
>
> Let me think about the text for this.
>
> > Otherwise, someone is going
> > to build an implementation that depends on pendclr,
>
> That presumably would be an implementation that requested a property
> change.
>
> > and some other
> > implementation that implemented the protocol blindly (ie without any
> > need for the pendclr value)
>
> The implementation it would be operating with would be one that
> responded to such requests.  And an implementation that responds to
> such requests and doesn't make them has no need for the pendclr,
> but it still should set it for those that need it.
>
>
> > In other words, I think there is useful feedback from an implementation
> > that does not need pendclr.
>
> I'm not sure what that feedback would be.  The fact implementation A has
> no use for it does not mean that information B has no need for it.
>
> > Perhaps only interop testing can sort it out.
>
> Perhaps.
>
>
> > > The protocol allows unsolicited changes
> > > but that doesn't mean they are going to be happening all that
> frequently.
>
> > A sane implementation of the currently proposed initial properties
> > probably won't use UPDXCH at all.
>
> I don't see much use of UPDXCH (by sane implementations) for a while but
> i think there will eventually be a need for it.
>
> > I'm trying to explore the corners
> > here and play devil's advocate; if an implementer is allowed to use
> > UPDXCH in a crazy way, we should try to anticipate that and either
> > prohibit it or make it safe.
>
> I don't think safety is the issue,  Even if it is safe, people using it
> the way you anticipate would disruptive, in that other implementations
> would be forced to implement the same function twice, to work
> with an implementation
>
>
> > To interoperate among implementations that depend on it and
> > ones that don't need it, though, I think the document needs
> > more precise language about how pendclr is to be used.
>
> I'll try to provide that.
>
>
>
> > (Much) earlier you suggested that a ULP might want to select its
> > own receive buffer size, for example. If two ULPs share the same
> > connection, then they could choose to adjust the connection in
> > contradictory ways. Perhaps you are taking that off the table
> > now, which is fair enough.
>
> Consider it off the table.
>
>
>
> > My remark here was not about two ULPs. The two directions of
> > transmission on a connection are not co-ordinated in any way. An
> > unsolicited UPDXCH can pass a REQXCH going in opposite directions.
>
> Yes.
>
> > IMO the document has to acknowledge that property value changes
> > have to be done carefully and with co-ordination with the regular
> > operation of the transport to avoid hazardous oscillations of
> > values.
>
> Agree.
>
> > The PROP protocol mechanism does not provide any kind of
> > serialization on its own.
>
> True.
>
>
>
>
> > > It then requests a smaller receiver buffer size.  as part of doing
> that it decides
> > > to not send requests or responses that are larger than this
> anticipated size
> > > limit.  I think the spec already discusses this but the basic point is
> that by making
> > > this request the sender is licensing the peer to reduce the buffer
> size.  If this
> > > request is completed, the sender will make that new size permanent (or
> at least until
> > > it hears about a change). On the other hand, if the peer rejects the
> request
> > > (either fairly soon via RESPXCH, or  after a while via UPDXCH), the
> sender would
> > > then adopt the current value, since the possibility of the
> > > peer eventually
> > > satisfying the orginal request is now gone. On the other hand, an
> update not including pendclr
> > > should not have that effect.  As long as the request is pending, he
> has to be prepared
> > > for its peer to adopt the lower value that it has requested.
>
> > I don't see how the REQuester's behavior would be different if pendclr
> > was false. It can't assume the requested setting takes effect until
> > it gets a REPXCH or UPDXCH with the new value, whether or not pendclr
> > is true, and there's nothing in the spec (that I recall) that says
> > otherwise (about pendclr being true is required before a new value is
> > trusted -- did I miss something? --
>
> It's not that it is not trusted.  It is that when you have a request
> outstanding to redeuce the the
> size to X, you need to take that account.  So, if you get an update that
> the size is now Y, you
> accept that the size is Y when that was sent but the possibility exists
> that it might be set
> down to X, since you requested that that be done.
>
> > > and if that is the case, then
> > > what's the value of sending an unsolicited UPDXCH ?)
>
> In general there is value in doing so, but there may be special
> circumstances in which a
> existence of a pending  request restricts the receiver from acting on it. .
>
> > There's nothing that prevents the REP side from sending "yes, I took
> > the new setting (pendclr true)" and then "oops, no I went back to the
> > old one (pendclr false)". That's not something an implementer might
> > do on purpose, but it sure can happen by accident (poorly-timed
> > administrator action, say).
>
> Yes, but in that case, the requester will find out that his request has
> been granted,
> and then that administrative action has superseded it.  the implementation
> should
> work OK although people might have a problem with the administrator's
> choices.
>
>
>
> > You could state that UPDXCH and REQXCH MUST/SHOULD NOT be used before
> > FIRSTPROP has been exchanged.
>
> MUST NOT is OK with me.  The next step is to state the same regarding any
> particular property.  I'm OK with SHOULD NOT for that.
>
>
>
>
> > > I don't think unsolicited reduction in receive buffer size (as opposed
> to the
> > > requested reduction discussed above) can be made to work.
>
> > It could, perhaps, if ...
>
> I was speaking of the protocol currently described.
>
> > the property had two values: one was the send
> > size limit, and the other was the receive buffer size.
>
> I think that work better as two separate properties.  Defining a
> new property
> is a lot easier than changing the structure of an existing one.
>
> > The send size
> > limit is not necessarily associated with a physical buffer size, if
> > Sends are done with a gather vector, so its relatively
> > straightforward to change it.
>
> > One peer could send a REQXCH to reduce the sender's send size limit.
> > If the other peer says "OK" then it is safe to reduce the receive
> > buffer size.
>
> > In fact, IMO both sides do want to know the send size limit of their
> > peer so they can trim receive buffers to a size that would be used.
> > It might also help a requester determine when a responder needs a
> > Reply chunk.
>
>
> > > Maybe I should add an explicit statement
> > > to that effect.
>
> > That would be implementation guidance at best, and might preclude
> > some future innovation that allows unsolicited receive buffer size
> > changes.
>
> I think I could draft something that would not run into the preclusion
> problem.
>
> You just have to point out that if the sendr is told that the receivers
> buffer are X
> long, he is going to be sending messages that long, making a reduction in
> receive buffer size problmatic in the absence of a mechasim to nsure that
> the
> sender aware of the new (lower) limit.
>
> > Should similar statements be made about the other initial properties
> > proposed in rpcrdma-xcharext?
>
> Probably.
>
> On Fri, Aug 26, 2016 at 11:39 AM, Chuck Lever <[e-mail]>
> wrote:
>
>> Hiya Dave-
>>
>> > On Aug 26, 2016, at 5:08 AM, David Noveck < <[e-mail]>
>> [e-mail]> wrote:
>> >
>> > > So, there is a protocol mechanism for reporting partial or full
>> > > completion of a property change request, yet no mechanism for
>> > > indicating which change request it is the receiver is completing.
>> > > But multiple property change requests are allowed to be in flight
>> > > concurrently.
>> >
>> > The protocol allows it, but particular implementations, may well
>> > restrict things so that they have only one in flight for each property
>> > at any particular time.
>> >
>> > > Without an XID, the only way this might work is by serializing
>> > > updates to each property.
>> >
>> > It is one way.  I don't think it is the only way.  The requester, if it
>> > is interested in knowing when there are no outstanding requests
>> > for a property can increment on requests and decrement on messages
>> > that indicate completion.
>> >
>> > > And, if the sender can send an "I'm done" update followed by any
>> > > number of "Oops, I changed this again" updates,
>> >
>> > The updates indicate that there has been an unsolicited change in
>> > the property:  no "Oops" and no "again".
>> >
>> > Whether such changes happen or not, depends on the
>> > implementation.  In any case, I would expect them to be
>> > rare.
>> >
>> > The proposed protocol provides a way for the peer to be
>> > informed of such changes.   I don't see how that undercuts
>> > the facilities that allow the completion of requested changes
>> > to be reported asynchronously.
>> >
>> > > I don't see the
>> > > point of a receiver ever remembering that it is waiting for a
>> > > pending change, or the protocol distinguishing between unsolicited
>> > > and pendclr.
>> >
>> > You seem to be saying the potential existence of unsolicited changes
>> > some time after the completion of a requested change somehow
>> > makes the notion of completion of the requested change useless.
>>
>> No, I'm saying that pendclr adds no value; the receiver knows
>> everything it needs to know by looking only at the new value.
>>
>> I don't have a problem with UPDXCH as a mechanism for unsolicited
>> change notifications, though I'm not sure we have a use case
>> where the same kind of notification cannot be done with REQXCH.
>>
>>
>> > I'm not sure why you believe that.
>>
>> I believe that, because I can't for the life of me think of a use
>> case where a receiver can make use of the information in pendclr.
>> (Yes, I probably lack quite a bit of imagination). Can you please
>> provide one?
>>
>> It's difficult, as a reviewer of proposed protocol, to understand
>> why we are reserving these 32 bits in the UPDXCH message without
>> having at least a little clue about why they are there and how a
>> receiver would make use of them.
>>
>> It's especially perplexing to me to see a facility for asynchronous
>> notification of completion when there is no real call-reply transaction
>> relationship between REQXCH and REPXCH, and there is no timeout when
>> sending REQXCH messages. A key question for me is "How is [UPDXCH
>> with pendclr set] different than a delayed REPXCH response?" If they
>> are semantically the same, then why have both?
>>
>> If the document moves forward with pendclr, I think implementers need
>> more guidance about how to use this field. Otherwise, someone is going
>> to build an implementation that depends on pendclr, and some other
>> implementation that implemented the protocol blindly (ie without any
>> need for the pendclr value) is not going to get it right (or will take
>> shortcuts, or whatever), and there will be an interoperability problem,
>> just add boiling water and fluff with fork.
>>
>> In other words, I think there is useful feedback from an implementation
>> that does not need pendclr. Perhaps only interop testing can sort it
>> out.
>>
>>
>> > The protocol allows unsolicited changes
>> > but that doesn't mean they are going to be happening all that
>> frequently.
>>
>> A sane implementation of the currently proposed initial properties
>> probably won't use UPDXCH at all. I'm trying to explore the corners
>> here and play devil's advocate; if an implementer is allowed to use
>> UPDXCH in a crazy way, we should try to anticipate that and either
>> prohibit it or make it safe.
>>
>>
>> > > it just needs to know the current value of the property.
>> >
>> > It certainly needs to know that.  Your implementaition might
>> > not be interested in anything else but the protocol provides it
>> > because it is easy for the sender to provide and useful in some
>> > cases.
>>
>> To interoperate among implementations that depend on it and
>> ones that don't need it, though, I think the document needs
>> more precise language about how pendclr is to be used.
>>
>>
>> > > If two or more ULPs are using the same connection, they might
>> > > send a sequence of possibly contradictory property change
>> > > requests.
>> >
>> > I don't understand why one might assume that ULP's
>> > are sending property change requests.  ULPs are RPC
>> > protocols.  They send rpc requests and receive replies.
>>
>> (Much) earlier you suggested that a ULP might want to select its
>> own receive buffer size, for example. If two ULPs share the same
>> connection, then they could choose to adjust the connection in
>> contradictory ways. Perhaps you are taking that off the table
>> now, which is fair enough.
>>
>>
>> > > Also, there's no co-ordination between the two senders on the
>> > > same connection; or an administrator might adjust these
>> > > properties.
>> >
>> > Any co-ordination requirement that a transport implementation
>> > might impose to deal with the case of multiple ULPs would be
>> > out of scope in this document.
>>
>> My remark here was not about two ULPs. The two directions of
>> transmission on a connection are not co-ordinated in any way. An
>> unsolicited UPDXCH can pass a REQXCH going in opposite directions.
>>
>> IMO the document has to acknowledge that property value changes
>> have to be done carefully and with co-ordination with the regular
>> operation of the transport to avoid hazardous oscillations of
>> values. The PROP protocol mechanism does not provide any kind of
>> serialization on its own.
>>
>>
>> > I don't see how you decide there is "no co-ordination".  If you build
>> > an implementation and give multiple ULPs the ability to request and
>> > make such changes independently, you are asking for (or begging for)
>> > trouble.  As an implementer, the choice regarding co-ordination (or not)
>> > is up to you.
>> >
>> > I think the entities making such changes are most likely to reflect
>> > administrative requests or internal transport optimization functions.
>> >
>> > I can't see why one would allow the ULP to do this itself.
>>
>> Yes, I'm talking about administrative requests or optimization
>> efforts, here.
>>
>>
>> > > I posit that the receiver cares only about the current effective
>> > > value of these properties.
>> >
>> > That's one of the things he cares about, but I believe it is not
>> > the only one.
>> >
>> > pendclr is completely unreliable.
>> >
>> > If you were to always set it to false, as you suggest below that you
>> > might, you can make it unreliable, but there is no good reason to do
>> that.
>> >
>> > > Can you provide a real world example of how pendclr might be used?
>> > > Because it appears to me to convey no actionable information, and
>> > > I don't see why I shouldn't always set it to false.
>> >
>> >
>> > Let's consider that a client implementation encounters a server
>> > which has a very large receive buffer size and feels that it would
>> prefer more
>> > smaller buffers.
>>
>> More likely, either the client or server is low on resources, and
>> wishes to reclaim receive buffer space. But, OK.
>>
>>
>> > It then requests a smaller receiver buffer size.  as part of doing that
>> it decides
>> > to not send requests or responses that are larger than this anticipated
>> size
>> > limit.  I think the spec already discusses this but the basic point is
>> that by making
>> > this request the sender is licensing the peer to reduce the buffer
>> size.  If this
>> > request is completed, the sender will make that new size permanent (or
>> at least until
>> > it hears about a change). On the other hand, if the peer rejects the
>> request
>> > (either fairly soon via RESPXCH, or  after a while via UPDXCH), the
>> sender would
>> > then adopt the current value, since the possibility of the
>> > peer eventually
>> > satisfying the orginal request is now gone. On the other hand, an
>> update not including pendclr
>> > should not have that effect.  As long as the request is pending, he has
>> to be prepared
>> > for its peer to adopt the lower value that it has requested.
>>
>> I don't see how the REQuester's behavior would be different if pendclr
>> was false. It can't assume the requested setting takes effect until
>> it gets a REPXCH or UPDXCH with the new value, whether or not pendclr
>> is true, and there's nothing in the spec (that I recall) that says
>> otherwise (about pendclr being true is required before a new value is
>> trusted -- did I miss something? -- and if that is the case, then
>> what's the value of sending an unsolicited UPDXCH ?).
>>
>> There's nothing that prevents the REP side from sending "yes, I took
>> the new setting (pendclr true)" and then "oops, no I went back to the
>> old one (pendclr false)". That's not something an implementer might
>> do on purpose, but it sure can happen by accident (poorly-timed
>> administrator action, say).
>>
>>
>> > > It's very easy to imagine an implementer deciding that UPDXCH is
>> > > better for his or her client or server than FIRSTPROP, and noting
>> > > stridently that the document does not prohibit or even warn against
>> > > this design.
>> >
>> > > IMO the document needs to make an explicit requirement not to
>> > > behave this way, or state that it is acceptable though not preferred.
>> >
>> > Are you sure that it is "requirement' and not a "REQUIREMENT" that you
>> > are looking for?
>> >
>> > I'm OK with saying you shouldn't do that, but I have problem saying it
>> is "acceptable".  I think it is noxious, although there is no "NOXIOUS".
>> >
>> > Maybe the right thing is to state the obvious, that the function of
>> UPDXCH is to
>> > convey changes, making its use to present values at connection time
>> dubious,
>> > while the function of ROPT_FIRSTPROP is to present the value at initial
>> connection.
>>
>> You could state that UPDXCH and REQXCH MUST/SHOULD NOT be used before
>> FIRSTPROP has been exchanged.
>>
>> Or even should not. I would go with MUST NOT: then receivers can depend
>> on seeing FIRSTPROP before seeing other changes.
>>
>>
>> > > For a client to reduce its receive buffers in mid-flight, all it
>> > > needs to do is send Reply chunks more often. That will work OK
>> > > if the server uses the Reply chunk whenever one is provided.
>> > > (Not all servers behave nicely in this regard, though).
>> >
>> > > For a server to reduce its receive buffers in mid-flight, the
>> > > server needs to notify the client of the change in receive
>> > > buffer size, the client would need to acknowledge the change,
>> > > and only then can the server reduce its receive buffer size.
>> > > How might that be done with the protocol proposed in
>> > > rpcrdma-xcharext ?
>> >
>> > I don't think unsolicited reduction in receive buffer size (as opposed
>> to the
>> > requested reduction discussed above) can be made to work.
>>
>> It could, perhaps, if the property had two values: one was the send
>> size limit, and the other was the receive buffer size. The send size
>> limit is not necessarily associated with a physical buffer size, if
>> Sends are done with a gather vector, so its relatively
>> straightforward to change it.
>>
>> One peer could send a REQXCH to reduce the sender's send size limit.
>> If the other peer says "OK" then it is safe to reduce the receive
>> buffer size.
>>
>> In fact, IMO both sides do want to know the send size limit of their
>> peer so they can trim receive buffers to a size that would be used.
>> It might also help a requester determine when a responder needs a
>> Reply chunk.
>>
>>
>> > Maybe I should add an explicit statement
>> > to that effect.
>>
>> That would be implementation guidance at best, and might preclude
>> some future innovation that allows unsolicited receive buffer size
>> changes.
>>
>> Should similar statements be made about the other initial properties
>> proposed in rpcrdma-xcharext?
>>
>> --
>> Chuck Lever
>>
>>
>>
>>
>
>
> _______________________________________________
> nfsv4 mailing [e-mail]://www.ietf.org/mailman/listinfo/nfsv4
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>