Re: [nfsv4] Preventing an NFSv4.1 client from destroying a migrated lease after TSM

[Chuck Lever <[e-mail]>]

> On Apr 18, 2017, at 3:30 PM, David Noveck <[e-mail]> wrote:
> 
> To avoid burying the lead, I'll respond waaay out of order:
> 
> > In sum, here are some options we've considered:
> 
> > 1a. destination asserts CONF_R, but client uses the
> > returned contrived slot sequence anyway
> 
> Clearly, migration-issues-13 needs to take account of this issue.
> I believe 1a is the best choice and will explain why below.
> 
> > We've explored this mechanism a little more, and now we seem
> > to be hamstrung on the first paragraph of RFC 5661, Section
> > 18.35.3:
> 
> Like much of RFCs 5661 and 7530, the paragraph in question
> was written without any consideration of issues related to 
> transparent state migration.  Sigh!
> 
> I think I have not been as explicit about this as I should have been
> but, as we have started to address a number of migration-related issues for
> NFSv4.1 it appears that we will need a standards-track document,
> updating RFC5661, paralleling RFC7931 for NFSv4.0.
> 
> RFC7931 had a replacement SETCLIENTID description and it now 
> appears that the new document will need a replacement for the
> EXCHANGE_ID  description.   Coincidence? ....  or something 
> else? :-)
> 
> >   The client uses the EXCHANGE_ID operation to register a particular
> >   client owner with the server.  
> 
> However, when the client_owner has been already been registered
> by other means (e.g. transparent state migration), the client may still
> use EXCHANGE_ID to obtain the clientid assigned.previously.
> 
> >  The client ID returned from this
> >   operation will be necessary for requests that create state on the
> >   server and will serve as a parent object to sessions created by the
> >   client.  
> 
> OK.
> 
> >   In order to confirm the client ID it must first be used,
> >   along with the returned eir_sequenceid, as arguments to
> >   CREATE_SESSION.  
> 
> In situations in which the registration of the clent_owner has not occurred 
> previously ...
> 
> > If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the
> > result, eir_flags, then eir_sequenceid MUST be ignored, as it has no
> >  relevancy.
> 
> It is not central to my argument but I believe that the "MUST" is not in
> accord with RFC2119.  The point here is that if a CREATE_SESSION 
> has been one, which this text assume is the case, that has already 
> established the proper sequence id to use.  This is explaning how the
> protocol works and not establishing an interoperability REQUIREMENT>
> 
> Anyway, I would go on:
> 
> If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the
> result, eir_flags, then it is an indication that the the registration
> of the client_owner has already occurred and that a further
> CREATE_SESSION is not  needed to confirm it.  Of course,
> subsequent CREATE_SESSIONs may be needed for other
> reasons.
> 
> The value eir_seqenceid is used to establish an initial
> sequence value associate with the clientid returned.  In cases
> in which a CREATE_SESSION has already been done, there
> is no need for this value, since sequencing of such request has 
> already been established and the client has no need for this
> value and will ignore it.
> 
> > The problem is the normative requirement in the last
> > sentence. 
> 
> I agree that this REQUIREMENT is a problem.
> 
> > If CONFIRMED_R is set, a client is required
> > to ignore the returned contrived slot sequence number.
> 
> According to RFC5661, yes.
> 
> > [ This is likely the reason that clients react to this
> > flag by purging the lease and starting over ].
> 
> I believe that the reason is that if the client thinks/knows a
> clientid is unconfirmed and he finds out that the server thinks
> otherwise, he figure something is really messed up, in which
> punting like this is to be preferred to a panic :-(
> 
> > Our first thought was to use the sequence number in the
> > migrated lease (in other words, the sequence number that
> > was established on the source server). A client knows
> > that sequence number by looking at the lease it has with
> > the source server.
> 
> In this case, you are moving over the single-slot 
> quasi-session associated with the client.
> 
> > However, there's no guarantee that the client will not
> > perform additional CREATE_SESSION operations against
> > the source server between the time the source server
> > copies the migrating lease and the client starts trunking
> > discovery for the destination server.
> 
> True.  The client can hack around this by starting with 
> the last sequence he has and backing up  if it is too
> high.  I'd rather not go there, though.
> 
> > Another thought was to use a well-known constant, like
> > zero, when creating the first session on the destination
> > server.
> 
> That is probably the best option if you is committed to respect
> the language of the current RFC5661 requirement.  However,
> you would still need to revise the EXCHANGE_ID description
> to provide a coherent description, in any case.
> 
> > We've established that trunking discovery works correctly
> > when the destination server does not assert CONFIRMED_R.
> 
> :-)
> 
> > This is because the client may use the contrived slot
> > sequence number provided by the destination server.
> 
> It's because he does use that slot sequence number.  To me, that is
> an additional reason he should be allowed to do so.
> 
> > If the destination server does not assert CONFIRMED_R,
> > then how does the client determine whether Transparent
> > State Migration has occurred? Can it simply start using
> > the open and lock state it has, and deal with BAD_STATEID
> > if the servers did not perform TSM?
> 
> I think that may be possible.  However, it is is ugly.  if you have another way
> to find out that state was transferred, you can look at the status bits from SEQUENCE,
> and only worry about lost stateids when there is an indication  that some
> state was lost.
> 
> > Confirmation of the lease means more than just that it
> > is present on the queried server
> 
> Confirmation means that the client is not uncertain about 
> whether it is exists.  Confirmation by means of CREATE_SESSION
> is one way to do that, by making sure that the EXCHANGE_ID was
> not a random retry, since abandoned. 
> 
> > it also means there is
> > now a cached CREATE_SESSION reply, and a known good
> > contrived slot sequence number. 
> 
> In many situations, that is a reasonable inference bu it doesn't
> change the meaning of "confirmation".
> 
> > Neither of those is true
> > for a migrated lease when the client's sessions were not
> > also migrated.
> 
> Yes, but I'm not certain that both of these are true in the case in which the client's
> session are migrated.
> 
> > Thus we should consider that a migrated lease is really
> > not confirmed at all, 
> 
> I consider it confirmed, but confirmed in a different manner.
> 
> > but in some intermediate state.
> 
> if it it were an intermediate state, there would be some way to get into 
> what you call the confirmed state (and I would call the 
> confirmed-by-CREATE_SESSION state).  You could do that by dong
> a CREATE_SESSION if you knew the sequenceid but I don't see the
> point in explicating the fine structure of the various confirmed clientid
> sub-states.
> 
> > A more ideal solution would be to create an additional
> > EXCHID4_FLAG that signifies the existence of a migrated
> > lease for the querying client.
> 
> If you were starting today (or had a time machine) you might do it that way.
> 
> 
> > In sum, here are some options we've considered:
> 
> > 1a. destination asserts CONF_R, but client uses the
> > returned contrived slot sequence anyway
> 
> 
> My first choice.

I believe that is what I have prototyped in the Linux
client, currently, and it is known to work as well as
a non-migration EXCHANGE_ID / CREATE_SESSION.

I don't object to this as long as the WG has blessed
this mechanism and documents it in an "NFSv4.1
migration update" as mentioned above.


> > 1b. destination asserts CONF_R, and client uses a fixed
> > constant starting slot sequence
> 
> My second choice.
> 
> > 2. destination does not assert CONF_R, and client is
> > prepared for BAD_STATEID if TSM did not occur
> 
> a real drag.
> 
> > 3. destination asserts a new EXCHID4_FLAG that signifies
> > a TSM has made the client's lease available on that
> > server
> 
> Unfortunately, NFSv4.1 is not an extensible minor version.
> 
> > 4. NFSv4.1 TSM cannot occur unless the client's sessions
> > are also migrated
> 
> 
> > Are there others?
> 
> There are others but none come to mind right now.
> 
> On Mon, Apr 17, 2017 at 2:45 PM, Chuck Lever <[e-mail]> wrote:
> 
> > On Mar 9, 2017, at 2:00 PM, David Noveck <[e-mail]> wrote:
> >
> > > I'm not certain what you mean by "handled ... as it
> > > would have been in 4.0". Server trunking discovery in
> > > NFSv4.0 requires an elaborate dance. In NFSv4.1,
> > > trunking discovery can be done by a single operation
> > > (EXCHANGE_ID).
> >
> > What I meant was that, apart from the greater
> > calories expended in v4.0 case, the result should have
> > been the same, i.e. that there is no trunking.
> >
> > > I'm speculating that if the server hadn't asserted
> > > CONFIRMED_R, this client could have recovered
> > > correctly from the migration event. Is that what you
> > > mean?
> >
> > It wasn't what I meant, but, now that you say it, I think is
> > probably right.
> 
> We've explored this mechanism a little more, and now we seem
> to be hamstrung on the first paragraph of RFC 5661, Section
> 18.35.3:
> 
>    The client uses the EXCHANGE_ID operation to register a particular
>    client owner with the server.  The client ID returned from this
>    operation will be necessary for requests that create state on the
>    server and will serve as a parent object to sessions created by the
>    client.  In order to confirm the client ID it must first be used,
>    along with the returned eir_sequenceid, as arguments to
>    CREATE_SESSION.  If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the
>    result, eir_flags, then eir_sequenceid MUST be ignored, as it has no
>    relevancy.
> 
> The problem is the normative requirement in the last
> sentence. If CONFIRMED_R is set, a client is required
> to ignore the returned contrived slot sequence number.
> [ This is likely the reason that clients react to this
> flag by purging the lease and starting over ].
> 
> Our first thought was to use the sequence number in the
> migrated lease (in other words, the sequence number that
> was established on the source server). A client knows
> that sequence number by looking at the lease it has with
> the source server.
> 
> However, there's no guarantee that the client will not
> perform additional CREATE_SESSION operations against
> the source server between the time the source server
> copies the migrating lease and the client starts trunking
> discovery for the destination server.
> 
> Another thought was to use a well-known constant, like
> zero, when creating the first session on the destination
> server.
> 
> What sequence number should be used for the first
> CREATE_SESSSION with the destination server? If it is
> the copied sequence number, what can prevent mutation
> of that sequence number while migration of that lease
> is not yet complete?
> 
> We've established that trunking discovery works correctly
> when the destination server does not assert CONFIRMED_R.
> This is because the client may use the contrived slot
> sequence number provided by the destination server.
> 
> If the destination server does not assert CONFIRMED_R,
> then how does the client determine whether Transparent
> State Migration has occurred? Can it simply start using
> the open and lock state it has, and deal with BAD_STATEID
> if the servers did not perform TSM?
> 
> 
> Confirmation of the lease means more than just that it
> is present on the queried server: it also means there is
> now a cached CREATE_SESSION reply, and a known good
> contrived slot sequence number. Neither of those is true
> for a migrated lease when the client's sessions were not
> also migrated.
> 
> Thus we should consider that a migrated lease is really
> not confirmed at all, but in some intermediate state.
> A more ideal solution would be to create an additional
> EXCHID4_FLAG that signifies the existence of a migrated
> lease for the querying client.
> 
> 
> In sum, here are some options we've considered:
> 
> 1a. destination asserts CONF_R, but client uses the
> returned contrived slot sequence anyway
> 
> 1b. destination asserts CONF_R, and client uses a fixed
> constant starting slot sequence
> 
> 2. destination does not assert CONF_R, and client is
> prepared for BAD_STATEID if TSM did not occur
> 
> 3. destination asserts a new EXCHID4_FLAG that signifies
> a TSM has made the client's lease available on that
> server
> 
> 4. NFSv4.1 TSM cannot occur unless the client's sessions
> are also migrated
> 
> Are there others?
> 
> --
> Chuck Lever
> 
> 
> 
> 

--
Chuck Lever