Re: [nfsv4] Preventing an NFSv4.1 client from destroying a migrated lease after TSM

[David Noveck <[e-mail]>]

> I believe that is what I have prototyped in the Linux
> client, currently, and it is known to work as well as
> a non-migration EXCHANGE_ID / CREATE_SESSION.

Good  to know.

> I don't object to this as long as the WG has blessed
> this mechanism

I don't know exactly what is involved in the WG
"bless[ing]" this mechanism.  Perhaps it is not distinct from:

> documents it in an "NFSv4.1
> migration update" as mentioned above.

In any case, we'll discuss the possibility of moving forward on
a standards-track document once migration-issues-13 is out.




On Wed, Apr 19, 2017 at 12:09 PM, Chuck Lever <[e-mail]>
wrote:

>
> > On Apr 18, 2017, at 3:30 PM, David Noveck <[e-mail]> wrote:
> >
> > To avoid burying the lead, I'll respond waaay out of order:
> >
> > > In sum, here are some options we've considered:
> >
> > > 1a. destination asserts CONF_R, but client uses the
> > > returned contrived slot sequence anyway
> >
> > Clearly, migration-issues-13 needs to take account of this issue.
> > I believe 1a is the best choice and will explain why below.
> >
> > > We've explored this mechanism a little more, and now we seem
> > > to be hamstrung on the first paragraph of RFC 5661, Section
> > > 18.35.3:
> >
> > Like much of RFCs 5661 and 7530, the paragraph in question
> > was written without any consideration of issues related to
> > transparent state migration.  Sigh!
> >
> > I think I have not been as explicit about this as I should have been
> > but, as we have started to address a number of migration-related issues
> for
> > NFSv4.1 it appears that we will need a standards-track document,
> > updating RFC5661, paralleling RFC7931 for NFSv4.0.
> >
> > RFC7931 had a replacement SETCLIENTID description and it now
> > appears that the new document will need a replacement for the
> > EXCHANGE_ID  description.   Coincidence? ....  or something
> > else? :-)
> >
> > >   The client uses the EXCHANGE_ID operation to register a particular
> > >   client owner with the server.
> >
> > However, when the client_owner has been already been registered
> > by other means (e.g. transparent state migration), the client may still
> > use EXCHANGE_ID to obtain the clientid assigned.previously.
> >
> > >  The client ID returned from this
> > >   operation will be necessary for requests that create state on the
> > >   server and will serve as a parent object to sessions created by the
> > >   client.
> >
> > OK.
> >
> > >   In order to confirm the client ID it must first be used,
> > >   along with the returned eir_sequenceid, as arguments to
> > >   CREATE_SESSION.
> >
> > In situations in which the registration of the clent_owner has not
> occurred
> > previously ...
> >
> > > If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the
> > > result, eir_flags, then eir_sequenceid MUST be ignored, as it has no
> > >  relevancy.
> >
> > It is not central to my argument but I believe that the "MUST" is not in
> > accord with RFC2119.  The point here is that if a CREATE_SESSION
> > has been one, which this text assume is the case, that has already
> > established the proper sequence id to use.  This is explaning how the
> > protocol works and not establishing an interoperability REQUIREMENT>
> >
> > Anyway, I would go on:
> >
> > If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the
> > result, eir_flags, then it is an indication that the the registration
> > of the client_owner has already occurred and that a further
> > CREATE_SESSION is not  needed to confirm it.  Of course,
> > subsequent CREATE_SESSIONs may be needed for other
> > reasons.
> >
> > The value eir_seqenceid is used to establish an initial
> > sequence value associate with the clientid returned.  In cases
> > in which a CREATE_SESSION has already been done, there
> > is no need for this value, since sequencing of such request has
> > already been established and the client has no need for this
> > value and will ignore it.
> >
> > > The problem is the normative requirement in the last
> > > sentence.
> >
> > I agree that this REQUIREMENT is a problem.
> >
> > > If CONFIRMED_R is set, a client is required
> > > to ignore the returned contrived slot sequence number.
> >
> > According to RFC5661, yes.
> >
> > > [ This is likely the reason that clients react to this
> > > flag by purging the lease and starting over ].
> >
> > I believe that the reason is that if the client thinks/knows a
> > clientid is unconfirmed and he finds out that the server thinks
> > otherwise, he figure something is really messed up, in which
> > punting like this is to be preferred to a panic :-(
> >
> > > Our first thought was to use the sequence number in the
> > > migrated lease (in other words, the sequence number that
> > > was established on the source server). A client knows
> > > that sequence number by looking at the lease it has with
> > > the source server.
> >
> > In this case, you are moving over the single-slot
> > quasi-session associated with the client.
> >
> > > However, there's no guarantee that the client will not
> > > perform additional CREATE_SESSION operations against
> > > the source server between the time the source server
> > > copies the migrating lease and the client starts trunking
> > > discovery for the destination server.
> >
> > True.  The client can hack around this by starting with
> > the last sequence he has and backing up  if it is too
> > high.  I'd rather not go there, though.
> >
> > > Another thought was to use a well-known constant, like
> > > zero, when creating the first session on the destination
> > > server.
> >
> > That is probably the best option if you is committed to respect
> > the language of the current RFC5661 requirement.  However,
> > you would still need to revise the EXCHANGE_ID description
> > to provide a coherent description, in any case.
> >
> > > We've established that trunking discovery works correctly
> > > when the destination server does not assert CONFIRMED_R.
> >
> > :-)
> >
> > > This is because the client may use the contrived slot
> > > sequence number provided by the destination server.
> >
> > It's because he does use that slot sequence number.  To me, that is
> > an additional reason he should be allowed to do so.
> >
> > > If the destination server does not assert CONFIRMED_R,
> > > then how does the client determine whether Transparent
> > > State Migration has occurred? Can it simply start using
> > > the open and lock state it has, and deal with BAD_STATEID
> > > if the servers did not perform TSM?
> >
> > I think that may be possible.  However, it is is ugly.  if you have
> another way
> > to find out that state was transferred, you can look at the status bits
> from SEQUENCE,
> > and only worry about lost stateids when there is an indication  that some
> > state was lost.
> >
> > > Confirmation of the lease means more than just that it
> > > is present on the queried server
> >
> > Confirmation means that the client is not uncertain about
> > whether it is exists.  Confirmation by means of CREATE_SESSION
> > is one way to do that, by making sure that the EXCHANGE_ID was
> > not a random retry, since abandoned.
> >
> > > it also means there is
> > > now a cached CREATE_SESSION reply, and a known good
> > > contrived slot sequence number.
> >
> > In many situations, that is a reasonable inference bu it doesn't
> > change the meaning of "confirmation".
> >
> > > Neither of those is true
> > > for a migrated lease when the client's sessions were not
> > > also migrated.
> >
> > Yes, but I'm not certain that both of these are true in the case in
> which the client's
> > session are migrated.
> >
> > > Thus we should consider that a migrated lease is really
> > > not confirmed at all,
> >
> > I consider it confirmed, but confirmed in a different manner.
> >
> > > but in some intermediate state.
> >
> > if it it were an intermediate state, there would be some way to get into
> > what you call the confirmed state (and I would call the
> > confirmed-by-CREATE_SESSION state).  You could do that by dong
> > a CREATE_SESSION if you knew the sequenceid but I don't see the
> > point in explicating the fine structure of the various confirmed clientid
> > sub-states.
> >
> > > A more ideal solution would be to create an additional
> > > EXCHID4_FLAG that signifies the existence of a migrated
> > > lease for the querying client.
> >
> > If you were starting today (or had a time machine) you might do it that
> way.
> >
> >
> > > In sum, here are some options we've considered:
> >
> > > 1a. destination asserts CONF_R, but client uses the
> > > returned contrived slot sequence anyway
> >
> >
> > My first choice.
>
> I believe that is what I have prototyped in the Linux
> client, currently, and it is known to work as well as
> a non-migration EXCHANGE_ID / CREATE_SESSION.
>
> I don't object to this as long as the WG has blessed
> this mechanism and documents it in an "NFSv4.1
> migration update" as mentioned above.
>
>
> > > 1b. destination asserts CONF_R, and client uses a fixed
> > > constant starting slot sequence
> >
> > My second choice.
> >
> > > 2. destination does not assert CONF_R, and client is
> > > prepared for BAD_STATEID if TSM did not occur
> >
> > a real drag.
> >
> > > 3. destination asserts a new EXCHID4_FLAG that signifies
> > > a TSM has made the client's lease available on that
> > > server
> >
> > Unfortunately, NFSv4.1 is not an extensible minor version.
> >
> > > 4. NFSv4.1 TSM cannot occur unless the client's sessions
> > > are also migrated
> >
> >
> > > Are there others?
> >
> > There are others but none come to mind right now.
> >
> > On Mon, Apr 17, 2017 at 2:45 PM, Chuck Lever <[e-mail]>
> wrote:
> >
> > > On Mar 9, 2017, at 2:00 PM, David Noveck <[e-mail]> wrote:
> > >
> > > > I'm not certain what you mean by "handled ... as it
> > > > would have been in 4.0". Server trunking discovery in
> > > > NFSv4.0 requires an elaborate dance. In NFSv4.1,
> > > > trunking discovery can be done by a single operation
> > > > (EXCHANGE_ID).
> > >
> > > What I meant was that, apart from the greater
> > > calories expended in v4.0 case, the result should have
> > > been the same, i.e. that there is no trunking.
> > >
> > > > I'm speculating that if the server hadn't asserted
> > > > CONFIRMED_R, this client could have recovered
> > > > correctly from the migration event. Is that what you
> > > > mean?
> > >
> > > It wasn't what I meant, but, now that you say it, I think is
> > > probably right.
> >
> > We've explored this mechanism a little more, and now we seem
> > to be hamstrung on the first paragraph of RFC 5661, Section
> > 18.35.3:
> >
> >    The client uses the EXCHANGE_ID operation to register a particular
> >    client owner with the server.  The client ID returned from this
> >    operation will be necessary for requests that create state on the
> >    server and will serve as a parent object to sessions created by the
> >    client.  In order to confirm the client ID it must first be used,
> >    along with the returned eir_sequenceid, as arguments to
> >    CREATE_SESSION.  If the flag EXCHGID4_FLAG_CONFIRMED_R is set in the
> >    result, eir_flags, then eir_sequenceid MUST be ignored, as it has no
> >    relevancy.
> >
> > The problem is the normative requirement in the last
> > sentence. If CONFIRMED_R is set, a client is required
> > to ignore the returned contrived slot sequence number.
> > [ This is likely the reason that clients react to this
> > flag by purging the lease and starting over ].
> >
> > Our first thought was to use the sequence number in the
> > migrated lease (in other words, the sequence number that
> > was established on the source server). A client knows
> > that sequence number by looking at the lease it has with
> > the source server.
> >
> > However, there's no guarantee that the client will not
> > perform additional CREATE_SESSION operations against
> > the source server between the time the source server
> > copies the migrating lease and the client starts trunking
> > discovery for the destination server.
> >
> > Another thought was to use a well-known constant, like
> > zero, when creating the first session on the destination
> > server.
> >
> > What sequence number should be used for the first
> > CREATE_SESSSION with the destination server? If it is
> > the copied sequence number, what can prevent mutation
> > of that sequence number while migration of that lease
> > is not yet complete?
> >
> > We've established that trunking discovery works correctly
> > when the destination server does not assert CONFIRMED_R.
> > This is because the client may use the contrived slot
> > sequence number provided by the destination server.
> >
> > If the destination server does not assert CONFIRMED_R,
> > then how does the client determine whether Transparent
> > State Migration has occurred? Can it simply start using
> > the open and lock state it has, and deal with BAD_STATEID
> > if the servers did not perform TSM?
> >
> >
> > Confirmation of the lease means more than just that it
> > is present on the queried server: it also means there is
> > now a cached CREATE_SESSION reply, and a known good
> > contrived slot sequence number. Neither of those is true
> > for a migrated lease when the client's sessions were not
> > also migrated.
> >
> > Thus we should consider that a migrated lease is really
> > not confirmed at all, but in some intermediate state.
> > A more ideal solution would be to create an additional
> > EXCHID4_FLAG that signifies the existence of a migrated
> > lease for the querying client.
> >
> >
> > In sum, here are some options we've considered:
> >
> > 1a. destination asserts CONF_R, but client uses the
> > returned contrived slot sequence anyway
> >
> > 1b. destination asserts CONF_R, and client uses a fixed
> > constant starting slot sequence
> >
> > 2. destination does not assert CONF_R, and client is
> > prepared for BAD_STATEID if TSM did not occur
> >
> > 3. destination asserts a new EXCHID4_FLAG that signifies
> > a TSM has made the client's lease available on that
> > server
> >
> > 4. NFSv4.1 TSM cannot occur unless the client's sessions
> > are also migrated
> >
> > Are there others?
> >
> > --
> > Chuck Lever
> >
> >
> >
> >
>
> --
> Chuck Lever
>
>
>
>