Re: [nfsv4] Preventing an NFSv4.1 client from destroying a migrated lease after TSM

[David Noveck <[e-mail]>]

> I'm not certain what you mean by "handled ... as it
> would have been in 4.0". Server trunking discovery in
> NFSv4.0 requires an elaborate dance. In NFSv4.1,
> trunking discovery can be done by a single operation
> (EXCHANGE_ID).

What I meant was that, apart from the greater
calories expended in v4.0 case, the result should have
been the same, i.e. that there is no trunking.

> I'm speculating that if the server hadn't asserted
> CONFIRMED_R, this client could have recovered
> correctly from the migration event. Is that what you
> mean?

It wasn't what I meant, but, now that you say it, I think is
probably right.

> I ask because that's not clear from RFC 5661, IMO.

it's one of a number of things that is not clear in RFC5661.

> 5661 implies that a lease is confirmed by a
> subsequent CREATE_SESSION.

I think it explicitly says that in the EXCHANGE_ID case; the
CREATE_SESSION is needed to to confirm creation of the
clientid.

Creation of a clientid through migration, without use of
EXCHANGE_ID was one of the things that were not dreamt
of in the RFC5661 philosophy.

> In this case, the client
> has done no prior CREATE_SESSION against server B.

or EXCHANGE_ID.

> No session exists at all when the client first
> contacts that server, because no session was
> migrated.

I can see how that puts things in an ambiguous
state.

I don't see rfc5661 getting fixed anytime soon, but
I can mention this as an issue to be resolved when
I produce migration-issues-12.

> I think you are saying that the CREATE_SESSION on
> server A confirms the lease until it expires or
> is destroyed, no matter where it resides.
> Confirmation of a lease is independent of the
> particular server that handles that confirmation.

That's what I think.

> If CONFIRMED_R is correctly asserted when the lease
> is migrated, couldn't the client use the setting of
> that flag to determine that TSM had been attempted,

If it was confirmed on A and the client sees it confirmed on
B, the only way it could have gotten to B is via migration.

> and then look at the SEQ4_STATUS flags to determine
> if any problems had occurred?

I hadn't thought of that but it sounds workable.

> > >how should the client determine
> > > it must perform BIND_CONN_TO_SESSION instead of
> > > CREATE_SESSION? (I guess just try BC2S first?)
> >
> > I think your guess is right.

>  This would be only when CONFIRMED_R was set and the
> client knows it is doing migration recovery?

I think I should mention this in migration-issues-12.

> Also, session migration promises EOS during migration
> events. But I'm not sure what happens if a client is
> still sending operations to the source server after
> the session is moved (for example, the server would
> have to return DELAY or MOVED, and the state of the
> session there would be different than the state of
> the session on the destination, wouldn't it). Don't
> we have a situation similar to lock state mutations
> after migration in NFSv4.0?

> Well, this is speculative, since we don't have an
> implementation that migrates sessions, so let's put
> it aside for now.

We don't have to solve this, but it should be mentioned
as an open issue in migration-issues-12.

On Thu, Mar 9, 2017 at 11:02 AM, Chuck Lever <[e-mail]> wrote:

>
> > On Mar 9, 2017, at 5:52 AM, David Noveck <[e-mail]> wrote:
> >
> > > One of Oracle's engineers, Xuan Qi, has been experimenting
> > > with NFSv4.1 Transparent State Migration. The
> > > implementations under test are a Solaris-based prototype
> > > NFS server, and a recent vintage migration-capable Linux
> > > NFS client. Xuan observed the following:
> >
> > > + While a simple workload is running, there is an FS
> > >  migration event from server A to server B. Transparent
> > >   State Migration is done as part of this migration. The
> > >   session that had been established between the client and
> > >   server A is NOT migrated.
> >
> > > + The client has never interacted with server B before the
> > >   migration event.
> >
> > In that case, there is no possibility that the clientid used to
> > access server A will be merged with one used to access
> > client B.  So it is likely that the clientid used to access
> > client A, together with the stateid's used to access the
> > migrated fs are now usable on B.
>
> I'm told that if the client has already mounted and used
> an FS that exists on server B before the migration, lease
> merging appears to work as expected.
>
>
> > > After the client retrieves fs_locations
> > > from server A, it begins trunking discovery with server B.
> >
> > I assume that it compares the server owners it got
> > from each server. Correct?
>
> It compares server owners to look for session trunking.
>
>
> > > + In the initial EXCHANGE_ID request with server B, the
> > >  client presents the same client owner and boot verifier
> > >  it previously presented to server A.
> >
> > Makes sense.
> >
> > > + In the initial EXCHANGE_ID reply, server B reports the
> > >  presence of the migrated lease by returning the same
> > >  clientID the client got from server A and asserting
> > >   EXCHGID4_FLAG_CONFIRMED_R.
> >
> > > + The client thus believes this lease is an old one it
> > >  forgot to clean up.
> >
> > I don't see why it should believe that.  It is aware of the
> > migration from A to B and thus the possibility that
> > a clientid has been migrated.  The fact that it is the
> > same id makes it even more likely that the lease is
> > migrated, although a coincidence is possible.
> >
> > > It purges the migrated lease by
> > >  sending a second EXCHANGE_ID with the same client owner
> > >  but an arbitrary boot verifier.
> >
> > OUCH!!. That sounds to me like a client bug.
>
> In the sense that it hasn't been explained to the client
> how to deal with NFSv4.1 migration recovery, this is
> probably a "missing functionality" bug.
>
>
> > >Server B returns a
> > >  second distinct client ID.
> >
> > Which is basically ignored I take it.
> >
> > > + The client issues a CREATE_SESSION with the second
> > >  client ID. This client ID and session is not used for
> > >  subsequent operations.
> >
> > OK.
> >
> > > + The client issues a DESTROY_SESSION and DESTROY_CLIENTID
> > >  to server A, using the original session and client ID.
> > >  How polite!
> >
> > Would it still do that if there was another non-migrated fs on A using
> > the same session?  I hope not.
> >
> > > + The client then establishes a fresh lease with server
> > >  B by sending a third EXCHANGE_ID with the same client
> > >  owner and the original boot verifier. Server B returns
> > >  a third distinct client ID.
> >
> > This clientid will have no stateids associated with it.
> >
> > > The client then uses this
> > >   new client ID for a CREATE_SESSION. This session is used
> > >   to issue RECLAIM_COMPLETE with no OPEN state recovery.
> >
> > Not clear why it is doing this, but it is not relevant.  The
> > damage was done when the existing migrated lock state
> > was thrown away.
>
> From this we can assume that the client's state recovery
> logic is aware that it did just purge its lease.
>
> However, since the client had OPEN state and just purged
> the lease on the server, why didn't client send any OPEN
> operations? State recovery is obviously needed here, but
> wasn't done. That's probably a client bug.
>
>
> > .> + The first WRITE operation to server B fails with
> >  > NFS4ERR_BAD_STATEID.
> >
> > Not surprising, since the session is associated with a client for
> > which there are no valid stateids.
> >
> > I don't understand why it thinks that stateid was valid, since it
> previously
> > invalidated it by doing an EXCHANGE_I'D with a new boot verifier.
>
> The client's write path assumes that migration/state
> recovery has worked properly, and resumes the workload
> without checking.
>
>
> > > The client performs no-grace
> > > stateid recovery, and the workload continues normally.
> >
> > Sigh!
>
> > > + Both servers report the same server scope, but different
> > >  server owners. The scope string is
> > >
> > >     "Solaris NFSv4.1 Server Scope"
> >
> > > RFC 5661 and draft-ietf-nfsv4-migration-issues are not
> > > clear about how servers and clients are supposed to
> > > interact after Transparent State Migration.
> >
> > probably not.  The basic assumption is that there needs
> > to be an update to bring rfc5661 into line with rfc7931, but
> > that hasn't been done yet.
> >
> > With regard to this specific situation, it doesn't seem that anything
> 4.1-specific
> > has gone on.  In particular, I haven't noticed any particular
> consquences due
> > to sessions or server name or server scope.  There has been an instance
> of
> > RECLAIM_COMPLETE but by that point, the damage had been done. It looks
> > to me that if this had been handled essentially as it would have been in
> 4.0,
> > things would have worked better. Maybe I've missed something.
>
> I'm not certain what you mean by "handled ... as it
> would have been in 4.0". Server trunking discovery in
> NFSv4.0 requires an elaborate dance. In NFSv4.1,
> trunking discovery can be done by a single operation
> (EXCHANGE_ID).
>
> I'm speculating that if the server hadn't asserted
> CONFIRMED_R, this client could have recovered
> correctly from the migration event. Is that what you
> mean?
>
>
> > >  For instance:
> >
> > > 1. Since the client has had no previous contact with
> > > server B, should the migrated lease be considered
> > > confirmed or not confirmed on server B?
> >
> > I'd say that if is confirmed on A and migrated to B,
> > then it should be confirmed on B as well.
>
> I ask because that's not clear from RFC 5661, IMO.
>
> 5661 implies that a lease is confirmed by a
> subsequent CREATE_SESSION. In this case, the client
> has done no prior CREATE_SESSION against server B.
> No session exists at all when the client first
> contacts that server, because no session was
> migrated.
>
> I think you are saying that the CREATE_SESSION on
> server A confirms the lease until it expires or
> is destroyed, no matter where it resides.
> Confirmation of a lease is independent of the
> particular server that handles that confirmation.
>
>
> > > 2. How should the client determine whether or not
> > > Transparent State Migration has occurred?
> >
> > I think one should follow the approach in rfc7931.  If the stateids
> > are  valid on the new server, then transparent state
> > migfation has occurred.
>
> If CONFIRMED_R is correctly asserted when the lease
> is migrated, couldn't the client use the setting of
> that flag to determine that TSM had been attempted,
> and then look at the SEQ4_STATUS flags to determine
> if any problems had occurred?
>
>
> > > 3. If the session was migrated (which is not the case
> > > in the current example), how should the client determine
> > > it must perform BIND_CONN_TO_SESSION instead of
> > > CREATE_SESSION? (I guess just try BC2S first?)
> >
> > I think your guess is right.
>
> This would be only when CONFIRMED_R was set and the
> client knows it is doing migration recovery?
>
> Also, session migration promises EOS during migration
> events. But I'm not sure what happens if a client is
> still sending operations to the source server after
> the session is moved (for example, the server would
> have to return DELAY or MOVED, and the state of the
> session there would be different than the state of
> the session on the destination, wouldn't it). Don't
> we have a situation similar to lock state mutations
> after migration in NFSv4.0?
>
> Well, this is speculative, since we don't have an
> implementation that migrates sessions, so let's put
> it aside for now.
>
>
> > >4. If TSM is detected, the client performs no state
> > > recovery. Should it avoid sending
> > > RECLAIM_COMPLETE in this case?
> >
> > I don't see why it should send RECLAIM_COMPLETE
> > in a situation in which there is no indication that
> >  state was lost.  Do we understand why it was sent in this case?
>
> Probably because a final RECLAIM_COMPLETE is always
> part of the Linux client's state recovery process.
> But, state was lost in this case, when the client
> purges its lease.
>
>
> --
> Chuck Lever
>
>
>
>