Re: [nfsv4] Preventing an NFSv4.1 client from destroying a migrated lease after TSM

[Chuck Lever <[e-mail]>]

> On Mar 9, 2017, at 5:52 AM, David Noveck <[e-mail]> wrote:
> 
> > One of Oracle's engineers, Xuan Qi, has been experimenting
> > with NFSv4.1 Transparent State Migration. The
> > implementations under test are a Solaris-based prototype
> > NFS server, and a recent vintage migration-capable Linux
> > NFS client. Xuan observed the following:
> 
> > + While a simple workload is running, there is an FS
> >  migration event from server A to server B. Transparent
> >   State Migration is done as part of this migration. The
> >   session that had been established between the client and
> >   server A is NOT migrated.
> 
> > + The client has never interacted with server B before the
> >   migration event. 
> 
> In that case, there is no possibility that the clientid used to 
> access server A will be merged with one used to access
> client B.  So it is likely that the clientid used to access 
> client A, together with the stateid's used to access the
> migrated fs are now usable on B.

I'm told that if the client has already mounted and used
an FS that exists on server B before the migration, lease
merging appears to work as expected.


> > After the client retrieves fs_locations
> > from server A, it begins trunking discovery with server B.
> 
> I assume that it compares the server owners it got
> from each server. Correct?

It compares server owners to look for session trunking.


> > + In the initial EXCHANGE_ID request with server B, the
> >  client presents the same client owner and boot verifier
> >  it previously presented to server A.
> 
> Makes sense.
> 
> > + In the initial EXCHANGE_ID reply, server B reports the
> >  presence of the migrated lease by returning the same
> >  clientID the client got from server A and asserting
> >   EXCHGID4_FLAG_CONFIRMED_R.
> 
> > + The client thus believes this lease is an old one it
> >  forgot to clean up. 
> 
> I don't see why it should believe that.  It is aware of the
> migration from A to B and thus the possibility that 
> a clientid has been migrated.  The fact that it is the
> same id makes it even more likely that the lease is 
> migrated, although a coincidence is possible.
> 
> > It purges the migrated lease by
> >  sending a second EXCHANGE_ID with the same client owner
> >  but an arbitrary boot verifier. 
> 
> OUCH!!. That sounds to me like a client bug.

In the sense that it hasn't been explained to the client
how to deal with NFSv4.1 migration recovery, this is
probably a "missing functionality" bug.


> >Server B returns a
> >  second distinct client ID.
> 
> Which is basically ignored I take it.
> 
> > + The client issues a CREATE_SESSION with the second
> >  client ID. This client ID and session is not used for
> >  subsequent operations.
> 
> OK.
> 
> > + The client issues a DESTROY_SESSION and DESTROY_CLIENTID
> >  to server A, using the original session and client ID.
> >  How polite!
> 
> Would it still do that if there was another non-migrated fs on A using
> the same session?  I hope not.
> 
> > + The client then establishes a fresh lease with server
> >  B by sending a third EXCHANGE_ID with the same client
> >  owner and the original boot verifier. Server B returns
> >  a third distinct client ID. 
> 
> This clientid will have no stateids associated with it.
> 
> > The client then uses this
> >   new client ID for a CREATE_SESSION. This session is used
> >   to issue RECLAIM_COMPLETE with no OPEN state recovery.
> 
> Not clear why it is doing this, but it is not relevant.  The
> damage was done when the existing migrated lock state 
> was thrown away.

From this we can assume that the client's state recovery
logic is aware that it did just purge its lease.

However, since the client had OPEN state and just purged
the lease on the server, why didn't client send any OPEN
operations? State recovery is obviously needed here, but
wasn't done. That's probably a client bug.


> .> + The first WRITE operation to server B fails with
>  > NFS4ERR_BAD_STATEID. 
> 
> Not surprising, since the session is associated with a client for 
> which there are no valid stateids.
> 
> I don't understand why it thinks that stateid was valid, since it previously 
> invalidated it by doing an EXCHANGE_I'D with a new boot verifier.

The client's write path assumes that migration/state
recovery has worked properly, and resumes the workload
without checking.


> > The client performs no-grace
> > stateid recovery, and the workload continues normally.
> 
> Sigh!

> > + Both servers report the same server scope, but different
> >  server owners. The scope string is
> >
> >     "Solaris NFSv4.1 Server Scope"
> 
> > RFC 5661 and draft-ietf-nfsv4-migration-issues are not
> > clear about how servers and clients are supposed to
> > interact after Transparent State Migration.
> 
> probably not.  The basic assumption is that there needs
> to be an update to bring rfc5661 into line with rfc7931, but
> that hasn't been done yet.
> 
> With regard to this specific situation, it doesn't seem that anything 4.1-specific 
> has gone on.  In particular, I haven't noticed any particular consquences due 
> to sessions or server name or server scope.  There has been an instance of 
> RECLAIM_COMPLETE but by that point, the damage had been done. It looks 
> to me that if this had been handled essentially as it would have been in 4.0, 
> things would have worked better. Maybe I've missed something.

I'm not certain what you mean by "handled ... as it
would have been in 4.0". Server trunking discovery in
NFSv4.0 requires an elaborate dance. In NFSv4.1,
trunking discovery can be done by a single operation
(EXCHANGE_ID).

I'm speculating that if the server hadn't asserted
CONFIRMED_R, this client could have recovered
correctly from the migration event. Is that what you
mean?


> >  For instance:
> 
> > 1. Since the client has had no previous contact with
> > server B, should the migrated lease be considered
> > confirmed or not confirmed on server B?
> 
> I'd say that if is confirmed on A and migrated to B,
> then it should be confirmed on B as well.

I ask because that's not clear from RFC 5661, IMO.

5661 implies that a lease is confirmed by a
subsequent CREATE_SESSION. In this case, the client
has done no prior CREATE_SESSION against server B.
No session exists at all when the client first
contacts that server, because no session was
migrated.

I think you are saying that the CREATE_SESSION on
server A confirms the lease until it expires or
is destroyed, no matter where it resides.
Confirmation of a lease is independent of the
particular server that handles that confirmation.


> > 2. How should the client determine whether or not
> > Transparent State Migration has occurred?
> 
> I think one should follow the approach in rfc7931.  If the stateids
> are  valid on the new server, then transparent state 
> migfation has occurred.

If CONFIRMED_R is correctly asserted when the lease
is migrated, couldn't the client use the setting of
that flag to determine that TSM had been attempted,
and then look at the SEQ4_STATUS flags to determine
if any problems had occurred?


> > 3. If the session was migrated (which is not the case
> > in the current example), how should the client determine
> > it must perform BIND_CONN_TO_SESSION instead of
> > CREATE_SESSION? (I guess just try BC2S first?)
> 
> I think your guess is right.

This would be only when CONFIRMED_R was set and the
client knows it is doing migration recovery?

Also, session migration promises EOS during migration
events. But I'm not sure what happens if a client is
still sending operations to the source server after
the session is moved (for example, the server would
have to return DELAY or MOVED, and the state of the
session there would be different than the state of
the session on the destination, wouldn't it). Don't
we have a situation similar to lock state mutations
after migration in NFSv4.0?

Well, this is speculative, since we don't have an
implementation that migrates sessions, so let's put
it aside for now.


> >4. If TSM is detected, the client performs no state
> > recovery. Should it avoid sending 
> > RECLAIM_COMPLETE in this case?
> 
> I don't see why it should send RECLAIM_COMPLETE
> in a situation in which there is no indication that
>  state was lost.  Do we understand why it was sent in this case?

Probably because a final RECLAIM_COMPLETE is always
part of the Linux client's state recovery process.
But, state was lost in this case, when the client
purges its lease.


--
Chuck Lever