Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Trond Myklebust <[e-mail]>]

On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
> On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem <[e-mail]>
> wrote:
> > 
> > On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
> > <[e-mail]> wrote:
> > > 
> > > On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
> > > 
> > > 
> > > 
> > > On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
> > > <[e-mail]> wrote:
> > > 
> > > On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
> > > <[e-mail]> wrote:
> > > > 
> > > > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
> > > > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck
> > > > > <[e-mail]>
> > > > > wrote:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> > > > > > <[e-mail]> wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > On Apr 15, 2024, at 18:31, Rick Macklem
> > > > > > > <[e-mail]>
> > > > > > > wrote:
> > > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I am wondering if others have found this a pita and are
> > > > > > > interested in
> > > > > > > a new operation for NFSv4.2 to get around it.
> > > > > > > 
> > > > > > > Right now DelegReturn requires a CFH and delegation
> > > > > > > stateid.
> > > > > > > I do not know why the CFH is needed, since it seems that
> > > > > > > the
> > > > > > > delegation stateid should be sufficient?
> > > > > > > 
> > > > > > > This new operation would be the same as DelegReturn
> > > > > > > except that
> > > > > > > it would not require a CFH argument.
> > > > > > > 
> > > > > > > Why am I interested in this?
> > > > > > > - Lets assume a NFSv4.2 server that supports persistent
> > > > > > > FHs and
> > > > > > >  multiple hard links (as most extant servers do).
> > > > > > > - A client mounted to this server is removing a file for
> > > > > > > which it
> > > > > > > holds
> > > > > > >  a write delegation and a bunch of dirty data.
> > > > > > > Right now, the client must:
> > > > > > > - Write the dirty data back to the server.
> > > > > > > - DelegReturn
> > > > > > > - Remove
> > > > > > > because the client has no way of knowing if the file will
> > > > > > > be
> > > > > > > deleted
> > > > > > > by the Remove (or alternately is just removing one of
> > > > > > > several
> > > > > > > hard links).
> > > > > > > 
> > > > > > > 
> > > > > > > If a client is holding a delegation, it will presumably
> > > > > > > have done
> > > > > > > a GETATTR at some point while holding that delegation so
> > > > > > > should
> > > > > > > know how many links remain. While it is true that the
> > > > > > > unlink
> > > > > > > might invalidate the filehandle, that’s not a situation
> > > > > > > you are
> > > > > > > able to fix with the new operation.
> > > > > > > 
> > > > > > > With a DelegReturn_NoFH, the client could:
> > > > > > > - Do a compound RPC with
> > > > > > >  Remove
> > > > > > >  PutFH of file
> > > > > > > - if the above returns NFS_OK, the file still exists and
> > > > > > > the
> > > > > > > client
> > > > > > >  can continue to use the delegation, writing the dirty
> > > > > > > data back
> > > > > > >  to the server sometime later (or maybe never, if the
> > > > > > > file gets
> > > > > > >   deleted by a subsequent Remove).
> > > > > > > or
> > > > > > > - If the above returns NFS4ERR_STALE for the PutFH, the
> > > > > > > file
> > > > > > >  has been deleted. It can throw the dirty data away and
> > > > > > >  DelegReturn_NoFH the delegation.
> > > > > > > 
> > > > > > > A similar situation exists for Rename where the
> > > > > > > destination
> > > > > > > already exists.
> > > > > > > 
> > > > > > > I find this scenario happens frequently during software
> > > > > > > builds.
> > > > > > > 
> > > > > > > So, what do others think  w.r.t. this new operation? rick
> > > > > > > 
> > > > > > > 
> > > > > > > Why would you need to do a DELEGRETURN if the PUTFH
> > > > > > > returns
> > > > > > > NFS4ERR_STALE?
> > > > > > 
> > > > > > 
> > > > > > To make sure the client and server agree on the state of
> > > > > > things.
> > > > > > 
> > > > > > > If the server isn’t willing to just invalidate the
> > > > > > > delegation on
> > > > > > > the final unlink of the file,
> > > > > > 
> > > > > > I'm not sure about "unwilling",  but I am uncomfortable
> > > > > > with
> > > > > > unilateral stare changes like this, even if the spec
> > > > > > indicated that
> > > > > > needed to be done, which is harder to effect than an
> > > > > > extension such
> > > > > > as Rick is proposing
> > > > > > 
> > > > > >  > then it can and should recall the delegation (and any
> > > > > > pNFS
> > > > > > layouts that might be outstanding).
> > > > > > 
> > > > > > If it did recall it/them, then there is no way to return
> > > > > > them
> > > > > > without something like Rick's proposed extension.
> > > > > I assumed Trond meant "recall during the REMOVE" while the FH
> > > > > is
> > > > > still valid.
> > > 
> > > 
> > > I didn't get that.  Apparently, I was mistaken.
> > > 
> > > > 
> > > > Yes. This is a clarification of how the server can deal with
> > > > the
> > > > problem (if it thinks this is a problem) within the confines of
> > > > the
> > > > existing spec
> > > 
> > > 
> > > That depends on what the problem is.  Your proposal does address
> > > the problem of recalling a delegation when it is incapable of
> > > being returned.  It doesn't, at least alone, solve Rick's
> > > problem  which is that he wants to be able to know, at the time
> > > of recall, whether he can throw away pending writes, as opposed
> > > to flushing them to the server.
> > > 
> > > 
> > > Please re-read what I said above (repeated here):
> > > 
> > > > > > > If a client is holding a delegation, it will presumably
> > > > > > > have done
> > > > > > > a GETATTR at some point while holding that delegation so
> > > > > > > should
> > > > > > > know how many links remain. While it is true that the
> > > > > > > unlink
> > > > > > > might invalidate the filehandle, that’s not a situation
> > > > > > > you are
> > > > > > > able to fix with the new operation.
> > I did not get this. Now I see that in LINK (sec. 18.9.3) it says
> > that
> > delegations must
> > be CB_RECALL'd (I never noticed that and assumed that a LINK could
> > be done by another client when a delegation is held. My Bad;-)
> > 
> > So, yes, the client that holds a delegation can track numlinks for
> > the
> > file. I didn't realize that was the case.
> I now think there is a glitch in this plan...
> I cannot see anywhere in RFC7530 that specifies LINK should recall
> delegations, so I am not sure if a NFSv4.0 LINK done by another
> client
> could increase numlinks without the delegation being recalled.
> (I knew there was a reason the FreeBSD NFSv4 server did not recall
> delegations for LINK. I will patch it to do so for NFSv4.1/4.2.)
> 

It doesn't just change nlink. It also bumps the change attribute (and
the ctime), which by itself implies that it must recall delegations
according to RFC8881.
Specifically in section 10.2:

   *  The semantics of the file system are crucial in defining when
      delegation recall is required.  If a particular change within a
      specific implementation causes change to a file attribute, then
      delegation recall is required, whether that operation has been
      specifically listed as requiring delegation recall.  Again, what
      is critical is whether the guarantees provided by the delegation
      are being invalidated.

The Linux client interprets that to mean that LINK will recall
delegations.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[e-mail]