Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Rick Macklem <[e-mail]>]

On Thu, Apr 18, 2024 at 8:53 AM Trond Myklebust <[e-mail]> wrote:
>
> On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
> > On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem <[e-mail]>
> > wrote:
> > >
> > > On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
> > > <[e-mail]> wrote:
> > > >
> > > > On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
> > > >
> > > >
> > > >
> > > > On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
> > > > <[e-mail]> wrote:
> > > >
> > > > On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
> > > > <[e-mail]> wrote:
> > > > >
> > > > > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
> > > > > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck
> > > > > > <[e-mail]>
> > > > > > wrote:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> > > > > > > <[e-mail]> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Apr 15, 2024, at 18:31, Rick Macklem
> > > > > > > > <[e-mail]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I am wondering if others have found this a pita and are
> > > > > > > > interested in
> > > > > > > > a new operation for NFSv4.2 to get around it.
> > > > > > > >
> > > > > > > > Right now DelegReturn requires a CFH and delegation
> > > > > > > > stateid.
> > > > > > > > I do not know why the CFH is needed, since it seems that
> > > > > > > > the
> > > > > > > > delegation stateid should be sufficient?
> > > > > > > >
> > > > > > > > This new operation would be the same as DelegReturn
> > > > > > > > except that
> > > > > > > > it would not require a CFH argument.
> > > > > > > >
> > > > > > > > Why am I interested in this?
> > > > > > > > - Lets assume a NFSv4.2 server that supports persistent
> > > > > > > > FHs and
> > > > > > > >  multiple hard links (as most extant servers do).
> > > > > > > > - A client mounted to this server is removing a file for
> > > > > > > > which it
> > > > > > > > holds
> > > > > > > >  a write delegation and a bunch of dirty data.
> > > > > > > > Right now, the client must:
> > > > > > > > - Write the dirty data back to the server.
> > > > > > > > - DelegReturn
> > > > > > > > - Remove
> > > > > > > > because the client has no way of knowing if the file will
> > > > > > > > be
> > > > > > > > deleted
> > > > > > > > by the Remove (or alternately is just removing one of
> > > > > > > > several
> > > > > > > > hard links).
> > > > > > > >
> > > > > > > >
> > > > > > > > If a client is holding a delegation, it will presumably
> > > > > > > > have done
> > > > > > > > a GETATTR at some point while holding that delegation so
> > > > > > > > should
> > > > > > > > know how many links remain. While it is true that the
> > > > > > > > unlink
> > > > > > > > might invalidate the filehandle, that’s not a situation
> > > > > > > > you are
> > > > > > > > able to fix with the new operation.
> > > > > > > >
> > > > > > > > With a DelegReturn_NoFH, the client could:
> > > > > > > > - Do a compound RPC with
> > > > > > > >  Remove
> > > > > > > >  PutFH of file
> > > > > > > > - if the above returns NFS_OK, the file still exists and
> > > > > > > > the
> > > > > > > > client
> > > > > > > >  can continue to use the delegation, writing the dirty
> > > > > > > > data back
> > > > > > > >  to the server sometime later (or maybe never, if the
> > > > > > > > file gets
> > > > > > > >   deleted by a subsequent Remove).
> > > > > > > > or
> > > > > > > > - If the above returns NFS4ERR_STALE for the PutFH, the
> > > > > > > > file
> > > > > > > >  has been deleted. It can throw the dirty data away and
> > > > > > > >  DelegReturn_NoFH the delegation.
> > > > > > > >
> > > > > > > > A similar situation exists for Rename where the
> > > > > > > > destination
> > > > > > > > already exists.
> > > > > > > >
> > > > > > > > I find this scenario happens frequently during software
> > > > > > > > builds.
> > > > > > > >
> > > > > > > > So, what do others think  w.r.t. this new operation? rick
> > > > > > > >
> > > > > > > >
> > > > > > > > Why would you need to do a DELEGRETURN if the PUTFH
> > > > > > > > returns
> > > > > > > > NFS4ERR_STALE?
> > > > > > >
> > > > > > >
> > > > > > > To make sure the client and server agree on the state of
> > > > > > > things.
> > > > > > >
> > > > > > > > If the server isn’t willing to just invalidate the
> > > > > > > > delegation on
> > > > > > > > the final unlink of the file,
> > > > > > >
> > > > > > > I'm not sure about "unwilling",  but I am uncomfortable
> > > > > > > with
> > > > > > > unilateral stare changes like this, even if the spec
> > > > > > > indicated that
> > > > > > > needed to be done, which is harder to effect than an
> > > > > > > extension such
> > > > > > > as Rick is proposing
> > > > > > >
> > > > > > >  > then it can and should recall the delegation (and any
> > > > > > > pNFS
> > > > > > > layouts that might be outstanding).
> > > > > > >
> > > > > > > If it did recall it/them, then there is no way to return
> > > > > > > them
> > > > > > > without something like Rick's proposed extension.
> > > > > > I assumed Trond meant "recall during the REMOVE" while the FH
> > > > > > is
> > > > > > still valid.
> > > >
> > > >
> > > > I didn't get that.  Apparently, I was mistaken.
> > > >
> > > > >
> > > > > Yes. This is a clarification of how the server can deal with
> > > > > the
> > > > > problem (if it thinks this is a problem) within the confines of
> > > > > the
> > > > > existing spec
> > > >
> > > >
> > > > That depends on what the problem is.  Your proposal does address
> > > > the problem of recalling a delegation when it is incapable of
> > > > being returned.  It doesn't, at least alone, solve Rick's
> > > > problem  which is that he wants to be able to know, at the time
> > > > of recall, whether he can throw away pending writes, as opposed
> > > > to flushing them to the server.
> > > >
> > > >
> > > > Please re-read what I said above (repeated here):
> > > >
> > > > > > > > If a client is holding a delegation, it will presumably
> > > > > > > > have done
> > > > > > > > a GETATTR at some point while holding that delegation so
> > > > > > > > should
> > > > > > > > know how many links remain. While it is true that the
> > > > > > > > unlink
> > > > > > > > might invalidate the filehandle, that’s not a situation
> > > > > > > > you are
> > > > > > > > able to fix with the new operation.
> > > I did not get this. Now I see that in LINK (sec. 18.9.3) it says
> > > that
> > > delegations must
> > > be CB_RECALL'd (I never noticed that and assumed that a LINK could
> > > be done by another client when a delegation is held. My Bad;-)
> > >
> > > So, yes, the client that holds a delegation can track numlinks for
> > > the
> > > file. I didn't realize that was the case.
> > I now think there is a glitch in this plan...
> > I cannot see anywhere in RFC7530 that specifies LINK should recall
> > delegations, so I am not sure if a NFSv4.0 LINK done by another
> > client
> > could increase numlinks without the delegation being recalled.
> > (I knew there was a reason the FreeBSD NFSv4 server did not recall
> > delegations for LINK. I will patch it to do so for NFSv4.1/4.2.)
> >
>
> It doesn't just change nlink. It also bumps the change attribute (and
> the ctime), which by itself implies that it must recall delegations
> according to RFC8881.
> Specifically in section 10.2:
>
>    *  The semantics of the file system are crucial in defining when
>       delegation recall is required.  If a particular change within a
>       specific implementation causes change to a file attribute, then
>       delegation recall is required, whether that operation has been
>       specifically listed as requiring delegation recall.  Again, what
>       is critical is whether the guarantees provided by the delegation
>       are being invalidated.
>
> The Linux client interprets that to mean that LINK will recall
> delegations.
Yes, I agree, for NFSv4.1/4.2.
(And admit the FreeBSD NFSv4.1/4.2 needs to be fixed to do this.)

But what about NFSv4.0?
I'm looking, but cannot find similar words in RFC7530.
(Sec. 10.4.4. deals with Recall  of an Open Delegation, but it
does not mention LINK and it does not mention "any operation
that modifies the attributes. The same words appear to be in
RFC3530, except under Sec. 9.4.4.) I do find mention that
holders of delegations can either not modify attributes or
only the attributes related to writing for write delegation.
However, I cannot see any mention of clients that do not
hold delegations.

Maybe you are arguing that it should have been obvious for
NFSv4.0, but it was not so for me (and, therefore, maybe other NFSv4.0
implementers?). There is also the case of NFSv3 LINK RPCs.

Although it would be nice to assume, I'm not convinced that
extant NFS servers will all provide the guarantee that numlinks will
not be changed (increased via LINK) when the client holds
a delegation.
(If the client could be sure the NFS server was a "pure NFSv4.1/4.2
server and did not support earlier NFS versions, I would agree with
you.)
I just think that, since RFC7530 Sec. 10.4.4. makes no mention of
LINK, it is risky to assume all NFSv4.0 server implementations recall
delegations when a LINK is done. (In particular, since the NFSv4.0
cannot know which client is performing the LINK.)

rick

>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [e-mail]
>
>