Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Rick Macklem <[e-mail]>]

On Thu, Apr 18, 2024 at 4:49 PM Trond Myklebust <[e-mail]> wrote:
>
> On Thu, 2024-04-18 at 13:25 -0700, Rick Macklem wrote:
> > On Thu, Apr 18, 2024 at 8:53 AM Trond Myklebust
> > <[e-mail]> wrote:
> > >
> > > On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
> > > > On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem
> > > > <[e-mail]>
> > > > wrote:
> > > > >
> > > > > On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
> > > > > <[e-mail]> wrote:
> > > > > >
> > > > > > On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
> > > > > > <[e-mail]> wrote:
> > > > > >
> > > > > > On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
> > > > > > <[e-mail]> wrote:
> > > > > > >
> > > > > > > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
> > > > > > > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck
> > > > > > > > <[e-mail]>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> > > > > > > > > <[e-mail]> wrote:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Apr 15, 2024, at 18:31, Rick Macklem
> > > > > > > > > > <[e-mail]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I am wondering if others have found this a pita and
> > > > > > > > > > are
> > > > > > > > > > interested in
> > > > > > > > > > a new operation for NFSv4.2 to get around it.
> > > > > > > > > >
> > > > > > > > > > Right now DelegReturn requires a CFH and delegation
> > > > > > > > > > stateid.
> > > > > > > > > > I do not know why the CFH is needed, since it seems
> > > > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > delegation stateid should be sufficient?
> > > > > > > > > >
> > > > > > > > > > This new operation would be the same as DelegReturn
> > > > > > > > > > except that
> > > > > > > > > > it would not require a CFH argument.
> > > > > > > > > >
> > > > > > > > > > Why am I interested in this?
> > > > > > > > > > - Lets assume a NFSv4.2 server that supports
> > > > > > > > > > persistent
> > > > > > > > > > FHs and
> > > > > > > > > >  multiple hard links (as most extant servers do).
> > > > > > > > > > - A client mounted to this server is removing a file
> > > > > > > > > > for
> > > > > > > > > > which it
> > > > > > > > > > holds
> > > > > > > > > >  a write delegation and a bunch of dirty data.
> > > > > > > > > > Right now, the client must:
> > > > > > > > > > - Write the dirty data back to the server.
> > > > > > > > > > - DelegReturn
> > > > > > > > > > - Remove
> > > > > > > > > > because the client has no way of knowing if the file
> > > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > deleted
> > > > > > > > > > by the Remove (or alternately is just removing one of
> > > > > > > > > > several
> > > > > > > > > > hard links).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If a client is holding a delegation, it will
> > > > > > > > > > presumably
> > > > > > > > > > have done
> > > > > > > > > > a GETATTR at some point while holding that delegation
> > > > > > > > > > so
> > > > > > > > > > should
> > > > > > > > > > know how many links remain. While it is true that the
> > > > > > > > > > unlink
> > > > > > > > > > might invalidate the filehandle, that’s not a
> > > > > > > > > > situation
> > > > > > > > > > you are
> > > > > > > > > > able to fix with the new operation.
> > > > > > > > > >
> > > > > > > > > > With a DelegReturn_NoFH, the client could:
> > > > > > > > > > - Do a compound RPC with
> > > > > > > > > >  Remove
> > > > > > > > > >  PutFH of file
> > > > > > > > > > - if the above returns NFS_OK, the file still exists
> > > > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > client
> > > > > > > > > >  can continue to use the delegation, writing the
> > > > > > > > > > dirty
> > > > > > > > > > data back
> > > > > > > > > >  to the server sometime later (or maybe never, if the
> > > > > > > > > > file gets
> > > > > > > > > >   deleted by a subsequent Remove).
> > > > > > > > > > or
> > > > > > > > > > - If the above returns NFS4ERR_STALE for the PutFH,
> > > > > > > > > > the
> > > > > > > > > > file
> > > > > > > > > >  has been deleted. It can throw the dirty data away
> > > > > > > > > > and
> > > > > > > > > >  DelegReturn_NoFH the delegation.
> > > > > > > > > >
> > > > > > > > > > A similar situation exists for Rename where the
> > > > > > > > > > destination
> > > > > > > > > > already exists.
> > > > > > > > > >
> > > > > > > > > > I find this scenario happens frequently during
> > > > > > > > > > software
> > > > > > > > > > builds.
> > > > > > > > > >
> > > > > > > > > > So, what do others think  w.r.t. this new operation?
> > > > > > > > > > rick
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Why would you need to do a DELEGRETURN if the PUTFH
> > > > > > > > > > returns
> > > > > > > > > > NFS4ERR_STALE?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > To make sure the client and server agree on the state
> > > > > > > > > of
> > > > > > > > > things.
> > > > > > > > >
> > > > > > > > > > If the server isn’t willing to just invalidate the
> > > > > > > > > > delegation on
> > > > > > > > > > the final unlink of the file,
> > > > > > > > >
> > > > > > > > > I'm not sure about "unwilling",  but I am uncomfortable
> > > > > > > > > with
> > > > > > > > > unilateral stare changes like this, even if the spec
> > > > > > > > > indicated that
> > > > > > > > > needed to be done, which is harder to effect than an
> > > > > > > > > extension such
> > > > > > > > > as Rick is proposing
> > > > > > > > >
> > > > > > > > >  > then it can and should recall the delegation (and
> > > > > > > > > any
> > > > > > > > > pNFS
> > > > > > > > > layouts that might be outstanding).
> > > > > > > > >
> > > > > > > > > If it did recall it/them, then there is no way to
> > > > > > > > > return
> > > > > > > > > them
> > > > > > > > > without something like Rick's proposed extension.
> > > > > > > > I assumed Trond meant "recall during the REMOVE" while
> > > > > > > > the FH
> > > > > > > > is
> > > > > > > > still valid.
> > > > > >
> > > > > >
> > > > > > I didn't get that.  Apparently, I was mistaken.
> > > > > >
> > > > > > >
> > > > > > > Yes. This is a clarification of how the server can deal
> > > > > > > with
> > > > > > > the
> > > > > > > problem (if it thinks this is a problem) within the
> > > > > > > confines of
> > > > > > > the
> > > > > > > existing spec
> > > > > >
> > > > > >
> > > > > > That depends on what the problem is.  Your proposal does
> > > > > > address
> > > > > > the problem of recalling a delegation when it is incapable of
> > > > > > being returned.  It doesn't, at least alone, solve Rick's
> > > > > > problem  which is that he wants to be able to know, at the
> > > > > > time
> > > > > > of recall, whether he can throw away pending writes, as
> > > > > > opposed
> > > > > > to flushing them to the server.
> > > > > >
> > > > > >
> > > > > > Please re-read what I said above (repeated here):
> > > > > >
> > > > > > > > > > If a client is holding a delegation, it will
> > > > > > > > > > presumably
> > > > > > > > > > have done
> > > > > > > > > > a GETATTR at some point while holding that delegation
> > > > > > > > > > so
> > > > > > > > > > should
> > > > > > > > > > know how many links remain. While it is true that the
> > > > > > > > > > unlink
> > > > > > > > > > might invalidate the filehandle, that’s not a
> > > > > > > > > > situation
> > > > > > > > > > you are
> > > > > > > > > > able to fix with the new operation.
> > > > > I did not get this. Now I see that in LINK (sec. 18.9.3) it
> > > > > says
> > > > > that
> > > > > delegations must
> > > > > be CB_RECALL'd (I never noticed that and assumed that a LINK
> > > > > could
> > > > > be done by another client when a delegation is held. My Bad;-)
> > > > >
> > > > > So, yes, the client that holds a delegation can track numlinks
> > > > > for
> > > > > the
> > > > > file. I didn't realize that was the case.
> > > > I now think there is a glitch in this plan...
> > > > I cannot see anywhere in RFC7530 that specifies LINK should
> > > > recall
> > > > delegations, so I am not sure if a NFSv4.0 LINK done by another
> > > > client
> > > > could increase numlinks without the delegation being recalled.
> > > > (I knew there was a reason the FreeBSD NFSv4 server did not
> > > > recall
> > > > delegations for LINK. I will patch it to do so for NFSv4.1/4.2.)
> > > >
> > >
> > > It doesn't just change nlink. It also bumps the change attribute
> > > (and
> > > the ctime), which by itself implies that it must recall delegations
> > > according to RFC8881.
> > > Specifically in section 10.2:
> > >
> > >    *  The semantics of the file system are crucial in defining when
> > >       delegation recall is required.  If a particular change within
> > > a
> > >       specific implementation causes change to a file attribute,
> > > then
> > >       delegation recall is required, whether that operation has
> > > been
> > >       specifically listed as requiring delegation recall.  Again,
> > > what
> > >       is critical is whether the guarantees provided by the
> > > delegation
> > >       are being invalidated.
> > >
> > > The Linux client interprets that to mean that LINK will recall
> > > delegations.
> > Yes, I agree, for NFSv4.1/4.2.
> > (And admit the FreeBSD NFSv4.1/4.2 needs to be fixed to do this.)
> >
> > But what about NFSv4.0?
> > I'm looking, but cannot find similar words in RFC7530.
> > (Sec. 10.4.4. deals with Recall  of an Open Delegation, but it
> > does not mention LINK and it does not mention "any operation
> > that modifies the attributes. The same words appear to be in
> > RFC3530, except under Sec. 9.4.4.) I do find mention that
> > holders of delegations can either not modify attributes or
> > only the attributes related to writing for write delegation.
> > However, I cannot see any mention of clients that do not
> > hold delegations.
> >
> > Maybe you are arguing that it should have been obvious for
> > NFSv4.0, but it was not so for me (and, therefore, maybe other
> > NFSv4.0
> > implementers?). There is also the case of NFSv3 LINK RPCs.
> >
> > Although it would be nice to assume, I'm not convinced that
> > extant NFS servers will all provide the guarantee that numlinks will
> > not be changed (increased via LINK) when the client holds
> > a delegation.
> > (If the client could be sure the NFS server was a "pure NFSv4.1/4.2
> > server and did not support earlier NFS versions, I would agree with
> > you.)
> > I just think that, since RFC7530 Sec. 10.4.4. makes no mention of
> > LINK, it is risky to assume all NFSv4.0 server implementations recall
> > delegations when a LINK is done. (In particular, since the NFSv4.0
> > cannot know which client is performing the LINK.)
> >
>
> What is the motivation for continuing to optimise NFSv4.0? While it
> would be nice to allow it to benefit too, that problem is not one that
> keeps me awake at night.
I certainly am not interested in improving NFSv4.0.

What I was trying to say is...
When an NFSv4.0 client does a LINK, does the NFSv4.[0,1,2] server
recall all delegations for the file?
If not, then the numlinks attribute could get increased without the
delegation being recalled. And if that is the case, then the NFSv4.1/4.2
client cannot trust its value of numlinks to indicate if the file being
REMOVE'd will be deleted on the server.

rick

>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [e-mail]
>
>