Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[David Noveck <[e-mail]>]

On Tue, Apr 16, 2024, 5:40 PM Rick Macklem <[e-mail]> wrote:

> On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust <[e-mail]>
> wrote:
> >
> > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
> > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck <[e-mail]>
> > > wrote:
> > > >
> > > >
> > > >
> > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> > > > <[e-mail]> wrote:
> > > > >
> > > > >
> > > > >
> > > > > On Apr 15, 2024, at 18:31, Rick Macklem <[e-mail]>
> > > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I am wondering if others have found this a pita and are
> > > > > interested in
> > > > > a new operation for NFSv4.2 to get around it.
> > > > >
> > > > > Right now DelegReturn requires a CFH and delegation stateid.
> > > > > I do not know why the CFH is needed, since it seems that the
> > > > > delegation stateid should be sufficient?
> > > > >
> > > > > This new operation would be the same as DelegReturn except that
> > > > > it would not require a CFH argument.
> > > > >
> > > > > Why am I interested in this?
> > > > > - Lets assume a NFSv4.2 server that supports persistent FHs and
> > > > >  multiple hard links (as most extant servers do).
> > > > > - A client mounted to this server is removing a file for which it
> > > > > holds
> > > > >  a write delegation and a bunch of dirty data.
> > > > > Right now, the client must:
> > > > > - Write the dirty data back to the server.
> > > > > - DelegReturn
> > > > > - Remove
> > > > > because the client has no way of knowing if the file will be
> > > > > deleted
> > > > > by the Remove (or alternately is just removing one of several
> > > > > hard links).
> > > > >
> > > > >
> > > > > If a client is holding a delegation, it will presumably have done
> > > > > a GETATTR at some point while holding that delegation so should
> > > > > know how many links remain. While it is true that the unlink
> > > > > might invalidate the filehandle, that’s not a situation you are
> > > > > able to fix with the new operation.
> > > > >
> > > > > With a DelegReturn_NoFH, the client could:
> > > > > - Do a compound RPC with
> > > > >  Remove
> > > > >  PutFH of file
> > > > > - if the above returns NFS_OK, the file still exists and the
> > > > > client
> > > > >  can continue to use the delegation, writing the dirty data back
> > > > >  to the server sometime later (or maybe never, if the file gets
> > > > >   deleted by a subsequent Remove).
> > > > > or
> > > > > - If the above returns NFS4ERR_STALE for the PutFH, the file
> > > > >  has been deleted. It can throw the dirty data away and
> > > > >  DelegReturn_NoFH the delegation.
> > > > >
> > > > > A similar situation exists for Rename where the destination
> > > > > already exists.
> > > > >
> > > > > I find this scenario happens frequently during software builds.
> > > > >
> > > > > So, what do others think  w.r.t. this new operation? rick
> > > > >
> > > > >
> > > > > Why would you need to do a DELEGRETURN if the PUTFH returns
> > > > > NFS4ERR_STALE?
> > > >
> > > >
> > > > To make sure the client and server agree on the state of things.
> > > >
> > > > > If the server isn’t willing to just invalidate the delegation on
> > > > > the final unlink of the file,
> > > >
> > > > I'm not sure about "unwilling",  but I am uncomfortable with
> > > > unilateral stare changes like this, even if the spec indicated that
> > > > needed to be done, which is harder to effect than an extension such
> > > > as Rick is proposing
> > > >
> > > >  > then it can and should recall the delegation (and any pNFS
> > > > layouts that might be outstanding).
> > > >
> > > > If it did recall it/them, then there is no way to return them
> > > > without something like Rick's proposed extension.
> > > I assumed Trond meant "recall during the REMOVE" while the FH is
> > > still valid.
>

I didn't get that.  Apparently, I was mistaken.

>
> > Yes. This is a clarification of how the server can deal with the
> > problem (if it thinks this is a problem) within the confines of the
> > existing spec
>

That depends on what the problem is.  Your proposal does address the
problem of recalling a delegation when it is incapable of being returned.
It doesn't, at least alone, solve Rick's problem  which is that he wants to
be able to know, at the time of recall, whether he can throw away pending
writes, as opposed to flushing them to the server.

I believe that, if the server acts as you suppose it might, we would have a
solution, if Rick ceased to depend on the legacy approach to remove
involving "silly rename".  If you used a more modern approach to remove in
which the fh invalidation is deferred to the last close, then the existence
of a delegation would prevent the invalidation until delegation return.
Since, until it was returned, there would be a potential open outstanding,
at delegation recall you could test the link count and throw away the
pending writes if the count is zero. If it isn't zero, you flush the
writes  before return.

Just fyi, I should be able to test the stuff I've posted here recently
> at the Bakeathon next week.
> - I have a client setup that tries to do Open/Claim_deleg_prev_fh
>   before ReclaimComplete.


That sounds like it would work OK.

It also does not DelegReturn a delegation
>   for a file it is about to Remove.
>

OK.

- I'll have the server configured so that it does not bother to CB_RECALL
>   a delegation issued to the same client as the one doing the Remove or
>   Rename.
>

In that case, I think the delegation will never be released and will hang
around as an unreturnable piece of state.  On the other hand, if the server
does what Trond proposes the delegation will be appropriately returned but
you will  be doing those writes unconditionally, which is a performance
issue, although not a major one.

>
> I'll report here if the testing finds anything interesting, rick
>

Thanks.



> >
> > --
> > Trond Myklebust
> > Linux NFS client maintainer, Hammerspace
> > [e-mail]
> >
> >
>