Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Rick Macklem <[e-mail]>]

On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem <[e-mail]> wrote:
>
> On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust <[e-mail]> wrote:
> >
> > On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
> >
> >
> >
> > On Tue, Apr 16, 2024, 5:40 PM Rick Macklem <[e-mail]> wrote:
> >
> > On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust <[e-mail]> wrote:
> > >
> > > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
> > > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck <[e-mail]>
> > > > wrote:
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> > > > > <[e-mail]> wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Apr 15, 2024, at 18:31, Rick Macklem <[e-mail]>
> > > > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am wondering if others have found this a pita and are
> > > > > > interested in
> > > > > > a new operation for NFSv4.2 to get around it.
> > > > > >
> > > > > > Right now DelegReturn requires a CFH and delegation stateid.
> > > > > > I do not know why the CFH is needed, since it seems that the
> > > > > > delegation stateid should be sufficient?
> > > > > >
> > > > > > This new operation would be the same as DelegReturn except that
> > > > > > it would not require a CFH argument.
> > > > > >
> > > > > > Why am I interested in this?
> > > > > > - Lets assume a NFSv4.2 server that supports persistent FHs and
> > > > > >  multiple hard links (as most extant servers do).
> > > > > > - A client mounted to this server is removing a file for which it
> > > > > > holds
> > > > > >  a write delegation and a bunch of dirty data.
> > > > > > Right now, the client must:
> > > > > > - Write the dirty data back to the server.
> > > > > > - DelegReturn
> > > > > > - Remove
> > > > > > because the client has no way of knowing if the file will be
> > > > > > deleted
> > > > > > by the Remove (or alternately is just removing one of several
> > > > > > hard links).
> > > > > >
> > > > > >
> > > > > > If a client is holding a delegation, it will presumably have done
> > > > > > a GETATTR at some point while holding that delegation so should
> > > > > > know how many links remain. While it is true that the unlink
> > > > > > might invalidate the filehandle, that’s not a situation you are
> > > > > > able to fix with the new operation.
> > > > > >
> > > > > > With a DelegReturn_NoFH, the client could:
> > > > > > - Do a compound RPC with
> > > > > >  Remove
> > > > > >  PutFH of file
> > > > > > - if the above returns NFS_OK, the file still exists and the
> > > > > > client
> > > > > >  can continue to use the delegation, writing the dirty data back
> > > > > >  to the server sometime later (or maybe never, if the file gets
> > > > > >   deleted by a subsequent Remove).
> > > > > > or
> > > > > > - If the above returns NFS4ERR_STALE for the PutFH, the file
> > > > > >  has been deleted. It can throw the dirty data away and
> > > > > >  DelegReturn_NoFH the delegation.
> > > > > >
> > > > > > A similar situation exists for Rename where the destination
> > > > > > already exists.
> > > > > >
> > > > > > I find this scenario happens frequently during software builds.
> > > > > >
> > > > > > So, what do others think  w.r.t. this new operation? rick
> > > > > >
> > > > > >
> > > > > > Why would you need to do a DELEGRETURN if the PUTFH returns
> > > > > > NFS4ERR_STALE?
> > > > >
> > > > >
> > > > > To make sure the client and server agree on the state of things.
> > > > >
> > > > > > If the server isn’t willing to just invalidate the delegation on
> > > > > > the final unlink of the file,
> > > > >
> > > > > I'm not sure about "unwilling",  but I am uncomfortable with
> > > > > unilateral stare changes like this, even if the spec indicated that
> > > > > needed to be done, which is harder to effect than an extension such
> > > > > as Rick is proposing
> > > > >
> > > > >  > then it can and should recall the delegation (and any pNFS
> > > > > layouts that might be outstanding).
> > > > >
> > > > > If it did recall it/them, then there is no way to return them
> > > > > without something like Rick's proposed extension.
> > > > I assumed Trond meant "recall during the REMOVE" while the FH is
> > > > still valid.
> >
> >
> > I didn't get that.  Apparently, I was mistaken.
> >
> > >
> > > Yes. This is a clarification of how the server can deal with the
> > > problem (if it thinks this is a problem) within the confines of the
> > > existing spec
> >
> >
> > That depends on what the problem is.  Your proposal does address the problem of recalling a delegation when it is incapable of being returned.  It doesn't, at least alone, solve Rick's problem  which is that he wants to be able to know, at the time of recall, whether he can throw away pending writes, as opposed to flushing them to the server.
> >
> >
> > Please re-read what I said above (repeated here):
> >
> > > > > > If a client is holding a delegation, it will presumably have done
> > > > > > a GETATTR at some point while holding that delegation so should
> > > > > > know how many links remain. While it is true that the unlink
> > > > > > might invalidate the filehandle, that’s not a situation you are
> > > > > > able to fix with the new operation.
> I did not get this. Now I see that in LINK (sec. 18.9.3) it says that
> delegations must
> be CB_RECALL'd (I never noticed that and assumed that a LINK could
> be done by another client when a delegation is held. My Bad;-)
>
> So, yes, the client that holds a delegation can track numlinks for the
> file. I didn't realize that was the case.
I now think there is a glitch in this plan...
I cannot see anywhere in RFC7530 that specifies LINK should recall
delegations, so I am not sure if a NFSv4.0 LINK done by another client
could increase numlinks without the delegation being recalled.
(I knew there was a reason the FreeBSD NFSv4 server did not recall
delegations for LINK. I will patch it to do so for NFSv4.1/4.2.)

rick

>
> >
> >
> > The point is that there are 2 problems here, and they are orthogonal to one another.
> >
> > One problem is the question of "is this the last link?", which the above addresses. Well.... Except that Rick cannot know for certain that the REMOVE will actually succeed. Until he has tried the operation, he can only hope...
> >
> > The second problem is the question of "is the filehandle now stale, and how do I return my delegation?", which the above does not address. For reasons best known to themselves, the authors of RFC3010, and subsequent NFSv4 specs decided that it is OK to have filehandles that are derived from an object pathname (e.g. RFC8881 section 4.2.1 and section 10.3.4), so presumably the REMOVE could end up invalidating the filehandle despite the file having several links remaining. It seems to me that such a server does need to either recall the delegation, or be prepared to administratively revoke it on invalidation of that filehandle.
>
> Isn't this covered by a persistent filehandle, which is what I specified
> in the original post (and have assumed throughout) and. maybe, support
> for numlinks? (It might be possible to construct a persistent FH using an
> object pathname, but only if multiple hard links are not supported, I think?)
>
> (To be honest, the FreeBSD NFSv4.1/4.2 client will be broken for non-persistent
> file handles and I have yet to hear reports of problems or done any work to fix
> it. It has never been clear to me what a client does when a volatile file handle
> becomes NFS4ERR_FHEXPIRED?)
>
> So, now that I know numlinks will not be changed by other clients
> while a delegation
> is held, I think that numlinks == 1 before a REMOVE will determine if
> the file will
> be deleted and I think that resolves this.
> (Assuming persistent file handles.)
>
> rick
>
> > The gentler approach would be to recall...
> >
> > --
> >
> > Trond Myklebust
> > Linux NFS client maintainer, Hammerspace
> > [e-mail]
> >
> >