Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Rick Macklem <[e-mail]>]

On Thu, Apr 18, 2024 at 6:36 PM Rick Macklem <[e-mail]> wrote:
>
> On Thu, Apr 18, 2024 at 6:24 PM Trond Myklebust <[e-mail]> wrote:
> >
> > On Thu, 2024-04-18 at 17:13 -0700, Rick Macklem wrote:
> > > On Thu, Apr 18, 2024 at 4:49 PM Trond Myklebust
> > > <[e-mail]> wrote:
> > > >
> > > > On Thu, 2024-04-18 at 13:25 -0700, Rick Macklem wrote:
> > > > > On Thu, Apr 18, 2024 at 8:53 AM Trond Myklebust
> > > > > <[e-mail]> wrote:
> > > > > >
> > > > > > On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
> > > > > > > On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem
> > > > > > > <[e-mail]>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
> > > > > > > > <[e-mail]> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
> > > > > > > > > <[e-mail]> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
> > > > > > > > > <[e-mail]> wrote:
> > > > > > > > > >
> > > > > > > > > > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
> > > > > > > > > > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck
> > > > > > > > > > > <[e-mail]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> > > > > > > > > > > > <[e-mail]> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Apr 15, 2024, at 18:31, Rick Macklem
> > > > > > > > > > > > > <[e-mail]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am wondering if others have found this a pita
> > > > > > > > > > > > > and
> > > > > > > > > > > > > are
> > > > > > > > > > > > > interested in
> > > > > > > > > > > > > a new operation for NFSv4.2 to get around it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Right now DelegReturn requires a CFH and
> > > > > > > > > > > > > delegation
> > > > > > > > > > > > > stateid.
> > > > > > > > > > > > > I do not know why the CFH is needed, since it
> > > > > > > > > > > > > seems
> > > > > > > > > > > > > that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > delegation stateid should be sufficient?
> > > > > > > > > > > > >
> > > > > > > > > > > > > This new operation would be the same as
> > > > > > > > > > > > > DelegReturn
> > > > > > > > > > > > > except that
> > > > > > > > > > > > > it would not require a CFH argument.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Why am I interested in this?
> > > > > > > > > > > > > - Lets assume a NFSv4.2 server that supports
> > > > > > > > > > > > > persistent
> > > > > > > > > > > > > FHs and
> > > > > > > > > > > > >  multiple hard links (as most extant servers do).
> > > > > > > > > > > > > - A client mounted to this server is removing a
> > > > > > > > > > > > > file
> > > > > > > > > > > > > for
> > > > > > > > > > > > > which it
> > > > > > > > > > > > > holds
> > > > > > > > > > > > >  a write delegation and a bunch of dirty data.
> > > > > > > > > > > > > Right now, the client must:
> > > > > > > > > > > > > - Write the dirty data back to the server.
> > > > > > > > > > > > > - DelegReturn
> > > > > > > > > > > > > - Remove
> > > > > > > > > > > > > because the client has no way of knowing if the
> > > > > > > > > > > > > file
> > > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > deleted
> > > > > > > > > > > > > by the Remove (or alternately is just removing
> > > > > > > > > > > > > one of
> > > > > > > > > > > > > several
> > > > > > > > > > > > > hard links).
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > If a client is holding a delegation, it will
> > > > > > > > > > > > > presumably
> > > > > > > > > > > > > have done
> > > > > > > > > > > > > a GETATTR at some point while holding that
> > > > > > > > > > > > > delegation
> > > > > > > > > > > > > so
> > > > > > > > > > > > > should
> > > > > > > > > > > > > know how many links remain. While it is true that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > unlink
> > > > > > > > > > > > > might invalidate the filehandle, that’s not a
> > > > > > > > > > > > > situation
> > > > > > > > > > > > > you are
> > > > > > > > > > > > > able to fix with the new operation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > With a DelegReturn_NoFH, the client could:
> > > > > > > > > > > > > - Do a compound RPC with
> > > > > > > > > > > > >  Remove
> > > > > > > > > > > > >  PutFH of file
> > > > > > > > > > > > > - if the above returns NFS_OK, the file still
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > client
> > > > > > > > > > > > >  can continue to use the delegation, writing the
> > > > > > > > > > > > > dirty
> > > > > > > > > > > > > data back
> > > > > > > > > > > > >  to the server sometime later (or maybe never, if
> > > > > > > > > > > > > the
> > > > > > > > > > > > > file gets
> > > > > > > > > > > > >   deleted by a subsequent Remove).
> > > > > > > > > > > > > or
> > > > > > > > > > > > > - If the above returns NFS4ERR_STALE for the
> > > > > > > > > > > > > PutFH,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > file
> > > > > > > > > > > > >  has been deleted. It can throw the dirty data
> > > > > > > > > > > > > away
> > > > > > > > > > > > > and
> > > > > > > > > > > > >  DelegReturn_NoFH the delegation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A similar situation exists for Rename where the
> > > > > > > > > > > > > destination
> > > > > > > > > > > > > already exists.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I find this scenario happens frequently during
> > > > > > > > > > > > > software
> > > > > > > > > > > > > builds.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So, what do others think  w.r.t. this new
> > > > > > > > > > > > > operation?
> > > > > > > > > > > > > rick
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Why would you need to do a DELEGRETURN if the
> > > > > > > > > > > > > PUTFH
> > > > > > > > > > > > > returns
> > > > > > > > > > > > > NFS4ERR_STALE?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > To make sure the client and server agree on the
> > > > > > > > > > > > state
> > > > > > > > > > > > of
> > > > > > > > > > > > things.
> > > > > > > > > > > >
> > > > > > > > > > > > > If the server isn’t willing to just invalidate
> > > > > > > > > > > > > the
> > > > > > > > > > > > > delegation on
> > > > > > > > > > > > > the final unlink of the file,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm not sure about "unwilling",  but I am
> > > > > > > > > > > > uncomfortable
> > > > > > > > > > > > with
> > > > > > > > > > > > unilateral stare changes like this, even if the
> > > > > > > > > > > > spec
> > > > > > > > > > > > indicated that
> > > > > > > > > > > > needed to be done, which is harder to effect than
> > > > > > > > > > > > an
> > > > > > > > > > > > extension such
> > > > > > > > > > > > as Rick is proposing
> > > > > > > > > > > >
> > > > > > > > > > > >  > then it can and should recall the delegation
> > > > > > > > > > > > (and
> > > > > > > > > > > > any
> > > > > > > > > > > > pNFS
> > > > > > > > > > > > layouts that might be outstanding).
> > > > > > > > > > > >
> > > > > > > > > > > > If it did recall it/them, then there is no way to
> > > > > > > > > > > > return
> > > > > > > > > > > > them
> > > > > > > > > > > > without something like Rick's proposed extension.
> > > > > > > > > > > I assumed Trond meant "recall during the REMOVE"
> > > > > > > > > > > while
> > > > > > > > > > > the FH
> > > > > > > > > > > is
> > > > > > > > > > > still valid.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I didn't get that.  Apparently, I was mistaken.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yes. This is a clarification of how the server can deal
> > > > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > problem (if it thinks this is a problem) within the
> > > > > > > > > > confines of
> > > > > > > > > > the
> > > > > > > > > > existing spec
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > That depends on what the problem is.  Your proposal does
> > > > > > > > > address
> > > > > > > > > the problem of recalling a delegation when it is
> > > > > > > > > incapable of
> > > > > > > > > being returned.  It doesn't, at least alone, solve Rick's
> > > > > > > > > problem  which is that he wants to be able to know, at
> > > > > > > > > the
> > > > > > > > > time
> > > > > > > > > of recall, whether he can throw away pending writes, as
> > > > > > > > > opposed
> > > > > > > > > to flushing them to the server.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Please re-read what I said above (repeated here):
> > > > > > > > >
> > > > > > > > > > > > > If a client is holding a delegation, it will
> > > > > > > > > > > > > presumably
> > > > > > > > > > > > > have done
> > > > > > > > > > > > > a GETATTR at some point while holding that
> > > > > > > > > > > > > delegation
> > > > > > > > > > > > > so
> > > > > > > > > > > > > should
> > > > > > > > > > > > > know how many links remain. While it is true that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > unlink
> > > > > > > > > > > > > might invalidate the filehandle, that’s not a
> > > > > > > > > > > > > situation
> > > > > > > > > > > > > you are
> > > > > > > > > > > > > able to fix with the new operation.
> > > > > > > > I did not get this. Now I see that in LINK (sec. 18.9.3) it
> > > > > > > > says
> > > > > > > > that
> > > > > > > > delegations must
> > > > > > > > be CB_RECALL'd (I never noticed that and assumed that a
> > > > > > > > LINK
> > > > > > > > could
> > > > > > > > be done by another client when a delegation is held. My
> > > > > > > > Bad;-)
> > > > > > > >
> > > > > > > > So, yes, the client that holds a delegation can track
> > > > > > > > numlinks
> > > > > > > > for
> > > > > > > > the
> > > > > > > > file. I didn't realize that was the case.
> > > > > > > I now think there is a glitch in this plan...
> > > > > > > I cannot see anywhere in RFC7530 that specifies LINK should
> > > > > > > recall
> > > > > > > delegations, so I am not sure if a NFSv4.0 LINK done by
> > > > > > > another
> > > > > > > client
> > > > > > > could increase numlinks without the delegation being
> > > > > > > recalled.
> > > > > > > (I knew there was a reason the FreeBSD NFSv4 server did not
> > > > > > > recall
> > > > > > > delegations for LINK. I will patch it to do so for
> > > > > > > NFSv4.1/4.2.)
> > > > > > >
> > > > > >
> > > > > > It doesn't just change nlink. It also bumps the change
> > > > > > attribute
> > > > > > (and
> > > > > > the ctime), which by itself implies that it must recall
> > > > > > delegations
> > > > > > according to RFC8881.
> > > > > > Specifically in section 10.2:
> > > > > >
> > > > > >    *  The semantics of the file system are crucial in defining
> > > > > > when
> > > > > >       delegation recall is required.  If a particular change
> > > > > > within
> > > > > > a
> > > > > >       specific implementation causes change to a file
> > > > > > attribute,
> > > > > > then
> > > > > >       delegation recall is required, whether that operation has
> > > > > > been
> > > > > >       specifically listed as requiring delegation recall.
> > > > > > Again,
> > > > > > what
> > > > > >       is critical is whether the guarantees provided by the
> > > > > > delegation
> > > > > >       are being invalidated.
> > > > > >
> > > > > > The Linux client interprets that to mean that LINK will recall
> > > > > > delegations.
> > > > > Yes, I agree, for NFSv4.1/4.2.
> > > > > (And admit the FreeBSD NFSv4.1/4.2 needs to be fixed to do this.)
> > > > >
> > > > > But what about NFSv4.0?
> > > > > I'm looking, but cannot find similar words in RFC7530.
> > > > > (Sec. 10.4.4. deals with Recall  of an Open Delegation, but it
> > > > > does not mention LINK and it does not mention "any operation
> > > > > that modifies the attributes. The same words appear to be in
> > > > > RFC3530, except under Sec. 9.4.4.) I do find mention that
> > > > > holders of delegations can either not modify attributes or
> > > > > only the attributes related to writing for write delegation.
> > > > > However, I cannot see any mention of clients that do not
> > > > > hold delegations.
> > > > >
> > > > > Maybe you are arguing that it should have been obvious for
> > > > > NFSv4.0, but it was not so for me (and, therefore, maybe other
> > > > > NFSv4.0
> > > > > implementers?). There is also the case of NFSv3 LINK RPCs.
> > > > >
> > > > > Although it would be nice to assume, I'm not convinced that
> > > > > extant NFS servers will all provide the guarantee that numlinks
> > > > > will
> > > > > not be changed (increased via LINK) when the client holds
> > > > > a delegation.
> > > > > (If the client could be sure the NFS server was a "pure
> > > > > NFSv4.1/4.2
> > > > > server and did not support earlier NFS versions, I would agree
> > > > > with
> > > > > you.)
> > > > > I just think that, since RFC7530 Sec. 10.4.4. makes no mention of
> > > > > LINK, it is risky to assume all NFSv4.0 server implementations
> > > > > recall
> > > > > delegations when a LINK is done. (In particular, since the
> > > > > NFSv4.0
> > > > > cannot know which client is performing the LINK.)
> > > > >
> > > >
> > > > What is the motivation for continuing to optimise NFSv4.0? While it
> > > > would be nice to allow it to benefit too, that problem is not one
> > > > that
> > > > keeps me awake at night.
> > > I certainly am not interested in improving NFSv4.0.
> > >
> > > What I was trying to say is...
> > > When an NFSv4.0 client does a LINK, does the NFSv4.[0,1,2] server
> > > recall all delegations for the file?
> > > If not, then the numlinks attribute could get increased without the
> > > delegation being recalled. And if that is the case, then the
> > > NFSv4.1/4.2
> > > client cannot trust its value of numlinks to indicate if the file
> > > being
> > > REMOVE'd will be deleted on the server.
> > >
> >
> > From RFC7830, section 10.4:
> >
> >    When a single client holds an OPEN_DELEGATE_READ delegation, it is
> >    assured that no other client may modify the contents or attributes of
> >    the file.  If more than one client holds an OPEN_DELEGATE_READ
> >    delegation, then the contents and attributes of that file are not
> >    allowed to change.  When a client has an OPEN_DELEGATE_WRITE
> >    delegation, it may modify the file data since no other client will be
> >    accessing the file's data.  The client holding an OPEN_DELEGATE_WRITE
> >    delegation may only affect file attributes that are intimately
> >    connected with the file data: size, time_modify, and change.
> >
> > Clearly, when reading the above, the assumption should be that a read
> > delegation MUST be recalled or revoked if the file attributes change.
> >
> > I do not find anything that says the same thing for a write delegation.
> > I suspect that the authors did intend to imply that, but I'm not
> > finding anything that explicitly says that a client that holds a write
> > delegation can expect attributes not to change due to the action of
> > another client. It can only expect that the data contents (and hence
> > the size attribute) won't change, and it is not allowed to modify
> > attributes other than those "intimately connected with the file
> > data"...
> For a simple example:
> - Client A has a NFSv4.2 mount against the server
>   - It holds a write delegation for "foo".
> - Client B has a NFSv4.0 mount against the server
>   - It does a "ln foo bar" (ie. creates a second hard link to the
>       same file called "bar")
> Does the server recall the write delegation that Client A holds
> as a part of the "ln foo bar"?
>
> I suspect you are right, in that the original intent would be "yes",
> but since LINK is not mentioned at the beginning of RFC7530 Sec. 10.4.4
> I, for one, missed it and need to patch the FreeBSD NFSv4 server.
>
> As a result, I am not comfortable assuming numlinks won't change
> while the client holds a write delegation for the file at this time.
>
> I will try and test this simple example at the Bakeathon next week, rick
At the Bakeathon last week, the non-FreeBSD NFSv4 servers I tested did
recall the delegation when another client did a "ln foo bar".
One of the servers recalled the delegation even when the same client with
a NFSv4.2 mount did the "ln foo bar".

I am still not sure if I am willing to "trust" the value of numlinks for all
servers, but using numlinks would have worked for the ones I tested.

Btw, I also found that all the servers I tested recalled the delegation
for Remove/Rename when done by the same client as the one holding
the delegation for a NFSv4.2 mount. This is what RFC8881 says,
although the words are verbatim from RFC3530 and I think the
requirement is "an oversight" for NFSv4.1/4.2, since the server does
know if the client doing the Remove/Rename is the same client.

Anyhow, I no longer think a DelegReturn_NoFH would be useful, rick

>
> >
> > --
> > Trond Myklebust
> > Linux NFS client maintainer, Hammerspace
> > [e-mail]
> >
> >