Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Rick Macklem <[e-mail]>]

On Thu, Apr 18, 2024 at 6:24 PM Trond Myklebust <[e-mail]> wrote:
>
> On Thu, 2024-04-18 at 17:13 -0700, Rick Macklem wrote:
> > On Thu, Apr 18, 2024 at 4:49 PM Trond Myklebust
> > <[e-mail]> wrote:
> > >
> > > On Thu, 2024-04-18 at 13:25 -0700, Rick Macklem wrote:
> > > > On Thu, Apr 18, 2024 at 8:53 AM Trond Myklebust
> > > > <[e-mail]> wrote:
> > > > >
> > > > > On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
> > > > > > On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem
> > > > > > <[e-mail]>
> > > > > > wrote:
> > > > > > >
> > > > > > > On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
> > > > > > > <[e-mail]> wrote:
> > > > > > > >
> > > > > > > > On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
> > > > > > > > <[e-mail]> wrote:
> > > > > > > >
> > > > > > > > On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
> > > > > > > > <[e-mail]> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
> > > > > > > > > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck
> > > > > > > > > > <[e-mail]>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> > > > > > > > > > > <[e-mail]> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Apr 15, 2024, at 18:31, Rick Macklem
> > > > > > > > > > > > <[e-mail]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I am wondering if others have found this a pita
> > > > > > > > > > > > and
> > > > > > > > > > > > are
> > > > > > > > > > > > interested in
> > > > > > > > > > > > a new operation for NFSv4.2 to get around it.
> > > > > > > > > > > >
> > > > > > > > > > > > Right now DelegReturn requires a CFH and
> > > > > > > > > > > > delegation
> > > > > > > > > > > > stateid.
> > > > > > > > > > > > I do not know why the CFH is needed, since it
> > > > > > > > > > > > seems
> > > > > > > > > > > > that
> > > > > > > > > > > > the
> > > > > > > > > > > > delegation stateid should be sufficient?
> > > > > > > > > > > >
> > > > > > > > > > > > This new operation would be the same as
> > > > > > > > > > > > DelegReturn
> > > > > > > > > > > > except that
> > > > > > > > > > > > it would not require a CFH argument.
> > > > > > > > > > > >
> > > > > > > > > > > > Why am I interested in this?
> > > > > > > > > > > > - Lets assume a NFSv4.2 server that supports
> > > > > > > > > > > > persistent
> > > > > > > > > > > > FHs and
> > > > > > > > > > > >  multiple hard links (as most extant servers do).
> > > > > > > > > > > > - A client mounted to this server is removing a
> > > > > > > > > > > > file
> > > > > > > > > > > > for
> > > > > > > > > > > > which it
> > > > > > > > > > > > holds
> > > > > > > > > > > >  a write delegation and a bunch of dirty data.
> > > > > > > > > > > > Right now, the client must:
> > > > > > > > > > > > - Write the dirty data back to the server.
> > > > > > > > > > > > - DelegReturn
> > > > > > > > > > > > - Remove
> > > > > > > > > > > > because the client has no way of knowing if the
> > > > > > > > > > > > file
> > > > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > deleted
> > > > > > > > > > > > by the Remove (or alternately is just removing
> > > > > > > > > > > > one of
> > > > > > > > > > > > several
> > > > > > > > > > > > hard links).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > If a client is holding a delegation, it will
> > > > > > > > > > > > presumably
> > > > > > > > > > > > have done
> > > > > > > > > > > > a GETATTR at some point while holding that
> > > > > > > > > > > > delegation
> > > > > > > > > > > > so
> > > > > > > > > > > > should
> > > > > > > > > > > > know how many links remain. While it is true that
> > > > > > > > > > > > the
> > > > > > > > > > > > unlink
> > > > > > > > > > > > might invalidate the filehandle, that’s not a
> > > > > > > > > > > > situation
> > > > > > > > > > > > you are
> > > > > > > > > > > > able to fix with the new operation.
> > > > > > > > > > > >
> > > > > > > > > > > > With a DelegReturn_NoFH, the client could:
> > > > > > > > > > > > - Do a compound RPC with
> > > > > > > > > > > >  Remove
> > > > > > > > > > > >  PutFH of file
> > > > > > > > > > > > - if the above returns NFS_OK, the file still
> > > > > > > > > > > > exists
> > > > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > client
> > > > > > > > > > > >  can continue to use the delegation, writing the
> > > > > > > > > > > > dirty
> > > > > > > > > > > > data back
> > > > > > > > > > > >  to the server sometime later (or maybe never, if
> > > > > > > > > > > > the
> > > > > > > > > > > > file gets
> > > > > > > > > > > >   deleted by a subsequent Remove).
> > > > > > > > > > > > or
> > > > > > > > > > > > - If the above returns NFS4ERR_STALE for the
> > > > > > > > > > > > PutFH,
> > > > > > > > > > > > the
> > > > > > > > > > > > file
> > > > > > > > > > > >  has been deleted. It can throw the dirty data
> > > > > > > > > > > > away
> > > > > > > > > > > > and
> > > > > > > > > > > >  DelegReturn_NoFH the delegation.
> > > > > > > > > > > >
> > > > > > > > > > > > A similar situation exists for Rename where the
> > > > > > > > > > > > destination
> > > > > > > > > > > > already exists.
> > > > > > > > > > > >
> > > > > > > > > > > > I find this scenario happens frequently during
> > > > > > > > > > > > software
> > > > > > > > > > > > builds.
> > > > > > > > > > > >
> > > > > > > > > > > > So, what do others think  w.r.t. this new
> > > > > > > > > > > > operation?
> > > > > > > > > > > > rick
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Why would you need to do a DELEGRETURN if the
> > > > > > > > > > > > PUTFH
> > > > > > > > > > > > returns
> > > > > > > > > > > > NFS4ERR_STALE?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > To make sure the client and server agree on the
> > > > > > > > > > > state
> > > > > > > > > > > of
> > > > > > > > > > > things.
> > > > > > > > > > >
> > > > > > > > > > > > If the server isn’t willing to just invalidate
> > > > > > > > > > > > the
> > > > > > > > > > > > delegation on
> > > > > > > > > > > > the final unlink of the file,
> > > > > > > > > > >
> > > > > > > > > > > I'm not sure about "unwilling",  but I am
> > > > > > > > > > > uncomfortable
> > > > > > > > > > > with
> > > > > > > > > > > unilateral stare changes like this, even if the
> > > > > > > > > > > spec
> > > > > > > > > > > indicated that
> > > > > > > > > > > needed to be done, which is harder to effect than
> > > > > > > > > > > an
> > > > > > > > > > > extension such
> > > > > > > > > > > as Rick is proposing
> > > > > > > > > > >
> > > > > > > > > > >  > then it can and should recall the delegation
> > > > > > > > > > > (and
> > > > > > > > > > > any
> > > > > > > > > > > pNFS
> > > > > > > > > > > layouts that might be outstanding).
> > > > > > > > > > >
> > > > > > > > > > > If it did recall it/them, then there is no way to
> > > > > > > > > > > return
> > > > > > > > > > > them
> > > > > > > > > > > without something like Rick's proposed extension.
> > > > > > > > > > I assumed Trond meant "recall during the REMOVE"
> > > > > > > > > > while
> > > > > > > > > > the FH
> > > > > > > > > > is
> > > > > > > > > > still valid.
> > > > > > > >
> > > > > > > >
> > > > > > > > I didn't get that.  Apparently, I was mistaken.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes. This is a clarification of how the server can deal
> > > > > > > > > with
> > > > > > > > > the
> > > > > > > > > problem (if it thinks this is a problem) within the
> > > > > > > > > confines of
> > > > > > > > > the
> > > > > > > > > existing spec
> > > > > > > >
> > > > > > > >
> > > > > > > > That depends on what the problem is.  Your proposal does
> > > > > > > > address
> > > > > > > > the problem of recalling a delegation when it is
> > > > > > > > incapable of
> > > > > > > > being returned.  It doesn't, at least alone, solve Rick's
> > > > > > > > problem  which is that he wants to be able to know, at
> > > > > > > > the
> > > > > > > > time
> > > > > > > > of recall, whether he can throw away pending writes, as
> > > > > > > > opposed
> > > > > > > > to flushing them to the server.
> > > > > > > >
> > > > > > > >
> > > > > > > > Please re-read what I said above (repeated here):
> > > > > > > >
> > > > > > > > > > > > If a client is holding a delegation, it will
> > > > > > > > > > > > presumably
> > > > > > > > > > > > have done
> > > > > > > > > > > > a GETATTR at some point while holding that
> > > > > > > > > > > > delegation
> > > > > > > > > > > > so
> > > > > > > > > > > > should
> > > > > > > > > > > > know how many links remain. While it is true that
> > > > > > > > > > > > the
> > > > > > > > > > > > unlink
> > > > > > > > > > > > might invalidate the filehandle, that’s not a
> > > > > > > > > > > > situation
> > > > > > > > > > > > you are
> > > > > > > > > > > > able to fix with the new operation.
> > > > > > > I did not get this. Now I see that in LINK (sec. 18.9.3) it
> > > > > > > says
> > > > > > > that
> > > > > > > delegations must
> > > > > > > be CB_RECALL'd (I never noticed that and assumed that a
> > > > > > > LINK
> > > > > > > could
> > > > > > > be done by another client when a delegation is held. My
> > > > > > > Bad;-)
> > > > > > >
> > > > > > > So, yes, the client that holds a delegation can track
> > > > > > > numlinks
> > > > > > > for
> > > > > > > the
> > > > > > > file. I didn't realize that was the case.
> > > > > > I now think there is a glitch in this plan...
> > > > > > I cannot see anywhere in RFC7530 that specifies LINK should
> > > > > > recall
> > > > > > delegations, so I am not sure if a NFSv4.0 LINK done by
> > > > > > another
> > > > > > client
> > > > > > could increase numlinks without the delegation being
> > > > > > recalled.
> > > > > > (I knew there was a reason the FreeBSD NFSv4 server did not
> > > > > > recall
> > > > > > delegations for LINK. I will patch it to do so for
> > > > > > NFSv4.1/4.2.)
> > > > > >
> > > > >
> > > > > It doesn't just change nlink. It also bumps the change
> > > > > attribute
> > > > > (and
> > > > > the ctime), which by itself implies that it must recall
> > > > > delegations
> > > > > according to RFC8881.
> > > > > Specifically in section 10.2:
> > > > >
> > > > >    *  The semantics of the file system are crucial in defining
> > > > > when
> > > > >       delegation recall is required.  If a particular change
> > > > > within
> > > > > a
> > > > >       specific implementation causes change to a file
> > > > > attribute,
> > > > > then
> > > > >       delegation recall is required, whether that operation has
> > > > > been
> > > > >       specifically listed as requiring delegation recall.
> > > > > Again,
> > > > > what
> > > > >       is critical is whether the guarantees provided by the
> > > > > delegation
> > > > >       are being invalidated.
> > > > >
> > > > > The Linux client interprets that to mean that LINK will recall
> > > > > delegations.
> > > > Yes, I agree, for NFSv4.1/4.2.
> > > > (And admit the FreeBSD NFSv4.1/4.2 needs to be fixed to do this.)
> > > >
> > > > But what about NFSv4.0?
> > > > I'm looking, but cannot find similar words in RFC7530.
> > > > (Sec. 10.4.4. deals with Recall  of an Open Delegation, but it
> > > > does not mention LINK and it does not mention "any operation
> > > > that modifies the attributes. The same words appear to be in
> > > > RFC3530, except under Sec. 9.4.4.) I do find mention that
> > > > holders of delegations can either not modify attributes or
> > > > only the attributes related to writing for write delegation.
> > > > However, I cannot see any mention of clients that do not
> > > > hold delegations.
> > > >
> > > > Maybe you are arguing that it should have been obvious for
> > > > NFSv4.0, but it was not so for me (and, therefore, maybe other
> > > > NFSv4.0
> > > > implementers?). There is also the case of NFSv3 LINK RPCs.
> > > >
> > > > Although it would be nice to assume, I'm not convinced that
> > > > extant NFS servers will all provide the guarantee that numlinks
> > > > will
> > > > not be changed (increased via LINK) when the client holds
> > > > a delegation.
> > > > (If the client could be sure the NFS server was a "pure
> > > > NFSv4.1/4.2
> > > > server and did not support earlier NFS versions, I would agree
> > > > with
> > > > you.)
> > > > I just think that, since RFC7530 Sec. 10.4.4. makes no mention of
> > > > LINK, it is risky to assume all NFSv4.0 server implementations
> > > > recall
> > > > delegations when a LINK is done. (In particular, since the
> > > > NFSv4.0
> > > > cannot know which client is performing the LINK.)
> > > >
> > >
> > > What is the motivation for continuing to optimise NFSv4.0? While it
> > > would be nice to allow it to benefit too, that problem is not one
> > > that
> > > keeps me awake at night.
> > I certainly am not interested in improving NFSv4.0.
> >
> > What I was trying to say is...
> > When an NFSv4.0 client does a LINK, does the NFSv4.[0,1,2] server
> > recall all delegations for the file?
> > If not, then the numlinks attribute could get increased without the
> > delegation being recalled. And if that is the case, then the
> > NFSv4.1/4.2
> > client cannot trust its value of numlinks to indicate if the file
> > being
> > REMOVE'd will be deleted on the server.
> >
>
> From RFC7830, section 10.4:
>
>    When a single client holds an OPEN_DELEGATE_READ delegation, it is
>    assured that no other client may modify the contents or attributes of
>    the file.  If more than one client holds an OPEN_DELEGATE_READ
>    delegation, then the contents and attributes of that file are not
>    allowed to change.  When a client has an OPEN_DELEGATE_WRITE
>    delegation, it may modify the file data since no other client will be
>    accessing the file's data.  The client holding an OPEN_DELEGATE_WRITE
>    delegation may only affect file attributes that are intimately
>    connected with the file data: size, time_modify, and change.
>
> Clearly, when reading the above, the assumption should be that a read
> delegation MUST be recalled or revoked if the file attributes change.
>
> I do not find anything that says the same thing for a write delegation.
> I suspect that the authors did intend to imply that, but I'm not
> finding anything that explicitly says that a client that holds a write
> delegation can expect attributes not to change due to the action of
> another client. It can only expect that the data contents (and hence
> the size attribute) won't change, and it is not allowed to modify
> attributes other than those "intimately connected with the file
> data"...
For a simple example:
- Client A has a NFSv4.2 mount against the server
  - It holds a write delegation for "foo".
- Client B has a NFSv4.0 mount against the server
  - It does a "ln foo bar" (ie. creates a second hard link to the
      same file called "bar")
Does the server recall the write delegation that Client A holds
as a part of the "ln foo bar"?

I suspect you are right, in that the original intent would be "yes",
but since LINK is not mentioned at the beginning of RFC7530 Sec. 10.4.4
I, for one, missed it and need to patch the FreeBSD NFSv4 server.

As a result, I am not comfortable assuming numlinks won't change
while the client holds a write delegation for the file at this time.

I will try and test this simple example at the Bakeathon next week, rick

>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [e-mail]
>
>