Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Rick Macklem <[e-mail]>]

On Thu, May 2, 2024 at 12:28 PM David Noveck <[e-mail]> wrote:
>
> Assuming that the quotes around the word "oversight" do not indicate that you think that the treatment of this issue in RFC8881 is OK, I will look at fixing this in the next draft of rfc5661bis.  In the case of an NFSv4.[12] write delegation, it is possible to  clearly indicate that recall is necessary when  the LINK is done by anyone other than the holder of the delegation including  NFSv4.0 and NFSv3 clients and local file accessors. Although direct discussion of theseblast actors are out-of-scope, I think we  can make it clear that others doing the LINK who cannot be holders of NFSv4.[12] delegations,  need the delegation to be returned before proceeding.

The "an oversight" comment was referring to RFC8881 Sec. 10.4.4, which
does not mention LINK
and does not state "sent by another client" for REMOVE and RENAME. (I,
personally, cannot see
why the delegation recall is required when the REMOVE or RENAME is
done by the same client
as the one holding the open delegation.  However, the text is clear
and that is what extant
servers seem to have implemented. As such, I cannot say that this
should be changed, although
I would have no problem with it being changed.)

I do think that not listing LINK in Sec. 10.4.4 is an oversight.

rick

>
> I'll send out proposed text soon.  If people are OK with it.  It will wind up in rfc5661bis-05.
>
>
>
>
> On Sun, Apr 28, 2024, 11:28 AM Rick Macklem <[e-mail]> wrote:
>>
>> On Thu, Apr 18, 2024 at 6:36 PM Rick Macklem <[e-mail]> wrote:
>> >
>> > On Thu, Apr 18, 2024 at 6:24 PM Trond Myklebust <[e-mail]> wrote:
>> > >
>> > > On Thu, 2024-04-18 at 17:13 -0700, Rick Macklem wrote:
>> > > > On Thu, Apr 18, 2024 at 4:49 PM Trond Myklebust
>> > > > <[e-mail]> wrote:
>> > > > >
>> > > > > On Thu, 2024-04-18 at 13:25 -0700, Rick Macklem wrote:
>> > > > > > On Thu, Apr 18, 2024 at 8:53 AM Trond Myklebust
>> > > > > > <[e-mail]> wrote:
>> > > > > > >
>> > > > > > > On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
>> > > > > > > > On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem
>> > > > > > > > <[e-mail]>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
>> > > > > > > > > <[e-mail]> wrote:
>> > > > > > > > > >
>> > > > > > > > > > On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
>> > > > > > > > > > <[e-mail]> wrote:
>> > > > > > > > > >
>> > > > > > > > > > On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
>> > > > > > > > > > <[e-mail]> wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
>> > > > > > > > > > > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck
>> > > > > > > > > > > > <[e-mail]>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
>> > > > > > > > > > > > > <[e-mail]> wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Apr 15, 2024, at 18:31, Rick Macklem
>> > > > > > > > > > > > > > <[e-mail]>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I am wondering if others have found this a pita
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > interested in
>> > > > > > > > > > > > > > a new operation for NFSv4.2 to get around it.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Right now DelegReturn requires a CFH and
>> > > > > > > > > > > > > > delegation
>> > > > > > > > > > > > > > stateid.
>> > > > > > > > > > > > > > I do not know why the CFH is needed, since it
>> > > > > > > > > > > > > > seems
>> > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > delegation stateid should be sufficient?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This new operation would be the same as
>> > > > > > > > > > > > > > DelegReturn
>> > > > > > > > > > > > > > except that
>> > > > > > > > > > > > > > it would not require a CFH argument.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Why am I interested in this?
>> > > > > > > > > > > > > > - Lets assume a NFSv4.2 server that supports
>> > > > > > > > > > > > > > persistent
>> > > > > > > > > > > > > > FHs and
>> > > > > > > > > > > > > >  multiple hard links (as most extant servers do).
>> > > > > > > > > > > > > > - A client mounted to this server is removing a
>> > > > > > > > > > > > > > file
>> > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > which it
>> > > > > > > > > > > > > > holds
>> > > > > > > > > > > > > >  a write delegation and a bunch of dirty data.
>> > > > > > > > > > > > > > Right now, the client must:
>> > > > > > > > > > > > > > - Write the dirty data back to the server.
>> > > > > > > > > > > > > > - DelegReturn
>> > > > > > > > > > > > > > - Remove
>> > > > > > > > > > > > > > because the client has no way of knowing if the
>> > > > > > > > > > > > > > file
>> > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > deleted
>> > > > > > > > > > > > > > by the Remove (or alternately is just removing
>> > > > > > > > > > > > > > one of
>> > > > > > > > > > > > > > several
>> > > > > > > > > > > > > > hard links).
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > If a client is holding a delegation, it will
>> > > > > > > > > > > > > > presumably
>> > > > > > > > > > > > > > have done
>> > > > > > > > > > > > > > a GETATTR at some point while holding that
>> > > > > > > > > > > > > > delegation
>> > > > > > > > > > > > > > so
>> > > > > > > > > > > > > > should
>> > > > > > > > > > > > > > know how many links remain. While it is true that
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > unlink
>> > > > > > > > > > > > > > might invalidate the filehandle, that’s not a
>> > > > > > > > > > > > > > situation
>> > > > > > > > > > > > > > you are
>> > > > > > > > > > > > > > able to fix with the new operation.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > With a DelegReturn_NoFH, the client could:
>> > > > > > > > > > > > > > - Do a compound RPC with
>> > > > > > > > > > > > > >  Remove
>> > > > > > > > > > > > > >  PutFH of file
>> > > > > > > > > > > > > > - if the above returns NFS_OK, the file still
>> > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > client
>> > > > > > > > > > > > > >  can continue to use the delegation, writing the
>> > > > > > > > > > > > > > dirty
>> > > > > > > > > > > > > > data back
>> > > > > > > > > > > > > >  to the server sometime later (or maybe never, if
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > file gets
>> > > > > > > > > > > > > >   deleted by a subsequent Remove).
>> > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > - If the above returns NFS4ERR_STALE for the
>> > > > > > > > > > > > > > PutFH,
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > file
>> > > > > > > > > > > > > >  has been deleted. It can throw the dirty data
>> > > > > > > > > > > > > > away
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > >  DelegReturn_NoFH the delegation.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > A similar situation exists for Rename where the
>> > > > > > > > > > > > > > destination
>> > > > > > > > > > > > > > already exists.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I find this scenario happens frequently during
>> > > > > > > > > > > > > > software
>> > > > > > > > > > > > > > builds.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > So, what do others think  w.r.t. this new
>> > > > > > > > > > > > > > operation?
>> > > > > > > > > > > > > > rick
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Why would you need to do a DELEGRETURN if the
>> > > > > > > > > > > > > > PUTFH
>> > > > > > > > > > > > > > returns
>> > > > > > > > > > > > > > NFS4ERR_STALE?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > To make sure the client and server agree on the
>> > > > > > > > > > > > > state
>> > > > > > > > > > > > > of
>> > > > > > > > > > > > > things.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > If the server isn’t willing to just invalidate
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > delegation on
>> > > > > > > > > > > > > > the final unlink of the file,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I'm not sure about "unwilling",  but I am
>> > > > > > > > > > > > > uncomfortable
>> > > > > > > > > > > > > with
>> > > > > > > > > > > > > unilateral stare changes like this, even if the
>> > > > > > > > > > > > > spec
>> > > > > > > > > > > > > indicated that
>> > > > > > > > > > > > > needed to be done, which is harder to effect than
>> > > > > > > > > > > > > an
>> > > > > > > > > > > > > extension such
>> > > > > > > > > > > > > as Rick is proposing
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >  > then it can and should recall the delegation
>> > > > > > > > > > > > > (and
>> > > > > > > > > > > > > any
>> > > > > > > > > > > > > pNFS
>> > > > > > > > > > > > > layouts that might be outstanding).
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > If it did recall it/them, then there is no way to
>> > > > > > > > > > > > > return
>> > > > > > > > > > > > > them
>> > > > > > > > > > > > > without something like Rick's proposed extension.
>> > > > > > > > > > > > I assumed Trond meant "recall during the REMOVE"
>> > > > > > > > > > > > while
>> > > > > > > > > > > > the FH
>> > > > > > > > > > > > is
>> > > > > > > > > > > > still valid.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > I didn't get that.  Apparently, I was mistaken.
>> > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > Yes. This is a clarification of how the server can deal
>> > > > > > > > > > > with
>> > > > > > > > > > > the
>> > > > > > > > > > > problem (if it thinks this is a problem) within the
>> > > > > > > > > > > confines of
>> > > > > > > > > > > the
>> > > > > > > > > > > existing spec
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > That depends on what the problem is.  Your proposal does
>> > > > > > > > > > address
>> > > > > > > > > > the problem of recalling a delegation when it is
>> > > > > > > > > > incapable of
>> > > > > > > > > > being returned.  It doesn't, at least alone, solve Rick's
>> > > > > > > > > > problem  which is that he wants to be able to know, at
>> > > > > > > > > > the
>> > > > > > > > > > time
>> > > > > > > > > > of recall, whether he can throw away pending writes, as
>> > > > > > > > > > opposed
>> > > > > > > > > > to flushing them to the server.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Please re-read what I said above (repeated here):
>> > > > > > > > > >
>> > > > > > > > > > > > > > If a client is holding a delegation, it will
>> > > > > > > > > > > > > > presumably
>> > > > > > > > > > > > > > have done
>> > > > > > > > > > > > > > a GETATTR at some point while holding that
>> > > > > > > > > > > > > > delegation
>> > > > > > > > > > > > > > so
>> > > > > > > > > > > > > > should
>> > > > > > > > > > > > > > know how many links remain. While it is true that
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > unlink
>> > > > > > > > > > > > > > might invalidate the filehandle, that’s not a
>> > > > > > > > > > > > > > situation
>> > > > > > > > > > > > > > you are
>> > > > > > > > > > > > > > able to fix with the new operation.
>> > > > > > > > > I did not get this. Now I see that in LINK (sec. 18.9.3) it
>> > > > > > > > > says
>> > > > > > > > > that
>> > > > > > > > > delegations must
>> > > > > > > > > be CB_RECALL'd (I never noticed that and assumed that a
>> > > > > > > > > LINK
>> > > > > > > > > could
>> > > > > > > > > be done by another client when a delegation is held. My
>> > > > > > > > > Bad;-)
>> > > > > > > > >
>> > > > > > > > > So, yes, the client that holds a delegation can track
>> > > > > > > > > numlinks
>> > > > > > > > > for
>> > > > > > > > > the
>> > > > > > > > > file. I didn't realize that was the case.
>> > > > > > > > I now think there is a glitch in this plan...
>> > > > > > > > I cannot see anywhere in RFC7530 that specifies LINK should
>> > > > > > > > recall
>> > > > > > > > delegations, so I am not sure if a NFSv4.0 LINK done by
>> > > > > > > > another
>> > > > > > > > client
>> > > > > > > > could increase numlinks without the delegation being
>> > > > > > > > recalled.
>> > > > > > > > (I knew there was a reason the FreeBSD NFSv4 server did not
>> > > > > > > > recall
>> > > > > > > > delegations for LINK. I will patch it to do so for
>> > > > > > > > NFSv4.1/4.2.)
>> > > > > > > >
>> > > > > > >
>> > > > > > > It doesn't just change nlink. It also bumps the change
>> > > > > > > attribute
>> > > > > > > (and
>> > > > > > > the ctime), which by itself implies that it must recall
>> > > > > > > delegations
>> > > > > > > according to RFC8881.
>> > > > > > > Specifically in section 10.2:
>> > > > > > >
>> > > > > > >    *  The semantics of the file system are crucial in defining
>> > > > > > > when
>> > > > > > >       delegation recall is required.  If a particular change
>> > > > > > > within
>> > > > > > > a
>> > > > > > >       specific implementation causes change to a file
>> > > > > > > attribute,
>> > > > > > > then
>> > > > > > >       delegation recall is required, whether that operation has
>> > > > > > > been
>> > > > > > >       specifically listed as requiring delegation recall.
>> > > > > > > Again,
>> > > > > > > what
>> > > > > > >       is critical is whether the guarantees provided by the
>> > > > > > > delegation
>> > > > > > >       are being invalidated.
>> > > > > > >
>> > > > > > > The Linux client interprets that to mean that LINK will recall
>> > > > > > > delegations.
>> > > > > > Yes, I agree, for NFSv4.1/4.2.
>> > > > > > (And admit the FreeBSD NFSv4.1/4.2 needs to be fixed to do this.)
>> > > > > >
>> > > > > > But what about NFSv4.0?
>> > > > > > I'm looking, but cannot find similar words in RFC7530.
>> > > > > > (Sec. 10.4.4. deals with Recall  of an Open Delegation, but it
>> > > > > > does not mention LINK and it does not mention "any operation
>> > > > > > that modifies the attributes. The same words appear to be in
>> > > > > > RFC3530, except under Sec. 9.4.4.) I do find mention that
>> > > > > > holders of delegations can either not modify attributes or
>> > > > > > only the attributes related to writing for write delegation.
>> > > > > > However, I cannot see any mention of clients that do not
>> > > > > > hold delegations.
>> > > > > >
>> > > > > > Maybe you are arguing that it should have been obvious for
>> > > > > > NFSv4.0, but it was not so for me (and, therefore, maybe other
>> > > > > > NFSv4.0
>> > > > > > implementers?). There is also the case of NFSv3 LINK RPCs.
>> > > > > >
>> > > > > > Although it would be nice to assume, I'm not convinced that
>> > > > > > extant NFS servers will all provide the guarantee that numlinks
>> > > > > > will
>> > > > > > not be changed (increased via LINK) when the client holds
>> > > > > > a delegation.
>> > > > > > (If the client could be sure the NFS server was a "pure
>> > > > > > NFSv4.1/4.2
>> > > > > > server and did not support earlier NFS versions, I would agree
>> > > > > > with
>> > > > > > you.)
>> > > > > > I just think that, since RFC7530 Sec. 10.4.4. makes no mention of
>> > > > > > LINK, it is risky to assume all NFSv4.0 server implementations
>> > > > > > recall
>> > > > > > delegations when a LINK is done. (In particular, since the
>> > > > > > NFSv4.0
>> > > > > > cannot know which client is performing the LINK.)
>> > > > > >
>> > > > >
>> > > > > What is the motivation for continuing to optimise NFSv4.0? While it
>> > > > > would be nice to allow it to benefit too, that problem is not one
>> > > > > that
>> > > > > keeps me awake at night.
>> > > > I certainly am not interested in improving NFSv4.0.
>> > > >
>> > > > What I was trying to say is...
>> > > > When an NFSv4.0 client does a LINK, does the NFSv4.[0,1,2] server
>> > > > recall all delegations for the file?
>> > > > If not, then the numlinks attribute could get increased without the
>> > > > delegation being recalled. And if that is the case, then the
>> > > > NFSv4.1/4.2
>> > > > client cannot trust its value of numlinks to indicate if the file
>> > > > being
>> > > > REMOVE'd will be deleted on the server.
>> > > >
>> > >
>> > > From RFC7830, section 10.4:
>> > >
>> > >    When a single client holds an OPEN_DELEGATE_READ delegation, it is
>> > >    assured that no other client may modify the contents or attributes of
>> > >    the file.  If more than one client holds an OPEN_DELEGATE_READ
>> > >    delegation, then the contents and attributes of that file are not
>> > >    allowed to change.  When a client has an OPEN_DELEGATE_WRITE
>> > >    delegation, it may modify the file data since no other client will be
>> > >    accessing the file's data.  The client holding an OPEN_DELEGATE_WRITE
>> > >    delegation may only affect file attributes that are intimately
>> > >    connected with the file data: size, time_modify, and change.
>> > >
>> > > Clearly, when reading the above, the assumption should be that a read
>> > > delegation MUST be recalled or revoked if the file attributes change.
>> > >
>> > > I do not find anything that says the same thing for a write delegation.
>> > > I suspect that the authors did intend to imply that, but I'm not
>> > > finding anything that explicitly says that a client that holds a write
>> > > delegation can expect attributes not to change due to the action of
>> > > another client. It can only expect that the data contents (and hence
>> > > the size attribute) won't change, and it is not allowed to modify
>> > > attributes other than those "intimately connected with the file
>> > > data"...
>> > For a simple example:
>> > - Client A has a NFSv4.2 mount against the server
>> >   - It holds a write delegation for "foo".
>> > - Client B has a NFSv4.0 mount against the server
>> >   - It does a "ln foo bar" (ie. creates a second hard link to the
>> >       same file called "bar")
>> > Does the server recall the write delegation that Client A holds
>> > as a part of the "ln foo bar"?
>> >
>> > I suspect you are right, in that the original intent would be "yes",
>> > but since LINK is not mentioned at the beginning of RFC7530 Sec. 10.4.4
>> > I, for one, missed it and need to patch the FreeBSD NFSv4 server.
>> >
>> > As a result, I am not comfortable assuming numlinks won't change
>> > while the client holds a write delegation for the file at this time.
>> >
>> > I will try and test this simple example at the Bakeathon next week, rick
>> At the Bakeathon last week, the non-FreeBSD NFSv4 servers I tested did
>> recall the delegation when another client did a "ln foo bar".
>> One of the servers recalled the delegation even when the same client with
>> a NFSv4.2 mount did the "ln foo bar".
>>
>> I am still not sure if I am willing to "trust" the value of numlinks for all
>> servers, but using numlinks would have worked for the ones I tested.
>>
>> Btw, I also found that all the servers I tested recalled the delegation
>> for Remove/Rename when done by the same client as the one holding
>> the delegation for a NFSv4.2 mount. This is what RFC8881 says,
>> although the words are verbatim from RFC3530 and I think the
>> requirement is "an oversight" for NFSv4.1/4.2, since the server does
>> know if the client doing the Remove/Rename is the same client.
>>
>> Anyhow, I no longer think a DelegReturn_NoFH would be useful, rick
>>
>> >
>> > >
>> > > --
>> > > Trond Myklebust
>> > > Linux NFS client maintainer, Hammerspace
>> > > [e-mail]
>> > >
>> > >
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/nfsv4