Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Rick Macklem <[e-mail]>]

On Thu, Apr 18, 2024 at 4:57 PM Thomas Haynes <[e-mail]> wrote:
>
>
>
> On Apr 18, 2024, at 1:25 PM, Rick Macklem <[e-mail]> wrote:
>
> On Thu, Apr 18, 2024 at 8:53 AM Trond Myklebust <[e-mail]> wrote:
>
>
> On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
>
> On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem <[e-mail]>
> wrote:
>
>
> On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
> <[e-mail]> wrote:
>
>
> On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
>
>
>
> On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
> <[e-mail]> wrote:
>
> On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
> <[e-mail]> wrote:
>
>
> On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
>
> On Tue, Apr 16, 2024 at 5:21 AM David Noveck
> <[e-mail]>
> wrote:
>
>
>
>
> On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
> <[e-mail]> wrote:
>
>
>
>
> On Apr 15, 2024, at 18:31, Rick Macklem
> <[e-mail]>
> wrote:
>
> Hi,
>
> I am wondering if others have found this a pita and are
> interested in
> a new operation for NFSv4.2 to get around it.
>
> Right now DelegReturn requires a CFH and delegation
> stateid.
> I do not know why the CFH is needed, since it seems that
> the
> delegation stateid should be sufficient?
>
> This new operation would be the same as DelegReturn
> except that
> it would not require a CFH argument.
>
> Why am I interested in this?
> - Lets assume a NFSv4.2 server that supports persistent
> FHs and
> multiple hard links (as most extant servers do).
> - A client mounted to this server is removing a file for
> which it
> holds
> a write delegation and a bunch of dirty data.
> Right now, the client must:
> - Write the dirty data back to the server.
> - DelegReturn
> - Remove
> because the client has no way of knowing if the file will
> be
> deleted
> by the Remove (or alternately is just removing one of
> several
> hard links).
>
>
> If a client is holding a delegation, it will presumably
> have done
> a GETATTR at some point while holding that delegation so
> should
> know how many links remain. While it is true that the
> unlink
> might invalidate the filehandle, that’s not a situation
> you are
> able to fix with the new operation.
>
> With a DelegReturn_NoFH, the client could:
> - Do a compound RPC with
> Remove
> PutFH of file
> - if the above returns NFS_OK, the file still exists and
> the
> client
> can continue to use the delegation, writing the dirty
> data back
> to the server sometime later (or maybe never, if the
> file gets
>  deleted by a subsequent Remove).
> or
> - If the above returns NFS4ERR_STALE for the PutFH, the
> file
> has been deleted. It can throw the dirty data away and
> DelegReturn_NoFH the delegation.
>
> A similar situation exists for Rename where the
> destination
> already exists.
>
> I find this scenario happens frequently during software
> builds.
>
> So, what do others think  w.r.t. this new operation? rick
>
>
> Why would you need to do a DELEGRETURN if the PUTFH
> returns
> NFS4ERR_STALE?
>
>
>
> To make sure the client and server agree on the state of
> things.
>
> If the server isn’t willing to just invalidate the
> delegation on
> the final unlink of the file,
>
>
> I'm not sure about "unwilling",  but I am uncomfortable
> with
> unilateral stare changes like this, even if the spec
> indicated that
> needed to be done, which is harder to effect than an
> extension such
> as Rick is proposing
>
> then it can and should recall the delegation (and any
>
> pNFS
> layouts that might be outstanding).
>
> If it did recall it/them, then there is no way to return
> them
> without something like Rick's proposed extension.
>
> I assumed Trond meant "recall during the REMOVE" while the FH
> is
> still valid.
>
>
>
> I didn't get that.  Apparently, I was mistaken.
>
>
> Yes. This is a clarification of how the server can deal with
> the
> problem (if it thinks this is a problem) within the confines of
> the
> existing spec
>
>
>
> That depends on what the problem is.  Your proposal does address
> the problem of recalling a delegation when it is incapable of
> being returned.  It doesn't, at least alone, solve Rick's
> problem  which is that he wants to be able to know, at the time
> of recall, whether he can throw away pending writes, as opposed
> to flushing them to the server.
>
>
> Please re-read what I said above (repeated here):
>
> If a client is holding a delegation, it will presumably
> have done
> a GETATTR at some point while holding that delegation so
> should
> know how many links remain. While it is true that the
> unlink
> might invalidate the filehandle, that’s not a situation
> you are
> able to fix with the new operation.
>
> I did not get this. Now I see that in LINK (sec. 18.9.3) it says
> that
> delegations must
> be CB_RECALL'd (I never noticed that and assumed that a LINK could
> be done by another client when a delegation is held. My Bad;-)
>
> So, yes, the client that holds a delegation can track numlinks for
> the
> file. I didn't realize that was the case.
>
> I now think there is a glitch in this plan...
> I cannot see anywhere in RFC7530 that specifies LINK should recall
> delegations, so I am not sure if a NFSv4.0 LINK done by another
> client
> could increase numlinks without the delegation being recalled.
> (I knew there was a reason the FreeBSD NFSv4 server did not recall
> delegations for LINK. I will patch it to do so for NFSv4.1/4.2.)
>
>
> It doesn't just change nlink. It also bumps the change attribute (and
> the ctime), which by itself implies that it must recall delegations
> according to RFC8881.
> Specifically in section 10.2:
>
>   *  The semantics of the file system are crucial in defining when
>      delegation recall is required.  If a particular change within a
>      specific implementation causes change to a file attribute, then
>      delegation recall is required, whether that operation has been
>      specifically listed as requiring delegation recall.  Again, what
>      is critical is whether the guarantees provided by the delegation
>      are being invalidated.
>
> The Linux client interprets that to mean that LINK will recall
> delegations.
>
> Yes, I agree, for NFSv4.1/4.2.
> (And admit the FreeBSD NFSv4.1/4.2 needs to be fixed to do this.)
>
> But what about NFSv4.0?
> I'm looking, but cannot find similar words in RFC7530.
> (Sec. 10.4.4. deals with Recall  of an Open Delegation, but it
> does not mention LINK and it does not mention "any operation
> that modifies the attributes. The same words appear to be in
> RFC3530, except under Sec. 9.4.4.) I do find mention that
> holders of delegations can either not modify attributes or
> only the attributes related to writing for write delegation.
> However, I cannot see any mention of clients that do not
> hold delegations.
>
> Maybe you are arguing that it should have been obvious for
> NFSv4.0, but it was not so for me (and, therefore, maybe other NFSv4.0
> implementers?). There is also the case of NFSv3 LINK RPCs.
>
> Although it would be nice to assume, I'm not convinced that
> extant NFS servers will all provide the guarantee that numlinks will
> not be changed (increased via LINK) when the client holds
> a delegation.
> (If the client could be sure the NFS server was a "pure NFSv4.1/4.2
> server and did not support earlier NFS versions, I would agree with
> you.)
>
>
> I was under the impression that a server was either NFSv4.0 or NFSv4.1/4.2, even if pNFS was not applied.
Hmm, not sure what you mean. For a given client mount, yes, the mount uses one
version of the protocol or the other. However, for a NFS server, the
same file system
can often be exported to clients doing NFSv4.0 mounts and other
clients doing NFSv4.1/4.2
mounts. (ie. Same exported files/file handles, but different clients
doing mounts with
different protocol versions.)

Heck, at the Bakeathons, some servers are configured to allow NFSv3
mounts as well.

The specific case I was discussing is "what happens w.r.t. delegation
recall when these
clients using different NFS versions for their mounts do a LINK?".

For what I understood, Trond was suggesting the NFSv4.1/4.2 client holding a
write delegation would be guaranteed that the numlinks attribute would
not change.
(For that to work, any LINK done by any NFS mount using any NFS version must
cause a recall of the delegation, unless the LINK is done by the same
client as the
one holding the delegation. And I am concerned that the NFS server
will not recall
the delegation when a NFSv4.0 mount does the LINK, since RFC7530 does not
mention recall of delegations for LINK.)

rick

>
> And again, I really wish we had bumped the major version to NFSv5.0 on that release….
>
>
>
>
> I just think that, since RFC7530 Sec. 10.4.4. makes no mention of
> LINK, it is risky to assume all NFSv4.0 server implementations recall
> delegations when a LINK is done. (In particular, since the NFSv4.0
> cannot know which client is performing the LINK.)
>
> rick
>
>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [e-mail]
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>