Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Thomas Haynes <[e-mail]>]

> On Apr 18, 2024, at 1:25 PM, Rick Macklem <[e-mail]> wrote:
> 
> On Thu, Apr 18, 2024 at 8:53 AM Trond Myklebust <[e-mail] <mailto:[e-mail]>> wrote:
>> 
>> On Wed, 2024-04-17 at 18:54 -0700, Rick Macklem wrote:
>>> On Wed, Apr 17, 2024 at 5:37 PM Rick Macklem <[e-mail]>
>>> wrote:
>>>> 
>>>> On Wed, Apr 17, 2024 at 4:56 PM Trond Myklebust
>>>> <[e-mail]> wrote:
>>>>> 
>>>>> On Wed, 2024-04-17 at 07:46 -0400, David Noveck wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Apr 16, 2024, 5:40 PM Rick Macklem
>>>>> <[e-mail]> wrote:
>>>>> 
>>>>> On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust
>>>>> <[e-mail]> wrote:
>>>>>> 
>>>>>> On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
>>>>>>> On Tue, Apr 16, 2024 at 5:21 AM David Noveck
>>>>>>> <[e-mail]>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
>>>>>>>> <[e-mail]> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Apr 15, 2024, at 18:31, Rick Macklem
>>>>>>>>> <[e-mail]>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I am wondering if others have found this a pita and are
>>>>>>>>> interested in
>>>>>>>>> a new operation for NFSv4.2 to get around it.
>>>>>>>>> 
>>>>>>>>> Right now DelegReturn requires a CFH and delegation
>>>>>>>>> stateid.
>>>>>>>>> I do not know why the CFH is needed, since it seems that
>>>>>>>>> the
>>>>>>>>> delegation stateid should be sufficient?
>>>>>>>>> 
>>>>>>>>> This new operation would be the same as DelegReturn
>>>>>>>>> except that
>>>>>>>>> it would not require a CFH argument.
>>>>>>>>> 
>>>>>>>>> Why am I interested in this?
>>>>>>>>> - Lets assume a NFSv4.2 server that supports persistent
>>>>>>>>> FHs and
>>>>>>>>> multiple hard links (as most extant servers do).
>>>>>>>>> - A client mounted to this server is removing a file for
>>>>>>>>> which it
>>>>>>>>> holds
>>>>>>>>> a write delegation and a bunch of dirty data.
>>>>>>>>> Right now, the client must:
>>>>>>>>> - Write the dirty data back to the server.
>>>>>>>>> - DelegReturn
>>>>>>>>> - Remove
>>>>>>>>> because the client has no way of knowing if the file will
>>>>>>>>> be
>>>>>>>>> deleted
>>>>>>>>> by the Remove (or alternately is just removing one of
>>>>>>>>> several
>>>>>>>>> hard links).
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> If a client is holding a delegation, it will presumably
>>>>>>>>> have done
>>>>>>>>> a GETATTR at some point while holding that delegation so
>>>>>>>>> should
>>>>>>>>> know how many links remain. While it is true that the
>>>>>>>>> unlink
>>>>>>>>> might invalidate the filehandle, that’s not a situation
>>>>>>>>> you are
>>>>>>>>> able to fix with the new operation.
>>>>>>>>> 
>>>>>>>>> With a DelegReturn_NoFH, the client could:
>>>>>>>>> - Do a compound RPC with
>>>>>>>>> Remove
>>>>>>>>> PutFH of file
>>>>>>>>> - if the above returns NFS_OK, the file still exists and
>>>>>>>>> the
>>>>>>>>> client
>>>>>>>>> can continue to use the delegation, writing the dirty
>>>>>>>>> data back
>>>>>>>>> to the server sometime later (or maybe never, if the
>>>>>>>>> file gets
>>>>>>>>>  deleted by a subsequent Remove).
>>>>>>>>> or
>>>>>>>>> - If the above returns NFS4ERR_STALE for the PutFH, the
>>>>>>>>> file
>>>>>>>>> has been deleted. It can throw the dirty data away and
>>>>>>>>> DelegReturn_NoFH the delegation.
>>>>>>>>> 
>>>>>>>>> A similar situation exists for Rename where the
>>>>>>>>> destination
>>>>>>>>> already exists.
>>>>>>>>> 
>>>>>>>>> I find this scenario happens frequently during software
>>>>>>>>> builds.
>>>>>>>>> 
>>>>>>>>> So, what do others think  w.r.t. this new operation? rick
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Why would you need to do a DELEGRETURN if the PUTFH
>>>>>>>>> returns
>>>>>>>>> NFS4ERR_STALE?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> To make sure the client and server agree on the state of
>>>>>>>> things.
>>>>>>>> 
>>>>>>>>> If the server isn’t willing to just invalidate the
>>>>>>>>> delegation on
>>>>>>>>> the final unlink of the file,
>>>>>>>> 
>>>>>>>> I'm not sure about "unwilling",  but I am uncomfortable
>>>>>>>> with
>>>>>>>> unilateral stare changes like this, even if the spec
>>>>>>>> indicated that
>>>>>>>> needed to be done, which is harder to effect than an
>>>>>>>> extension such
>>>>>>>> as Rick is proposing
>>>>>>>> 
>>>>>>>>> then it can and should recall the delegation (and any
>>>>>>>> pNFS
>>>>>>>> layouts that might be outstanding).
>>>>>>>> 
>>>>>>>> If it did recall it/them, then there is no way to return
>>>>>>>> them
>>>>>>>> without something like Rick's proposed extension.
>>>>>>> I assumed Trond meant "recall during the REMOVE" while the FH
>>>>>>> is
>>>>>>> still valid.
>>>>> 
>>>>> 
>>>>> I didn't get that.  Apparently, I was mistaken.
>>>>> 
>>>>>> 
>>>>>> Yes. This is a clarification of how the server can deal with
>>>>>> the
>>>>>> problem (if it thinks this is a problem) within the confines of
>>>>>> the
>>>>>> existing spec
>>>>> 
>>>>> 
>>>>> That depends on what the problem is.  Your proposal does address
>>>>> the problem of recalling a delegation when it is incapable of
>>>>> being returned.  It doesn't, at least alone, solve Rick's
>>>>> problem  which is that he wants to be able to know, at the time
>>>>> of recall, whether he can throw away pending writes, as opposed
>>>>> to flushing them to the server.
>>>>> 
>>>>> 
>>>>> Please re-read what I said above (repeated here):
>>>>> 
>>>>>>>>> If a client is holding a delegation, it will presumably
>>>>>>>>> have done
>>>>>>>>> a GETATTR at some point while holding that delegation so
>>>>>>>>> should
>>>>>>>>> know how many links remain. While it is true that the
>>>>>>>>> unlink
>>>>>>>>> might invalidate the filehandle, that’s not a situation
>>>>>>>>> you are
>>>>>>>>> able to fix with the new operation.
>>>> I did not get this. Now I see that in LINK (sec. 18.9.3) it says
>>>> that
>>>> delegations must
>>>> be CB_RECALL'd (I never noticed that and assumed that a LINK could
>>>> be done by another client when a delegation is held. My Bad;-)
>>>> 
>>>> So, yes, the client that holds a delegation can track numlinks for
>>>> the
>>>> file. I didn't realize that was the case.
>>> I now think there is a glitch in this plan...
>>> I cannot see anywhere in RFC7530 that specifies LINK should recall
>>> delegations, so I am not sure if a NFSv4.0 LINK done by another
>>> client
>>> could increase numlinks without the delegation being recalled.
>>> (I knew there was a reason the FreeBSD NFSv4 server did not recall
>>> delegations for LINK. I will patch it to do so for NFSv4.1/4.2.)
>>> 
>> 
>> It doesn't just change nlink. It also bumps the change attribute (and
>> the ctime), which by itself implies that it must recall delegations
>> according to RFC8881.
>> Specifically in section 10.2:
>> 
>>   *  The semantics of the file system are crucial in defining when
>>      delegation recall is required.  If a particular change within a
>>      specific implementation causes change to a file attribute, then
>>      delegation recall is required, whether that operation has been
>>      specifically listed as requiring delegation recall.  Again, what
>>      is critical is whether the guarantees provided by the delegation
>>      are being invalidated.
>> 
>> The Linux client interprets that to mean that LINK will recall
>> delegations.
> Yes, I agree, for NFSv4.1/4.2.
> (And admit the FreeBSD NFSv4.1/4.2 needs to be fixed to do this.)
> 
> But what about NFSv4.0?
> I'm looking, but cannot find similar words in RFC7530.
> (Sec. 10.4.4. deals with Recall  of an Open Delegation, but it
> does not mention LINK and it does not mention "any operation
> that modifies the attributes. The same words appear to be in
> RFC3530, except under Sec. 9.4.4.) I do find mention that
> holders of delegations can either not modify attributes or
> only the attributes related to writing for write delegation.
> However, I cannot see any mention of clients that do not
> hold delegations.
> 
> Maybe you are arguing that it should have been obvious for
> NFSv4.0, but it was not so for me (and, therefore, maybe other NFSv4.0
> implementers?). There is also the case of NFSv3 LINK RPCs.
> 
> Although it would be nice to assume, I'm not convinced that
> extant NFS servers will all provide the guarantee that numlinks will
> not be changed (increased via LINK) when the client holds
> a delegation.
> (If the client could be sure the NFS server was a "pure NFSv4.1/4.2
> server and did not support earlier NFS versions, I would agree with
> you.)

I was under the impression that a server was either NFSv4.0 or NFSv4.1/4.2, even if pNFS was not applied.

And again, I really wish we had bumped the major version to NFSv5.0 on that release….




> I just think that, since RFC7530 Sec. 10.4.4. makes no mention of
> LINK, it is risky to assume all NFSv4.0 server implementations recall
> delegations when a LINK is done. (In particular, since the NFSv4.0
> cannot know which client is performing the LINK.)
> 
> rick
> 
>> 
>> --
>> Trond Myklebust
>> Linux NFS client maintainer, Hammerspace
>> [e-mail] <mailto:[e-mail]>
>> 
>> 
> 
> _______________________________________________
> nfsv4 mailing list
> [e-mail] <mailto:[e-mail]>
> https://www.ietf.org/mailman/listinfo/nfsv4