Re: [nfsv4] RFC: New DelegReturn_NoFH operation for NFSv4.2?

[Rick Macklem <[e-mail]>]

On Wed, Apr 17, 2024 at 4:46 AM David Noveck <[e-mail]> wrote:
>
>
>
> On Tue, Apr 16, 2024, 5:40 PM Rick Macklem <[e-mail]> wrote:
>>
>> On Tue, Apr 16, 2024 at 7:44 AM Trond Myklebust <[e-mail]> wrote:
>> >
>> > On Tue, 2024-04-16 at 07:06 -0700, Rick Macklem wrote:
>> > > On Tue, Apr 16, 2024 at 5:21 AM David Noveck <[e-mail]>
>> > > wrote:
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Apr 15, 2024, 8:06 PM Trond Myklebust
>> > > > <[e-mail]> wrote:
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Apr 15, 2024, at 18:31, Rick Macklem <[e-mail]>
>> > > > > wrote:
>> > > > >
>> > > > > Hi,
>> > > > >
>> > > > > I am wondering if others have found this a pita and are
>> > > > > interested in
>> > > > > a new operation for NFSv4.2 to get around it.
>> > > > >
>> > > > > Right now DelegReturn requires a CFH and delegation stateid.
>> > > > > I do not know why the CFH is needed, since it seems that the
>> > > > > delegation stateid should be sufficient?
>> > > > >
>> > > > > This new operation would be the same as DelegReturn except that
>> > > > > it would not require a CFH argument.
>> > > > >
>> > > > > Why am I interested in this?
>> > > > > - Lets assume a NFSv4.2 server that supports persistent FHs and
>> > > > >  multiple hard links (as most extant servers do).
>> > > > > - A client mounted to this server is removing a file for which it
>> > > > > holds
>> > > > >  a write delegation and a bunch of dirty data.
>> > > > > Right now, the client must:
>> > > > > - Write the dirty data back to the server.
>> > > > > - DelegReturn
>> > > > > - Remove
>> > > > > because the client has no way of knowing if the file will be
>> > > > > deleted
>> > > > > by the Remove (or alternately is just removing one of several
>> > > > > hard links).
>> > > > >
>> > > > >
>> > > > > If a client is holding a delegation, it will presumably have done
>> > > > > a GETATTR at some point while holding that delegation so should
>> > > > > know how many links remain. While it is true that the unlink
>> > > > > might invalidate the filehandle, that’s not a situation you are
>> > > > > able to fix with the new operation.
>> > > > >
>> > > > > With a DelegReturn_NoFH, the client could:
>> > > > > - Do a compound RPC with
>> > > > >  Remove
>> > > > >  PutFH of file
>> > > > > - if the above returns NFS_OK, the file still exists and the
>> > > > > client
>> > > > >  can continue to use the delegation, writing the dirty data back
>> > > > >  to the server sometime later (or maybe never, if the file gets
>> > > > >   deleted by a subsequent Remove).
>> > > > > or
>> > > > > - If the above returns NFS4ERR_STALE for the PutFH, the file
>> > > > >  has been deleted. It can throw the dirty data away and
>> > > > >  DelegReturn_NoFH the delegation.
>> > > > >
>> > > > > A similar situation exists for Rename where the destination
>> > > > > already exists.
>> > > > >
>> > > > > I find this scenario happens frequently during software builds.
>> > > > >
>> > > > > So, what do others think  w.r.t. this new operation? rick
>> > > > >
>> > > > >
>> > > > > Why would you need to do a DELEGRETURN if the PUTFH returns
>> > > > > NFS4ERR_STALE?
>> > > >
>> > > >
>> > > > To make sure the client and server agree on the state of things.
>> > > >
>> > > > > If the server isn’t willing to just invalidate the delegation on
>> > > > > the final unlink of the file,
>> > > >
>> > > > I'm not sure about "unwilling",  but I am uncomfortable with
>> > > > unilateral stare changes like this, even if the spec indicated that
>> > > > needed to be done, which is harder to effect than an extension such
>> > > > as Rick is proposing
>> > > >
>> > > >  > then it can and should recall the delegation (and any pNFS
>> > > > layouts that might be outstanding).
>> > > >
>> > > > If it did recall it/them, then there is no way to return them
>> > > > without something like Rick's proposed extension.
>> > > I assumed Trond meant "recall during the REMOVE" while the FH is
>> > > still valid.
>
>
> I didn't get that.  Apparently, I was mistaken.
>
>> >
>> > Yes. This is a clarification of how the server can deal with the
>> > problem (if it thinks this is a problem) within the confines of the
>> > existing spec
>
>
> That depends on what the problem is.  Your proposal does address the problem of recalling a delegation when it is incapable of being returned.  It doesn't, at least alone, solve Rick's problem  which is that he wants to be able to know, at the time of recall, whether he can throw away pending writes, as opposed to flushing them to the server.
>
> I believe that, if the server acts as you suppose it might, we would have a solution, if Rick ceased to depend on the legacy approach to remove involving "silly rename".  If you used a more modern approach to remove in which the fh invalidation is deferred to the last close, then the existence of a delegation would prevent the invalidation until delegation return. Since, until it was returned, there would be a potential open outstanding, at delegation recall you could test the link count and throw away the pending writes if the count is zero. If it isn't zero, you flush the writes  before return.

Yes, if a server implements OPEN4_RESULT_PRESERVE_UNLINKED
and the client retained an Open as well as a Delegation, then I think this
would work. (ie. If numlinks == 0, the close will delete the file and no new
links can be created, so the dirty data can be thrown away.)

The FreeBSD server does not do OPEN4_RESULT_PRESERVE_UNLINKED
and I am not planning on implementing it, since it requires "clever work" so
that the file is not deleted when the server reboots.
I wonder what extant NFSv4.1/4.2 servers support OPEN4_RESULT_PRESERVE_UNLINKED?

>
>> Just fyi, I should be able to test the stuff I've posted here recently
>> at the Bakeathon next week.
>> - I have a client setup that tries to do Open/Claim_deleg_prev_fh
>>   before ReclaimComplete.
>
>
> That sounds like it would work OK.
>
>> It also does not DelegReturn a delegation
>>   for a file it is about to Remove.
>
>
> OK.
>
>> - I'll have the server configured so that it does not bother to CB_RECALL
>>   a delegation issued to the same client as the one doing the Remove or
>>   Rename.
>
>
> In that case, I think the delegation will never be released and will hang around as an unreturnable piece of state.  On the other hand, if the server does what Trond proposes the delegation will be appropriately returned but you will  be doing those writes unconditionally, which is a performance issue, although not a major one.

I can try to determine what extant servers do when files with outstanding
delegations are deleted during Bakeathon testing next week. I can at least
see of the servers CB_RECALL. If there are some that do not, it may be
possible for the server testers to determine if the delegation was thrown
away.

I will have an experimental server up that does what I think Trond suggested.
That is, it will throw away the delegation instead of CB_RECALL'ng it when
a file is deleted, for the case where the delegation is issued to the
client doing
the REMOVE.

rick

>>
>>
>> I'll report here if the testing finds anything interesting, rick
>
>
> Thanks.
>
>
>>
>> >
>> > --
>> > Trond Myklebust
>> > Linux NFS client maintainer, Hammerspace
>> > [e-mail]
>> >
>> >