Re: [nfsv4] RFC: Is Open/Claim_Delegate_Prev done before/after Reclaim_complete?

[Rick Macklem <[e-mail]>]

On Fri, Apr 5, 2024 at 8:02 PM Trond Myklebust <[e-mail]> wrote:
>
> On Fri, 2024-04-05 at 16:13 -0700, Rick Macklem wrote:
> > On Thu, Apr 4, 2024 at 9:18 PM Trond Myklebust
> > <[e-mail]> wrote:
> > >
> > > On Thu, 2024-04-04 at 16:23 -0700, Rick Macklem wrote:
> > > > Hi,
> > > >
> > > > I am in the process of implementing Open/Claim_Delegate_Prev and
> > > > have
> > > > run
> > > > into a question I cannot seem to answer from reading RFC8881.
> > > >
> > > > There is this snippet:
> > > >    enum open_claim_type4 {
> > > >            /*
> > > >             * Not a reclaim.
> > > >             */
> > > >            CLAIM_NULL              = 0,
> > > >
> > > >            CLAIM_PREVIOUS          = 1,
> > > >            CLAIM_DELEGATE_CUR      = 2,
> > > >            CLAIM_DELEGATE_PREV     = 3,
> > > >
> > > >            /*
> > > >             * Not a reclaim.
> > > >             *
> > > >             * Like CLAIM_NULL, but object identified
> > > >             * by the current filehandle.
> > > >             */
> > > >            CLAIM_FH                = 4, /* new to v4.1 */
> > > > The comments in the above seem to "hint" that CLAIM_DELEGATE_PREV
> > > > is
> > > > a
> > > > reclaim type operation.
> > > > If that is the case, I'd assume it/should be performed by the
> > > > client
> > > > before it does a
> > > > Reclaim_Complete?
> > > >
> > > > Then there is this snippet:
> > > >    For OPEN requests that reach the server during the grace
> > > > period,
> > > > the
> > > >    server returns an error of NFS4ERR_GRACE.  The following claim
> > > > types
> > > >    are exceptions:
> > > >
> > > >    *  OPEN requests specifying the claim type CLAIM_PREVIOUS are
> > > > devoted
> > > >       to reclaiming opens after a server restart and are
> > > > typically
> > > > only
> > > >       valid during the grace period.
> > > >
> > > >    *  OPEN requests specifying the claim types CLAIM_DELEGATE_CUR
> > > > and
> > > >       CLAIM_DELEG_CUR_FH are valid both during and after the
> > > > grace
> > > >       period.  Since the granting of the delegation that they are
> > > >       subordinate to assures that there is no conflict with locks
> > > > to
> > > > be
> > > >       reclaimed by other clients, the server need not return
> > > >       NFS4ERR_GRACE when these are received during the grace
> > > > period.
> > > > This clearly states that CLAIM_DELEGATE_PREV cannot be performed
> > > > during the server's grace period.
> > > >
> > > > The above two statements are not necessarily contradictory but it
> > > > does seem
> > > > that, if the server is in its grace period, the Reclaim_Complete
> > > > needs to be
> > > > done before any Open/Claim_Delegate_Prev, since Reclaim_Complete
> > > > tells
> > > > the server when to end its grace period.
> > > > Note that, usually, a client would not be rebooting when a server
> > > > is
> > > > in its
> > > > grace period, but that could happen.
> > > >
> > > > Bottom line...
> > > > Should a client perform Open/Claim_Delegate_Prev before or after
> > > > Reclaim_Complete?
> > > > And, if the answer is "after", how does the server know when a
> > > > client
> > > > is done recovering opens/delegations via
> > > > Open/Claim_Delegate_Prev?
> > > >
> > > > Thanks for any help with this, rick
> > >
> > > CLAIM_DELEGATE_PREV has now survived at least 24 years as part of a
> > > "standard" and has yet to be properly described. I can find
> > > references
> > > to it in RFC3010 and all NFSv4 descriptions since then, but I have
> > > yet
> > > to meet anybody who can claim to have implemented it in a product.
> > >
> > > IMO, it is time to drop it, and start afresh with something that
> > > has
> > > actually been road tested before someone started putting pen to
> > > paper.
> > Well, I am going to respond to David's post in some more detail, but
> > I will
> > note here that, with the exception of whether to do it before or
> > after
> > Reclaim_complete,
> > I have not found serious issues with it. (Btw, the FreeBSD server has
> > had a "dormant"
> > implementation for about 20years, although I am just now trying to
> > use it by the
> > FreeBSD client. I was wondering if I would find any other server
> > implementation at
> > the Bakeathon in a couple of weeks.)
>
> I see. So how do you plan to deal with the case where the server denies
> the request for a delegation? Will your client just toss away the dirty
> data?
Lets assume the client had a write delegation and dirty data for the file
in a non-volatile cache in the client before it crashed/rebooted.

The more I think about it, the more I think you are correct, at least
by default.
Why?
Because when a client crashes/reboots and there is dirty data in the
RAM cache it gets
tossed, so tossing dirty data in the non-volatile cache when it cannot safely
write it back to the server would be consistent with what happens now.

For the case where the server is in grace, the client can use
Open/Claim_previous
and then, if the reply indicates an immediate recall of the
delegation, it can write the dirty data
back to the server before doing a Delegreturn.

For the case where the server is not in grace and the Open/Claim_delegate_prev
fails, I think there is another way that the client can safely write
the dirty data back.
- If the client keeps track of the server's value of the change
attribute for the file
  in non-volatile storage.
  - The client can try an
Open/Claim_null/Open_share_access_write/Open_share_deny_write
     followed by a Getattr of change in the same compound.
     - if this succeeds and the change attribute is the same as the
stored one, then I think the client can
       safely write the dirty data back to the server using the open_stateid
       - then close the open_stateid

An option other than tossing the data would be to flag it and let a
sysadmin/user decide if
merging it back to the server's file makes sense. Ugly but doable when
the dirty data is in
client non-volatile storage.

rick

>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [e-mail]
>
>