Re: [nfsv4] RFC: Is Open/Claim_Delegate_Prev done before/after Reclaim_complete?

[Rick Macklem <[e-mail]>]

On Sun, Apr 7, 2024 at 8:33 AM Rick Macklem <[e-mail]> wrote:
>
> On Sun, Apr 7, 2024 at 7:52 AM David Noveck <[e-mail]> wrote:
> >>
> >>
> >>
> >> The more I think about it, the more I think you are correct, at least
> >> by default.
> I was referring to the case where the client cannot reacquire the delegation,
> such as when the Open/Claim_delegate_prev gets a reply of NFS4ERR_RECLAIM_BAD.
> (Btw, Sec. 15.1.9.4 states "before the server restart or file system
> migration event".
> If we are going to use NFS4ERR_RECLAIM_BAD instead of NFS4ERR_EXPIRED, these
> words will need to be tweaked.)
>
> I am fine with either error return, but the current wording in Sec.
> 15.1.9.4 defines it for
> specific cases that are not this one.
>
> >
> >
> > Not fully correct.  Some of what I told you is wrong and will be corrected in -04.    The current target for submission is 4/23 but it is likely I canpull that forward by a week.   In any case, I hope the following fragment will be helpful:
> >
> > /*
> >  * Includes multiple types of operations:
> >  *
> >  *    - Non-reclaim operations valid independent of grace period status.
> >  *    - Reclaim operations only used during a grace period.
> >  *    - Reclaim operations only used during a special delegation recovery
> >  *      period.
> >  */
> >
> > enum open_claim_type4 {
> >
> >         CLAIM_NULL              = 0,    /* Non-reclaim operation, */
> >         CLAIM_PREVIOUS          = 1,    /* Reclaim operation -- grace
> >                                                                period only. */
> >         CLAIM_DELEGATE_CUR      = 2,    /* Non-reclaim operation. */
> >         CLAIM_DELEGATE_PREV     = 3,    /* Reclaim operation -- special
> >                                                                       delegation recovery period
> >                                                                       only. */
> Sounds correct to me, assuming that "special delegation recovery period" is
> described somewhere. Otherwise I'd just say something like
> "delegation reclaim - not during
> grace period".
>
> >         /*
> >          * Beyond this point, all values are new to v4.1.
> >          */
> >
> >        /*
> >          * Like CLAIM_NULL, but object identified
> >          * by the current filehandle.
> >          */
> >         CLAIM_FH                = 4,    /* Non-reclaim operation. */
> >
> >         /*
> >          * Like CLAIM_DELEGATE_CUR, but object identified
> >          * by current filehandle.
> >          */
> >         CLAIM_DELEG_CUR_FH      = 5,    /* Non-reclaim operation. */
> >
> >         /*
> >          * Like CLAIM_DELEGATE_PREV, but object identified
> >          * by current filehandle.
> >          */
> >         CLAIM_DELEG_PREV_FH     = 6     /* Reclaim operation -- special
> >                                                                       delgation recovery period
> >                                                                        only. */
> > };
> >
> >> Why?
> >> Because when a client crashes/reboots and there is dirty data in the
> >> RAM cache it gets
> >> tossed, so tossing dirty data in the non-volatile cache when it cannot safely
> >> write it back to the server would be consistent with what happens now.
> >
> >
> > Yes, but the idea is to do better than that, when it can be written safely.
> >>
> >>
> >> For the case where the server is in grace, the client can use
> >> Open/Claim_previous
> >> and then, if the reply indicates an immediate recall of the
> >> delegation, it can write the dirty data
> >> back to the server before doing a Delegreturn.
> >
> >
> > True but that is an unusual case.
> >>
> >>
> >> For the case where the server is not in grace and the Open/Claim_delegate_prev
> >> fails, I think there is another way that the client can safely write
> >> the dirty data back.
> >> - If the client keeps track of the server's value of the change
> >> attribute for the file
> >>   in non-volatile storage.
> >>   - The client can try an
> >> Open/Claim_null/Open_share_access_write/Open_share_deny_write
> >>      followed by a Getattr of change in the same compound.
> >>      - if this succeeds and the change attribute is the same as the
> >> stored one, then I think the client can
> >
> >
> > If the  Open/Claim_delegate_prev fails, it would be because the delegation was revoked, in which
> > case, change attribute will be different.
> Why would the change attribute be different? It changes when there is
> a data/metadata change
> done to the file. Typically, a delegation would be revoked (instead of
> recalled) if there is a conflicting Open
> request and the lease has expired. This does not imply a change in the
> change attribute, does it?
> Of course, if any data/metadata changes are applied to the file by
> other clients (such as the one
> that acquired the conflicting open), then change would change.
> Or, I do not understand what you mean by revoke?
>
> >  Also, in that case, checking change attribute and proceding
> > to write is not safe, since the attribute can change after the check.  You need continuous posession
> > of the delegation which is why clain-delege-prev was created.
> Wouldn't a OPEN4_SHARE_DENY_WRITE be sufficient to guarantee that
> other clients are
> not writing to the file during the write-back? Once change is checked
> and found the same after the Open,
> there shouldn't have been any writes done by other clients already
> done and the deny_write should guarantee
> that no other clients write to the file until the open_stateid is
> closed (or there is something ugly
> like a network partitioning/lease expiration). For this case, all the
> client is trying to do is write the dirty data
> back to the server safely, since it was not able to re-acquire the
> delegation. It will no longer be able to use
> the cached data once the open_stateid is closed.
> I have not tried this yet, so there might be issues that I have not
> recognized w.r.t. doing this.
>
> I agree that reclaiming the delegation is preferable, since caching of
> the file can continue and
> the write-back does not need to be done right away.
>
> >
> >>        safely write the dirty data back to the server using the open_stateid
> >>        - then close the open_stateid
> >>
> >> An option other than tossing the data would be to flag it and let a
> >> sysadmin/user decide if
> >> merging it back to the server's file makes sense. Ugly but doable when
> >> the dirty data is in
> >> client non-volatile storage.
> >
> >
> > He could decide but I'm not sure on what basis he could decide, especially since telling
> > him all the relevant information could be a security hole a megaparsec wide. :-(
> Well, I wore a sysadmin hat for 30+ years and one of my more enjoyable
> work items
> went something like...
> A prof. would email saying they had made a bunch of changes to a file
> and then lost them.
> Can you get the changes back?
> - The attempt usually involved finding the most recent backup of the
> file, along with stuff
>    like an editor tmp file left around to try and get back what I could.
> So, yes, for the user it could easily be a security issue (it happens
> that this caching mechanism
> uses a separate file for each file being cached, so access to the
> cache for the file could be
> the same as the file), but for a sysadmin it is just a bothersome manual action.
Btw, back when everything was UFS, I could sometimes even find some of what
the had accidentally deleted in free blocks. They'd usually do this
when they were
trying to get a paper submitted by a deadline and I'd find out what
the paper was
talking about, so I'd know what to look for in recently free'd data blocks.

It could be fun or it could be a bother, but for a sysadmin that knew how this
caching mechanism worked (each cache file is for a file on the server and is
named by the server's file handle, although the name it was looked up with is
also recorded).  So, finding the cached copy and providing that to someone who
has write permissions on the server's file wouldn't be that difficult.
Basically a bit
of a search on the NFS server to figure out the FH for the file.
Again, definitely something joe user couldn't do, but if I was the
sysadmin for such
clients, I might configure them so the cached data file gets flagged
when a delegation
cannot be re-acquired and has dirty data in it.

rick

>
> rick
>
> >>
> >>
> >> rick
> >>
> >> >
> >> > --
> >> > Trond Myklebust
> >> > Linux NFS client maintainer, Hammerspace
> >> > [e-mail]
> >> >
> >> >
> >>
> >> _______________________________________________
> >> nfsv4 mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/nfsv4