[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Rick Macklem <[e-mail]>]

On Tue, Aug 20, 2024 at 4:49 AM Jeff Layton <[e-mail]> wrote:

> On Thu, 2024-08-08 at 13:36 -0700, Rick Macklem wrote:
> > Hi,
> >
> > Over the years, I've run into cases where it would be really
> > nice to be able to perform multiple NFSv4 operations on a
> > file without other operations done by other clients
> > "gumming up the works" by changing the file's data/metadata
> > between the operations in the compound.
> >
> > So, what do others think about an extension to NFSv4.2 that
> > adds 2 new operations:
> >   MUTEX_BEGIN(CFH)
> >   MUTEX_END(CFH)
> > Both would use the CFH as argument, and no other client would
> > be allowed to perform operations on the CFH between the MUTEX_BEGIN
> > and MUTEX_END.
> >
> > I think there would need to be a couple of properties for these:
> > - There would need to be an "implicit" MUTEX_END when any operation
> >   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
> > - I think you would want a restriction of only one mutex for one CFH
> >   at a time in a compound. Without that, there could easily be deadlocks
> >   caused by other compounds acquiring mutexes on the same CFHs in a
> >   different order.
> > - Only one compound can hold a mutex on a given CFH at any time.
> > - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
> >   is the first operation.
> > - All mutexes are discarded by a server when it crashes/recovers.
> >   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
> >   That way RPC retries after a server reboot should work ok, I think?
> >
> > I am not sure what the semantics for reading data/metadata should be,
> > but I was thinking that would be allowed to be done by compounds for
> > other clients for the CFH. If a client wanted to serialize against
> > other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
> >
> > I see this as useful in a variety of ways:
> > - The example in the previous email of:
> >   MUTEX_BEGIN
> >   NVERIFY acl_truform ACL_MODEL_NFS4
> >   SETATTR posix_access_acl
> >   MUTEX_END
> > - Append writing:
> >   MUTEX_BEGIN
> >   VERIFY size "offset in WRITE that follows"
> >   WRITE "offset" etc
> >   MUTEX_END
> > - A bunch of cases where NFSv4 lacks the postop_attributes
> >   that were in NFSv3.
> >   MUTEX_BEGIN
> >   WRITE
> >   GETATTR size, change,..
> >   MUTEX_END
> >
> > So, what do others think?
> > (This was obviously not possible without sessions.)
> >
>
> It's an interesting idea. Quite a bit different than LOCK/LOCKU for
> sure.
>
> The difficulty here will be in defining what activity this mutex
> blocks. Just other MUTEX_BEGINs, or is this a mandatory lock for all
> access to that filehandle?

I think that only other MUTEX_BEGINs would not be particularly
useful, since there will be clients doing RPCs without any MUTEX_BEGINs
for a long time.

My original thinking was all operations in other compounds that use
the FH (either CFH or SAVEFH) would block until MUTEX_END.
(I'd even like to include NFSv3 RPCs, but that might be a bridge too far?)

Assuming the latter, would READ or GETATTR
> activity also be blocked? If not, then we'll have to have a list of
> operations that are blocked by the mutex, and new operations will have
> to have their semantics spelled out clearly vs. MUTEX_BEGIN.
>
I suppose only blocking operations that modify the file (where the change
attribute changes) is possible. I do think this makes it more complex
to implement (a lot more difficult to implement for FreeBSD, at least)
and I am not sure which would be preferred?

For example, suppose a compound does:
MUTEX_BEGIN
SETATTR (size of 0)
WRITE
GETATTR (size, change,...)
MUTEX_END
And then another compound does:
READ
GETATTR (size, change,...)
--> If the READ is not blocked until MUTEX_END, it might see EOF
    for the file size of 0 and then see a size of non-zero for
    the GETATTR.
    --> Could happen now (without MUTEX_BEGIN/MUTEX_END),
        so not a big deal, but it *might* be
        better to block the READ in this case?

As far as implementing it, I do not know what other server
setups are, but for FreeBSD, doing the "block all operations
on the FH by other RPCs" is by far the easiest.
- FreeBSD uses a shared/exclusive lock on vnodes (think inodes)
  during each operation in a compound for CFH (and SAVEFH, if it
  exists). All the MUTEX_BEGIN needs to do is acquire the exclusive
  vnode lock for CFH and keep it until MUTEX_END (normally, it is
  unlocked after the operation).
  --> Using a shared lock on the vnode would allow READ/GETATTR etc
      to be done by other compounds when between MUTEX_BEGIN/MUTEX_END
      but, at least for FreeBSD, a safe upgrade to an exclusive lock
      (required by operations like WRITE) cannot be done.
      --> So, implementing the "only block operations that change change"
          would require some other (rather odd) locking mechanism layered
          on top of the vnode locking. Doable, but not fun.

Then there is the issue of avoiding deadlock...
I think there are operations, like RENAME, which cannot be put between
MUTEX_BEGIN/MUTEX_END without risk of deadlock (if the RENAME is using
two different directories).
I'll admit I have not worked through too many of these yet, but it looks
like the simplest way to avoid deadlock problems is to not allow
MUTEX_BEGIN to be done on directory FHs.
Does this sound like too severe a restriction?

rick




> --
> Jeff Layton <[e-mail]>
>