[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Jeff Layton <[e-mail]>]

On Tue, 2024-08-20 at 13:35 -0700, Rick Macklem wrote:
> 
> 
> On Tue, Aug 20, 2024 at 1:29 PM Rick Macklem <[e-mail]> wrote:
> > 
> > 
> > On Tue, Aug 20, 2024 at 4:49 AM Jeff Layton <[e-mail]> wrote:
> > > On Thu, 2024-08-08 at 13:36 -0700, Rick Macklem wrote:
> > > > Hi,
> > > > 
> > > > Over the years, I've run into cases where it would be really
> > > > nice to be able to perform multiple NFSv4 operations on a
> > > > file without other operations done by other clients
> > > > "gumming up the works" by changing the file's data/metadata
> > > > between the operations in the compound.
> > > > 
> > > > So, what do others think about an extension to NFSv4.2 that
> > > > adds 2 new operations:
> > > >   MUTEX_BEGIN(CFH)
> > > >   MUTEX_END(CFH)
> > > > Both would use the CFH as argument, and no other client would
> > > > be allowed to perform operations on the CFH between the MUTEX_BEGIN
> > > > and MUTEX_END.
> > > > 
> > > > I think there would need to be a couple of properties for these:
> > > > - There would need to be an "implicit" MUTEX_END when any operation
> > > >   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
> > > > - I think you would want a restriction of only one mutex for one CFH
> > > >   at a time in a compound. Without that, there could easily be deadlocks
> > > >   caused by other compounds acquiring mutexes on the same CFHs in a
> > > >   different order.
> > > > - Only one compound can hold a mutex on a given CFH at any time.
> > > > - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
> > > >   is the first operation.
> > > > - All mutexes are discarded by a server when it crashes/recovers.
> > > >   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
> > > >   That way RPC retries after a server reboot should work ok, I think?
> > > > 
> > > > I am not sure what the semantics for reading data/metadata should be,
> > > > but I was thinking that would be allowed to be done by compounds for
> > > > other clients for the CFH. If a client wanted to serialize against
> > > > other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
> > > > 
> > > > I see this as useful in a variety of ways:
> > > > - The example in the previous email of:
> > > >   MUTEX_BEGIN
> > > >   NVERIFY acl_truform ACL_MODEL_NFS4
> > > >   SETATTR posix_access_acl
> > > >   MUTEX_END
> > > > - Append writing:
> > > >   MUTEX_BEGIN
> > > >   VERIFY size "offset in WRITE that follows"
> > > >   WRITE "offset" etc
> > > >   MUTEX_END
> > > > - A bunch of cases where NFSv4 lacks the postop_attributes
> > > >   that were in NFSv3.
> > > >   MUTEX_BEGIN
> > > >   WRITE
> > > >   GETATTR size, change,..
> > > >   MUTEX_END
> > > > 
> > > > So, what do others think?
> > > > (This was obviously not possible without sessions.)
> > > > 
> > > 
> > > It's an interesting idea. Quite a bit different than LOCK/LOCKU for
> > > sure.
> > > 
> > > The difficulty here will be in defining what activity this mutex
> > > blocks. Just other MUTEX_BEGINs, or is this a mandatory lock for all
> > > access to that filehandle? 
> > > 
> > 
> > I think that only other MUTEX_BEGINs would not be particularly
> > useful, since there will be clients doing RPCs without any MUTEX_BEGINs
> > for a long time.
> > 
> > My original thinking was all operations in other compounds that use
> > the FH (either CFH or SAVEFH) would block until MUTEX_END.
> > (I'd even like to include NFSv3 RPCs, but that might be a bridge too far?)
> > 
> > > Assuming the latter, would READ or GETATTR
> > > activity also be blocked? If not, then we'll have to have a list of
> > > operations that are blocked by the mutex, and new operations will have
> > > to have their semantics spelled out clearly vs. MUTEX_BEGIN.
> > > 
> > 
> > I suppose only blocking operations that modify the file (where the change
> > attribute changes) is possible. I do think this makes it more complex
> > to implement (a lot more difficult to implement for FreeBSD, at least)
> > and I am not sure which would be preferred?
> > 
> > For example, suppose a compound does:
> > MUTEX_BEGIN
> > SETATTR (size of 0)
> > WRITE
> > GETATTR (size, change,...)
> > MUTEX_END
> > And then another compound does:
> > READ
> > GETATTR (size, change,...)
> > --> If the READ is not blocked until MUTEX_END, it might see EOF
> >     for the file size of 0 and then see a size of non-zero for
> >     the GETATTR.
> >     --> Could happen now (without MUTEX_BEGIN/MUTEX_END),
> >         so not a big deal, but it *might* be
> >         better to block the READ in this case?
> > 
> > As far as implementing it, I do not know what other server
> > setups are, but for FreeBSD, doing the "block all operations
> > on the FH by other RPCs" is by far the easiest.
> > - FreeBSD uses a shared/exclusive lock on vnodes (think inodes)
> >   during each operation in a compound for CFH (and SAVEFH, if it
> >   exists). All the MUTEX_BEGIN needs to do is acquire the exclusive
> >   vnode lock for CFH and keep it until MUTEX_END (normally, it is
> >   unlocked after the operation).
> >   --> Using a shared lock on the vnode would allow READ/GETATTR etc
> >       to be done by other compounds when between MUTEX_BEGIN/MUTEX_END
> >       but, at least for FreeBSD, a safe upgrade to an exclusive lock
> >       (required by operations like WRITE) cannot be done.
> >       --> So, implementing the "only block operations that change change"
> >           would require some other (rather odd) locking mechanism layered
> >           on top of the vnode locking. Doable, but not fun.
> > 
> > Then there is the issue of avoiding deadlock...
> > I think there are operations, like RENAME, which cannot be put between
> > MUTEX_BEGIN/MUTEX_END without risk of deadlock (if the RENAME is using
> > two different directories).
> > I'll admit I have not worked through too many of these yet, but it looks
> > like the simplest way to avoid deadlock problems is to not allow
> > MUTEX_BEGIN to be done on directory FHs.
> > Does this sound like too severe a restriction?
> > 
> 
> 
> Oh, and I think operations like PUTFH, which changes the CFH need to be
> avoided between MUTEX_BEGIN/MUTEX_END to avoid deadlock, as well.
> If a compound does:
> MUTEX_BEGIN (FH for foo)
> PUTFH (FH for bar)
> GETATTR
> MUTEX_END
> and another compound does:
> MUTEX_BEGIN (FH for bar)
> PUTFH (FH for foo)
> GETATTR
> MUTEX_END
> --> They could deadlock.
> Does not allowing PUTFH/PUTROOTFH/PUTPUBFH/RESTOREFH between MUTEX_BEGIN
> and MUTEX_END sound like too severe a restriction?

I don't think so, but there are other operations that can change the
current_fh too -- OPEN, for instance. If I do a MUTEX_BEGIN on a
directory and then do an OPEN (by name) in it, what happens to the
mutex when the current_fh changes?

As far as deadlocks go, the simplest way to avoid them would be to just
deny any nesting whatsoever, and only allow locking a single mutex in
the compound.

We'd probably have to implement this as a per-fh thing in the Linux
server too. One thing thing I've been wanting to build is a way to do a
gated write using the change attr. This could make that possible:

COMPOUND1:
GETATTR (get the change attr)
READ

(modify the data in memory)

COMPOUND2:
MUTEX_BEGIN
VERIFY
WRITE
MUTEX_END

...if the write fails, do the whole thing again.

Synchronized, multi-host I/O without using file locks!
-- 
Jeff Layton <[e-mail]>