[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Rick Macklem <[e-mail]>]

On Tue, Aug 20, 2024 at 2:57 PM Jeff Layton <[e-mail]> wrote:

> On Tue, 2024-08-20 at 13:35 -0700, Rick Macklem wrote:
> >
> >
> > On Tue, Aug 20, 2024 at 1:29 PM Rick Macklem <[e-mail]>
> wrote:
> > >
> > >
> > > On Tue, Aug 20, 2024 at 4:49 AM Jeff Layton <[e-mail]>
> wrote:
> > > > On Thu, 2024-08-08 at 13:36 -0700, Rick Macklem wrote:
> > > > > Hi,
> > > > >
> > > > > Over the years, I've run into cases where it would be really
> > > > > nice to be able to perform multiple NFSv4 operations on a
> > > > > file without other operations done by other clients
> > > > > "gumming up the works" by changing the file's data/metadata
> > > > > between the operations in the compound.
> > > > >
> > > > > So, what do others think about an extension to NFSv4.2 that
> > > > > adds 2 new operations:
> > > > >   MUTEX_BEGIN(CFH)
> > > > >   MUTEX_END(CFH)
> > > > > Both would use the CFH as argument, and no other client would
> > > > > be allowed to perform operations on the CFH between the MUTEX_BEGIN
> > > > > and MUTEX_END.
> > > > >
> > > > > I think there would need to be a couple of properties for these:
> > > > > - There would need to be an "implicit" MUTEX_END when any operation
> > > > >   between MUTEX_BEGIN and MUTEX_END returns a status other than
> NFS_OK.
> > > > > - I think you would want a restriction of only one mutex for one
> CFH
> > > > >   at a time in a compound. Without that, there could easily be
> deadlocks
> > > > >   caused by other compounds acquiring mutexes on the same CFHs in a
> > > > >   different order.
> > > > > - Only one compound can hold a mutex on a given CFH at any time.
> > > > > - MUTEX_BEGIN/MUTEX_END can only be used in compounds where
> SEQUENCE
> > > > >   is the first operation.
> > > > > - All mutexes are discarded by a server when it crashes/recovers.
> > > > >   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
> > > > >   That way RPC retries after a server reboot should work ok, I
> think?
> > > > >
> > > > > I am not sure what the semantics for reading data/metadata should
> be,
> > > > > but I was thinking that would be allowed to be done by compounds
> for
> > > > > other clients for the CFH. If a client wanted to serialize against
> > > > > other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
> > > > >
> > > > > I see this as useful in a variety of ways:
> > > > > - The example in the previous email of:
> > > > >   MUTEX_BEGIN
> > > > >   NVERIFY acl_truform ACL_MODEL_NFS4
> > > > >   SETATTR posix_access_acl
> > > > >   MUTEX_END
> > > > > - Append writing:
> > > > >   MUTEX_BEGIN
> > > > >   VERIFY size "offset in WRITE that follows"
> > > > >   WRITE "offset" etc
> > > > >   MUTEX_END
> > > > > - A bunch of cases where NFSv4 lacks the postop_attributes
> > > > >   that were in NFSv3.
> > > > >   MUTEX_BEGIN
> > > > >   WRITE
> > > > >   GETATTR size, change,..
> > > > >   MUTEX_END
> > > > >
> > > > > So, what do others think?
> > > > > (This was obviously not possible without sessions.)
> > > > >
> > > >
> > > > It's an interesting idea. Quite a bit different than LOCK/LOCKU for
> > > > sure.
> > > >
> > > > The difficulty here will be in defining what activity this mutex
> > > > blocks. Just other MUTEX_BEGINs, or is this a mandatory lock for all
> > > > access to that filehandle?
> > > >
> > >
> > > I think that only other MUTEX_BEGINs would not be particularly
> > > useful, since there will be clients doing RPCs without any MUTEX_BEGINs
> > > for a long time.
> > >
> > > My original thinking was all operations in other compounds that use
> > > the FH (either CFH or SAVEFH) would block until MUTEX_END.
> > > (I'd even like to include NFSv3 RPCs, but that might be a bridge too
> far?)
> > >
> > > > Assuming the latter, would READ or GETATTR
> > > > activity also be blocked? If not, then we'll have to have a list of
> > > > operations that are blocked by the mutex, and new operations will
> have
> > > > to have their semantics spelled out clearly vs. MUTEX_BEGIN.
> > > >
> > >
> > > I suppose only blocking operations that modify the file (where the
> change
> > > attribute changes) is possible. I do think this makes it more complex
> > > to implement (a lot more difficult to implement for FreeBSD, at least)
> > > and I am not sure which would be preferred?
> > >
> > > For example, suppose a compound does:
> > > MUTEX_BEGIN
> > > SETATTR (size of 0)
> > > WRITE
> > > GETATTR (size, change,...)
> > > MUTEX_END
> > > And then another compound does:
> > > READ
> > > GETATTR (size, change,...)
> > > --> If the READ is not blocked until MUTEX_END, it might see EOF
> > >     for the file size of 0 and then see a size of non-zero for
> > >     the GETATTR.
> > >     --> Could happen now (without MUTEX_BEGIN/MUTEX_END),
> > >         so not a big deal, but it *might* be
> > >         better to block the READ in this case?
> > >
> > > As far as implementing it, I do not know what other server
> > > setups are, but for FreeBSD, doing the "block all operations
> > > on the FH by other RPCs" is by far the easiest.
> > > - FreeBSD uses a shared/exclusive lock on vnodes (think inodes)
> > >   during each operation in a compound for CFH (and SAVEFH, if it
> > >   exists). All the MUTEX_BEGIN needs to do is acquire the exclusive
> > >   vnode lock for CFH and keep it until MUTEX_END (normally, it is
> > >   unlocked after the operation).
> > >   --> Using a shared lock on the vnode would allow READ/GETATTR etc
> > >       to be done by other compounds when between MUTEX_BEGIN/MUTEX_END
> > >       but, at least for FreeBSD, a safe upgrade to an exclusive lock
> > >       (required by operations like WRITE) cannot be done.
> > >       --> So, implementing the "only block operations that change
> change"
> > >           would require some other (rather odd) locking mechanism
> layered
> > >           on top of the vnode locking. Doable, but not fun.
> > >
> > > Then there is the issue of avoiding deadlock...
> > > I think there are operations, like RENAME, which cannot be put between
> > > MUTEX_BEGIN/MUTEX_END without risk of deadlock (if the RENAME is using
> > > two different directories).
> > > I'll admit I have not worked through too many of these yet, but it
> looks
> > > like the simplest way to avoid deadlock problems is to not allow
> > > MUTEX_BEGIN to be done on directory FHs.
> > > Does this sound like too severe a restriction?
> > >
> >
> >
> > Oh, and I think operations like PUTFH, which changes the CFH need to be
> > avoided between MUTEX_BEGIN/MUTEX_END to avoid deadlock, as well.
> > If a compound does:
> > MUTEX_BEGIN (FH for foo)
> > PUTFH (FH for bar)
> > GETATTR
> > MUTEX_END
> > and another compound does:
> > MUTEX_BEGIN (FH for bar)
> > PUTFH (FH for foo)
> > GETATTR
> > MUTEX_END
> > --> They could deadlock.
> > Does not allowing PUTFH/PUTROOTFH/PUTPUBFH/RESTOREFH between MUTEX_BEGIN
> > and MUTEX_END sound like too severe a restriction?
>
> I don't think so, but there are other operations that can change the
> current_fh too -- OPEN, for instance. If I do a MUTEX_BEGIN on a
> directory and then do an OPEN (by name) in it, what happens to the
> mutex when the current_fh changes?
>
Yes, I think that any operation that changes CFH or uses both
CFH and SAVED_FH as arguments introduces the risk of deadlock.


> As far as deadlocks go, the simplest way to avoid them would be to just
> deny any nesting whatsoever, and only allow locking a single mutex in
> the compound.
>
If operations as above are not allowed between MUTEX_BEGIN/MUTEX_END
plus no nesting (I think I have mentioned that before), then I believe
deadlocks are avoied.


> We'd probably have to implement this as a per-fh thing in the Linux
> server too. One thing thing I've been wanting to build is a way to do a
> gated write using the change attr. This could make that possible:
>
> COMPOUND1:
> GETATTR (get the change attr)
> READ
>
> (modify the data in memory)
>
> COMPOUND2:
> MUTEX_BEGIN
> VERIFY
> WRITE
> MUTEX_END
>
> ...if the write fails, do the whole thing again.
>
> Synchronized, multi-host I/O without using file locks!
>
Yep, and you can do an append write using the same COMPOUND2,
but checking size == write-offset in the VERIFY.

Another one I am interested in is...
I really like Thomas's Layout_WCC, but I do like the fact
that it only works for NFSv3 DSs. This would allow a NFSv4
DS (assuming the DS supports MUTEX_BEGIN/MUTEX_END) to do
the same thing, I think?

rick

-- 
> Jeff Layton <[e-mail]>
>