[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Rick Macklem <[e-mail]>]

On Tue, Aug 20, 2024 at 1:29 PM Rick Macklem <[e-mail]> wrote:

>
>
> On Tue, Aug 20, 2024 at 4:49 AM Jeff Layton <[e-mail]>
> wrote:
>
>> On Thu, 2024-08-08 at 13:36 -0700, Rick Macklem wrote:
>> > Hi,
>> >
>> > Over the years, I've run into cases where it would be really
>> > nice to be able to perform multiple NFSv4 operations on a
>> > file without other operations done by other clients
>> > "gumming up the works" by changing the file's data/metadata
>> > between the operations in the compound.
>> >
>> > So, what do others think about an extension to NFSv4.2 that
>> > adds 2 new operations:
>> >   MUTEX_BEGIN(CFH)
>> >   MUTEX_END(CFH)
>> > Both would use the CFH as argument, and no other client would
>> > be allowed to perform operations on the CFH between the MUTEX_BEGIN
>> > and MUTEX_END.
>> >
>> > I think there would need to be a couple of properties for these:
>> > - There would need to be an "implicit" MUTEX_END when any operation
>> >   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
>> > - I think you would want a restriction of only one mutex for one CFH
>> >   at a time in a compound. Without that, there could easily be deadlocks
>> >   caused by other compounds acquiring mutexes on the same CFHs in a
>> >   different order.
>> > - Only one compound can hold a mutex on a given CFH at any time.
>> > - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
>> >   is the first operation.
>> > - All mutexes are discarded by a server when it crashes/recovers.
>> >   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
>> >   That way RPC retries after a server reboot should work ok, I think?
>> >
>> > I am not sure what the semantics for reading data/metadata should be,
>> > but I was thinking that would be allowed to be done by compounds for
>> > other clients for the CFH. If a client wanted to serialize against
>> > other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
>> >
>> > I see this as useful in a variety of ways:
>> > - The example in the previous email of:
>> >   MUTEX_BEGIN
>> >   NVERIFY acl_truform ACL_MODEL_NFS4
>> >   SETATTR posix_access_acl
>> >   MUTEX_END
>> > - Append writing:
>> >   MUTEX_BEGIN
>> >   VERIFY size "offset in WRITE that follows"
>> >   WRITE "offset" etc
>> >   MUTEX_END
>> > - A bunch of cases where NFSv4 lacks the postop_attributes
>> >   that were in NFSv3.
>> >   MUTEX_BEGIN
>> >   WRITE
>> >   GETATTR size, change,..
>> >   MUTEX_END
>> >
>> > So, what do others think?
>> > (This was obviously not possible without sessions.)
>> >
>>
>> It's an interesting idea. Quite a bit different than LOCK/LOCKU for
>> sure.
>>
>> The difficulty here will be in defining what activity this mutex
>> blocks. Just other MUTEX_BEGINs, or is this a mandatory lock for all
>> access to that filehandle?
>
> I think that only other MUTEX_BEGINs would not be particularly
> useful, since there will be clients doing RPCs without any MUTEX_BEGINs
> for a long time.
>
> My original thinking was all operations in other compounds that use
> the FH (either CFH or SAVEFH) would block until MUTEX_END.
> (I'd even like to include NFSv3 RPCs, but that might be a bridge too far?)
>
> Assuming the latter, would READ or GETATTR
>> activity also be blocked? If not, then we'll have to have a list of
>> operations that are blocked by the mutex, and new operations will have
>> to have their semantics spelled out clearly vs. MUTEX_BEGIN.
>>
> I suppose only blocking operations that modify the file (where the change
> attribute changes) is possible. I do think this makes it more complex
> to implement (a lot more difficult to implement for FreeBSD, at least)
> and I am not sure which would be preferred?
>
> For example, suppose a compound does:
> MUTEX_BEGIN
> SETATTR (size of 0)
> WRITE
> GETATTR (size, change,...)
> MUTEX_END
> And then another compound does:
> READ
> GETATTR (size, change,...)
> --> If the READ is not blocked until MUTEX_END, it might see EOF
>     for the file size of 0 and then see a size of non-zero for
>     the GETATTR.
>     --> Could happen now (without MUTEX_BEGIN/MUTEX_END),
>         so not a big deal, but it *might* be
>         better to block the READ in this case?
>
> As far as implementing it, I do not know what other server
> setups are, but for FreeBSD, doing the "block all operations
> on the FH by other RPCs" is by far the easiest.
> - FreeBSD uses a shared/exclusive lock on vnodes (think inodes)
>   during each operation in a compound for CFH (and SAVEFH, if it
>   exists). All the MUTEX_BEGIN needs to do is acquire the exclusive
>   vnode lock for CFH and keep it until MUTEX_END (normally, it is
>   unlocked after the operation).
>   --> Using a shared lock on the vnode would allow READ/GETATTR etc
>       to be done by other compounds when between MUTEX_BEGIN/MUTEX_END
>       but, at least for FreeBSD, a safe upgrade to an exclusive lock
>       (required by operations like WRITE) cannot be done.
>       --> So, implementing the "only block operations that change change"
>           would require some other (rather odd) locking mechanism layered
>           on top of the vnode locking. Doable, but not fun.
>
> Then there is the issue of avoiding deadlock...
> I think there are operations, like RENAME, which cannot be put between
> MUTEX_BEGIN/MUTEX_END without risk of deadlock (if the RENAME is using
> two different directories).
> I'll admit I have not worked through too many of these yet, but it looks
> like the simplest way to avoid deadlock problems is to not allow
> MUTEX_BEGIN to be done on directory FHs.
> Does this sound like too severe a restriction?
>

Oh, and I think operations like PUTFH, which changes the CFH need to be
avoided between MUTEX_BEGIN/MUTEX_END to avoid deadlock, as well.
If a compound does:
MUTEX_BEGIN (FH for foo)
PUTFH (FH for bar)
GETATTR
MUTEX_END
and another compound does:
MUTEX_BEGIN (FH for bar)
PUTFH (FH for foo)
GETATTR
MUTEX_END
--> They could deadlock.
Does not allowing PUTFH/PUTROOTFH/PUTPUBFH/RESTOREFH between MUTEX_BEGIN
and MUTEX_END sound like too severe a restriction?

rick


> rick
>
>
>
>
>> --
>> Jeff Layton <[e-mail]>
>>
>