[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Rick Macklem <[e-mail]>]

On Thu, Aug 8, 2024 at 2:11 PM Tom Haynes <[e-mail]> wrote:

> I've been thinking of ATOMIC_BEGIN/ATOMIC_END.
>
> The difference for me is that they are not per-FH, but signifying all ops
> in between have to be done atomically.
>
I do not have a strong preference. The only concern I would have with a
global lock would be the
performance hit. For a busy server handling 100s of clients, I would think
blocking all the others
from doing operations could be a big hit. (Even using MUTEX_BEGIN for a
single FH could be a big hit
for something like WRITE/FILE_SYNC4.)

I think clients would have to be very careful to only atomic lock around
fast operations.


> Not much difference, except to work on two files, MUTEX_* has to do a
> dance with PUTFH.
>
> I.e., if you want COPY or CLONE to lock both files. Or you want to lock
> directories for a rename....
>
COPY can take a long time (the asynchronous case might be ok). I recently
had a bug report
for the FreeBSD server, where a COPY was taking 25sec. This turned out to
be a ZFS configuration
problem, but the time a synchronous COPY takes can still be pretty large.

The case of multiple concurrent mutex's is definitely asking for deadlock.
I'm not sure what locking
directories for rename would do. It is not a case I have thought about.


> Oh, how about a try until available flag. So if two compounds arrive at
> the same time, the second waits until the first is done?
>
I think a short wait is fine (in fact I just forgot to suggest that for the
previous post), but I think
setting an upper bound of something like 50msec would be appropriate.


> In general I support this.
>
Be careful, you might end up stuck writing it up. I hate writing these
drafts;-)

rick


> On Thu, Aug 8, 2024, 13:47 Rick Macklem <[e-mail]> wrote:
>
>> Oh, one thing I missed...
>>
>> On Thu, Aug 8, 2024 at 1:36 PM Rick Macklem <[e-mail]>
>> wrote:
>>
>>> Hi,
>>>
>>> Over the years, I've run into cases where it would be really
>>> nice to be able to perform multiple NFSv4 operations on a
>>> file without other operations done by other clients
>>> "gumming up the works" by changing the file's data/metadata
>>> between the operations in the compound.
>>>
>>> So, what do others think about an extension to NFSv4.2 that
>>> adds 2 new operations:
>>>   MUTEX_BEGIN(CFH)
>>>   MUTEX_END(CFH)
>>> Both would use the CFH as argument, and no other client would
>>> be allowed to perform operations on the CFH between the MUTEX_BEGIN
>>> and MUTEX_END.
>>>
>>> I think there would need to be a couple of properties for these:
>>> - There would need to be an "implicit" MUTEX_END when any operation
>>>   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
>>>
>> - When a mutex is held by another compound in progress for a an FH,
>>   MUTEX_BEGIN would reply with an error (NFS4ERR_LOCKED maybe).
>>
>>> - I think you would want a restriction of only one mutex for one CFH
>>>   at a time in a compound. Without that, there could easily be deadlocks
>>>   caused by other compounds acquiring mutexes on the same CFHs in a
>>>   different order.
>>> - Only one compound can hold a mutex on a given CFH at any time.
>>> - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
>>>   is the first operation.
>>> - All mutexes are discarded by a server when it crashes/recovers.
>>>   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
>>>   That way RPC retries after a server reboot should work ok, I think?
>>>
>>> I am not sure what the semantics for reading data/metadata should be,
>>> but I was thinking that would be allowed to be done by compounds for
>>> other clients for the CFH. If a client wanted to serialize against
>>> other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
>>>
>>> I see this as useful in a variety of ways:
>>> - The example in the previous email of:
>>>   MUTEX_BEGIN
>>>   NVERIFY acl_truform ACL_MODEL_NFS4
>>>   SETATTR posix_access_acl
>>>   MUTEX_END
>>> - Append writing:
>>>   MUTEX_BEGIN
>>>   VERIFY size "offset in WRITE that follows"
>>>   WRITE "offset" etc
>>>   MUTEX_END
>>> - A bunch of cases where NFSv4 lacks the postop_attributes
>>>   that were in NFSv3.
>>>   MUTEX_BEGIN
>>>   WRITE
>>>   GETATTR size, change,..
>>>   MUTEX_END
>>>
>>> So, what do others think?
>>> (This was obviously not possible without sessions.)
>>>
>>> rick
>>>
>>>
>>> _______________________________________________
>> nfsv4 mailing list -- [e-mail]
>> To unsubscribe send an email to [e-mail]
>>
>