[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Rick Macklem <[e-mail]>]

On Fri, Aug 9, 2024 at 1:12 PM Rick Macklem <[e-mail]> wrote:

>
>
> On Fri, Aug 9, 2024 at 8:21 AM Chuck Lever III <[e-mail]>
> wrote:
>
>>
>>
>> > On Aug 8, 2024, at 4:36 PM, Rick Macklem <[e-mail]>
>> wrote:
>> >
>> > Hi,
>> >
>> > Over the years, I've run into cases where it would be really
>> > nice to be able to perform multiple NFSv4 operations on a
>> > file without other operations done by other clients
>> > "gumming up the works" by changing the file's data/metadata
>> > between the operations in the compound.
>> >
>> > So, what do others think about an extension to NFSv4.2 that
>> > adds 2 new operations:
>> >   MUTEX_BEGIN(CFH)
>> >   MUTEX_END(CFH)
>> > Both would use the CFH as argument, and no other client would
>> > be allowed to perform operations on the CFH between the MUTEX_BEGIN
>> > and MUTEX_END.
>> >
>> > I think there would need to be a couple of properties for these:
>> > - There would need to be an "implicit" MUTEX_END when any operation
>> >   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
>> > - I think you would want a restriction of only one mutex for one CFH
>> >   at a time in a compound. Without that, there could easily be deadlocks
>> >   caused by other compounds acquiring mutexes on the same CFHs in a
>> >   different order.
>> > - Only one compound can hold a mutex on a given CFH at any time.
>> > - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
>> >   is the first operation.
>> > - All mutexes are discarded by a server when it crashes/recovers.
>> >   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
>> >   That way RPC retries after a server reboot should work ok, I think?
>>
>> I favor a narrow reservation, as described here, rather than
>> blocking all server operations. The usage for multiple files
>> would be more complicated, as Tom says, but my intuition is
>> that the server's implementation of a mutex might be more
>> efficient for the single file, which might be the common case.
>>
>> Also, I think this needs to be explicitly tied to the client's
>> lease so that a mutex reservation evaporates automatically if
>> the client goes away.
>>
> Agreed. I did say something about NFS4ERR_STALE_CLIENTID,
> but probably didn't make it clear.
>
>
>> This could be used as a DoS vector; care should be taken to
>> describe how a server mitigates or prevents clients from
>> accidentally or maliciously blocking operations on a FH
>> permanently.
>>
> I mentioned an "implicit" MUTEX_END when an operation
> returns anything other than NFS_OK.
> This should also apply to any other case where the processing
> of the compound ends before a MUTEX_END (such as a client
> not putting one in the compound.
>
Another one is if the file object is deleted, there needs to be
an implicit MUTEX_END.

>
> I don't know if this is sufficient, but with a relatively short
> (<= 50msec) timeout on MUTEX_BEGIN, it might?
>
>
>>
>> > I am not sure what the semantics for reading data/metadata should be,
>> > but I was thinking that would be allowed to be done by compounds for
>> > other clients for the CFH. If a client wanted to serialize against
>> > other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
>>
> There is also the issue of what to do about NFSv4.0/4.1 (and NFSv3)?
> - All I can think of is that they would be allowed to continue processing
>   operations irrespective of the mutex, but others might have better ideas.
>
I am now thinking that requiring processing of any operations
NFSv4.0/4.1/4.2 that
uses the file object (CFH or SAVEFH) should block while another compound
holds
the mutex for the file. (Without this, the MUTEX would be pretty useless
when
clients using multiple minor versions are mounted on the server.)
I don't know about other server implementations, but this should be pretty
easy
to do for FreeBSD via vnode (represents a file) locking.

Since there seems to be some interest, I'll try prototyping it on FreeBSD.

Thanks for the comments, rick

And then there is the question of "does it apply to other compounds done
> by the same ClientID?
> - I am now thinking it should apply. I can see a client wanting to throw a
>   bunch of writes at the server, but would like to get "change,
> time_modify"
>   that applies to that write, by doing:
>    MUTEX_BEGIN
>    WRITE(UNSTABLE4)
>    GETATTR(of change, time_modify)
>    MUTEX_END
>   (A WRITE(UNSTABLE4) will not take long.)
>
> rick
>
> >
>> > I see this as useful in a variety of ways:
>> > - The example in the previous email of:
>> >   MUTEX_BEGIN
>> >   NVERIFY acl_truform ACL_MODEL_NFS4
>> >   SETATTR posix_access_acl
>> >   MUTEX_END
>> > - Append writing:
>> >   MUTEX_BEGIN
>> >   VERIFY size "offset in WRITE that follows"
>> >   WRITE "offset" etc
>> >   MUTEX_END
>> > - A bunch of cases where NFSv4 lacks the postop_attributes
>> >   that were in NFSv3.
>> >   MUTEX_BEGIN
>> >   WRITE
>> >   GETATTR size, change,..
>> >   MUTEX_END
>> >
>> > So, what do others think?
>> > (This was obviously not possible without sessions.)
>> >
>> > rick
>> >
>> >
>> > _______________________________________________
>> > nfsv4 mailing list -- [e-mail]
>> > To unsubscribe send an email to [e-mail]
>>
>> --
>> Chuck Lever
>>
>>
>>