[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Rick Macklem <[e-mail]>]

On Tue, Aug 20, 2024 at 3:50 PM Trond Myklebust <[e-mail]>
wrote:

> On Thu, 2024-08-08 at 13:36 -0700, Rick Macklem wrote:
>
> Hi,
>
> Over the years, I've run into cases where it would be really
> nice to be able to perform multiple NFSv4 operations on a
> file without other operations done by other clients
> "gumming up the works" by changing the file's data/metadata
> between the operations in the compound.
>
> So, what do others think about an extension to NFSv4.2 that
> adds 2 new operations:
>   MUTEX_BEGIN(CFH)
>   MUTEX_END(CFH)
> Both would use the CFH as argument, and no other client would
> be allowed to perform operations on the CFH between the MUTEX_BEGIN
> and MUTEX_END.
>
> I think there would need to be a couple of properties for these:
> - There would need to be an "implicit" MUTEX_END when any operation
>   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
> - I think you would want a restriction of only one mutex for one CFH
>   at a time in a compound. Without that, there could easily be deadlocks
>   caused by other compounds acquiring mutexes on the same CFHs in a
>   different order.
> - Only one compound can hold a mutex on a given CFH at any time.
> - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
>   is the first operation.
> - All mutexes are discarded by a server when it crashes/recovers.
>   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
>   That way RPC retries after a server reboot should work ok, I think?
>
>
> I think you also need to apply:
>
> - The mutex is discarded as soon as the COMPOUND completes.
>
> Otherwise, you're relying on the client to be a good citizen and clean up
> if one of the operations in the COMPOUND generated an error.
>
Yes. I did mention that in one of the emails.
It also needs to be discarded when the server reboots, so that
retries of RPCs won't be a problem after the reboot.

There is also the question of what to do w.r.t. very slow compounds?
Not sure if I have a good answer for that one yet.


>
> I am not sure what the semantics for reading data/metadata should be,
> but I was thinking that would be allowed to be done by compounds for
> other clients for the CFH. If a client wanted to serialize against
> other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
>
> I see this as useful in a variety of ways:
> - The example in the previous email of:
>   MUTEX_BEGIN
>   NVERIFY acl_truform ACL_MODEL_NFS4
>   SETATTR posix_access_acl
>   MUTEX_END
> - Append writing:
>   MUTEX_BEGIN
>   VERIFY size "offset in WRITE that follows"
>   WRITE "offset" etc
>   MUTEX_END
> - A bunch of cases where NFSv4 lacks the postop_attributes
>   that were in NFSv3.
>   MUTEX_BEGIN
>   WRITE
>   GETATTR size, change,..
>   MUTEX_END
>
> So, what do others think?
> (This was obviously not possible without sessions.)
>
>
> Questions:
>
> What happens in the case where another client holds a write delegation? Do
> you recall the delegation? I think you must, since the client is otherwise
> authoritative for some of these attributes that you want to keep stable.
>
Good point. I hadn't thought of this. I think you are correct that if the
server sees a GETATTR
between MUTEX_BEGIN/MUTEX_END for a file write delegated to another client,
requesting the attributes
handled by the client when write delegated (size/change/time_modify), it
might need to CB_RECALL.
(Of course, for the above examples using WRITE, another client couldn't be
holding a write delegation.)

Or maybe using the results of a CB_GETATTR might be sufficient, since that
client is
now authoritative for the attributes?
It is definitely a case that needs further thought.

What do others think?


> What happens if a pNFS layout is outstanding? Do you recall the layout?
> Again, I think you have to, since otherwise you cannot guarantee exclusive
> access or attribute stability.
>
Another case I haven't really thought about. I think you are correct.
If a RW layout has been issued to another client, I think the layout needs
to be recalled
when a WRITE/GETATTR/SETATTR (and maybe others) is done between
MUTEX_BEGIN/MUTEX_END and
no new RW layout issued to other clients until after the MUTEX_END.
But I definitely need to think about this some more.

Again, what do others think?


> Speaking of pNFS, how do you ensure that the above WRITE+GETATTR is
> possible for clients that are using pNFS? Do we need a
> 'LAYOUTGET_WITH_MUTEX' operation? Since layouts are recallable, it might be
> possible...
>
Another good point. RFC7862 allows some additional operations be done by
a NFSv4.2 DS (Sec. 3.3.1). Maybe this extension could add
MUTEX_BEGIN/MUTEX_END/GETATTR
to that list?

That way a client that is pNFS aware and using a NFSv4.2 DS could do the
same
things on the DS that it would otherwise do on the MDS. If the DS is not
NFSv4.2,
it would just live without this extension.

Any other ideas?

rick


>
> --
>
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [e-mail] <[e-mail]>
>
>
>