[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Trond Myklebust <[e-mail]>]

On Thu, 2024-08-08 at 13:36 -0700, Rick Macklem wrote:
Hi,

Over the years, I've run into cases where it would be really
nice to be able to perform multiple NFSv4 operations on a
file without other operations done by other clients
"gumming up the works" by changing the file's data/metadata
between the operations in the compound.

So, what do others think about an extension to NFSv4.2 that
adds 2 new operations:
  MUTEX_BEGIN(CFH)
  MUTEX_END(CFH)
Both would use the CFH as argument, and no other client would
be allowed to perform operations on the CFH between the MUTEX_BEGIN
and MUTEX_END.

I think there would need to be a couple of properties for these:
- There would need to be an "implicit" MUTEX_END when any operation
  between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
- I think you would want a restriction of only one mutex for one CFH
  at a time in a compound. Without that, there could easily be deadlocks
  caused by other compounds acquiring mutexes on the same CFHs in a
  different order.
- Only one compound can hold a mutex on a given CFH at any time.
- MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
  is the first operation.
- All mutexes are discarded by a server when it crashes/recovers.
  (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
  That way RPC retries after a server reboot should work ok, I think?

I think you also need to apply:

- The mutex is discarded as soon as the COMPOUND completes.

Otherwise, you're relying on the client to be a good citizen and clean up if one of the operations in the COMPOUND generated an error.


I am not sure what the semantics for reading data/metadata should be,
but I was thinking that would be allowed to be done by compounds for
other clients for the CFH. If a client wanted to serialize against
other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.

I see this as useful in a variety of ways:
- The example in the previous email of:
  MUTEX_BEGIN
  NVERIFY acl_truform ACL_MODEL_NFS4
  SETATTR posix_access_acl
  MUTEX_END
- Append writing:
  MUTEX_BEGIN
  VERIFY size "offset in WRITE that follows"
  WRITE "offset" etc
  MUTEX_END
- A bunch of cases where NFSv4 lacks the postop_attributes
  that were in NFSv3.
  MUTEX_BEGIN
  WRITE
  GETATTR size, change,..
  MUTEX_END

So, what do others think?
(This was obviously not possible without sessions.)

Questions:

What happens in the case where another client holds a write delegation? Do you recall the delegation? I think you must, since the client is otherwise authoritative for some of these attributes that you want to keep stable.

What happens if a pNFS layout is outstanding? Do you recall the layout? Again, I think you have to, since otherwise you cannot guarantee exclusive access or attribute stability.

Speaking of pNFS, how do you ensure that the above WRITE+GETATTR is possible for clients that are using pNFS? Do we need a 'LAYOUTGET_WITH_MUTEX' operation? Since layouts are recallable, it might be possible...



--

Trond Myklebust
Linux NFS client maintainer, Hammerspace
[e-mail]<mailto:[e-mail]>