[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Rick Macklem <[e-mail]>]

On Fri, Aug 9, 2024 at 2:00 AM Pali Rohár <[e-mail]>
wrote:

> Hello, if I understand correctly then this functionality could be
> already possible via NFS v4.1 by using persistent reply cache of the
> session.
>
> RFC 8881 section 2.10.6.5. says:
>
> "A persistent reply cache places certain demands on the server. The
> execution of the sequence of operations (starting with SEQUENCE) and
> placement of its results in the persistent cache MUST be atomic."
>
> It should be enough for a NFS v4.1 client to create a new session with
> flag CREATE_SESSION4_FLAG_PERSIST and place those "atomic" operations
> into that session.
>
Hmm. I think that "atomic" w.r.t. persistent sessions refers to the
changes done to session state and the file system are committed to
storage at one time, using some sort of log that allows partially completed
compounds to be "rolled back" after a reboot such that the session either
represents the state before the compound began or after it has been
completed.
As an implementer I will note that many (maybe all extant) servers that do
not
support persistent sessions. (I know that the FreeBSD server does not.)

However, I do not think the above precludes another client from performing
operations on the same file concurrently (interspersed) with the operations
on the compound.
An example case that MUTEX_BEGIN/MUTEX_END is meant to address might be
an attempt to implement append writing (Client A and B both have the CFH
set to the same file):
(in temporal ordering)
Client A                             Client B
- without MUTEX_BEGIN/MUTEX_END
VERIFY (size == N) returns NFS_OK
                                     VERIFY (size == N) returns NFS_OK
WRITE (at offset N) returns NFS_OK
                                     WRITE (at offset N) returns NFS_OK
--> Client B overwrites Client A's write
- with MUTEX_BEGIN/MUTEX_END
MUTEX_BEGIN returns NFS_OK
                                     MUTEX_BEGIN - blocks (or replies
NFS4ERR_LOCKED)
VERIFY (size == N) returns NFS_OK
WRITE (at offset N) returns NFS_OK
MUTEX_END returns NFS_OK
                                     MUTEX_BEGIN - returns NFS_OK
                                     VERIFY (size == N) returns
NFS4ERR_NOTSAME
                                     --> this causes an implicit MUTEX_END

Now, it is true that, for writing, byte range locking could be used,
but that adds overhead and requires that all writing do the locking
(since most/all extant servers do advisory byte locking and the client
has no way of knowing whether or not a server is doing advisory vs mandatory
byte range locking).

rick


> On Thursday 08 August 2024 13:36:25 Rick Macklem wrote:
> > Hi,
> >
> > Over the years, I've run into cases where it would be really
> > nice to be able to perform multiple NFSv4 operations on a
> > file without other operations done by other clients
> > "gumming up the works" by changing the file's data/metadata
> > between the operations in the compound.
> >
> > So, what do others think about an extension to NFSv4.2 that
> > adds 2 new operations:
> >   MUTEX_BEGIN(CFH)
> >   MUTEX_END(CFH)
> > Both would use the CFH as argument, and no other client would
> > be allowed to perform operations on the CFH between the MUTEX_BEGIN
> > and MUTEX_END.
> >
> > I think there would need to be a couple of properties for these:
> > - There would need to be an "implicit" MUTEX_END when any operation
> >   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
> > - I think you would want a restriction of only one mutex for one CFH
> >   at a time in a compound. Without that, there could easily be deadlocks
> >   caused by other compounds acquiring mutexes on the same CFHs in a
> >   different order.
> > - Only one compound can hold a mutex on a given CFH at any time.
> > - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
> >   is the first operation.
> > - All mutexes are discarded by a server when it crashes/recovers.
> >   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
> >   That way RPC retries after a server reboot should work ok, I think?
> >
> > I am not sure what the semantics for reading data/metadata should be,
> > but I was thinking that would be allowed to be done by compounds for
> > other clients for the CFH. If a client wanted to serialize against
> > other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
> >
> > I see this as useful in a variety of ways:
> > - The example in the previous email of:
> >   MUTEX_BEGIN
> >   NVERIFY acl_truform ACL_MODEL_NFS4
> >   SETATTR posix_access_acl
> >   MUTEX_END
> > - Append writing:
> >   MUTEX_BEGIN
> >   VERIFY size "offset in WRITE that follows"
> >   WRITE "offset" etc
> >   MUTEX_END
> > - A bunch of cases where NFSv4 lacks the postop_attributes
> >   that were in NFSv3.
> >   MUTEX_BEGIN
> >   WRITE
> >   GETATTR size, change,..
> >   MUTEX_END
> >
> > So, what do others think?
> > (This was obviously not possible without sessions.)
> >
> > rick
>