[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Pali Rohár <[e-mail]>]

On Friday 09 August 2024 08:12:39 Rick Macklem wrote:
> On Fri, Aug 9, 2024 at 2:00 AM Pali Rohár <[e-mail]>
> wrote:
> 
> > Hello, if I understand correctly then this functionality could be
> > already possible via NFS v4.1 by using persistent reply cache of the
> > session.
> >
> > RFC 8881 section 2.10.6.5. says:
> >
> > "A persistent reply cache places certain demands on the server. The
> > execution of the sequence of operations (starting with SEQUENCE) and
> > placement of its results in the persistent cache MUST be atomic."
> >
> > It should be enough for a NFS v4.1 client to create a new session with
> > flag CREATE_SESSION4_FLAG_PERSIST and place those "atomic" operations
> > into that session.
> >
> Hmm. I think that "atomic" w.r.t. persistent sessions refers to the
> changes done to session state and the file system are committed to
> storage at one time, using some sort of log that allows partially completed
> compounds to be "rolled back" after a reboot such that the session either
> represents the state before the compound began or after it has been
> completed.

Yes, it sounds like that, it requires something like what databases are
using for consistency during execution of transactions.

> As an implementer I will note that many (maybe all extant) servers that do
> not
> support persistent sessions. (I know that the FreeBSD server does not.)

I think that I have not saw NFS v4.1 server which implements persistent
sessions properly, so it is really rare thing.

> However, I do not think the above precludes another client from performing
> operations on the same file concurrently (interspersed) with the operations
> on the compound.

I think that this is questionable as the description of "The execution
of the sequence of operations (starting with SEQUENCE) ... MUST be
atomic." is not fully clear. If we take atomicity as the real strict
atomicity then in your example between VERIFY and WRITE cannot be
executed some other operation which may change state of VERIFY and
WRITE.

Some future clarification for this functionality would be nice.

> An example case that MUTEX_BEGIN/MUTEX_END is meant to address might be
> an attempt to implement append writing (Client A and B both have the CFH
> set to the same file):
> (in temporal ordering)
> Client A                             Client B
> - without MUTEX_BEGIN/MUTEX_END
> VERIFY (size == N) returns NFS_OK
>                                      VERIFY (size == N) returns NFS_OK
> WRITE (at offset N) returns NFS_OK
>                                      WRITE (at offset N) returns NFS_OK
> --> Client B overwrites Client A's write
> - with MUTEX_BEGIN/MUTEX_END
> MUTEX_BEGIN returns NFS_OK
>                                      MUTEX_BEGIN - blocks (or replies
> NFS4ERR_LOCKED)
> VERIFY (size == N) returns NFS_OK
> WRITE (at offset N) returns NFS_OK
> MUTEX_END returns NFS_OK
>                                      MUTEX_BEGIN - returns NFS_OK
>                                      VERIFY (size == N) returns
> NFS4ERR_NOTSAME
>                                      --> this causes an implicit MUTEX_END
> 
> Now, it is true that, for writing, byte range locking could be used,
> but that adds overhead and requires that all writing do the locking
> (since most/all extant servers do advisory byte locking and the client
> has no way of knowing whether or not a server is doing advisory vs mandatory
> byte range locking).

For this purpose there are shared reservations. You can take DENY_WRITE
share reservations and you can be sure that nobody overwrite file during
holding of your shared reservation.

> rick
> 
> 
> > On Thursday 08 August 2024 13:36:25 Rick Macklem wrote:
> > > Hi,
> > >
> > > Over the years, I've run into cases where it would be really
> > > nice to be able to perform multiple NFSv4 operations on a
> > > file without other operations done by other clients
> > > "gumming up the works" by changing the file's data/metadata
> > > between the operations in the compound.
> > >
> > > So, what do others think about an extension to NFSv4.2 that
> > > adds 2 new operations:
> > >   MUTEX_BEGIN(CFH)
> > >   MUTEX_END(CFH)
> > > Both would use the CFH as argument, and no other client would
> > > be allowed to perform operations on the CFH between the MUTEX_BEGIN
> > > and MUTEX_END.
> > >
> > > I think there would need to be a couple of properties for these:
> > > - There would need to be an "implicit" MUTEX_END when any operation
> > >   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
> > > - I think you would want a restriction of only one mutex for one CFH
> > >   at a time in a compound. Without that, there could easily be deadlocks
> > >   caused by other compounds acquiring mutexes on the same CFHs in a
> > >   different order.
> > > - Only one compound can hold a mutex on a given CFH at any time.
> > > - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
> > >   is the first operation.
> > > - All mutexes are discarded by a server when it crashes/recovers.
> > >   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
> > >   That way RPC retries after a server reboot should work ok, I think?
> > >
> > > I am not sure what the semantics for reading data/metadata should be,
> > > but I was thinking that would be allowed to be done by compounds for
> > > other clients for the CFH. If a client wanted to serialize against
> > > other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
> > >
> > > I see this as useful in a variety of ways:
> > > - The example in the previous email of:
> > >   MUTEX_BEGIN
> > >   NVERIFY acl_truform ACL_MODEL_NFS4
> > >   SETATTR posix_access_acl
> > >   MUTEX_END
> > > - Append writing:
> > >   MUTEX_BEGIN
> > >   VERIFY size "offset in WRITE that follows"
> > >   WRITE "offset" etc
> > >   MUTEX_END
> > > - A bunch of cases where NFSv4 lacks the postop_attributes
> > >   that were in NFSv3.
> > >   MUTEX_BEGIN
> > >   WRITE
> > >   GETATTR size, change,..
> > >   MUTEX_END
> > >
> > > So, what do others think?
> > > (This was obviously not possible without sessions.)
> > >
> > > rick
> >