[nfsv4] Re: RFC: new MUTEX_BEGIN/MUTEX_END operations

[Rick Macklem <[e-mail]>]

On Tue, Aug 20, 2024 at 2:03 PM Pali Rohár <[e-mail]>
wrote:

> On Thursday 08 August 2024 13:36:25 Rick Macklem wrote:
> > Hi,
> >
> > Over the years, I've run into cases where it would be really
> > nice to be able to perform multiple NFSv4 operations on a
> > file without other operations done by other clients
> > "gumming up the works" by changing the file's data/metadata
> > between the operations in the compound.
> >
> > So, what do others think about an extension to NFSv4.2 that
> > adds 2 new operations:
> >   MUTEX_BEGIN(CFH)
> >   MUTEX_END(CFH)
> > Both would use the CFH as argument, and no other client would
> > be allowed to perform operations on the CFH between the MUTEX_BEGIN
> > and MUTEX_END.
> >
> > I think there would need to be a couple of properties for these:
> > - There would need to be an "implicit" MUTEX_END when any operation
> >   between MUTEX_BEGIN and MUTEX_END returns a status other than NFS_OK.
> > - I think you would want a restriction of only one mutex for one CFH
> >   at a time in a compound. Without that, there could easily be deadlocks
> >   caused by other compounds acquiring mutexes on the same CFHs in a
> >   different order.
> > - Only one compound can hold a mutex on a given CFH at any time.
> > - MUTEX_BEGIN/MUTEX_END can only be used in compounds where SEQUENCE
> >   is the first operation.
> > - All mutexes are discarded by a server when it crashes/recovers.
> >   (Any time a client receives a NFS4ERR_STALE_CLIENTID.)
> >   That way RPC retries after a server reboot should work ok, I think?
> >
> > I am not sure what the semantics for reading data/metadata should be,
> > but I was thinking that would be allowed to be done by compounds for
> > other clients for the CFH. If a client wanted to serialize against
> > other compounds for the CFH, it could do a MUTEX_BEGIN/MUTEX_END.
> >
> > I see this as useful in a variety of ways:
> > - The example in the previous email of:
> >   MUTEX_BEGIN
> >   NVERIFY acl_truform ACL_MODEL_NFS4
> >   SETATTR posix_access_acl
> >   MUTEX_END
> > - Append writing:
> >   MUTEX_BEGIN
> >   VERIFY size "offset in WRITE that follows"
> >   WRITE "offset" etc
> >   MUTEX_END
> > - A bunch of cases where NFSv4 lacks the postop_attributes
> >   that were in NFSv3.
> >   MUTEX_BEGIN
> >   WRITE
> >   GETATTR size, change,..
> >   MUTEX_END
> >
> > So, what do others think?
> > (This was obviously not possible without sessions.)
> >
> > rick
>
> Hello,
>
> Now I'm thinking more about this and those mutexes looks to be
> performance killer for any NFS4 server which allows thousands of
> parallel operations. And this can open also vectors for DDOS attacks
> if servers are implemented not securely where malicious clients take
> mutexes and blocks any operations.


> Would not it be better to address existing problems by separate
> mechanisms? For example append issue can be solved by new append
> operation which NFS4 server can implement more optimized (e.g. if its
> storage or API already provides append operation, which applies for all
> POSIX systems via open/O_APPEND).
>
An Append_Write is not something I know how to do correctly.
After a server reboot, there can be a retry, which can result in
duplicate writes. I'm assuming you are talking about a Write that
is defined as "done at EOF" instead of at "byte offset N".


> Also this mutex mechanism does not solve atomicity of operations in
> multiprotocol environment where other protocols without this kind of
> mutex co-exist together on the same storage (e.g. interop with SMB or
> NFS3). If you have implementation of NFS4 and Samba in different
> userspace processes then what can result is that the NFS4 mutex will
> hold other NFS4 compound operations, but would not hold Samba
> operations. So atomicity would not be guaranteed at all.
>
I suppose that is similar to NFSv3 RPCs. It is doable, but is
left up to the server implementer, I think?


> NFS4 servers do not have to be implemented in kernel where they can lock
> inode to prevent any other operation by other system parts (which
> implements mutex on filehandle). And from POSIX userspace such inode
> locking is highly impossible to implement.
>
If the NFS4 server is in userspace, it can implement some kind of
lock on the file handles (or however it handles file handles).
The kernel vnode lock is just a convenient mechanism for the kernel
based FreeBSD server.


> I just cannot imagine to how implement NFS4 server with support for this
> mutex with system fs storage just by using POSIX API.
>
pthreads have various locking primitives. (It is true that the locking
is only visible to the NFSv4 server and not to the rest of the server
system. Having said this, it would be someone familiar with a userspace
server like Ganesha to answer this.)


> Pali
>