Re: [nfsv4] Progressing RFC errata for RFC 5661

[Trond Myklebust <[e-mail]>]

On Thu, 19 Sep 2019 at 00:07, Rick Macklem <[e-mail]> wrote:
>
> Trond Myklebust wrote:
> >Rick,
> >
> >That errata predates most of the Linux pNFS client implementation. We
> >wrote the implementation to conform to the errata.
> >
> >So no. It's not a bug. It's a deliberate design based on a decision
> >that was discussed in the IETF WG, on the mailing list
> >
> >https://mailarchive.ietf.org/arch/msg/nfsv4/_KTtO6uz-MvRoStbhPuOXWZr6yI
> This actually appears to be a discussion related to the offset and length arguments for LayoutCommit, but...
> >

See Spencer's email:
https://mailarchive.ietf.org/arch/msg/nfsv4/tj4_tBSTLMEKqf43Wg_bv1K0TeI

It may have focussed on the offset and length stuff, but it was
intended to cover all the layoutcommit issues that were being
discussed at the time.

> >and in a special session of the IETF:
> >
> >https://mailarchive.ietf.org/arch/msg/nfsv4/Rpw9XCwCARxfaU4ym5L2TauV6ao
> Ok. This was long before I got around to implementing it, so I wouldn't have
> understood the implications.
> --> I would have been interested in hearing the rationale behind not doing
>       LayoutCommit for FILE_SYNC4 writes, since it seems to me that RFC-5661
>       had gotten it right when it required them.

It doesn't mention anything about FILE_SYNC4 in the context of
pNFS/files. All it says is that the server is indicating that the data
+ metadata changes have been persisted. Since the MDS/DS back end
protocol is not part of the pNFS/files protocol specification, but is
required to provide strong coupling of NFSv4.1 state, etc between the
MDS and DS, it allows the server to implement strong policies w.r.t.
data+metadata persistence. Since indeed the only server implementation
available at the time actually did persist data + metadata when it
returned FILE_SYNC4, then we needed that clarification.

Note that pNFS/flexfiles does not impose the same interpretation of
the WRITE flags, because the MDS/DS back end protocol is loosely
coupled (and is designed so that NFS can act as that back end
protocol).

Also note that the requirements of POSIX when it comes to O_SYNC
write()s, provides additional motivation here. O_SYNC requires both
data+metadata to be persisted when the write() system call returns.
You cannot get scale out behaviour if each O_SYNC write requires both
I/O to the DS, and then a LAYOUTCOMMIT to the MDS in order to persist
the file metadata. In fact, you can't even get basic NFSv4 protocol
consistency if you allow the file data to visibly change on the DS but
don't record that fact in the change attribute on the MDS. The generic
pNFS spec in RFC5661 section 12.5.4. is extremely oily about this
scenario and seems to put the onus on the clients for ensuring
data/metadata consistency. However it does not at all discuss what
should happen in the case where the client does not do this, or is
incapable of doing so due to a reboot, network partition or crash
other than to note in section 12.7.1. that some data may be lost (Yes,
but what about the metadata, Jim?).

> As I said, the FreeBSD server can handle this case, it just results in a lot of
> overhead synchronizing Size, Change, Time_Modify between MDS and DS
> whenever a RW layout is issued to a client for the file.
>
> Thanks for pointing this out, rick
>
> On Wed, 18 Sep 2019 at 12:39, Rick Macklem <[e-mail]> wrote:
> >
> > Mkrtchyan, Tigran wrote:
> > [stuff snipped]
> > >Hi Rick,
> > >
> > >here is the public link to errata
> > >
> > >https://www.rfc-editor.org/errata/eid2751
> > >
> > >Tigran.
> > Thanks Tigran.
> >
> > Ok, so now that I've read it I have to admit I think it is rewriting the RFC
> > to conform with what the Linux client does.
> >
> > I think this para. from Sec. 13.10 of RFC-5661 is clear:
> >    The NFSv4.1 protocol only provides close-to-open file data cache
> >    semantics; meaning that when the file is closed, all modified data is
> >    written to the server.  When a subsequent OPEN of the file is done,
> >    the change attribute is inspected for a difference from a cached
> >    value for the change attribute.  For the case above, this means that
> >    a LAYOUTCOMMIT will be done at close (along with the data WRITEs) and
> >    will update the file's size and change attribute.  Access from
> >    another client after that point will result in the appropriate size
> >    being returned.
> >
> > It states "will be done". It doesn't say anything about UNSTABLE4 vs FILE_SYNC4.
> > (I think most POSIX-like clients would consider the fsync(2) syscall to require
> >  the same treatment as "close" above, but that is a POSIX-specific client issue.)
> > I can see the argument that, since there is no "must" in the statement, that a
> > client can choose not to do this, but that would also imply that the client will
> > need to live with the consequences of it.
> >
> > I think the second sentence of the first para. of the errata is bogus:
> > For file layouts, WRITEs to a Data Server that return a stable_how4 value of
> > FILE_SYNC4 guarantee that data and file system metadata are on stable
> > storage.  This means that a LAYOUTCOMMIT is not needed in order to make the
> > data and metadata visible to the metadata server and other clients.
> >
> > Why?
> > The FILE_SYNC4 was returned by the DS. This would imply the DS
> > has committed data and metadata to stable storage on the DS.
> > However, I am not aware of anything in RFC-5661 that would imply that
> > the Size, Time_Modify and Change attributes or anything else must have
> > been updated or in stable storage on the MDS at this time.
> >
> > If a server does not require LayoutCommit operations for correct behaviour
> > then it can simply reply NFS4ERR_NOTSUPP (as I believe the Netapp filer
> > does) and the client then no longer needs to do them.
> >
> > If there is somewhere in RFC-5661 that it is stated that LayoutCommits are
> > not required when the DS replies FILE_SYNC4, then I missed it and there
> > is a problem with RFC-5661 that needs to be addressed.
> >
> > Otherwise, sorry, but it seems that the bug is in the Linux client implementation
> > and not RFC-5661.
> >
> > Is there a File Layout pNFS server implementation where the DSs return
> > FILE_SYNC4 that will break if the client does a LayoutCommit for this case?
> > (If so, then something may need to be done.)
> >
> > rick
> >
> >
> >