Re: [nfsv4] RFC 1813

I agree and Linux has definitely interpreted it smartly, here. 

To some extent, even making VERF cookie per mount basis is also just one more interpretation of 
" cookie must be  consistent during a single boot session" when the same NFS server can serve multiple exports.

I opened an errata against 7530 and 5661 RFCs to make it bit more clear for server implementations.

Thanks,
Viral

-----Original Message-----
From: Trond Myklebust [mailto:trondmy@gmail.com] 
Sent: Thursday, June 21, 2018 9:35 PM
To: Rick Macklem
Cc: Noveck, David; Mehta, Viral; brent.callaghan@eng.sun.com; Pawlowski, Brian; peter.staubach@eng.sun.com; Michael R Eisler; IETF NFSv4 WG Mailing List
Subject: Re: [nfsv4] RFC 1813 - NFS questions on write verifier

I’m not aware of anything in the current specs that state the client is allowed to assume that the writeverf is a global constant, which is why the Linux kernel tracks on a per-file basis. However it is clear from the semantics of COMMIT that the writeverf is required to be at least global to the file, since the client is required to match the cookies returned by each WRITE to those returned by other WRITEs and the final COMMIT.

That said, if there are clients such as FreeBSD out there that assume the verifier is global, then perhaps we should introduce errata to RFC5661 and RFC7530 that clarify that this is now a requirement upon server implementations.

> On Jun 21, 2018, at 08:24, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> 
> Just a data point...The FreeBSD client keeps the write verifier on a 
> per-mount basis and assumes that it will be the same for all Writes 
> unless the server has rebooted and requires the Writes to be redone.
> 
> rick
> 
> ________________________________________
> From: nfsv4 <nfsv4-bounces@ietf.org> on behalf of Noveck, David 
> <David.Noveck@netapp.com>
> Sent: Thursday, June 21, 2018 7:37:47 AM
> To: Mehta, Viral; brent.callaghan@eng.sun.com; Pawlowski, Brian; 
> peter.staubach@eng.sun.com; mike@eisler.com
> Cc: nfsv4@ietf.org
> Subject: Re: [nfsv4] RFC 1813 - NFS questions on write verifier
> 
> First of all, I’ve cc’d the v4 working group, which is a good idea in general when you ask questions about these specs, especially since most of the email addresses on your to-list are probably no longer active (e.g. anything ending eng.sun.com).
> 
> 
>  *   RFC 7530 is bit more clear,
> 
> I think it is less clear.  RFC1813 says you can’t do what you want to do.  (it say “must be consistent rather than “MUST BE consistent”, but RFC1813 is not a standards-track document).  In any case the write verifier handling was not designed to support what you want to do, even though client implementations might well work as you would wish.
> 
> 
>  *   The server returns a write verifier upon successful completion of the
>  *   COMMIT..  The write verifier is used by the client to determine if the
>  *   server has restarted or rebooted between the initial WRITE(s) and the
>  *   COMMIT..
> 
> Note that it says “the server”  implying there is just one.
> 
>  *   The client does this by comparing the write verifier
>  *   returned from the initial writes and the verifier returned by the
>  *   COMMIT operation.
> 
> “The write verifier” (singular) “returned from the initial writes” 
> only make sense if the restriction specified in RFC1813 is adhered to here as well.
> 
> 
>  *   So, the client would compare COMMIT verifier with Previous WRITE verifier.
> 
> Except that that are multiple previous WRITE verifiers.
> 
>  *   And that allows SERVER to send “different” VERIFIER on different WRITE Operations
> 
> It can send them, but it is not clear how clients are supposed to deal with them since the assumption (in both specs) is that a single server will return the same one on successive writes.
> 
> From: Mehta, Viral <Viral.Mehta@dell.com>
> Sent: Thursday, June 21, 2018 4:50 AM
> To: brent.callaghan@eng.sun.com; Pawlowski, Brian 
> <Brian.Pawlowski@netapp.com>; peter.staubach@eng.sun.com; Noveck, 
> David <David.Noveck@netapp.com>; mike@eisler.com
> Subject: RE: RFC 1813 - NFS questions on write verifier
> 
> RFC 7530 is bit more clear,
> 
> The server returns a write verifier upon successful completion of the
>   COMMIT.  The write verifier is used by the client to determine if the
>   server has restarted or rebooted between the initial WRITE(s) and the
>   COMMIT.  The client does this by comparing the write verifier
>   returned from the initial writes and the verifier returned by the
>   COMMIT operation.
> 
> 
> So, the client would compare COMMIT verifier with Previous WRITE verifier.
> And that allows SERVER to send “different” VERIFIER on different WRITE Operations.
> 
> From: Mehta, Viral
> Sent: Tuesday, June 12, 2018 10:32 AM
> To: 'brent.callaghan@eng.sun.com'; 'beepy@netapp.com'; 'peter.staubach@eng.sun.com'
> Cc: 'David.Noveck@netapp.com'; 'mike@eisler.com'
> Subject: RFC 1813 - NFS questions on write verifier
> 
> Hi,
> 
> I had a question on NFSv3 server behaviour for unstable WRITEs in clustered environment.
> 
> As we know,
> NFS client sends WRITE op with stable or unstable flag. If the client 
> sends WRITE op with UNSTABLE flag, and NFS server responses with the 
> same flag, it also sends VERIFIER cookie. Later, when NFS server receives COMMIT operation, it sends the same VERIFIER cookie back and so the NFS client is happy.
> 
> If in between WRITE and COMMIT operations, NFS server was crashed and 
> rebooted, it sends the new VERIFIER (most of the implementation choose to send boot time of the NFS server) and so the client should perform WRITE again and it is again happy.
> 
> It all works if there is single node where NFS server is running. What if NFS server is clusterized / running in scale-out environment ? I wanted to understand how is this VERIFIER maintained ?
> 
> For e.g., if one node “N1” in scale out receives NFS WRITE request and the UNSTABLE WRITE actually happened on other node “N2”.
> i.e., Node “N1” forwarded WRITE request to node “N2” since data was on the node “N2”.
> Now, if node “N2” crashes before it receives COMMIT, which VERIFIER is 
> sent back to the client ? RFC-1813 says that
> 
> 
> “This cookie must be
>         consistent during a single boot session and must be
>         unique between instances of the NFS version 3 protocol
>         server where uncommitted data may be lost.
> “
> 
> Since there may not be single NFS server instance, and if NFS server process is running on all scale out nodes. What is the verifier cookie that is sent back ?
> And when is that verifier cookie updated ?
> 
> PS – It looks like real NFS client implementations are tracking COOKIE on PER WRITE basis and not PER NFS Server instance bases as mentioned in RFC.
>         But, while implementing NFS server, is that fair enough to assume above scenario ?
> 
> 
> Thanks,
> Viral
> 
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4