Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close

["Sandeep Joshi" <[e-mail]>]

It seems like we agree that laycommit will be sent for file layout,
correct?
Or Should I file a defect on this?

For reference my original email below.

// START
In certain cases, I don't see layoutcommit on a file at all even after
doing many writes.



Client side operations:

open
write(s)
close


On server side (observed operations):

open
layoutget's
close


But, I do not see laycommit at all. In terms data written by client it
is about 4-5MB.

When does client issue laycommit? 
 
// END


Regards,

Sandeep

-----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf
Of Trond Myklebust
Sent: Thursday, July 08, 2010 2:16 PM
To: [e-mail]
Cc: [e-mail]; [e-mail]; [e-mail];
[e-mail]; [e-mail]; [e-mail]
Subject: Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close

On Thu, 2010-07-08 at 16:30 -0400, [e-mail] wrote:
> > Note that a LAYOUTRETURN can arrive without LAYOUTCOMMIT if the 
> > client hasn't written to the file.  I'm not sure what about the 
> > blocks case though, do you implicitly free up any provisionally 
> > allocated blocks that the client had not explicitly committed using
LAYOUTCOMMIT?
> 
> In principle, yes as the blocks are no longer promised to the client, 
> although lazy evaluation of this is an obvious optimization.
> 
> > >> "Upon receiving an OPEN, LOCK or a WANT_DELEGATION, the server 
> > >> must check that it has received LAYOUTCOMMITs from any other 
> > >> clients that may have the file open for writing. If it hasn't, 
> > >> then it MUST take some action to ensure that any file data 
> > >> changes are accompanied by a change
> > >                            ^ potentially visible
> > >> attribute update."
> > 
> > That should be OK as long as it's not for every GETATTR for the 
> > change, mtime, or size attributes.
> > 
> > >>
> > >> Then you can add the above suggestion without the offending 
> > >> caveat. Note however that it does break the "SHOULD NOT" 
> > >> admonition in section 18.32.4.
> > 
> > Better be safe than sorry in this rare error case.
> 
> I concur with Benny on both of the above - in essence, the unrecovered
client failure is a reason to potentially ignore the "SHOULD" (server
can't know whether it actually ignored the "SHOULD", hence better safe
than sorry).  We probably ought to find a someplace appropriate to add a
paragraph or two explaining this in one of the 4.2 documents.

Right. I'm only interested in fixing the close-to-open case. The case of
general GETATTR calls might be nice to fix too, but it should not be
essential in order to ensure that well-behaved applications continue to
work as expected.

Note, however, that legacy support for stateless protocols like NFSv2
and NFSv3 may be problematic: there is no equivalent of OPEN, and so the
server may have to do the above check on all NFSPROC2_GETATTR,
NFSPROC3_GETATTR, NFSPROC2_LOOKUP and NFSPROC3_LOOKUP requests.

   Trond

> Thanks,
> --David
> 
> 
> > -----Original Message-----
> > From: Benny Halevy [mailto:[e-mail]] On Behalf Of 
> > Benny Halevy
> > Sent: Thursday, July 08, 2010 12:00 PM
> > To: Trond Myklebust
> > Cc: Black, David; Noveck, David; Muntz, Daniel; 
> > [e-mail]; [e-mail]; [e-mail]; 
> > [e-mail]; [e-mail]
> > Subject: Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close
> > 
> > On Jul. 08, 2010, 2:14 +0300, Trond Myklebust
<[e-mail]> wrote:
> > > On Wed, 2010-07-07 at 19:09 -0400, Trond Myklebust wrote:
> > >> On Wed, 2010-07-07 at 18:52 -0400, Trond Myklebust wrote:
> > >>> On Wed, 2010-07-07 at 18:44 -0400, [e-mail] wrote:
> > >>>> Let me try this ...
> > >>>>
> > >>>> A correct client will always send LAYOUTCOMMIT.
> > >>>> Assume that the client is correct.
> > >>>> Hence if the LAYOUTCOMMIT doesn't arrive, something's failed.
> > >>>>
> > >>>> Important implication: No LAYOUTCOMMIT is an error/failure 
> > >>>> case.  It just has to work; it doesn't have to be fast.
> > >>>>
> > 
> > Note that a LAYOUTRETURN can arrive without LAYOUTCOMMIT if the 
> > client hasn't written to the file.  I'm not sure what about the 
> > blocks case though, do you implicitly free up any provisionally 
> > allocated blocks that the client had not explicitly committed using
LAYOUTCOMMIT?
> > 
> > >>>> Suggestion: If a client dies while holding writeable layouts 
> > >>>> that permit write-in-place, and the client doesn't reappear or 
> > >>>> doesn't reclaim those layouts, then the server should assume 
> > >>>> that the files involved were written before the client died, 
> > >>>> and set the file attributes accordingly as part of internally 
> > >>>> reclaiming the layout that the client has abandoned.
> > 
> > Of course. That's part of the server recovery.
> > 
> > >>>>
> > >>>> Caveat: It may take a while for the server to determine that 
> > >>>> the client has abandoned a layout.
> > 
> > That's two lease times after a respective CB_LAYOUTRECALL.
> > 
> > >>>>
> > >>>> This can result in false positives (file appears to be modified

> > >>>> when it
> > >>>> wasn't) but won't yield false negatives (file does not appear 
> > >>>> to be modified even though it was modified).
> > >>>
> > >>> OK... So we're going to have to turn off client side file 
> > >>> caching entirely for pNFS? I can do that...
> > >>>
> > >>> The above won't work. Think readahead...
> > >>
> > >> So... What can work, is if you modify it to work explicitly for 
> > >> close-to-open
> > >>
> > >> "Upon receiving an OPEN, LOCK or a WANT_DELEGATION, the server 
> > >> must check that it has received LAYOUTCOMMITs from any other 
> > >> clients that may have the file open for writing. If it hasn't, 
> > >> then it MUST take some action to ensure that any file data 
> > >> changes are accompanied by a change
> > >                            ^ potentially visible
> > >> attribute update."
> > 
> > That should be OK as long as it's not for every GETATTR for the 
> > change, mtime, or size attributes.
> > 
> > >>
> > >> Then you can add the above suggestion without the offending 
> > >> caveat. Note however that it does break the "SHOULD NOT" 
> > >> admonition in section 18.32.4.
> > 
> > Better be safe than sorry in this rare error case.
> > 
> > Benny
> > 
> > >>
> > >> Trond
> > >>
> > >>
> > >>> Trond
> > >>>
> > >>>> Thanks,
> > >>>> --David
> > >>>>
> > >>>>> -----Original Message-----
> > >>>>> From: [e-mail] [mailto:[e-mail]] 
> > >>>>> On Behalf
> > >>>> Of [e-mail]
> > >>>>> Sent: Wednesday, July 07, 2010 6:04 PM
> > >>>>> To: [e-mail]; Muntz, Daniel
> > >>>>> Cc: [e-mail]; [e-mail]; 
> > >>>>> [e-mail];
> > >>>> [e-mail];
> > >>>>> [e-mail]; [e-mail]
> > >>>>> Subject: Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close
> > >>>>>
> > >>>>>> Yes. I would agree that the client cannot rely on the updates

> > >>>>>> being
> > >>>> made
> > >>>>>> visible if it fails to send the LAYOUTCOMMIT. My point was 
> > >>>>>> simply
> > >>>> that a
> > >>>>>> compliant server MUST also have a valid strategy for dealing 
> > >>>>>> with
> > >>>> the
> > >>>>>> case where the client doesn't send it.
> > >>>>>
> > >>>>> So you are saying the updates "MUST be made visible" through 
> > >>>>> the server's valid strategy.  Is that right.
> > >>>>>
> > >>>>> And that the client cannot rely on that.  Why not, if the 
> > >>>>> server must have a valid strategy.
> > >>>>>
> > >>>>> Is this just prudent "belt and suspenders" design or what?
> > >>>>>
> > >>>>> It seems to me that if one side here is MUST (and the spec 
> > >>>>> needs to be clearer about what might or might not constitute a

> > >>>>> valid strategy),
> > >>>> then
> > >>>>> the other side should be SHOULD.
> > >>>>>
> > >>>>> If both sides are "MUST", then if things don't work out then 
> > >>>>> the
> > >>>> client
> > >>>>> and server can equally point to one another and say "It's his
fault".
> > >>>>>
> > >>>>> Am I missing something here?
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> -----Original Message-----
> > >>>>> From: [e-mail] [mailto:[e-mail]] 
> > >>>>> On Behalf Of Trond Myklebust
> > >>>>> Sent: Wednesday, July 07, 2010 5:01 PM
> > >>>>> To: Muntz, Daniel
> > >>>>> Cc: [e-mail]; [e-mail]; 
> > >>>>> [e-mail]; [e-mail]; [e-mail]; 
> > >>>>> [e-mail]
> > >>>>> Subject: Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close
> > >>>>>
> > >>>>> On Wed, 2010-07-07 at 16:39 -0400, [e-mail] wrote:
> > >>>>>> To bring this discussion full circle, since we agree that a
> > >>>> compliant
> > >>>>>> server can implement a scheme where written data does not 
> > >>>>>> become
> > >>>>> visible
> > >>>>>> until after a LAYOUTCOMMIT, do we also agree that 
> > >>>>>> LAYOUTCOMMIT is a "MUST" from a compliant client (independent
of layout type)?
> > >>>>>
> > >>>>> Yes. I would agree that the client cannot rely on the updates 
> > >>>>> being
> > >>>> made
> > >>>>> visible if it fails to send the LAYOUTCOMMIT. My point was 
> > >>>>> simply that
> > >>>> a
> > >>>>> compliant server MUST also have a valid strategy for dealing 
> > >>>>> with the case where the client doesn't send it.
> > >>>>>
> > >>>>> Cheers
> > >>>>>   Trond
> > >>>>>
> > >>>>>>   -Dan
> > >>>>>>
> > >>>>>>> -----Original Message-----
> > >>>>>>> From: [e-mail] [mailto:[e-mail]]

> > >>>>>>> On Behalf Of Trond Myklebust
> > >>>>>>> Sent: Wednesday, July 07, 2010 7:04 AM
> > >>>>>>> To: Benny Halevy
> > >>>>>>> Cc: [e-mail]; [e-mail]; Garth 
> > >>>>>>> Gibson; Brent Welch; NFSv4
> > >>>>>>> Subject: Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close
> > >>>>>>>
> > >>>>>>> On Wed, 2010-07-07 at 16:51 +0300, Benny Halevy wrote:
> > >>>>>>>> On Jul. 07, 2010, 16:18 +0300, Trond Myklebust
> > >>>>>>> <[e-mail]> wrote:
> > >>>>>>>>> On Wed, 2010-07-07 at 09:06 -0400, Trond Myklebust wrote:
> > >>>>>>>>>> On Wed, 2010-07-07 at 15:05 +0300, Benny Halevy wrote:
> > >>>>>>>>>>> On Jul. 06, 2010, 23:40 +0300, Trond Myklebust
> > >>>>>>> <[e-mail]> wrote:
> > >>>>>>>>>>>> On Tue, 2010-07-06 at 15:20 -0400, [e-mail]
> > >>>>> wrote:
> > >>>>>>>>>>>>> The COMMIT to the DS, ttbomk, commits data on the DS. 
> > >>>>>>>>>>>>> I
> > >>>> see it as
> > >>>>>>>>>>>>> orthogonal to updating the metadata on the MDS (but
> > >>>> perhaps I'm wrong).
> > >>>>>>>>>>>>> As sjoshi@bluearc mentioned, the LAYOUTCOMMIT provides

> > >>>>>>>>>>>>> a
> > >>>> synchronization
> > >>>>>>>>>>>>> point, so even if the non-clustered server does not 
> > >>>>>>>>>>>>> want
> > >>>> to update
> > >>>>>>>>>>>>> metadata on every DS I/O, the LAYOUTCOMMIT could also 
> > >>>>>>>>>>>>> be a
> > >>>> trigger to
> > >>>>>>>>>>>>> execute whatever synchronization mechanism the 
> > >>>>>>>>>>>>> implementer
> > >>>> wishes to put
> > >>>>>>>>>>>>> in the control protocol.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> As far as I'm aware, there are no exceptions in RFC5661
> > >>>> that would allow
> > >>>>>>>>>>>> pNFS servers to break the rule that any visible change 
> > >>>>>>>>>>>> to
> > >>>> the data must
> > >>>>>>>>>>>> be atomically accompanied with a change attribute
update.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Trond, I'm not sure how this rule you mentioned is
> > >>>> specified.
> > >>>>>>>>>>>
> > >>>>>>>>>>> See more in section 12.5.4 and 12.5.4.1. LAYOUTCOMMIT 
> > >>>>>>>>>>> and
> > >>>> change/time_modify
> > >>>>>>>>>>> in particular:
> > >>>>>>>>>>>
> > >>>>>>>>>>>    For some layout protocols, the storage device is able

> > >>>>>>>>>>> to
> > >>>> notify the
> > >>>>>>>>>>>    metadata server of the occurrence of an I/O; as a 
> > >>>>>>>>>>> result,
> > >>>> the change
> > >>>>>>>>>>>    and time_modify attributes may be updated at the 
> > >>>>>>>>>>> metadata
> > >>>> server.
> > >>>>>>>>>>>    For a metadata server that is capable of monitoring
> > >>>> updates to the
> > >>>>>>>>>>>    change and time_modify attributes, LAYOUTCOMMIT
> > >>>> processing is not
> > >>>>>>>>>>>    required to update the change attribute.  In this 
> > >>>>>>>>>>> case,
> > >>>> the metadata
> > >>>>>>>>>>>    server must ensure that no further update to the data

> > >>>>>>>>>>> has
> > >>>> occurred
> > >>>>>>>>>>>    since the last update of the attributes; file-based
> > >>>> protocols may
> > >>>>>>>>>>>    have enough information to make this determination or

> > >>>>>>>>>>> may
> > >>>> update the
> > >>>>>>>>>>>    change attribute upon each file modification.  This 
> > >>>>>>>>>>> also
> > >>>> applies for
> > >>>>>>>>>>>    the time_modify attribute.  If the server 
> > >>>>>>>>>>> implementation
> > >>>> is able to
> > >>>>>>>>>>>    determine that the file has not been modified since 
> > >>>>>>>>>>> the
> > >>>> last
> > >>>>>>>>>>>    time_modify update, the server need not update
> > >>>> time_modify at
> > >>>>>>>>>>>    LAYOUTCOMMIT.  At LAYOUTCOMMIT completion, the 
> > >>>>>>>>>>> updated
> > >>>> attributes
> > >>>>>>>>>>>    should be visible if that file was modified since the
> > >>>> latest previous
> > >>>>>>>>>>>    LAYOUTCOMMIT or LAYOUTGET
> > >>>>>>>>>>
> > >>>>>>>>>> I know. However the above paragraph does not state that 
> > >>>>>>>>>> the
> > >>>> server
> > >>>>>>>>>> should make those changes visible to clients other than 
> > >>>>>>>>>> the
> > >>>> one that is
> > >>>>>>>>>> writing.
> > >>>>>>>>>>
> > >>>>>>>>>> Section 18.32.4 states that writes will cause the
> > >>>> time_modified and
> > >>>>>>>>>> change attributes to be updated (if and only if the file 
> > >>>>>>>>>> data
> > >>>> is
> > >>>>>>>>>> modified). Several other sections rely on this behaviour,
> > >>>> including
> > >>>>>>>>>> section 10.3.1, section 11.7.2.2, and section 11.7.7.
> > >>>>>>>>>>
> > >>>>>>>>>> The only 'special behaviour' that I see allowed for pNFS 
> > >>>>>>>>>> is
> > >>>> in section
> > >>>>>>>>>> 13.10, which states that clients can't expect to see 
> > >>>>>>>>>> changes immediately, but that they must be able to expect
> > >>>> close-to-open
> > >>>>>>>>>> semantics to work. Again, if this is to be the case, then

> > >>>>>>>>>> the
> > >>>> server
> > >>>>>>>>>> _must_ be able to deal with the case where client 1 dies
> > >>>> before it can
> > >>>>>>>>>> issue the LAYOUTCOMMIT.
> > >>>>>>>>
> > >>>>>>>> Agreed.
> > >>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>> As I see it, if your server allows one client to read 
> > >>>>>>>>>>>> data
> > >>>> that may have
> > >>>>>>>>>>>> been modified by another client that holds a WRITE 
> > >>>>>>>>>>>> layout
> > >>>> for that range
> > >>>>>>>>>>>> then (since that is a visible data change) it should
> > >>>> provide a change
> > >>>>>>>>>>>> attribute update irrespective of whether or not a
> > >>>> LAYOUTCOMMIT has been
> > >>>>>>>>>>>> sent.
> > >>>>>>>>>>>
> > >>>>>>>>>>> the requirement for the server in WRITE's implementation
> > >>>> section
> > >>>>>>>>>>> is quite weak: "It is assumed that the act of writing 
> > >>>>>>>>>>> data
> > >>>> to a file will
> > >>>>>>>>>>> cause the time_modified and change attributes of the 
> > >>>>>>>>>>> file to
> > >>>> be updated."
> > >>>>>>>>>>>
> > >>>>>>>>>>> The difference here is that for pNFS the written data is

> > >>>>>>>>>>> not
> > >>>> guaranteed
> > >>>>>>>>>>> to be visible until LAYOUTCOMMIT.  In a broader sense,
> > >>>> assuming the clients
> > >>>>>>>>>>> are caching dirty data and use a write-behind cache,
> > >>>> application-written data
> > >>>>>>>>>>> may be visible to other processes on the same host but 
> > >>>>>>>>>>> not
> > >>>> to others until
> > >>>>>>>>>>> fsync() or close() - open-to-close semantics are the 
> > >>>>>>>>>>> only
> > >>>> thing the client
> > >>>>>>>>>>> guarantees, right?  Issuing LAYOUTCOMMIT on fsync() and
> > >>>> close() ensure the
> > >>>>>>>>>>> data is committed to stable storage and is visible to 
> > >>>>>>>>>>> all
> > >>>> other clients in
> > >>>>>>>>>>> the cluster.
> > >>>>>>>>>>
> > >>>>>>>>>> See above. I'm not disputing your statement that 'the 
> > >>>>>>>>>> written
> > >>>> data is
> > >>>>>>>>>> not guaranteed to be visible until LAYOUTCOMMIT'. I am
> > >>>> disputing an
> > >>>>>>>>>> assumption that 'the written data may be visible without 
> > >>>>>>>>>> an
> > >>>> accompanying
> > >>>>>>>>>> change attribute update'.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> In other words, I'd expect the following scenario to give 
> > >>>>>>>>> the
> > >>>> same
> > >>>>>>>>> results in NFSv4.1 w/pNFS as it does in NFSv4:
> > >>>>>>>>
> > >>>>>>>> That's a strong requirement that may limit the scalability 
> > >>>>>>>> of
> > >>>> the server.
> > >>>>>>>>
> > >>>>>>>> The spirit of the pNFS operations, at least from Panasas
> > >>>> perspective was that
> > >>>>>>>> the data is transient until LAYOUTCOMMIT, meaning it may or

> > >>>>>>>> may
> > >>>> not be visible
> > >>>>>>>> to clients other than the one who wrote it, and its 
> > >>>>>>>> associated
> > >>>> metadata MUST
> > >>>>>>>> be updated and describe the new data only on LAYOUTCOMMIT 
> > >>>>>>>> and
> > >>>> until then it's
> > >>>>>>>> undefined, i.e. it's up to the server implementation 
> > >>>>>>>> whether to
> > >>>> update it or not.
> > >>>>>>>>
> > >>>>>>>> Without locking, what do the stronger semantics buy you?
> > >>>>>>>> Even if a client verified the change_attribute new data may
> > >>>> become visible
> > >>>>>>>> at any time after the GETATTR if the file/byte range aren't
> > >>>> locked.
> > >>>>>>>
> > >>>>>>> There is no locking needed in the scenario below: it is 
> > >>>>>>> ordinary close-to-open semantics.
> > >>>>>>>
> > >>>>>>> The point is that if you remove the one and only way that 
> > >>>>>>> clients
> > >>>> have
> > >>>>>>> to determine whether or not their data caches are valid, 
> > >>>>>>> then they
> > >>>> can
> > >>>>>>> no longer cache data at all, and server scalability will be 
> > >>>>>>> shot
> > >>>> to
> > >>>>>>> smithereens anyway.
> > >>>>>>>
> > >>>>>>> Trond
> > >>>>>>>
> > >>>>>>>> Benny
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Client 1			Client 2
> > >>>>>>>>> ========			========
> > >>>>>>>>>
> > >>>>>>>>> OPEN foo
> > >>>>>>>>> READ
> > >>>>>>>>> CLOSE
> > >>>>>>>>> 				OPEN
> > >>>>>>>>> 				LAYOUTGET ...
> > >>>>>>>>> 				WRITE via DS
> > >>>>>>>>> 				<dies>...
> > >>>>>>>>> OPEN foo
> > >>>>>>>>> verify change_attr
> > >>>>>>>>> READ if above WRITE is visible CLOSE
> > >>>>>>>>>
> > >>>>>>>>> Trond
> > >>>>>>>>> _______________________________________________
> > >>>>>>>>> nfsv4 mailing list
> > >>>>>>>>> [e-mail]
> > >>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> _______________________________________________
> > >>>>>>> nfsv4 mailing list
> > >>>>>>> [e-mail]
> > >>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> nfsv4 mailing list
> > >>>>> [e-mail]
> > >>>>> https://www.ietf.org/mailman/listinfo/nfsv4
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> nfsv4 mailing list
> > >>>>> [e-mail]
> > >>>>> https://www.ietf.org/mailman/listinfo/nfsv4
> > >>>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe 
> > >> linux-nfs" in the body of a message to [e-mail] 
> > >> More majordomo info at  
> > >> http://vger.kernel.org/majordomo-info.html
> > >
> > >
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe 
> > > linux-nfs" in the body of a message to [e-mail] 
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4



_______________________________________________
nfsv4 mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/nfsv4