Re: [nfsv4] Write-behind caching

[sfaibish <[e-mail]>]

On Wed, 27 Oct 2010 10:52:00 -0400, <[e-mail]> wrote:

> Trond's item 3 seems good to me but I'm not understanding some
> assumptions that are being made.
>
> So suppose you have these partially updated blocks where you are
> writing.  It to me to be that it is OK to write in various ways under
> the following conditions.
>
> 1) You have the block layout not recalled:
>
>      a) It is OK to do an NFS WRITE of a partial block.
>
>      b) It is dangerous to do a block write because of the
>      possibility of corruption.
>
>      c) It would be foolish/erroneous to do an NFS WRITE of
>      the full block including the modified part and the part
>      read earlier.
>
> 2) When the block layout is being recalled:
>
>      a) It is still OK to do an NFS WRITE of the partial
>      block.
>
>      b) It is dangerous to do a block write because of the
>      possibility of corruption.
>
>      c) It would be foolish/erroneous to do an NFS WRITE of
>      the full block including the modified part and the part
>      read earlier.
>
> 3) When the block layout is not present
>
>      a) It is definitely OK to do an NFS WRITE of the partial
>      block.
>
>      b) Block write is not OK.
>
>      c) It would be foolish/erroneous to do an NFS WRITE of
>      the full block including the modified part and the part
>      read earlier.
>
> So to me the message is that when you have partial writes in your buffer
> cache you should keep track of them (small bit mask in these cases) and
> do the partial write.  When you give up your layout, you retain the
> right do the partial NFS WRITE.
>
> So it sounds to me there are two cases.
>
> If having the delegation gives you exclusive access (as if you had a
> delegation), then (2b) causes a possible delay problem and the
> appropriate warning is to avoid gathering up too many of those and
> delaying recalls only applies to partial writes.
>
> On the other hand, if it doesn't, then (1b) and (2b) are corruption
> sources anyway and that rather than do (2b), the important thing is the
> write when actually done be a (3a) rather than a (3b).
>
> I think the fundamental point is the one others have been making, which
> is that there is a difference between,
>
>      "If there are dirty blocks where specific
>      circumstances, make it advisable that they be
>      written before returning a layout, care should
>      be taken to avoid a significant accumulation of such
>      blocks which might unduly delay a pending recall
>      of a layout.  If possible, the client should ensure
>      that the writes can safely be done after the recall
>      is completed, so as avoid this sort of delay."
>
> And
>
>      "If there are dirty blocks, they have to be written
>      before returning a layout.  Care should be taken to
>      avoid an undue accumulation of dirty blocks since
>      writing these before return the layout might unduly
>      delay a pending recall of layout."
I can see an additional problem with this statement; depending on the  
amount
of dirty pages that are written before returning the layout the server  
might
get nervous (timeout pending) and either resend the layoutrecall or simply
start fencing the I/Os to DS regardless of the amount of dirty pages
accumulated at the client and that the client expects to write before
returning the layout. So, we might want to address to this problem
in the recommendation.

/Sorin

>
> With the first being more complicated and detailed but also accurate.
>
>
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf
> Of Trond Myklebust
> Sent: Wednesday, October 27, 2010 9:40 AM
> To: Jason Glasgow
> Cc: [e-mail]
> Subject: Re: [nfsv4] Write-behind caching
>
>
>
> On Wed, 2010-10-27 at 09:05 -0400, Jason Glasgow wrote:
>> With regards to block layout there are some specific implementation
>> issues that historically made it important to flush dirty data before
>> returning a layout.  Let me describe one that arose when implementing
>> MPFS at EMC.  I have not followed the UNIX implementations of the pNFS
>> block layout recently enough to know if this is still a concern.
>>
>>
>> Assume a page cache with page size 4K, a file system with block size
>> 8K and a file that is 8K long.  In many UNIX like operating systems
>> NFS3/MPFS implementations, a 2K write to the range 2K-4K would obtain
>> a layout for reading and writing, read the first 8K of the file into 2
>> pages in the kernel page cache.  It would then copy the 2K of data
>> from userspace and overwrite the later half of the first page.  When
>> it was time to write out the page, MPFS would write out the complete
>> 8K block.  If a layout were recalled between the time the data was
>> read from disk, and when it was written, it is possible that another
>> client would modify the range of the file from 0K to 2K.  Unless
>> specific care was taken when flushing the entire 8K block later, data
>> corruption would occur.
>>
>> There are two ways to avoid the problem.
>>
>>
>> 1. Keep track of the bytes ranges of a page (or file system block)
>> that are dirty and only perform the read-modify-write cycle while the
>> client holds the layout.  This can get messy if a clients writes every
>> other byte on a page.
>> 2.  Do not return a layout containing a dirty page until that page has
>> been written.
>
> 3. In order to ensure you can safely do read-modify-write in the block
> pNFS case, never request layout sizes that are not a multiple of the
> block size.
>
>> Perhaps this sheds some light on the original motivation.
>
> Yes, but the problem you are describing does not justify a _requirement_
> that the client flush out its data. Only that it consider the
> implications of corner cases such as the above.
>
> Trond
>
>
>> Regarding the block layout, I am entirely sympathetic to arguments
>> that a layout recall should only wait for outstanding writes complete,
>> and should not cause the client to initiate new writes.
>>
>>
>> -Jason
>>
>>
>>
>>
>> On Wed, Oct 27, 2010 at 1:07 AM, Dean Hildebrand
>> <[e-mail]> wrote:
>>         I remember at one time there was a thought that all dirty data
>>         would have to be written to disk when it receives a
>>         layoutrecall.  Once the data was written, it would send a
>>         layoutreturn.  I think this was the thinking before all the
>>         timing issues and other such things cropped up.  I assume
>>         someone wrote that as general advice, somehow thinking that
>>         responding to a layoutrecall was more important than actually
>>         achieving good write performance.
>>
>>         In this light, the analogy with delegreturn makes sense if you
>>         take a very specific example, but obviously not in general.
>>
>>         I would vote to just cut this text, as I think it is simply
>>         outdated.
>>         Dean
>>
>>
>>
>>         On 10/26/2010 3:34 AM, [e-mail] wrote:
>>                 That makes sense.  Let me take on this issue with
>>                 regard to the file
>>                 layout.  Are there volunteers to address it with
>>                 regard to block and
>>                 object?  It would be great if we could get together in
>>                 Beijing, discuss
>>                 this, and come to a joint conclusion to present to the
>>                 working group
>>                 (via email I mean).  I'm not planning trying to do
>>                 this before the
>>                 working group meeting.  In any case, I'm pretty sure
>>                 there won't be any
>>                 time during the working group meeting.
>>
>>                 -----Original Message-----
>>                 From: Spencer Shepler [mailto:[e-mail]]
>>                 Sent: Monday, October 25, 2010 11:34 PM
>>                 To: Noveck, David; [e-mail]
>>                 Subject: RE: [nfsv4] Write-behind caching
>>
>>
>>                 Fair enough.  I haven't looked to see if the layout
>>                 types
>>                 address this specific, needed, behavior.  Obviously
>>                 the
>>                 statement you reference and the individual layout
>>                 descriptions
>>                 should be tied together.  Again, I don't remember but
>>                 there
>>                 may be layout specific steps needed in the case of
>>                 handling
>>                 layoutreturns.
>>
>>                 In any case, we can handle the eventual conclusion as
>>                 an errata.
>>
>>                 Spencer
>>
>>
>>                         -----Original Message-----
>>                         From: [e-mail]
>>                         [mailto:[e-mail]]
>>                         Sent: Monday, October 25, 2010 8:25 PM
>>                         To: Spencer Shepler; [e-mail]
>>                         Subject: RE: [nfsv4] Write-behind caching
>>
>>                         I agree that the intent was to cover a variety
>>                         of layout types.
>>
>>                         I think what you are saying about the issue of
>>                         different throughputs
>>                 for
>>                         having and not having layouts also makes
>>                         sense.  It may in some way
>>                 have
>>                         led to the statement in RFC5661 but those
>>                         statements are by no means
>>                 the
>>                         same.  They have different consequences.  I
>>                         take it that you are
>>                 saying
>>                         (correctly) something like:
>>
>>                              However, write-behind implementations
>>                         will generally need to
>>                 bound
>>                              the amount of unwritten date so that
>>                         given the bandwidth of the
>>                              output path, the data can be written in a
>>                         reasonable time.
>>                 Clients
>>                              which have layouts should avoid keeping
>>                         larger amounts to reflect
>>                 a
>>                              situation in which a layout provides a
>>                         write path of higher
>>                         bandwidth.
>>                              This is because a CB_LAYOUTRECALL may be
>>                         received.  The client
>>                              should not delay returning the layout so
>>                         as to use that higher-
>>                         bandwidth
>>                              path, so it is best if it assumes, in
>>                         limiting the amount of data
>>                              to be written, that the write bandwidth
>>                         is only what is available
>>                              without the layout, and that it uses this
>>                         bandwidth assumption
>>                 even
>>                              if it does happen to have a layout.
>>
>>                         This differs from the text in RFC5661 in a few
>>                         respects.
>>
>>                                First it says that the amount of dirty
>>                         data should be the same
>>                 when
>>                                you have the layout and when you don't,
>>                         rather than simply
>>                 saying it
>>                                should be small when you have the
>>                         layout, possibly implying that
>>                 it
>>                                should be smaller than when you don't
>>                         have a layout.
>>
>>                                Second the text now in RFC5661 strongly
>>                         implies that when you
>>                 get
>>                                CB_LAYOUTRECALL, you would normally
>>                         start new IO's, rather than
>>                               simply drain the pending IO's and return
>>                         the layout ASAP.
>>
>>                         So I don't agree that what is in RFC5661 is
>>                         good implementation
>>                 advice,
>>                         particularly in suggesting that clients should
>>                         delay the LAYOUTRETURN
>>                         while doing a bunch of IO, including starting
>>                         new IO's.
>>
>>
>>                         -----Original Message-----
>>                         From: [e-mail]
>>                         [mailto:[e-mail]] On Behalf
>>                 Of
>>                         Spencer Shepler
>>                         Sent: Monday, October 25, 2010 10:07 PM
>>                         To: Noveck, David; [e-mail]
>>                         Subject: Re: [nfsv4] Write-behind caching
>>
>>
>>                         Since this description is part of the general
>>                         pNFS description, the
>>                 intent
>>                         may have been to cover a variety of layout
>>                         types.  However, I agree
>>                 that
>>                         the client is not guaranteed access to the
>>                         layout and is fully capable
>>                 of
>>                         writing the data via the MDS if all else fails
>>                         (inability to obtain
>>                 the
>>                         layout after a return); it may not be the most
>>                         performant path but it
>>                         should be functional.  And maybe that is the
>>                         source of the statement
>>                 that
>>                         the client should take care in managing its
>>                         dirty pages given the lack
>>                 of
>>                         guarantee of access to the supposed, higher
>>                         throughput path for
>>                 writing
>>                         data.
>>
>>                         As implementation guidance it seems okay but
>>                         truly a requirement for
>>                         correct function.
>>
>>                         Spencer
>>
>>                                 -----Original Message-----
>>                                 From: [e-mail]
>>                                 [mailto:[e-mail]] On
>>                 Behalf
>>                         Of
>>                                 [e-mail]
>>                                 Sent: Monday, October 25, 2010 6:58 PM
>>                                 To: [e-mail]
>>                                 Subject: [nfsv4] Write-behind caching
>>
>>                                 The following statement appears at the
>>                                 bottom of page 292 of
>>                 RFC5661.
>>                                    However, write-behind caching may
>>                                 negatively
>>                                    affect the latency in returning a
>>                                 layout in response to a
>>                                    CB_LAYOUTRECALL; this is similar to
>>                                 file delegations and the
>>                 impact
>>                                    that file data caching has on
>>                                 DELEGRETURN.  Client
>>                 implementations
>>                                    SHOULD limit the amount of
>>                                 unwritten data they have outstanding
>>                 at
>>                                    any one time in order to prevent
>>                                 excessively long responses to
>>                                    CB_LAYOUTRECALL.
>>
>>                                 This does not seem to make sense to
>>                                 me.
>>
>>                                 First of all the analogy between
>>                                 DELEGRETURN and
>>                                 CB_LAYOUTRECALL/LAYOUTRETURN doesn't
>>                                 seem to me to be correct.  In
>>                 the
>>                                 case of DELEGRETURN, at least if the
>>                                 file in question has been
>>                 closed,
>>                                 during the pendency of the delegation,
>>                                 you do need to write all of
>>                 the
>>                                 dirty data associated with those
>>                                 previously open files.  Normally,
>>                         clients
>>                                 just write all dirty data.
>>
>>                                 LAYOUTRETURN does not have that sort
>>                                 of requirement.  If it is valid
>>                         to
>>                                 hold the dirty data when you do have
>>                                 the layout, it is just as valid
>>                         to
>>                                 hold it when you don't.  You could
>>                                 very well return the layout and
>>                 get
>>                         it
>>                                 again before some of those dirty
>>                                 blocks are written.  Having a
>>                 layout
>>                                 grants you the right to do IO using a
>>                                 particular means (different
>>                         based on
>>                                 the mapping type), but if you don't
>>                                 have the layout, you still have
>>                 a
>>                         way
>>                                 to do the writeback, and there is no
>>                                 particular need to write back
>>                 all
>>                         the
>>                                 data before returning the layout.  As
>>                                 mentioned above, you may well
>>                         get
>>                                 the layout again before there is any
>>                                 need to actually do the
>>                         write-back.
>>                                 You have to wait until IO's that are
>>                                 in flight are completed before
>>                         you
>>                                 return the layout.  However, I don't
>>                                 see why you would have to or
>>                 want
>>                         to
>>                                 start new IO's using the layout if you
>>                                 have received a
>>                         CB_LAYOUTRECALL..
>>                                 Am I missing something?  Is there some
>>                                 valid reason for this
>>                         statement?
>>                                 Or should this be dealt with via the
>>                                 errata mechanism?
>>
>>                                 What do existing clients actually do
>>                                 with pending writeback data
>>                 when
>>                         they
>>                                 get a CB_LAYOUTRECALL?  Do they start
>>                                 new IO's using the layout?
>>                                 If so, is there any other reason other
>>                                 than the paragraph above?
>>
> _______________________________________________
>>                                 nfsv4 mailing list
>>                                 [e-mail]
>>
> https://www.ietf.org/mailman/listinfo/nfsv4
>>
> _______________________________________________
>>                         nfsv4 mailing list
>>                         [e-mail]
>>                         https://www.ietf.org/mailman/listinfo/nfsv4
>>
>>
>>                 _______________________________________________
>>                 nfsv4 mailing list
>>                 [e-mail]
>>                 https://www.ietf.org/mailman/listinfo/nfsv4
>>         _______________________________________________
>>         nfsv4 mailing list
>>         [e-mail]
>>         https://www.ietf.org/mailman/listinfo/nfsv4
>>
>>
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/nfsv4
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>



-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Unified Storage Division
         EMC²
where information lives

Phone: [phone]
Cellphone: 617-510-0422
Email : [e-mail]