Re: [nfsv4] Write-behind caching

[Benny Halevy <[e-mail]>]

On 2010-11-01 19:49, [e-mail] wrote:
> I think that want to address this issue without using the words
> "unrealistic" or "optimal".  Things that you think are unrealistic
> sometimes, in some sorts of environments, turn out to be common.  Trying
> to decide what approaches are optimal are also troubling.  In different
> situations, different approaches may be better or worse.  The protocol
> needs to define the rules that the client and server have to obey and
> they may make choices that result in results from optimal to pessimal.
> We can make suggestions on doing things better but in unusual situations
> the performance considerations may be different.  The point is we have
> to be clear when the client can and can't do A and similarly for B and
> sometimes the choice of A or B is simply up to the client.
> 
> So I'm going to put down the following numbered propositions and we'll
> see where people disagree  with me.  Please be specific.  I'm going to
> assume the anything you don't argue with numerically below the point of
> disagreement is something we can agree on.
> 
> 1) Normally, when a client is writing data covered by a layout, it may
> write using the layout or to the MDS, but unless there is a particular
> reason (e.g. slow or inconsistent response using the layout), it SHOULD
> write using the layout.

Agreed.

> 
> 2) When a layout is recalled, the protocol itself does not nor should it
> require that dirty blocks in the cache be written before returning the
> layout.  If a client chooses to do writes using the recalled layout, it
> is doing so as an attempt to improve performance, given its judgment of
> the relative performance of IO using the layout and IO through the MDS.

Agreed.

> 
> 3) Particularly in the case in which clora_changed is 0, clients MAY
> choose to take advantage of the higher-performance layout path to write
> that data, while it is available.  However, since doing that delays the
> return of the layout, it is possible that by delaying the return of the
> layout, performance of others waiting for the layout may be reduced.   

In some cases trading latency for throughput (i.e. batching) results in
higher system performance, so the trade off is not as simple as you
describe.  If the client has little enough data to write, the cost
of getting the layout again later to write it may be higher than using
the layout already in hand to flush the data before returning the layout.

Also, in a case there's contention and the layout ping-pongs (or flip-flops :)
between clients, they can choose to make forward progress, or they
can fall back to writing through the MDS.  This heuristic decision is
really layout specific.

Another case if for objects RAID-5 where we would definitely want to complete
writing a full stripe if we started writing parts of it but have not pulled
the trigger yet on the rest.  In this case, interrupting the stripe write
will result in inconsistent parity and I consider the whole stripe-wise
operation "in-flight" even if not all I/Os were actually started.

> 
> 4) When writing of dirty blocks is done using a layout being recalled,
> the possibility exists that the layout will be revoked before all the
> blocks are successfully written.  The client MUST be prepared to rewrite
> those dirty blocks whose layouts write failed to the MDS in such cases. 

Absolutely.

> 
> 5) Clients that want to write dirty blocks associated with recalled
> layouts MAY choose to restrict the size of the set of dirty blocks they
> keep in order to make it relatively unlikely that the layout will be
> revoked during recall.  

Another approach documented in the spec is periodically returning the 
layout in segments as the client clears its cache.  This will allow it to
show progress and prevent layout revocation.

> On the other hand, for applications, in which
> having a large set of dirty blocks in the cache reduces the IO actually
> done, such restriction may result in poorer performance, even though the
> specific IO path used is more performant. 

I didn't get that, sorry...

> 
> 6) Note that if a large set of dirty blocks can be kept by the client
> when a layout is not held, it should be possible to keep a set that at
> least that size a set of dirty blocks when a layout is held.  Even if
> the client should choose to write those blocks as part of the layout
> recall, any that it is not able to write in an appropriate time, will be
> a subset of an amount which, by hypothesis, can be appropriately held
> when the only means of writing them is to the MDS. 

Makes sense.

> 
> Another way of looking at this is that we have the following questions
> which I'm going to present as multiple choice.  I have missed a few
> choices but,
> 
> Q1) When a recall of a layout occurs what do you about dirty blocks?
>     A) Nothing.  The IO's to write them are like any other IO and
>        you don't do IO using layout under recall.
>     B) You should write all dirty blocks as part of the recall if
>        clora_changed is 0.
      B') like (B), just s/all/some/
      B'') like (B'), and send LAYOUTCOMMITs and LAYOUTRETURNs periodically
           as you go.

>     C) You should do (A), but have the option of doing (B) but you 
>        are responsible for the consequences.

      C' and C'' respectively :)

> 
> Q2) How many dirty blocks should you keep covered by a layout?
>     A) As many as you want.  It doesn't matter.

I don't think it matters much if if you do A, B' or B''.

>     B) A small number so that you can be sure that they can be
>        written as part of layout recall (Assuming Q1=B).
>     C) If there is a limit, it must be at least as great as the limit 
>        that would be in effect if there is no layout present, since
>        that number is OK, once the layout does go back.
> 
> There are pieces of the spec that are assuming (A) and the answer to
> these and pieces assuming (B).
> 
> So I guess I was arguing before that the answers to Q1 and Q2 should be
> (A).
> 
> My understanding is that Benny is arguing for (B) as the answer to Q1
> and Q2.

I hope my answers above make it clearer.

Benny

> 
> So I'm now willing to compromise slightly and answer (C) to both of
> those, but I think that still leaves me and Benny quite a ways apart.
> 
> I'm not sure what Trond's answer is, but I'd interested in understanding
> his view in terms of (1)-(6) and Q1 and Q2. 
> 
> 
> -----Original Message-----
> From: Trond Myklebust [mailto:[e-mail]] 
> Sent: Friday, October 29, 2010 6:04 PM
> To: faibish, sorin
> Cc: Noveck, David; [e-mail]; [e-mail];
> [e-mail]
> Subject: Re: [nfsv4] Write-behind caching
> 
> On Fri, 2010-10-29 at 17:48 -0400, Trond Myklebust wrote:
>> On Fri, 2010-10-29 at 17:32 -0400, sfaibish wrote:
>>> On Fri, 29 Oct 2010 13:39:55 -0400, Trond Myklebust  
>>> <[e-mail]> wrote:
>>>
>>>> On Fri, 2010-10-29 at 13:20 -0400, [e-mail] wrote:
>>>>> There are two issues here with regard to handling of layout
> recall.
>>>>>
>>>>> One is with regard to in-flight IO.  As Benny points out, you
> cannot be  
>>>>> sure that the in-flight IO can be completed in time to avoid the
> MDS  
>>>>> losing patience.  That should rarely be the case though, if
> things are  
>>>>> working right.  The client has to be prepared to deal with IO
> failures  
>>>>> due to layout revocation.  Any IO that was in flight and failed
> because  
>>>>> of layout revocation will need to be handled by being reissued to
> the  
>>>>> MDS.  Is there anybody that disagrees with that?
>>>>>
>>>>> The second issue concerns IO not in-flight (in other words, not
> IO's  
>>>>> yet but potential IO's) when the recall is received.  I just
> don't see  
>>>>> that it reasonable to start IO's using layout segments being
> recalled  
>>>>> (whether for dirty buffers or anything else).  Doing IO's to the
> MDS is  
>>>>> fine but there is no real need for the layout recall to specially
> 
>>>>> trigger them, whether clora_changed is set or not.
>>>>
>>>> This should be _very_ rare. Any cases where 2 clients are trying
> to do
>>>> conflicting I/O on the same data is likely to be either a
> violation of
>>>> the NFS cache consistency rules, or a scenario where it is in any
> case
>>>> more efficient to go through the MDS (e.g. writing to adjacent
> records
>>>> that share the same extent).
>>> Well this is a different discussion: what was the reason for the
> recall in
>>> the first place. This is one usecase but there could be other
> usecases
>>> for the recall and we discuss here how to implement the protcol more
> than
>>> how to solve a real problem. My 2c
>>
>> I strongly disagree. If this is an unrealistic scenario, then we don't
>> have to care about devising an optimal strategy for it. The 'there
> could
>> be other usecases' scenario needs to be fleshed out before we can deal
>> with it.
> 
> To clarify a bit what I mean: we MUST devise optimal strategies for
> realistic and useful scenarios. It is entirely OPTIONAL to devise
> optimal strategies for unrealistic ones.
> 
> If writing back all data before returning the layout causes protocol
> issues because the server cannot distinguish between a bad client and
> one that is waiting for I/O to complete, then my argument is that we're
> in the second case: we don't have to optimise for it, and so it is safe
> for the server to assume 'bad client'...
> 
>    Trond
> 
>