Re: [nfsv4] Write-behind caching

[<[e-mail]>]

I think that want to address this issue without using the words
"unrealistic" or "optimal".  Things that you think are unrealistic
sometimes, in some sorts of environments, turn out to be common.  Trying
to decide what approaches are optimal are also troubling.  In different
situations, different approaches may be better or worse.  The protocol
needs to define the rules that the client and server have to obey and
they may make choices that result in results from optimal to pessimal.
We can make suggestions on doing things better but in unusual situations
the performance considerations may be different.  The point is we have
to be clear when the client can and can't do A and similarly for B and
sometimes the choice of A or B is simply up to the client.

So I'm going to put down the following numbered propositions and we'll
see where people disagree  with me.  Please be specific.  I'm going to
assume the anything you don't argue with numerically below the point of
disagreement is something we can agree on.

1) Normally, when a client is writing data covered by a layout, it may
write using the layout or to the MDS, but unless there is a particular
reason (e.g. slow or inconsistent response using the layout), it SHOULD
write using the layout.

2) When a layout is recalled, the protocol itself does not nor should it
require that dirty blocks in the cache be written before returning the
layout.  If a client chooses to do writes using the recalled layout, it
is doing so as an attempt to improve performance, given its judgment of
the relative performance of IO using the layout and IO through the MDS.

3) Particularly in the case in which clora_changed is 0, clients MAY
choose to take advantage of the higher-performance layout path to write
that data, while it is available.  However, since doing that delays the
return of the layout, it is possible that by delaying the return of the
layout, performance of others waiting for the layout may be reduced.   

4) When writing of dirty blocks is done using a layout being recalled,
the possibility exists that the layout will be revoked before all the
blocks are successfully written.  The client MUST be prepared to rewrite
those dirty blocks whose layouts write failed to the MDS in such cases. 

5) Clients that want to write dirty blocks associated with recalled
layouts MAY choose to restrict the size of the set of dirty blocks they
keep in order to make it relatively unlikely that the layout will be
revoked during recall.  On the other hand, for applications, in which
having a large set of dirty blocks in the cache reduces the IO actually
done, such restriction may result in poorer performance, even though the
specific IO path used is more performant. 

6) Note that if a large set of dirty blocks can be kept by the client
when a layout is not held, it should be possible to keep a set that at
least that size a set of dirty blocks when a layout is held.  Even if
the client should choose to write those blocks as part of the layout
recall, any that it is not able to write in an appropriate time, will be
a subset of an amount which, by hypothesis, can be appropriately held
when the only means of writing them is to the MDS. 

Another way of looking at this is that we have the following questions
which I'm going to present as multiple choice.  I have missed a few
choices but,

Q1) When a recall of a layout occurs what do you about dirty blocks?
    A) Nothing.  The IO's to write them are like any other IO and
       you don't do IO using layout under recall.
    B) You should write all dirty blocks as part of the recall if
       clora_changed is 0.
    C) You should do (A), but have the option of doing (B) but you 
       are responsible for the consequences.

Q2) How many dirty blocks should you keep covered by a layout?
    A) As many as you want.  It doesn't matter.
    B) A small number so that you can be sure that they can be
       written as part of layout recall (Assuming Q1=B).
    C) If there is a limit, it must be at least as great as the limit 
       that would be in effect if there is no layout present, since
       that number is OK, once the layout does go back.

There are pieces of the spec that are assuming (A) and the answer to
these and pieces assuming (B).

So I guess I was arguing before that the answers to Q1 and Q2 should be
(A).

My understanding is that Benny is arguing for (B) as the answer to Q1
and Q2.

So I'm now willing to compromise slightly and answer (C) to both of
those, but I think that still leaves me and Benny quite a ways apart.

I'm not sure what Trond's answer is, but I'd interested in understanding
his view in terms of (1)-(6) and Q1 and Q2. 


-----Original Message-----
From: Trond Myklebust [mailto:[e-mail]] 
Sent: Friday, October 29, 2010 6:04 PM
To: faibish, sorin
Cc: Noveck, David; [e-mail]; [e-mail];
[e-mail]
Subject: Re: [nfsv4] Write-behind caching

On Fri, 2010-10-29 at 17:48 -0400, Trond Myklebust wrote:
> On Fri, 2010-10-29 at 17:32 -0400, sfaibish wrote:
> > On Fri, 29 Oct 2010 13:39:55 -0400, Trond Myklebust  
> > <[e-mail]> wrote:
> > 
> > > On Fri, 2010-10-29 at 13:20 -0400, [e-mail] wrote:
> > >> There are two issues here with regard to handling of layout
recall.
> > >>
> > >> One is with regard to in-flight IO.  As Benny points out, you
cannot be  
> > >> sure that the in-flight IO can be completed in time to avoid the
MDS  
> > >> losing patience.  That should rarely be the case though, if
things are  
> > >> working right.  The client has to be prepared to deal with IO
failures  
> > >> due to layout revocation.  Any IO that was in flight and failed
because  
> > >> of layout revocation will need to be handled by being reissued to
the  
> > >> MDS.  Is there anybody that disagrees with that?
> > >>
> > >> The second issue concerns IO not in-flight (in other words, not
IO's  
> > >> yet but potential IO's) when the recall is received.  I just
don't see  
> > >> that it reasonable to start IO's using layout segments being
recalled  
> > >> (whether for dirty buffers or anything else).  Doing IO's to the
MDS is  
> > >> fine but there is no real need for the layout recall to specially

> > >> trigger them, whether clora_changed is set or not.
> > >
> > > This should be _very_ rare. Any cases where 2 clients are trying
to do
> > > conflicting I/O on the same data is likely to be either a
violation of
> > > the NFS cache consistency rules, or a scenario where it is in any
case
> > > more efficient to go through the MDS (e.g. writing to adjacent
records
> > > that share the same extent).
> > Well this is a different discussion: what was the reason for the
recall in
> > the first place. This is one usecase but there could be other
usecases
> > for the recall and we discuss here how to implement the protcol more
than
> > how to solve a real problem. My 2c
> 
> I strongly disagree. If this is an unrealistic scenario, then we don't
> have to care about devising an optimal strategy for it. The 'there
could
> be other usecases' scenario needs to be fleshed out before we can deal
> with it.

To clarify a bit what I mean: we MUST devise optimal strategies for
realistic and useful scenarios. It is entirely OPTIONAL to devise
optimal strategies for unrealistic ones.

If writing back all data before returning the layout causes protocol
issues because the server cannot distinguish between a bad client and
one that is waiting for I/O to complete, then my argument is that we're
in the second case: we don't have to optimise for it, and so it is safe
for the server to assume 'bad client'...

   Trond