Re: [nfsv4] More layout stuff: layout return on close

"Spencer Shepler" <spencer.shepler@gmail.com> Tue, 30 November 2010 23:00 UTC

Return-Path: <spencer.shepler@gmail.com>
X-Original-To: nfsv4@core3.amsl.com
Delivered-To: nfsv4@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 886F33A6C25 for <nfsv4@core3.amsl.com>; Tue, 30 Nov 2010 15:00:07 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.599
X-Spam-Level:
X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1vYHN4pXGx0j for <nfsv4@core3.amsl.com>; Tue, 30 Nov 2010 15:00:06 -0800 (PST)
Received: from mail-gx0-f172.google.com (mail-gx0-f172.google.com [209.85.161.172]) by core3.amsl.com (Postfix) with ESMTP id 0D9383A6B28 for <nfsv4@ietf.org>; Tue, 30 Nov 2010 15:00:05 -0800 (PST)
Received: by gxk28 with SMTP id 28so2859676gxk.31 for <nfsv4@ietf.org>; Tue, 30 Nov 2010 15:01:18 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:references :in-reply-to:subject:date:message-id:mime-version:content-type :content-transfer-encoding:x-mailer:thread-index:content-language; bh=lLZ1oosY7Asfx0vah7Xo7bZMk7J4u1FQQj8/v9r9HgY=; b=Ez3PCz54rm7G3hh8IqTZBsuk2scLQUQx1g1iCv7wrt5B1IRVArnNC1zw6wIUXaiFXp zIYDIYfT01DCZ+P6orNhlEoHK8V0SxEBZL5hWeSaZKPczLg9/RfAKC1+kG6dcsFs7sXQ 2pZJzXrdo0JbMCMzGSElyFPjVk/pWZ1SrqCW8=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:references:in-reply-to:subject:date:message-id :mime-version:content-type:content-transfer-encoding:x-mailer :thread-index:content-language; b=OLxualvjdEER6HYz6at6k+VzZwmqPP9a/UeV1WIEneGBQVHqPArA1+OYeqey7tHc7V ny6vDYbEtRUfoCRhocwB1lLh6+xAGQwV4qQT/alNXzLLOdKcL16kOEl99OAFL/4NNTsx Fhy2r/G5YYNvhbDzLuqP45WxwF5Ce0oufs3RA=
Received: by 10.100.107.18 with SMTP id f18mr5731067anc.61.1291158077920; Tue, 30 Nov 2010 15:01:17 -0800 (PST)
Received: from ssheplermobile (tide521.microsoft.com [131.107.0.91]) by mx.google.com with ESMTPS id t39sm7508225ano.7.2010.11.30.15.01.15 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 30 Nov 2010 15:01:16 -0800 (PST)
From: Spencer Shepler <spencer.shepler@gmail.com>
To: david.noveck@emc.com, Trond.Myklebust@netapp.com, iisaman@citi.umich.edu
References: <1290552459.30508.4.camel@heimdal.trondhjem.org><4CECC382.5020207@panasas.com> <4CECE104.5080009@desy.de><1290610592.3234.20.camel@heimdal.trondhjem.org><4CEE283D.5040903@desy.de><1290692948.2911.7.camel@heimdal.trondhjem.org><4CEE8AA9.1030109@desy.de><BF3BB6D12298F54B89C8DCC1E4073D8002A5F2B6@CORPUSMX50A.corp.emc.com><4CF25270.5050609@panasas.com><1290968847.12027.6.camel@heimdal.trondhjem.org><BF3BB6D12298F54B89C8DCC1E4073D8002A5F323@CORPUSMX50A.corp.emc.com><4CF514BA.50901@panasas.com><1291130418.5123.8.camel@heimdal.trondhjem.org><1291130781.5123.11.camel@heimdal.trondhjem.org><AANLkTi=oq5ZN6q8hAtyYCBPC9yYNX9EQBB0sbWGXyfdb@mail.gmail.com> <1291155032.2998.28.camel@heimdal.trondhjem.org> <BF3BB6D12298F54B89C8DCC1E4073D8002A5F6B9@CORPUSMX50A.corp.emc.com>
In-Reply-To: <BF3BB6D12298F54B89C8DCC1E4073D8002A5F6B9@CORPUSMX50A.corp.emc.com>
Date: Tue, 30 Nov 2010 15:01:13 -0800
Message-ID: <004b01cb90e2$7e15fd10$7a41f730$@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQLdJo3VhcQPmb8Q4mW0PRrEeIHisAHToPcPAo+fepICWD0BWQINto7IAgDjyJUByIbemwCjofnLAya3URYCYxZzzgLLQSj3AVjkVqoA4m/k2wFhU8p7AMIJgwECles11QNSmNbYkGjjY8A=
Content-Language: en-us
Cc: bhalevy@panasas.com, nfsv4@ietf.org
Subject: Re: [nfsv4] More layout stuff: layout return on close
X-BeenThere: nfsv4@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: NFSv4 Working Group <nfsv4.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/nfsv4>
List-Post: <mailto:nfsv4@ietf.org>
List-Help: <mailto:nfsv4-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 30 Nov 2010 23:00:07 -0000

Agreed.  The server is responsible for the state transitions and
as you point out, the client is given access to the content so the
ordering of its processing can be made to match the ordering the
server has provided.  Given the various latencies in network and processing
this is needed.

Spencer

> -----Original Message-----
> From: nfsv4-bounces@ietf.org [mailto:nfsv4-bounces@ietf.org] On Behalf Of
> david.noveck@emc.com
> Sent: Tuesday, November 30, 2010 2:48 PM
> To: Trond.Myklebust@netapp.com; iisaman@citi.umich.edu
> Cc: bhalevy@panasas.com; nfsv4@ietf.org
> Subject: Re: [nfsv4] More layout stuff: layout return on close
> 
> Let me just say I have to say with no quoting.  Once you have five sets of
> greater-than's, it just gets impossible.
> 
> We're talking about scrapping RFC5661?!  What I mean is Trond is talking
> about that, and we do feel his pain :-)  But still, it seems a bit, well,
> extreme.
> 
> My interpretation was (and maybe there is something in the spec which
> contradicts this) or it is in some way wrong is:
> 
>      The purpose of the seqid in the CB_LAYOUTRECALL
>      stateid is to define what version of the layout
>      state corpus being referred to, since, if
>      you the client don't know, you can't know what
>      I (the server) am talking about.
> 
>      So if you have a LAYOUTGET or return in flight
>      and the CB_LAYOUTRECALL is beyond the one I have
>      received (i.e. it is more than one more than the
>      last one I have received), I cannot process the
>      recall and should wait until I can.
> 
>      If I don't do that, my no-matching-layout is
>      of no value because it may mean that I just haven't
>      seen the layout that was recalled yet and so the
>      server could not deduce anything from my
>      no-matching-layout response.  Me being forgetful
>      is different from me lacking precognition :-)
> 
>      As has been noted, the CB_LAYOUTRECALL doesn't
>      change state, so it doesn't really make sense for
>      it to change the seqid.  I know this seems wrong to
>      Fred, and maybe it is wrong, but if it is, I think
>      we need a stronger argument, of the form "If you
>      don't do this, then this case breaks".
> 
>      The seqid sequencing applies to state transitions
>      that are made on the server (and provides for ordering
>      those successive versions of *server* layout state).
>      So, having somebody other than the server change the
>      seqid, seems wrong to me.  Maybe I'm wrong.
> 
>      If the server and the client are both changing the
>      seqid, I need good strong drink, before somebody
>      explains to me how that can result in anything but
>      chaos.  (Just one drink;  I really will listen).
> 
> It seems to me from the stuff I have heard recently is that many of the
> problems are not in the protocol per se (i.e. it is not full of mistakes),
> but rather in the explanation, which does need work to be clearer.  I
> certainly could be proven wrong.
> 
> 
> 
> -----Original Message-----
> From: nfsv4-bounces@ietf.org [mailto:nfsv4-bounces@ietf.org] On Behalf Of
> Trond Myklebust
> Sent: Tuesday, November 30, 2010 5:11 PM
> To: Fred Isaman
> Cc: Benny Halevy; nfsv4@ietf.org
> Subject: Re: [nfsv4] More layout stuff: layout return on close
> 
> On Tue, 2010-11-30 at 10:39 -0500, Fred Isaman wrote:
> > On Tue, Nov 30, 2010 at 10:26 AM, Trond Myklebust
> > <trond.myklebust@fys.uio.no> wrote:
> > > On Tue, 2010-11-30 at 10:20 -0500, Trond Myklebust wrote:
> > >> On Tue, 2010-11-30 at 17:14 +0200, Benny Halevy wrote:
> > >> > On 2010-11-29 01:23, david.noveck@emc.com wrote:
> > >> > > I think these two case are different.
> > >> > >
> > >> > > I think the CB_LAYOUTRECALL case does have sequencing issues
> but they
> > >> > > are dealt with, at least in the case of LAYOUTRECALL4_FILE, by
> the fact
> > >> > > that that the seqid in lor_stateid tells the client what
> revision of
> > >> > > layout state is being referred to, and thus implicitly tells
> the server
> > >> > > what layout changes in flight are or are not covered by the
> > >> > > NFS4ERR_NOMATCHING_LAYOUT.
> > >> > >
> > >> > > As Benny points out, the CLOSE doesn't have that, but I don't
> see an
> > >> > > ordering problem.  The seqid ordering mechanism is to make sure
> that the
> > >> > > client and the server agree on the corpus of layout-segments
> with
> > >> > > return-on-close best thought of as attribute for each segment
> part of
> > >> > > that layout segment corpus.  The close doesn't change that
> corpus in a
> > >> > > way that interferes with ongoing LAYOUT operations.  It merely
> > >> > > eliminates from active use those bearing the return-on-close
> attribute
> > >> > > and that shift can be done independently on the client and the
> server.
> > >> > >
> > >> > > Let's get more concrete here.  I don't see how there should be
> ordering
> > >> > > issues between close and LAYOUTRETURNs.  If the segment is
> returned
> > >> > > before the close or after the close, it is still gone.  I don't
> see any
> > >> > > perceivable difference.  Now let's take the case of LAYOUTGET
> and close.
> > >> > > Suppose you get a LAYOUTGET response, and it overlaps (for
> simplicity
> > >> > > let use say it is identical but I don't think the overlap case
> is really
> > >> > > any different).  What determines whether the client should
> treat this as
> > >> > > a segment he still has (post-close), is whether it has the
> > >> > > return-on-close flag set.  If it does, he should assume it is
> gone (and
> > >> > > it must have been sent before the close was processed).  If it
> doesn't
> > >> > > have that set he should assume it still exists, apart from
> ordering.  If
> > >> > > it was sent before the close, then the layout segment was
> upgraded to
> > >> > > non-open-specific, so it doesn't go away at close.  OTOH, if it
> was sent
> > >> > > after the close, then the segment went away at the close and
> was added
> > >> > > afterwards.  However, in either case, he (the client) should
> assume he
> > >> > > has still has it and the that server matches in that regard.
> > >> >
> > >> > How about the resulting layout stateid?
> > >> > What should the client use on the next layout operation?
> > >>
> > >> The same layout stateid he would have used if the server had
> recalled
> > >> that particular layout segment and he'd replied NOMATCHING_LAYOUT.
> > >>
> >
> > The CB_LAYOUTRECALL supplies a stateid.  Are you suggesting that the
> > client should not update its seqid to that of the recall when it
> > replies NOMATCHING_LAYOUT?  Because that seems wrong.
> 
> Oh crap! You're kidding me....
> 
> We invented a whole new damned convention for bloody stateid seqids ONLY
> for use in the damned CB_LAYOUTRECALL?
> 
> I've got a better idea. Let's scrap RFC5661 and start over again. This
> time, we pay attention to the existing conventions for stateids. One of
> those conventions is that if the seqid changes occur _only_ when a state
> change occurs.
> 
> No wonder we're having problems...
> 
> > >> IOW: the most recent stateid returned by LAYOUTGET, unless the
> client
> > >> has forgotten all layout segments for that file (i.e. the whole
> layout),
> > >> in which case it needs to use the open stateid.
> > >
> > > So, the thing to remember here is that the layout stateid is only
> useful
> > > for ordering LAYOUTGET w.r.t. LAYOUTRETURN. It has no other purpose.
> > >
> >
> > It is also for ordering of LAYOUTGET wrt CB_LAYOUTRECALL.  For all
> > intents and purposes, a return of NOMATCHING_LAYOUT to a
> > CB_LAYOUTRECALL is a LAYOUTRETURN, and ordering needs to be paid
> > attention to.
> 
> Right, but there is no state transition for a layoutrecall. Just like a
> delegation recall, it leaves the state unchanged as far as the client is
> concerned...
> 
> IOW: if we had made the damned LAYOUTRECALL take a stateid with the _same_
> seqid as the most recent LAYOUTGET that is being recalled, then we
> wouldn't be in this ridiculous debate.
> 
> 
> The point is: as far as the client is concerned, forgetting a layout
> segment is not a seqid-changing state transition. Apparently we have to do
> the seqid change on LAYOUTRECALL, but that is an aberration due to
> apparent blindness on the part of the spec reviewers (of which I was one,
> yes). There is no reason to introduce the same insanity everywhere else.
> 
> Trond
> --
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> Trond.Myklebust@netapp.com
> www.netapp.com
> 
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4