Re: [nfsv4] Possible change to Flex Files layout

Rick Macklem <rmacklem@uoguelph.ca> Fri, 11 March 2016 01:23 UTC

Return-Path: <rmacklem@uoguelph.ca>
X-Original-To: nfsv4@ietfa.amsl.com
Delivered-To: nfsv4@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 776D612DC5A for <nfsv4@ietfa.amsl.com>; Thu, 10 Mar 2016 17:23:07 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.222
X-Spam-Level:
X-Spam-Status: No, score=-4.222 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 19Sd8XboLceb for <nfsv4@ietfa.amsl.com>; Thu, 10 Mar 2016 17:23:04 -0800 (PST)
Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by ietfa.amsl.com (Postfix) with ESMTP id 49F4912DBD4 for <nfsv4@ietf.org>; Thu, 10 Mar 2016 17:23:04 -0800 (PST)
IronPort-PHdr: 9a23:d47juxepE2K1cb6tSgTP5636lGMj4u6mDksu8pMizoh2WeGdxc69YR7h7PlgxGXEQZ/co6odzbGG7OawCSdZvsnJmUtBWaIPfidNsd8RkQ0kDZzNImzAB9muURYHGt9fXkRu5XCxPBsdMs//Y1rPvi/6tmZKSV3BPAZ4bt74BpTVx5zukbvipNuDM04R22X1SIgxBSv1hD2ZjtMRj4pmJ/R54TryiVwMRd5rw3h1L0mYhRf265T41pdi9yNNp6BprJYYAu3SNp41Rr1ADTkgL3t9pIiy7UGCHkOz4S42X2MQg1JoCgzI8gCyCp3xuy/3qPtV1yyTPMmwRrcxD2eM9aBuHSXpgyRPEjcy82Xaj4QklqdSqxGlqhlX3onbfYyRLPo4daqLLoBSfnZIQssED38JOYi7dYZaSrNZZes=
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DPAQA3HeJW/61jaINehBBtBrohAQ2BbRcKhW4CgXgUAQEBAQEBAQFjJ4ItghQBAQEDAQEBASAERwsFCwIBCA4KAgINGQICJzACBBOIHAgOrjyPIwEBAQEBAQEBAgEBAQEBAQEVBHyFHIF7gkeEIgKDFoE6BY4oiRWFbIlzh2yFL4YCiGgCHgEBQoQCHi4BiRY8fgEBAQ
X-IronPort-AV: E=Sophos;i="5.24,318,1454994000"; d="scan'208";a="270315647"
Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 10 Mar 2016 20:23:02 -0500
Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C105F15F55D; Thu, 10 Mar 2016 20:23:02 -0500 (EST)
Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id h0PMP6vPmDmI; Thu, 10 Mar 2016 20:23:01 -0500 (EST)
Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 66A5B15F56E; Thu, 10 Mar 2016 20:23:01 -0500 (EST)
X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca
Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id fEUfQaLAmzu5; Thu, 10 Mar 2016 20:23:01 -0500 (EST)
Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 4866915F55D; Thu, 10 Mar 2016 20:23:01 -0500 (EST)
Date: Thu, 10 Mar 2016 20:23:01 -0500
From: Rick Macklem <rmacklem@uoguelph.ca>
To: Benny Halevy <bhalevy@gmail.com>
Message-ID: <1985271727.13393755.1457659381248.JavaMail.zimbra@uoguelph.ca>
In-Reply-To: <CA+isNR+w9Qw39jfayOnmPsU74kF+-bEUL0qTG-esqA=p1QqUew@mail.gmail.com>
References: <1244659329.10251491.1457494754382.JavaMail.zimbra@uoguelph.ca> <CAABAsM7hN_FSC7VUCvjOpCoJvii0PK7LvJjFy1u_dRiDtzxNZQ@mail.gmail.com> <2092494483.11935609.1457581578961.JavaMail.zimbra@uoguelph.ca> <CA+isNR+w9Qw39jfayOnmPsU74kF+-bEUL0qTG-esqA=p1QqUew@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Originating-IP: [172.17.95.12]
X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF44 (Win)/8.0.9_GA_6191)
Thread-Topic: Possible change to Flex Files layout
Thread-Index: errFvlnWN4AmKWljmzIFd2d3I+GGJw==
Archived-At: <http://mailarchive.ietf.org/arch/msg/nfsv4/3Lok2eSfmwHUlIQWJA9SVUwpRFs>
Cc: nfsv4 <nfsv4@ietf.org>
Subject: Re: [nfsv4] Possible change to Flex Files layout
X-BeenThere: nfsv4@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: NFSv4 Working Group <nfsv4.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nfsv4/>
List-Post: <mailto:nfsv4@ietf.org>
List-Help: <mailto:nfsv4-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 11 Mar 2016 01:23:07 -0000

Benny Halevy wrote:
> 
> 
> On Thu, Mar 10, 2016 at 5:46 AM, Rick Macklem < rmacklem@uoguelph.ca > wrote:
> 
> 
> 
> Trond Myklebust wrote:
> > Hi Rick,
> > 
> > On Tue, Mar 8, 2016 at 10:39 PM, Rick Macklem < rmacklem@uoguelph.ca >
> > wrote:
> > > Hi,
> > > 
> > > I am building an NFSv4.1 pNFS for FreeBSD using Flex Files and
> > > I've run into a case where it would be nice to make a small
> > > addition to the layout.
> > > 
> > > I have a case where the files are replicated over K of the N
> > > servers in the cluster. Which K of the N is on a per file basis.
> > > (I am using GlusterFS as a backend and this is its replication
> > > strategy, if I understand it correctly.)
> > > 
> > > When there are replicas, the files are basically mirrored, but
> > > writes only need to be done to one of the replicas/mirrors.
> > > (Again, if I understand what GlusterFS does.)
> > > 
> > > -- So, what do others think of adding a boolean for the layout
> > > that says "only one mirror needs to be written VS all mirrors
> > > need to be written".
> > > 
> > > I can live without this, but I think I would find it useful and
> > > I don't see a problem with adding it.
> > > 
> > > Thanks for your consideration of this, rick
> > 
> > I don't see any major problems with this request. We have the
> > ffl_flags field in the layout, and so adding a flag for this would be
> > trivial, and could be done without breaking existing clients. Those
> > existing implementations would presumably be inefficient, since they
> > would ignore the flag, and continue to write to all mirrors, but they
> > would not break.
> > 
> > One question I have, though, is how this would affect the
> > ffds_efficiency? The current expectation for that field is that it
> > applies only to the mirrored read case, where you want to select one
> > of many mirrors. Wouldn't we also want to extend that scope of that
> > field so that it can apply to writes?
> Although I don't have a strong opinion, I just stated it would be used
> for writes the same as reads for this case in the diff. (At least that
> was what I tried to say;-)
> 
> I also stated that the storage device had to make sure that all mirrors
> were updated before replying successfully to the write (but they don't
> have to have the data committed to stable storage until the associated
> Commit). Does this sound reasonable?
> 
> Hi Rick,
> 
> If the DS that the client wrote to responds with STABLE* then file data have
> to be
> on stable storage under any failure scenario. Bare in mind that the client
> will mark
> the dirty page as clean in this case and is free to drop it out of its data
> cache so it won't be able
> to rewrite it in a server restart case. Therefore, I wouldn't allow data on
> mirrors to be unstable
> in this case.
> 
To me, requiring the DS to reply STABLE to the write (which implies written to
stable storage on all mirrors) seems to be a very strong requirement.

Now, if you are saying that the DS may reply STABLE to indicate that all mirrors
have been updated, including to stable storage or the DS may reply UNSTABLE to
indicate that the mirrors may not be updated until the Commit, then I would
be fine with that.

After all, write sharing is a rare case and the client can use Commit to ensure
the update is done on all mirrors when that rare case is handled, I'll admit I
like the idea of that option being available. (I'd guess that most clients would
do the Commit before unlocking an exclusive byte range lock that overlaps the
byte range written, since write-sharing clients will normally use byte-range locking
to handle concurrency.)
--> Without the FF_FLAGS_WRITE_ONE_MIRROR flag, the client will need to write to
    all mirrors (and do Commits if the write reply is UNSTABLE), so I don't see
    the UNSTABLE write to one mirror too "soft" a requirement.

To be honest, at this time I have no idea if the NFSv3 server built into GlusterFS
does STABLE or UNSTABLE writes and what guarantees it provides w.r.t. when replicas
are updated, so I am trying to define this in such a way that I can support it.
(Btw, I don't really need this change. The default GlusterFS setup doesn't do
 replication and, therefore, won't use mirrors. I just thought it would be a nice
 feature to support.)

rick

> Benny
> 
> 
> 
> > Either way I think we'd need a proposal for how we might modify the
> > description in
> > https://tools.ietf.org/html/draft-ietf-nfsv4-flex-files-06#section-8.1
> > 
> I've attached a diff of the .txt version as a first stab at the change.
> 
> Thanks for your comments, rick
> 
> > Cheers
> > Trond
> > 
> 
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
>