Re: [nfsv4] Possible change to Flex Files layout

David Noveck <davenoveck@gmail.com> Sat, 12 March 2016 15:34 UTC

Return-Path: <davenoveck@gmail.com>
X-Original-To: nfsv4@ietfa.amsl.com
Delivered-To: nfsv4@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id ADEBE12D5EA for <nfsv4@ietfa.amsl.com>; Sat, 12 Mar 2016 07:34:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.699
X-Spam-Level:
X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z0_M0IKVV8cu for <nfsv4@ietfa.amsl.com>; Sat, 12 Mar 2016 07:34:09 -0800 (PST)
Received: from mail-oi0-x231.google.com (mail-oi0-x231.google.com [IPv6:2607:f8b0:4003:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 34BC412D58C for <nfsv4@ietf.org>; Sat, 12 Mar 2016 07:34:09 -0800 (PST)
Received: by mail-oi0-x231.google.com with SMTP id m82so106114036oif.1 for <nfsv4@ietf.org>; Sat, 12 Mar 2016 07:34:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=+ZNiHxGvBkgYr3yu/iwvySIRGa4e7dvkso/Z4i262hc=; b=0gvEtuxezuyZ0giC5E5TCSKv9Qmdr5WPSt1/Lpv33HWBqoCZbV/g5e2L+xy0Ru4k2H yjQp17RDDZdiCUW09VuhXnbAmUxUzCCiwdOu3CVzNEx4dxUo7BTd0s1lf1RBDtjtKCkb AVB7OSxZB2llN9249TidpwXIUIBkl6vCwlDBYStt/oOY92Z8Lu+lbz4oeIpGz47hEIZw ZBNf9puWqQNX6jA/Ut57pEeFfRjH7dPnDHi/y0dy2C4Baiv0hfhceHBiZ4Nbk5d+BXLg LSot1h/6jfhkTd8CC39ytrRWNe5jrQPxXLxba5k1NEiI1pepoBFivxd9U+Z3SQcju5Su ndUA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=+ZNiHxGvBkgYr3yu/iwvySIRGa4e7dvkso/Z4i262hc=; b=NuG7EONVDzDUPP9UO7WWTPaLneeSmBlaWrJRcKk9M6xiS7iJwraOhz0fuiOHJ4xIqk DUZwq8lGLq0MGzPWMb1QfqGt+DXC1iY3ioxAEHF2PZkup0dyTlh1xMrLtgjt0bu7PAxl Kig/BHp11D+m3i+RWIt6R5dIOw5ALOHBU87Df2R6s+1jA+3g1+zIppMeB9FhO/u/Li0p 4jYYS2SDiHwPsH/yaZoQw+ICaS3KNzvAVa+Y69QrEnc276eBEo8OaG9GigROjJ/AC++H dSx3i+G36w0u6lvGN1Ezh/lCMzATYwcwlhaCGdx6rvZSvfxE+iTEGhmnRNossW6fZqgZ EsSA==
X-Gm-Message-State: AD7BkJIoW0iV9U0+qiLFib0yuoU6h3Sm5bdssNQtM95wElQK4bBB/nFNGkMP2VOV/B1HoVUPK3DJadDA+S0vNw==
MIME-Version: 1.0
X-Received: by 10.202.215.86 with SMTP id o83mr8839876oig.55.1457796848544; Sat, 12 Mar 2016 07:34:08 -0800 (PST)
Received: by 10.182.29.38 with HTTP; Sat, 12 Mar 2016 07:34:08 -0800 (PST)
In-Reply-To: <CA+isNRKnR8QdvDTk+_wcHJ2Gbg39PtR7kimG1SjUqt8aDgCQ6A@mail.gmail.com>
References: <1244659329.10251491.1457494754382.JavaMail.zimbra@uoguelph.ca> <CAABAsM7hN_FSC7VUCvjOpCoJvii0PK7LvJjFy1u_dRiDtzxNZQ@mail.gmail.com> <2092494483.11935609.1457581578961.JavaMail.zimbra@uoguelph.ca> <CA+isNR+w9Qw39jfayOnmPsU74kF+-bEUL0qTG-esqA=p1QqUew@mail.gmail.com> <1985271727.13393755.1457659381248.JavaMail.zimbra@uoguelph.ca> <CA+isNRJbnPe+eFHw+k-kAii_2BcC5qiwA0oQEy31iKasoipPdQ@mail.gmail.com> <1344440686.14694101.1457748366315.JavaMail.zimbra@uoguelph.ca> <CA+isNRKnR8QdvDTk+_wcHJ2Gbg39PtR7kimG1SjUqt8aDgCQ6A@mail.gmail.com>
Date: Sat, 12 Mar 2016 10:34:08 -0500
Message-ID: <CADaq8jf-_mWr1ibLqq7UtyDTto2aOmY4YVTvAMSpVRaksRndQA@mail.gmail.com>
From: David Noveck <davenoveck@gmail.com>
To: Benny Halevy <bhalevy@gmail.com>
Content-Type: multipart/alternative; boundary="001a113d3c304ff64c052ddbc70a"
Archived-At: <http://mailarchive.ietf.org/arch/msg/nfsv4/BOMSlszciRW2_Jw3HeTIYFcAGW0>
Cc: Rick Macklem <rmacklem@uoguelph.ca>, nfsv4 <nfsv4@ietf.org>
Subject: Re: [nfsv4] Possible change to Flex Files layout
X-BeenThere: nfsv4@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: NFSv4 Working Group <nfsv4.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nfsv4/>
List-Post: <mailto:nfsv4@ietf.org>
List-Help: <mailto:nfsv4-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 12 Mar 2016 15:34:12 -0000

> Aside for the stability of written data there is the requirement for
visibility.

Or at least potential requirements as to visibility.

> That is, a clustered server needs to mimic as much as possible the
functional behavior of a monolithic server.

True, but a lot of the deployments will not be clustered servers.  In fact,
in many cases in which one is replicating data, the two replicas will be
far enough apart that a natural disaster is unlikely to simultaneously
affect both.

In any case, pNFS does not strive to precisely emulate clustered server
semantics.  Instead, it seeks to provide many of the benefits of clustered
operation, while allowing requirements that depend on the clients being
aware of the distributed  nature of the arrangement.  Use of LAYOUTCOMMIT
is an example.

>With that in mind, the newly written data is visible to all clients in a
monolithic server regardless if its stable_how value. The application level
relies on that for close-to-open semantics.

True.

> This implies that in a clustered server configuration with multiple DSs,
the MDS must make sure all mirrored copies will return the new data before
the WRITE reply is sent.

It would imply that if an MDS with multiple DS's were a clustered server
configuration.

Since it isn't, I'd like us to consider how we might avoid a requirement to
immediately propagate writes to all replicas.  Given that there may be good
reasons to have some of these hundreds or thousands of miles away from the
others, I think we want to avoid a situation in which the client writes to
the closest replica but has to wait for all  replicas to receive and reply
to the write.  This adds a latency of 16μs/mi based on the distance to the
farthest replica.

I'm thinking we need something like the following to deal with this issue:

When stable writes are done to the MDS or to a single replica (if allowed
by the appropriate  layout flag), it is the responsibility of the receiving
node to propagate the written data stably, before replying to the client.

In the corresponding cases in which unstable writes are done, the receiving
node does not have any such obligation, although it may choose to
asynchronously propagate the updates.  However, once a COMMIT is replied
to, all replicas must reflect the writes that have been done, and these
data must have been committed to stable storage on all replicas.

In order to avoid situations in which stale data is read from replicas to
which writes have not been propagated:


   - A client which has outstanding unstable writes made to single node
   (MDS or DS) MUST do all reads from that same node.
   - When writes are flushed to the server, for example to implement,
   close-to-open semantics, a COMMIT must be done by the client to ensure that
   up-to-date written data will be available irrespective of the particular
   replica read.



On Sat, Mar 12, 2016 at 2:57 AM, Benny Halevy <bhalevy@gmail.com> wrote:

>
> On Mar 12, 2016 04:06, "Rick Macklem" <rmacklem@uoguelph.ca> wrote:
> >
> > Benny Halevy wrote:
> > >
> > >
> > >
> > > On Mar 11, 2016 3:23 AM, "Rick Macklem" < rmacklem@uoguelph.ca >
> wrote:
> > > >
> > > > Benny Halevy wrote:
> > > > >
> > > > >
> > > > > On Thu, Mar 10, 2016 at 5:46 AM, Rick Macklem <
> rmacklem@uoguelph.ca >
> > > > > wrote:
> > > > >
> > > > >
> > > > >
> > > > > Trond Myklebust wrote:
> > > > > > Hi Rick,
> > > > > >
> > > > > > On Tue, Mar 8, 2016 at 10:39 PM, Rick Macklem <
> rmacklem@uoguelph.ca >
> > > > > > wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I am building an NFSv4.1 pNFS for FreeBSD using Flex Files and
> > > > > > > I've run into a case where it would be nice to make a small
> > > > > > > addition to the layout.
> > > > > > >
> > > > > > > I have a case where the files are replicated over K of the N
> > > > > > > servers in the cluster. Which K of the N is on a per file
> basis.
> > > > > > > (I am using GlusterFS as a backend and this is its replication
> > > > > > > strategy, if I understand it correctly.)
> > > > > > >
> > > > > > > When there are replicas, the files are basically mirrored, but
> > > > > > > writes only need to be done to one of the replicas/mirrors.
> > > > > > > (Again, if I understand what GlusterFS does.)
> > > > > > >
> > > > > > > -- So, what do others think of adding a boolean for the layout
> > > > > > > that says "only one mirror needs to be written VS all mirrors
> > > > > > > need to be written".
> > > > > > >
> > > > > > > I can live without this, but I think I would find it useful and
> > > > > > > I don't see a problem with adding it.
> > > > > > >
> > > > > > > Thanks for your consideration of this, rick
> > > > > >
> > > > > > I don't see any major problems with this request. We have the
> > > > > > ffl_flags field in the layout, and so adding a flag for this
> would be
> > > > > > trivial, and could be done without breaking existing clients.
> Those
> > > > > > existing implementations would presumably be inefficient, since
> they
> > > > > > would ignore the flag, and continue to write to all mirrors, but
> they
> > > > > > would not break.
> > > > > >
> > > > > > One question I have, though, is how this would affect the
> > > > > > ffds_efficiency? The current expectation for that field is that
> it
> > > > > > applies only to the mirrored read case, where you want to select
> one
> > > > > > of many mirrors. Wouldn't we also want to extend that scope of
> that
> > > > > > field so that it can apply to writes?
> > > > > Although I don't have a strong opinion, I just stated it would be
> used
> > > > > for writes the same as reads for this case in the diff. (At least
> that
> > > > > was what I tried to say;-)
> > > > >
> > > > > I also stated that the storage device had to make sure that all
> mirrors
> > > > > were updated before replying successfully to the write (but they
> don't
> > > > > have to have the data committed to stable storage until the
> associated
> > > > > Commit). Does this sound reasonable?
> > > > >
> > > > > Hi Rick,
> > > > >
> > > > > If the DS that the client wrote to responds with STABLE* then file
> data
> > > > > have
> > > > > to be
> > > > > on stable storage under any failure scenario. Bare in mind that the
> > > > > client
> > > > > will mark
> > > > > the dirty page as clean in this case and is free to drop it out of
> its
> > > > > data
> > > > > cache so it won't be able
> > > > > to rewrite it in a server restart case. Therefore, I wouldn't
> allow data
> > > > > on
> > > > > mirrors to be unstable
> > > > > in this case.
> > > > >
> > > > To me, requiring the DS to reply STABLE to the write (which implies
> written
> > > > to
> > > > stable storage on all mirrors) seems to be a very strong requirement.
> > >
> > > It is indeed. Similarly, think of a server with mirrored write cache.
> If it
> > > replied STABLE before the mirrored copy was protected by stable
> storage (and
> > > that includes battery backup of the cache) and crashed while losing
> the user
> > > data or redundant data, then you'd expect it to return EIO when
> accessing
> > > these areas that are potentially corrupt. Returning stale data post
> STABLE
> > > write or COMMIT can seriously mess up applications that rely on stable
> > > storage semantics.
> > >
> > I wasn't questioning the case where the DS chooses to reply STABLE. When
> it
> > does so, it must have committed to stable storage (and this applies to
> all
> > mirrors when writing to the metadata server or a DS when
> FF_FLAGS_WRITE_ONE_MIRROR
> > is set).
> >
> > The case I meant was where the client specifies UNSTABLE.
> > I thought you were saying that the DS would be required to do this as a
> > STABLE write and reply STABLE. This seemed an unreasonably strong
> requirement
> > for the server.
>
> OK, that makes more sense.
> Still, the DS that was target to write better respond with UNSTABLE in
> this case to ensure that the client will issue COMMIT before dropping the
> dirty data out of its cache and before returning success to the
> application's respective system call (a-la sync, flush, or close)
>
> >
> > > >
> > > > Now, if you are saying that the DS may reply STABLE to indicate that
> all
> > > > mirrors
> > > > have been updated, including to stable storage or the DS may reply
> UNSTABLE
> > > > to
> > > > indicate that the mirrors may not be updated until the Commit, then
> I would
> > > > be fine with that.
> > >
> > > This is trivially possible when the client asks for UNSTABLE however
> (and
> > > sadly IMO) the protocol does not allow the server to demote a write
> from
> > > stable to unstable.
> > >
> > Sure, but I assumed the client did the write to the DS as UNSTABLE. Sorry
> > about the confusion.
> >
> > > >
> > > > After all, write sharing is a rare case and the client can use
> Commit to
> > > > ensure
> > > > the update is done on all mirrors when that rare case is handled,
> I'll
> > > > admit I
> > > > like the idea of that option being available. (I'd guess that most
> clients
> > > > would
> > > > do the Commit before unlocking an exclusive byte range lock that
> overlaps
> > > > the
> > > > byte range written, since write-sharing clients will normally use
> > > > byte-range locking
> > > > to handle concurrency.)
> > >
> > > Agreed.
> > >
> > > > --> Without the FF_FLAGS_WRITE_ONE_MIRROR flag, the client will need
> to
> > > > write to
> > > > all mirrors (and do Commits if the write reply is UNSTABLE), so I
> don't see
> > > > the UNSTABLE write to one mirror too "soft" a requirement.
> > >
> > > I'd consider leaving that semantic choice to the admin as an export
> flag,
> > > based on the application, as the behavior you described is reasonable
> and
> > > practical for e.g. interactive usage, similar to other server tweaks
> that
> > > allow divergence from the strict protocol (and that better be clearly
> > > documented of course)
> > >
> > Sure, although I tend to think the client`s choice of STABLE vs UNSTABLE
> in
> > the write request could be sufficient, since it is the client that needs
> > to redo writes for the UNSTABLE case and it is the client that will know
> > when any exclusive byte range lock is released.
>
> If that is the case then no export option is needed.
>
> >
> > Btw, I think the current draft needs to clarify what the semantics are
> for
> > the same question, but when applied to writes to the metadata server.
> > Section 8 Pg 19:
> > - clearly states that writes to the metadata server must be done on all
> >   mirrors.
> >   - What it doesn't seem to say is how quickly this needs to be done?
> >     (Which was the question that I had for the case where a write to a
> >      single DS will write to all mirrors.)
> >
> > If we assume that the STABLE/UNSTABLE variants are adequately handled by
> > the STABLE/UNSTABLE in the write reply, there is still the question of
> > when all mirrors must be updated for the UNSTABLE case (2 possiblilities
> > I can think of. Are there others):
> > 1 - All mirrors must be updated before the Write reply is sent to the
> client,
> >     and to stable storage iff STABLE is replied.
> > OR
> > 2 - All mirrors must be updated before the Write reply is sent to the
> client,
> >     including to stable storage, if the write reply specifies STABLE.
> >     or
> >     All mirrors must be updated before the Commit reply is sent to the
> client,
> >     including to stable storage, if the write reply specifies UNSTABLE.
> >
> > I am now leaning towards (2) for both the writes to the metadata server
> or
> > the write done to one DS iff the new flag FF_FLAGS_WRITE_ONE_MIRROR is
> added
> > to the spec. and is set.
> > I lean towards (2) because updating a lot of mirrors could take a while
> and
> > the client can choose by specifying STABLE vs UNSTABLE whether it
> requires the
> > server to update all mirrors right away or can wait until it does a
> Commit.
> >
> > So, what do others think, rick
>
> Aside for the stability of written data there is the requirement for
> visibility.
> That is, a clustered server needs to mimic as much as possible the
> functional behavior of a monolithic server.  With that in mind, the newly
> written data is visible to all clients in a monolithic server regardless if
> its stable_how value. The application level relies on that for
> close-to-open semantics.  This implies that in a clustered server
> configuration with multiple DSs, the MDS must make sure all mirrored copies
> will return the new data before the WRITE reply is sent.
>
> There could be multiple ways to ensure that. One is to actually write the
> new data to all mirrors. But for example, in a shared storage architecture
> the MDS might only need to invalidate the caches in the DSs.
>
> Benny
>
> >
> > > Benny
> > >
> > > >
> > > > To be honest, at this time I have no idea if the NFSv3 server built
> into
> > > > GlusterFS
> > > > does STABLE or UNSTABLE writes and what guarantees it provides
> w.r.t. when
> > > > replicas
> > > > are updated, so I am trying to define this in such a way that I can
> support
> > > > it.
> > > > (Btw, I don't really need this change. The default GlusterFS setup
> doesn't
> > > > do
> > > > replication and, therefore, won't use mirrors. I just thought it
> would be a
> > > > nice
> > > > feature to support.)
> > > >
> > > > rick
> > > >
> > > > > Benny
> > > > >
> > > > >
> > > > >
> > > > > > Either way I think we'd need a proposal for how we might modify
> the
> > > > > > description in
> > > > > >
> https://tools.ietf.org/html/draft-ietf-nfsv4-flex-files-06#section-8.1
> > > > > >
> > > > > I've attached a diff of the .txt version as a first stab at the
> change.
> > > > >
> > > > > Thanks for your comments, rick
> > > > >
> > > > > > Cheers
> > > > > > Trond
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > nfsv4 mailing list
> > > > > nfsv4@ietf.org
> > > > > https://www.ietf.org/mailman/listinfo/nfsv4
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > nfsv4 mailing list
> > > > > nfsv4@ietf.org
> > > > > https://www.ietf.org/mailman/listinfo/nfsv4
> > > > >
> > >
> > > _______________________________________________
> > > nfsv4 mailing list
> > > nfsv4@ietf.org
> > > https://www.ietf.org/mailman/listinfo/nfsv4
> > >
>
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>