Re: [nfsv4] Possible change to Flex Files layout

[David Noveck <[e-mail]>]

Cool.

On Tue, Jul 18, 2017 at 5:16 AM, Thomas Haynes <[e-mail]>
wrote:

>
> On Mar 12, 2016, at 4:34 PM, David Noveck <[e-mail]> wrote:
>
> > Aside for the stability of written data there is the requirement for
> visibility.
>
> Or at least potential requirements as to visibility.
>
> > That is, a clustered server needs to mimic as much as possible the
> functional behavior of a monolithic server.
>
> True, but a lot of the deployments will not be clustered servers.  In
> fact, in many cases in which one is replicating data, the two replicas will
> be far enough apart that a natural disaster is unlikely to simultaneously
> affect both.
>
> In any case, pNFS does not strive to precisely emulate clustered server
> semantics.  Instead, it seeks to provide many of the benefits of clustered
> operation, while allowing requirements that depend on the clients being
> aware of the distributed  nature of the arrangement.  Use of LAYOUTCOMMIT
> is an example.
>
> >With that in mind, the newly written data is visible to all clients in a
> monolithic server regardless if its stable_how value. The application level
> relies on that for close-to-open semantics.
>
> True.
>
> > This implies that in a clustered server configuration with multiple DSs,
> the MDS must make sure all mirrored copies will return the new data before
> the WRITE reply is sent.
>
> It would imply that if an MDS with multiple DS's were a clustered server
> configuration.
>
> Since it isn't, I'd like us to consider how we might avoid a requirement
> to immediately propagate writes to all replicas.  Given that there may be
> good reasons to have some of these hundreds or thousands of miles away from
> the others, I think we want to avoid a situation in which the client writes
> to the closest replica but has to wait for all  replicas to receive and
> reply to the write.  This adds a latency of 16μs/mi based on
> the distance to the farthest replica.
>
> I'm thinking we need something like the following to deal with this issue:
>
> When stable writes are done to the MDS or to a single replica (if allowed
> by the appropriate  layout flag), it is the responsibility of the receiving
> node to propagate the written data stably, before replying to the client.
>
> In the corresponding cases in which unstable writes are done, the
> receiving node does not have any such obligation, although it may choose to
> asynchronously propagate the updates.  However, once a COMMIT is replied
> to, all replicas must reflect the writes that have been done, and these
> data must have been committed to stable storage on all replicas.
>
> In order to avoid situations in which stale data is read from replicas to
> which writes have not been propagated:
>
>
>
>    - A client which has outstanding unstable writes made to single node
>    (MDS or DS) MUST do all reads from that same node.
>    - When writes are flushed to the server, for example to implement,
>    close-to-open semantics, a COMMIT must be done by the client to ensure that
>    up-to-date written data will be available irrespective of the particular
>    replica read.
>
>
>
>
>
> Better late than never, draft-11 will encompass Rick’s change
> for FF_FLAGS_WRITE_ONE_MIRROR, your change above, and my change for error
> handling in this case.
>
>
>
>
>