Re: [nfsv4] Possible change to Flex Files layout

[Thomas Haynes <[e-mail]>]

On Mar 12, 2016, at 4:34 PM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:


> Aside for the stability of written data there is the requirement for visibility.

Or at least potential requirements as to visibility.

> That is, a clustered server needs to mimic as much as possible the functional behavior of a monolithic server.

True, but a lot of the deployments will not be clustered servers.  In fact, in many cases in which one is replicating data, the two replicas will be far enough apart that a natural disaster is unlikely to simultaneously affect both.

In any case, pNFS does not strive to precisely emulate clustered server semantics.  Instead, it seeks to provide many of the benefits of clustered operation, while allowing requirements that depend on the clients being aware of the distributed  nature of the arrangement.  Use of LAYOUTCOMMIT is an example.

>With that in mind, the newly written data is visible to all clients in a monolithic server regardless if its stable_how value. The application level relies on that for close-to-open semantics.

True.

> This implies that in a clustered server configuration with multiple DSs, the MDS must make sure all mirrored copies will return the new data before the WRITE reply is sent.

It would imply that if an MDS with multiple DS's were a clustered server configuration.

Since it isn't, I'd like us to consider how we might avoid a requirement to immediately propagate writes to all replicas.  Given that there may be good reasons to have some of these hundreds or thousands of miles away from the others, I think we want to avoid a situation in which the client writes to the closest replica but has to wait for all  replicas to receive and reply to the write.  This adds a latency of 16μs/mi based on the distance to the farthest replica.

I'm thinking we need something like the following to deal with this issue:


When stable writes are done to the MDS or to a single replica (if allowed by the appropriate  layout flag), it is the responsibility of the receiving node to propagate the written data stably, before replying to the client.

In the corresponding cases in which unstable writes are done, the receiving node does not have any such obligation, although it may choose to asynchronously propagate the updates.  However, once a COMMIT is replied to, all replicas must reflect the writes that have been done, and these data must have been committed to stable storage on all replicas.

In order to avoid situations in which stale data is read from replicas to which writes have not been propagated:


  *   A client which has outstanding unstable writes made to single node (MDS or DS) MUST do all reads from that same node.
  *   When writes are flushed to the server, for example to implement, close-to-open semantics, a COMMIT must be done by the client to ensure that up-to-date written data will be available irrespective of the particular replica read.




Better late than never, draft-11 will encompass Rick’s change for FF_FLAGS_WRITE_ONE_MIRROR, your change above, and my change for error handling in this case.