Re: [nfsv4] Benjamin Kaduk's Discuss on draft-ietf-nfsv4-mv0-trunking-update-03: (with DISCUSS and COMMENT)

[Benjamin Kaduk <[e-mail]>]

On Fri, Jan 18, 2019 at 12:47:42PM -0500, David Noveck wrote:
> The following is the response to your DISCUSS.   The response to your
> additional COMMENTs
> wIll be sent soon.
> 
> > That said, this document (along with the pieces of 7530 and 7931 that I
> > read along the way) still leave me uncertain about how some things are
> > supposed to work.
> 
> That's unfortunate.  It shouldn't happen.
> 
> > (If it's clarified in parts of those documents that I
> > didn't read, I'll happily clear and apologize for the disruption, of
> > course.)
> 
> There is no disruption.   This discussion seems to me fully in line with
> the way that the process needs to work.
> 
> I believe there are some relevant clarifications that derive from existing
> documents, but it does not appear to me, that you missed anything.   The
> relevant text does not, on its own, provide sufficient clarification, so it
> appears that we will need to clarify things so that future readers do not
> run into the difficulties that you saw.
> 
> > To start with, I'm still lacking a clear high-level picture of why a client
> > needs to care about trunking detection vs. just treating all listed
> > addresses as replicas.  There are some parts in the body where we talk
> > about, e.g., lock state and similar maintenance, but I don't have a clear
> > picture of what the risks and benefits of (not) tracking trunking are, and
> > this would be a fine opportunity to add some text.
> 
> The issues that arise when you are not aware of the trunking relationship
> between two paths do have to with management of locking state.   RFC7530
> (following on from similar text in RFC3530) cited the confusions regarding
> locking that would arise if a client thought it was talking to two
> different servers (i.e. replicas) when it was in fact talking to a single
> system over two different paths.   Basically, a client who is unaware of
> this distinction, might work successfully  sometimes, but in most cases you
> have to know about trunking relationships that exist and it would be
> extremely hard to write locking code that worked without being aware of
> trunking relationships.
> 
> In fact, this is why:
> 
>    - RFC5661 added a means of trunking detection to NFSv4.1
>    - RFCs7530 and 3530 advised NFSv4.0 clients to actively prevent the
>    occurrence of trunking by adopting what is described in RFC7931 as the
>    "non-uniform client string model".
>    - RFC7931 provided a means of trunking detection usable by NFSv4.0
> 
> -04 will need to clearly reference the importantance of state/locking
> considerations in making it necessary that the client and server agree as
> to trunking relationships.  I anticipate we will do this by replacing the
> third paragraph of the introduction by the following two paragraphs;
> 
> As part of addressing this need, [RFC7931] introduces trunking into
> NFS version 4.0 along with a trunking detection mechanism.  A trunking
> detection mechanism enables a client to determine whether two distinct
> network addresses are connected to the same NFS version 4.0 server
> instance.  This knowledge is necessary since, without it, a client
> unaware of a trnking relation ship between paths it is using

nit: "trunking relationship"

> simultaneous is likely to become confused in ways described in
> [RF7530].
> 
> NFSv4.1 was defined with an integral means of trunking detection
> described in [RFC5661], while NFSv4.0 initially did not have one, with
> it being added by [RFC7931].  Nevertheless, the use of the concept of
> server-trunkability is the same in both protocol versions.

That would be a big help; thanks.

> 
> > Specifically, in
> > Section 5.2.1, we just say  that "[a] client may use file system location
> > elements simultaneously to provide higher-performance access to the target
> > file system"; most of the focus of this document makes me think that this
> > statement was intended to apply only to trunking,
> 
> That was the intention and given that this is a trunking-motivated
> document, it seemed reasonable.  However, as you point out, there are cases
> in which those those words could reasonably be read as applying to
> replication.
> 
> > but I also think there
> > are supposed to be replication-only scenarios that provide performance
> > gains.  I'm not sure if we need to clarify the distinction in that location
> > as well as the high-level overview.
> 
> There are such scenarios but they are not as important because:
> 
>    - They only apply to a subset of workloads, basically read-only file
>    systems, and very low-change read-write filesystems where the application
>    takes special care to propagate updates to all replicas.
>    - The client (or the application) needs to be aware of the fact that
>    spcial state mangement actions need to be taken.  Foe example, if you are
>    to stripe your read from a given file between/amon a set of replicas, there
>    needs to be.
> 
> It's also relevant to consider the original purpose of the replication
> feature in NFSv4.0.   The intention was to provide alternative locations
> that could be used if the primary location became unavailable.   This has
> not prevented its use to provide parallel acces to multiple replicas to
> increase performance, in a limited set of contexts, primarily to provide
> read-only access, but it is important in deciding how to describe this
> option in -04.
> 
> I believe we can address this issue in -04 by replacing the current fourth
> paragraph of section 5.2.1 by the following two paragraphs:
> 
> A client may use file system location elements simultaneously to
> provide higher-performance access to the target file system.  This is
> most simply be done using trunling although the use of multiple

nit: "is most simply done using trunking"

> replicas simultaneously is possible. To enable this simultaneous
> acces, the client utilizes trunking detection and/or discovery,

nit: "access"

> further described in Section 5.2.2 of the current document, to
> determine a set of network paths that are server-trunkable with the
> one currently being used to access the file system.   Once this
> determination is made, requests may be routed across multiple paths,
> using the existing state management mechanism
> 
> Multiple replicas may also be used simultaneously, typicalls in

nit: "typically"

> handling read-only file systems.   In this case, each replica has its
> own state management which the client needs to be aware of, doin

nit: "doing"

> multiple file opens to enable a file to be read simultaneoudly from
> multiple replicas.

SGTM.

> 
> > It's also unclear to me what parts of migration flows are under the control
> > of the client vs. the server.  It's clear that the server has to initiate
> > migration via NFS4ERR_MOVED,
> 
> The server has to precipitate the client's response to the migration event
> by informing it of the unusability of the (previously) current instance by
> returning NFS4ERR_MOVED.  However, in almost all cases, this is not the
> start of the server's work.
> 
> The server must, before informing the client of the shift, make sure that
> when the client acts on it, the successor server (or servers) are prepared
> to respond appropriately:
> 
>    - The destination server needs to have a fully up-to-date copy of the
>    source file system.  Where the file system is read-write, this means that
>    all update made to the sotrce file system must be propagated to the
>    destination(s).
>    - When transparent state migration is implemented, locking data needs to
>    be similarly propagated.
> 
> > but my current understanding is just that this
> > prompts the client to look at fs_locations, and the client has control over
> > which alternate location to move to.
> 
> When multiple locations are provided, the client has this choice.

Okay, thanks for confirming that.

> > But there's also a lot of discussion
> > in all three documents about the servers migrating state along with
> > migration, so it seems like the server should be controlling where the
> > client goes.  Is this just supposed to be by limiting the fs_locations data
> > to the specific migration target chosen by the server?
> 
> As a practical matter there are difficulties propagating locking state and
> other updates to multiple desinations so that I expect most implementations
> to designate a single successor, at least for a while.  However, the
> protocol does allow multiple choices to be provided.  If the server does
> so, then it would have to propagate information to multiple targets and be
> prepared for any valid choice the client might make.
> 
> > (If so, this would
> > probably have potential for poor interaction with the implicit filesystem
> > discovery described in Section 5.3.)
> 
> I'm not sure of what problems you foresee.  Perhaps I need to better understand
> your interpretation of section 5.3.

I don't have a detailed description of a problematic scenario, here, so my
"probably" is to be treated as speculative.

Basically, my reading of 5.3 is that it provides a way for a client to
know, implicitly, that server Y serves file system B because server Y and
server X are trunked, and the client knows that server X serves file system
B (because the client was already talking to server X for file system A and
got back a list of filesystems on X).  So if the client is currently
talking to server Z for file system B but then B tries to initiate
migration to server X, the client could well be within its rights to talk
to server Y instead.  This would be problematic if Z has only propagated
locking state to X and not Y.

I'm happy to learn what misunderstandings are present in the above :)

> > On the other hand, Section 5.2.6
> > talks about the server putting entries "that represent addresses usable
> > with the current server or a migration target before those associated with
> > replicas", which seems to imply that there is some other way to know what
> > the migration target is.
> 
> Not sure what you mean by "some other way".

I'm not sure either :)

The point being that if 5.2.6 allows a server to include both migration
targets and (non-migration-target) replicas in the filesystem location
attribute, a client trying migration would probably want to know where the
boundary between migration targets and non-migration-targets is.  From the
discussion above, it sounds like the client doesn't have "some other way"
of knowing this, and having the server list non-migration-targets is a bad
idea, at least when initiating migration.

> I think the server has to know what addresses are usable with current
> server because it is the one that provides the appropriate service.  With
> regard to migration targets, as we have seen, the sending server has to
> know this because he is typically involved in preparing (through
> information progation) the destination server to provide the needed
> service.
> 
> > Section 5.2.6 also tells the client to rely on that ordering:
> >
> >                                   To keep this process as short as
> >   possible, Servers are REQUIRED to place file system location entries
> >   that represent addresses usable with the current server or a
> >   migration target before those associated with replicas.  A client can
> >   then cease scanning for trunkable file system location entries once
> >   it encounters a file system location element whose fs_name differs
> >
> > but I don't think a client actually can do so, since the client has no way
> > to know that the server implements this document as opposed to stock
> > 7530+7931 (at least, no way that I saw).
> 
> The basic assumption behind this paragraph as written is that no previous
> document has provided guidance as to the use of the location attribute to
> provide information relevant to trunking discovery.   However, in
> considering interactions with previous servers as you have done (nice
> catch!) it appears that it would be possible for a server to present two
> trunkable aaddresses even without any awareness of the fact that they are
> trunkable, so that a client following the guidance above might deny himself
> knowledge of some piece of information related to trunking discovery.
> 
> In light of this, it appears that -04 will have to rewrite the material in
> the last paragraph of section 5.2.6 to be something like the following:
> 
> Because a file system location attribute may include entries relating
> to the current server, the migration destination, and possible
> replicas to use, scanning for available network addresses thst might
> be trunkable with addresses already seen could potentially be a long
> process.  In order keep this process as short as possible, Servers
> that provide infornation about trunkable network paths when the exist
> are REQUIRED to place file system location entries that represent
> addresses usable with the current server or a migration target before
> those associated with replicas.
> 
> This ordering allows a client to cease scanning for trunkable file
> system location entries once it encounters a file system location
> element whose fs_name differs from the current fs_name, or whose
> address is not server-trunkable with the one it is currently using.
> While the possibility exists, that a client might prematurely cease
> scanning for trunkable addresses when receiving a location attribute
> from a an older server not following the order constaint above, the
> harm is expected to be limited since such servers would not be
> expected to present infomation about trunkble server access paths.

Works for me.

> 
> > Finally, removing the last paragraph of Section 8.5 of RFC 7530 could have
> > negative operational impact if updated clients interact with non-updated
> > servers/environments that are misconfigured in the described fashion.  It's
> > probably worth stating in the top-level Section 5 that such misconfigured
> > servers are believed to no longer exist (if that's in fact true, of
> > course; if not, we'd need to reconsider the change).
> 
> It turns out this deletion was inadvertant.  The paragraph will be added
> back in -04.

That works too :)

-Benjamin