Re: [nfsv4] (belated) copy of my review of draft-ietf-nfsv4-layout-types-03 (part 2)

[Thomas Haynes <[e-mail]>]

Here is part 2:

On Jul 16, 2015, at 4:21 PM, David Noveck <[e-mail]<mailto:[e-mail]>> wrote:




Control Protocol

I think this group of sections need to be re-organized. The basic problem is that the control protocol only deals with the interaction of the MDS and storage devices while, if you look at the requirements in section 3.1 and the non-requirements in section 3.2 you see a number of different patterns:


The control protocol is used between the metadata server and storage device to present a consistent interface to the clients.

I would rewrite the paragraph in Section 3 as:

   In Section 12.2.6 of [RFC5661], the control protocol is introduced.
   There have been no specifications for control protocols, and indeed
   there need not be such a protocol in use for any given
   implementation.  The control protocol is actually a set of
   requirements provided to describe the interaction between the
   metadata server and the storage device such that they present
   a consistent interface to the clients.  When specifying a new layout
   type, the defining document MUST show how it meets these
   requirements, especially with respect to the security implications.

I’m not sure if consistent or coherent is better here.

In any event, I decided to take your rewrite, but with the addition of this point.


  *   (1) of section 3.1 primarily specifies a requirement for client-MDS interaction. While this may, depending of how it is implemented have implications for MDS-storage-device interaction, it is not necessary to think of this as part of control protocol.
  *   (3) and (4) of section 3.1 are requirements for correct inter-client interaction.
  *   (2) and (5) of section 3.1 concern MDS-data storage interaction and are appropriately considered as part of the control protocol.
  *   The items in section 3.2 deal with matters relating to the interaction of clients with storage-devices.

Suggest the following replacement for the text introducing the concept of a control protocol:


In Section 12.2.6 of [RFC5661], the concept of a control protocol was introduced. There have been no published specifications for control protocols as yet.  The control protocol denotes any mechanism used to meet the requirements that apply to the interaction between the metadata server and the storage device.  Particular implementations my satisfy this requirement in any manner they choose and the mechanism chosen may not be described as a protocol.  Specifications defining Layout Types need to clearly show how implementations can meet the requirements discussed below, especially with respect to the those that have security implications.  In addition, such specifications may find it necessary to impose requirements on implementations of the layout type to ensure appropriate interoperability.


In some cases, there may be no control protocol other than the storage protocol.  This is often described as using a "loose coupling" model.  In such cases, the assumption is that the

I groaned when I saw “loose coupling’, but you do it without talking about flex files, so my response
would be to add loose and tight coupling to the definitions.

Which I have done...


Metadata Server, data storage devices, and client may be changed independently  and that the implementation requirements in the layout type specification need to ensure this degree of interoperability.  This model is used in the block and object layout type specification.


In other cases, it is assumed that there may be purpose-built control protocol which may be different for different implementations of the MDS and data server.  In such cases, the assumption is that the Metadata server and data servers are designed and implemented as a unit and interoperability needs to be assured between clients and MDS-DS pairs, developed independently. This is the model used for the files layout.


Another possibility, not so far realized, is for the definition of a control protocol to be specified in a standards-track document.   There are two subcases to consider:

  *   A new layout type includes a definition of a particular control protocol whose use is obligatory for MDS's and data storage devices implementing the layout type.  In this case the interoperability model is similar to the first case above and the defining document should assure interoperability among MDS's, data storage devices and clients developed independently.
  *   A control protocol is defined in a standards-track document which meets the control protocol requirements for one of the existing layout types.  In this case, the new document's job is to assure interoperability between MDS's and data storage devices developed separately.  The existing  definition document for the selected layout type retains the function of assuring interoperability between clients and a given collection of MDS and data storage devices.  In his context, implementations tht implement the new protocol are treated in the same way as those that use an internal control protocol or a functional equivalent.




I have to say I really appreciate the second point above, It really ties this document together with the goals of the IETF.




Protocol Requirements


My general comment is that for some of the requirements (e.g. 2, 3, 4, 5) some degree of client co-operation/good-behavior may be required.  Unfortunately, it seems that explicitly allowing general requirements for co-operation among DS/MDS/client would allow things I don't think we want to allow.  For example, it might vitiate (1).


In the first sentence, suggest deletion of the word "broad".  Also, since all of these use "MUST", should they be "REQUIREMENTS"?

As to individual requirements:


(1):     The big issue here is to make clear that this is not a requirement imposed on the client

^^ good point


and to make clear that the
           client may do this at any point of its choosing, if that is indeed the requirement, I suggest the following replacement:


NFSv4.1 clients MUST be able to access a file by directing IO requests to the metadata server rather than to the storage device.  When  a client chooses to do this, the metadata  must be able to retrieve the data from the constituent storage devices and present it back to the client.


Whether the metadata server allows access over other protocols (e.g., NFSv3, Server Message Block (SMB), etc) is strictly an implementation choice, just as it is in the case of any other (i.e. non-pNFS-supporting) NFSv4.1 server.


Edited in place


(2): Propose the following changes for clarification:

  *   In the second sentence, suggest replacing "fails to renew its lease in time" by "a client's lease is expired due to non-renewal"
  *   In the third sentence, suggest replacing "state" by "locking state or access permissions".
  *   Also in the third sentence, suggest deleting "with the client".
  *   Propose adding the following paragraph:

Effective revocation may require client co-operation in using a particular stateid (files layout) or principal (e,g. flexible files layout) when performing IO.



Ack


(3):      I think there are some substantial issues that need to be addressed here.

I've been unable to come up with an alternative since I think the working group needs to discuss the issues so we get some sort of sense which way people want to go here.

  *   As written this doesn't apply to anything except the file layout.  Block and object devices do not have ACL's. and NFSv3 DS's (for flexible files) typically don't either.

I don’t disagree with your analysis.


  *   The stuff about "file open modes" really belongs in (4).

Tell that to the RFC 5661 authors :-)

Page 308:

   pNFS implementations MUST NOT remove NFSv4.1's access controls.  The
   combination of clients, storage devices, and the metadata server are
   responsible for ensuring that all client-to-storage-device file data
   access respects NFSv4.1's ACLs and file open modes.

Although that does clarify your first point above.

I think the confusion here is that point 3 is a security concern whereas point 4 is a correctness concern.




  *   If the focus moves from the means of enforcing access restrictions (e.g ACLs) to the restrictions themselves, we run into the question of what sort of client co-operation is valid.  The issues relating to what might be loosely called "principal impersonation" were discussed in connection with the review of the flex-files draft but I don't think the working group ever came to a consensus in that regard.

I’ve updated the text to include the point about client cooperation:

   (3)  A pNFS impelementation MUST NOT remove NFSv4.1's access
        controls: ACLs and file open modes.  While Section 12.9 of
        [RFC5661] specifically lays this burden on the combination of
        clients, storage devices, and the metadata server, depending on
        the implementation, there might be a requirement that the
        metadata server update the storage device such that it can
        enforce security.

        The file layout requires the storage device to enforce access
        whereas the flex file layout requires both the storage device
        and the client to enforce security.


(4):      This requires some clarification, if, as I assume is the case, open modes are part of the locking state that must be respected.  The following issues need to be addressed:

  *   If the metadata server supports mandatory locking, IO operations that conflict with existing byte-range locks must noT be done by other clients or by other lockowners on the same client.
  *   When a  delegation is held by one client conflicting IO operations initiated my other clients must not occur, until the delegation is recalled and returned, or evoked.
  *   When an open specifying a deny mode is in effect, Io operations of the denied type must not be done by ther clients or by other openowners in the same client.

Is it sufficient to list these?


(5):      There are a number of problems with this point:

  *   For most layout types, the DS has no conception of modify time, change attribute, or end- of-file (EOF) position.

The key word here is agree.

I’ll try to illustrate with an example from flex files.

The client opens a file, gets a layout, does a GETATTR to get the mtime, does a series of
WRITEs to the storage device, and does a GETATTR to get the mtime.

The mds and the storage device may or may not have their times synced. But even if they do, there
is always network delay.

What we do know is that before the layout is given to the client, the mds has a
mtime of Xmds and the storage device has a mtime of Xsd. Further, the mds could
as part of the LAYOUTGET, send a NFSv3 GETATTR to the storage device to get Xsd.

As long as the client makes no request to the mds to query one of the time, size, or space
attributes, the mds and the storage device need not synchronize. At the moment the client
does send such a GETATTR, the mds sends a NFSv3 GETATTR to storage device to get
a mtime of Ysd. If Ysd != Xsd, then the mds knows it must update its mtime to be Ymds, where
Ymds is the current time on the mds. I.e., it need not be in perfect synch with the storage device.

The next time the client sends a GETATTR, the mds would compare the returned attrs from
the storage device, Zsd, to Ysd.

So, with the storage device being without a concept of mtime for the metadata of the file,
the storage device and the mds can work together to agree on the attribute.

Putting that down in this document? Nope.

Okay, how about this?

   (5)  The metadata server and the storage devices MUST agree on
        attributes like modify time, the change attribute, and the end-
        of-file (EOF) position.

        (a)  "Agree" in the sense that some while state changes need not
             be propagated immediately, they must be propagated when
             accessed by the client.  This access is typically in
             response to a GETATTR of those attributes.

        (b)  A particular storage device might be striped such it knows
             nothing about the EOF position.  It still meets the
             requirement of agreeing on that fact with the metadata
             server.

        (c)  Both clock skew and network delay can lead to the metadata
             server and the storage device having different concepts of
             the time attributes.  As long as those differences can be
             accounted for what is presented to the client in a GETATTR,
             then the two "agree".




  *   The existing text first says "MUST agree" but then undercuts that so as to make it effectively meaningless.
  *   There s no reference to LAYOUTCOMMIT and it should be stated that  when a LAYOUTCOMMIT is done, it is required  that DS-generated changes in attributes be reflected at the MDS by completion of the operation.

Note that this might be optional to a given layout type because it has made LAYOUTCOMMIT optional.

But:

        (d)  A LAYOUTCOMMIT requires that storage device generated
             changes in attributes need be reflected in the metadata
             server by the completion of the operation.



As to the last paragraph:

  *   Suggest replacing the first sentence by "These requirements may be satisfied in different ways by different Layout Types".
  *   In the third paragraph we reach the crux of the issue we must face.  Some things to note:
     *   While the control protocol is specified as relating to coordination of MDS and DS, here we are suddenly speaking of interaction among client, MDS, and DS.

I think this is addressed by the earlier edits I have made to include the client.


     *   If you were to allow arbitrary client participation, it isn't clear  whether the result would be acceptable.  In particular, the client might assure that any request made to the MDS is properly honored by simply making no such requests.  It is clear that that would represent a sharp change from RFC5661.
     *   Although this says "the other Layout Types MUST document" the required interaction, what actually must do the documenting is the layout type specification, so maybe this belongs in section 3.3

I think the first paragraph of Section 3.3 addresses this point.

But I also broke this last line away from the previous paragraph and used the language of "standards-track document"



     *   What the layout type specification needs to do is not only document how this interaction might happen but specify the implementation requirements which the DS, MDS, and pNFS clients must abide by,  in order to ensure that these overall pNFS requirements are met.  In other words, it is a requirement (for layout type specifications) that they specify REQUIREMENTS (for implementations of that layout type).

With regard to item (3) of this section, i can see this is not particularly relevant  to the main issues you are concerned with.  Nevertheless, these items are one of the primary functions of control protocols and,  when this section is reorganized, you need to say something about these functions.   I think the best model for this is to require the document defining a layout type to specify, at least in general terms, how these co-ordination functions are performed in the context of the specific layout type.

Non-Protocol Requirements

Some general comments on this section:

  *   I'm guessing that a more appropriate title for the contents as it stands would be Protocol Non-requirements.  Is that correct?

Undocumented Protocol REQUIREMENTS



  *   Stating non-requirements (i.e. things that are not in RFC5661) is inherently confusing.
  *   Although these things do not in fact appear in RFC5661, that does not dispose of the matter given that we are updating RFC6661.  Although there are good reasons that many of these items cannot be enforced in many of the mapping types, the appropriate facts are not referred to.
  *   What we wind up saying, "SHOULD if at all possible" is dangerously unclear and give implementations and/or layout types too much leeway to create non-interoperable sets of implementations.

Every loosely coupled system is going to violate the second point in this Section. So if we make it a MUST, we are screwed. If however we go with a SHOULD, then the protocol designer can chose to not honor this requirement.

I *think* you agree with that below - so must find caffeine ...



I think it is appropriate to re-orient this section away from the validations which might or might not be done and put the focus on what you are trying to ensure.

I agree - especially as I can’t see how anything other than a tightly coupled system can meet the requirements as stated in this section.


  I believe that it is clear that someone has to ensure:

  *   that clients do not perform IO if they do not have layouts for the files in question.
  *   that client do not perform IO operations outside the ranges specified.
  *   that clients do not perform IO operations inconsistent with the iomodes specified by the layout held.

'While it is true that, for some layout types, the data storage device is not in a position to enforce these conditions, it seems to me that the proper response is not to weaken requirements into recommendations.  I'm pretty sure that the overall pNFS protocol requirements laid out in the previous section cannot be met in the absence of someone making sure you have an appropriate layout to do IO,

Instead a better approach is to allow the different layout types to take different approaches to allocating responsibilities between the data storage device and the client as to how the overall requirements are to be met.  Some possibilities in this regard:

  *   The storage device has sufficient knowledge about existing layouts to determine whether a given io is valid issued, according to what the layouts specify.  This is the approach in effect when validation is possible.
  *   The client has the primary responsibility for only issuing IO appropriate to the layouts held.  The role of the MDS is ensuring that a client which may not be aware of needed changes in available layouts is prevented from inappropriately accessing the storage device.


I’ve taken a stab at a rewrite along the lines you suggested.

You can review it in draft-05. :-)1



Editorial Requirements

Regarding the introductory paragraph, I don't think it is the function of this section to introduce new  (i.e. "additional") requirements.

These are not protocol requirements, they are requirements on how the protocol requirements are met.

Fencing is by far the number one issue I face in talking to people about flex files.

And I shouldn’t have to mention Security - the IETF process as a whole mandates it is covered.

I’m not opposed to your suggestions here, I’m just loathe to give up driving home the point that the fencing model needs to be explained.

I’ll edit the example to include fencing and not just mandatory byte-range locking.


Here's a possible replacement:


This section discusses how the protocol requirements discussed above need to be addressed in documents specifying a new layout Type.  Depending on the interoperability model for the layout type in question, this may involve the imposition of layout-type-specific requirements that ensure appropriate interoperability of pNFS components developed separately..


I don't agree with the material beginning "at a minimum', primarily because it vitiates the essentially correct statement which starts the last paragraph of this section. i.e. someone might think he is absolved from providing other required information because he has met the "minimum". I think the basic reason for doing this document is to eliminate that sort of confusion.

I would prefer to strengthen the last paragraph and I think it can be made appropriately strong without capitalized RFC2119 terms, underlining, or 72-point boldface Arial Black text :--) Here's the kind of thing I'm thinking of:


The specification of the Layout Type needs to make clear how the client, metadata server, and storage device act together to meet the protocol requirements discussed previously. For example, if the metadata server implements mandatory byte-range locking when accessed directly by the client, it must do so when data is read or written using the designated data storage protocol.  If the document does not impose implementation requirements sufficient to ensure that these semantic requirements are met, it is not appropriate for the working group to allow the document to move forward.