Re: [nfsv4] Review of draft-ietf-nfsv4-rfc5667bis-09

[Tom Talpey <[e-mail]>]

On 4/27/2017 11:44 AM, Chuck Lever wrote:
>
>> On Apr 27, 2017, at 7:20 AM, Tom Talpey <[e-mail]> wrote:
>>
>>> snip:
>>>    Such implementation limits can constrain the complexity of NFS
>>>    version 4 COMPOUNDs, limit the number of elements in scatter-gather
>>>    operations, or prevent the use of Kerberos integrity or privacy
>>>    services.
>>
>> I like the approach, and the lead-in language looks good. The text
>> quoted above is just a little bit dark, especially that bit about
>> preventing krb5i/krb5p. I'd suggest a more active statement to replace
>> the above, including the more prescriptive SHOULD rather than "can".
>> How about:
>>
>> "Client implementations SHOULD be prepared to provide mechanisms for
>>  reporting the above errors, and optionally provide configuration to
>>  limit the complexity of NFS version 4 COMPOUNDs, limit the number
>>  of elements in scatter-gather operations, and to avoid other possible
>>  sources of RPC-over-RDMA chunk overruns at the peer.
>>
>>  These become especially important when Kerberos integrity or privacy
>>  is in place for the RPC connection. These facilities add payload to
>>  the RPC headers, potentially increasing the complexity of the chunk
>>  manipulation, independent of the upper layer NFS operation. The
>>  implementation SHOULD consider such RPC payload requirements in
>>  addition to the NFS considerations."
>
> Sure, I can work this in.
>
> When you say "Client implementations SHOULD ... [report] the above
> errors" you are talking about reporting them to administrators
> and/or RPC consumers? I don't think we can use SHOULD in that case.

I am agnostic about who to inform. The important thing is that some
visibility of the error be surfaced. I absolutely don't think an
arbitrary GARBAGE_ARGS returned to an application that may simply
choke on it, qualifies.

> This feels like implementation advice, not protocol.

Correct. But since the protocol creates the problem, the protocol
definition needs to say something about dealing with it. So I believe
SHOULD is best.

> Would "recommend" be enough for this section?

The RFC2119 term RECOMMENDED is basically a synonym for SHOULD. It's
perfectly permissible.

Tom.

>
>
>> Feel free to wordsmith further.
>>
>> Tom.
>>
>>
>> On 4/26/2017 12:18 PM, Chuck Lever wrote:
>>>
>>>> On Apr 24, 2017, at 7:30 AM, Chuck Lever <[e-mail]> wrote:
>>>>
>>>>>
>>>>> On Apr 24, 2017, at 7:56 AM, Tom Talpey <[e-mail]> wrote:
>>>>>
>>>>> On 4/21/2017 10:43 AM, Chuck Lever wrote:
>>>>>> I agree that SHOULD/MAY makes things cloudier, and does not
>>>>>> seem to align with well-defined RFC2119 usage.
>>>>>>
>>>>>> Another way we've dealt with similar disagreements between
>>>>>> specification and implementation is to decide that one of
>>>>>> the implementations is incorrect.
>>>>>>
>>>>>> Can we agree that:
>>>>>>
>>>>>> - GARBAGE_ARGS is a bit of a layering violation, though it's
>>>>>> understandable why it might be returned
>>>>>>
>>>>>> - RPC clients are already prepared for GARBAGE_ARGS
>>>>>
>>>>> Are you certain of this?
>>>>
>>>> GARBAGE_ARGS has been part of the RPC protocol for decades.
>>>> The two Unix-flavored clients that have NFS/RDMA support can
>>>> both handle this error.
>>>
>>> I've confirmed that the only other known NFS/RDMA client
>>> (Oracle dNFS) properly recognizes GARBAGE_ARGS.
>>>
>>>
>>>>> And out of curiosity, what is returned
>>>>> to the consumer for GARBAGE_ARGS versus ERR_CHUNK?
>>>>
>>>> RFC 5531:
>>>>> GARBAGE_ARGS  = 4, /* procedure can’t decode params   */
>>>>
>>>>
>>>> GARBAGE_ARGS is an RPC-level error. The reply is "accepted"
>>>> with accept_stat GARBAGE_ARGS. An XID is available in the
>>>> header.
>>>>
>>>> rfc5666bis:
>>>>> If the rdma_vers field contains a recognized value, but an
>>>>> XDR parsing error occurs, the responder MUST reply with an
>>>>> RDMA_ERROR procedure and set the rdma_err value to ERR_CHUNK.
>>>>
>>>>
>>>> ERR_CHUNK is a transport level error. An XID is available
>>>> in the header.
>>>>
>>>> The difference is that the RPC layer v. the transport layer
>>>> are reporting they don't understand the contents of the
>>>> message (Call). There is nothing more in either type of
>>>> message.
>>>>
>>>>
>>>>>> - In RPC-over-RDMA Version One, we are not trying to recover
>>>>>> (in the sense of resending a simpler COMPOUND) but are rather
>>>>>> trying to ensure the offending RPC is properly terminated on
>>>>>> the client, and does not further block other RPCs or deadlock
>>>>>> the transport
>>>>>>
>>>>>> Thus I claim it is harmless if a server returns GARBAGE_ARGS
>>>>>> instead of ERR_CHUNK.
>>>>>
>>>>> "Harmless" is a bit relative. The operation fails, through no fault
>>>>> of the consumer. And, frankly, in a very mysterious way.
>>>>
>>>> We have no richer way of communicating failure in RPC-over-RDMA
>>>> Version One. We are not looking for recovery here, so I don't
>>>> believe any more information would be useful. If the server
>>>> wishes, it can log the failure with a message explaining what
>>>> went wrong.
>>>>
>>>>
>>>>> Again, I think there is more to say here. It's a limitation of the
>>>>> protocol whose implications should be made clear (contraining the
>>>>> complexity of COMPOUNDs, limiting scatter/gather lengths, etc).
>>>>
>>>> I'd welcome any suggested text.
>>>>
>>>> Honestly, I'm not sure what can be said. Neither NFSv4.0 nor
>>>> RPC-over-RDMA have a sophisticated mechanism to communicate this
>>>> kind of limitation. The best an NFSv4 server can do is return
>>>> NFS4ERR_RESOURCE, which also carries little extra information
>>>> about what a client should do to recover.
>>>>
>>>> So are you comfortable with eliminating GARBAGE_ARGS if we can
>>>> come up with more detail about the impact of not knowing how
>>>> complex a COMPOUND can be?
>>>
>>> I've come up with some possible replacement text for
>>> the final two paragraphs of S5.4.1 in an attempt to
>>> address comments from Tom, David, and Karen. The
>>> normative requirements have been removed, and a (brief)
>>> discussion of the consequences of not handling complex
>>> COMPOUNDs was introduced.
>>>
>>>
>>> 5.4.2.  Complexity Considerations
>>>
>>>   As mentioned above, an NFS version 4 COMPOUND procedure can contain
>>>   more than one operation that carries a DDP-eligible data item.  The
>>>   RPC-over-RDMA Version One protocol does not place any limit on the
>>>   number of chunks that may appear in the Read or Write lists.
>>>   Therefore an NFS version 4 client MAY construct an RPC-over-RDMA
>>>   Version One message containing more than one Read chunk or Write
>>>   chunk.
>>>
>>>   However, implementations have practical limits on the number of
>>>   chunks or segments they are prepared to process in one of these
>>>   lists.  There are several ways an NFS Version 4 server might indicate
>>>   that an RPC Call message constructed by a client is valid but cannot
>>>   be processed because of implementation limitations:
>>>
>>>   o  If the problem is detected in the upper layer (i.e., by the NFS
>>>      version 4 implementation), the server returns an NFS status of
>>>      NFS4ERR_RESOURCE.
>>>
>>>   o  If the problem is detected during XDR decoding of the request
>>>      (e.g., during re-assembly of the Call message by the RPC layer),
>>>      the server returns an RPC accept_stat of GARBAGE_ARGS.
>>>
>>>   o  If the problem is detected at the transport layer (i.e., during
>>>      transport header processing), the server returns an RDMA_ERROR
>>>      message with the err_code field set to ERR_CHUNK.
>>>
>>>   Such implementation limits can constrain the complexity of NFS
>>>   version 4 COMPOUNDs, limit the number of elements in scatter-gather
>>>   operations, or prevent the use of Kerberos integrity or privacy
>>>   services.
>>>
>>>
>>> Comments, opinions on this approach?
>>>
>>>
>>>>> Tom.
>>>>>
>>>>>
>>>>>>
>>>>>> As a result, I can change the Read list text in S5.4.1 to be
>>>>>> the same as the Write list text, removing the mention of
>>>>>> GARBAGE_ARGS.
>>>>>>
>>>>>> Would that sit comfortably with everyone?
>>>>>>
>>>>>>
>>>>>>> On Apr 20, 2017, at 7:21 PM, David Noveck <[e-mail]> wrote:
>>>>>>>
>>>>>>>> The "or" is a similar situation, it prescribes a choice, which
>>>>>>>> does not define a protocol.
>>>>>>>
>>>>>>> Fair enough, but the point that needs to be made is that, with
>>>>>>> regard to Version One, Chuck and  the working group is not
>>>>>>> free to define a protocol.  As a result we have the kind of
>>>>>>> ugliness you object to, but it is inherent in the choice to try to
>>>>>>> revive Version One as-is.
>>>>>>>
>>>>>>>> If an NFS version 4 client sends an RPC Call with a Read list that
>>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>>> process, the server SHOULD reject the request by responding with an
>>>>>>>> RDMA_ERROR message with the rdma_err value set to ERR_CHUNK. The
>>>>>>>> server MAY reject the RPC with an RDMA_MSG message containing an RPC
>>>>>>>> Reply with an accept status of GARBAGE_ARGS.
>>>>>>>
>>>>>>> I think I know what you intend here and I've seen stuff like this in RFCs but I don't
>>>>>>> wthink e can do this because this is not in line with the definitions of "SHOULD"
>>>>>>> and "MAY" that appear in RFC2119.
>>>>>>>
>>>>>>> "SHOULD" means that you are supposed to do something but can avoid it if
>>>>>>> you have a good reason and are aware of the consequences of not doing it.
>>>>>>> In this case the "good" reason is that someone coded the implementation
>>>>>>> to do something else, which is not all that good a reason.  The consequences of
>>>>>>> returning the GARBAGEARGS are exactly zero, since the client has to be prepared
>>>>>>> for either it or ERR_CHUNK.
>>>>>>>
>>>>>>> "MAY" means the implementation can choose to do the action or not, which is line
>>>>>>> with the reality here but essentially contradicts the SHOULD.
>>>>>>>
>>>>>>>> This at least makes it clear which response is "preferred".
>>>>>>>
>>>>>>> But it is isn't really the job of the RFC2119 terms to say which is "preferred" or
>>>>>>> "'preferred'".  These terms are supposed to describe interoperability and the
>>>>>>> interoperability situation is that the server MUST return ERR_CHUNK or
>>>>>>> GARBAGEARGS and the client needs to be prepared for either.  That is the
>>>>>>> unpleasant reality.  If you want to indicate a preference, you can say something
>>>>>>> like:
>>>>>>> 	• Returning ERR_CHUNK is preferrable.
>>>>>>> 	• Returinng ERR_CHUNK is more in line with the appropriate protocol layering since this issue relates to a limitation of the transport implementation.
>>>>>>> 	• Use of GARBAGEARGS is an unfortunate artifact of inappropriately layered implementations and is only allowed for reasons of compatibility with existing implementations.  It is desirable to avoid it.
>>>>>>>> And one would hope a future draft would decide.
>>>>>>>
>>>>>>> Not sure what draft you are thinking of.  I don't see us doing an rfc5667bisbis (rfc5667tris).
>>>>>>>
>>>>>>> By the time we did that, the implementations with these restrictions will probably be gone.
>>>>>>>
>>>>>>>> I have a second question though. How does the client determine what is
>>>>>>>> the actual error? As in, how many chunks were allowed?
>>>>>>>
>>>>>>> This is not fixable in Version One.  It would be in Version Two, but by then
>>>>>>> the need will probably be gone.
>>>>>>>
>>>>>>>> Does the upper
>>>>>>>> layer have to recover, and if so what are the implications?
>>>>>>>
>>>>>>> I think something could be put in to indicate that clients should break up COMPOUNDS
>>>>>>> so the only have a single chunk each.
>>>>>>>
>>>>>>>> Yes, I know 5667 did not explore this very well.
>>>>>>>
>>>>>>> It didn't explore it at all.  And 5666's error reporting facilities were extremely limited.
>>>>>>>
>>>>>>>> Mea culpa.
>>>>>>>
>>>>>>> I don'tt think you have anything to apologize for.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 20, 2017 at 5:28 PM, Tom Talpey <[e-mail]> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 4/19/2017 11:14 AM, Chuck Lever wrote:
>>>>>>> Hi Tom-
>>>>>>>
>>>>>>> On Apr 18, 2017, at 11:08 PM, Tom Talpey <[e-mail]> wrote:
>>>>>>>
>>>>>>> I noticed the same thing, and I'll add that the MUST reject condition
>>>>>>> is very confusing because it allows an "or". In my opinion a MUST is
>>>>>>> always a single requirement, never ambiguous.
>>>>>>>
>>>>>>> I agree this kind of thing is tricky. I wrote it as "the server MUST
>>>>>>> reject the RPC". That's the single requirement. The choice is how the
>>>>>>> rejection is conveyed to the client.
>>>>>>>
>>>>>>> The statement "MUST reject" is not testable. So, while it may be
>>>>>>> understood what is intended, there is nothing implementable in the
>>>>>>> MUST. The "or" is a similar situation, it prescribes a choice, which
>>>>>>> does not define a protocol.
>>>>>>>
>>>>>>> Is there some reason you want to allow such a choice? I think you'll
>>>>>>> find that, worded properly, it becomes actually much less implementable
>>>>>>> and interoperable than you may think.
>>>>>>>
>>>>>>> The Solaris server can return an RPC-level error in cases like this.
>>>>>>>
>>>>>>> Well, this is happening because the Solaris server is (probably) just
>>>>>>> handing the chunk list up to the RPC layer, and it's the RPC (XDR)
>>>>>>> processing that detects any problem.
>>>>>>>
>>>>>>> On the other hand, an implementation could do the opposite, it could
>>>>>>> process the chunks at the lower layer, before ever invoking RPC
>>>>>>> processing. This would naturally lead to a non-RPC error.
>>>>>>>
>>>>>>> The challenge in defining the protocol is to hide these possibilities.
>>>>>>>
>>>>>>> I think there are similar choices allowed in rfc5666bis. Let's say
>>>>>>> that in a perfect world, I would go with only ERR_CHUNK, but I'm
>>>>>>> documenting existing implementation behavior here.
>>>>>>>
>>>>>>> I'm not sure it matters to the client: both errors are permanent and
>>>>>>> the RPC is terminated on the client.
>>>>>>>
>>>>>>> I'm open to alternatives.
>>>>>>>
>>>>>>> The icky way to do this is to split into two weak requirements.
>>>>>>>
>>>>>>> If an NFS version 4 client sends an RPC Call with a Read list that
>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>> process, the server SHOULD reject the request by responding with an
>>>>>>> RDMA_ERROR message with the rdma_err value set to ERR_CHUNK. The
>>>>>>> server MAY reject the RPC with an RDMA_MSG message containing an RPC
>>>>>>> Reply with an accept status of GARBAGE_ARGS.
>>>>>>>
>>>>>>> This at least makes it clear which response is "preferred". And one
>>>>>>> would hope a future draft would decide.
>>>>>>>
>>>>>>> I have a second question though. How does the client determine what is
>>>>>>> the actual error? As in, how many chunks were allowed? Does the upper
>>>>>>> layer have to recover, and if so what are the implications?
>>>>>>>
>>>>>>> Yes, I know 5667 did not explore this very well. Mea culpa.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Tom.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 4/18/2017 6:32 PM, karen deitke wrote:
>>>>>>> Hi Chuck, its unclear what you mean by "is prepared to process" in the text below.
>>>>>>> Other than that, looks good.
>>>>>>>
>>>>>>> Karen
>>>>>>>
>>>>>>> 5.4.1
>>>>>>> If an NFS version 4 client sends an RPC Call with a Write list that
>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>> process, the server MUST reject the RPC by responding with an
>>>>>>> RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>>>>>>>
>>>>>>>
>>>>>>> If an NFS version 4 client sends an RPC Call with a Read list that
>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>> process, the server MUST reject the RPC by responding with an
>>>>>>> RDMA_MSG message containing an RPC Reply with an accept status of
>>>>>>> GARBAGE_ARGS, or with an RDMA_ERROR message with the rdma_err value
>>>>>>> set to ERR_CHUNK.
>>>>>>>
>>>>>>>
>>>>>>> On 4/18/2017 1:21 PM, David Noveck wrote:
>>>>>>> *Overall Evaluation*
>>>>>>> *
>>>>>>> *
>>>>>>> Major improvement over RFC5667.  Almost ready to ship.  No technical
>>>>>>> issues.
>>>>>>>
>>>>>>> A lot of my comments are basically editorial and are offered on a
>>>>>>> take-it-or-lease-it basis.
>>>>>>>
>>>>>>> I think some clarification in Section 5.4.1 is needed although not
>>>>>>> necessarily in the ways suggested below,
>>>>>>>
>>>>>>> *Comments by Section*
>>>>>>> *5.4.1. Multiple DDP-eligible Data Items*
>>>>>>> Giventhat READ_PLUS no longer has any DDP-eligible data items, the
>>>>>>> situation described in the fifth bullet can no longer arise. I suggest
>>>>>>> deleting the bullet.
>>>>>>> The penultimate paragraph can be read as applying to some situations
>>>>>>> in which it shouldn't and where the extra chunks would very naturally
>>>>>>> ignored. For example, if you had on write chunk together with a READ
>>>>>>> operation which failed, the server would have more chunks (i.e. one)
>>>>>>> than the number it is prepared to process (i.e. zero). Suggest, as a
>>>>>>> possible replacement:
>>>>>>>
>>>>>>> Normally, when an NFS version 4 client sends an RPC Call with a
>>>>>>> Write list that contains multiple chunks. each such, when matched
>>>>>>> with a DDP-eligible data item in the response, directs the
>>>>>>> placement of the data item as specified by
>>>>>>> [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items
>>>>>>> matched to write chunks that an NFS version 4 server is not
>>>>>>> prepared to process, the server MUST reject the RPC by responding
>>>>>>> with an RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>>>>>>>
>>>>>>> With regard to the last paragraph, I am curious that this paragraph,
>>>>>>> unlike the previous one, allows GARBGEARGS. Is this so because that
>>>>>>> would be allowed if the chunks in question had offsets other than
>>>>>>> those that correspond to DDP-eligible data items? If so, please
>>>>>>> consider the following possible replacement.
>>>>>>>
>>>>>>> Normally, when an NFS version 4 client sends an RPC Call with a
>>>>>>> Read list that contains multiple chunks, each such, when properly
>>>>>>> matched with a DDP-eliigible data item in the request, directs the
>>>>>>> fetching of the the data item as specified by
>>>>>>> [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items
>>>>>>> matched to read chunks that an NFS version 4 server is not
>>>>>>> prepared to process, the server MUST reject the RPC by responding
>>>>>>> with an RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>>>>>>>
>>>>>>> *5.6. Session-Related Considerations*
>>>>>>> In the third sentence of the second paragraph, suggest replacing "no
>>>>>>> different" by "not different".
>>>>>>> In the last sentence of the last paragraph, suggest replacing "is not"
>>>>>>> by "were not"
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> nfsv4 mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> nfsv4 mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> nfsv4 mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>
>>>>>>> --
>>>>>>> Chuck Lever
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> nfsv4 mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> nfsv4 mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> nfsv4 mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>
>>>>>> --
>>>>>> Chuck Lever
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> nfsv4 mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> nfsv4 mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>
>>>> --
>>>> Chuck Lever
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> nfsv4 mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>
>>> --
>>> Chuck Lever
>>>
>>>
>>>
>>> _______________________________________________
>>> nfsv4 mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/nfsv4
>
> --
> Chuck Lever
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4
>