Re: [nfsv4] Review of draft-ietf-nfsv4-rfc5667bis-09

[Tom Talpey <[e-mail]>]

On 4/27/2017 12:01 PM, karen deitke wrote:
>
>
> On 4/27/2017 9:13 AM, Tom Talpey wrote:
>> On 4/27/2017 10:38 AM, karen deitke wrote:
>>>
>>>
>>> On 4/27/2017 8:20 AM, Tom Talpey wrote:
>>>> >snip:
>>>> >    Such implementation limits can constrain the complexity of NFS
>>>> >    version 4 COMPOUNDs, limit the number of elements in
>>>> scatter-gather
>>>> >    operations, or prevent the use of Kerberos integrity or privacy
>>>> >    services.
>>>>
>>>> I like the approach, and the lead-in language looks good. The text
>>>> quoted above is just a little bit dark, especially that bit about
>>>> preventing krb5i/krb5p. I'd suggest a more active statement to replace
>>>> the above, including the more prescriptive SHOULD rather than "can".
>>>> How about:
>>>>
>>>>  "Client implementations SHOULD be prepared to provide mechanisms for
>>>>   reporting the above errors, and optionally provide configuration to
>>>>   limit the complexity of NFS version 4 COMPOUNDs, limit the number
>>>>   of elements in scatter-gather operations, and to avoid other possible
>>>>   sources of RPC-over-RDMA chunk overruns at the peer.
>>>>
>>>>   These become especially important when Kerberos integrity or privacy
>>>>   is in place for the RPC connection. These facilities add payload to
>>>>   the RPC headers, potentially increasing the complexity of the chunk
>>>>   manipulation, independent of the upper layer NFS operation. The
>>>>   implementation SHOULD consider such RPC payload requirements in
>>>>   addition to the NFS considerations."
>>>  This wording about preventing the use of krb5 is confusing to me. Are
>>> you saying that clients whose krb5 implementation causes "extraneous"
>>> number of chunks in an rdma header may not be supported by a server and
>>> thus prevent krb usage?
>>
>> As the protocol stands today, yes, there is no requirement that the
>> server MUST be prepared to handle any certain number of chunks. And,
>> since krb can increase the size of RPC headers, potentially pushing
>> them beyond the maximum message size, that can potentially add new
>> payload to the chunk list.
>>
>> The implication is that NFS COMPOUNDs may need to shrink, if this is
>> the case. However, the "limit" is server implementation dependent,
>> and the chunks are built at the client. So my suggestion, short of
>> writing a new protocol, is to inform the client implementer of this
>> possibility.
> Or the client can simply send the entire packet in 1 offset 0 read chuck
> as the solaris client does.  I'm guessing that the linux client can not
> do this based on where the packets are created, but just pointing out
> there are other ways to accomplish this.

Indeed it can, but the penalty is high since it requires registration
of the message, and five trips on the wire even before transferring
any bulk data. That penalty will be incurred for every request, plus,
servers may choose to not implement the exchange.

It's worth mentioning. But I feel it is far short of a solution.

Tom.


>
> Karen
>>
>>>
>>> Also:
>>>
>>>  "However, implementations have practical limits on the number of
>>>   chunks_or_  _segments_  they are prepared to process in one of these
>>>   lists."
>>>
>>> Agreed, but what is the practical limit?  There are cases where a
>>> solaris client can send up to 16 segments in one write chunk. WOuld
>>> that be considered under the "practical limit"?
>>
>> I'm afraid there is not enough experience to define such a limit. This
>> document could attempt to define one, but that may be problematic. The
>> stated goal of this -bis update is to clarify, not to make new
>> requirements.
>>
>> Tom.
>>
>>>
>>> Karen
>>>
>>>
>>>>
>>>> Feel free to wordsmith further.
>>>>
>>>> Tom.
>>>>
>>>>
>>>> On 4/26/2017 12:18 PM, Chuck Lever wrote:
>>>>>
>>>>>> On Apr 24, 2017, at 7:30 AM, Chuck Lever <[e-mail]>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> On Apr 24, 2017, at 7:56 AM, Tom Talpey <[e-mail]> wrote:
>>>>>>>
>>>>>>> On 4/21/2017 10:43 AM, Chuck Lever wrote:
>>>>>>>> I agree that SHOULD/MAY makes things cloudier, and does not
>>>>>>>> seem to align with well-defined RFC2119 usage.
>>>>>>>>
>>>>>>>> Another way we've dealt with similar disagreements between
>>>>>>>> specification and implementation is to decide that one of
>>>>>>>> the implementations is incorrect.
>>>>>>>>
>>>>>>>> Can we agree that:
>>>>>>>>
>>>>>>>> - GARBAGE_ARGS is a bit of a layering violation, though it's
>>>>>>>> understandable why it might be returned
>>>>>>>>
>>>>>>>> - RPC clients are already prepared for GARBAGE_ARGS
>>>>>>>
>>>>>>> Are you certain of this?
>>>>>>
>>>>>> GARBAGE_ARGS has been part of the RPC protocol for decades.
>>>>>> The two Unix-flavored clients that have NFS/RDMA support can
>>>>>> both handle this error.
>>>>>
>>>>> I've confirmed that the only other known NFS/RDMA client
>>>>> (Oracle dNFS) properly recognizes GARBAGE_ARGS.
>>>>>
>>>>>
>>>>>>> And out of curiosity, what is returned
>>>>>>> to the consumer for GARBAGE_ARGS versus ERR_CHUNK?
>>>>>>
>>>>>> RFC 5531:
>>>>>>> GARBAGE_ARGS  = 4, /* procedure can’t decode params   */
>>>>>>
>>>>>>
>>>>>> GARBAGE_ARGS is an RPC-level error. The reply is "accepted"
>>>>>> with accept_stat GARBAGE_ARGS. An XID is available in the
>>>>>> header.
>>>>>>
>>>>>> rfc5666bis:
>>>>>>> If the rdma_vers field contains a recognized value, but an
>>>>>>> XDR parsing error occurs, the responder MUST reply with an
>>>>>>> RDMA_ERROR procedure and set the rdma_err value to ERR_CHUNK.
>>>>>>
>>>>>>
>>>>>> ERR_CHUNK is a transport level error. An XID is available
>>>>>> in the header.
>>>>>>
>>>>>> The difference is that the RPC layer v. the transport layer
>>>>>> are reporting they don't understand the contents of the
>>>>>> message (Call). There is nothing more in either type of
>>>>>> message.
>>>>>>
>>>>>>
>>>>>>>> - In RPC-over-RDMA Version One, we are not trying to recover
>>>>>>>> (in the sense of resending a simpler COMPOUND) but are rather
>>>>>>>> trying to ensure the offending RPC is properly terminated on
>>>>>>>> the client, and does not further block other RPCs or deadlock
>>>>>>>> the transport
>>>>>>>>
>>>>>>>> Thus I claim it is harmless if a server returns GARBAGE_ARGS
>>>>>>>> instead of ERR_CHUNK.
>>>>>>>
>>>>>>> "Harmless" is a bit relative. The operation fails, through no fault
>>>>>>> of the consumer. And, frankly, in a very mysterious way.
>>>>>>
>>>>>> We have no richer way of communicating failure in RPC-over-RDMA
>>>>>> Version One. We are not looking for recovery here, so I don't
>>>>>> believe any more information would be useful. If the server
>>>>>> wishes, it can log the failure with a message explaining what
>>>>>> went wrong.
>>>>>>
>>>>>>
>>>>>>> Again, I think there is more to say here. It's a limitation of the
>>>>>>> protocol whose implications should be made clear (contraining the
>>>>>>> complexity of COMPOUNDs, limiting scatter/gather lengths, etc).
>>>>>>
>>>>>> I'd welcome any suggested text.
>>>>>>
>>>>>> Honestly, I'm not sure what can be said. Neither NFSv4.0 nor
>>>>>> RPC-over-RDMA have a sophisticated mechanism to communicate this
>>>>>> kind of limitation. The best an NFSv4 server can do is return
>>>>>> NFS4ERR_RESOURCE, which also carries little extra information
>>>>>> about what a client should do to recover.
>>>>>>
>>>>>> So are you comfortable with eliminating GARBAGE_ARGS if we can
>>>>>> come up with more detail about the impact of not knowing how
>>>>>> complex a COMPOUND can be?
>>>>>
>>>>> I've come up with some possible replacement text for
>>>>> the final two paragraphs of S5.4.1 in an attempt to
>>>>> address comments from Tom, David, and Karen. The
>>>>> normative requirements have been removed, and a (brief)
>>>>> discussion of the consequences of not handling complex
>>>>> COMPOUNDs was introduced.
>>>>>
>>>>>
>>>>> 5.4.2.  Complexity Considerations
>>>>>
>>>>>    As mentioned above, an NFS version 4 COMPOUND procedure can contain
>>>>>    more than one operation that carries a DDP-eligible data item.  The
>>>>>    RPC-over-RDMA Version One protocol does not place any limit on the
>>>>>    number of chunks that may appear in the Read or Write lists.
>>>>>    Therefore an NFS version 4 client MAY construct an RPC-over-RDMA
>>>>>    Version One message containing more than one Read chunk or Write
>>>>>    chunk.
>>>>>
>>>>>    However, implementations have practical limits on the number of
>>>>>    chunks or segments they are prepared to process in one of these
>>>>>    lists.  There are several ways an NFS Version 4 server might
>>>>> indicate
>>>>>    that an RPC Call message constructed by a client is valid but
>>>>> cannot
>>>>>    be processed because of implementation limitations:
>>>>>
>>>>>    o  If the problem is detected in the upper layer (i.e., by the NFS
>>>>>       version 4 implementation), the server returns an NFS status of
>>>>>       NFS4ERR_RESOURCE.
>>>>>
>>>>>    o  If the problem is detected during XDR decoding of the request
>>>>>       (e.g., during re-assembly of the Call message by the RPC layer),
>>>>>       the server returns an RPC accept_stat of GARBAGE_ARGS.
>>>>>
>>>>>    o  If the problem is detected at the transport layer (i.e., during
>>>>>       transport header processing), the server returns an RDMA_ERROR
>>>>>       message with the err_code field set to ERR_CHUNK.
>>>>>
>>>>>    Such implementation limits can constrain the complexity of NFS
>>>>>    version 4 COMPOUNDs, limit the number of elements in scatter-gather
>>>>>    operations, or prevent the use of Kerberos integrity or privacy
>>>>>    services.
>>>>>
>>>>>
>>>>> Comments, opinions on this approach?
>>>>>
>>>>>
>>>>>>> Tom.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> As a result, I can change the Read list text in S5.4.1 to be
>>>>>>>> the same as the Write list text, removing the mention of
>>>>>>>> GARBAGE_ARGS.
>>>>>>>>
>>>>>>>> Would that sit comfortably with everyone?
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Apr 20, 2017, at 7:21 PM, David Noveck <[e-mail]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The "or" is a similar situation, it prescribes a choice, which
>>>>>>>>>> does not define a protocol.
>>>>>>>>>
>>>>>>>>> Fair enough, but the point that needs to be made is that, with
>>>>>>>>> regard to Version One, Chuck and  the working group is not
>>>>>>>>> free to define a protocol.  As a result we have the kind of
>>>>>>>>> ugliness you object to, but it is inherent in the choice to try to
>>>>>>>>> revive Version One as-is.
>>>>>>>>>
>>>>>>>>>> If an NFS version 4 client sends an RPC Call with a Read list
>>>>>>>>>> that
>>>>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>>>>> process, the server SHOULD reject the request by responding
>>>>>>>>>> with an
>>>>>>>>>> RDMA_ERROR message with the rdma_err value set to ERR_CHUNK. The
>>>>>>>>>> server MAY reject the RPC with an RDMA_MSG message containing an
>>>>>>>>>> RPC
>>>>>>>>>> Reply with an accept status of GARBAGE_ARGS.
>>>>>>>>>
>>>>>>>>> I think I know what you intend here and I've seen stuff like this
>>>>>>>>> in RFCs but I don't
>>>>>>>>> wthink e can do this because this is not in line with the
>>>>>>>>> definitions of "SHOULD"
>>>>>>>>> and "MAY" that appear in RFC2119.
>>>>>>>>>
>>>>>>>>> "SHOULD" means that you are supposed to do something but can
>>>>>>>>> avoid it if
>>>>>>>>> you have a good reason and are aware of the consequences of not
>>>>>>>>> doing it.
>>>>>>>>> In this case the "good" reason is that someone coded the
>>>>>>>>> implementation
>>>>>>>>> to do something else, which is not all that good a reason.  The
>>>>>>>>> consequences of
>>>>>>>>> returning the GARBAGEARGS are exactly zero, since the client has
>>>>>>>>> to be prepared
>>>>>>>>> for either it or ERR_CHUNK.
>>>>>>>>>
>>>>>>>>> "MAY" means the implementation can choose to do the action or
>>>>>>>>> not, which is line
>>>>>>>>> with the reality here but essentially contradicts the SHOULD.
>>>>>>>>>
>>>>>>>>>> This at least makes it clear which response is "preferred".
>>>>>>>>>
>>>>>>>>> But it is isn't really the job of the RFC2119 terms to say which
>>>>>>>>> is "preferred" or
>>>>>>>>> "'preferred'".  These terms are supposed to describe
>>>>>>>>> interoperability and the
>>>>>>>>> interoperability situation is that the server MUST return
>>>>>>>>> ERR_CHUNK or
>>>>>>>>> GARBAGEARGS and the client needs to be prepared for either.  That
>>>>>>>>> is the
>>>>>>>>> unpleasant reality.  If you want to indicate a preference, you
>>>>>>>>> can say something
>>>>>>>>> like:
>>>>>>>>>     • Returning ERR_CHUNK is preferrable.
>>>>>>>>>     • Returinng ERR_CHUNK is more in line with the appropriate
>>>>>>>>> protocol layering since this issue relates to a limitation of the
>>>>>>>>> transport implementation.
>>>>>>>>>     • Use of GARBAGEARGS is an unfortunate artifact of
>>>>>>>>> inappropriately layered implementations and is only allowed for
>>>>>>>>> reasons of compatibility with existing implementations.  It is
>>>>>>>>> desirable to avoid it.
>>>>>>>>>> And one would hope a future draft would decide.
>>>>>>>>>
>>>>>>>>> Not sure what draft you are thinking of.  I don't see us doing an
>>>>>>>>> rfc5667bisbis (rfc5667tris).
>>>>>>>>>
>>>>>>>>> By the time we did that, the implementations with these
>>>>>>>>> restrictions will probably be gone.
>>>>>>>>>
>>>>>>>>>> I have a second question though. How does the client determine
>>>>>>>>>> what is
>>>>>>>>>> the actual error? As in, how many chunks were allowed?
>>>>>>>>>
>>>>>>>>> This is not fixable in Version One.  It would be in Version Two,
>>>>>>>>> but by then
>>>>>>>>> the need will probably be gone.
>>>>>>>>>
>>>>>>>>>> Does the upper
>>>>>>>>>> layer have to recover, and if so what are the implications?
>>>>>>>>>
>>>>>>>>> I think something could be put in to indicate that clients should
>>>>>>>>> break up COMPOUNDS
>>>>>>>>> so the only have a single chunk each.
>>>>>>>>>
>>>>>>>>>> Yes, I know 5667 did not explore this very well.
>>>>>>>>>
>>>>>>>>> It didn't explore it at all.  And 5666's error reporting
>>>>>>>>> facilities were extremely limited.
>>>>>>>>>
>>>>>>>>>> Mea culpa.
>>>>>>>>>
>>>>>>>>> I don'tt think you have anything to apologize for.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Apr 20, 2017 at 5:28 PM, Tom Talpey <[e-mail]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 4/19/2017 11:14 AM, Chuck Lever wrote:
>>>>>>>>> Hi Tom-
>>>>>>>>>
>>>>>>>>> On Apr 18, 2017, at 11:08 PM, Tom Talpey <[e-mail]> wrote:
>>>>>>>>>
>>>>>>>>> I noticed the same thing, and I'll add that the MUST reject
>>>>>>>>> condition
>>>>>>>>> is very confusing because it allows an "or". In my opinion a
>>>>>>>>> MUST is
>>>>>>>>> always a single requirement, never ambiguous.
>>>>>>>>>
>>>>>>>>> I agree this kind of thing is tricky. I wrote it as "the server
>>>>>>>>> MUST
>>>>>>>>> reject the RPC". That's the single requirement. The choice is how
>>>>>>>>> the
>>>>>>>>> rejection is conveyed to the client.
>>>>>>>>>
>>>>>>>>> The statement "MUST reject" is not testable. So, while it may be
>>>>>>>>> understood what is intended, there is nothing implementable in the
>>>>>>>>> MUST. The "or" is a similar situation, it prescribes a choice,
>>>>>>>>> which
>>>>>>>>> does not define a protocol.
>>>>>>>>>
>>>>>>>>> Is there some reason you want to allow such a choice? I think
>>>>>>>>> you'll
>>>>>>>>> find that, worded properly, it becomes actually much less
>>>>>>>>> implementable
>>>>>>>>> and interoperable than you may think.
>>>>>>>>>
>>>>>>>>> The Solaris server can return an RPC-level error in cases like
>>>>>>>>> this.
>>>>>>>>>
>>>>>>>>> Well, this is happening because the Solaris server is (probably)
>>>>>>>>> just
>>>>>>>>> handing the chunk list up to the RPC layer, and it's the RPC (XDR)
>>>>>>>>> processing that detects any problem.
>>>>>>>>>
>>>>>>>>> On the other hand, an implementation could do the opposite, it
>>>>>>>>> could
>>>>>>>>> process the chunks at the lower layer, before ever invoking RPC
>>>>>>>>> processing. This would naturally lead to a non-RPC error.
>>>>>>>>>
>>>>>>>>> The challenge in defining the protocol is to hide these
>>>>>>>>> possibilities.
>>>>>>>>>
>>>>>>>>> I think there are similar choices allowed in rfc5666bis. Let's say
>>>>>>>>> that in a perfect world, I would go with only ERR_CHUNK, but I'm
>>>>>>>>> documenting existing implementation behavior here.
>>>>>>>>>
>>>>>>>>> I'm not sure it matters to the client: both errors are
>>>>>>>>> permanent and
>>>>>>>>> the RPC is terminated on the client.
>>>>>>>>>
>>>>>>>>> I'm open to alternatives.
>>>>>>>>>
>>>>>>>>> The icky way to do this is to split into two weak requirements.
>>>>>>>>>
>>>>>>>>>  If an NFS version 4 client sends an RPC Call with a Read list
>>>>>>>>> that
>>>>>>>>>  contains more chunks than an NFS version 4 server is prepared to
>>>>>>>>>  process, the server SHOULD reject the request by responding
>>>>>>>>> with an
>>>>>>>>>  RDMA_ERROR message with the rdma_err value set to ERR_CHUNK. The
>>>>>>>>>  server MAY reject the RPC with an RDMA_MSG message containing an
>>>>>>>>> RPC
>>>>>>>>>  Reply with an accept status of GARBAGE_ARGS.
>>>>>>>>>
>>>>>>>>> This at least makes it clear which response is "preferred". And
>>>>>>>>> one
>>>>>>>>> would hope a future draft would decide.
>>>>>>>>>
>>>>>>>>> I have a second question though. How does the client determine
>>>>>>>>> what is
>>>>>>>>> the actual error? As in, how many chunks were allowed? Does the
>>>>>>>>> upper
>>>>>>>>> layer have to recover, and if so what are the implications?
>>>>>>>>>
>>>>>>>>> Yes, I know 5667 did not explore this very well. Mea culpa.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Tom.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 4/18/2017 6:32 PM, karen deitke wrote:
>>>>>>>>> Hi Chuck, its unclear what you mean by "is prepared to process"
>>>>>>>>> in the text below.
>>>>>>>>> Other than that, looks good.
>>>>>>>>>
>>>>>>>>> Karen
>>>>>>>>>
>>>>>>>>> 5.4.1
>>>>>>>>> If an NFS version 4 client sends an RPC Call with a Write list
>>>>>>>>> that
>>>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>>>> process, the server MUST reject the RPC by responding with an
>>>>>>>>> RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If an NFS version 4 client sends an RPC Call with a Read list that
>>>>>>>>> contains more chunks than an NFS version 4 server is prepared to
>>>>>>>>> process, the server MUST reject the RPC by responding with an
>>>>>>>>> RDMA_MSG message containing an RPC Reply with an accept status of
>>>>>>>>> GARBAGE_ARGS, or with an RDMA_ERROR message with the rdma_err
>>>>>>>>> value
>>>>>>>>> set to ERR_CHUNK.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 4/18/2017 1:21 PM, David Noveck wrote:
>>>>>>>>> *Overall Evaluation*
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> Major improvement over RFC5667.  Almost ready to ship. No
>>>>>>>>> technical
>>>>>>>>> issues.
>>>>>>>>>
>>>>>>>>> A lot of my comments are basically editorial and are offered on a
>>>>>>>>> take-it-or-lease-it basis.
>>>>>>>>>
>>>>>>>>> I think some clarification in Section 5.4.1 is needed although not
>>>>>>>>> necessarily in the ways suggested below,
>>>>>>>>>
>>>>>>>>> *Comments by Section*
>>>>>>>>> *5.4.1. Multiple DDP-eligible Data Items*
>>>>>>>>> Giventhat READ_PLUS no longer has any DDP-eligible data items, the
>>>>>>>>> situation described in the fifth bullet can no longer arise. I
>>>>>>>>> suggest
>>>>>>>>> deleting the bullet.
>>>>>>>>> The penultimate paragraph can be read as applying to some
>>>>>>>>> situations
>>>>>>>>> in which it shouldn't and where the extra chunks would very
>>>>>>>>> naturally
>>>>>>>>> ignored. For example, if you had on write chunk together with a
>>>>>>>>> READ
>>>>>>>>> operation which failed, the server would have more chunks (i.e.
>>>>>>>>> one)
>>>>>>>>> than the number it is prepared to process (i.e. zero). Suggest,
>>>>>>>>> as a
>>>>>>>>> possible replacement:
>>>>>>>>>
>>>>>>>>>  Normally, when an NFS version 4 client sends an RPC Call with a
>>>>>>>>>  Write list that contains multiple chunks. each such, when matched
>>>>>>>>>  with a DDP-eligible data item in the response, directs the
>>>>>>>>>  placement of the data item as specified by
>>>>>>>>>  [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items
>>>>>>>>>  matched to write chunks that an NFS version 4 server is not
>>>>>>>>>  prepared to process, the server MUST reject the RPC by responding
>>>>>>>>>  with an RDMA_ERROR message with the rdma_err value set to
>>>>>>>>> ERR_CHUNK.
>>>>>>>>>
>>>>>>>>> With regard to the last paragraph, I am curious that this
>>>>>>>>> paragraph,
>>>>>>>>> unlike the previous one, allows GARBGEARGS. Is this so because
>>>>>>>>> that
>>>>>>>>> would be allowed if the chunks in question had offsets other than
>>>>>>>>> those that correspond to DDP-eligible data items? If so, please
>>>>>>>>> consider the following possible replacement.
>>>>>>>>>
>>>>>>>>>  Normally, when an NFS version 4 client sends an RPC Call with a
>>>>>>>>>  Read list that contains multiple chunks, each such, when properly
>>>>>>>>>  matched with a DDP-eliigible data item in the request, directs
>>>>>>>>> the
>>>>>>>>>  fetching of the the data item as specified by
>>>>>>>>>  [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items
>>>>>>>>>  matched to read chunks that an NFS version 4 server is not
>>>>>>>>>  prepared to process, the server MUST reject the RPC by responding
>>>>>>>>>  with an RDMA_ERROR message with the rdma_err value set to
>>>>>>>>> ERR_CHUNK.
>>>>>>>>>
>>>>>>>>> *5.6. Session-Related Considerations*
>>>>>>>>> In the third sentence of the second paragraph, suggest
>>>>>>>>> replacing "no
>>>>>>>>> different" by "not different".
>>>>>>>>> In the last sentence of the last paragraph, suggest replacing "is
>>>>>>>>> not"
>>>>>>>>> by "were not"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> nfsv4 mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> nfsv4 mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> nfsv4 mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Chuck Lever
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> nfsv4 mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> nfsv4 mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> nfsv4 mailing list
>>>>>>>>> [e-mail]
>>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>
>>>>>>>> --
>>>>>>>> Chuck Lever
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> nfsv4 mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> nfsv4 mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>>
>>>>>> --
>>>>>> Chuck Lever
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> nfsv4 mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>
>>>>> --
>>>>> Chuck Lever
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> nfsv4 mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>>
>>>>
>>>> _______________________________________________
>>>> nfsv4 mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>
>>> _______________________________________________
>>> nfsv4 mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/nfsv4
>
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/nfsv4