Re: [nfsv4] [Technical Errata Reported] RFC8166 (6528)

[Chuck Lever III <[e-mail]>]

> On Apr 21, 2021, at 9:51 AM, Tom Talpey <[e-mail]> wrote:
> 
> On 4/21/2021 9:25 AM, Chuck Lever III wrote:
>>> On Apr 21, 2021, at 9:20 AM, Tom Talpey <[e-mail]> wrote:
>>> 
>>> On 4/20/2021 1:13 PM, Chuck Lever III wrote:
>>>>> On Apr 20, 2021, at 11:36 AM, Tom Talpey <[e-mail]> wrote:
>>>>> 
>>>>> On 4/20/2021 11:17 AM, Chuck Lever III wrote:
>>>>>> Sorry for the delay, I wasn't sure where to begin further conversation
>>>>>> on this errata.
>>>>>>> On Apr 12, 2021, at 12:40 PM, RFC Errata System <[e-mail]> wrote:
>>>>>>> 
>>>>>>> The following errata report has been submitted for RFC8166,
>>>>>>> "Remote Direct Memory Access Transport for Remote Procedure Call Version 1".
>>>>>>> 
>>>>>>> --------------------------------------
>>>>>>> You may review the report below and at:
>>>>>>> https://www.rfc-editor.org/errata/eid6528
>>>>>>> 
>>>>>>> --------------------------------------
>>>>>>> Type: Technical
>>>>>>> Reported by: David Noveck <[e-mail]>
>>>>>>> 
>>>>>>> Section: 4.2.4
>>>>>>> 
>>>>>>> Original Text
>>>>>>> -------------
>>>>>>> A Requester MUST NOT send an RPC-over-RDMA header with the RDMA_ERROR
>>>>>>> procedure.  A Responder MUST silently discard RDMA_ERROR procedures
>>>>>> Stated as a co-author of this document, I agree that this text is
>>>>>> problematic for version negotiation, which uses the RDMA_ERROR
>>>>>> procedure/header type.
>>>>>> One vote for "verified".
>>>>>>> Corrected Text
>>>>>>> --------------
>>>>>>> An RDMA_ERROR header cannot be sent without an assurance that, the receiver
>>>>>>> has posted a receive operation which its sending will satisfy.  In most cases,
>>>>>>> this means that a Requester (i.e. one sending RPC Calls) MUST NOT send an
>>>>>>> RPC-over-RDMA header with the RDMA_ERROR procedure.  Similarly, a Responder
>>>>>>> (i.e. one sending RPC Replies) MUST silently discard RDMA_ERROR procedures.
>>>>>> I don't recall that the issue that prompted the addition of the
>>>>>> original text was the requirement for a posted Receive. That
>>>>>> might be slightly revisionist.
>>>>>> Though it is true sending an RDMA_ERROR when there is no Receive
>>>>>> waiting will likely result in loss of connection, I think the
>>>>>> typical scenario is that RDMA_ERROR is in response to a previous
>>>>>> message; thus missing a Receive would be an implementation bug.
>>>>> 
>>>>> Why is this particular RDMA_ERROR so important? It seems to me like
>>>>> a ton of normative text for a version mismatch, which basically
>>>>> should never happen.
>>>> It will happen now that we have rpcrdma-version-two.
>>>>> If anything, changing "silently discard" to
>>>>> "close the connection" would be just as appripriate.
>>>> I prefer to avoid "close the connection" as a first resort since
>>>> it can impact other RPC transactions that are ongoing. That might
>>>> not be a concern for ERR_VERS, but it could very well be a
>>>> problem for ERR_CHUNK.
>>>>> Can someone motivate the scenario for me?
>>>> The problem is that RFC 8166 defines only forward-direction
>>>> operation, where the Requester/Responder roles are clear. With
>>>> the introduction of reverse-direction operation and control
>>>> plane operations, it becomes impossible to implement the
>>>> requirements stipulated in S4.2.4 because there are now scenarios
>>>> where the Receiver has no way to determine whether it is a
>>>> Requester or a Responder.
>>> 
>>> Oh. So, the idea is that the ULP reverse channel might attempt to
>>> establish an RPCRDMA version which might be different from the forward
>>> channel?
>> Or it could be due to a bug. The point is this text in RFC 8166
>> can't work.
>>> How is that useful, or even plausible? I think the answer will
>>> guide the proposed fix.
>>> 
>>> I don't think, in any case, that it's at all useful to allow the
>>> version to change after both sides have exchanged messages with
>>> matching versions. If that happens, closing the connection is the
>>> only valid action.
>> This errata deals strictly with RFC 8166. For an RPC/RDMA v1 client,
>> why wouldn't sending an ERR_VERS work for v1?
>> For later versions, we'll need a new error code to indicate that
>> the Receiver detected a supported but unexpected version.
> 
> I'll find time to take a closer look. I fully agree that an erratum
> needs to be tightly scoped.

Thanks for having a look!


> Longer term, I'm skeptical of the value
> of extending error codes for this. A v1 RDMA_ERROR reply seems like
> the most universal approach, on the face of it. But heroically
> attempting to recover from this is, to me, a complete non-goal.

I'm not necessarily interested in recovery. There are two basic
principals that guide behavior regarding protocol errors:

- Try hard not to break the connection because that can perturb
other ongoing RPC transactions.

- Try hard not to leave pending RPC transactions hanging on the
Requester. In general, dropping a connection does nothing to
help the Requester retire the erroneous transaction.

The purpose of an explicit error response here would be to enable
the Requester to terminate the transaction instead of leaving it
hanging. RDMA2_ERR_SYSTEM would be adequate for that, but that
would imply that the Responder is at fault.

More extensive recovery is a quality-of-implementation issue, and
I tend to think heroic recovery in this case is probably not of
high value.


> Tom.
> 
>>>> I think Dave is concerned about how these requirements impact the
>>>> operation of versions subsequent to v1, or during version negotiation
>>>> between a v1-only implementation and a polyglot.
>>>>>> "In most cases," is not appropriate with MUST/MUST NOT. We could
>>>>>> stipulate that the exact cases where these requirements hold is
>>>>>> when sending ERR_CHUNK. These requirements do not hold for ERR_VERS.
>>>>> 
>>>>> I'll go further - "in most cases" isn't appropriate for a spec.
>>>>> 
>>>>> Tom.
>>>>> 
>>>>>>> However, in the case of providing an RDMA_ERROR headers containing an error
>>>>>>> code of ERR_VERS, such a schema is not realizable, since there is no way for
>>>>>>> a receiver who does not support a particular version, to determine whether
>>>>>>> an RPC Call or Reply is being sent, leaving the receiver uncertain as to whether
>>>>>>> it is being Addressed as a Requester or a Responder, leaving it unable to
>>>>>>> participate in version negotiation.
>>>>>> The particular issue is that either:
>>>>>> - the Sender of the message with the unsupported version might be
>>>>>> using a procedure of the new protocol version that is not supported
>>>>>> in the protocol version the Receiver supports, or
>>>>>> - the Sender of the message with the unsupported version might be
>>>>>> using a procedure that does not bear an RPC message or any other
>>>>>> indication of message direction.
>>>>>> Thus the Receiver would not be able to determine the direction of
>>>>>> the incoming message.
>>>>>> I'm not convinced that RFC 8166 Section 4.2 is an appropriate place
>>>>>> to explain this reasoning. (Stating it here in the errata is OK).
>>>>>> The above could be dropped in the Corrected text.
>>>>>>> In the case of version errors, the
>>>>>>> implementation is to rely on the assumption that forward direction requests
>>>>>>> are being done and reserve direction requests only done once the version is
>>>>>>> properly negotiated.
>>>>>> Shorter:
>>>>>> "For the purpose of version negotiation, the receiver is to assume
>>>>>> that reverse-direction operation is started only once the connection's
>>>>>> transport protocol version has been fixed using forward direction
>>>>>> operations."
>>>>>> One problem with this is that reverse-direction operation is defined
>>>>>> in a separate RFC (8167).
>>>>>>> As a result, such messages MUST NOT be sent by the
>>>>>>> client and MUST be silently discarded by servers.
>>>>>> I'd like to confirm that what the reporter intended in the second
>>>>>> sentence above was "client" and "servers" and not "Requester" and
>>>>>> "Responders".
>>>>>>> Notes
>>>>>>> -----
>>>>>>> Even if one feels that this is not an appropriate correction, the existing text must be fixed somehow.   In assuming that the terms Requester and Responder can be used this way in this context is likely to result in some implementers concluding that version errors can never be sent while other might be unabble to coonclude that given the effort expended in the spec to make such errors be interpretable by anf rpc-over-rdma version.
>>>>>>> 
>>>>>>> Instructions:
>>>>>>> -------------
>>>>>>> This erratum is currently posted as "Reported". If necessary, please
>>>>>>> use "Reply All" to discuss whether it should be verified or
>>>>>>> rejected. When a decision is reached, the verifying party
>>>>>>> can log in to change the status and edit the report, if necessary.
>>>>>>> 
>>>>>>> --------------------------------------
>>>>>>> RFC8166 (draft-ietf-nfsv4-rfc5666bis-11)
>>>>>>> --------------------------------------
>>>>>>> Title               : Remote Direct Memory Access Transport for Remote Procedure Call Version 1
>>>>>>> Publication Date    : June 2017
>>>>>>> Author(s)           : C. Lever, Ed., W. Simpson, T. Talpey
>>>>>>> Category            : PROPOSED STANDARD
>>>>>>> Source              : Network File System Version 4
>>>>>>> Area                : Transport
>>>>>>> Stream              : IETF
>>>>>>> Verifying Party     : IESG
>>>>>> --
>>>>>> Chuck Lever
>>>>>> _______________________________________________
>>>>>> nfsv4 mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>>> 
>>>>> _______________________________________________
>>>>> nfsv4 mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>>> --
>>>> Chuck Lever
>> --
>> Chuck Lever

--
Chuck Lever