Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

[Karen <[e-mail]>]

On 5/20/16 12:44 PM, Chuck Lever wrote:
>> On May 20, 2016, at 2:30 PM, Karen <[e-mail]> wrote:
>>
>>
>>
>> On 5/20/16 10:41 AM, Chuck Lever wrote:
>>> Hi Karen-
>>>
>>> Thanks for the review. I'll respond below to questions
>>> and non-editorial comments. I'll merge the other comments
>>> before the next revision.
>>>
>>>
>>>> On May 20, 2016, at 12:11 PM, Karen <[e-mail]> wrote:
>>>>
>>>> Hi Chuck,
>>>>
>>>> Here is some feedback on draft-ietf-nfsv4-rpcrdma-bidirection-03
>>>> Karen
>>>>
>>>>
>>>> Abstract section:
>>>>
>>>> The use of the word "recent" seems relative to time.  5 years later, recent will no longer be recent.  Not sure how much of a big deal this is, just seems maybe instead of putting "recent minor versions of NFSv4", a more precise explanation would be "With the introduction of NFSv4.0".
>>>>
>>>>
>>>> 2.1
>>>>
>>>> "This traditional form of ONC RPC message passing is referred to as operation in the "forward direction"."
>>>>
>>>> "as operation in the forward direction" seems awkward.  "is referred to as the "forward direction"." seems a bit more concise and clearer.
>>>> 2.2
>>>> Same as in section 2.2 in "This form of message passin is referred to as operation in the "backward direction."
>>>>
>>>> Also again the use of the word recently, as in "Not until recently", seems that "Not until NFSV4.0" would be clearer.
>>>>
>>>> 3.1
>>>> "An NFSv4 "delegation" is simply a promise made by a server that it will notify a client before another agent is allowed access to a file"
>>>>
>>>> another agent? seems "a different client" would be clearer.
>>> A server may also host application programs, which can
>>> open files too. When a local application opens a file
>>> delegated to a client, the server would recall that
>>> delegation.
>>>
>>> So "agent" means other clients, or programs running on
>>> the server.
>> Ah, I see, ok, thanks, the term agent in this sense was confusing to me.
>>>
>>>> 3.2
>>>>
>>>> "Without a backward direction RPC-over-RDMA capability, an NFSv4.1 client must additionally connect using a transport with backward direction capability to use as a backchannel."
>>>>
>>>> I understand the point, and agree, but the sentence seems mumbled.  Maybe its confusing to me because of the use of the word backward direction.  Maybe backward direction needs to be defined as a connection that is initiated by a requestor, but also used to accept requests from the respondor, in which the requestor then reponds?  Not sure if that wording is any clearer, but the idea that the requests are sent from the side that did not create the connection?
>>> Isn't the definition in section 2.2 sufficient?
>> Re-read that section, yes this is sufficient.
>>>
>>>> 4
>>>>
>>>> "For an RDMA Send operation to work, the receiving peer must have posted an RDMA Receive Work Request (WR) to provide a receive buffer in which to land the incoming message.
>>>>
>>>> why is the posted receive buffer being called a posted receive work request? Seems this would be clearer
>>>> "For an RDMA Send operation to work, the receiving peer must have posted an RDMA receive buffer as the location in which the RDMA send request will land."
>>> Each receive buffer is posted by posting a Receive Work
>>> Request on the QP's receive queue. "Posting a receive
>>> buffer" is a more colloquial way of saying it, posting
>>> a Receive Work Request is perhaps more precise.
>>> An argument could be made for switching to one terminology
>>> or the other throughout the document. But which one is
>>> more clear? Opinions?
>> Right, both are used and should be consistent.  Speaking for me, "receive buffer" is clearer than "receive work request".
>>>
>>>> And similarly
>>>>
>>>>
>>>> "If a receiver hasn't posted enough Receive WRs to land incoming Send operations..."
>>>>
>>>> If a receiver hasn't posted enough receive buffers to accept incoming send requests"
>>>>
>>>> (Why is Receive capitalized?)
>>>>
>>>>
>>>> 4.1
>>>>
>>>> "When message direction is not fully determined by context"
>>>>
>>>> "fully determined by context" thats confusing, what does that really mean?  I think this means when the rdma header does not directly indicate if the message is a call or reply, but the wording is confusing.
>>> That means that in some cases the receiver can guess
>>> accurately which direction the message was going, based
>>> on the context of the operation, even without having
>>> an RPC message payload.
>>>
>>> I'm open to suggestions.
>>>
>>> I think some prefer that the document simply state that
>>> direction is always unknown in cases where an RPC
>>> message payload is not present. That kind of opens a
>>> can of worms with RDMA_ERROR, which is needed to report
>>> that the client does not support backward direction
>>> operation.
>> I don't think that it can always absolutely be clear which direction without the RPC header. Or am I misunderstanding RPC payload? I'm taking that to mean the RPC header, but maybe you are referring to the NFS data in the rpc payload?
> "Context" here means that if the receiver can tell
> by other means (like, there are no other outstanding
> operations). So no, it's not always going to be clear.
> In those cases, direction is not known.
I don't think there is ever a 100% way to know the direction without the 
rpc header's call or reply field.  Even if there is an outstanding 
request, and it is for the xid that is defined in the rdma header, being 
that the xid is not unique between client and server, there still 
exists, though extremely small, a possiblity that this is a new request 
that just so happens to have the same xid and is not actually the reply 
to an outstanding operation.
>
>
>>>> 4.2
>>>>
>>>> "If a sender transmits a message for a receiver which has no prepared receive buffer"
>>>>
>>>> the word "posted" not "prepared" was used earlier, to be consisted "posted" would read more clearly than "prepared"
>>>>
>>>> 4.2.2
>>>> "A forward direction RPC-over-RDMA service endpoint"
>>>>
>>>>
>>>> did you mean "server" instead of "service"?
>>>>
>>>> "To receive incoming backward direction replies, an RPC-over-RDMA server endpoint must pre-post a receive buffer for each backward direction Call it sends"
>>>>
>>>> What prevents a buggy, or hacking client from sending more requests that forward credits and preventing a reply in the backward direction from arriving? Could a buggy or hacky client cause a denial of service by sending more than credit number of requests?  Guess that could happen in the standard single direction and would result in the connection closing.  Maybe not an issue?
>>> The same issue exists in the forward direction.
>> Right, agreed, guess its part of the nature of ib.
>>>
>>>> 5.1
>>>>
>>>> last sentence, "and the header's msg_type field MUST conatin the value CALL",
>>>>
>>>> for more clarification "and the RPC header's msg_type field", this is a nit...
>>>>
>>>> 5.4
>>>>
>>>> "If an ONC RPC client has no work to do, it may be some time before it re-establishes a transport connection. Backward direction Requesters must be prepared to wait indefinitely for a connection to be established before a pending backward direction ONC RPC Call can be retransmitted."
>>>>
>>>> I don't believe there are currently an non-idempotent backchannel operations, but if there ever were to be, seems that the client should detect if it has any drc replies and re-create the bi-directional, or even one way backchannel connection, right after it detects the connection is  gone.
>>> Does backward direction RPC/TCP do this?
>> No, but only because there aren't any non-idempotent bc ops, and as such, I'm not sure we even had a backchannel drc.
>>> It certainly would be an issue for a backward-only connection.
>>> The client would have to notice that the connection was down
>>> and re-establish it quickly.
>> Yes, but again, we'd need a ulp that needs drc in the backward direction.  Not sure there are any, but not away of any upcoming nfs bc changes either.
> A DRC or session is going to be an upper layer affair;
> that is, IMO not the transport's responsibility.
>
> And for NFSv4.1, detecting loss of the backchannel is
> also an upper layer thing: the server asserts
> SEQ4_STATUS_CB_PATH_DOWN.
Agreed.
>
> The issue in this document is that somehow, the client
> has to detect the missing connection quickly in case
> the server wants to send it backchannel requests. The
> exact mechanism by which this occurs must be specified
> somewhere else.
>
> I'll review this text and see if it can be improved.
Ok, thanks.
>
>>>> Not addressed, and obvious to the implementer, but should it be mentioned that send buffers in the backward direction MUST be the same size as the posted receive buffers on the receiving side?  That is the backward connection must use the same send/recv buffer sizes and this can not be negotiated?
>>> Well, the I-D used to say that the buffer sizes are the
>>> same in both direction, but I guess I chopped it out at
>>> some point. I'll have to go back and look for it.
>> Ok.
>>
>> Karen
>>>
>>> --
>>> Chuck Lever
> --
> Chuck Lever
>
>
>