Re: [nfsv4] Credit management and one-way messages

[Chuck Lever <[e-mail]>]

> On Aug 1, 2017, at 1:51 PM, David Noveck <[e-mail]> wrote:
> 
> Ouch!!
> 
> If i was addressing an unduly narrow problem, it was because it was framed around the difficulties that might be added by adding one-way messages such as those added by Version 2.  If you consider, as I guess makes sense, RPCs that are not replied to as effectively one-way messages, that puts an entirely different spin (of possibly high angular momentum) on the problem.
> 
> Let me try to start addressing this larger problem.  I hope it doesn't make anyone (else) dizzy.
> 
> One reason that this is hard to address is that we have different views regarding the roles of the credit management logic regarding replies.  In my view, there is no point in a requester trying to give a credit to the responder allowing the responder to send the response to the requester.  I thought that the right thing was for the requester to ensure that the responders response could be read was for the requester to post a receive (or ensure that one was present) before sending the request.  I though that this happened outside the ambit of credit management because there is no real need for it to be within the scope of credit  management.  The requester can be responsible for providing receives corresponding with in-flight requests without involving the responder in the decision, which involves complexity, especially when multipeRPC directions are involved.  I never found anythng in the spec that told me I was wrong in that, but i just may be missing it.
> 
> I think we discussed this a while ago but no change in the spec was made to either make it clear I was wrong, or indicate that I was right.  My recollection is that you treated my observation as correct/valid implementation advice and we never really addressed our different interpretations of the spec.  It seems to me that the spec was ambiguous on this point.   Of course, I might not be seeing something that is there and you might be simply assuming that something is implied because it seems natural, even if it is not explcitly there.

Another interpretation is that the spec describes only protocol,
and that after some trial-and-error, implementers will conclude
that there are only one or two ways to implement that protocol.


> If the spec still is not clear on this point, it might need to be fixed, if not to mandate my approach but at least to indicate how the requester can safeguard itself against the possibility that a response sent by the responder will cause a disconnect.
> 
> With this approach, RPC's not replied to would still be a problem, but the problem cannot result in a disconnect.  For each not-replied-to request, there would be a receive that was not included in the credits, which would be devoted in this model to receives for receiving requests (and things intended to be one-way messages).   Of course, with 1K or 4K buffers that would not be such a big deal and you could defer any connection reset until a time when it was easy to do.  

A requester is responsible for posting enough receive buffers to
catch Replies for all outstanding RPC Calls. So either:

A. It can post a receive buffer as part of preparing to send a Call.
If a Reply is missed, that receive is still posted. The requester
has to accommodate for that.

B. It can batch-post receive buffers (say, as part of receive
completion handling) in order to keep more than enough available.

The problem, as I see it, occurs when a responder wants to send a
one-way message to a requester. In scenario A, that knocks down a
receive buffer that was intended to catch a Reply. The requester
is responsible for getting a replacement receive buffer posted,
but there's a possibility that it could not do so fast enough
to catch other incoming messages.

This is the fundamental issue with one-way messages. A sender
has no way to know when the receiver is prepared to receive
additional messages because there is no message acknowledgement
in RPC-over-RDMA version 1, other than RPC Call and Reply messages.


A responder is responsible for posting enough receive buffers to
catch "credit" RPC Calls. It relies on the fact that requesters
can't keep more than "credit" RPC Calls in flight, to know exactly
how many receive buffers it needs to keep posted. So either:

A. It posts "credit" receive buffers when accepting a connection,
and then it posts a receive buffer as part of preparing to send a
Reply. If sending fails, that receive is still posted, and the
responder has to accommodate for that.

B. It can batch-post receive buffers in order to keep more than
enough available.

The problem, as I see it, occurs when a requester sends enough
one-way messages that it prevents the transmission of more
RPC Calls. The responder cannot send Replies to those one-way
messages, so the requester has no way to know when the responder
is ready to receive another RPC Call.


I think that credit management will have to take a different
form if we believe one-way messages are a necessary part of the
future of RPC-over-RDMA.


--
Chuck Lever