Re: [nfsv4] Credit management and one-way messages

[David Noveck <[e-mail]>]

*Ouch!!*

If i was addressing an unduly narrow problem, it was because it was framed
around the difficulties that might be added by adding one-way messages such
as those added by Version 2.  If you consider, as I guess makes sense, RPCs
that are not replied to as effectively one-way messages, that puts an
entirely different spin (of possibly high angular momentum) on the problem.

Let me try to start addressing this larger problem.  I hope it doesn't make
anyone (else) dizzy.

One reason that this is hard to address is that we have different views
regarding the roles of the credit management logic regarding replies.  In
my view, there is no point in a requester trying to give a credit to the
responder allowing the responder to send the response to the requester.  I
thought that the right thing was for the requester to ensure that the
responders response could be read was for the requester to post a receive
(or ensure that one was present) before sending the request.  I though that
this happened outside the ambit of credit management because there is no
real need for it to be within the scope of credit  management.  The
requester can be responsible for providing receives corresponding with
in-flight requests without involving the responder in the decision, which
involves complexity, especially when multipeRPC directions are involved.  I
never found anythng in the spec that told me I was wrong in that, but i
just may be missing it.

I think we discussed this a while ago but no change in the spec was made to
either make it clear I was wrong, or indicate that I was right.  My
recollection is that you treated my observation as correct/valid
implementation advice and we never really addressed our different
interpretations of the spec.  It seems to me that the spec was ambiguous on
this point.   Of course, I might not be seeing something that is there and
you might be simply assuming that something is implied because it seems
natural, even if it is not explcitly there.  If the spec still is not clear
on this point, it might need to be fixed, if not to mandate my approach but
at least to indicate how the requester can safeguard itself against the
possibility that a response sent by the responder will cause a disconnect.

With this approach, RPC's not replied to would still be a problem, but the
problem cannot result in a disconnect.  For each not-replied-to request,
there would be a receive that was not included in the credits, which would
be devoted in this model to receives for receiving requests (and things
intended to be one-way messages).   Of course, with 1K or 4K buffers that
would not be such a big deal and you could defer any connection reset until
a time when it was easy to do.

On Tue, Aug 1, 2017 at 11:49 AM, Chuck Lever <[e-mail]> wrote:

>
> > On Jul 31, 2017, at 2:47 PM, David Noveck <[e-mail]> wrote:
> >
> > One issue that has been mentioned as a possible impediment to advnacing
> RPC-over-RDMA to be a working group document, in its current form, concerns
> its use of one-way messages to convey transport properties.  These concerns
> seem to arise from the presentation of credit managemt as a "request-grant
> protocol".  If the credit logic in fact depended on that pairing, then
> there would be no roon for one-way meesages.
> >
> > In fact, it doesn't. If you look at the actual operation of credit
> management, one sees a different picture from what is stated in the
> introductory paragraph of Section 3.1.1 of RFC 8166:
> >
> > Flow control for RDMA Send operations directed to the Responder is
> implemented as a simple request/grant protocol in the RPC-over-RDMA header
> associated with each RPC message.
> >
> > what follows focuses on the role of the grant. giving extensive
> information but how it is computed and nd why it is vsli for the requester
> to use it.
> >
> > While the text says "Practically speaking, the critical value is the
> granted value", it is not clear what the exact role of the request is.  It
> aapears to be a hintwhich the receiver is under no obligation to take any
> notice of.  So it appears that the presentation of the creddit management
> approach as a request-grant protocol., is a reflection of the fact that,
> when it is used to transport RPC Requests and Replies, this pairing always
> exists.  When grants are sent with one-way messages, there is no problem
> that arises from the fact that there is no corresponding credit request.
> The sender simply informs the receiver about his own receive resources and
> thus his ability to accept further sends.
>
> The request-grant protocol sets an upper bound for the number of
> messages that can be in flight at once. You are addressing the
> narrow issue of how a receiver should interpret the value in the
> rdma_credits field to set the negotiated credit limit.
>
> The purpose of this limit is to prevent a sender from transmitting
> more messages than the receiver has available receive buffers.
>
> For example, a requester cannot RPC Call without first ensuring
> there is a receive buffer available to catch the RPC Reply. A
> responder cannot send an RPC Reply without first ensuring there
> are an appropriate number of receive buffers ready for the granted
> number of credits, minus the number of RPC transactions it is
> currently processing. A requester computes the number of remote
> receive buffers that are available, and thus how many more RPC
> Calls it can send before waiting, by observing the number of
> Replies it has received.
>
> In other words, the credit limit is _enforced_ via the two-way
> interchange. The underlying and more significant issue is therefore
> that, when unidirectional messages are introduced, neither side can
> properly compute the number of new messages that may be sent
> relative to the negotiated upper bound.
>
> Another header field could be introduced that contains the exact
> number of receive buffers available on the sender when that message
> was sent. Or such a field could replace rdma_credits. Either would
> be another step away from general compatibility with RPC-over-RDMA
> version 1.
>
>
> > There is one potential issue connected with use of one-way messages to
> be addressed.  If the recever has N credits and then N one-way messages are
> sent without any traffic in the opposite direction, then it is possble for
> a deadlock to result, since there would be no way for the sender to find
> out about the receive resources.  For most one-way messages, this is not a
> problem, since many one-way messages naturally give rise to messages in the
> opposite direction even if the relationship is not formalized witin an RPC
> paradigm.  For example:
> >       • The RDMA2_CONNPROP sent by the client to the server is paired
> with an RDMA2_CONNPROP in the opposite direction.
> >       • An RDMA2_REQPROP results in RDMA2_RESPROP send in response.
> > RDMA2_UPDPROP is an exception.  In the unlikely event that there are a a
> lage series of such messages sent in one irection while there are no RPC's
> being sent to the receiver it is possible for a ealock to arise.  This
> would result in a situation in which the receiver would have to send some
> message back in the other to provide a credit grant.   The most likely way
> to to that is for it to choose to send an RDMA2_UPDPROP with an empty
> property set in the reverse direction.
> >
> > One other problem with one-way messages as they stand in
> rpcrdma-version-two concerns section 7.2.2 in which the following item in
> the bullet list is poblematic:
> >       • When the rdma_proc field has the value RDMA2_OPTIONAL and no RPC
> message payload is present, a Requester MUST set the value of the
> rdma_optdir field to CALL, and a Responder MUST set the value of the
> rdma_optdir field to REPLY.  The Requester chooses a value for the rdma_xid
> field from the XID space that matches the message's direction.  Requesters
> and Responders set the rdma_credit field in a similar fashion: a value is
> set that is appropriate for the direction of the message.
> > This cannot be acted on, because, in the context of a one-way message,
> it is not clear which party is the requester and which is the responder.
> While the roles of client and server are fixed and clear, the roles of
> requester and responder vary from RPC to RPC.  If you are not in an RPC
> context, then any decision as to who is the requestor or responder is
> arbitrary.
> > I think the best way to address these issues is for rpcrdma-version-two
> would be to:
> >       • Provide an explanation of credit management not so tied to the
> RPC paradigm.
>
> That is appropriate to explore, but is much easier said than
> done.
>
> But first, we need to define what a unidirectional message is,
> and even that's harder than it looks. Here are some examples.
>
>  + An RPC Call that does not expect a Reply is unidirectional.
>
>  + A one-way control message is unidirectional.
>
>  + An RPC Call that is dropped or ignored by a responder is
>    unidirectional.
>
>  + An RPC retransmission without a connection break makes
>    either the original or the retransmitted message
>    unidirectional.
>
>  + An RPC Call becomes unidirectional when the matching RPC
>    Reply is lost for some reason.
>
>  + RDMA_DONE would be unidirectional. (RDMA_DONE is proposed
>    as a mechanism to manage Read chunks in RPC Replies in
>    draft-cel-nfsv4-rpcrdma-reliable-reply-00).
>
>  + An extension might introduce some new form of unidirectional
>    message if the problematic text in section 7.2.2 is modified
>    or removed.
>
> These are some of the ways a requester or responder can lose
> synchronization of outstanding credits.
>
>
> >       • Add a new value for the direction field for one-way messages
>
> In some of the above cases, the sender does not know that a
> unidirectional message is being sent, and thus cannot properly
> set the direction field.
>
>
> >       • Provide that one-way messages always contain a credit grant
> rather than a credit request
>
> It doesn't make sense to me to send a message from a requester to
> a responder with a "grant" credit value. What value would the
> requester put in the rdma_credits field in the case where the
> responder and requester have a different number of receive
> buffers available?
>
> It might be stronger to have a special value for either the
> direction or the rdma_credits field which means "the value in the
> rdma_credits field should be ignored".
>
> But see above: sometimes a sender cannot know in advance whether
> a message is unidirectional.
>
>
> >       • Explain how the potential deadlock with RDMA2_UPDPROP can be
> avoided.
>
> In addition, to handle cases where a message becomes unidirectional
> after it is sent, RPC-over-RDMA needs a reliable and non-destructive
> mechanism for resynchronizing the number of outstanding credits at
> the sender and receiver.
>
> Today that mechanism is to drop and re-establish a connection (which
> is 100% reliable but is not non-destructive).
>
> We are introducing mechanisms that can build up state associated
> with a connection (using unidirectional messages, possibly). That
> state is lost and must be re-established if the connection is
> dropped. Not to mention all the outstanding RPCs that have to be
> retransmitted.
>
>
> --
> Chuck Lever
>
>
>
>