[nfsv4] Credit management and one-way messages

[David Noveck <[e-mail]>]

One issue that has been mentioned as a possible impediment to advnacing
RPC-over-RDMA to be a working group document, in its current form, concerns
its use of one-way messages to convey transport properties.  These concerns
seem to arise from the presentation of credit managemt as a "request-grant
protocol".  If the credit logic in fact depended on that pairing, then
there would be no roon for one-way meesages.

In fact, it doesn't. If you look at the actual operation of credit
management, one sees a different picture from what is stated in the
introductory paragraph of Section 3.1.1 of RFC 8166:

Flow control for RDMA Send operations directed to the Responder is implemented
as a simple request/grant protocol in the RPC-over-RDMA header associated
with each RPC message.

what follows focuses on the role of the grant. giving extensive information
but how it is computed and nd why it is vsli for the requester to use it.

While the text says "Practically speaking, the critical value is the
granted value", it is not clear what the exact role of the request is.  It
aapears to be a hintwhich the receiver is under no obligation to take any
notice of.  So it appears that the presentation of the creddit management
approach as a request-grant protocol., is a reflection of the fact that,
when it is used to transport RPC Requests and Replies, this pairing always
exists.  When grants are sent with one-way messages, there is no problem
that arises from the fact that there is no corresponding credit request.
The sender simply informs the receiver about his own receive resources and
thus his ability to accept further sends.

There is one potential issue connected with use of one-way messages to be
addressed.  If the recever has N credits and then N one-way messages are
sent without any traffic in the opposite direction, then it is possble for
a deadlock to result, since there would be no way for the sender to find
out about the receive resources.  For most one-way messages, this is not a
problem, since many one-way messages naturally give rise to messages in the
opposite direction even if the relationship is not formalized witin an RPC
paradigm.  For example:

   - The RDMA2_CONNPROP sent by the client to the server is paired with an
   RDMA2_CONNPROP in the opposite direction.
   -

   An RDMA2_REQPROP results in RDMA2_RESPROP send in response.


RDMA2_UPDPROP is an exception. In the unlikely event that there are a a
lage series of such messages sent in one irection while there are no RPC's
being sent to the receiver it is possible for a ealock to arise. This would
result in a situation in which the receiver would have to send some message
back in the other to provide a credit grant. The most likely way to to that
is for it to choose to send an RDMA2_UPDPROP with an empty property set in
the reverse direction.

One other problem with one-way messages as they stand in rpcrdma-version-two
concerns section 7.2.2 in which the following item in the bullet list is
poblematic:


   - When the rdma_proc field has the value RDMA2_OPTIONAL and no RPC
message payload is present, a Requester MUST set the value of the
rdma_optdir field to CALL, and a Responder MUST set the value of the
rdma_optdir field to REPLY.  The Requester chooses a value for the
rdma_xid field from the XID space that matches the message's
direction.  Requesters and Responders set the rdma_credit field in a
similar fashion: a value is set that is appropriate for the direction
of the message.

This cannot be acted on, because, in the context of a one-way message,
it is not clear which party is the requester and which is the
responder.  While the roles of client and server are fixed and clear,
the roles of requester and responder vary from RPC to RPC.  If you are
not in an RPC context, then any decision as to who is the requestor or
responder is arbitrary.

I think the best way to address these issues is for rpcrdma-version-two
would be to:

   - Provide an explanation of credit management not so tied to the RPC
   paradigm.
   - Add a new value for the direction field for one-way messages
   - Provide that one-way messages always contain a credit grant rather
   than a creit request
   - Explan how the potential deadlock with RDMA2_UPDPROP can be avoided.