Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

[Chuck Lever <[e-mail]>]

> On Jun 3, 2016, at 10:33 AM, David Noveck <[e-mail]> wrote:
> 
> A week ago I wrote:
> > The problem is that, in a multi-direction context, a given implementation only knows whether it is a client or 
> > a server.  It only knows if it is a requester or a responder based on the message it is processing at any given 
> > instant and if it can't figure that out, it is unsure whether it is a requester or responder and may be neither at
> > particular times.
> 
> It's now been at least two weeks since last call was supposed to end for the Version One RDMA documents
> and a week since Chuck posted any necessary updates.  I'm not sure why the documents are still listed as in 
> WGLC, but I wanted to make it clear that my comments above should not be holding these up.
> 
> Those comments refer to a problem that is inherent in Version One  and cannot be fixed within the constraints
> of a Version One no-XDR-change update.  Since the Working group has decided to do a Version One no-XDR-change
> update, the three existing RDMA documents should now go forward.  I don;t believe there is any disagreement on 
> that point.

I concur that rfc5666bis and rfc5666-implementation-experience
are ready to move forward.

However, I've been thinking about an earlier comment Dave made
in this thread:

> My only concern is with regard to a possible future extension in which multiple RPC messages are carried in a single SEND (i.e., the precise opposite of message continuation).  When the language for the new field is drafted, we should make sure that it doesn't assume all the messages in a SEND are all related to a single direction of operation. 


There seem to be two related issues, and one of them has
ramifications for rpcrdma-bidirection.


I. Ambiguity of the meaning of the credit field

In the rfc5666bis world, the credit field in all messages
flowing on a connection had an unambiguous meaning. If
the message was going from requester to responder, the
credit field was a credit request. In the other direction,
it was a credit grant.

rpcrdma-bidirection introduced the idea of two separate
credit flows, which depend on whether the message was
part of forward RPC operation or backward RPC operation
on that connection.

This is the source of the problem.

The meaning of the credit field depends on a field in
the upper layer, so it's a layering violation, to say
the least.

Also, the original concept of credit was to manage RDMA
receive buffers, but rpcrdma-bidirection interpretation
of the credit value is one-credit-per-RPC.

Now if we want to add support for partial RPC message
per credit (message continuation) or multiple RPC
messages per credit (as proposed above), or NO RPC
message per credit (for RDMA2_OPTIONAL control messages)
the credit field is unusable.

Would we add an independent pool of credits for each of
these transmission mechanisms? Probably not.

A logical course of action, then, would be to alter the
rpcrdma-bidirection I-D so that forward and backward
direction operation use the same pool of credits.

The question is whether it is feasible for the two
directions to share the credit pools without deadlocking.
Some prototyping and/or careful thought would be needed
to answer it.

(Spencer, I may be changing my mind about the readiness
of rpcrdma-bidirection).


II. Ambiguity of the meaning of the XID field

Today we have a one-to-one match between the XID value
in the RPC-over-RDMA header and the XID associated with
the RPC payload in that message.

With an extension for message continuation, there is
still a one-to-one match: the RPC-over-RDMA XID can
match the XID of the partial RPC payload.

But for multiple RPC payloads per Send, or no RPC
payloads per Send (control messages), we no longer have
a match. How should the RPC-over-RDMA header's XID field
be set in those cases?

RFC 5666 and rfc5666bis have similar language regarding
how the XID field in the RPC-over-RDMA header is to be
set. Here's rfc5666bis, Section 5.2.1:

>    The
>    receiver MAY perform its processing based solely on the XID in the
>    RPC-over-RDMA header, and thereby ignore the XID in the RPC message,
>    if it so chooses.


In other words, implementations are already allowed to
rely on the value of the RPC-over-RDMA header's XID
field, and ignore the XID field in the payload.

This issue has to be addressed for rpcrdma-version-two
to support extensions that enable multiple or no RPCs
per message.

I propose that here, an independent XID space for
RDMA2_OPTIONAL messages makes sense. This would allow
receivers to match RDMA2_OPTIONAL call and reply messages,
but would not tie the header's XID value to the upper
layer payload in any way.


--
Chuck Lever