Re: [nfsv4] Clean up for rpcrdma-version-two Read chunks

[Chuck Lever <[e-mail]>]

> On May 4, 2020, at 8:59 PM, David Noveck <[e-mail]> wrote:
> 
>> On Mon, May 4, 2020, 12:13 PM Chuck Lever <[e-mail]> wrote:
>> 
>>> RPC/RDMA v1 allows a position zero Read chunk to appear in an RDMA_MSG type Call.
>>> Where does a Responder put the inline portion of such a message?
>> 
>> I propose that in RPC/RDMA version 2, a Responder MUST return RDMA2_ERR_BAD_XDR if
>> a Requester sends a Read list containing a position zero Read chunk as part of
>> header type other than RDMA2_NOMSG.
> 
> Agree.
> 
> 
>>> RPC/RDMA v1 does not explicitly require an RDMA_NOMSG type Call to have a position
>>> zero Read chunk. Does such a message have gaps? Are they zero-filled?
>> 
>> I propose that in RPC/RDMA version 2, a Responder MUST return RDMA2_ERR_BAD_XDR if
>> a Requester sends an RDMA2_NOMSG header type whose Read list does not include a
>> position zero Read chunk.
> 
> As stated, this would forbid NOMSG bring used to send a long reply.

Nit: rpcrdma-version-two no longer uses the term Long message, see
Section 4.4.3.


> I think the text to address this needs to be careful not to foreclose that. Your text above uses the word "requester" assuming this is sufficient but the only way a peer receiving message could determine whether it was sent by a requester or responder is my looking at the message, which, in this case, does not exist.

The RPC/RDMA version two header has the RDMA2_F_RESPONSE flag (Section
6.2.2.1) which was introduced to enable a receiver to distinguish the
roles of the sender and receiver peers without sniffing the RPC layer
payload.

A while back I had envisioned using Read chunks in Responder-to-
Requester messages, and even wrote an I-D about it. But now that we
have both Reply chunks and Message chaining, it seems unnecessary to
hold the door open for using a Read chunk for a Reply, especially
given how arcane Read chunks are. Did you have a particular use case in
mind?

Also, I think the proposed text above would prevent the use of the
RDMA2_NOMSG type for asynchronous credit grants (Section 4.2.1.2) so I
ought to restate the requirement as:

>> A Responder MUST return RDMA2_ERR_BAD_XDR if a Requester sends an
>> RDMA2_NOMSG header type with a non-empty Read list that does not
>> include a position zero Read chunk.



>>> RPC/RDMA v1 does not prevent or prohibit overlapping Read chunks. Is the correct
>>> response ERR_CHUNK?
>> 
>> A protocol change would be needed to totally prevent the expression of overlapping
>> Read chunks. Maybe it's a little too late to address that in RPC/RDMA version 2.
> 
> I think you mean version 1.

No, I meant version 2. My sense of the virtual room two weeks ago was
that no-one had the stomach for major surgery on the RPC/RDMA version 2
data structures at this point. That's why I've limited the above proposals
to simple requirements that the Responder recognize badly formed Read
lists and respond to them as errors.

If we were to "go there," my thought about how to address the gap/overlap
issue would be to eliminate Read chunks and structure the Read list the
same way that the Write list is structured; ie, as a list of arrays of
RDMA segments, but each array would have a position field.

It might be interesting to use position fields in the Write list as well,
filled in by the Responder, to help disambiguate the position of result
data items in a Reply message.

But at this point we have escaped the orbit of RPC/RDMA version one entirely.


> Nobody seems up to do rfc8166bis.
> 
>> I propose that in RPC/RDMA version 2, a Responder MUST return RDMA2_ERR_BAD_XDR if
>> a Requester sends a Read list with chunks whose offsets and lengths result in the
>> same message byte position appearing in more than one Read chunk.
> 
> This would require sorting by the receiver.

It would indeed. The alternative is kind of disastrous.

Gaps would probably contain data from a previous use of the Read sink
buffer.

The exact contents of overlap regions would depend on the order that the
RNIC completed the RDMA Read operations.

Thus IMO a good quality Responder has to perform some sanity checking on
the position values and lengths in incoming Read segments. I'm not sure
how it could avoid some sorting-like behavior to perform this check.


> It might be better to place responsibility on the sender to sort these.

IMO the goal of RPC/RDMA is to reduce host processing on the Requester.
Sorting on the Responder follows that paradigm more closely than the
converse.

For a given position value, RPC/RDMA already requires that Read segments
have to be in the order they appear in the reconstructed RPC message. But
there's currently no requirement that the Read list's position values have
to appear in monotonically increasing order. In fact I think RPC/RDMA
permits a Requester to interleave Read segments at different positions, as
long as they are in the Read list in the order they should be used to
reconstruct the RPC Call.

I'm working on some changes to the Linux NFS/RDMA implementation that might
perform Responder-side Read list sorting in order to deal properly with
Read lists that contain segments with more than one Position value. For
example, a Read list that contains Read segments with position zero and
Read segments with a non-zero position could be re-sorted so that all of
the segments are in byte order at position zero. This enables the Responder
to set up the Read sink buffer pages so at Read completion the message is
already in proper segment and byte order.


--
Chuck Lever