Re: [nfsv4] Clean up for rpcrdma-version-two Read chunks

[Chuck Lever <[e-mail]>]

> On May 5, 2020, at 3:04 PM, David Noveck <[e-mail]> wrote:
> 
>       > This would require sorting by the receiver.
> 
> > It would indeed. 
> 
> It would require that if they could arrive unsorted.

Even if the Requester is "REQUIRED to send these in order" there can always be Requester implementation bugs
which mis-order the segments. That's why the Responder implementation has to be defensive anyway.


> > The alternative is kind of disastrous.
> 
> I think a reasonable alternative is to ask the sender to send them in order.  I understand you don't agree with that but can't see how it is "disastrous".

I don't have a problem with suggesting or requiring the sender to organize the Read list in a particular order. I just don't
think it's a full solution. The possibility of Requester bugs or malicious intent necessitates that Responders be protective
anyway. A Read list ordering requirement doesn't make the need for Responder defensiveness go away. At the very
least the Responder still has to recognize mis-sorted Read lists and bat them down.

It's the difference between "the protocol requires receivers to recognize and reject badness" versus "the protocol makes
it impossible to express badness".


> > Gaps would probably contain data from a previous use of the Read sink
> > buffer.
> 
> I don't see how.  If you have multiple read chunks, it is probably due to multiple operations, each of which contain a DDP-eligible data area, since there is no op with more than one DDp-eligible argument. In this cse the non-DDP_eligible data appears inline.
> 
> I guess you are focused on the case of a mix of PZRCS and PNZRCS, which is kind of a special case.

For the foreseeable future, it is a common special case. This case arises frequently for krb5, for example.


> > The exact contents of overlap regions would depend on the order that the
> > RNIC completed the RDMA Read operations.
> 
> I think we agree about overlapping chunks being bad but only disagree about
> how best to prevent them.
> 
> > Thus IMO a good quality Responder has to perform some sanity checking on
> > the position values and lengths in incoming Read segments. 
> 
> I Agree.
> 
> > I'm not sure
> > how it could avoid some sorting-like behavior to perform this check.
> 
> That's true only if it is valld to send them out-of-order which I don't think
> should be the case.
> 
> > It might be better to place responsibility on the sender to sort these.
> 
> I should have said "send these in sorted order".  Realistically, requesters
> will send these in sorted order and no actual sort would be required.

True enough for Requesters we know of with sane implementations. However,
as we well know, that doesn't mean this will always be the case. We need to
somehow document our assumptions and requirements here.


> > IMO the goal of RPC/RDMA is to reduce host processing on the Requester.
> 
> That's an important goal but don't agree that it is *the* goal.
> 
> > Sorting on the Responder follows that paradigm more closely than the
> > converse.
> 
> I think requring them being in sorted order avoids a lot of complexity.
> I haven't seen any case where it makes sense to send them other than in sorted order.
> 
> With regard to this goal/paradigm, alll I can say is that T. S. Kuhn has lot to answer for :-)
> 
> > For a given position value, RPC/RDMA already requires that Read segments
> > have to be in the order they appear in the reconstructed RPC message. But
> > there's currently no requirement that the Read list's position values have
> > to appear in monotonically increasing order. In fact I think RPC/RDMA
> > permits a Requester to interleave Read segments at different positions, as
> > long as they are in the Read list in the order they should be used to
> > reconstruct the RPC Call.
> 
> True but I don't see why any requester would find it necessary/helpful
>  to interleave read segments in this way, except perhaps in the special case
> a PZRC/PNZRC mix.

Again, that's the particular case that is likely to be frequent and impact RPC/RDMA v1
as well as v2.


> > I'm working on some changes to the Linux NFS/RDMA implementation that might
> > perform Responder-side Read list sorting in order to deal properly with
> > Read lists that contain segments with more than one Position value. For
> > example, a Read list that contains Read segments with position zero and
> > Read segments with a non-zero position could be re-sorted so that all of
> > the segments are in byte order at position zero. 
> 
> You are talking about doing some thing that goes beyond sorting, even though
> sorting is a part of it and it appears to me that you are sorting by someting
> apporoximating chunk position.
> You are reorganizing all the read chunks including 
> changing non-PZ chunks to position-zero chunks.   In addition the sorting is not by
> the position field of the chunk but by expected position.  I can see why you moght
> do if you receive somethning like that that but I can't see why the reqester is using a 
> PZRC to send this, given that, in version two, you can avoid the PZRC and use 
> message continuation to send the request.

A v2 Requester is now able to use Message chaining but it is currently not /required/ to use it.
A v1 Requester is still allowed to send a PZRC and a normal Read chunk.

RPC/RDMA v2 would have to explicitly forbid the use of PZRCs with normal Read chunks.
Is there a sane way to do that?

We could require that RDMA2_NOMSG includes only position zero Read segments.

For situations that call for normal Read chunks with an inline body that is larger than the
transport's inline threshold, the transport is required then to use RDMA2_MSG and Message
chaining.

If the transport's Message chaining limits can't accommodate that arrangement, then the
Requester has to use a PZRC (ie, no normal Read chunks).


--
Chuck Lever