Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

[Spencer Shepler <[e-mail]>]

Thanks, Chuck.  Will do.

-----Original Message-----
From: Chuck Lever [mailto:[e-mail]] 
Sent: Thursday, June 9, 2016 11:15 AM
To: Spencer Shepler <[e-mail]>
Cc: David Noveck <[e-mail]>; NFSv4 <[e-mail]>
Subject: Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review

Hi Spencer-

Let's proceed with revision 05. I'm clearing my recent comment about the readiness of rpcrdma-bidirection.



> On Jun 3, 2016, at 12:20 PM, Spencer Shepler <[e-mail]> wrote:
> 
> 
> Thanks for the discussion.  I will wait on you, Chuck, to "clear" the comment you made below on readiness.
> 
> Just to be clear, the WGLC is complete and from my point of view we have consensus but want to ensure the author is comfortable with moving forward.
> 
> -----Original Message-----
> From: Chuck Lever [mailto:[e-mail]]
> Sent: Friday, June 3, 2016 8:05 AM
> To: David Noveck <[e-mail]>; Spencer Shepler 
> <[e-mail]>
> Cc: NFSv4 <[e-mail]>
> Subject: Re: [nfsv4] draft-ietf-nfsv4-rpcrdma-bidirection-03 review
> 
> 
>> On Jun 3, 2016, at 10:33 AM, David Noveck <[e-mail]> wrote:
>> 
>> A week ago I wrote:
>>> The problem is that, in a multi-direction context, a given 
>>> implementation only knows whether it is a client or a server.  It 
>>> only knows if it is a requester or a responder based on the message 
>>> it is processing at any given instant and if it can't figure that out, it is unsure whether it is a requester or responder and may be neither at particular times.
>> 
>> It's now been at least two weeks since last call was supposed to end 
>> for the Version One RDMA documents and a week since Chuck posted any 
>> necessary updates.  I'm not sure why the documents are still listed as in WGLC, but I wanted to make it clear that my comments above should not be holding these up.
>> 
>> Those comments refer to a problem that is inherent in Version One  
>> and cannot be fixed within the constraints of a Version One 
>> no-XDR-change update.  Since the Working group has decided to do a 
>> Version One no-XDR-change update, the three existing RDMA documents should now go forward.  I don;t believe there is any disagreement on that point.
> 
> I concur that rfc5666bis and rfc5666-implementation-experience are ready to move forward.
> 
> However, I've been thinking about an earlier comment Dave made in this thread:
> 
>> My only concern is with regard to a possible future extension in which multiple RPC messages are carried in a single SEND (i.e., the precise opposite of message continuation).  When the language for the new field is drafted, we should make sure that it doesn't assume all the messages in a SEND are all related to a single direction of operation. 
> 
> 
> There seem to be two related issues, and one of them has ramifications for rpcrdma-bidirection.
> 
> 
> I. Ambiguity of the meaning of the credit field
> 
> In the rfc5666bis world, the credit field in all messages flowing on a connection had an unambiguous meaning. If the message was going from requester to responder, the credit field was a credit request. In the other direction, it was a credit grant.
> 
> rpcrdma-bidirection introduced the idea of two separate credit flows, which depend on whether the message was part of forward RPC operation or backward RPC operation on that connection.
> 
> This is the source of the problem.
> 
> The meaning of the credit field depends on a field in the upper layer, so it's a layering violation, to say the least.
> 
> Also, the original concept of credit was to manage RDMA receive buffers, but rpcrdma-bidirection interpretation of the credit value is one-credit-per-RPC.
> 
> Now if we want to add support for partial RPC message per credit (message continuation) or multiple RPC messages per credit (as proposed above), or NO RPC message per credit (for RDMA2_OPTIONAL control messages) the credit field is unusable.
> 
> Would we add an independent pool of credits for each of these transmission mechanisms? Probably not.
> 
> A logical course of action, then, would be to alter the rpcrdma-bidirection I-D so that forward and backward direction operation use the same pool of credits.
> 
> The question is whether it is feasible for the two directions to share the credit pools without deadlocking.
> Some prototyping and/or careful thought would be needed to answer it.
> 
> (Spencer, I may be changing my mind about the readiness of rpcrdma-bidirection).
> 
> 
> II. Ambiguity of the meaning of the XID field
> 
> Today we have a one-to-one match between the XID value in the RPC-over-RDMA header and the XID associated with the RPC payload in that message.
> 
> With an extension for message continuation, there is still a one-to-one match: the RPC-over-RDMA XID can match the XID of the partial RPC payload.
> 
> But for multiple RPC payloads per Send, or no RPC payloads per Send (control messages), we no longer have a match. How should the RPC-over-RDMA header's XID field be set in those cases?
> 
> RFC 5666 and rfc5666bis have similar language regarding how the XID field in the RPC-over-RDMA header is to be set. Here's rfc5666bis, Section 5.2.1:
> 
>>   The
>>   receiver MAY perform its processing based solely on the XID in the
>>   RPC-over-RDMA header, and thereby ignore the XID in the RPC message,
>>   if it so chooses.
> 
> 
> In other words, implementations are already allowed to rely on the value of the RPC-over-RDMA header's XID field, and ignore the XID field in the payload.
> 
> This issue has to be addressed for rpcrdma-version-two to support extensions that enable multiple or no RPCs per message.
> 
> I propose that here, an independent XID space for RDMA2_OPTIONAL messages makes sense. This would allow receivers to match RDMA2_OPTIONAL call and reply messages, but would not tie the header's XID value to the upper layer payload in any way.
> 
> 
> --
> Chuck Lever
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [e-mail]
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.i
> etf.org%2fmailman%2flistinfo%2fnfsv4&data=01%7c01%7csshepler%40microso
> ft.com%7cc7fb211893674020da6d08d39091ec90%7c72f988bf86f141af91ab2d7cd0
> 11db47%7c1&sdata=KiMVRSXc2Sj8%2blfR0vjWLJFQ4KAYKx95%2bMkReAuNQjA%3d

--
Chuck Lever