Re: [nfsv4] Expert review of NFSv4 RDMA private data draft (draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05)

> On Dec 12, 2019, at 5:06 PM, Black, David <David.Black@dell.com> wrote:
> 
> With credit to Magnus (AD responsible for nfsv4 WG) for pinging me behind the scenes, I was able to obtain an expert review (see below) for the RDMA private data draft.
>  
> Please keep Dave Minturn  and Sean Hefty on cc: and take it easy on email volume, as they’re doing the WG a favor.

My thanks to the reviewers. A few comments inline.

> Thanks, --David
>  
>  <>From: Hefty, Sean <sean.hefty@intel.com <mailto:sean.hefty@intel.com>> 
> Sent: Monday, December 9, 2019 6:08 PM
> To: Minturn, Dave B <dave.b.minturn@intel.com <mailto:dave.b.minturn@intel.com>>
> Subject: RE: Question regarding RDMA CM Private Data
>  
> Just reading your comments, CM private data is the correct name.  The amount of data present is defined by the spec, with a portion of the data consumed by the RDMA CM protocol itself.  I *think* there’s something like 48 bytes to play with on connection requests.

If that 48-byte limit is documented somewhere, this draft can easily reference it. I was never able to find such a document.

> I agree that the format specifier is useless.  There are many existing users which could be setting the private data to anything.  You really need to rely on the port numbers being correct, with checks on the other fields detecting mismatched protocols.

Other reviewers have been ambivalent about the format identifier, but we don't have a better solution at the moment. The problem is the port numbers are all arbitrary. AFAIK they are not used in a way that identify the connection as RPC.

The 2049 destination port number means "NFS" but that's a layer above RPC/RDMA.

RPC/RDMA also does not define a ToS. That would be a sure way to distinguish the traffic the connection is to carry.

> The private data is and has been exposed directly to user space applications for years, so anything there is fair play.
>  
> In order to standardize the private data format, you would need to change the version of the RDMA CM header carried in the private data of the underlying transport.

Which is outside the purview of the IETF.

> The field sizes for the send/recv sizes look too small to me, unless those fields are scaled somehow.

The values in the inline threshold fields are scaled; see Section 5.2. The actual size range of these buffers is 1KB to 256KB.

> From: Minturn, Dave B <dave.b.minturn@intel.com <mailto:dave.b.minturn@intel.com>> 
> Sent: Monday, December 9, 2019 5:03 PM
> To: Hefty, Sean <sean.hefty@intel.com <mailto:sean.hefty@intel.com>>
> Subject: Question regarding RDMA CM Private Data
>  
> Hi Sean,
>  
> David Black contacted me and asked if I could review an IETF RFC for NFS/RDMA’s use of RDMA CM’s private data.  The RFC reference is:  https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.pdf <https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.pdf>
> It would be great if you could take a look.  I called out some items I saw but you are the expert in this area. 
> ..Dave
>  
> Here are some items that I had questions/concerns with: (Section 4)
>                 
> When an RPC-over-RDMA version 1 transport connection is
> established, the client (which actively establishes connections) and
> the server (which passively accepts connections) populate the CM
> Private Data field exchanged as part of CM connection establishment.
>  
> >>  Is “CM Private Data” the proper name? 
>  
> For RPC-over-RDMA version 1, the CM Private Data field is formatted
> as described in the following subsection. RPC clients and servers
> use the same format. If the capacity of the Private Data field is
> too small to contain this message format, the underlying RDMA
> transport is not managed by a Connection Manager, or the underlying
> RDMA transport uses Private Data for its own purposes, the CM Private
> Data field cannot be used on behalf of RPC-over-RDMA version 1.
>  
> >> This statement seemed unnecessary because it’s transparent to the RFC ULP if the RDMA transport
> >> is using Private Data.  Seems more appropriate to say something about the <capacity> of the Private Data
> >> field length being RDMA transport dependent. 
> 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> | Format Identifier                                                                                |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> | Version           | Flags         | Send Size               | Receive Size      |
> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> Format Identifier: This field contains a fixed 32-bit value that
> identifies the content of the Private Data field as an RPC-over-
> RDMA version 1 CM Private Data message. In RPC-over-RDMA version
> 1 Private Data, the value of this field is always 0xf6ab0e18, in
> network byte order. The use of this field is further expanded
> upon in Section 4.1.
> 4.1.2. Amongst Implementations of Other Upper-Layer Protocols
> The Format Identifier field in the message format defined in this
> document is provided to enable implementations to distinguish RPC-
> over-RDMA version 1 Private Data from application-specific private
> data inserted by applications other than RPC-over RDMA version 1.
> Examples of other applications that make use of CM Private Data
> include iWARP, via the MPA enhancement described in [RFC6581], and
> iSCSI extensions for RDMA (iSER), as defined in [RFC7145].
> During connection establishment, an implementation of the extension
> described in this document checks the Format Identifier field before
> decoding subsequent fields. If the RPC-over-RDMA version 1 CM
> Private Data Format Identifier is not present as the first 4 octets,
> an RPC-over-RDMA version 1 receiver MUST ignore the CM Private Data,
> behaving as if no RPC-over-RDMA version 1 Private Data has been
> provided (see above).
> >>  It seems wrong to assume that no other RDMA ULP will put that “magic #” in.   Maybe
> >> if combined with a well know port, but even then it’s for sure.  Now that I said that,
> >> I found the following text proposing using IANA to standardize the RDMA_CM private data
> >> 
> . IANA Considerations
> In accordance with [RFC8126], the author requests that IANA create a
> new registry in the "Remote Direct Data Placement" Protocol Category
> Group. The new registry is to be called the "RDMA-CM Private Data
> Lever Expires May 11, 2020 [Page 8]
> Internet-Draft RPC-Over-RDMA CM Private Data November 2019
> Identifier Registry".. This is a registry of 32-bit numbers that
> identify the upper-layer protocol associated with data that appears
> in the application-specific RDMA-CM Private Data area. The fields in
> this registry include: Format Identifier, Description, and Reference.
> The initial contents of this registry are a single entry:
> +------------------+------------------------------------+-----------+
> | Format | Format Description | Reference |
> | Identifier | | |
> +------------------+------------------------------------+-----------+
> | 0xf6ab0e18 | RPC-over-RDMA version 1 CM Private | [RFC-TBD] |
> | | Data | |
> +------------------+------------------------------------+-----------+
> Table 1: RDMA-CM Private Data Identifier Registry
> IANA is to assign subsequent new entries in this registry using the
> Expert Review policy as defined in Section 4.5 of [RFC8126].
>  
>  
>  
>  
>  
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org <mailto:nfsv4@ietf.org>
> https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>
--
Chuck Lever