[nfsv4] Expert review of NFSv4 RDMA private data draft (draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05)

["Black, David" <[e-mail]>]

With credit to Magnus (AD responsible for nfsv4 WG) for pinging me behind the scenes, I was able to obtain an expert review (see below) for the RDMA private data draft.

Please keep Dave Minturn  and Sean Hefty on cc: and take it easy on email volume, as they're doing the WG a favor.

Thanks, --David

From: Hefty, Sean <[e-mail]<mailto:[e-mail]>>
Sent: Monday, December 9, 2019 6:08 PM
To: Minturn, Dave B <[e-mail]<mailto:[e-mail]>>
Subject: RE: Question regarding RDMA CM Private Data

Just reading your comments, CM private data is the correct name.  The amount of data present is defined by the spec, with a portion of the data consumed by the RDMA CM protocol itself.  I *think* there's something like 48 bytes to play with on connection requests.

I agree that the format specifier is useless.  There are many existing users which could be setting the private data to anything.  You really need to rely on the port numbers being correct, with checks on the other fields detecting mismatched protocols.  The private data is and has been exposed directly to user space applications for years, so anything there is fair play.

In order to standardize the private data format, you would need to change the version of the RDMA CM header carried in the private data of the underlying transport.

The field sizes for the send/recv sizes look too small to me, unless those fields are scaled somehow.


From: Minturn, Dave B <[e-mail]<mailto:[e-mail]>>
Sent: Monday, December 9, 2019 5:03 PM
To: Hefty, Sean <[e-mail]<mailto:[e-mail]>>
Subject: Question regarding RDMA CM Private Data

Hi Sean,

David Black contacted me and asked if I could review an IETF RFC for NFS/RDMA's use of RDMA CM's private data.  The RFC reference is:  https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.pdf
It would be great if you could take a look.  I called out some items I saw but you are the expert in this area.
..Dave

Here are some items that I had questions/concerns with: (Section 4)

When an RPC-over-RDMA version 1 transport connection is
established, the client (which actively establishes connections) and
the server (which passively accepts connections) populate the CM
Private Data field exchanged as part of CM connection establishment.

>>  Is "CM Private Data" the proper name?

For RPC-over-RDMA version 1, the CM Private Data field is formatted
as described in the following subsection. RPC clients and servers
use the same format. If the capacity of the Private Data field is
too small to contain this message format, the underlying RDMA
transport is not managed by a Connection Manager, or the underlying
RDMA transport uses Private Data for its own purposes, the CM Private
Data field cannot be used on behalf of RPC-over-RDMA version 1.

>> This statement seemed unnecessary because it's transparent to the RFC ULP if the RDMA transport
>> is using Private Data.  Seems more appropriate to say something about the <capacity> of the Private Data
>> field length being RDMA transport dependent.
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Format Identifier                                                                                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version           | Flags         | Send Size               | Receive Size      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Format Identifier: This field contains a fixed 32-bit value that
identifies the content of the Private Data field as an RPC-over-
RDMA version 1 CM Private Data message. In RPC-over-RDMA version
1 Private Data, the value of this field is always 0xf6ab0e18, in
network byte order. The use of this field is further expanded
upon in Section 4.1.
4.1.2. Amongst Implementations of Other Upper-Layer Protocols
The Format Identifier field in the message format defined in this
document is provided to enable implementations to distinguish RPC-
over-RDMA version 1 Private Data from application-specific private
data inserted by applications other than RPC-over RDMA version 1.
Examples of other applications that make use of CM Private Data
include iWARP, via the MPA enhancement described in [RFC6581], and
iSCSI extensions for RDMA (iSER), as defined in [RFC7145].
During connection establishment, an implementation of the extension
described in this document checks the Format Identifier field before
decoding subsequent fields. If the RPC-over-RDMA version 1 CM
Private Data Format Identifier is not present as the first 4 octets,
an RPC-over-RDMA version 1 receiver MUST ignore the CM Private Data,
behaving as if no RPC-over-RDMA version 1 Private Data has been
provided (see above).
>>  It seems wrong to assume that no other RDMA ULP will put that "magic #" in.   Maybe
>> if combined with a well know port, but even then it's for sure.  Now that I said that,
>> I found the following text proposing using IANA to standardize the RDMA_CM private data
>>
. IANA Considerations
In accordance with [RFC8126], the author requests that IANA create a
new registry in the "Remote Direct Data Placement" Protocol Category
Group. The new registry is to be called the "RDMA-CM Private Data
Lever Expires May 11, 2020 [Page 8]
Internet-Draft RPC-Over-RDMA CM Private Data November 2019
Identifier Registry". This is a registry of 32-bit numbers that
identify the upper-layer protocol associated with data that appears
in the application-specific RDMA-CM Private Data area. The fields in
this registry include: Format Identifier, Description, and Reference.
The initial contents of this registry are a single entry:
+------------------+------------------------------------+-----------+
| Format | Format Description | Reference |
| Identifier | | |
+------------------+------------------------------------+-----------+
| 0xf6ab0e18 | RPC-over-RDMA version 1 CM Private | [RFC-TBD] |
| | Data | |
+------------------+------------------------------------+-----------+
Table 1: RDMA-CM Private Data Identifier Registry
IANA is to assign subsequent new entries in this registry using the
Expert Review policy as defined in Section 4.5 of [RFC8126].