Re: [nfsv4] Benjamin Kaduk's No Objection on draft-ietf-nfsv4-rpcrdma-cm-pvt-data-07: (with COMMENT)

Chuck Lever <> Thu, 20 February 2020 15:15 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 9CDA412003E; Thu, 20 Feb 2020 07:15:36 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -4.3
X-Spam-Status: No, score=-4.3 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id a58kWn79XhYa; Thu, 20 Feb 2020 07:15:33 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 4F74D120013; Thu, 20 Feb 2020 07:15:33 -0800 (PST)
Received: from pps.filterd ( []) by ( with SMTP id 01KFE8LR003212; Thu, 20 Feb 2020 15:15:31 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; h=content-type : mime-version : subject : from : in-reply-to : date : cc : content-transfer-encoding : message-id : references : to; s=corp-2020-01-29; bh=hXmf15ZgsZPo2uAPc2I1/yLKw+ZfMCnFNt9MSd8TlXU=; b=RToBKNktGgAl5v043v1IJ/HH0J6xnxPVl72fuurCBfop+aP7/gU5X3OBGviJW0K0KWd9 CPy43egMWTlLzSpnt3oSG/84xFvM1em4xMfGPy2nBLmgBhxo7RDKQ05Y54pbgavXfW6b RISW5p0vMIXF4yDJzMgpsp23f5pixW+4zyL4iaWYGR4CR88bKDBZrcv8aMLjBDtxpEKb pF2oE4DEQGebVEF+Q3Y9adYLMzYGa98Wq6g5R+l9OvS87HTV7SeGh+0crGwOZxA08f7V FLoO6UM/7UUKtCH1EZorhc3DUsHR+Ogp5ritEnji6OFItwFu1H8MC3HhK4754nHUhN4b mA==
Received: from ( []) by with ESMTP id 2y8ud1abbe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 20 Feb 2020 15:15:29 +0000
Received: from pps.filterd ( []) by ( with SMTP id 01KFBpt9130097; Thu, 20 Feb 2020 15:13:29 GMT
Received: from ( []) by with ESMTP id 2y8ud67erm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 20 Feb 2020 15:13:28 +0000
Received: from ( []) by (8.14.4/8.14.4) with ESMTP id 01KFDR5J019357; Thu, 20 Feb 2020 15:13:27 GMT
Received: from (/ by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 20 Feb 2020 07:13:27 -0800
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Chuck Lever <>
In-Reply-To: <>
Date: Thu, 20 Feb 2020 10:13:26 -0500
Cc: The IESG <>,, Tom Haynes <>, Spencer Shepler <>, Brian Pawlowski <>,,
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <>
To: Benjamin Kaduk <>
X-Mailer: Apple Mail (2.3445.104.11)
X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9536 signatures=668685
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 mlxscore=0 adultscore=0 spamscore=0 suspectscore=0 malwarescore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002200113
X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9536 signatures=668685
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 malwarescore=0 suspectscore=0 bulkscore=0 spamscore=0 priorityscore=1501 phishscore=0 impostorscore=0 mlxlogscore=999 clxscore=1015 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002200113
Archived-At: <>
Subject: Re: [nfsv4] Benjamin Kaduk's No Objection on draft-ietf-nfsv4-rpcrdma-cm-pvt-data-07: (with COMMENT)
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NFSv4 Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 20 Feb 2020 15:15:37 -0000

Hi Ben, thanks for your comments! Responses and questions below.

> On Feb 19, 2020, at 8:10 PM, Benjamin Kaduk via Datatracker <> wrote:
> Benjamin Kaduk has entered the following ballot position for
> draft-ietf-nfsv4-rpcrdma-cm-pvt-data-07: No Objection
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
> Please refer to
> for more information about IESG DISCUSS and COMMENT positions.
> The document, along with other ballot positions, can be found here:
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> I agree with Alissa.

I assume by this you mean that you are seconding the requested change
to the registry policy to "Specification Required" ? I have that change
lined up for the Editor's copy waiting for confirmation that it would
not require another WGLC. See my reply yesterday to Alissa.

AI: Chuck

> Section 4
>   For RPC-over-RDMA version 1, the CM Private Data field is formatted
>   as described in the following subsection.  RPC clients and servers
>   use the same format.  If the capacity of the Private Data field is
>   too small to contain this message format, the underlying RDMA
>   transport is not managed by a Connection Manager, or the underlying
>   RDMA transport uses Private Data for its own purposes, the CM Private
>   Data field cannot be used on behalf of RPC-over-RDMA version 1.
> How will an implementation know if "the underlying RDMA transport uses
> Private Data for its own purposes"?

That's what the Format Identifier is for. It indicates where the
RPC/RDMA-specific data sits in the CM Private Data buffer. A
compliant RPC/RDMA implementation does not assume it is the only
user of the CM Private Data.

The "Interoperability Amongst RDMA Transports" section is intended
to explain this. I'm happy to clarify that text if needed.

> Section 5
>   Although it is possible to reorganize the last three of the eight
>   bytes in the existing format, extended formats are unlikely to do so.
>   New formats would take the form of extensions of the format described
>   in this document with added fields starting at byte eight of the
>   format and changes to the definition of previously reserved flags.
> I would suggest making it a (mandatory) invariant of this format to
> retain these last three bytes' interpretation, requiring the use of a
> different "magic word" for future versions that need to diverge from it.
> The current text does not really give an implementation anything that it
> can rely on.

Our experience with RPC/RDMA version 1 itself is that the only field
that needs to be at a fixed location is the Version field. In fact,
we've found the need to change the semantics and data type of one or
more of the first four fields in the RPC/RDMA version 1 header.

We've had to work around the requirement that they remain the same in
all future versions of RPC/RDMA. The text in Section 5 reflects that

Can you elaborate on the risks that might arise if we were to keep
the existing text?

> Section 6
>   The RPC-over-RDMA version 1 protocol framework depends on the
>   semantics of the Reliable Connected (RC) queue pair (QP) type, as
>   defined in Section 9.7.7 of [IBA].  The integrity of CM Private Data
> It's interesting to see such a reference to [IBA], when IIUC the RFC
> 8166 protocol is not limited to Infiniband as the underlying transport.

Correct, RPC/RDMA can operate on iWARP and RoCE as well. However:

 - both of those RDMA fabric types also support RC QPs, and
 - [IBA] appears to be the only publication where "RC QP type"
   is defined, though I could be wrong.

I don't read this text as suggesting that RPC/RDMA cannot operate
on those alternate fabrics, but perhaps I am "too close to the

>   Additional analysis of RDMA transport security appears in the
>   Security Considerations section of [RFC5042].  That document
> nit: the actual analysis isn't *in* the security considerations section
> (but is referenced from it).

I can revisit RFC 5042 and extract those references.

AI: Chuck

>   Improperly setting one of the fields in a version 1 Private Message
>   can result in an increased risk of disconnection (i.e., self-imposed
>   Denial of Service).  There is no additional risk of exposing upper-
>   layer payloads after exchanging the Private Message format defined in
>   the current document.
> I'm not entirely sure where or how one might have expected such
> additional exposures to occur.

Agreed. However, I was asked to make this statement here to make it
clear that the proposed CM Private Data format does not put data
further at risk. I'm open to suggested rewording.

> We should probably mention the risk that some (other) CM-private data
> item might inadvertenly produce in its payload the "magic number" that
> we use to identify this protocol's data structure.  I *think* (but
> please confirm) that erroneously doing so would lead only to (likely)
> RDMA-channel disconnection and could not introduce (e.g.) data
> corruption.

Yes, my expectation is that it would have the same consequences as
someone tampering with the RPC/RDMA CM Private Data.

I will see about mentioning this failure mode in the text.

AI: Chuck

>   In addition to describing the structure of a new format version, any
>   document that extends the Private Data format described in the
>   current document must discuss security considerations of new data
>   items exchanged between connection peers.
> In a similar vein, future extensions should consider what the risks of
> erroneously identifying "random" data as the new format would be.

AI: Chuck

> Section 7
> Should the registry also include the length of the private data?

I can add that.

AI: Chuck

> Similarly to the previous section's comments, should prospective
> registrations also be analyzing the risks to their protocol of
> interpreting "random" data as the data structure (as would happen upon
> an inadvertent match of the "magic number")?
> Section 7.1
>   The new Reference field should contain a reference to that
>   documentation.  The DE can assign new Format Identifiers at random as
>   long as they do not conflict with existing entries in this registry.
> Random may not be the best choice for this, if there are ways to produce
> values that are less-likely-than-random to occur inadvertently in the
> payload of any of the registered formats.

I chose the current Format Identifier for RPC/RDMA CM Private Data
by extracting the middle bits from a randomly-generated UUID. That's
why the text uses the "random" wording.

IMO we want to avoid choosing values like 0x00000007, precisely because
they are likely to look like a payload. I think we are agreeing here.

Another sentence might be necessary to explain that a new Identifier
should be chosen so that it will not look like a payload of other
registered format. That might be challenging for DEs to carry out in

AI: Chuck

Chuck Lever