Re: [nfsv4] WG last call for draft-ietf-nfsv4-rpcrdma-cm-pvt-data-02.txt (Ends May 31st)

Chuck Lever <chuck.lever@oracle.com> Wed, 12 June 2019 19:07 UTC

Return-Path: <chuck.lever@oracle.com>
X-Original-To: nfsv4@ietfa.amsl.com
Delivered-To: nfsv4@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 99E081201E9 for <nfsv4@ietfa.amsl.com>; Wed, 12 Jun 2019 12:07:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.31
X-Spam-Level:
X-Spam-Status: No, score=-4.31 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_DKIMWL_WL_HIGH=-0.01, UNPARSEABLE_RELAY=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=oracle.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NllqZBeQqm10 for <nfsv4@ietfa.amsl.com>; Wed, 12 Jun 2019 12:07:25 -0700 (PDT)
Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 63A061201DB for <nfsv4@ietf.org>; Wed, 12 Jun 2019 12:07:25 -0700 (PDT)
Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x5CJ3muu107451; Wed, 12 Jun 2019 19:07:24 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=content-type : mime-version : subject : from : in-reply-to : date : cc : content-transfer-encoding : message-id : references : to; s=corp-2018-07-02; bh=1/Dm2sQZ+6Xx2TxxgZaZcy4Hbb/m6bD1FRTUCLMQ7M0=; b=IujbBoXY+JsoNX6dEvJ9Mj+8Cynodo20FY11eT783bYAi1Qg6D5YFd862l+9eF5rk2CE tKsLuJTOCear+U2izMn3HtfvsZ5CDgMkiJnIyBuhSZnZKlIej6LSfjax3vrmnlTmbWNz UmIEvSgvBOZU+E5Z16O7BiJqrfl96U3rw5coGRUmvutXZQ5K633edCnwG9YR0Z1ugj5n ORf6WiNeAceJWLetOBRxz6c/WSI2RVjw7/PE4VOb2MBYMTHpRrzhM/zTnKer8Ky1fzQw C7m4ddcOaaJ0dB6o4vTHMuwko/wwqscdSrP5hOtlHj6Y0+PAPMPBh5UPztJPBAvr7k8v jw==
Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 2t05nqwd18-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 12 Jun 2019 19:07:24 +0000
Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x5CJ6pZ8150822; Wed, 12 Jun 2019 19:07:23 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userp3020.oracle.com with ESMTP id 2t1jpj68g9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 12 Jun 2019 19:07:23 +0000
Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x5CJ7MKY019388; Wed, 12 Jun 2019 19:07:23 GMT
Received: from anon-dhcp-171.1015granger.net (/68.61.232.219) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 12 Jun 2019 12:07:22 -0700
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
From: Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <7e509bb4-461c-55f9-9e0d-3469daee84d2@talpey.com>
Date: Wed, 12 Jun 2019 15:07:17 -0400
Cc: nfsv4@ietf.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <4170A375-7008-4319-9AA3-34560CBCBF14@oracle.com>
References: <CAFt6BanN_L9EyLyf3Hfpd1nBHaRNh5-jppcXNEGugCfiFvOKww@mail.gmail.com> <3d25b43c-cbb7-1681-e647-fc4b3c6ba0ba@talpey.com> <90FF728D-F043-4CB5-B3D8-D30FB5B95B6B@oracle.com> <7e509bb4-461c-55f9-9e0d-3469daee84d2@talpey.com>
To: Tom Talpey <tom@talpey.com>
X-Mailer: Apple Mail (2.3445.104.11)
X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9286 signatures=668687
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906120129
X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9286 signatures=668687
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906120129
Archived-At: <https://mailarchive.ietf.org/arch/msg/nfsv4/8sAq1EkX6JOXjFEqJRHR72AHftU>
Subject: Re: [nfsv4] WG last call for draft-ietf-nfsv4-rpcrdma-cm-pvt-data-02.txt (Ends May 31st)
X-BeenThere: nfsv4@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: NFSv4 Working Group <nfsv4.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nfsv4/>
List-Post: <mailto:nfsv4@ietf.org>
List-Help: <mailto:nfsv4-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/nfsv4>, <mailto:nfsv4-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Jun 2019 19:07:28 -0000

[ Already "good" replacements have been snipped ]

> On Jun 11, 2019, at 4:35 PM, Tom Talpey <tom@talpey.com> wrote:
> 
> On 6/11/2019 3:20 PM, Chuck Lever wrote:
> 
>>> In section 3.2 third paragraph two comments. First, it describes an
>>> advantage of avoiding a context switch. This is a local implementation
>>> matter, and not really something the protocol can or cannot proscribe.
>>> It would be state the benefit in protocol terms, such as "avoids
>>> processing of the invalidate completion", then go on to say which may
>>> enable avoiding interrupt(s) and context switch(es). Second, a nit - the
>>> word "upshot" is slang. Suggest "benefit".
>> I plan to replace the third paragraph of Section 3.2:
>> OLD TEXT:
>> An RPC-over-RDMA responder can use remote invalidation
>> when replying to an RPC request that provided Read or
>> Write chunks. The requester thus avoids dispatching an
>> extra Work Request, the resulting context switch, and
>> the invalidation completion interrupt as part of
>> completing an RPC transaction that uses chunks. The
>> upshot is faster completion of RPC transactions that
>> involve RDMA data transfer.
>> REPLACEMENT TEXT:
>> An RPC-over-RDMA responder can use remote invalidation
>> when replying to an RPC request that provided chunks.
>> The requester thus avoids processing an invalidation
>> completion for that chunk. The benefit is faster
>> completion of RPC transactions that involve RDMA data
>> transfer.
> 
> The words "The requester thus avoids processing..." don't quite
> capture the benefit, I feel. In particular, "processing" is a
> rather passive term and doesn't capture the full benefit.
> 
> Here's a swag:
> 
> "This extensions provides a means for the RPC-over-RDMA requester
> to signal the responder to optionally use this remote invalidation
> when replying to an RPC request that provided chunks. In this way,
> the RDMA provider performs the invalidation in the process of
> delivering the response, which allows the requester to avoid
> required additional, and possibly expensive, invalidation prior
> to finalizing the results of the RPC."
> 
> I admit, it's tricky language to get right. Maybe it can be stated
> more simply.

The preceding paragraphs already provide the same context
that you suggested above. I've updated the third paragraph
again. Here is how Section 3.2 starts now:


After an RDMA data transfer operation completes, an RDMA peer can use
remote invalidation to request that the remote peer RNIC invalidate
an STag associated with the data transfer [RFC5042].

An RDMA consumer requests remote invalidation by posting an RDMA Send
With Invalidate Work Request in place of an RDMA Send Work Request.
Each RDMA Send With Invalidate carries one STag to invalidate.  The
receiver of an RDMA Send With Invalidate performs the requested
invalidation and then reports that invalidation as part of the
completion of a waiting Receive Work Request.

An RPC-over-RDMA responder uses remote invalidation when replying to
an RPC request that provided chunks.  Because one of the chunks has
already been invalidated, finalizing the results of the RPC is made
simpler and faster.


>>> At the end of section 3.2, the final paragraph is quite mysterious. It
>>> refers to "a simple signaling mechanism" and "some NFS/RDMA client".
>>> Really, something more needs to be said here. If this signaling
>>> mechanism involves a protocol payload, then definitely so. Can you
>>> elaborate? I admit I don't fully understand what is meant here.
>> I plan to replace the final paragraph of Section 3.2:
>> OLD TEXT:
>> However, it is possible to provide a simple signaling mechanism
>> for a requester to indicate it can deal with remote invalidation
>> of any STag it has presented to a responder.
>> There are some NFS/RDMA client implementations that
>> can successfully make use of such a signaling mechanism.
>> REPLACEMENT TEXT:
>> There are some NFS/RDMA client implementations whose STags
>> are always safe to invalidate remotely.
>> For such clients, indicating to the responder that remote
>> invalidation is always safe can allow such invalidation
>> without the need for more complex communication between the
>> requester and responder.
> 
> "...without the need for additional protocol to be defined for
> management of STags between requester and responder."
> 
> Maybe?

This works for me:

There are some NFS/RDMA client implementations whose STags are always
safe to invalidate remotely.  For such clients, indicating to the
responder that remote invalidation is always safe can allow such
invalidation without the need for additional protocol to be
defined.


>>> In section 7, the vulnerability is in the RDMA-CM protocol, not this
>>> protocol. And, private data is carried differently by different lower
>>> layers - for example in the IETF iWARP family, it's part of MPA.
>>> Honestly, I'm not sure this is relevant to this document. It basically
>>> inherits all the considerations of the protocols it extends.
>> I've replaced the entirety of the Security Considerations
>> section as follows:
>> OLD TEXT:
>> RDMA-CM Private Data typically traverses the link layer in
>> the clear. A man-in-the-middle attack could alter the
>> settings exchanged at connect time such that one or both
>> peers might perform operations that result in premature
>> termination of the connection.
>> REPLACEMENT TEXT:
>> The private data extension specified in this document
>> inherits the security considerations of the link layer
>> protocols it extends (e.g., the RDMA-CM protocol).
> 
> Hmm, this will trigger the next logical question "where is this protocol
> defined?", and the answer ("outside the IETF") will cause trouble.
> 
> Also, it's not strictly the link layer. All the IETF RDMA protocols
> operate above transport.
> 
> I'd suggest keeping the example to the IETF-owned MPA protocol, which
> is the one governing RDMA connection establishment and which carries
> private data. I.e, RFC 5044 (MPA) and RFC6581 (MPA extensions). MPA, in turn, makes its own security considerations and refers to RFC5042 (the
> original RDMA Security analysis), which may be a useful reference but
> is not specifically about connection establishment.

In it's entirety, the Security Considerations section now reads:

The private data extension specified in this document inherits the
security considerations of the link layer protocols it extends; e.g.,
the MPA protocol, as specified in [RFC5044] and extended in
[RFC6581].  Additional relevant analysis of RDMA security appears in
the Security Considerations section of [RFC5042] but it is not
specifically about connection establishment.


--
Chuck Lever