[storm] more comments on http://tools.ietf.org/html/draft-ietf-storm-rdmap-ext-00

Bernard Metzler <BMT@zurich.ibm.com> Tue, 05 July 2011 22:49 UTC

Return-Path: <BMT@zurich.ibm.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id DF87921F88EC for <storm@ietfa.amsl.com>; Tue, 5 Jul 2011 15:49:13 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.489
X-Spam-Status: No, score=-5.489 tagged_above=-999 required=5 tests=[AWL=1.110, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id pSrcbYsSEYbo for <storm@ietfa.amsl.com>; Tue, 5 Jul 2011 15:49:13 -0700 (PDT)
Received: from mtagate2.uk.ibm.com (mtagate2.uk.ibm.com []) by ietfa.amsl.com (Postfix) with ESMTP id E32D521F88EB for <storm@ietf.org>; Tue, 5 Jul 2011 15:49:12 -0700 (PDT)
Received: from d06nrmr1707.portsmouth.uk.ibm.com (d06nrmr1707.portsmouth.uk.ibm.com []) by mtagate2.uk.ibm.com (8.13.1/8.13.1) with ESMTP id p65MnBei008269 for <storm@ietf.org>; Tue, 5 Jul 2011 22:49:11 GMT
Received: from d06av08.portsmouth.uk.ibm.com (d06av08.portsmouth.uk.ibm.com []) by d06nrmr1707.portsmouth.uk.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p65Mn4ho1974508 for <storm@ietf.org>; Tue, 5 Jul 2011 23:49:11 +0100
Received: from d06av08.portsmouth.uk.ibm.com (loopback []) by d06av08.portsmouth.uk.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p65Mn3MU010452 for <storm@ietf.org>; Tue, 5 Jul 2011 23:49:04 +0100
Received: from d12mc302.megacenter.de.ibm.com (d12mc302.megacenter.de.ibm.com []) by d06av08.portsmouth.uk.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id p65Mn3gm010449; Tue, 5 Jul 2011 23:49:03 +0100
To: storm@ietf.org
MIME-Version: 1.0
X-KeepSent: 68B8817A:D6C70115-C12578C4:0052797F; type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5.1 September 28, 2009
Message-ID: <OF68B8817A.D6C70115-ONC12578C4.0052797F-C12578C4.007D5757@ch.ibm.com>
From: Bernard Metzler <BMT@zurich.ibm.com>
Date: Wed, 6 Jul 2011 00:49:03 +0200
X-MIMETrack: Serialize by Router on D12MC302/12/M/IBM(Release 8.5.2FP1HF342 | May 5, 2011) at 06/07/2011 00:49:03, Serialize complete at 06/07/2011 00:49:03
Content-Type: text/plain; charset="US-ASCII"
Subject: [storm] more comments on http://tools.ietf.org/html/draft-ietf-storm-rdmap-ext-00
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Jul 2011 22:49:14 -0000

Some comments on the RDMA Protocol Extension draft
(sorry when partially overlapping with Arkady's email
from 3 May).

The terms "Data Source" and "Data Sink" are used for Immediate
Data and Atomic Operations as well. For Atomics I found it
confusing and would prefer to have the 'Requester' and 'Responder'
terms used throughout the whole document.

Immediate Data
The specification introduces a new RDMAP message format.
Section 1 and 6 are limiting its usage to Writes, eg.:
"Immediate Data messages allow the ULP at the sender to
provide a small amount of data following an RDMA Write payload."
Why not Sends, while SendWithImm is defined for other RDMA
transports and is part of a well established RDMA verbs
interface (OpenFabrics)?

While potentially out of the primary business of the
specification of a wire protocol - I assume Immediate Data
will consume a peers Receive Queue Element?
A short discussion including appropriate RQ sizing might
be helpful for the reader (and implementer). It might go
into section 6.1.

5.1.1: I found it difficult to understand the FetchAdd decription.
In particular, the statement of
"The setting of "Add Mask" field to 0x0000000000000000 results
in Atomic Add of 64-bit Original Remote Data Value and 64-bit
"Add Data"."
was surprising to me. I would have expected all bits set to 1.

FetchAdd sometimes extends to FetchAndAdd. I would prefer one term

The abbreviation of RMW seem to refer to ReadModifyWrite.
Better to be written in full length? It could be confused
with RemoteMemoryWindow.

Ordering and Completion Table
Section 7 provides a table of operation ordering.
For Immediate Data, I found the term "Placement" inappropriate.
Immediate Data are not placed in a target buffer. Immediate Data
are received and delivered to the ULP. Maybe exchanging "Immediate
Data is Placed at Remote Peer" with "Immediate Data is received
at Remote Peer" would clarify? I see the same issue for Atomic
Responses, which are also not Placed but received.

The discussion of the two consecutive Atomic operations (page 22)
seem to allow re-ordering of Atomic responses. Is that true?
It might complicate response processing. 

It might be a good idea to try an implementation of
the proposed RDMAP extension. The SoftiWARP stack might
be a good candidate for that.