[storm] comments on http://tools.ietf.org/html/draft-ietf-storm-rdmap-ext-00

arkady kanevsky <arkady.kanevsky@gmail.com> Tue, 03 May 2011 16:42 UTC

Return-Path: <arkady.kanevsky@gmail.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id B5459E087F for <storm@ietfa.amsl.com>; Tue, 3 May 2011 09:42:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.998
X-Spam-Status: No, score=-2.998 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_46=0.6, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id Qh8K7XP8o-cB for <storm@ietfa.amsl.com>; Tue, 3 May 2011 09:42:27 -0700 (PDT)
Received: from mail-pw0-f44.google.com (mail-pw0-f44.google.com []) by ietfa.amsl.com (Postfix) with ESMTP id 0DBB2E0843 for <storm@ietf.org>; Tue, 3 May 2011 09:42:24 -0700 (PDT)
Received: by pwi5 with SMTP id 5so161165pwi.31 for <storm@ietf.org>; Tue, 03 May 2011 09:42:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=7PRjr5WAwVvvlRZFhGK/UY03Wk3m1v2qKtN8ETf4ezI=; b=aCUChMrzzKlZelD+OHIYUL5RpjBJO9T3ybMEEJUUgj8L4Ue5vVAXmEZL+O/EHuU98N IwZE1B+VJkPIMMifkJei3fPGNfsYY5Odc5NHdPyO3PNr4RopK+4cv2OmEMkfHq0PbrGW k/AvECwgKzormYq3QWM6aaUdQKZhLntI0v+ok=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=AFEbXKy0nP+Lp5uxViknyJblTasXeNzqbwpTp7gYKnsZnbpRuhTHICFZ/Osa4OGQQ4 zuzhdp748G+AkOQTpLW3KYbnr4TlTJkI04N8xCRV34M5SXA4k494FACqfejgcxO++oMI F82PYjGWYh3fKb/mnozjnFkvQtFr4IdznyPR8=
MIME-Version: 1.0
Received: by with SMTP id 12mr24478wfg.181.1304440944512; Tue, 03 May 2011 09:42:24 -0700 (PDT)
Received: by with HTTP; Tue, 3 May 2011 09:42:24 -0700 (PDT)
Date: Tue, 3 May 2011 12:42:24 -0400
Message-ID: <BANLkTiku2Q9uFi34szED0PnPwLe=6CwsKA@mail.gmail.com>
From: arkady kanevsky <arkady.kanevsky@gmail.com>
To: storm@ietf.org, "Sharp, Robert O" <robert.o.sharp@intel.com>
Content-Type: multipart/alternative; boundary=00504502bc6320cf7c04a261d320
Subject: [storm] comments on http://tools.ietf.org/html/draft-ietf-storm-rdmap-ext-00
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 03 May 2011 16:42:28 -0000

Here are a few comments on the draft:
In Glossary it is stated:

Atomic Operation - is an operation that results in an execution of a
   64-bit operation at a specific address on a remote node. The
   consumer can use atomic operations to read, modify and write at the
   destination address while at the same time guarantee that no other
   read or write operation will occur across any other RDMAP/DDP
   Streams on an RNIC at the Data Sink.

later in section 5 it is stated:
   operations atomically read, modify and write back the contents of
   the destination address and guarantee that atomic operations on this
   address by other Queue Pairs (QPs) on the same RNIC do not occur
   between the read and the write.

5.3. Atomicity Guarantees   Atomicity of the RMW on the responder's
node by the Atomic Operation
   SHALL be assured in the presence of concurrent atomic accesses by
   other QPs on the same RNIC.

The three statements are very different. The cases that need to be handled are:
other operation of the same QP, the same RNIC, other devices, local ULP access.

2. It is stated:

The discovery of whether the
   atomic operations are implemented or not is outside the scope of
   this specification and it should be handled by the ULPs or

Is this really the best way? Should the connection setup extension draft
which I submitted be extended to exchange this info also? That is which
version of RDMA protocol the remote side supports or requests to support.
The reason I am asking it it not clear to me what option does ULP have if
atomic or immediate data is not supported.

3. What is the Endian(ness) format of Add Data for fetchadd? ditto for

4. Is there a separate memory registration (Stag creation) for atomic
operation? Or any registered memory can be used for atomic op?

5. How does the requester knows that Stag+remote tagged offset does
corresponds to  a naturally aligned buffer address?

6. "The Swap atomic operation result is unknown when the buffer address is
not naturally aligned." feels a bit loose. I would assume that RNIC will at
least not change any data outside memory specified by Stag.
Also in 5.2.1 #4 states:

At the Remote Peer, when an invalid Atomic Operation Request
      Message is delivered to the Remote Peer's RDMAP layer, an error
      is surfaced.

It sounds like breaking natural alignment is not an error. That is it will
break the connection.


6. At the Remote Peer, when an invalid Atomic Operation Response
      Message is delivered to the Remote Peer's RDMAP layer, an error
      is surfaced.

Does the error carries the request identifier?

7. Any guidance for sizing ORD/IRD for atomic ops? Does this doc requires my
doc with ORD/IRD negotiation at connect time?

8. If the same memory has two Stag (overlapping memory regions) is atomic op
ordering only preserved for a single QP/connection? What happens if natural
alignment is less then 64-bit? Is there any order guaranteed per Stag (if
two QPs use the same Stag for atomic or RDMA ops)?

9. It is strange that for atomic op we are using 64-bit terminology but for
immediate data we use 8-byte terminology.

10.Suggest to specify which queues and buffers immediate data message uses?


Arkady Kanevsky