[storm] comments on http://tools.ietf.org/html/draft-ietf-storm-rdmap-ext-00

[arkady kanevsky <[e-mail]>]

Here are a few comments on the draft:
In Glossary it is stated:

Atomic Operation - is an operation that results in an execution of a
   64-bit operation at a specific address on a remote node. The
   consumer can use atomic operations to read, modify and write at the
   destination address while at the same time guarantee that no other
   read or write operation will occur across any other RDMAP/DDP
   Streams on an RNIC at the Data Sink.

later in section 5 it is stated:
The
   operations atomically read, modify and write back the contents of
   the destination address and guarantee that atomic operations on this
   address by other Queue Pairs (QPs) on the same RNIC do not occur
   between the read and the write.

in
5.3. Atomicity Guarantees   Atomicity of the RMW on the responder's
node by the Atomic Operation
   SHALL be assured in the presence of concurrent atomic accesses by
   other QPs on the same RNIC.

The three statements are very different. The cases that need to be handled are:
other operation of the same QP, the same RNIC, other devices, local ULP access.

2. It is stated:

The discovery of whether the
   atomic operations are implemented or not is outside the scope of
   this specification and it should be handled by the ULPs or
   applications.

Is this really the best way? Should the connection setup extension draft
which I submitted be extended to exchange this info also? That is which
version of RDMA protocol the remote side supports or requests to support.
The reason I am asking it it not clear to me what option does ULP have if
atomic or immediate data is not supported.

3. What is the Endian(ness) format of Add Data for fetchadd? ditto for
cmpswp.

4. Is there a separate memory registration (Stag creation) for atomic
operation? Or any registered memory can be used for atomic op?

5. How does the requester knows that Stag+remote tagged offset does
corresponds to  a naturally aligned buffer address?

6. "The Swap atomic operation result is unknown when the buffer address is
not naturally aligned." feels a bit loose. I would assume that RNIC will at
least not change any data outside memory specified by Stag.
Also in 5.2.1 #4 states:

At the Remote Peer, when an invalid Atomic Operation Request
      Message is delivered to the Remote Peer's RDMAP layer, an error
      is surfaced.

It sounds like breaking natural alignment is not an error. That is it will
break the connection.

Also:

6. At the Remote Peer, when an invalid Atomic Operation Response
      Message is delivered to the Remote Peer's RDMAP layer, an error
      is surfaced.


Does the error carries the request identifier?

7. Any guidance for sizing ORD/IRD for atomic ops? Does this doc requires my
doc with ORD/IRD negotiation at connect time?

8. If the same memory has two Stag (overlapping memory regions) is atomic op
ordering only preserved for a single QP/connection? What happens if natural
alignment is less then 64-bit? Is there any order guaranteed per Stag (if
two QPs use the same Stag for atomic or RDMA ops)?

9. It is strange that for atomic op we are using 64-bit terminology but for
immediate data we use 8-byte terminology.

10.Suggest to specify which queues and buffers immediate data message uses?

Arkady

-- 
Cheers,
Arkady Kanevsky