Re: [storm] iSER draft

[Tom Talpey <[e-mail]>]

Yes, understood on the ZBVA/Infiniband issue. Regarding the VA term’s presence in an earlier draft, I did not see it in RFC5046 and I did not review the expired draft, so consider these comments to be on the overall change, not the last revision.

Let me try a different approach to perhaps make myself clearer.

First, it seems to me that a “virtual address” has no real meaning to the network protocol. It begins, perhaps, as some value in an address space on the initiator, but it’s not meaningful on the target in this way, it’s only used to request that the remote RNIC perform a transfer to that originally registered remote address. So calling it a Virtual Address as an iSER protocol object seems, to me, an artificial and somewhat leading convention.

Second, the Virtual Address goes out from the initiator to the target in a Control PDU, but where does it come back? The RDMA Read or Write as depicted in (xx) shows only a Tagged Offset. So, it’s not clear what its protocol meaning is.

Third, I don’t ever see a Tagged Buffer described by a fully qualified four-tuple. I see it appearing as either { Stag, TO, length } or { Stag, VA, length }, depending on the addressing mode.

I think the non-ZBVA mode is really just a special case of the existing one, but where the meaning of TO has changed from a small offset to some other token, generated and managed only at the initiator. So, it seems artificial to define it as distinct, and document it as possessing some new properties. Isn’t it just a Target Offset, still?

Tom.



From: Alexander Nezhinsky [mailto:[e-mail]]
Sent: Thursday, June 30, 2011 2:08 PM
To: Michael Ko
Cc: Tom Talpey; [e-mail]
Subject: Re: [storm] iSER draft


On Thu, Jun 30, 2011 at 12:26 AM, Michael Ko <[e-mail]<mailto:[e-mail]>> wrote:
Tom,

As originally written, the iSER spec always assumes ZBVA (zero-based virtual address).  But when iSER is used in conjunction with transports other than iWARP, there is a need to include the virtual address for the tagged buffer.  Such is the case for Infiniband.  Accordingly, the Virtual Address field has been added to the iSCSI Control Type PDUs (sec. 9.2).  Alex pointed out that the tagged buffer should now be identified by the 4-tuple (STag, VA, TO, and length) instead.  This should be updated everywhere where the term "tagged buffer" is mentioned in the spec.  For example, in sec. 1.1 under Advertisement, the sentence "A typical method would be for the iSER Layer to embed the Tagged Buffer's STag, TO, and buffer length in a Send Message destined for the remote iSER Layer" should be modified to "A typical method would be for the iSER Layer to embed the Tagged Buffer's STag, VA, TO, and buffer length in a Send Message destined for the remote iSER Layer."  BTW, this sentence also answers your question as to whether VA is passed on the wire.


Agree, using VA was in the draft already, I only suggested to clarify its use (and modes of use) from the level of definitions and up,
so that it won't "suddenly" appear in the header format description section.

The Virtual Address is literally the virtual address of the tagged buffer.  The tagged offset is the displacement relative to the beginning of the buffer starting at the virtual address.

We could rephrase the definition of Virtual Address in sec. 1.1 as follows:

"Virtual Address (VA) - This is the virtual address of the Tagged Buffer on a Node.  When set to 0, it indicates that a virtual address is not needed to locate the Tagged Buffer."


Read and write buffers are identified by the combination of STag and VA.

OK. I tried to make the same point in the following:
Both fields should be required for any transport. This requirement does not preclude
using ZBVA where possible. When VA is not used (as in iWARP) it is set to 0.

Below is the actually used format:

struct iser_hdr {
  uint8_t   flags;
  uint8_t   rsvd[3];
  uint32_t  write_stag; /* write rkey in IB */
  uint64_t  write_va;
  uint32_t  read_stag;  /* read rkey in IB */
  uint64_t  read_va;
}
I would like to add that InfiniBand does have ZBVA mode. It supposedly provides less than optimal performance with IB.
And it is not exported through the current rdma-cm API, which supports RDMA/Ethernet as well, and is the only option
to use RDMA in user space linux apps, as of today.

So you could look at the additional fields as a kind of "reserved" in case of standard ZBVA implementation,
used to practically support certain APIs and get enhanced performance in some cases.

Alex