Re: [storm] iSER draft

[Alexander Nezhinsky <[e-mail]>]

On Thu, Jun 30, 2011 at 12:26 AM, Michael Ko <[e-mail]>wrote:

> **
> Tom,
>
> As originally written, the iSER spec always assumes ZBVA (zero-based
> virtual address).  But when iSER is used in conjunction with transports
> other than iWARP, there is a need to include the virtual address for the
> tagged buffer.  Such is the case for Infiniband.  Accordingly, the Virtual
> Address field has been added to the iSCSI Control Type PDUs (sec. 9.2).
> Alex pointed out that the tagged buffer should now be identified by the
> 4-tuple (STag, VA, TO, and length) instead.  This should be updated
> everywhere where the term "tagged buffer" is mentioned in the spec.  For
> example, in sec. 1.1 under Advertisement, the sentence "A typical method
> would be for the iSER Layer to embed the Tagged Buffer's STag, TO, and
> buffer length in a Send Message destined for the remote iSER Layer" should
> be modified to "A typical method would be for the iSER Layer to embed the
> Tagged Buffer's STag, VA, TO, and buffer length in a Send Message destined
> for the remote iSER Layer."  BTW, this sentence also answers your question
> as to whether VA is passed on the wire.
>
>

Agree, using VA was in the draft already, I only suggested to clarify its
use (and modes of use) from the level of definitions and up,
so that it won't "suddenly" appear in the header format description section.


> The Virtual Address is literally the virtual address of the tagged buffer.
> The tagged offset is the displacement relative to the beginning of the
> buffer starting at the virtual address.
>
> We could rephrase the definition of Virtual Address in sec. 1.1 as follows:
>
> "Virtual Address (VA) - This is the virtual address of the Tagged Buffer on
> a Node.  When set to 0, it indicates that a virtual address is not needed to
> locate the Tagged Buffer."
>
>

> Read and write buffers are identified by the combination of STag and VA.
>

OK. I tried to make the same point in the following:

> Both fields should be required for any transport. This requirement does not
> preclude
> using ZBVA where possible. When VA is not used (as in iWARP) it is set to
> 0.
>
> Below is the actually used format:
>
> struct iser_hdr {
>   uint8_t   flags;
>   uint8_t   rsvd[3];
>   uint32_t  write_stag; /* write rkey in IB */
>   uint64_t  write_va;
>   uint32_t  read_stag;  /* read rkey in IB */
>   uint64_t  read_va;
> }
>
> I would like to add that InfiniBand does have ZBVA mode. It supposedly
provides less than optimal performance with IB.
And it is not exported through the current rdma-cm API, which supports
RDMA/Ethernet as well, and is the only option
to use RDMA in user space linux apps, as of today.

So you could look at the additional fields as a kind of "reserved" in case
of standard ZBVA implementation,
used to practically support certain APIs and get enhanced performance in
some cases.

Alex