Re: [storm] iSER draft

Alexander Nezhinsky <> Thu, 30 June 2011 18:07 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 1BA5011E823C for <>; Thu, 30 Jun 2011 11:07:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -3.598
X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 8cfP4HUAAa3k for <>; Thu, 30 Jun 2011 11:07:51 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id C3CA111E8240 for <>; Thu, 30 Jun 2011 11:07:46 -0700 (PDT)
Received: by ywp31 with SMTP id 31so1261056ywp.31 for <>; Thu, 30 Jun 2011 11:07:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=V56kMUeXYDjUIeeM8yVTjokrNIlm4NoGQsjcH5qC6K0=; b=yEhJklZyoZ61bOiccqCPuOoncCzVVn0nWGjnBCghCg3TK0V9lnogYuMlat2eBPo+Ss S9VZyK3l+hN/rC/e9jc4AZgKG0oRqz+dNmE/0GBOYyFZDW/5oD9fvyr+ndO1dMLe+ofc 8tb0xB1MuGHIeM5P7GK/BNjCQdwIropSvbQZU=
MIME-Version: 1.0
Received: by with SMTP id f41mr2968696yhe.233.1309457266283; Thu, 30 Jun 2011 11:07:46 -0700 (PDT)
Received: by with HTTP; Thu, 30 Jun 2011 11:07:46 -0700 (PDT)
In-Reply-To: <>
References: <> <> <>
Date: Thu, 30 Jun 2011 21:07:46 +0300
Message-ID: <>
From: Alexander Nezhinsky <>
To: Michael Ko <>
Content-Type: multipart/alternative; boundary=20cf300514fa3492cb04a6f1c730
Subject: Re: [storm] iSER draft
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 30 Jun 2011 18:07:52 -0000

On Thu, Jun 30, 2011 at 12:26 AM, Michael Ko <>wrote;wrote:

> **
> Tom,
> As originally written, the iSER spec always assumes ZBVA (zero-based
> virtual address).  But when iSER is used in conjunction with transports
> other than iWARP, there is a need to include the virtual address for the
> tagged buffer.  Such is the case for Infiniband.  Accordingly, the Virtual
> Address field has been added to the iSCSI Control Type PDUs (sec. 9.2).
> Alex pointed out that the tagged buffer should now be identified by the
> 4-tuple (STag, VA, TO, and length) instead.  This should be updated
> everywhere where the term "tagged buffer" is mentioned in the spec.  For
> example, in sec. 1.1 under Advertisement, the sentence "A typical method
> would be for the iSER Layer to embed the Tagged Buffer's STag, TO, and
> buffer length in a Send Message destined for the remote iSER Layer" should
> be modified to "A typical method would be for the iSER Layer to embed the
> Tagged Buffer's STag, VA, TO, and buffer length in a Send Message destined
> for the remote iSER Layer."  BTW, this sentence also answers your question
> as to whether VA is passed on the wire.

Agree, using VA was in the draft already, I only suggested to clarify its
use (and modes of use) from the level of definitions and up,
so that it won't "suddenly" appear in the header format description section.

> The Virtual Address is literally the virtual address of the tagged buffer.
> The tagged offset is the displacement relative to the beginning of the
> buffer starting at the virtual address.
> We could rephrase the definition of Virtual Address in sec. 1.1 as follows:
> "Virtual Address (VA) - This is the virtual address of the Tagged Buffer on
> a Node.  When set to 0, it indicates that a virtual address is not needed to
> locate the Tagged Buffer."

> Read and write buffers are identified by the combination of STag and VA.

OK. I tried to make the same point in the following:

> Both fields should be required for any transport. This requirement does not
> preclude
> using ZBVA where possible. When VA is not used (as in iWARP) it is set to
> 0.
> Below is the actually used format:
> struct iser_hdr {
>   uint8_t   flags;
>   uint8_t   rsvd[3];
>   uint32_t  write_stag; /* write rkey in IB */
>   uint64_t  write_va;
>   uint32_t  read_stag;  /* read rkey in IB */
>   uint64_t  read_va;
> }
> I would like to add that InfiniBand does have ZBVA mode. It supposedly
provides less than optimal performance with IB.
And it is not exported through the current rdma-cm API, which supports
RDMA/Ethernet as well, and is the only option
to use RDMA in user space linux apps, as of today.

So you could look at the additional fields as a kind of "reserved" in case
of standard ZBVA implementation,
used to practically support certain APIs and get enhanced performance in
some cases.