Re: [storm] WG Last Call - iSER - through December 12 - comments

<> Wed, 14 December 2011 14:44 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 6CA1521F8AB8 for <>; Wed, 14 Dec 2011 06:44:17 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -106.512
X-Spam-Status: No, score=-106.512 tagged_above=-999 required=5 tests=[AWL=0.086, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id r4ZYFy+uTz90 for <>; Wed, 14 Dec 2011 06:44:12 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTP id B294121F86EE for <>; Wed, 14 Dec 2011 06:44:11 -0800 (PST)
Received: from ( []) by (Switch-3.4.3/Switch-3.4.3) with ESMTP id pBEEi6vu024301 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 Dec 2011 09:44:07 -0500
Received: from ( []) by (RSA Interceptor); Wed, 14 Dec 2011 09:43:49 -0500
Received: from ( []) by (Switch-3.4.3/Switch-3.4.3) with ESMTP id pBEEhmn8021817; Wed, 14 Dec 2011 09:43:48 -0500
Received: from ([]) by ([]) with mapi; Wed, 14 Dec 2011 09:43:48 -0500
From: <>
To: <>, <>
Date: Wed, 14 Dec 2011 09:43:46 -0500
Thread-Topic: WG Last Call - iSER - through December 12 - comments
Thread-Index: AcyQIZ9GrcM5E+JESbe7RIuFF0Mv9AYYaKUgBCRY0oAAVaTO8A==
Message-ID: <>
References: <> <>
In-Reply-To: <>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: multipart/alternative; boundary="_000_7C4DFCE962635144B8FAE8CA11D0BF1E05A000D506MX14Acorpemcc_"
MIME-Version: 1.0
Subject: Re: [storm] WG Last Call - iSER - through December 12 - comments
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 14 Dec 2011 14:44:17 -0000


Thank you for the detailed review.

The minor comments appear to all be editorial, so I'll leave them to Mike to resolve.

On the major comments:

Items 11 and 12 are the same issue, as I also found item 11, and the text quoted in item 12 was supposed to resolve it.  Obviously, that text is not sufficiently clear ;-), so please work with Mike to come up with a clearer paragraph.

Item 13 looks like a useful clarification - the non-TCP connection may have to be torn down in order to establish a TCP connection, and IB is an example where there is necessary.

Items 14 and 15 represent a deliberate change because the implementations did something different from what's in RFC 5046, as noted in Appendix A (Section 14).

   8. Added two 64-bit fields in iSER header in section 9.2 for the Read
     Base Offset and the Write Base Offset to accommodate a non-zero
     Base Offset.  This allows one implementation such as the OFED
     stack to be used in both the Infiniband and the iWARP environment.
     Changes were made in the definition of Base Offset, Advertisement,
     and Tagged Buffer.  Changes were also made in sections 2.4.1, 2.5,
     2.6, 7.3.1, 7.3.3, 7.3.5, 7.3.6, 9.1, 9.3, 9.4, 9.5.1, and 9.5.2.
     This change is not backward compatible with RFC 5046.

I would suggest changing the last sentence to:

     This change is not backward compatible with RFC 5046, but is part
     of all known implementations of iSER at the time this document was

If you know of an implementation that did *NOT* do this, please speak up now!

Item 16 - I would prefer not to add the iSER InfiniBand message formats to this draft, but instead leave them to the IBTA to document.  OTOH, if there's no good documentation of these for InfiniBand, they could be added.  What's the state of IB documentation for iSER?

Item 17 - It looks like this is covered by item 9 in Annex A:

   9. Remove iWARP specific behavior.  Changes were made in the
     definition section on RDMA Operation and Send Message Type.
     Changes affecting Send with Invalidate were made in sections
     2.4.1, 2.5, 2.6, 4.1, and 7.3.2.  Changes affecting Terminate were
     made in sections and  Changes were made in
     section 15 to remove iWARP headers.

The notion of Solicited Event (SE) is iWARP-specific; the generalization above seems useful.  OTOH, RFC 5046 contained requirements for use of SendSE and SendInvSE, and those requirements appear to have been removed.  I believe that they need to be restored, possibly in a new section that is specifically about iSER over iWARP.


From: Hemal Shah []
Sent: Monday, December 12, 2011 8:06 PM
To: Black, David;
Subject: RE: WG Last Call - iSER - through December 12 - comments

David and Mike,

Below are my comments on the iSER draft.


 1.  Section 1.1: iSCSI data-type PDU definition should clearly say at what layer the data transfer is happening. For example, the following sentence "An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection." should be rephrased (added text in red) as "An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer via RDMA operations at the iSER layer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection. I believe that the definition is already being revised based on David's comments.
 2.  Section 1.1: The definition of iSCSI/iSER session needs to have further description. The current description is insufficient. Suggest to add the following to the definition "All connections of an iSCSI/iSER session are iSCSI/iSER connections."
 3.  SNIC term is not defined but used in this draft. This needs to be addressed.
 4.  Sections 2.4.3 and 2.4.4 use terms "sending side", "receiving side", "remote node", etc. It is confusing to use multiple terms (some of them are not defined in the draft) in these two sections. Suggest to use "Local Peer/Remote Peer" or "Data Source/Data Sink" terms consistently in these two sections.
 5.  The use of term input qualifiers in Section 3 is misleading as we are actually referring to the input parameters for the operational primitives. Suggest to replace term "input qualifier" with "input parameter" in Section 3.
 6.  Section 5.1.3: In the following sentence the second "iSER-ORD" should be replaced with "ORD". "Upon receiving the iSER Hello Message, the iSER Layer at the target MUST set the iSER-ORD value to the minimum of the iSER-ORD value at the target and the iSER-IRD value declared by the initiator."
 7.  Section 9.2: Do WSV and RSV flags indicate the validity of WBO and RBO fields? The description of these flags should state it.
 8.  Section 15 should be titled as "iWARP Message Formats for iSER".
 9.  Sections 15.3-15.5 should be formatted as iWARP messages.
 10. I do not understand the reason behind removing the iWARP Message Format for RDMA Read Request, iWARP Message Format for Solicited SCSI Write Data, and iWARP Message Format for SCSI Response PDU in Section 15. They should be kept in Section 15.


 1.  According to the iSCSI spec, an iSCSI connection has an underlying TCP connection. Based on that, an iSER-assisted iSCSI connection requires an underlying TCP connection to be present. For iWARP, an iSER-assisted iSCSI connection has an underlying TCP connection. For iSER-assisted iSCSI connection over InfiniBand transports, there is no underlying TCP connection. In this case, what does an iSER-assisted iSCSI connection mean? What IB transport service types are used for the iSER-assisted iSCSI connection? The draft is not clear on what an iSER-assisted iSCSI connection mean for the InfiniBand transports. The draft needs to clearly define iSER-assisted iSCSI connection for InfiniBand transports and its mapping over underlying InfiniBand transport connections and transport service types.
 2.  Section 4.1: "For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode." What is the meaning of this sentence? What message delivery capability is being referred here? Does this sentence infer that for IB transports, IB messages are allowed prior to enabling the iSER-assisted mode? The spec is not clear on how the iSCSI PDUs are transferred over IB connections prior to enabling iSER-assisted mode. Are iSCSI PDUs transferred over iSCSI/TCP/IPoIB connection or iSCSI/IB Reliable Connection prior to enabling iSER-assisted mode (I think it is the later)?
 3.  Section 5.1 has the following text "If the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementation (such as [IB]), the connection may need to be re-established in TCP capable mode.  (For InfiniBand this will require an [IPoIB] type connection.) " What is the implication of re-establishing connection in TCP capable mode for InfiniBand? Does it require tearing down the exisiting iSCSI/IB connection, establishing a new TCP/IPoIB connection, and performing iSCSI login without RDMAExtensions key on that TCP connection? The spec does not describe this behavior clearly.
 4.  Section 9.2: iSER header and iSER messages defined in Section 9.2 are not backward compatible with RFC 5046 defined iSER messages. I consider this as a significant issue. I suggest to add a single flag (or two flags) in iSCSI-control type PDU to indicate the presence of WBO and RBO by using one (or two) of the reserved bits. If this flag is set to 1, then the format is as defined in Figure 3 of this draft. If this flag is cleared, then the format of the iSER header for iSCSI control-type PDU is same as the one defined in RFC5046. I strongly suggest to keep the formats of iSER Hello and HelloReply Messages to be identical to the formats defined in RFC5046. These proposed modifications to the header formats in Section 9.2 make this draft backward compatible with RFC5046 when WBO and RBO are not used.
 5.  The headers in Sections 9.3 and 9.4 need to be reformatted if the change suggested in the previous bullet is made.
 6.  We should add an appendix for "InfiniBand Message Formats for iSER" because they are different from "iWARP Message Formats for iSER".
 7.  Solicited Event: RFC5046 describes the use of Send with Solicited Event (SendSE) and Send with Solicited Event and Invalidate (SendSEInv) Messages. This draft removes the use of SendSE and SendSEInv Messages. What was the rationale behind this removal? The draft should allow the use of SendSE and SendSEInv unless we have good reasons not to.



-----Original Message-----
From: [] On Behalf Of
Sent: Monday, November 21, 2011 11:32 AM
Subject: [storm] WG Last Call - iSER - through December 12
Importance: High

This message is to announce the STORM working group Last Call on the following document:

                  iSCSI Extensions for RDMA Specification

Last Call comments are due by Monday, December 12 at midnight Eastern time.

Please send all technical comments to the storm mailing list:
Editorial comments may be sent directly to the draft authors:

I'll even start this Working Group Last Call with an initial comment:

The iSER draft currently references RFC 3720 for iSCSI - that reference will need to be updated to at least the new consolidated iSCSI draft, and a reference to the iSCSI new features (SAM) draft should probably be added.

There's no need for a new version of the draft now, but a new version will be needed before the RFC publication request can be submitted.

Thanks in advance for the WG's prompt review. :-)

--David (storm WG co-chair)
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786        Mobile: +1 (978) 394-7754

storm mailing list