Re: [storm] WG Last Call - iSER - through December 12 - comments

"Hemal Shah" <> Tue, 13 December 2011 01:06 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id DEEC411E8082 for <>; Mon, 12 Dec 2011 17:06:19 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -5.299
X-Spam-Status: No, score=-5.299 tagged_above=-999 required=5 tests=[AWL=1.299, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id tlW6yQvlB5iy for <>; Mon, 12 Dec 2011 17:06:16 -0800 (PST)
Received: from ( []) by (Postfix) with ESMTP id 04D3D21F84C3 for <>; Mon, 12 Dec 2011 17:06:15 -0800 (PST)
Received: from [] by with ESMTP (Broadcom SMTP Relay (Email Firewall v6.3.2)); Mon, 12 Dec 2011 17:14:03 -0800
X-Server-Uuid: D3C04415-6FA8-4F2C-93C1-920E106A2031
Received: from ([]) by ([]) with mapi; Mon, 12 Dec 2011 17:06:09 -0800
From: "Hemal Shah" <>
To: "''" <>, "" <>
Date: Mon, 12 Dec 2011 17:06:07 -0800
Thread-Topic: WG Last Call - iSER - through December 12 - comments
Thread-Index: AcyQIZ9GrcM5E+JESbe7RIuFF0Mv9AYYaKUgBCRY0oA=
Message-ID: <>
References: <>
In-Reply-To: <>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
MIME-Version: 1.0
X-WSS-ID: 62F879513GG9066343-01-01
Content-Type: multipart/alternative; boundary=_000_76DBE161893C324BA6D4BB507EE4C38495ABAD8328IRVEXCHCCR02c_
Subject: Re: [storm] WG Last Call - iSER - through December 12 - comments
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 13 Dec 2011 01:06:20 -0000

David and Mike,

Below are my comments on the iSER draft.


1.      Section 1.1: iSCSI data-type PDU definition should clearly say at what layer the data transfer is happening. For example, the following sentence "An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection." should be rephrased (added text in red) as "An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer via RDMA operations at the iSER layer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection. I believe that the definition is already being revised based on David's comments.
2.      Section 1.1: The definition of iSCSI/iSER session needs to have further description. The current description is insufficient. Suggest to add the following to the definition "All connections of an iSCSI/iSER session are iSCSI/iSER connections."
3.      SNIC term is not defined but used in this draft. This needs to be addressed.
4.      Sections 2.4.3 and 2.4.4 use terms "sending side", "receiving side", "remote node", etc. It is confusing to use multiple terms (some of them are not defined in the draft) in these two sections. Suggest to use "Local Peer/Remote Peer" or "Data Source/Data Sink" terms consistently in these two sections.
5.      The use of term input qualifiers in Section 3 is misleading as we are actually referring to the input parameters for the operational primitives. Suggest to replace term "input qualifier" with "input parameter" in Section 3.
6.      Section 5.1.3: In the following sentence the second "iSER-ORD" should be replaced with "ORD". "Upon receiving the iSER Hello Message, the iSER Layer at the target MUST set the iSER-ORD value to the minimum of the iSER-ORD value at the target and the iSER-IRD value declared by the initiator."
7.      Section 9.2: Do WSV and RSV flags indicate the validity of WBO and RBO fields? The description of these flags should state it.
8.      Section 15 should be titled as "iWARP Message Formats for iSER".
9.      Sections 15.3-15.5 should be formatted as iWARP messages.
10.     I do not understand the reason behind removing the iWARP Message Format for RDMA Read Request, iWARP Message Format for Solicited SCSI Write Data, and iWARP Message Format for SCSI Response PDU in Section 15. They should be kept in Section 15.

11.     According to the iSCSI spec, an iSCSI connection has an underlying TCP connection. Based on that, an iSER-assisted iSCSI connection requires an underlying TCP connection to be present. For iWARP, an iSER-assisted iSCSI connection has an underlying TCP connection. For iSER-assisted iSCSI connection over InfiniBand transports, there is no underlying TCP connection. In this case, what does an iSER-assisted iSCSI connection mean? What IB transport service types are used for the iSER-assisted iSCSI connection? The draft is not clear on what an iSER-assisted iSCSI connection mean for the InfiniBand transports. The draft needs to clearly define iSER-assisted iSCSI connection for InfiniBand transports and its mapping over underlying InfiniBand transport connections and transport service types.
12.     Section 4.1: "For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode." What is the meaning of this sentence? What message delivery capability is being referred here? Does this sentence infer that for IB transports, IB messages are allowed prior to enabling the iSER-assisted mode? The spec is not clear on how the iSCSI PDUs are transferred over IB connections prior to enabling iSER-assisted mode. Are iSCSI PDUs transferred over iSCSI/TCP/IPoIB connection or iSCSI/IB Reliable Connection prior to enabling iSER-assisted mode (I think it is the later)?
13.     Section 5.1 has the following text "If the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementation (such as [IB]), the connection may need to be re-established in TCP capable mode.  (For InfiniBand this will require an [IPoIB] type connection.) " What is the implication of re-establishing connection in TCP capable mode for InfiniBand? Does it require tearing down the exisiting iSCSI/IB connection, establishing a new TCP/IPoIB connection, and performing iSCSI login without RDMAExtensions key on that TCP connection? The spec does not describe this behavior clearly.
14.     Section 9.2: iSER header and iSER messages defined in Section 9.2 are not backward compatible with RFC 5046 defined iSER messages. I consider this as a significant issue. I suggest to add a single flag (or two flags) in iSCSI-control type PDU to indicate the presence of WBO and RBO by using one (or two) of the reserved bits. If this flag is set to 1, then the format is as defined in Figure 3 of this draft. If this flag is cleared, then the format of the iSER header for iSCSI control-type PDU is same as the one defined in RFC5046. I strongly suggest to keep the formats of iSER Hello and HelloReply Messages to be identical to the formats defined in RFC5046. These proposed modifications to the header formats in Section 9.2 make this draft backward compatible with RFC5046 when WBO and RBO are not used.
15.     The headers in Sections 9.3 and 9.4 need to be reformatted if the change suggested in the previous bullet is made.
16.     We should add an appendix for "InfiniBand Message Formats for iSER" because they are different from "iWARP Message Formats for iSER".
17.     Solicited Event: RFC5046 describes the use of Send with Solicited Event (SendSE) and Send with Solicited Event and Invalidate (SendSEInv) Messages. This draft removes the use of SendSE and SendSEInv Messages. What was the rationale behind this removal? The draft should allow the use of SendSE and SendSEInv unless we have good reasons not to.



-----Original Message-----
From: [] On Behalf Of
Sent: Monday, November 21, 2011 11:32 AM
Subject: [storm] WG Last Call - iSER - through December 12
Importance: High

This message is to announce the STORM working group Last Call on the following document:

                  iSCSI Extensions for RDMA Specification

Last Call comments are due by Monday, December 12 at midnight Eastern time.

Please send all technical comments to the storm mailing list:
Editorial comments may be sent directly to the draft authors:

I'll even start this Working Group Last Call with an initial comment:

The iSER draft currently references RFC 3720 for iSCSI - that reference will need to be updated to at least the new consolidated iSCSI draft, and a reference to the iSCSI new features (SAM) draft should probably be added.

There's no need for a new version of the draft now, but a new version will be needed before the RFC publication request can be submitted.

Thanks in advance for the WG's prompt review. :-)

--David (storm WG co-chair)
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786        Mobile: +1 (978) 394-7754

storm mailing list