Re: [storm] iSER update request from Bob Russell

[<[e-mail]>]

Paul,
 
[WG chair hat on this time]
 
We may be looking at an old "rough consensus" that iSER ought to be able
to fall back to plain iSCSI if needed vs. current "running code" that
doesn't do that (and at least on IB probably never will).  If lack of
fallback "ain't broken" in practice in iSER "running code", that may
be acceptable.  If this is the case, then here (and for any other case
in which all the implementations diverged from the spec), the new draft
will need to be very clear that the spec is being revised incompatibly
*because* all the implementations looked at the old RFC and did
something
different, and it's interoperability with those implementations that
matters.

Thanks,
--David


 


________________________________

	From: Paul Koning [mailto:[e-mail]] 
	Sent: Friday, October 16, 2009 8:50 PM
	To: Black, David; [e-mail]; [e-mail]
	Subject: RE: [storm] iSER update request from Bob Russell
	
	

	I thought the intent of the "start in normal mode" rule was to
allow interop with non-iSER iSCSI implementations.  If that is not
required - i.e., attempts for an iSER initiator to talk to a plain iSCSI
target, or vice versa, are a configuration error and will just fail -
then the change seems ok.  If that's not acceptable, then the spec is
right and the implementations are wrong.

	 

	Either that or I'm confused about what this is all about.

	 

	                paul 

	 

	From: [e-mail] [mailto:[e-mail]] On
Behalf Of [e-mail]
	Sent: Friday, October 16, 2009 8:38 PM
	To: [e-mail]; [e-mail]
	Subject: Re: [storm] iSER update request from Bob Russell

	 

	There's a lot of material in here.  With WG chair hat off, my

	opinions are:

	 

	- If all the implementations start the connection immediately

	    in RDMA mode, then the RFC should be revised to reflect that

	    "running code".  I do hope the initial MPA Request and Reply

	    frames are being used.

	- If all the implementations are using the Expanded iSER header,

	    then the RFC ought to be revised to reflect "running code".

	- ZBVA was originally left out of the RFC because it was thought

	    to be IB-specific.  If that concept also applies to iWARP,

	    it belongs in the new iSER RFC ("running code" again).

	Thanks,
	--David

	 

		 

________________________________

		From: [e-mail]
[mailto:[e-mail]] On Behalf Of Mike Ko
		Sent: Thursday, August 20, 2009 6:25 PM
		To: STORM
		Subject: [storm] iSER update request from Bob Russell

		On March 22, 2009, Bob Russell posted the following at
the IPS mailing list.  I have embedded my responses below prefixed by
<mk>.

		 

		Mike

		 

		-----  from Bob Russell  -----

		 

		There are 2 issues I would like to suggest for
discussion at the BOF
		meeting later this week.  Both have to do with the iSER
spec, RFC 5046.

		 

		1. At the present time, as far as I know, no existing
hardware,
		   neither Infiniband nor iWARP, is capable of opening a
connection
		   in "normal" TCP mode and then transitioning it into
zero-copy mode.
		   Unfortunately, the iSER spec requires that.
		   Can't we just replace that part of the iSER spec?
		   Otherwise, all hardware and all implementations are
non-standard.

		 

		<mk> When I wrote the Supplement to Infiniband
Architecture Specification Annex A12 (Support for iSCSI Extensions for
RDMA), there was no mention of transitioning the connection from TCP
mode to RDMA mode.  I can update update RFC 5046 to remove this
requirement if that is the consensus of the group.

		 

		2. The OFED stack is used to access both Infiniband and
iWARP hardware.
		   This software requires 2 extra 64-bit fields for
addressing
		   on both Infiniband and iWARP hardware, but these
fields
		   are not allowed for in the current iSER Header
Format.
		   Can't we just add those extra fields to the iSER
spec?
		   If someday some other implementation doesn't need
those
		   fields, they can be just set to 0 (which is what is
implied by
		   the current iSER standard anyway).  Again, by not
doing this,
		   all implementations are non-standard.

		 

		<mk> I assume you are suggesting that we should take the
"Expanded iSER Header for Supporting Virtual Address" as defined in
table 4 of Annex A12 and update RFC 5046 accordingly.  Again, I am fine
with this if that is the consensus of the group.

		 

		In other words, I'm suggesting that we consider
replacing the relevant
		parts of the current iSER specs with the current OFED
specs on these
		2 issues.

		 

		Thanks for your consideration,
		Bob Russell

		 

		Note: The following (old) posting by Mike Ko states that
the
		extra header fields are needed only by IB, not by IETF
		(i.e., iWARP), because IB uses nZBVA, whereas iWARP uses
ZBVA.

		 

		But are there any IETF/iWARP implementations out there
that actually
		use ZBVA with iWARP RNICs?  (I don't mean software
simulations of
		the iWARP protocol.)  We have built an iSER
implementation that
		uses the OFED stack to access both IB and iWARP
hardware, and for
		both of them we need to use the extra iSER header fields
(nZBVA).
		Perhaps this is an issue with the design of the OFED
stack, which
		was built primarily to access IB hardware and therefore
reflects
		the needs of the IB hardware.  But we found that the
only way to
		access iWARP hardware via the OFED stack was to used the
expanded
		(nZBVA) iSER header (and to use a meaningful value in
the extra field,
		NOT to just set it to zero).

		 

		In any case, rather than have 2 different versions of
the iSER header,
		it would be better to have just one, regardless of the
underlying
		technology involved (after all, isn't that what a
standard is for??).
		This is especially relevant when using the OFED stack,
because,
		as we have demonstrated, software built on top of the
OFED stack can
		(AND SHOULD!) be able to run with EITHER IB or iWARP
hardware,
		with NO change to that software.  Having 2 different
iSER headers
		does NOT make that possible!

		 

		 

		 

		    2008/4/15 Mike Ko <mako at almaden.ibm.com>:

		 

		    VA is a concept introduced in an Infiniband annex to
support iSER. It
		    appears in the expanded iSER header for Infiniband
use only to support the
		    non-Zero Based Virtual Address (non-ZBVA) used in
Infiniband vs the ZBVA
		    used in IETF.

		 

		Mike - could you please put me in contact with someone
who has actually
		implemented iSER on top of IETF/iWARP hardware NICs
using ZBVA?

		 

		<mk> Perhaps vendors who have built iSER stacks can
comment on this.

		
		    "The DataDescriptorOut describes the I/O buffer
starting with the immediate
		    unsolicited data (if any), followed by the
non-immediate unsolicited data
		    (if any) and solicited data." If non-ZBVA mode is
used, then VA points to
		    the beginning of this buffer. So in your example,
the VA field in the
		    expanded iSER header will be zero. Note that for
IETF, ZBVA is assumed and
		    there is no provision to specify a different VA in
the iSER header.

		 

		Mike - I believe this VA field in the expanded iSER
header is almost
		NEVER zero -- it is always an actual virtual address.

		 

		<mk> So if the OFED iSER stack is based on Annex 12,
then it already has the means to select ZBVA or non-ZBVA as specified in
the "iSER CM REQ Message Private Data Format" of table 2.  So rather
than having to change the OFED implementation and update the Infiniband
Annex, I suggest that we leave the ZBVA/non-ZBVA option alone even
though ZBVA is never used as you said.

		
		    Tagged offset (TO) refers to the offset within a
tagged buffer in RDMA Write
		    and RDMA Read Request Messages. When sending
non-immediate unsolicited
		    data, Send Message types are used and the TO field
is not present. Instead,
		    the buffer offset is appropriately represented by
the Buffer Offset field in
		    the SCSI Data-Out PDU. Note that Tagged Offset is
not the same as write VA
		    and it does not appear in the iSER header.

		 

		    Mike