Re: [storm] iSER draft

[Mallikarjun Chadalapaka <[e-mail]>]

Hi Mike,

A few responses to your comments, :)

>[mk] The use of the Steering Address is up to the RCaP.

Understood.  But iSER layer has differing protocol requirements (see my
bullets 2-b-c, 3-b-c), keyed to RCaP semantics.  So IMHO, it would be useful
to formally spec these differences and relate them to an iSER-level protocol
parameter.

>[mk] Both the Steering Address and the Tagged Offset will be required.

Sure, I agree for control-type PDUs.  That's the reason I said (emphasis
added)  " Steering Address ("Address") or Tagged Offset ("Offset") is
required by the RCaP in the *RDMA requests*."  RDMA Reads/Writes generated
due to iSCSI Data-type PDUs contain only one 64-bit reference - for iWARP,
it is an Offset when used with ZB-VA; for IB, it's a full Virtual Address
(although I am not an IB-expert, that's what I understand so far)

>[mk] I don't think a new connection attribute is needed.  

While adding clarification text in several places in the draft might address
the issue that Tom pointed out, I think a formal iSCSI or iSER state
variable (either through text key negotiation or iSER Hello negotiation)
gives a hook for you to hang those semantics off of in a formal way - both
to spec, and validate wire protocol behavior for compliance later on.
E.g., let's say the attribute name is "Reference Type" with 0 - for Address,
and 1 for Offset, you could say "When the operational Reference Type is 1
for the connection, initiator iSER layer MUST.....".  Finally, this is
probably better modeled as an iSER Hello parameter rather than iSCSI text
key, to minimize layer "rototilling".  For the RCaP that does not have a
Hello negotiation (e.g. IB), you could specify in the spec that Reference
Type=0 is the default iSER behavior.

My 0.02.

Mallikarjun




>
>From: Michael Ko [mailto:[e-mail]] 
>Sent: Thursday, July 14, 2011 6:50 PM
>To: Mallikarjun Chadalapaka; Tom Talpey; 'Caitlin Bestler'
>Cc: [e-mail]
>Subject: Re: [storm] iSER draft
>
>Mallikarjun,
> 
>Nice to hear from you!
> 
>I have embedded my comments in your response.
> 
>Mike
>----- Original Message ----- 
>From: Mallikarjun Chadalapaka 
>To: 'Michael Ko' ; Tom Talpey ; 'Caitlin Bestler' 
>Cc: [e-mail] 
>Sent: Thursday, July 14, 2011 6:26 PM
>Subject: RE: [storm] iSER draft
>
>I suggest we have to be explicit about the responsibilities of each
protocol
>layer because the phrase "initiator side needs to translate the Tagged
>Offset" sounds vague to me.  
> 
>[mk] That statement is only meant to convey what would be done by the
initiator if it uses address for referencing the buffer rather than using
buffer offset to reference the buffer, and the Steering Address is not sent
to the target as currently defined in RFC 5046.  It does not appear in the
spec.
> 
>Also, it may be worth formalizing the notion of
>"if Virtual Address was used for this command" via a negotiated connection
>attribute. 
> 
>[mk] The use of the Steering Address is up to the RCaP.  For iWARP, the
Steering Address is set to zero.  By adding the explanation (as I indicated
in my previous note) on how the Tagged Offset is computed from the Steering
Address and the Buffer Offset, it is up to the RCaP to determine what they
want to do.
>
>If I followed this thread right, it seems like we are talking roughly about
>the following:
>
>1) A shared iSCSI/iSER-level connection attribute (connection-scoped text
>key?) that indicates whether Steering Address ("Address") or Tagged Offset
>("Offset") is required by the RCaP in the RDMA requests on the wire for
that
>connection.  How the RCaP API exposes this capability to its local iSER
>layer is implementation-dependent, and we do not need spec it.
> 
>[mk] Both the Steering Address and the Tagged Offset will be required.
When the Steering Address is set to zero, we have the iWARP mode of behavior
as defined in RFC 5046.
>
>2) If the connection attribute = "Offset:,
> 
>[mk] I don't think a new connection attribute is needed.  The RCaP just
needs to set the Steering Address to the appropriate value.  So for iWARP,
set the Steering Address to zero.
> 
>a) the semantic requirement on the RCaP layer on the initiator is
>that it treats the incoming RDMA memory references as offsets, and does the
>appropriate base+offset local translation for RDMA Reads and RDMA Writes.  
>b) The initiator iSER layer must only use ZB-VA, and must set the
>Steering Address to 0 in appropriate control-type PDUs. 
>c) The target iSER layer must assume that zero TO for the advertised
>STag points to the beginning of the initiator I/O Buffer in all the RDMA
>Writes and RDMA Reads that it issues (same as now).
>3) If the connection attribute = "Address":
>a)  the semantic requirement on the RCaP layer on the initiator is
>that it treats incoming RDMA memory references as complete Steering
>Addresses in RDMA Reads and RDMA Writes.  
>b) The initiator iSER layer must use non-ZB-VA, and must set the
>Steering Address to the RCaP-API-returned Virtual Address in appropriate
>control-type PDUs.
>c) The target iSER layer must assume that the Steering Address for
>the advertised STag points to the beginning of the initiator I/O Buffer,
and
>compute the Tagged Offset in all the RDMA Writes and RDMA Reads that it
>issues by adding the Steering Address to the Buffer Offset received from
the
>Data_Descriptor of Datamover API.
>
>
>Mallikarjun
>
>  
>
>
>
>
>
>
>From: [e-mail] [mailto:[e-mail]] On Behalf Of
>Michael Ko
>Sent: Thursday, July 14, 2011 1:53 PM
>To: Tom Talpey; Caitlin Bestler
>Cc: [e-mail]
>Subject: Re: [storm] iSER draft
>
>For clarity, I need to update in the next revision of the spec everywhere
>where "Tagged Offset", "Steering Tag", etc. are mentioned to specify how
the
>Tagged Offset is computed from the "Steering Address" and the buffer
>offset. 
>
>So for example, in the example you cited in section 7.3.5, the target
>computes the Tagged Offset using the Steering Address and the Buffer
>Offset. For iWARP, the Steering Address is zero, and so the Tagged Offset
>is the Buffer Offset in the SCSI Data-in PDU as defined in RFC 5046.
>
>Mike
>----- Original Message ----- 
>From: Tom Talpey 
>To: Michael Ko ; Caitlin Bestler 
>Cc: Alexander Nezhinsky ; [e-mail] 
>Sent: Thursday, July 14, 2011 1:16 PM
>Subject: RE: [storm] iSER draft
>
>Changing the name doesn't address the core issues here. There are no
>processing rules defined in the draft to support the arithmetic you
describe
>below, or even to use the Steering/Virtual Address.
>
>For example, section 7.3.5 states the target MUST use the Buffer Offset of
>the Data-In PDU as the Tagged Offset of an RDMA Write. Where does the new
>Steering Address in the control PDU header get folded into the Tagged
>Offset? And what protocol state would trigger the target to do so,
including
>perform the arithmetic? Same question for several other sections.
>
>
>
>From: Michael Ko [mailto:[e-mail]] 
>Sent: Thursday, July 14, 2011 2:39 PM
>To: Caitlin Bestler; Tom Talpey
>Cc: Alexander Nezhinsky; [e-mail]
>Subject: Re: [storm] iSER draft
>
>When "Virtual Address" is not used, as in iWARP, the initiator side needs
to
>translate the "Tagged Offset" which is just the offset into the buffer,
into
>a usable address by adding it to the starting base address of the buffer. 
>If the starting base address is communicated to the target side, as is done
>in some RCaPs, then the "Tagged Offset" is the usable address itself at the
>initiator where data is to be fetched or stored. Alex suggested that
>perhaps we can change the name "Virtual Address" to "Steering Address". 
>Then it can be defined in the glossary without tripping over the common
term
>"virtual address".
>
>Mike
>----- Original Message ----- 
>From: Caitlin Bestler 
>To: Tom Talpey 
>Cc: Alexander Nezhinsky ; Michael Ko ; [e-mail] 
>Sent: Wednesday, July 13, 2011 9:49 PM
>Subject: Re: [storm] iSER draft
>
>
>On Jul 13, 2011, at 8:12 AM, Tom Talpey wrote:
>
>> Yes, understood on the ZBVA/Infiniband issue. Regarding the VA term's
>presence in an earlier draft, I did not see it in RFC5046 and I did not
>review the expired draft, so consider these comments to be on the overall
>change, not the last revision.
>> 
>> Let me try a different approach to perhaps make myself clearer.
>> 
>> First, it seems to me that a "virtual address" has no real meaning to the
>network protocol. It begins, perhaps, as some value in an address space on
>the initiator, but it's not meaningful on the target in this way, it's only
>used to request that the remote RNIC perform a transfer to that originally
>registered remote address. So calling it a Virtual Address as an iSER
>protocol object seems, to me, an artificial and somewhat leading
convention.
>> 
>> Second, the Virtual Address goes out from the initiator to the target in
a
>Control PDU, but where does it come back? The RDMA Read or Write as
depicted
>in (xx) shows only a Tagged Offset. So, it's not clear what its protocol
>meaning is.
>> 
>> Third, I don't ever see a Tagged Buffer described by a fully qualified
>four-tuple. I see it appearing as either { Stag, TO, length } or { Stag,
VA,
>length }, depending on the addressing mode.
>> 
>> I think the non-ZBVA mode is really just a special case of the existing
>one, but where the meaning of TO has changed from a small offset to some
>other token, generated and managed only at the initiator. So, it seems
>artificial to define it as distinct, and document it as possessing some new
>properties. Isn't it just a Target Offset, still?
>> 
>> Tom.
>> 
>> 
>
>I agree, so far we have not seen a protocol justification for the need to
>add "Virtual Address" to the glossary as something distinct from Target
>Offset
>for the purpose of defining an IETF protocol.
>
>I am sympathetic to the interoperability issues raised, but I don't think
>those can justify something that has *no* justification in the protocol.
>
>If someone could site a class of implementation where there is a real need
>for this distinction in an iSER adapter, but as far as I can see the
>adapters have
>to be able to translate to TO *anyway* in order to use an RDMA Write or
RDMA
>Read.
>
>Local Interface compatibility with IB can make a *lot* of sense, but why
>does it have to make its way into the *wire* protocol?
>
>--
>Caitlin Bestler
>[e-mail]
>http://www.asomi.com/CaitlinBestlerResume.html
>
>