Re: [storm] iSER draft

Michael Ko <Michael@huaweisymantec.com> Fri, 15 July 2011 01:50 UTC

Return-Path: <Michael@huaweisymantec.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8D03721F87A2 for <storm@ietfa.amsl.com>; Thu, 14 Jul 2011 18:50:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.877
X-Spam-Level:
X-Spam-Status: No, score=-1.877 tagged_above=-999 required=5 tests=[AWL=0.121, BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_46=0.6]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dozTxJ+z8itc for <storm@ietfa.amsl.com>; Thu, 14 Jul 2011 18:50:29 -0700 (PDT)
Received: from mta2.huaweisymantec.com (mta2.huaweisymantec.com [218.17.155.15]) by ietfa.amsl.com (Postfix) with ESMTP id 33D7121F879F for <storm@ietf.org>; Thu, 14 Jul 2011 18:50:24 -0700 (PDT)
MIME-version: 1.0
Content-type: multipart/alternative; boundary="Boundary_(ID_rY2at5EF9D90x9YlgSnn5A)"
Received: from hstml01-in.huaweisymantec.com ([172.26.3.42]) by hstga02-in.huaweisymantec.com (Sun Java(tm) System Messaging Server 6.3-8.03 (built Apr 24 2009; 32bit)) with ESMTP id <0LOC00MM1QFX9LA0@hstga02-in.huaweisymantec.com> for storm@ietf.org; Fri, 15 Jul 2011 09:50:22 +0800 (CST)
Received: from m90003900a ([69.199.248.19]) by hstml01-in.huaweisymantec.com (Sun Java(tm) System Messaging Server 6.3-8.03 (built Apr 24 2009; 32bit)) with ESMTPA id <0LOC008O3QGWS610@hstml01-in.huaweisymantec.com> for storm@ietf.org; Fri, 15 Jul 2011 09:51:03 +0800 (CST)
Message-id: <D74343B13CC640229AB1AFCDEE9F9883@china.huawei.com>
From: Michael Ko <Michael@huaweisymantec.com>
To: Mallikarjun Chadalapaka <cbm@chadalapaka.com>, Tom Talpey <ttalpey@microsoft.com>, 'Caitlin Bestler' <cait@asomi.com>
References: <BANLkTi=GVHUBdsZiwsRXxETd_rU=F8f85w@mail.gmail.com> <F83812DF4B59B9499C1BC978336D91745EDE207F@TK5EX14MBXC118.redmond.corp.microsoft.com> <971779FD6B314A40B1A3658E521C0D4A@china.huawei.com> <BANLkTik0rG2=orD0S0L3s62gBSJsLapA+w@mail.gmail.com> <F83812DF4B59B9499C1BC978336D91745EE0279D@TK5EX14MBXC111.redmond.corp.microsoft.com> <B443B9D1-754C-4D74-B43A-858A6031850A@asomi.com> <1680F0051D794095A86AF16343CAD400@china.huawei.com> <F83812DF4B59B9499C1BC978336D91745EE039F9@TK5EX14MBXC111.redmond.corp.microsoft.com> <70CC5C8CCDD74EA0A9CEFF6DDC8145ED@china.huawei.com> <SNT131-ds105B54D3D6E874C31C27A9A0490@phx.gbl>
Date: Thu, 14 Jul 2011 18:50:12 -0700
X-Priority: 3
X-MSMail-priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5931
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.6109
Cc: storm@ietf.org
Subject: Re: [storm] iSER draft
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Jul 2011 01:50:33 -0000

Mallikarjun,

Nice to hear from you!

I have embedded my comments in your response.

Mike
  ----- Original Message ----- 
  From: Mallikarjun Chadalapaka 
  To: 'Michael Ko' ; Tom Talpey ; 'Caitlin Bestler' 
  Cc: storm@ietf.org 
  Sent: Thursday, July 14, 2011 6:26 PM
  Subject: RE: [storm] iSER draft


  I suggest we have to be explicit about the responsibilities of each protocol
  layer because the phrase "initiator side needs to translate the Tagged
  Offset" sounds vague to me.  

  [mk] That statement is only meant to convey what would be done by the initiator if it uses address for referencing the buffer rather than using buffer offset to reference the buffer, and the Steering Address is not sent to the target as currently defined in RFC 5046.  It does not appear in the spec.

  Also, it may be worth formalizing the notion of
  "if Virtual Address was used for this command" via a negotiated connection
  attribute. 

  [mk] The use of the Steering Address is up to the RCaP.  For iWARP, the Steering Address is set to zero.  By adding the explanation (as I indicated in my previous note) on how the Tagged Offset is computed from the Steering Address and the Buffer Offset, it is up to the RCaP to determine what they want to do.

  If I followed this thread right, it seems like we are talking roughly about
  the following:

  1) A shared iSCSI/iSER-level connection attribute (connection-scoped text
  key?) that indicates whether Steering Address ("Address") or Tagged Offset
  ("Offset") is required by the RCaP in the RDMA requests on the wire for that
  connection.  How the RCaP API exposes this capability to its local iSER
  layer is implementation-dependent, and we do not need spec it.

  [mk] Both the Steering Address and the Tagged Offset will be required.  When the Steering Address is set to zero, we have the iWARP mode of behavior as defined in RFC 5046.

  2) If the connection attribute = "Offset:,

  [mk] I don't think a new connection attribute is needed.  The RCaP just needs to set the Steering Address to the appropriate value.  So for iWARP, set the Steering Address to zero.
   
  a) the semantic requirement on the RCaP layer on the initiator is
  that it treats the incoming RDMA memory references as offsets, and does the
  appropriate base+offset local translation for RDMA Reads and RDMA Writes.  
  b) The initiator iSER layer must only use ZB-VA, and must set the
  Steering Address to 0 in appropriate control-type PDUs. 
  c) The target iSER layer must assume that zero TO for the advertised
  STag points to the beginning of the initiator I/O Buffer in all the RDMA
  Writes and RDMA Reads that it issues (same as now).

  3) If the connection attribute = "Address":
  a)  the semantic requirement on the RCaP layer on the initiator is
  that it treats incoming RDMA memory references as complete Steering
  Addresses in RDMA Reads and RDMA Writes.  
  b) The initiator iSER layer must use non-ZB-VA, and must set the
  Steering Address to the RCaP-API-returned Virtual Address in appropriate
  control-type PDUs.
  c) The target iSER layer must assume that the Steering Address for
  the advertised STag points to the beginning of the initiator I/O Buffer, and
  compute the Tagged Offset in all the RDMA Writes and RDMA Reads that it
  issues by adding the Steering Address to the Buffer Offset received from the
  Data_Descriptor of Datamover API.


  Mallikarjun

    






  From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On Behalf Of
  Michael Ko
  Sent: Thursday, July 14, 2011 1:53 PM
  To: Tom Talpey; Caitlin Bestler
  Cc: storm@ietf.org
  Subject: Re: [storm] iSER draft

  For clarity, I need to update in the next revision of the spec everywhere
  where "Tagged Offset", "Steering Tag", etc. are mentioned to specify how the
  Tagged Offset is computed from the "Steering Address" and the buffer
  offset. 

  So for example, in the example you cited in section 7.3.5, the target
  computes the Tagged Offset using the Steering Address and the Buffer
  Offset. For iWARP, the Steering Address is zero, and so the Tagged Offset
  is the Buffer Offset in the SCSI Data-in PDU as defined in RFC 5046.

  Mike
  ----- Original Message ----- 
  From: Tom Talpey 
  To: Michael Ko ; Caitlin Bestler 
  Cc: Alexander Nezhinsky ; storm@ietf.org 
  Sent: Thursday, July 14, 2011 1:16 PM
  Subject: RE: [storm] iSER draft

  Changing the name doesn't address the core issues here. There are no
  processing rules defined in the draft to support the arithmetic you describe
  below, or even to use the Steering/Virtual Address.

  For example, section 7.3.5 states the target MUST use the Buffer Offset of
  the Data-In PDU as the Tagged Offset of an RDMA Write. Where does the new
  Steering Address in the control PDU header get folded into the Tagged
  Offset? And what protocol state would trigger the target to do so, including
  perform the arithmetic? Same question for several other sections.



  From: Michael Ko [mailto:Michael@huaweisymantec.com] 
  Sent: Thursday, July 14, 2011 2:39 PM
  To: Caitlin Bestler; Tom Talpey
  Cc: Alexander Nezhinsky; storm@ietf.org
  Subject: Re: [storm] iSER draft

  When "Virtual Address" is not used, as in iWARP, the initiator side needs to
  translate the "Tagged Offset" which is just the offset into the buffer, into
  a usable address by adding it to the starting base address of the buffer. 
  If the starting base address is communicated to the target side, as is done
  in some RCaPs, then the "Tagged Offset" is the usable address itself at the
  initiator where data is to be fetched or stored. Alex suggested that
  perhaps we can change the name "Virtual Address" to "Steering Address". 
  Then it can be defined in the glossary without tripping over the common term
  "virtual address".

  Mike
  ----- Original Message ----- 
  From: Caitlin Bestler 
  To: Tom Talpey 
  Cc: Alexander Nezhinsky ; Michael Ko ; storm@ietf.org 
  Sent: Wednesday, July 13, 2011 9:49 PM
  Subject: Re: [storm] iSER draft


  On Jul 13, 2011, at 8:12 AM, Tom Talpey wrote:

  > Yes, understood on the ZBVA/Infiniband issue. Regarding the VA term's
  presence in an earlier draft, I did not see it in RFC5046 and I did not
  review the expired draft, so consider these comments to be on the overall
  change, not the last revision.
  > 
  > Let me try a different approach to perhaps make myself clearer.
  > 
  > First, it seems to me that a "virtual address" has no real meaning to the
  network protocol. It begins, perhaps, as some value in an address space on
  the initiator, but it's not meaningful on the target in this way, it's only
  used to request that the remote RNIC perform a transfer to that originally
  registered remote address. So calling it a Virtual Address as an iSER
  protocol object seems, to me, an artificial and somewhat leading convention.
  > 
  > Second, the Virtual Address goes out from the initiator to the target in a
  Control PDU, but where does it come back? The RDMA Read or Write as depicted
  in (xx) shows only a Tagged Offset. So, it's not clear what its protocol
  meaning is.
  > 
  > Third, I don't ever see a Tagged Buffer described by a fully qualified
  four-tuple. I see it appearing as either { Stag, TO, length } or { Stag, VA,
  length }, depending on the addressing mode.
  > 
  > I think the non-ZBVA mode is really just a special case of the existing
  one, but where the meaning of TO has changed from a small offset to some
  other token, generated and managed only at the initiator. So, it seems
  artificial to define it as distinct, and document it as possessing some new
  properties. Isn't it just a Target Offset, still?
  > 
  > Tom.
  > 
  > 

  I agree, so far we have not seen a protocol justification for the need to
  add "Virtual Address" to the glossary as something distinct from Target
  Offset
  for the purpose of defining an IETF protocol.

  I am sympathetic to the interoperability issues raised, but I don't think
  those can justify something that has *no* justification in the protocol.

  If someone could site a class of implementation where there is a real need
  for this distinction in an iSER adapter, but as far as I can see the
  adapters have
  to be able to translate to TO *anyway* in order to use an RDMA Write or RDMA
  Read.

  Local Interface compatibility with IB can make a *lot* of sense, but why
  does it have to make its way into the *wire* protocol?

  --
  Caitlin Bestler
  cait@asomi.com
  http://www.asomi.com/CaitlinBestlerResume.html