[storm] iSER - problem with unsolicited NOP-IN right after final Login Response

Alexander Nezhinsky <nezhinsky@gmail.com> Mon, 21 May 2012 13:15 UTC

Return-Path: <nezhinsky@gmail.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id E1D7221F85F9 for <storm@ietfa.amsl.com>; Mon, 21 May 2012 06:15:35 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.184
X-Spam-Status: No, score=-1.184 tagged_above=-999 required=5 tests=[BAYES_40=-0.185, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id RNg4f3HBykMc for <storm@ietfa.amsl.com>; Mon, 21 May 2012 06:15:35 -0700 (PDT)
Received: from mail-yw0-f44.google.com (mail-yw0-f44.google.com []) by ietfa.amsl.com (Postfix) with ESMTP id 0865821F85F7 for <storm@ietf.org>; Mon, 21 May 2012 06:15:34 -0700 (PDT)
Received: by yhq56 with SMTP id 56so5150541yhq.31 for <storm@ietf.org>; Mon, 21 May 2012 06:15:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=XYiwemVcOWbda/sC0ZJb9jhLIXvpZ6zb8XvncqdLoJ4=; b=Zm0HEZ0J7AzJRDpR10XArMbUP/GanEVJwFLAYiNay5/nr41ETWsTCue8YKp6PB/xhT Gj3uJcAbtc6h44f+OHlZryBCqLSL1rjBJ7O+kGOcoOnyWXR2NR8avssRLy9eClV8ZtDT 1457oT7cKKBMa4aNs3FxfrtOQ5ZYiCLvQB1AMAlmNM4fUzn29gBt86aztdS7yAHq8e/S fWgjJaZU20R3y6GLwVfn5Ca+4oqAwtk/AXg/DOTmtdKqtGoPNW1UoZgipfxs8ySircai DhBZ0k5dmn+YEUcKWvNn7GUZnK8Rq7RiroZhIPL3Jsi4ISL7wXpp/b3y1AecDqSHn1gk Zjdw==
MIME-Version: 1.0
Received: by with SMTP id p9mr8640159icp.43.1337606134026; Mon, 21 May 2012 06:15:34 -0700 (PDT)
Received: by with HTTP; Mon, 21 May 2012 06:15:34 -0700 (PDT)
Date: Mon, 21 May 2012 16:15:34 +0300
Message-ID: <CAEkHY=egdu9RNYojRq2jNSe1205VZxTa8fizM-sZaA8aGh_FNg@mail.gmail.com>
From: Alexander Nezhinsky <nezhinsky@gmail.com>
To: "storm@ietf.org" <storm@ietf.org>, "david.black@emc.com" <david.black@emc.com>, mkosjc@gmail.com
Content-Type: multipart/alternative; boundary=20cf30244ca377d08a04c08bb24a
Cc: Or Gerlitz <ogerlitz@mellanox.com>, Mike Christie <michaelc@cs.wisc.edu>
Subject: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 21 May 2012 13:15:36 -0000


I understand that it is a bad timing for sending this kind of mail, now
that iSER draft was submitted,
but actually we still have a small problem.

It is related to the final Login Response handling and the transition to
Full-Featured phase on the initiator side in
Infiniband setups.

When the target receives the final Login Request it send the final Login
Response and from its perspective
the connection is now in Full Featured Phase (assuming that it agreed to
transition in the Login Response being sent).

This means that the target is ready to accept SCSI commands, Text Requests
etc. sent by the initiator.
It also means that the target is eligible to send some unsolicited
PDUs, notably unsolicited NOP-INs.

With IB sending NOP-IN periodically is the easiest (an almost only
feasible) way to determine closed connections
reliably, because this kind of error is delivered to user only in response
to a previously initiated TX operation.

This leaves the initiator in a dubious position. It posts its RX buffers
for that connection only when the final
Login Response arrives. But during that time (after the target had sent the
Last Login Response but before
the Full Featured phase related RX-buffers are posted on the initiator
side) the target may send
the first NOP-IN as it considers the connection in Full Featured phase
already and NumOfUnsolicited PDUs
accounting for NOP-INs has been negotiated to a non-zero value.

If the initiator works with a single RX-buffer posted during the entire
login phase (which is a logical thing to do
judging by the login exchange protocol) then an error occurs, as no buffers
are posted when the NOP-IN arrives
and the connection is shut down.

Posting a single extra buffer before sending the last Login Request only
alleviates the problem. Although this
often solves it in practical terms (as the target most probably sends the
next NOP-IN only after some timeout
period measuring seconds or hundreds of milliseconds), it does not solves
it in terms of protocol completeness,
as the target MAY theoretically send more than one NOP-IN until FF buffers
are posted.

This issue was encountered recently in linux iscsi/iser initiator and the
above solution has been applied to solve
it against the existing target implementation (STGT), but the initiator
remains exposed to this kind of errors.

The solution is actually quite simple (theoretically) - if we bring back
the requirement for iSER Hello exchange
then the iSER assisted Full Featured phase does not commence until
HelloReply PDU arrives at the target
and the initiator has a definitive point in time when it can safely post
its RX buffers - after the final LoginResponse
returns but before it sends iSER Hello PDU.

In practical terms it means that iSER Hello support requirement should be
brought back to spec, which is a hassle.

Should we decide on this now?


P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux
iSCSI and iSER initiator for raising the issue.