Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response

Michael Ko <> Mon, 21 May 2012 14:22 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 0376421F85B5 for <>; Mon, 21 May 2012 07:22:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -3.598
X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id f47zcQSqL069 for <>; Mon, 21 May 2012 07:22:36 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 101F021F8567 for <>; Mon, 21 May 2012 07:22:35 -0700 (PDT)
Received: by qcsq13 with SMTP id q13so3981951qcs.31 for <>; Mon, 21 May 2012 07:22:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=DRGJxtf/F8PMXQbis0fbkQXSHfQC3e6TCkzUE6PHW7M=; b=E3CmPnVZ08wCH/n1c8L/f9ubfixk+yjPNk6kIP81NXMNUwtBdHR1nHUexMUCVDkcQs lmQjo3ZsF3UkdWZXN+/b2EW6iT/u7VsepwsVaXHLzhELjJBblSUDpK6a+HQ0wSdDiDdF 3O54yXubmhlb+qwH3a/OCbWT7tIBGuWLl/s4SDOGi3D4PWhfbyyA/UVLmYvcQIYeuTH4 NloXAf2wIvr/WgJZA7iaIdGV17sthKaqZ914ScMyMu5hS3/FRH1EdlbVIiqFAAMN/y41 1fB7lv/zsPEGJPZYGcQWjX11oJPjxnJI85J57xURZHq/hM449zyDj8gc7DvQemC2dONG mZiQ==
MIME-Version: 1.0
Received: by with SMTP id cl19mr38401157qab.52.1337610155476; Mon, 21 May 2012 07:22:35 -0700 (PDT)
Received: by with HTTP; Mon, 21 May 2012 07:22:35 -0700 (PDT)
In-Reply-To: <>
References: <>
Date: Mon, 21 May 2012 07:22:35 -0700
Message-ID: <>
From: Michael Ko <>
To: Alexander Nezhinsky <>
Content-Type: multipart/alternative; boundary=20cf302d4e3c2a478404c08ca2ca
Cc: Mike Christie <>, Or Gerlitz <>, "" <>
Subject: Re: [storm] iSER - problem with unsolicited NOP-IN right after final Login Response
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 21 May 2012 14:22:39 -0000


The iSER Hello support has never been removed in the latest spec.  Only its
use is made optional.  So during login negotiation, just negotiate
iSERHelloRequired to Yes.


On Mon, May 21, 2012 at 6:15 AM, Alexander Nezhinsky <>wrote;wrote:

> Hi
> I understand that it is a bad timing for sending this kind of mail, now
> that iSER draft was submitted,
> but actually we still have a small problem.
> It is related to the final Login Response handling and the transition to
> Full-Featured phase on the initiator side in
> Infiniband setups.
> When the target receives the final Login Request it send the final Login
> Response and from its perspective
> the connection is now in Full Featured Phase (assuming that it agreed to
> transition in the Login Response being sent).
> This means that the target is ready to accept SCSI commands, Text Requests
> etc. sent by the initiator.
> It also means that the target is eligible to send some unsolicited
> PDUs, notably unsolicited NOP-INs.
> With IB sending NOP-IN periodically is the easiest (an almost only
> feasible) way to determine closed connections
> reliably, because this kind of error is delivered to user only in response
> to a previously initiated TX operation.
> This leaves the initiator in a dubious position. It posts its RX buffers
> for that connection only when the final
> Login Response arrives. But during that time (after the target had sent
> the Last Login Response but before
> the Full Featured phase related RX-buffers are posted on the initiator
> side) the target may send
> the first NOP-IN as it considers the connection in Full Featured phase
> already and NumOfUnsolicited PDUs
> accounting for NOP-INs has been negotiated to a non-zero value.
> If the initiator works with a single RX-buffer posted during the entire
> login phase (which is a logical thing to do
> judging by the login exchange protocol) then an error occurs, as no
> buffers are posted when the NOP-IN arrives
> and the connection is shut down.
> Posting a single extra buffer before sending the last Login Request only
> alleviates the problem. Although this
> often solves it in practical terms (as the target most probably sends the
> next NOP-IN only after some timeout
> period measuring seconds or hundreds of milliseconds), it does not solves
> it in terms of protocol completeness,
> as the target MAY theoretically send more than one NOP-IN until FF buffers
> are posted.
> This issue was encountered recently in linux iscsi/iser initiator and the
> above solution has been applied to solve
> it against the existing target implementation (STGT), but the initiator
> remains exposed to this kind of errors.
> The solution is actually quite simple (theoretically) - if we bring back
> the requirement for iSER Hello exchange
> then the iSER assisted Full Featured phase does not commence until
> HelloReply PDU arrives at the target
> and the initiator has a definitive point in time when it can safely post
> its RX buffers - after the final LoginResponse
> returns but before it sends iSER Hello PDU.
> In practical terms it means that iSER Hello support requirement should be
> brought back to spec, which is a hassle.
> Should we decide on this now?
> Alexander
> P.S. : Thanx to Mike Christie and Or Gerlitz, the maintainers of linux
> iSCSI and iSER initiator for raising the issue.