From: pat_thaler@agilent.com
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Date: Thu, 31 Jul 2003 13:42:00 -0600
Message-ID: <1BEBA5E8600DD4119A50009027AF54A0132E0465@axcs04.cos.agilent.com>
Thread-Topic: iSCSI/iWARP drafts and flow control
Thread-Index: AcNXjWQfPYoXpsOAEdesjwCQJ6pbUAAA9FVw
To: <cait@asomi.com>, <pat_thaler@agilent.com>
Cc: <cbm@rose.hp.com>, <ips@ece.cmu.edu>, <rddp@ietf.org>
Content-Transfer-Encoding: quoted-printable
Subject: [rddp] RE: iSCSI/iWARP drafts and flow control
Sender: rddp-admin@ietf.org
Errors-To: rddp-admin@ietf.org
Precedence: bulk
Content-Transfer-Encoding: quoted-printable
Content-Transfer-Encoding: quoted-printable

All, sorry for the empty reply - I'm not sure how that happened.

Caitlin,

You asked some questions about how the other messages are flow =
controlled in iSCSI over TCP. The answer is that they aren't flow =
controlled. If iSCSI gets a PDU it cannot handle, it drops it and there =
are provisions to trigger it to be resent depending on the kind of =
recovery level supported. The only control for PDUs to the target is on =
non-immediate commands (both SCSI Command and Task Management Function =
Requeset PDUs). Note that when unsolicited non-immediate data is =
permitted, iSCSI allows the command to generate a command PDU plus an =
unknown number of SCSI Data-out PDUs to carry the unsolicted data. For =
iSER, we require that the unsolicted SCSI Data-out PDUs be full when =
there is enough unsolicted data to fill them (and we created a key to =
negotiate that size). Therefore, when operating over iSER the target =
does know the maximum number of PDUs that the initiator might send per =
SCSI command.

There is no deadlock in existing iSCSI because there is no flow control =
on NOP-In and the target can always send a NOP-In to advance MaxCmdSN.

To summarize, in current iSCSI, each opening in CmdSN window allows from =
1 to ? PDUs while in iSCSI over iSER, each opening in CmdSN window =
allows from 1 to n PDUs where n is the amount of unsolicited data =
divided by data per PDU (rounded up of course).

Note also that the CmdSN window is across a session. If you have =
connections in a session that are running over separate RNICs and are =
using CmdSN for flow control, each RNIC will have to have access to =
enough buffers for the whole window to land on it.=20

Between these two factors, CmdSN flow control will require over =
provisioning buffers much of the time. Perhaps memory is cheap enough =
that for an RNIC with a small number of connections this is acceptable =
in exchange for using an existing mechanism. On the other hand, we will =
have to create a mechanism to handle immediate commands and other PDUs =
that aren't covered by CmdSN so it isn't clear to me whether this is the =
right answer. The downside is overprovisioning buffers because of =
sessions spanning adapters and because each command might be a write =
with unsolicited data but many commands are reads. The upside is that =
CmdSN window can be managed to respond to changes in load while one has =
a less responsive simple mechanism to deal with the rest of the traffic.

From target to initiator, the initiator knows that each command will =
generate a response PDU so it can provision that before it sends a =
command.=20

Login and text negotiation PDUs both ways (other than perhaps the first =
PDU to open text negotiation) also have a form of iSCSI flow control. =
For these, having sent a PDU, one can't send another until one has =
gotten the response.

R2T PDUs are replaced by RDMA Reads in iSER so they are coverd by the =
RDMA Read flow control. SCSI Data-out for solicited and SCSI Data-in =
PDUs are replaced by RDMA operations so we don't have to cover them.


What isn't flow controlled by iSCSI:
initiator to target:
immediate command PDUs - existing iSCSI allows for the target to drop =
these if it gets more than it can handle and the initiator can only =
count on buffering for two, but the initiator can send more than that =
and hope the target has buffering. One can't count on how many of these =
there might be.

the first Text Request PDU - there can only be one

SNACK request - in theory, one shouldn't need to send this when =
operating over iSER, but an initiator might send one if a timeout =
occurs.

NOP-Out - one doesn't expect a lot of them, but there is no limit placed =
on them.=20

target to initiator:
Asynchronous Message

NOP-In - one doesn't expect a lot of them, but there is no limit placed =
on them.

While there usually won't be a lot of these PDUs, one has no way to put =
a maximum number on how many there will be.

There may be times when these PDUs are being sent and there is little or =
no command traffic. Also the replenishment of buffers for these has =
little relationship to the command processing. So I don't think one =
should link the replenishment to command related activities, e.g. =
advances in MaxCmdSN or reception of a command response.
=20
That is the problem space.

Does one put a flow control into iSER that looks at opcode and, for =
commands, at the I bit to control just the PDUs that aren't limited by =
MaxCmdSN or does one put in a mechanism that covers all PDUs going =
through iSER?

If one does the former, what happens when the transmitting iSER gets a =
PDU it have to stop because of the flow control and there are MaxCmdSN =
controlled PDUs after it?=20
Is iSCSI suppose to be flow control aware and manage the PDUs so that =
doesn't happen?=20
Does the whole connection transmit stop until more credit arrives?
Are the MaxCmdSN controlled PDUs allowed to pass the flow controlled PDU =
and be transmitted?

Is there a mechanism to disable flow control when the receiver doesn't =
require it, e.g. large shared buffer pool with statistical provisioning?

Pat

_______________________________________________
rddp mailing list
rddp@ietf.org
https://www1.ietf.org/mailman/listinfo/rddp