Re: [storm] SPC-2 reserve/release text - version 6
"Knight, Frederick" <Frederick.Knight@netapp.com> Fri, 28 September 2012 16:40 UTC
Return-Path: <Frederick.Knight@netapp.com>
X-Original-To: storm@ietfa.amsl.com
Delivered-To: storm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0229521F851E for <storm@ietfa.amsl.com>; Fri, 28 Sep 2012 09:40:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.599
X-Spam-Level:
X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1QMLLzE5yazC for <storm@ietfa.amsl.com>; Fri, 28 Sep 2012 09:40:13 -0700 (PDT)
Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 65B1E21F851C for <storm@ietf.org>; Fri, 28 Sep 2012 09:40:13 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.80,502,1344236400"; d="scan'208";a="695287335"
Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 28 Sep 2012 09:40:12 -0700
Received: from vmwexceht05-prd.hq.netapp.com (vmwexceht05-prd.hq.netapp.com [10.106.77.35]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id q8SGeBve026428; Fri, 28 Sep 2012 09:40:12 -0700 (PDT)
Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.102]) by vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) with mapi id 14.02.0309.002; Fri, 28 Sep 2012 09:40:11 -0700
From: "Knight, Frederick" <Frederick.Knight@netapp.com>
To: "Black, David" <david.black@emc.com>
Thread-Topic: [storm] SPC-2 reserve/release text - version 6
Thread-Index: Ac2cNcys4MzRK8PXQu651fVKNuEIxQAXDfsAABnkfIAADjDW0AAa4S6gACikjGD//nJjgIAAECkggAAHbKCAAAZiEIAABrHggAAIv6CAAAR6sP///AFA
Date: Fri, 28 Sep 2012 16:40:10 +0000
Message-ID: <FFEE311F0EC80C4EA64EC8757CAD0CD208EA89CE@SACEXCMBX04-PRD.hq.netapp.com>
References: <8D3D17ACE214DC429325B2B98F3AE7120DE797C4@MX15A.corp.emc.com> <4AE113CD-96D7-414D-8AAB-36ECF7FD89EA@satran.net> <8D3D17ACE214DC429325B2B98F3AE7120DE798F9@MX15A.corp.emc.com> <FFEE311F0EC80C4EA64EC8757CAD0CD208EA8077@SACEXCMBX04-PRD.hq.netapp.com> <8D3D17ACE214DC429325B2B98F3AE7120DE7993C@MX15A.corp.emc.com> <FFEE311F0EC80C4EA64EC8757CAD0CD208EA8277@SACEXCMBX04-PRD.hq.netapp.com> <EF9A0D09-7415-4460-9224-C9334EA71A68@satran.net> <FFEE311F0EC80C4EA64EC8757CAD0CD208EA83F9@SACEXCMBX04-PRD.hq.netapp.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B42@MX15A.corp.emc.com> <FFEE311F0EC80C4EA64EC8757CAD0CD208EA88C7@SACEXCMBX04-PRD.hq.netapp.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79B71@MX15A.corp.emc.com> <FFEE311F0EC80C4EA64EC8757CAD0CD208EA8922@SACEXCMBX04-PRD.hq.netapp.com> <8D3D17ACE214DC429325B2B98F3AE7120DE79BB8@MX15A.corp.emc.com>
In-Reply-To: <8D3D17ACE214DC429325B2B98F3AE7120DE79BB8@MX15A.corp.emc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.106.53.53]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Cc: "storm@ietf.org" <storm@ietf.org>
Subject: Re: [storm] SPC-2 reserve/release text - version 6
X-BeenThere: storm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Storage Maintenance WG <storm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/storm>, <mailto:storm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/storm>
List-Post: <mailto:storm@ietf.org>
List-Help: <mailto:storm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/storm>, <mailto:storm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 28 Sep 2012 16:40:15 -0000
Timers don't resolve the situation (they are just a tradeoff). In my opinion, a device that allows that sequence: > Initiator A - RESERVE command (GOOD status) > Initiator B - WRITE (RESERVATION CONFLICT status) > Initiator A gets disconnected/reconnected and RESERVE state is lost > Initiator B - WRITE (GOOD status) Is in violation of the text in SPC-2 that says: "Reserve/release managed reservations are retained by the device server until released or until reset by mechanisms specified in this standard." Since - disconnect/reconnect is NOT specified in SPC-2, some form of reset is the only model left (and resets establish UAs). Anyway, I'm glad we could settle on some text. Fred -----Original Message----- From: Black, David [mailto:david.black@emc.com] Sent: Friday, September 28, 2012 11:44 AM To: Knight, Frederick Cc: storm@ietf.org Subject: RE: [storm] SPC-2 reserve/release text - version 6 Fred, > Here's a suggestion that removes the "IF" and doesn't add any new must/should > requirements: > > Unit Attention conditions that result from session reestablishment > indicate whether nexus state was preserved, see Section 6.3.4 of [SAM4]. That text is acceptable to me. I don't think we need anything beyond that text in the draft, but here's the rest of the explanation ... An important aspect of what's going on here is that with respect to the use of the term "re-establishment" for an I_T Nexus in Section 6.3.4 of [SAM4], after Time2Retain expires, an iSCSI target is entitled to consider ISID re-use to not be a "re-establishment" of the I_T Nexus. This is because when Time2Retain expires, the target is allowed to discard all of its state for that nexus, including the target's record of the ISID value. Avoidance of the problematic scenario involving a reservation loss is a factor to consider in selecting Time2Retain values: > Initiator A - RESERVE command (GOOD status) > Initiator B - WRITE (RESERVATION CONFLICT status) > Initiator A gets disconnected/reconnected and RESERVE state is lost > Initiator B - WRITE (GOOD status) Thanks, --David > -----Original Message----- > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > Sent: Friday, September 28, 2012 11:09 AM > To: Black, David > Cc: storm@ietf.org > Subject: RE: [storm] SPC-2 reserve/release text - version 6 > > The IF part of the statement: > > > > >> If a Unit Attention condition results from session reestablishment, > > > >> the specific Unit Attention condition indicates whether nexus state > > > >> was preserved, see Section 6.3.4 of [SAM4]. > > Is what is broken. > > IF the target decides to establish a UA, then it must follow SAM4 otherwise, > if the target decides not to establish a UA, then everything is fine. > Consider the following sequence: > > Initiator A - RESERVE command (GOOD status) > Initiator B - WRITE (RESERVATION CONFLICT status) > Initiator A gets disconnected/reconnected and RESERVE state is lost > Initiator B - WRITE (GOOD status) > > The target decided not to establish a UA as a result of the session > reestablishment, so everything just proceeds on its merry way? Here's a > suggestion that removes the "IF" and doesn't add any new must/should > requirements: > > Unit Attention conditions that result from session reestablishment > indicate whether nexus state was preserved, see Section 6.3.4 of [SAM4]. > > Fred > > -----Original Message----- > From: Black, David [mailto:david.black@emc.com] > Sent: Friday, September 28, 2012 10:34 AM > To: Knight, Frederick > Cc: storm@ietf.org > Subject: RE: [storm] SPC-2 reserve/release text - version 6 > > I'm not sure what's BROKEN, but a statement requiring the UAs would be a post- > SAM-2 requirements change that would have to go in the > -sam- draft so that it's subject to the negotiation of the text key in that > draft. I have no problem with a blanket statement that negotiating that text > key to a value that promises the functionality in the -sam- draft requires > that the implementation adhere to everything in SAM-4 that may differ from > prior versions of SAM, including I_T Nexus loss UA behavior, but there will > need to be some language in there about what happens after Time2Retain expires > before the ISID is reused. > > OTOH, I am strongly opposed to revising the baseline requirements in the > consolidated draft in a fashion that may break "running code" > that's working fine, especially when the change is based on a point of > principle that does not exist in the specifications against which that > "running code" was developed (i.e., SAM-2). > > Thanks, > --David > > > -----Original Message----- > > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > Sent: Friday, September 28, 2012 10:21 AM > > To: Black, David > > Cc: storm@ietf.org > > Subject: RE: [storm] SPC-2 reserve/release text - version 6 > > > > But we also shouldn't codify BROKEN designs in a brand new RFC. It > > would better (in my opinion) to leave it alone than sanction and > > describe how to build a broken implementation. Adding the "IF" is wrong. > > > > We've fixed the issues raised by RFC5048 by incorporating it into the new > RFC. > > > > Fred > > > > -----Original Message----- > > From: Black, David [mailto:david.black@emc.com] > > Sent: Friday, September 28, 2012 9:51 AM > > To: Knight, Frederick > > Cc: storm@ietf.org > > Subject: RE: [storm] SPC-2 reserve/release text - version 6 > > > > Fred, > > > > > If state is retained (it's an old "I" reconnecting), then which UA > > > is defined by SAM-3. > > > > > > If state is not retained (it's a new "I"), then the UA is defined by SAM- > 3. > > > > > > Since both old "I"s and new "I"s result in the "IF" condition being > > > true, what is the condition that would make the "IF" false? > > > > You almost answered your own question ;-). > > > > The answer is .... (drum roll) ... a target implementation based on > > RFC 3720 SAM-2, although I would hope that such implementations at > > least picked up RFC 5048's changes. > > > > The I_T Nexus Loss concept on which this UA discussion is based is > > part of the SCSI events and event notifications model that was > > introduced in SAM-3, but neither RFC 3720 nor RFC 5048 references SAM-3. > > > > There were definitely implementations of iSCSI based on SAM-2 - I > > agree with Julian that we should *not* be introducing a UA requirement > > in this draft that might break such implementations. > > > > Thanks, > > --David > > > > > -----Original Message----- > > > From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > > Sent: Friday, September 28, 2012 8:38 AM > > > To: Julian Satran > > > Cc: Black, David; storm@ietf.org > > > Subject: RE: [storm] SPC-2 reserve/release text - version 6 > > > > > > OK, that's what I thought. > > > > > > If state is retained (it's an old "I" reconnecting), then which UA > > > is defined by SAM-3. > > > > > > If state is not retained (it's a new "I"), then the UA is defined by SAM- > 3. > > > > > > Since both old "I"s and new "I"s result in the "IF" condition being > > > true, what is the condition that would make the "IF" false? > > > > > > That "IF" language allows a target to loose state and not deliver any UA. > > > > > > Fred > > > > > > > > > -----Original Message----- > > > From: Julian Satran [mailto:julian@satran.net] > > > Sent: Friday, September 28, 2012 2:03 AM > > > To: Knight, Frederick > > > Cc: Black, David; storm@ietf.org > > > Subject: Re: [storm] SPC-2 reserve/release text - version 6 > > > > > > There is no such thing. And the reason the timeout is there is to > > > limit the amount of state the targets have to keep. A target may > > > choose to accept very large timeouts and then do what you wanted > > > done but beyond that only the power-up ua is required. > > > > > > julo > > > > > > Sent from my iPad > > > > > > On 28 בספט 2012, at 00:29, "Knight, Frederick" > > > <Frederick.Knight@netapp.com> > > > wrote: > > > > > > > I would think if the ISID is not recognized as an old one, then it > > > > must be > > > recognized as a new one. If it is a new one, then it will get a UA: > > > 29/00 reflecting the first access after power on from the "new" initiator. > > > > > > > > Is there something other than new or old? Is there such a thing > > > > as a > > > middle-aged initiator? > > > > > > > > Fred > > > > > > > > -----Original Message----- > > > > From: Black, David [mailto:david.black@emc.com] > > > > Sent: Thursday, September 27, 2012 11:56 AM > > > > To: Knight, Frederick > > > > Cc: storm@ietf.org > > > > Subject: RE: [storm] SPC-2 reserve/release text - version 6 > > > > > > > > Fred, > > > > > > > > The problem is not that the device does not raise a UA, it's that > > > > the UA may > > > never be delivered. > > > > > > > > A summary of how this can happen is that the new connection with > > > > the same > > > ISID may not be recognized as reinstating or continuing the session > > > at the target if the timeouts have expired - the UA gets > > > established, but nobody shows up to pick it up in time, so the UA is > > > discarded along with the rest of the session state and the target > > > fails to recognize the initiator's subsequent ISID reuse even though > > > the initiator believes that it's re- establishing the session. > > > > > > > > This is further complicated by the UA that indicates nexus state > > > preservation not being specified in SAM-2. > > > > > > > > Thanks, > > > > --David > > > > > > > > > > > >> -----Original Message----- > > > >> From: Knight, Frederick [mailto:Frederick.Knight@netapp.com] > > > >> Sent: Thursday, September 27, 2012 11:11 AM > > > >> To: Black, David; Julian Satran > > > >> Cc: storm@ietf.org > > > >> Subject: RE: [storm] SPC-2 reserve/release text - version 6 > > > >> > > > >> I afraid I have to contest a little on this - if I understand > > > >> this discussion correctly. > > > >> > > > >> If a SCSI device loses any state and does not raise a UNIT > > > >> ATTENTION, then it is BROKEN. It is not a matter of anything in > > > >> iSCSI (3720); I think SAM and SPC are pretty clear on this. If > > > >> there is no UNIT ATTENTION, then state SHALL be preserved; > > > >> otherwise, all sorts of stuff will never work (mode page > > > >> settings, RESERVE commands, all sorts of > > > stuff). > > > >> > > > >> Is someone really suggesting that some devices will drop state > > > >> and NOT tell the host, and somehow that it is OK to do that? > > > >> > > > >> The suggest rewording says this is possible. > > > >> > > > >> Fred > > > >> > > > >> -----Original Message----- > > > >> From: storm-bounces@ietf.org [mailto:storm-bounces@ietf.org] On > > > >> Behalf Of Black, David > > > >> Sent: Thursday, September 27, 2012 10:47 AM > > > >> To: Julian Satran > > > >> Cc: storm@ietf.org > > > >> Subject: Re: [storm] SPC-2 reserve/release text - version 6 > > > >> > > > >> Julian, > > > >> > > > >>> The text in 4.4.3 states the obvious (there is no other speced > > > >>> method for continuation or reinstatement). Moreover if the > > > >>> session is reinstated after time2wait you create a need for a UA > > > >>> (or at least suggest one) and that was not there. > > > >> > > > >> The UAs are courtesy of SAM-3 (and SAM-4). To encompass systems > > > >> that may not deliver these UAs and situations in which they don't > > > >> occur (e.g., the target doesn't recognize what happened as nexus > > > >> reestablishment, independent of what the initiator thinks it's > > > >> doing) I > > > think the text should be rephrased as: > > > >> > > > >> If a Unit Attention condition results from session reestablishment, > > > >> the specific Unit Attention condition indicates whether nexus state > > > >> was preserved, see Section 6.3.4 of [SAM4]. > > > >> > > > >>> Just stating that creation/continuation of a session done > > > >>> according 6.x includes preservation of reserve/release state > > > >>> -that was already there implied by conformance to spc2 and the > > > >>> relation between it nexus and session. > > > >> > > > >> Unfortunately, SPC-2 is sufficiently imprecise on this subject > > > >> that such an implication is at best a "preferred implementation > approach". > > > >> It's not a requirement that can be safely stated in any stronger > > > >> fashion without the risk of rendering existing implementations > > > >> non- > > > compliant. > > > >> > > > >>> The new text you suggest to 4.4 is merely a best practices > > > >>> recommendations and will need additions once the new "locking" > > > >>> through compare-and-swap get in widespread use. > > > >> > > > >> It's definitely best practices recommendations - no argument there. > > > >> > > > >> For context, what Julian is referring to is use of the COMPARE > > > >> AND WRITE command that performs an atomic operation at the target > > > >> instead of using RESERVE and RELEASE to create a critical section > > > >> at the initiator based on read and write commands. > > > >> > > > >> COMPARE AND WRITE is in use - a sentence mentioning that command > > > >> could be added, although this best practices text is aimed > > > >> primarily at uses of reserve/release for long-term reservations. > > > >> I think there's an implicit "when a reservation is the desired > behavior" > > > >> implication to the "persistent reservations SHOULD be used instead" > > > >> text that I don't think needs to be made explicit. OTOH, that "SHOULD" > > > >> could be changed to a "should", or it might be even better to > > > >> rewrite that text > > > >> as: > > > >> > > > >> persistent reservations are suggested as an alternative, see Annex B > > > >> of [SPC4]. > > > >> > > > >> In contrast, "SHOULD NOT be used" is clearly justified for > > > >> reserve/release reservations by [SPC3]. > > > >> > > > >> Thanks, > > > >> --David > > > >> > > > >> > > > >>> -----Original Message----- > > > >>> From: Julian Satran [mailto:julian@satran.net] > > > >>> Sent: Wednesday, September 26, 2012 10:25 PM > > > >>> To: Black, David > > > >>> Cc: storm@ietf.org > > > >>> Subject: Re: [storm] SPC-2 reserve/release text - version 6 > > > >>> > > > >>> David, > > > >>> > > > >>> The text in 4.4.3 states the obvious (there is no other speced > > > >>> method for continuation or reinstatement). Moreover if the the > > > >>> session is reinstated after time2wait you create a need for a UA > > > >>> (or at least suggest one) and that was not there. Just stating > > > >>> that creation/continuation of a session done according 6.x > > > >>> includes preservation of reserve/release state -that was already > > > >>> there implied by conformance to spc2 and the relation between it > > > >>> nexus and > > > session. > > > >>> > > > >>> The new text you suggest to 4.4 is merely a best practices > > > >>> recomendations and will need additions once the new "locking" > > > >>> through compare-and-swap get in widespread use. > > > >>> If instead we refrain from stating anything specific and state > > > >>> that target should maintain whatever state is required by SPC > > > >>> while a session is continued and may discard state when a > > > >>> session is terminated and may indicate that it discarded state > > > >>> through a ua you are on safer ground and do not create new > requirement. > > > >>> > > > >>> Sent from my iPad > > > >>> > > > >>> On 27 בספט 2012, at 00:25, "Black, David" <david.black@emc.com> wrote: > > > >>> > > > >>>> Here's where I think we've arrived (I've made a few minor > > > >>>> editorial tweaks to what's been discussed). Despite all the > > > >>>> detailed discussion, I think this version 6 comes close to > > > >>>> Julian's suggestion > > > >> that: > > > >>>> > > > >>>>> i would mention only that the state that must be maintained > > > >>>>> while the session is maintained includes the reserve/release > > > >>>>> and leave it to > > > >>> that. > > > >>>> > > > >>>> Although we can't actually say "must" as that would be a new > > > >>>> requirement, and implementations differ in what they do about > > > >>>> reserve/release reservation state. The current language > > > >>>> characterizes retention of that state as the "preferred > > > >>>> implementation > > > >> approach". > > > >>>> > > > >>>> Beyond that, I think the two additional caveats about > > > >>>> reserve/release reservations (possible need for a reset, > > > >>>> potentially unexpected interaction with persistent > > > >>>> reservations, see below) are useful > > > >> to include. > > > >>>> > > > >>>> ----------------------------------------- > > > >>>> > > > >>>> [1] For clarity, the following change should be made in > > > >>>> 4.4.3.1: > > > >>>> > > > >>>> OLD > > > >>>> That is, it should perform session recovery as described in > > > >>>> Chapter 6. > > > >>>> NEW > > > >>>> That is, it should reinstate the session via iSCSI session > > > >>>> reinstatement (Section 6.3.5) or continue via session > > > >>>> continuation (Section 6.3.6). > > > >>>> END > > > >>>> > > > >>>> [2] Add the following sentence to the end of 4.4.3.1: > > > >>>> > > > >>>> The specific Unit Attention condition that results from session > > > >>>> reestablishment indicates whether nexus state was preserved, > > > >>>> see Section 6.3.4 of [SAM4]. > > > >>>> > > > >>>> [3] NEW TEXT: > > > >>>> > > > >>>> 4.4.3.2. Reservations > > > >>>> > > > >>>> There are two reservation management methods defined in the > > > >>>> SCSI standards, reserve/release reservations, based on the > > > >>>> RESERVE and RELEASE commands [SPC2], and persistent > > > >>>> reservations, based on the PERSISTENT RESERVE IN and PERSISTENT > RESERVE OUT commands [SPC3]. > > > >>>> Reserve/release reservations are obsolete [SPC3] and SHOULD NOT > > > >>>> be used; persistent reservations SHOULD be used instead. > > > >>>> > > > >>>> State for persistent reservations is required to persist > > > >>>> through changes and failures at the iSCSI layer that result in > > > >>>> I_T Nexus failures, see [SPC3] for details and specific requirements. > > > >>>> > > > >>>> In contrast, [SPC2] does not specify detailed persistence > > > >>>> requirements for reserve/release reservation state after an I_T > > > >>>> Nexus failure. Nonetheless, when reserve/release reservations > > > >>>> are supported by an iSCSI target, the preferred implementation > > > >>>> approach is to preserve reserve/release reservation state for > > > >>>> iSCSI session reinstatement (see Section 6.3.5) or session > > > >>>> continuation (see Section 6.3.6). > > > >>>> > > > >>>> Two additional caveats apply to reserve/release reservations: > > > >>>> > > > >>>> - When connection failure causes the iSCSI session to fail, and > > > >>>> the session is not reinstated or continued, target retention > > > >>>> of that session's reserve/release reservation state may > > > >>>> require an initiator to issue a reset (e.g., LOGICAL UNIT RESET, > > > >>>> see section 11.5) in order to remove that reservation state. > > > >>>> > > > >>>> - Reserve/release reservations may not behave as expected when > > > >>>> persistent reservations are also used on the same logical unit; > > > >>>> see the discussion of "Exceptions to SPC-2 RESERVE and RELEASE > > > >>>> behavior" in [SPC4]. > > > >>>> > > > >>>> Thanks, > > > >>>> --David > > > >>>> ---------------------------------------------------- > > > >>>> David L. Black, Distinguished Engineer EMC Corporation, 176 > > > >>>> South St., Hopkinton, MA 01748 > > > >>>> +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > > >>>> david.black@emc.com Mobile: +1 (978) 394-7754 > > > >>>> ---------------------------------------------------- > > > >>>> > > > >>>> > > > >>>> _______________________________________________ > > > >>>> storm mailing list > > > >>>> storm@ietf.org > > > >>>> https://www.ietf.org/mailman/listinfo/storm > > > >> > > > >> _______________________________________________ > > > >> storm mailing list > > > >> storm@ietf.org > > > >> https://www.ietf.org/mailman/listinfo/storm > > > > _______________________________________________ > > > > storm mailing list > > > > storm@ietf.org > > > > https://www.ietf.org/mailman/listinfo/storm
- [storm] SPC-2 reserve/release text - version 6 Black, David
- Re: [storm] SPC-2 reserve/release text - version 6 Julian Satran
- Re: [storm] SPC-2 reserve/release text - version 6 Black, David
- Re: [storm] SPC-2 reserve/release text - version 6 Knight, Frederick
- Re: [storm] SPC-2 reserve/release text - version 6 Black, David
- Re: [storm] SPC-2 reserve/release text - version 6 Knight, Frederick
- Re: [storm] SPC-2 reserve/release text - version 6 Julian Satran
- Re: [storm] SPC-2 reserve/release text - version 6 Knight, Frederick
- Re: [storm] SPC-2 reserve/release text - version 6 Black, David
- Re: [storm] SPC-2 reserve/release text - version 6 Knight, Frederick
- Re: [storm] SPC-2 reserve/release text - version 6 Black, David
- Re: [storm] SPC-2 reserve/release text - version 6 Knight, Frederick
- Re: [storm] SPC-2 reserve/release text - version 6 Black, David
- Re: [storm] SPC-2 reserve/release text - version 6 Knight, Frederick