Re: [bmwg] Murray Kucherawy's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

[Scott Bradner <[e-mail]>]

speaking as the editor of RFC 2119

nice chart of how to misuse SHOULD but - imo - SHOULD is a MUST with an escape clause

I do not think it is useful to use SHOULD without specifically saying what the escape clause is - 
i.e. specifically say when its OK to not act as if it was a MUST

so, if you cannot say when the SHOULD should not be a MUST then it should be a MUST or a MAY

Scott

(if I had to do it all over again I would not have included SHOULD/SHOULD NOT)

> On Feb 11, 2022, at 1:03 AM, Murray S. Kucherawy <[e-mail]> wrote:
> 
> On Thu, Feb 10, 2022 at 2:52 PM Carsten Rossenhoevel <[e-mail]> wrote:
> Apologies for the late response to your review.  
> 
> Not at all, I appreciate the attention and thought you're giving to it. 
> Upon re-reading the draft, we agree that the number of SHOULD statements is indeed quite high.  In our defense :-), we wanted to create well-defined benchmarking methodology that does not leave uncertainty (or allow compliant but unrealistic behavior that would give the user an unfair advantage in benchmarking comparisons).  Anyway, throughout the discussions in 20 drafts over three years, some of these statements got changed in a way that their intended meaning got distorted. 
> 
> 
> I am by no means an expert on crafting test documents.  I am instead going by the text of RFC 2119 which guides our use of "SHOULD" and friends to describe requirements for protocol interoperability.  As applied to a test description, I have to extrapolate a bit, but I would imagine use of them as follows:
> 
> "SHALL" or "MUST" means if you don't do exactly what this says when constructing your test or test framework, the test is ruined and its results cannot be meaningfully applied.
> 
> "RECOMMENDED" or "SHOULD" means you really ought to do this (perhaps because it will yield a simulation closer to reality), but if you don't, the test is still considered viable.  And oh, by the way, here are some reasons why you might want or need to deviate.
> 
> "OPTIONAL" or "MAY" means the test constructor has full discretion here; whatever you choose is very unlikely to affect the meaningful outcome of the test.  (The same would go for any aspect or parameter of the test not specifically addressed by the document.). One thing to consider is whether such a parameter needs to be described at all.  Maybe you want to explicitly say "We thought about this, and yes, it really is optional" as opposed to omitting it, where the reader might wonder if this aspect of the test got any consideration.
> 
> With so many SHOULDs in there, however, I begin to wonder about just how much variability this test framework has.
> 
> Ergo:
> 
> 
> With help from Al Morton, we concluded that the SHOULD statements in the document seem to fall into a couple of categories: 
> 
> Category
> Explanation
> Suggested Authors' Action
> Conditional
> a specific way is recommended, but other more or less harmful ways would also be possible
> Review text to ensure that alternatives and their consequences are described explicitly
> Aspirational
> recommending that the industry will adopt a specific behavior or methodology
> Keep
> Pleonastic
> statements that are without viable technical alternative
> Eliminate and transfer into indicative text form
> Hidden MUSTs
> editors frowned upon the use of MUST and chose SHOULD instead
> Convert into explicit MUST?
> 
> First, nice work on such classifications. 
> Q1) Do you agree with this classification and the suggested actions?
> 
> For "Conditional", yes.
> 
> For "Aspirational", I suggest not using the key words at all, but instead including a section (perhaps early in the document) addressed to the industry advising the methodology you think it should adopt (and why), an example of which is the test framework described by this document.
> 
> For "Pleonastic", if by that explanation you mean they really ought to be MUSTs, then I would make that conversion.
> 
> For "Hidden MUST", I would consider why there was the preference for the weaker assertion.  Some of them might deserve MUST, some might deserve something else, some might rightly remain SHOULD.  But I think given your chart above, you're on the right track to thinking about these already.
> Q2) Do you recommend that we go through each of the SHOULDs in the whole document and modify if needed, or focus only on the explicit occurrences that you quoted in your review?
> 
> I would recommend going through them all.  I found them by typing SHOULD into the search tool of my browser and then just selecting them in sequence.  There are indeed a lot, but you could perhaps divide and conquer.  The good news is that there are a lot of them that are just fine.  :-)
> 
> The ones I cited in my DISCUSS stood out to me as thoroughly confusing and I would definitely reconsider them.  But I can give you some other examples where I paused while reviewing:
> 
> * the first SHOULD in Section 4.2 -- why isn't that a MUST?  is the test still viable if I don't do that?  if it stays SHOULD, what deviations might be acceptable?
> * the SHOULD in Section 4.2.1 refers to some minima (500 CVEs in the last 10 years), but why those numbers?  what if I only do 100?  what about one or none?  what about 499?
> * in 4.3.1.2 you have a SHOULD covering a bullet list, and a SHOULD on the first bullet; this second SHOULD is thus redundant to the first
> 
> On the other hand, for example, all but the last SHOULD in 4.3.1.1, and all of the ones in 4.3.1.3, look perfectly reasonable to me (modulo your action under "Conditional" above).
> Q3) Should we state explicit "MUST" in category 4 cases -- do you think this is permissible for an informational RFC -- or do you have any other suggested action?
> 
> I think since you've cited BCP 14 (RFC 2119 and RFC 8174), you may as well use them.  It is a little weird for an Informational document to read like a protocol specification, but this is hardly the first document we've seen like this.
> 
> For "Hidden MUSTs", I think the action is to figure out why the editors balked at MUST, re-evaluate, and commit by either making it a MUST, or leaving it SHOULD but treating it as you say you will for "Conditional".
> 
> You bring up an interesting point though.  One of my mentors demonstrated to me that it's possible to write normative text without using our MUST/SHOULD/MAY key words.  For example, here are two instances of normative text, arguably of equal force, copied from your Section 4:
> 
> OLD:
>    Test setup defined in this document applies to all benchmarking tests
>    described in 
> Section 7
> .  The test setup MUST be contained within an
>    Isolated Test Environment (see 
> Section 3 of [RFC6815]).
> NEW:
>    Test setup defined in this document applies to all benchmarking tests
>    described in 
> Section 7.  A test setup compliant with this specification
>    is contained within an Isolated Test Environment (see Section 3 of [RFC6815]).
> So if you want to avoid having to pick one of them and slot it within the definitions we use for those key words, you definitely have some latitude with your prose without losing the force of the message.
> 
> I hope this has been helpful.  Please let me know if I can provide further guidance.
> 
> -MSK
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg