Re: [bmwg] Murray Kucherawy's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

["Murray S. Kucherawy" <[e-mail]>]

On Thu, Feb 10, 2022 at 2:52 PM Carsten Rossenhoevel <[e-mail]> wrote:

> Apologies for the late response to your review.
>

Not at all, I appreciate the attention and thought you're giving to it.

> Upon re-reading the draft, we agree that the number of SHOULD statements
> is indeed quite high.  In our defense :-), we wanted to create well-defined
> benchmarking methodology that does not leave uncertainty (or allow
> compliant but unrealistic behavior that would give the user an unfair
> advantage in benchmarking comparisons).  Anyway, throughout the discussions
> in 20 drafts over three years, some of these statements got changed in a
> way that their intended meaning got distorted.
>

I am by no means an expert on crafting test documents.  I am instead going
by the text of RFC 2119 which guides our use of "SHOULD" and friends to
describe requirements for protocol interoperability.  As applied to a test
description, I have to extrapolate a bit, but I would imagine use of them
as follows:

"SHALL" or "MUST" means if you don't do exactly what this says when
constructing your test or test framework, the test is ruined and its
results cannot be meaningfully applied.

"RECOMMENDED" or "SHOULD" means you really ought to do this (perhaps
because it will yield a simulation closer to reality), but if you don't,
the test is still considered viable.  And oh, by the way, here are some
reasons why you might want or need to deviate.

"OPTIONAL" or "MAY" means the test constructor has full discretion here;
whatever you choose is very unlikely to affect the meaningful outcome of
the test.  (The same would go for any aspect or parameter of the test not
specifically addressed by the document.). One thing to consider is whether
such a parameter needs to be described at all.  Maybe you want to
explicitly say "We thought about this, and yes, it really is optional" as
opposed to omitting it, where the reader might wonder if this aspect of the
test got any consideration.

With so many SHOULDs in there, however, I begin to wonder about just how
much variability this test framework has.

Ergo:

With help from Al Morton, we concluded that the SHOULD statements in the
> document seem to fall into a couple of categories:
> *Category*
> *Explanation*
> *Suggested Authors' Action*
> Conditional
> a specific way is recommended, but other more or less harmful ways would
> also be possible
> Review text to ensure that alternatives and their consequences are
> described explicitly
> Aspirational
> recommending that the industry will adopt a specific behavior or
> methodology
> Keep
> Pleonastic
> statements that are without viable technical alternative
> Eliminate and transfer into indicative text form
> Hidden MUSTs
> editors frowned upon the use of MUST and chose SHOULD instead
> Convert into explicit MUST?
>
> First, nice work on such classifications.

> Q1) Do you agree with this classification and the suggested actions?
>
For "Conditional", yes.

For "Aspirational", I suggest not using the key words at all, but instead
including a section (perhaps early in the document) addressed to the
industry advising the methodology you think it should adopt (and why), an
example of which is the test framework described by this document.

For "Pleonastic", if by that explanation you mean they really ought to be
MUSTs, then I would make that conversion.

For "Hidden MUST", I would consider why there was the preference for the
weaker assertion.  Some of them might deserve MUST, some might deserve
something else, some might rightly remain SHOULD.  But I think given your
chart above, you're on the right track to thinking about these already.

> Q2) Do you recommend that we go through each of the SHOULDs in the whole
> document and modify if needed, or focus only on the explicit occurrences
> that you quoted in your review?
>
I would recommend going through them all.  I found them by typing SHOULD
into the search tool of my browser and then just selecting them in
sequence.  There are indeed a lot, but you could perhaps divide and
conquer.  The good news is that there are a lot of them that are just
fine.  :-)

The ones I cited in my DISCUSS stood out to me as thoroughly confusing and
I would definitely reconsider them.  But I can give you some other examples
where I paused while reviewing:

* the first SHOULD in Section 4.2 -- why isn't that a MUST?  is the test
still viable if I don't do that?  if it stays SHOULD, what deviations might
be acceptable?
* the SHOULD in Section 4.2.1 refers to some minima (500 CVEs in the last
10 years), but why those numbers?  what if I only do 100?  what about one
or none?  what about 499?
* in 4.3.1.2 you have a SHOULD covering a bullet list, and a SHOULD on the
first bullet; this second SHOULD is thus redundant to the first

On the other hand, for example, all but the last SHOULD in 4.3.1.1, and all
of the ones in 4.3.1.3, look perfectly reasonable to me (modulo your action
under "Conditional" above).

> Q3) Should we state explicit "MUST" in category 4 cases -- do you think
> this is permissible for an informational RFC -- or do you have any other
> suggested action?
>
I think since you've cited BCP 14 (RFC 2119 and RFC 8174), you may as well
use them.  It is a little weird for an Informational document to read like
a protocol specification, but this is hardly the first document we've seen
like this.

For "Hidden MUSTs", I think the action is to figure out why the editors
balked at MUST, re-evaluate, and commit by either making it a MUST, or
leaving it SHOULD but treating it as you say you will for "Conditional".

You bring up an interesting point though.  One of my mentors demonstrated
to me that it's possible to write normative text without using our
MUST/SHOULD/MAY key words.  For example, here are two instances of
normative text, arguably of equal force, copied from your Section 4:

OLD:

   Test setup defined in this document applies to all benchmarking tests
   described in Section 7
<https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-ngfw-performance-13#section-7>.
The test setup MUST be contained within an
   Isolated Test Environment (see Section 3 of [RFC6815]
<https://datatracker.ietf.org/doc/html/rfc6815#section-3>).

NEW:

   Test setup defined in this document applies to all benchmarking tests
   described in Section 7
<https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-ngfw-performance-13#section-7>.
A test setup compliant with this specification
   is contained within an Isolated Test Environment (see Section 3 of
[RFC6815] <https://datatracker.ietf.org/doc/html/rfc6815#section-3>).

So if you want to avoid having to pick one of them and slot it within the
definitions we use for those key words, you definitely have some latitude
with your prose without losing the force of the message.

I hope this has been helpful.  Please let me know if I can provide further
guidance.

-MSK