Re: [bmwg] Murray Kucherawy's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

[Carsten Rossenhoevel <[e-mail]>]

Dear Murray,

Apologies for the late response to your review.

Upon re-reading the draft, we agree that the number of SHOULD statements 
is indeed quite high.  In our defense :-), we wanted to create 
well-defined benchmarking methodology that does not leave uncertainty 
(or allow compliant but unrealistic behavior that would give the user an 
unfair advantage in benchmarking comparisons).  Anyway, throughout the 
discussions in 20 drafts over three years, some of these statements got 
changed in a way that their intended meaning got distorted.

With help from Al Morton, we concluded that the SHOULD statements in the 
document seem to fall into a couple of categories:

*Category*
	*Explanation*
	*Suggested Authors' Action*
Conditional
	a specific way is recommended, but other more or less harmful ways 
would also be possible
	Review text to ensure that alternatives and their consequences are 
described explicitly
Aspirational
	recommending that the industry will adopt a specific behavior or 
methodology
	Keep
Pleonastic
	statements that are without viable technical alternative
	Eliminate and transfer into indicative text form
Hidden MUSTs
	editors frowned upon the use of MUST and chose SHOULD instead
	Convert into explicit MUST?

Q1) Do you agree with this classification and the suggested actions?

Q2) Do you recommend that we go through each of the SHOULDs in the whole 
document and modify if needed, or focus only on the explicit occurrences 
that you quoted in your review?

Q3) Should we state explicit "MUST" in category 4 cases -- do you think 
this is permissible for an informational RFC -- or do you have any other 
suggested action?

Best regards, Carsten



Am 2/3/2022 um 6:24 AM schrieb Murray Kucherawy via Datatracker:
> Murray Kucherawy has entered the following ballot position for
> draft-ietf-bmwg-ngfw-performance-13: Discuss
>
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
>
>
> Please refer tohttps://www.ietf.org/blog/handling-iesg-ballot-positions/
> for more information about how to handle DISCUSS and COMMENT positions.
>
>
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/
>
>
>
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
>
> I may be wandering into unfamiliar territory here, i.e., how benchmarking specs
> are typically written, but this is sufficiently confusing that I'd like to
> discuss it.
>
> I note that RFC 3511, which this document obsoletes, didn't cite RFC 2119 (BCP
> 14) but rather defined those same key words on its own.  Then it used SHOULD
> rather liberally, in a way that seems kind of peculiar to me (especially
> compared to the text of Section 6 of RFC 2119).  Do any of them matter to the
> outcome of the benchmark being constructed or executed?  If so and they would
> spoil the test, shouldn't they be MUSTs?  If not, why include them?  Or in the
> alternative, why might I, as someone setting up a test, legitimately do
> something contrary to the SHOULD in each case (which "SHOULD" expressly
> permits)?
>
> This document does cite BCP 14 directly, and then seems to take that curious
> pattern to the next level.  Among the 130+ SHOULDs in here, I'm particularly
> confused by stuff like this in Section 4.3.1:
>
>     This section specifies which parameters SHOULD be considered while
>     configuring clients using test equipment.
>
> I have no idea what this means to the test.  If I've simply thought about these
> parameters, have I met the burden here?
>
> This in Section 4.3.1.1 ("TCP Stack Attributes") seems an odd thing to have to
> stipulate:
>
>    The client SHOULD initiate and close TCP connections.
>
> Then Section 7.1.3, which contains subsections about each of the test
> parameters for the benchmark described in Section 7.1, consists of this text:
>
>     In this section, the benchmarking test specific parameters SHOULD be
>     defined.
>
> As I read it, this is a self-referential SHOULD about this document!  I'm very
> confused.  This happens again in Section 7.2.3, 7.3.3, etc., up to 7.9.3, and
> even Appendix A.3.  I think in each case you just want:
>
>     This section defines test-specific parameters for this benchmark.
>
>
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
>
> Nits not yet mentioned by others:
>
> Section 4.2:
>
> * "... users SHOULD configure their device ..." -- s/device/devices/ (unless
> all users share one device)
>
> Section 6.3:
>
> * "The value SHOULD be expressed in millisecond." -- s/millisecond/milliseconds/
>
>
>
-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax [phone], mobile [phone]
[e-mail],https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025