[bmwg] Murray Kucherawy's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

[Murray Kucherawy via Datatracker <[e-mail]>]

Murray Kucherawy has entered the following ballot position for
draft-ietf-bmwg-ngfw-performance-13: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)


Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/
for more information about how to handle DISCUSS and COMMENT positions.


The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/



----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I may be wandering into unfamiliar territory here, i.e., how benchmarking specs
are typically written, but this is sufficiently confusing that I'd like to
discuss it.

I note that RFC 3511, which this document obsoletes, didn't cite RFC 2119 (BCP
14) but rather defined those same key words on its own.  Then it used SHOULD
rather liberally, in a way that seems kind of peculiar to me (especially
compared to the text of Section 6 of RFC 2119).  Do any of them matter to the
outcome of the benchmark being constructed or executed?  If so and they would
spoil the test, shouldn't they be MUSTs?  If not, why include them?  Or in the
alternative, why might I, as someone setting up a test, legitimately do
something contrary to the SHOULD in each case (which "SHOULD" expressly
permits)?

This document does cite BCP 14 directly, and then seems to take that curious
pattern to the next level.  Among the 130+ SHOULDs in here, I'm particularly
confused by stuff like this in Section 4.3.1:

   This section specifies which parameters SHOULD be considered while
   configuring clients using test equipment.

I have no idea what this means to the test.  If I've simply thought about these
parameters, have I met the burden here?

This in Section 4.3.1.1 ("TCP Stack Attributes") seems an odd thing to have to
stipulate:

  The client SHOULD initiate and close TCP connections.

Then Section 7.1.3, which contains subsections about each of the test
parameters for the benchmark described in Section 7.1, consists of this text:

   In this section, the benchmarking test specific parameters SHOULD be
   defined.

As I read it, this is a self-referential SHOULD about this document!  I'm very
confused.  This happens again in Section 7.2.3, 7.3.3, etc., up to 7.9.3, and
even Appendix A.3.  I think in each case you just want:

   This section defines test-specific parameters for this benchmark.


----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Nits not yet mentioned by others:

Section 4.2:

* "... users SHOULD configure their device ..." -- s/device/devices/ (unless
all users share one device)

Section 6.3:

* "The value SHOULD be expressed in millisecond." -- s/millisecond/milliseconds/