Review of draft-ietf-ippm-model-based-metrics-10

[Robert Sparks <[e-mail]>]

Reviewer: Robert Sparks
Review result: Almost Ready

I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed
by the IESG for the IETF Chair.  Please treat these comments just
like any other last call comments.

For more information, please see the FAQ at

<https://trac.ietf.org/trac/gen/wiki/GenArtfaq>.

Document: draft-ietf-ippm-model-based-metrics-10
Reviewer: Robert Sparks
Review Date: 2017-03-13
IETF LC End Date: 2017-03-14
IESG Telechat date: Not scheduled for a telechat

Status: Almost ready, but has significant issues to address before
publication as an Informational RFC

I'm guessing the goal for this document changed substantially during
its
development. The point and tone if the document changes depending on
where you are in it. I found it very difficult to read (and I did see
the
shepherd's report call out that the document has already had a strong
editorial pass).  It really feels to me that the purpose this document
is
trying to serve could be better achieved with two much shorter
documents.
Please consider pulling out the most important observations into its
own
informational document, and an experimental document that succinctly
presents the tests, the framework, and the feedback you want from
people
wo try these things.

Major issues:

There are issues with the equations in section 7.2. Please look at
the
line containing "Xa < defect_count(n) < Xb". Where is Xb defined? Was
it supposed to be Xr from further up the page? (or perhaps that Xr
should
have been Xb?

The document starts using 2119 MUSTs in section 7.3. None of the uses
are appropriate 2119 keywords - they aren't talking about protocol.
Please
rewrite the prose without using the 2119 keywords.

Section 8.2.4 seems to be missing an actual description/definition of
the 
test it is trying to talk about.


Minor issues:


The abstract says this document does not define tests. The rest of
the
document says it does. (See phrases like "The tests described in this
note")

Where the document claims it is neccessary to "suppress the effects
of
TCP congestion control", it should say "avoid the effects". The
mechanism
of using precalculated traffic patterns discussed in the document
avoids,
but does not suppress.

At the end of section 1, you call out "multiple standardized versions
of
TCP". Please clarify what you're trying to point to?

In the definition of "traffic patterns", you note that the goal is to
"mimic the range of common patterns". Consider qualifying that to
"current
common patterns", and add some discussion later in the document when
you
talk about pregenerating traffic patterns to point out that these
change
over time. 

At the next to last paragraph of section 4.1, you say "metrics have
entirely thwarted the analytic framework". I don't think that's what
you
mean. I suspect you mean to say attempts to create metrics have been
unsuccessful (I suggest you avoid the word "thwarted").

At the 3rd paragraph on page 19, you say "There is no model". Do you
mean
no model exists in the world for this, or only that this document does
not
provide a model.

I strongly object to the attempt to use "infinitesimal" to describe
the
conditions on the test network you are trying to convey. At the very
least,
you are dealing with a finite (if arbitrarily large) set of states
that
network can be in, not a continuum. You should be looking to raid
terms
from discrete mathematics instead, but I don't think even that's the
right
thing to do. Please replace the infinitesimal concept here with it's
definition (which you have to call out anyway). Just _say_ "has the
tightest available constraints that allow the tests to pass." 

Consider providing some rational for the numbers you came up with in
the
strawman in the second paragraph on page 33. 1mS seems pretty
arbitrary
otherwise. I'm not asking for you to _defend_ the choice as the right
number - just talk about why you picked it instead of something else.


Nits:

- Consider pointing to the models you are actually using in the first
  sentence of the first paragraph on page 9

- The third paragraph of the abstract is particularly hard to read.
Please
  consider breaking the compound sentences into simpler ones.

- You use "engineering test" before you say what you mean by that
(see
  sections 8.2.4 and 8.3.2 for examples)

- There are several places where you talk about tests being
inconclusive
  because the precomputed traffic was not accurately generated. Are
you
  trying to say "if you have an inconclusive test and can't otherwise
figure
  out why, look closely at your precomputed traffic", or "Double-check
your
  precomputed traffic before you run your test or risk
garbage-in/garbage
  out"?