Re: [bmwg] I-D Action: draft-ietf-bmwg-ca-bench-meth-02.txt

[Al Morton <[e-mail]>]

At 02:21 PM 7/31/2012, Tom Alexander wrote:
>Hello, Mike,
>
>Thanks for incorporating my feedback. The -03 draft looks very good 
>and I have no further comments.
>
>Best regards,
>
>- Tom

Mike and Sarah,

I read the draft on the flight-out, and have the following comments.
(these are comments on the version 02 text, sorry I couldn't keep
up with your really fast update...)

Al (as a participant)

-=-=-=-=-

end of Section 1:
    ...
    The nature of application-driven networks is such that a single
    dropped packet inherently changes the input stimulus from a network
    perspective.  While application flows will be specified in great
    detail, it simply is not practical to require totally repeatable
    input stimulus.

It seems that the input stimulus *should* be repeatable when there
is no loss.  Whether the measured results are repeatable or not depends
on the DUT.

When there is loss (in overload) the input will change (TCP connections
will react, etc.), and the offered load will change thus the measured
result will change. Therefore several trials at the same conditions will
be needed along with averaging.

-----

Section 2 Scope, 2nd para.
...
    None of the metrics captured through this methodology are specific to
    a device and the results are DUT implementation independent.
Suggest
    a device and the format of the results are DUT implementation independent.
                 ^^^^^^^^^^^^^^
Question of this sentence:
    If this wasn't what you were thinking, then I don't understand what
we are testing - the results must be dependent on the DUT implementation,
but the design of the test must not be.

Further down, para 3:
...
          While this list may become obsolete,
    these are a subset of devices that fall under this scope of testing.
s/of devices/of all devices/

-------

In 3.7.3.  TCP Stack Considerations,

It seems to me that the scenario of DUT as TCP Middlebox is a likely
candidate for these methods, and probably should be mentioned explicitly
here (since TCP options and other parameters might change and affect
the testing - these facts need to be measured and noted).


-------

In 4.1.1.  Objective

    To determine the maximum rate through which a device is able to
s/through/at/
    establish and complete application flows as defined by
    draft-ietf-bmwg-ca-bench-term-00.
Is this under loss-less conditions? failure-less conditions?

-------

In  4.1.3.  Procedure

...
    This procedure MAY be repeated any reasonable number of times with
    the results being averaged together.
and the Standard Deviation SHOULD also be reported (I think)

------