Re: [bmwg] Eric Rescorla's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)

[Eric Rescorla <[e-mail]>]

On Wed, Apr 18, 2018 at 1:26 AM, <[e-mail]>
wrote:

> Hi Eric,
>
> Thank you for your feedback on this draft. Please find below my response
> inline with tag [Bhuvan]
>
> @Al, Thanks for taking care of Eric's last feedback.
>
> Best regards,
> Bhuvan
>
> On Wed, Apr 18, 2018 at 02:44 AM, Eric Rescorla <[e-mail]> wrote:
>
> Eric Rescorla has entered the following ballot position for
> draft-ietf-bmwg-sdn-controller-benchmark-meth-08: No Objection
>
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
>
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
>
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-
> controller-benchmark-meth/
>
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
>
> Rich version of this review at:
> https://mozphab-ietf.devsvcdev.mozaws.net/D3948
>
> COMMENTS
>
> reported.
>
> 4.7. Test Repeatability
>
> To increase the confidence in measured result, it is recommended
> that each test SHOULD be repeated a minimum of 10 times.
>
>
>
> Nit: you might be happier with "RECOMMENDED that each test be repeated
> ...."
>
> Also, where does 10 come from? Generally, the number of trials you
> need depends on the variance of each trial.
> [Bhuvan] The RECOMMENDED number 10 was arrived based on our experience
> during the benchmarking. I will discuss with other authors for changing
> SHOULD to RECOMMENDED.
>
>
The SHOULD versus RECOMMENDED has the same normative force in 2119. It's
just editorial and I thought it
would read better.

Test Reporting
>
> Each test has a reporting format that contains some global and
> identical reporting components, and some individual components that
> are specific to individual tests. The following test configuration
> parameters and controller settings parameters MUST be reflected in
>
>
>
> This is an odd MUST, as it's not required for interop.
>
> [Bhuvan] The intent of specifying MUST is to capture relevant test
> parameters to enable Apple to Apple comparison of test test results across
> two testers/test runs.
>

OK



> 5. Stop the trial when the discovered topology information matches
> the deployed network topology, or when the discovered topology
> information return the same details for 3 consecutive queries.
> 6. Record the time last discovery message (Tmn) sent to controller
> from the forwarding plane test emulator interface (I1) when the
> trial completed successfully. (e.g., the topology matches).
>
>
>
> How large is the TD usually? How much does 3 seconds compare to that?
> [Bhuvan] The test duration varies depends on the size of the test
> topology. For a smaller topology (3 - 10) the TD was within a minute. So we
> kept the query interval of 3 seconds to accomodate smaller and larger
> topologies.
>
>
So, 3 seconds is a pretty big fraction of that. It introduces non-trivial
random (I think) error.

As for n-1, I *think* it's the right one here, but I'm not sure. It's what
you use for "sample variance" typically. Have you talked to a statistician?

-Ekr



Total Trials
>
> SUM[SQUAREOF(Tri-TDm)]
> Topology Discovery Time Variance (TDv)  ----------------------
> Total Trials -1
>
>
>
> You probably don't need to specify individual formulas for mean and
> variance. However, you probably do want to explain why you are using
> the n-1 sample variance formula.
>
>
>
> [Bhuvan] We have added both formulas based on the feedback received in the
> mailing list.  We are using n-1, as it is commonly used variance measure.
> Do we need an explanation here or providing any reference to this is
> sufficient?
>
>
Well, my point was that you could specify mean and variance in one place
and not repeat them over and over

Well, n-1 is typically used for sample variance. This is something a little
different.

> Measurement:
>
> (R1-T1) + (R2-T2)..(Rn-Tn)
> Asynchronous Message Processing Time Tr1 = -----------------------
> Nrx
>
>
>
> Incidentally, this formula is the same as \sum_i{R_i} - \sum_i{T_i}
>
>
>
> [Bhuvan]  Good suggestion, we will incorporate in the next revision.
>
>
>
> messages transmitted to the controller.
>
> If this test is repeated with varying number of nodes with same
> topology, the results SHOULD be reported in the form of a graph. The
> X coordinate SHOULD be the Number of nodes (N), the Y coordinate
> SHOULD be the average Asynchronous Message Processing Time.
>
>
>
> This is an odd metric because an implementation which handled overload
> by dropping every other message would look better than one which
> handled overload by queuing.
>
>
>
> [Bhuvan] Believe Al has clarified this feedback. As suggested by Al, we
> will update the BCP language in the reporting section.
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg
>
> DISCLAIMER: Privileged and/or Confidential information may be contained in
> this message. If you are not the addressee of this message, you may not
> copy, use or deliver this message to anyone. In such event,you should
> destroy the message and kindly notify the sender by reply e-mail. It is
> understood that opinions or conclusions that do not relate to the official
> business of the company are neither given nor endorsed by the company.
>