Re: [bmwg] Benoit Claise's No Objection on draft-ietf-bmwg-dcbench-methodology-15: (with COMMENT)

[Lucien <[e-mail]>]

Hi Benoit,

Thank you for the comment! ( I did address also Tim's comment by updating
the draft and answering his email)

Please see inline for your comments:

On Wed, Jun 21, 2017 at 8:49 AM, Benoit Claise <[e-mail]> wrote:

> Benoit Claise has entered the following ballot position for
> draft-ietf-bmwg-dcbench-methodology-15: No Objection
>
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
>
>
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
>
>
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-bmwg-dcbench-methodology/
>
>
>
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
>
> At some point in time, I was wondering if this document was about
> (benchmarking) terminology or about (benchmarking) performance metric
> definition? And I was wondering what the connection was with the RFC 6390
> PMOL
> template and with
> https://tools.ietf.org/html/draft-ietf-ippm-metric-registry-11? When I
> look at
> the section 3 "jitter", there are obvious elements from the PMOL template.
> Thinking about it, whether a performance metric is used for benchmark or
> not,
> we need a formal definition. Is it time for: 1. IPPM to finish up
> draft-ietf-ippm-metric-registry 2. start populating the registry 3. BMWG
> to
> start using/specifying those performance metrics
>
> Thoughts?
>

Initially when we started this over 4 years ago i wanted to do 1 document
for both terminology and method of benchmarking. It became a very long and
intense publication, and I believe back then when you were also our AD, the
feedback we kept getting was to make this easier and break it down in two
publications wich we did. Now looking back, I am glad we did because it
provides a much better structure and it's more digestable and succinct.

The document terminology is about defining the terms specific for data
center benchmarking. It's not about the performance per say. But it's
related to the implication of performance testing.

The document methodology (on the same last call), is about how and what to
benchmark (using the terminologies defined in the other document).

I think it would be a great idea to harmonize all performance metrics
accross WGs, but that's really beyond the scope of what our goal is in
those two publications. I am happy to help participate in such incentive /
project if there is critical mass willing to do it.





>
> Below is Tim Chown's OPS DIR review:
> verall, the document is well written, and I believe it to be Ready with
> minor
> nits.
>
> General comment:
>
> It would be interesting to see an Appendix with an example of a recorded
> test
> using the language defined here.
>
> Specific comments:
>
> Section 1:
>
> “- Low amount of buffer (in the Mb range)”
> Change to MB?  (given later you refer to KB/MB/GB as the measurement unit
> for
> buffers)
>
> Section 2.1
>
> Expand DUT on first use.
>
> Section 3.1
>
> Perhaps clarify relationship of Delay and Latency, since you focus on
> Latency
> in the document and not Delay?
>
> Last para, you say “If” here, but for Latency the FILO timestamp was a
> MUST in
> Section 2?  This doesn’t seem consistent?
>
> Expand PDV on first use.
>
> Section 6.1.1
>
> “1518 bytes frames” -> “1518 byte frames”
>
> Section 7.1,
>
> Why is ‘and’ in quotes here?  Not sure you can say the balance is defined
> by
> goodput?  Do you mean that goodput is an indication of the balance?  For
> standard TCP, a very small loss can have a dramatic effect on application
> throughput.
>
> The second para should follow the first, change “[RFC2647].  Goodput…” to
> “[RFC2647], i.e., goodput…”
>
> Section 7.3
>
> I don’t understand how the example given correlates to G = S/Ft ?
>
> There’s a few typos in this section; please re-check.
>
>
>