Re: [bmwg] draft-ietf-bmwg-dcbench-methodology-07

[Lucien Avramov <[e-mail]>]

Hi Scott, Al, IESG reviewers, BMWG!

After a good amount of work and feedback from Scott yesterday, I am pleased
to report that we have posted the final versions now.

The changes to look for in the methodology draft are about:

1) clarifying terms and pointing to the terminology draft when appropriate
when new terms were defined in the terminology draft (such as for example
intensity )
2) clarifying the iteration process for all tests, explaining briefly how
to read the test sequence each time
3) clarifying when TCP type of testing and UDP type of testing is needed
(for goodput type of testing)

Many thanks for all the work from so many folks getting us together to
mature documents.

Regards,
Lucien

On Sun, Jun 18, 2017 at 10:00 AM, Lucien Avramov <[e-mail]>
wrote:

> Hi Al, Scott
>
> thanks for the comments, updated both drafts just now with all comments!
>
> Thanks,
> Lucien
>
> On Sun, Jun 18, 2017 at 9:51 AM, MORTON, ALFRED C (AL) <[e-mail]>
> wrote:
>
>> adding OPS-DIR,
>> > -----Original Message-----
>> > From: bmwg [mailto:[e-mail]] On Behalf Of Scott Bradner
>> > Sent: Sunday, June 18, 2017 9:53 AM
>> > To: MORTON, ALFRED C (AL)
>> > Cc: [e-mail]; bmwg-
>> > [e-mail]; [e-mail]
>> > Subject: Re: [bmwg] draft-ietf-bmwg-dcbench-methodology-07
>> >
>> > my issue was that I read the current text as talking about the result of
>> > the testing not the offered load
>> >
>> > so I was just going to suggest coming similar to what Al suggested
>> >
>> > but I’d add specific pointers to the explanation in the terminology
>> > document as to why
>> >
>> >
>> > e.g.
>> >
>> >    For all tests, the test traffic generator sending rate MUST be
>> >    less than or equal to 99.98% of the nominal value of Line Rate
>> >    (with no further PPM adjustment to account for interface clock
>> >    tolerances), to ensure stressing the DUT in reasonable worst case
>> >    conditions. (see RFC XXX section ??? - note to RFC editor replace XXX
>> >    with number of terminology RFC)
>> >
>> [ACM]
>> good addition, Scott.
>>
>> > I have not reviewed the rest of the ID to see if similar misreading is
>> > possible so that
>> > a similar change would be useful
>> [ACM]
>> Let's assume for now that other Area reviewers and IESG review will
>> catch some more of these, if needed.
>>
>> thanks,
>> Al
>>
>> >
>> > Scott
>> >
>> >
>> > > On Jun 18, 2017, at 1:03 AM, MORTON, ALFRED C (AL) <[e-mail]>
>> > wrote:
>> > >
>> > > Hi Lucien, Scott, and all,
>> > >
>> > >
>> > >
>> > > When I read this material, I assumed the goal was to
>> > >
>> > > limit the maximum sending rate of the test traffic generator
>> > >
>> > > in accordance with the Ethernet clock tolerances,
>> > >
>> > > effectively choosing the low-end of clock tolerance
>> > >
>> > > as the maximum sending rate with a factor of 2 margin.
>> > >
>> > > 99.98% of the nominal line rate would be -200 ppm,
>> > >
>> > > and any conforming interface clock should work fine.
>> > >
>> > >
>> > >
>> > > If I understood your goal correctly,
>> > >
>> > > I wonder if a small clarification will help future
>> > >
>> > > readers.
>> > >
>> > >
>> > >
>> > > When section 2.2 says:
>> > >
>> > >    For all tests, the percentage of traffic per port capacity sent
>> > MUST
>> > >
>> > >                                                 ^^^^^^^^^^^^^^^^^^
>> > >
>> > >    be 99.98% at most, with no PPM adjustment to ensure stressing the
>> > DUT
>> > >
>> > >       +++++++++++++++
>> > >
>> > >    in worst case conditions.
>> > >
>> > >
>> > >
>> > > You’ve defined the concept of Port Capacity in terminology Section 5,
>> > >
>> > > under the Line Rate definition, as
>> > >
>> > >
>> > >
>> > >    The term "port capacity" defines the maximum speed capability for
>> > the
>> > >
>> > >    given port; for example 1GE, 10GE, 40GE, 100GE etc.
>> > >
>> > >
>> > >
>> > > Actually, the port capacity examples are the *nominal value of Line
>> > Rate*,
>> > >
>> > > before the clock tolerances are considered.
>> > >
>> > >
>> > >
>> > > Here’s a suggestion for Section 2.2 of the meth:
>> > >
>> > >
>> > >
>> > >    For all tests, the test traffic generator sending rate MUST be
>> > >
>> > >    less than or equal to 99.98% of the nominal value of Line Rate
>> > >
>> > >    (with no further PPM adjustment to account for interface clock
>> > >
>> > >    tolerances), to ensure stressing the DUT in reasonable worst case
>> > >
>> > >    conditions.
>> > >
>> > >
>> > >
>> > > Changing
>> > >
>> > > s/port capacity/nominal value of Line Rate/
>> > >
>> > > here, requires an additional substitution in section 2.3 of the meth
>> > draft,
>> > >
>> > > and a single substitution in section 5.1 of the term draft.
>> > >
>> > >
>> > >
>> > > Does this help?
>> > >
>> > > Al
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > From: Lucien Avramov [mailto:[e-mail]]
>> > > Sent: Saturday, June 17, 2017 1:49 PM
>> > > To: Scott Bradner
>> > > Cc: [e-mail]; [e-mail];
>> > [e-mail]
>> > > Subject: Re: draft-ietf-bmwg-dcbench-methodology-07
>> > >
>> > >
>> > >
>> > > Hi Scott!
>> > >
>> > > ·         The draft you are reviewing is the ''methodology'' draft
>> > which explains the methodology for line rate performance testing. The
>> > exact reason onto why it is 99.98% with 0 PPM adjustment is covered
>> > extensively in the terminology draft, section 5:
>> > https://urldefense.proofpoint.com/v2/url?u=https-
>> > 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Ddcbench-2Dterminology-
>> > 2D12-23section-2D5&d=DwIGaQ&c=LFYZ-
>> > o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=cY2eIWfAaujzPr7
>> A49lWCjuJ_Vx
>> > 0ywhEDDmsWPBcTJA&s=96T3Kqxhm3c-gASDa1w_XDRAcXYh30m7fyjPEm28HCE&e= .
>> That
>> > draft dependency is covered in the introduction and also in the
>> > reference lists and is referenced throughout the document
>> > >
>> > > ·         There is a clock drift difference between tester and DUT
>> > explained in section 5.2 of the terminology draft
>> > >
>> > > In our view BMWG has not morphed, our goal has been to be very precise
>> > and accurate with performance testing, and this is not about conformance
>> > testing at all. These definitions that you see, have not changed in 4
>> > years and have been proven themselves in a very large amount of testing
>> > done since we started publishing.
>> > >
>> > >
>> > >
>> > > As you may remember, we have been working on this draft actively for
>> > over 4 years now. We've incorporated numerous of your feedbacks on these
>> > drafts over the IETF sessions (since IETF 86 - captured in the meeting
>> > minutes of the IETF sessions).  On April 2016, we had your blessing on
>> > the list also regarding our drafts.
>> > >
>> > >
>> > >
>> > > We appreciate as always your feedback, Scott!
>> > >
>> > >
>> > >
>> > > Lucien
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Fri, Jun 16, 2017 at 7:33 PM, Scott Bradner <[e-mail]> wrote:
>> > >
>> > > try again - my new mac book often sends mail before I want it to
>> > >
>> > >
>> > > I have been asked to do a OPS-DIR review of draft-ietf-bmwg-dcbench-
>> > methodology-07
>> > >
>> > > I quickly became confused - a paragraph in section 2.2 reads
>> > >
>> > > For all tests, the percentage of traffic per port capacity sent MUST
>> > >   be 99.98% at most, with no PPM adjustment to ensure stressing the
>> > DUT
>> > >   in worst case conditions. Tests results at a lower rate MAY be
>> > >   provided for better understanding of performance increase in terms
>> > of
>> > >   latency and jitter when the rate is lower than 99.98%. The receiving
>> > >   rate of the traffic SHOULD be captured during this test in % of line
>> > >   rate.
>> > >
>> > >
>> > > Thus reads to me like a conformance test not a performance test
>> > (saying what is “passing”)  unless
>> > > BMWG has morphed a lot since I was last involved conformance and
>> > pass/fail are not things that BMWG
>> > > should be doing -
>> > >
>> > > what am I missing?
>> > >
>> > > Scott
>> > >
>> > >
>> > >
>> > >
>> >
>> > _______________________________________________
>> > bmwg mailing list
>> > [e-mail]
>> > https://urldefense.proofpoint.com/v2/url?u=https-
>> > 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwIGaQ&c=LFYZ-
>> > o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=cY2eIWfAaujzPr7
>> A49lWCjuJ_Vx
>> > 0ywhEDDmsWPBcTJA&s=NwNczwTUN_yQqjl4Aw4jBTqk3iMbJpTrJMDiuFT8g20&e=
>>
>
>