Re: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Vratko, Thanks for your review!

I just now re-discovered your comments, and have
addressed them in the working version of the draft
(where I had implemented changes to address Maciek's
comments!). I will submit the new draft when 
submission opens-up again - sorry for the omission!

Al
(as a participant/author)


> -----Original Message-----
> From: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco)
> [mailto:[e-mail]]
> Sent: Thursday, July 4, 2019 1:42 PM
> To: MORTON, ALFRED C (AL) <[e-mail]>
> Cc: [e-mail]
> Subject: RE: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
> 
> Hi Al.
> 
> Sorry for being late with the review.
> I wanted to think some things through,
> and then did not find big enough chunk of time
> to write everything down.
> 
> First, narrow comments.
> 
> * Second line of 1. Introduction:
> 
> > in[RFC2544], supported by the terms and definitions in [RFC1242], and
> 
> Missing space after the first "in".
> 
> * Point 4 in chapter 3 seem to be referring to two different estimates:
> 
> > It was found that the actual buffer time in the DUT could be
> > estimated using results from the Throughput tests conducted
> > according to Section 26.1 of [RFC2544], because it appears that
> > the DUT's frame processing rate may tend to increase the estimate.
> 
> Which estimate is increased, relative to what?
> Is it "implied estimate" tending to be larger than "corrected estimate"?
> 
[acm] 
Thanks for pointing out this ambiguity - I have re-worded this long sentence
for clarity.

> * Within 5.4:
> 
> > (assuming a simple model of the DUT
> > composed of a buffer and a forwarding function)
> 
> A reasonable simplification, but could be mentioned in Motivation,
> so 5.4 does not need a sentence in parentheses.
[acm] 
I used this as an opportunity to remind the reader 
with a reference to Section 3.

> 
> * Still within 5.4:
> 
> > Corrected DUT Buffer Time
> 
> This is a big one.
> The correct name for this quantity could be
> "DUT Buffer Time Correction".
[acm] 
If we were talking about a 'correction factor' I would agree,
but we have two DUT buffer times distinguished by adjectives,
which seems more straightforward to me.

> 
> * still 5.4:
> 
> > and the Buffer size is more accurately estimated by excluding them.
> 
> Can be turned into a formula, where
> "Corrected DUT Buffer Time" can be used in a more natural meaning.
> 
>   Corrected DUT Buffer Time = Implied DUT Buffer Time - DUT Buffer Time
> Correction
[acm] 
You are defining a new term here (?), DUT Buffer Time Correction (factor),
which could be the unit-less term:

    Measured Throughput
  --------------------------
  Max Theoretical Frame Rate

But we only use this term once, and it contains terms we've defined
using quantities easily understood, so that the meaning and limitations
of this factor are clear (when numerator and denominator are equal,
there's no correction needed).


> 
> * In 6. table:
> 
> > Min,Max,StdDev
> 
> Not sure if it is clear these refer to B2B length statistics.
> Include information that these quantities are to be expressed in frames.
[acm] 
Ok

> 
> 
> And now some broader comments.
> 
> * Suspended processor in production:
> 
> > useful to estimate whether frame losses will occur if
> > DUT forwarding is temporarily suspended in a production deployment
> 
> This is the main benefit, and Introduction already mentions
> "compensating for disruptions in the software packet processor".
> 
> But I think it would be good to classify different scenarios
> leading to frame loss, that could explain both two buffer times
> (implied and corrected) and how to use them outside back-to-back load.
> 
> In back-to-back test we have processor processing,
> while the buffer is being filled by maximal traffic.
> The time it takes to fill is the "implied buffer time".
> 
> In a hypothetic scenario where processor is suspended
> and buffer is being filled by maximal traffic,
> the time to fill the buffer is shorter, the "corrected buffer time".
> 
> A scenario occuring in practice has suspended processor,
> but the buffer is being filled more slowly, say at throughput rate.
> Deployers wishing to predict the time for the buffer to fill up
> can use this formula:
>   Real Buffer Time = Corrected Buffer Time * B2B Frame Rate / Real Frame
> Rate
> That is why reporting corrected (instead of implied) buffer time is
> useful.
[acm] 
Yes, vCPU operation can be suspended, and processors can appear suspended 
while they handle higher priority processes that interrupt the 
data plane operations for a significant amount of time.
But orchestrating these suspensions/interrupts for measurements isn't 
practical.  

Nevertheless, the calculation above is useful, I think, so I added it
below the table of results, but I think the correction should be based on
the Measured Throughput we used before (B2B would be = Max Theoretical).
We have the corrected time using Measured Throughput and add time to 
that value.

> 
> Also, it would be nice to name the scenarios and rename the buffer times.
> For example, "running buffer time" and "suspended buffer time"
> (instead of "implied" and "corrected" respectively).
[acm] 
I see what you are going, but the term "suspended buffer time"
seems anti-intuitive. The packet forwarding is running or suspended,
not the buffer, so we need a longer term like 
"suspended forwarding buffer time" which if we measured correctly,
is just the accurate estimate of "buffer time".

> 
> * B2B processor rate:
> 
> For the computation of the corrected buffer time to be correct,
> real processor frame processing rate (average during B2B test)
> should be used instead Measured Throughput in the 5.4 formula.
> 
> The process rate is not easy to measure directly,
> especially if the immediate rate varies over the duration of B2B traffic.
> I agree that Throughput is a reasonable approximation,
> but there may be other quantities (e.g. FRMOL [1]),
> that are either a better approximation, or at least easier to measure.
> 
> Not sure how much attention other such quantities should get in the draft,
> as Throughput has the advantage of avoiding some frame sizes.
[acm] 
Right, that's a big advantage.
I have some sympathy for using other approximations/measurements of the 
"real" forwarding rate. But FRMOL may not be the best alternative,
because overload behavior could be degenerative.
The definition of FRMOL goes on to say:

   Discussion:

      Forwarding rate at maximum offered load may be less than the
      maximum rate at which a device can be observed to successfully
      forward traffic.  This will be the case when the ability of a
      device to forward frames degenerates when offered traffic at
      maximum load.

> 
> * DUT vs SUT:
> 
> This is related to final items of 4. Prerequisities.
> 
> > Therefore, sources of packet loss
> > that are un-related to consistent evaluation of buffer size SHOULD be
> > identified and removed or mitigated.
> 
> Do we have a separate document discussing differences
> between testing DUT and SUT? We should have.
> 
> Usually I prefer testing SUT (meaning no extra mitigations),
> but in this case, for the analysis of the three aforementioned scenarios
> to work correctly, we need to make reasonably sure
> the processor is not going to get suspended during B2B test.
[acm] 
In BMWG's literature, a SUT is composed of multiple DUTs in
an expected arrangement, or typical of a planned deployment.

> 
> Also, I agree that an average result of Binary Search with Loss
> Verification
> gives a more realistic process rate estimate
> than an average result without loss verification.
[acm] 
Good.  Thanks again for your comments!
> 
> Vratko.
> 
> [1] https://tools.ietf.org/html/rfc2285#page-16
> 
> 
> -----Original Message-----
> From: bmwg <[e-mail]> On Behalf Of [e-mail]
> Sent: Thursday, 2019-July-04 17:48
> To: [e-mail]
> Cc: [e-mail]
> Subject: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
> 
> 
> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
> This draft is a work item of the Benchmarking Methodology WG of the IETF.
> 
>         Title           : Updates for the Back-to-back Frame Benchmark in
> RFC 2544
>         Author          : Al Morton
> 	Filename        : draft-ietf-bmwg-b2b-frame-00.txt
> 	Pages           : 12
> 	Date            : 2019-07-04
> 
> Abstract:
>    Fundamental Benchmarking Methodologies for Network Interconnect
>    Devices of interest to the IETF are defined in RFC 2544.  This memo
>    updates the procedures of the test to measure the Back-to-back frames
>    Benchmark of RFC 2544, based on further experience.
> 
>    This memo updates Section 26.4 of RFC 2544.
> 
> 
> 
> The IETF datatracker status page for this draft is:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_draft-2Dietf-2Dbmwg-2Db2b-
> 2Dframe_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=KGWJ5PiKSpydvqF2Zjj7nU84zzXKmdTmrJVljmaDKyY&e=
> 
> There are also htmlized versions available at:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Db2b-2Dframe-
> 2D00&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=vmpSsfloMMDHBQgWaiRbxfoOhIrOdkSjyx3xkUez7Wk&e=
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_html_draft-2Dietf-2Dbmwg-2Db2b-2Dframe-
> 2D00&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=HDt6i-tc3YgzN2BD1gWqMk3d_5AfJpoCAaiagAbTfi0&e=
> 
> 
> Please note that it may take a couple of minutes from the time of
> submission until the htmlized version and diff are available at
> tools.ietf.org.
> 
> Internet-Drafts are also available by anonymous FTP at:
> https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.ietf.org_internet-
> 2Ddrafts_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=Nb-hM-c8mHzaS1JVAeHe-B7fZfodV4DaDZrSHn6Yff4&e=
> 
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=KIjOdOro8S84nxchOIFL22xaljYGQE7P2F6GyHYB0xA&e=