Re: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Vratko,

Thanks for your comments, both narrow and general, 
and not too late - we are not pouring concrete yet!  

There is some overlap with Maciek's points,
and I've just submitted a draft to address his comments.

I will try to push another change before the 
deadline, but can't promise that ... I need some
time to think about the suspended processor scenario, too.
It seems somehow related to one of Yoshiaki-san's 
experiments and suggested methods (in slides, discussed
last year, should be in our meeting proceedings with 
discussion on the list - IIRC Yoshiaki was pausing 
ports instead of the processor). 

We have experimental evaluation of the existing methods 
in the draft, and maybe some people interested in trying new 
approaches?

regards,
Al
(as a participant)

> -----Original Message-----
> From: Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco)
> [mailto:[e-mail]]
> Sent: Thursday, July 4, 2019 1:42 PM
> To: MORTON, ALFRED C (AL) <[e-mail]>
> Cc: [e-mail]
> Subject: RE: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
> 
> Hi Al.
> 
> Sorry for being late with the review.
> I wanted to think some things through,
> and then did not find big enough chunk of time
> to write everything down.
> 
> First, narrow comments.
> 
> * Second line of 1. Introduction:
> 
> > in[RFC2544], supported by the terms and definitions in [RFC1242], and
> 
> Missing space after the first "in".
> 
> * Point 4 in chapter 3 seem to be referring to two different estimates:
> 
> > It was found that the actual buffer time in the DUT could be
> > estimated using results from the Throughput tests conducted
> > according to Section 26.1 of [RFC2544], because it appears that
> > the DUT's frame processing rate may tend to increase the estimate.
> 
> Which estimate is increased, relative to what?
> Is it "implied estimate" tending to be larger than "corrected estimate"?
> 
> * Within 5.4:
> 
> > (assuming a simple model of the DUT
> > composed of a buffer and a forwarding function)
> 
> A reasonable simplification, but could be mentioned in Motivation,
> so 5.4 does not need a sentence in parentheses.
> 
> * Still within 5.4:
> 
> > Corrected DUT Buffer Time
> 
> This is a big one.
> The correct name for this quantity could be
> "DUT Buffer Time Correction".
> 
> * still 5.4:
> 
> > and the Buffer size is more accurately estimated by excluding them.
> 
> Can be turned into a formula, where
> "Corrected DUT Buffer Time" can be used in a more natural meaning.
> 
>   Corrected DUT Buffer Time = Implied DUT Buffer Time - DUT Buffer Time
> Correction
> 
> * In 6. table:
> 
> > Min,Max,StdDev
> 
> Not sure if it is clear these refer to B2B length statistics.
> Include information that these quantities are to be expressed in frames.
> 
> 
> And now some broader comments.
> 
> * Suspended processor in production:
> 
> > useful to estimate whether frame losses will occur if
> > DUT forwarding is temporarily suspended in a production deployment
> 
> This is the main benefit, and Introduction already mentions
> "compensating for disruptions in the software packet processor".
> 
> But I think it would be good to classify different scenarios
> leading to frame loss, that could explain both two buffer times
> (implied and corrected) and how to use them outside back-to-back load.
> 
> In back-to-back test we have processor processing,
> while the buffer is being filled by maximal traffic.
> The time it takes to fill is the "implied buffer time".
> 
> In a hypothetic scenario where processor is suspended
> and buffer is being filled by maximal traffic,
> the time to fill the buffer is shorter, the "corrected buffer time".
> 
> A scenario occuring in practice has suspended processor,
> but the buffer is being filled more slowly, say at throughput rate.
> Deployers wishing to predict the time for the buffer to fill up
> can use this formula:
>   Real Buffer Time = Corrected Buffer Time * B2B Frame Rate / Real Frame
> Rate
> That is why reporting corrected (instead of implied) buffer time is
> useful.
> 
> Also, it would be nice to name the scenarios and rename the buffer times.
> For example, "running buffer time" and "suspended buffer time"
> (instead of "implied" and "corrected" respectively).
> 
> * B2B processor rate:
> 
> For the computation of the corrected buffer time to be correct,
> real processor frame processing rate (average during B2B test)
> should be used instead Measured Throughput in the 5.4 formula.
> 
> The process rate is not easy to measure directly,
> especially if the immediate rate varies over the duration of B2B traffic.
> I agree that Throughput is a reasonable approximation,
> but there may be other quantities (e.g. FRMOL [1]),
> that are either a better approximation, or at least easier to measure.
> 
> Not sure how much attention other such quantities should get in the draft,
> as Throughput has the advantage of avoiding some frame sizes.
> 
> * DUT vs SUT:
> 
> This is related to final items of 4. Prerequisities.
> 
> > Therefore, sources of packet loss
> > that are un-related to consistent evaluation of buffer size SHOULD be
> > identified and removed or mitigated.
> 
> Do we have a separate document discussing differences
> between testing DUT and SUT? We should have.
> 
> Usually I prefer testing SUT (meaning no extra mitigations),
> but in this case, for the analysis of the three aforementioned scenarios
> to work correctly, we need to make reasonably sure
> the processor is not going to get suspended during B2B test.
> 
> Also, I agree that an average result of Binary Search with Loss
> Verification
> gives a more realistic process rate estimate
> than an average result without loss verification.
> 
> Vratko.
> 
> [1] https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_rfc2285-23page-2D16&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=vrtgQUlH3DsfwGQUVH6iVjE_vTTg8PyrBnpdstCBL6g&e=
> 
> 
> -----Original Message-----
> From: bmwg <[e-mail]> On Behalf Of [e-mail]
> Sent: Thursday, 2019-July-04 17:48
> To: [e-mail]
> Cc: [e-mail]
> Subject: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
> 
> 
> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
> This draft is a work item of the Benchmarking Methodology WG of the IETF.
> 
>         Title           : Updates for the Back-to-back Frame Benchmark in
> RFC 2544
>         Author          : Al Morton
> 	Filename        : draft-ietf-bmwg-b2b-frame-00.txt
> 	Pages           : 12
> 	Date            : 2019-07-04
> 
> Abstract:
>    Fundamental Benchmarking Methodologies for Network Interconnect
>    Devices of interest to the IETF are defined in RFC 2544.  This memo
>    updates the procedures of the test to measure the Back-to-back frames
>    Benchmark of RFC 2544, based on further experience.
> 
>    This memo updates Section 26.4 of RFC 2544.
> 
> 
> 
> The IETF datatracker status page for this draft is:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_draft-2Dietf-2Dbmwg-2Db2b-
> 2Dframe_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=KGWJ5PiKSpydvqF2Zjj7nU84zzXKmdTmrJVljmaDKyY&e=
> 
> There are also htmlized versions available at:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Db2b-2Dframe-
> 2D00&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=vmpSsfloMMDHBQgWaiRbxfoOhIrOdkSjyx3xkUez7Wk&e=
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_html_draft-2Dietf-2Dbmwg-2Db2b-2Dframe-
> 2D00&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=HDt6i-tc3YgzN2BD1gWqMk3d_5AfJpoCAaiagAbTfi0&e=
> 
> 
> Please note that it may take a couple of minutes from the time of
> submission until the htmlized version and diff are available at
> tools.ietf.org.
> 
> Internet-Drafts are also available by anonymous FTP at:
> https://urldefense.proofpoint.com/v2/url?u=ftp-3A__ftp.ietf.org_internet-
> 2Ddrafts_&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=Nb-hM-c8mHzaS1JVAeHe-B7fZfodV4DaDZrSHn6Yff4&e=
> 
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwIGaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=RRx2aBMGbtbHkbyawC23-
> eDvM2V9bObmslTPERRbsZc&s=KIjOdOro8S84nxchOIFL22xaljYGQE7P2F6GyHYB0xA&e=