Re: [bmwg] [Newsletter] Re: Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

[Gábor LENCSE <[e-mail]>]

Dear Alex Samonte,

Is it absolutely necessary to use only a single value? Perhaps using 
multiple latency values at a few different working points (like 25%, 
50%, 75%) could give more information about the behavior of the system.

Best regards,

Gábor Lencse

On 6/3/2020 21:53, Alex Samonte wrote:
> The main rationale is to see what the nominal latency is..
>
> As we all know the latency, will rise dramatically (and be 
> unpredictable) when something gets close to bottlenecking. That 
> doesn't mean things will fail, but you start getting ugly behavior.
>
> from 0%-XX% the latency is fairly reliable (XX to be guessed at 
> later).  This means the latency doesn't change dramatically as you 
> increase that percentage.  Usually it's something linear with a slope 
> of much less than 1.    When you get to XX%-100%, the latency becomes 
> unstable and it varies wildly as various components of the system have 
> a bottleneck. When that happens, other things compensate, and then 
> they become a bottleneck (when the first thing is not) and it 
> oscillates all over the place.  This is what leads to the unstable 
> latency.  The difference of the latency at 1% vs XX% is small compared 
> to the latency at XX% and some of the unstable latency at 100%.
>
> So the question is how to we determine XX%?  Unfortunately, all DUTs 
> are different, have different limits, bottlenecks, and features that 
> may change XX%.   It's not the same from vendor to vendor, nor even 
> model to model within the same vendor.
>
> A couple big examples to point out.
> 1) multi core CPUs.  Because of the way that most stateful devices 
> behave, a single session will probably be handled by a single 
> core/thread of a CPU.  If a device has 8 cores, if there is only 1 
> session, 7 cores will likely go unused.   If there are 9 sessions, 1 
> core will be handling 2 sessions, and the others will be handling 1 
> (50% difference).  If there are 8001 sessions, then it's 1001, and 
> others 1000, and the uneveness does not matter.   However, if one of 
> those sessions is 'harder' or more difficult to process.  Or the 
> number of sessions on a single core may be harder to process, that 
> core may reach 100% capacity before the others do.   Lets' say I have 
> 100%, 50%, 50%, 50% in a 4 core case.   Here we will experience 
> unstable latency.  The sessions that go through the core at 100% will 
> experience additional latency, where the sessions going through the 
> other 3 cores will not, leading to unstable latency.
>
> 2) internal buffers.  Depending on the protocol different protocols 
> may have different buffers associated with them, so when buffers fill 
> up, more latency is added, but because they can be differently 
> allocated for different protocols, you will see variable latency 
> depending on which is exhausted and which is not.
>
> 3) interface (internal or external) limits.   When you approach 100% 
> of a certain speed link (be it internal or external to the DUT) you 
> can also experience additional latency.  It's hard to determine what a 
> particular DUT's internal architecture is(nor do we really want to try 
> to figure that out), So we generally do not want to push it to 100% 
> because we know we will incur
>
>
> So what we've said is that:
> 0%-XX% the latencies are reliable / not unstable
> XX%-100% the latency are unstable
>
> People looking to benchmark are generally interested in the largest 
> stable value (XX).   100% (or near it) is generally not a 
> representative value of the normal case of traffic.   0% (or 1%) is 
> also generally not representative of the normal case for traffic.
>
> Ideally XX% is what we're trying to get, but it's different for each 
> model.   Since 0-XX% represents the stable side, i'd rather err on 
> that side, than the XX%-100%.
>
> When doing the benchmark we're finding the various max values (without 
> failure), but that does not necessarily mean stable latency.   In 
> order to get ot that range we're going to back off from 100%.
>
> Now what could that value be?  95%  75%  50%?   This was purposefully 
> set low so we would not encounter any other bottlenecks that would 
> make the latency unstable.
>
> I've tested lots of different devices over the years, both my own and 
> other vendors, and the lower you make that number, the higher the 
> number of devices will stay in the 'stable' area of latency.
>
> so 50% was chosen as the least likely to cause unstable latency, and 
> still representative of real customer usage (plenty of customers have 
> no issue of using a device that is at 50% capacity).   At 75% and much 
> more so at 95% customers are looking to change/upgrade their equipment 
> so they are not nearing the top of what the device is capable of.   So 
> again it makes me less interested in the latency there.
>
>
> Hopefully that makes sense, and while it could be shorter in the 
> explanation, i'm not sure I could make it into a 2 sentence one.
>
> -Alex
>
>
> On Wed, Jun 3, 2020 at 9:12 AM MORTON, ALFRED C (AL) 
> <[e-mail] <mailto:[e-mail]>> wrote:
>
>     Thanks for your question, Simon, and the history, Brian.
>
>     I confess that this question (why 50%?) has occurred to me in
>     other contexts, and it may help to add a sentence to two of
>     rationale. So if the technical folks can help with suggestions,
>     that would be great!
>
>     regards,
>
>     Al
>
>     *From:* bmwg [mailto:[e-mail]
>     <mailto:[e-mail]>] *On Behalf Of
>     *[e-mail] <mailto:[e-mail]>
>     *Sent:* Wednesday, June 3, 2020 10:56 AM
>     *To:* 'Simon Edwards' <[e-mail] <mailto:[e-mail]>>
>     *Cc:* [e-mail] <mailto:[e-mail]>
>     *Subject:* Re: [bmwg] Query about 50% values in [Benchmarking
>     Methodology for Network Security Device Performance draft 02]
>
>     Simon,
>
>     This requirement was agreed to and adopted by the working group
>     within NetSecOPEN. It first appeared in the IETF individual draft
>     on October 14, 2018.
>     (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05
>     <https://urldefense..proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>).
>     It was clarified and expanded on March 5, 2019 in
>     https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>.
>     It’s form has largely been unchanged since then.
>
>     I will leave it to the technical folks to expand on this more.
>     However, I am saying all of this because it has been in the
>     position to be reviewed and commented on by the BMWG community for
>     awhile. Our assumption is that given no one appears to have an
>     issue with it that we hit the mark.
>
>     Brian
>
>     *From:*bmwg <[e-mail] <mailto:[e-mail]>>
>     *On Behalf Of *Simon Edwards
>     *Sent:* June 3, 2020 5:10 AM
>     *To:* [e-mail] <mailto:[e-mail]>
>     *Subject:* [bmwg] Query about 50% values in [Benchmarking
>     Methodology for Network Security Device Performance draft 02]
>
>     Hi all,
>
>     In a number of sections, but specifically '7.8.3.2.  Test
>     Equipment Configuration Parameters', there are requirements to
>     measure with 50% of the maximum connections/ sec measured in the
>     HTTP/S throughput tests
>
>     E.g. "Target objective for scenarios 1 and 2: 50% of the maximum
>     connections per second measured in test scenario..."
>
>     I'm sure this 50% value is the product of much thought and
>     discussion, rather than an arbitrary choice. Is anyone able to
>     explain the reason for the specific '50%' value (as opposed to
>     25%, 75% or whatever) or could you please point to documentation
>     around that decision made by the group?
>
>     I'm asking just to understand. I don't disagree with the decision : )
>
>     Very best wishes,
>
>     Simon
>
>     _______________________________________________
>     bmwg mailing list
>     [e-mail] <mailto:[e-mail]>
>     https://www.ietf.org/mailman/listinfo/bmwg
>
>
>
> -- 
>
> Alex Samonte
> Director Of Technical Architecture
>
> FortinetLogosig1550861636.png 
> <http://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=fortinet-logo> 
>
> E: [e-mail] <mailto:[e-mail]>
> M: [phone]
> Skype: asamonteFN
> www.fortinet.com 
> <https://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr-company-url&utm_campaign=sd-wan> 
> Fortinetsocialtwitternew1550866064.png 
> <https://twitter.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-twitter> 
> Fortinetsociallinkedinnew1550866125.png 
> <http://www.linkedin.com/company/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-linkedin> 
> Fortinetsocialfbnew1550866196.png 
> <http://www.facebook.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-facebook> 
> Fortinetsocialinstagramnew1550866291.png 
> <https://www.instagram.com/behindthefirewall/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-instagram> 
> Fortinetsocialblognew1550866342.png 
> <https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog> 
>
>
> Secure Remote Access for Your Workforce at Scale 
> <http://signatures.fortinet.com/uc/5e868f6230170c1cb4a71ffb>
>
> *
> ------------------------------------------------------------------------
> *** Please note that this message and any attachments may contain 
> confidential and proprietary material and information and are intended 
> only for the use of the intended recipient(s). If you are not the 
> intended recipient, you are hereby notified that any review, use, 
> disclosure, dissemination, distribution or copying of this message and 
> any attachments is strictly prohibited. If you have received this 
> email in error, please immediately notify the sender and destroy this 
> e-mail and any attachments and all copies, whether electronic or 
> printed. Please also note that any views, opinions, conclusions or 
> commitments expressed in this message are those of the individual 
> sender and do not necessarily reflect the views of Fortinet, Inc., its 
> affiliates, and emails are not binding on Fortinet and only a writing 
> manually signed by Fortinet's General Counsel can be a binding 
> commitment of Fortinet to Fortinet's customers or partners. Thank you. 
> ***
> ------------------------------------------------------------------------
> *
>
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg