Re: [bmwg] [Newsletter] Re: [Newsletter] Re: Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

[Alex Samonte <[e-mail]>]

Certainly possible to use multiple values, but I'm not sure you would truly
gain that much more information.
We were not trying to increase the test load for latency (or anything else
that used a %age of previous traffic) by a factor of 3 (in this case).

But those numbers (25%, 50%, 75%) and the number of them (3) is just as
arbitrary as what we chose.   If someone wants to do a study (or can point
to one) where it would gain enough information to be useful we can consider
that for a future update.

-Alex


On Wed, Jun 3, 2020 at 1:53 PM Gábor LENCSE <[e-mail]> wrote:

> Dear Alex Samonte,
>
> Is it absolutely necessary to use only a single value? Perhaps using
> multiple latency values at a few different working points (like 25%, 50%,
> 75%) could give more information about the behavior of the system.
>
> Best regards,
>
> Gábor Lencse
> On 6/3/2020 21:53, Alex Samonte wrote:
>
> The main rationale is to see what the nominal latency is..
>
> As we all know the latency, will rise dramatically (and be unpredictable)
> when something gets close to bottlenecking.  That doesn't mean things will
> fail, but you start getting ugly behavior.
>
> from 0%-XX% the latency is fairly reliable (XX to be guessed at later).
> This means the latency doesn't change dramatically as you increase that
> percentage.  Usually it's something linear with a slope of much less than
> 1.    When you get to XX%-100%, the latency becomes unstable and it varies
> wildly as various components of the system have a bottleneck.  When that
> happens, other things compensate, and then they become a bottleneck (when
> the first thing is not) and it oscillates all over the place.  This is what
> leads to the unstable latency.  The difference of the latency at 1% vs XX%
> is small compared to the latency at XX% and some of the unstable latency at
> 100%.
>
> So the question is how to we determine XX%?   Unfortunately, all DUTs are
> different, have different limits, bottlenecks, and features that may change
> XX%.   It's not the same from vendor to vendor, nor even model to model
> within the same vendor.
>
> A couple big examples to point out.
> 1) multi core CPUs.  Because of the way that most stateful devices behave,
> a single session will probably be handled by a single core/thread of a
> CPU.  If a device has 8 cores, if there is only 1 session, 7 cores will
> likely go unused.   If there are 9 sessions, 1 core will be handling 2
> sessions, and the others will be handling 1 (50% difference).  If there are
> 8001 sessions, then it's 1001, and others 1000, and the uneveness does not
> matter.   However, if one of those sessions is 'harder' or more difficult
> to process.  Or the number of sessions on a single core may be harder to
> process, that core may reach 100% capacity before the others do.   Lets'
> say I have 100%, 50%, 50%, 50% in a 4 core case.   Here we will experience
> unstable latency.  The sessions that go through the core at 100% will
> experience additional latency, where the sessions going through the other 3
> cores will not, leading to unstable latency.
>
> 2) internal buffers.  Depending on the protocol different protocols may
> have different buffers associated with them, so when buffers fill up, more
> latency is added, but because they can be differently allocated for
> different protocols, you will see variable latency depending on which is
> exhausted and which is not.
>
> 3) interface (internal or external) limits.   When you approach 100% of a
> certain speed link (be it internal or external to the DUT) you can also
> experience additional latency.  It's hard to determine what a particular
> DUT's internal architecture is(nor do we really want to try to figure that
> out), So we generally do not want to push it to 100% because we know we
> will incur
>
>
> So what we've said is that:
> 0%-XX% the latencies are reliable / not unstable
> XX%-100% the latency are unstable
>
> People looking to benchmark are generally interested in the largest stable
> value (XX).   100% (or near it) is generally not a representative value of
> the normal case of traffic.   0% (or 1%) is also generally not
> representative of the normal case for traffic.
>
> Ideally XX% is what we're trying to get, but it's different for each
> model.   Since 0-XX% represents the stable side, i'd rather err on that
> side, than the XX%-100%.
>
> When doing the benchmark we're finding the various max values (without
> failure), but that does not necessarily mean stable latency.   In order to
> get ot that range we're going to back off from 100%.
>
> Now what could that value be?  95%  75%  50%?   This was purposefully set
> low so we would not encounter any other bottlenecks that would make the
> latency unstable.
>
> I've tested lots of different devices over the years, both my own and
> other vendors, and the lower you make that number, the higher the number of
> devices will stay in the 'stable' area of latency.
>
> so 50% was chosen as the least likely to cause unstable latency, and still
> representative of real customer usage (plenty of customers have no issue of
> using a device that is at 50% capacity).   At 75% and much more so at 95%
> customers are looking to change/upgrade their equipment so they are not
> nearing the top of what the device is capable of.   So again it makes me
> less interested in the latency there.
>
>
> Hopefully that makes sense, and while it could be shorter in the
> explanation, i'm not sure I could make it into a 2 sentence one.
>
> -Alex
>
>
> On Wed, Jun 3, 2020 at 9:12 AM MORTON, ALFRED C (AL) <[e-mail]>
> wrote:
>
>> Thanks for your question, Simon, and the history, Brian.
>>
>>
>>
>> I confess that this question (why 50%?) has occurred to me in other
>> contexts, and it may help to add a sentence to two of rationale. So if the
>> technical folks can help with suggestions, that would be great!
>>
>>
>>
>> regards,
>>
>> Al
>>
>>
>>
>> *From:* bmwg [mailto:[e-mail]] *On Behalf Of *
>> [e-mail]
>> *Sent:* Wednesday, June 3, 2020 10:56 AM
>> *To:* 'Simon Edwards' <[e-mail]>
>> *Cc:* [e-mail]
>> *Subject:* Re: [bmwg] Query about 50% values in [Benchmarking
>> Methodology for Network Security Device Performance draft 02]
>>
>>
>>
>> Simon,
>>
>>
>>
>> This requirement was agreed to and adopted by the working group within
>> NetSecOPEN. It first appeared in the IETF individual draft on October 14,
>> 2018. (
>> https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05
>> <https://urldefense..proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>).
>> It was clarified and expanded on March 5, 2019 in
>> https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>.
>> It’s form has largely been unchanged since then.
>>
>>
>>
>> I will leave it to the technical folks to expand on this more. However, I
>> am saying all of this because it has been in the position to be reviewed
>> and commented on by the BMWG community for awhile. Our assumption is that
>> given no one appears to have an issue with it that we hit the mark.
>>
>>
>>
>> Brian
>>
>>
>>
>> *From:* bmwg <[e-mail]> *On Behalf Of *Simon Edwards
>> *Sent:* June 3, 2020 5:10 AM
>> *To:* [e-mail]
>> *Subject:* [bmwg] Query about 50% values in [Benchmarking Methodology
>> for Network Security Device Performance draft 02]
>>
>>
>>
>> Hi all,
>>
>>
>>
>> In a number of sections, but specifically '7.8.3.2.  Test Equipment
>> Configuration Parameters', there are requirements to measure with 50% of
>> the maximum connections/ sec measured in the HTTP/S throughput tests
>>
>>
>>
>> E.g. "Target objective for scenarios 1 and 2: 50% of the maximum
>> connections per second measured in test scenario..."
>>
>>
>>
>> I'm sure this 50% value is the product of much thought and discussion,
>> rather than an arbitrary choice. Is anyone able to explain the reason for
>> the specific '50%' value (as opposed to 25%, 75% or whatever) or could you
>> please point to documentation around that decision made by the group?
>>
>>
>>
>> I'm asking just to understand. I don't disagree with the decision : )
>>
>>
>>
>> Very best wishes,
>>
>> Simon
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/bmwg
>>
>
>
> --
>
> Alex Samonte
> Director Of Technical Architecture
>
> [image: FortinetLogosig1550861636.png]
> <http://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=fortinet-logo>
> E: [e-mail]
> M: [phone]
> Skype: asamonteFN
> www.fortinet.com
> <https://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr-company-url&utm_campaign=sd-wan>
>            [image: Fortinetsocialtwitternew1550866064.png]
> <https://twitter.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-twitter>
>   [image: Fortinetsociallinkedinnew1550866125.png]
> <http://www.linkedin.com/company/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-linkedin>
>   [image: Fortinetsocialfbnew1550866196.png]
> <http://www.facebook.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-facebook>
>  [image: Fortinetsocialinstagramnew1550866291.png]
> <https://www.instagram.com/behindthefirewall/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-instagram>
>   [image: Fortinetsocialblognew1550866342.png]
> <https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog> [image:
> Secure Remote Access for Your Workforce at Scale]
> <http://signatures.fortinet.com/uc/5e868f6230170c1cb4a71ffb>
>
>
>
> * ------------------------------ *** Please note that this message and any
> attachments may contain confidential and proprietary material and
> information and are intended only for the use of the intended recipient(s).
> If you are not the intended recipient, you are hereby notified that any
> review, use, disclosure, dissemination, distribution or copying of this
> message and any attachments is strictly prohibited. If you have received
> this email in error, please immediately notify the sender and destroy this
> e-mail and any attachments and all copies, whether electronic or printed.
> Please also note that any views, opinions, conclusions or commitments
> expressed in this message are those of the individual sender and do not
> necessarily reflect the views of Fortinet, Inc., its affiliates, and emails
> are not binding on Fortinet and only a writing manually signed by
> Fortinet's General Counsel can be a binding commitment of Fortinet to
> Fortinet's customers or partners. Thank you. ***
> ------------------------------*
>
> _______________________________________________
> bmwg mailing [e-mail]://www.ietf.org/mailman/listinfo/bmwg
>
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg
>


-- 

Alex Samonte
Director Of Technical Architecture

[image: FortinetLogosig1550861636.png]
<http://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=fortinet-logo>
E: [e-mail]
M: [phone]
Skype: asamonteFN
www.fortinet.com
<https://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr-company-url&utm_campaign=sd-wan>
           [image: Fortinetsocialtwitternew1550866064.png]
<https://twitter.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-twitter>
  [image: Fortinetsociallinkedinnew1550866125.png]
<http://www.linkedin.com/company/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-linkedin>
  [image: Fortinetsocialfbnew1550866196.png]
<http://www.facebook.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-facebook>
 [image: Fortinetsocialinstagramnew1550866291.png]
<https://www.instagram.com/behindthefirewall/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-instagram>
  [image: Fortinetsocialblognew1550866342.png]
<https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog>[image:
Secure Remote Access for Your Workforce at Scale]
<http://signatures.fortinet.com/uc/5e868f6230170c1cb4a71ffb>



***  Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. ***