Re: [bmwg] [Newsletter] Re: Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

["Van Den Breekel, Jurrie" <[e-mail]>]

Hi Gábor,

That is a good suggestion, the challenge however is that the latency tests are already 12 out of the total of 35 tests in the spec. The 12 tests for latency are Throughput and CPS  for both HTTP and HTTPS with 3 object sizes each. Relative to the other tests its already on the high side.

Best regards,
Jurrie

From: bmwg <[e-mail]> on behalf of Gábor LENCSE <[e-mail]>
Date: Wednesday, June 3, 2020 at 1:54 PM
To: "[e-mail]" <[e-mail]>
Subject: Re: [bmwg] [Newsletter] Re: Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

[Warning] This email comes from an external source. Be careful of any embedded links and attachments.

Dear Alex Samonte,

Is it absolutely necessary to use only a single value? Perhaps using multiple latency values at a few different working points (like 25%, 50%, 75%) could give more information about the behavior of the system.

Best regards,

Gábor Lencse
On 6/3/2020 21:53, Alex Samonte wrote:
The main rationale is to see what the nominal latency is..

As we all know the latency, will rise dramatically (and be unpredictable) when something gets close to bottlenecking.  That doesn't mean things will fail, but you start getting ugly behavior.

from 0%-XX% the latency is fairly reliable (XX to be guessed at later).  This means the latency doesn't change dramatically as you increase that percentage.  Usually it's something linear with a slope of much less than 1.    When you get to XX%-100%, the latency becomes unstable and it varies wildly as various components of the system have a bottleneck.  When that happens, other things compensate, and then they become a bottleneck (when the first thing is not) and it oscillates all over the place.  This is what leads to the unstable latency.  The difference of the latency at 1% vs XX% is small compared to the latency at XX% and some of the unstable latency at 100%.

So the question is how to we determine XX%?   Unfortunately, all DUTs are different, have different limits, bottlenecks, and features that may change XX%.   It's not the same from vendor to vendor, nor even model to model within the same vendor.

A couple big examples to point out.
1) multi core CPUs.  Because of the way that most stateful devices behave, a single session will probably be handled by a single core/thread of a CPU.  If a device has 8 cores, if there is only 1 session, 7 cores will likely go unused.   If there are 9 sessions, 1 core will be handling 2 sessions, and the others will be handling 1 (50% difference).  If there are 8001 sessions, then it's 1001, and others 1000, and the uneveness does not matter.   However, if one of those sessions is 'harder' or more difficult to process.  Or the number of sessions on a single core may be harder to process, that core may reach 100% capacity before the others do.   Lets' say I have 100%, 50%, 50%, 50% in a 4 core case.   Here we will experience unstable latency.  The sessions that go through the core at 100% will experience additional latency, where the sessions going through the other 3 cores will not, leading to unstable latency.

2) internal buffers.  Depending on the protocol different protocols may have different buffers associated with them, so when buffers fill up, more latency is added, but because they can be differently allocated for different protocols, you will see variable latency depending on which is exhausted and which is not.

3) interface (internal or external) limits.   When you approach 100% of a certain speed link (be it internal or external to the DUT) you can also experience additional latency.  It's hard to determine what a particular DUT's internal architecture is(nor do we really want to try to figure that out), So we generally do not want to push it to 100% because we know we will incur


So what we've said is that:
0%-XX% the latencies are reliable / not unstable
XX%-100% the latency are unstable

People looking to benchmark are generally interested in the largest stable value (XX).   100% (or near it) is generally not a representative value of the normal case of traffic.   0% (or 1%) is also generally not representative of the normal case for traffic.

Ideally XX% is what we're trying to get, but it's different for each model.   Since 0-XX% represents the stable side, i'd rather err on that side, than the XX%-100%.

When doing the benchmark we're finding the various max values (without failure), but that does not necessarily mean stable latency.   In order to get ot that range we're going to back off from 100%.

Now what could that value be?  95%  75%  50%?   This was purposefully set low so we would not encounter any other bottlenecks that would make the latency unstable.

I've tested lots of different devices over the years, both my own and other vendors, and the lower you make that number, the higher the number of devices will stay in the 'stable' area of latency.

so 50% was chosen as the least likely to cause unstable latency, and still representative of real customer usage (plenty of customers have no issue of using a device that is at 50% capacity).   At 75% and much more so at 95% customers are looking to change/upgrade their equipment so they are not nearing the top of what the device is capable of.   So again it makes me less interested in the latency there.


Hopefully that makes sense, and while it could be shorter in the explanation, i'm not sure I could make it into a 2 sentence one.

-Alex


On Wed, Jun 3, 2020 at 9:12 AM MORTON, ALFRED C (AL) <[e-mail]<mailto:[e-mail]>> wrote:
Thanks for your question, Simon, and the history, Brian.

I confess that this question (why 50%?) has occurred to me in other contexts, and it may help to add a sentence to two of rationale. So if the technical folks can help with suggestions, that would be great!

regards,
Al

From: bmwg [mailto:[e-mail]<mailto:[e-mail]>] On Behalf Of [e-mail]<mailto:[e-mail]>
Sent: Wednesday, June 3, 2020 10:56 AM
To: 'Simon Edwards' <[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]>
Subject: Re: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Simon,

This requirement was agreed to and adopted by the working group within NetSecOPEN. It first appeared in the IETF individual draft on October 14, 2018. (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05<https://urldefense..proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>). It was clarified and expanded on March 5, 2019 in https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>. It’s form has largely been unchanged since then.

I will leave it to the technical folks to expand on this more. However, I am saying all of this because it has been in the position to be reviewed and commented on by the BMWG community for awhile. Our assumption is that given no one appears to have an issue with it that we hit the mark.

Brian

From: bmwg <[e-mail]<mailto:[e-mail]>> On Behalf Of Simon Edwards
Sent: June 3, 2020 5:10 AM
To: [e-mail]<mailto:[e-mail]>
Subject: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Hi all,

In a number of sections, but specifically '7.8.3.2.  Test Equipment Configuration Parameters', there are requirements to measure with 50% of the maximum connections/ sec measured in the HTTP/S throughput tests

E.g. "Target objective for scenarios 1 and 2: 50% of the maximum connections per second measured in test scenario..."

I'm sure this 50% value is the product of much thought and discussion, rather than an arbitrary choice. Is anyone able to explain the reason for the specific '50%' value (as opposed to 25%, 75% or whatever) or could you please point to documentation around that decision made by the group?

I'm asking just to understand. I don't disagree with the decision : )

Very best wishes,
Simon
_______________________________________________
bmwg mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/bmwg


--

Alex Samonte
Director Of Technical Architecture

[cid:~WRD2717.jpg]<http://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=fortinet-logo>
E: [e-mail]<mailto:[e-mail]>
M: [phone]
Skype: asamonteFN
www.fortinet.com<https://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr-company-url&utm_campaign=sd-wan>             [cid:~WRD2717.jpg] <https://twitter.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-twitter>     [cid:~WRD2717.jpg] <http://www.linkedin.com/company/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-linkedin>     [cid:~WRD2717.jpg] <http://www.facebook.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-facebook>    [cid:~WRD2717.jpg] <https://www.instagram.com/behindthefirewall/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-instagram>     [cid:~WRD2717.jpg] <https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog>
[Image removed by sender. Secure Remote Access for Your Workforce at                     Scale]<http://signatures.fortinet.com/uc/5e868f6230170c1cb4a71ffb>


________________________________
*** Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. ***
________________________________



_______________________________________________

bmwg mailing list

[e-mail]<mailto:[e-mail]>

https://www.ietf.org/mailman/listinfo/bmwg