Re: [bmwg] [Newsletter] Re: Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Gábor LENCSE <lencse@hit.bme.hu> Wed, 03 June 2020 20:54 UTC

Return-Path: <lencse@hit.bme.hu>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6F2EF3A0F95 for <bmwg@ietfa.amsl.com>; Wed, 3 Jun 2020 13:54:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.889
X-Spam-Level:
X-Spam-Status: No, score=-1.889 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JF5KxtdO7A9S for <bmwg@ietfa.amsl.com>; Wed, 3 Jun 2020 13:54:18 -0700 (PDT)
Received: from frogstar.hit.bme.hu (frogstar.hit.bme.hu [IPv6:2001:738:2001:4020::2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 94E5C3A0F96 for <bmwg@ietf.org>; Wed, 3 Jun 2020 13:54:17 -0700 (PDT)
Received: from [192.168.1.101] (host-79-121-42-113.kabelnet.hu [79.121.42.113]) (authenticated bits=0) by frogstar.hit.bme.hu (8.15.2/8.15.2) with ESMTPSA id 053Ks2HD050230 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for <bmwg@ietf.org>; Wed, 3 Jun 2020 22:54:07 +0200 (CEST) (envelope-from lencse@hit.bme.hu)
X-Authentication-Warning: frogstar.hit.bme.hu: Host host-79-121-42-113.kabelnet.hu [79.121.42.113] claimed to be [192.168.1.101]
To: bmwg@ietf.org
References: <DBBPR09MB30618401FAD3184921486E79B6880@DBBPR09MB3061.eurprd09.prod.outlook.com> <027801d639b7$0d4ac290$27e047b0$@netsecopen.org> <4D7F4AD313D3FC43A053B309F97543CF0108A5F74A@njmtexg5.research.att.com> <CAGiD7R7UeKa79pe-Y0BEXCvrDhiAnm329scxWkfu9GZ=7Jh5cA@mail.gmail.com>
From: =?UTF-8?Q?G=c3=a1bor_LENCSE?= <lencse@hit.bme.hu>
Message-ID: <dd3ac998-feba-7a4d-838e-8c1c79e28971@hit.bme.hu>
Date: Wed, 3 Jun 2020 22:53:56 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.8.1
MIME-Version: 1.0
In-Reply-To: <CAGiD7R7UeKa79pe-Y0BEXCvrDhiAnm329scxWkfu9GZ=7Jh5cA@mail.gmail.com>
Content-Type: multipart/alternative; boundary="------------8B546F37EC9F21EC48331992"
Content-Language: en-US
X-Virus-Scanned: clamav-milter 0.101.2 at frogstar.hit.bme.hu
X-Virus-Status: Clean
Received-SPF: pass (frogstar.hit.bme.hu: authenticated connection) receiver=frogstar.hit.bme.hu; client-ip=79.121.42.113; helo=[192.168.1.101]; envelope-from=lencse@hit.bme.hu; x-software=spfmilter 2.001 http://www.acme.com/software/spfmilter/ with libspf2-1.2.10;
X-DCC--Metrics: frogstar.hit.bme.hu; whitelist
X-Scanned-By: MIMEDefang 2.79 on 152.66.248.44
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/FC2VOpS-4HqDUBzZAYtkmJUHsPU>
Subject: Re: [bmwg] [Newsletter] Re: Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 03 Jun 2020 20:54:22 -0000

Dear Alex Samonte,

Is it absolutely necessary to use only a single value? Perhaps using 
multiple latency values at a few different working points (like 25%, 
50%, 75%) could give more information about the behavior of the system.

Best regards,

Gábor Lencse

On 6/3/2020 21:53, Alex Samonte wrote:
> The main rationale is to see what the nominal latency is..
>
> As we all know the latency, will rise dramatically (and be 
> unpredictable) when something gets close to bottlenecking. That 
> doesn't mean things will fail, but you start getting ugly behavior.
>
> from 0%-XX% the latency is fairly reliable (XX to be guessed at 
> later).  This means the latency doesn't change dramatically as you 
> increase that percentage.  Usually it's something linear with a slope 
> of much less than 1.    When you get to XX%-100%, the latency becomes 
> unstable and it varies wildly as various components of the system have 
> a bottleneck. When that happens, other things compensate, and then 
> they become a bottleneck (when the first thing is not) and it 
> oscillates all over the place.  This is what leads to the unstable 
> latency.  The difference of the latency at 1% vs XX% is small compared 
> to the latency at XX% and some of the unstable latency at 100%.
>
> So the question is how to we determine XX%?  Unfortunately, all DUTs 
> are different, have different limits, bottlenecks, and features that 
> may change XX%.   It's not the same from vendor to vendor, nor even 
> model to model within the same vendor.
>
> A couple big examples to point out.
> 1) multi core CPUs.  Because of the way that most stateful devices 
> behave, a single session will probably be handled by a single 
> core/thread of a CPU.  If a device has 8 cores, if there is only 1 
> session, 7 cores will likely go unused.   If there are 9 sessions, 1 
> core will be handling 2 sessions, and the others will be handling 1 
> (50% difference).  If there are 8001 sessions, then it's 1001, and 
> others 1000, and the uneveness does not matter.   However, if one of 
> those sessions is 'harder' or more difficult to process.  Or the 
> number of sessions on a single core may be harder to process, that 
> core may reach 100% capacity before the others do.   Lets' say I have 
> 100%, 50%, 50%, 50% in a 4 core case.   Here we will experience 
> unstable latency.  The sessions that go through the core at 100% will 
> experience additional latency, where the sessions going through the 
> other 3 cores will not, leading to unstable latency.
>
> 2) internal buffers.  Depending on the protocol different protocols 
> may have different buffers associated with them, so when buffers fill 
> up, more latency is added, but because they can be differently 
> allocated for different protocols, you will see variable latency 
> depending on which is exhausted and which is not.
>
> 3) interface (internal or external) limits.   When you approach 100% 
> of a certain speed link (be it internal or external to the DUT) you 
> can also experience additional latency.  It's hard to determine what a 
> particular DUT's internal architecture is(nor do we really want to try 
> to figure that out), So we generally do not want to push it to 100% 
> because we know we will incur
>
>
> So what we've said is that:
> 0%-XX% the latencies are reliable / not unstable
> XX%-100% the latency are unstable
>
> People looking to benchmark are generally interested in the largest 
> stable value (XX).   100% (or near it) is generally not a 
> representative value of the normal case of traffic.   0% (or 1%) is 
> also generally not representative of the normal case for traffic.
>
> Ideally XX% is what we're trying to get, but it's different for each 
> model.   Since 0-XX% represents the stable side, i'd rather err on 
> that side, than the XX%-100%.
>
> When doing the benchmark we're finding the various max values (without 
> failure), but that does not necessarily mean stable latency.   In 
> order to get ot that range we're going to back off from 100%.
>
> Now what could that value be?  95%  75%  50%?   This was purposefully 
> set low so we would not encounter any other bottlenecks that would 
> make the latency unstable.
>
> I've tested lots of different devices over the years, both my own and 
> other vendors, and the lower you make that number, the higher the 
> number of devices will stay in the 'stable' area of latency.
>
> so 50% was chosen as the least likely to cause unstable latency, and 
> still representative of real customer usage (plenty of customers have 
> no issue of using a device that is at 50% capacity).   At 75% and much 
> more so at 95% customers are looking to change/upgrade their equipment 
> so they are not nearing the top of what the device is capable of.   So 
> again it makes me less interested in the latency there.
>
>
> Hopefully that makes sense, and while it could be shorter in the 
> explanation, i'm not sure I could make it into a 2 sentence one.
>
> -Alex
>
>
> On Wed, Jun 3, 2020 at 9:12 AM MORTON, ALFRED C (AL) 
> <acm@research.att.com <mailto:acm@research.att.com>> wrote:
>
>     Thanks for your question, Simon, and the history, Brian.
>
>     I confess that this question (why 50%?) has occurred to me in
>     other contexts, and it may help to add a sentence to two of
>     rationale. So if the technical folks can help with suggestions,
>     that would be great!
>
>     regards,
>
>     Al
>
>     *From:* bmwg [mailto:bmwg-bounces@ietf.org
>     <mailto:bmwg-bounces@ietf.org>] *On Behalf Of
>     *bmonkman@netsecopen.org <mailto:bmonkman@netsecopen.org>
>     *Sent:* Wednesday, June 3, 2020 10:56 AM
>     *To:* 'Simon Edwards' <simon@selabs.uk <mailto:simon@selabs.uk>>
>     *Cc:* bmwg@ietf.org <mailto:bmwg@ietf.org>
>     *Subject:* Re: [bmwg] Query about 50% values in [Benchmarking
>     Methodology for Network Security Device Performance draft 02]
>
>     Simon,
>
>     This requirement was agreed to and adopted by the working group
>     within NetSecOPEN. It first appeared in the IETF individual draft
>     on October 14, 2018.
>     (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05
>     <https://urldefense..proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>).
>     It was clarified and expanded on March 5, 2019 in
>     https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>.
>     It’s form has largely been unchanged since then.
>
>     I will leave it to the technical folks to expand on this more.
>     However, I am saying all of this because it has been in the
>     position to be reviewed and commented on by the BMWG community for
>     awhile. Our assumption is that given no one appears to have an
>     issue with it that we hit the mark.
>
>     Brian
>
>     *From:*bmwg <bmwg-bounces@ietf.org <mailto:bmwg-bounces@ietf.org>>
>     *On Behalf Of *Simon Edwards
>     *Sent:* June 3, 2020 5:10 AM
>     *To:* bmwg@ietf.org <mailto:bmwg@ietf.org>
>     *Subject:* [bmwg] Query about 50% values in [Benchmarking
>     Methodology for Network Security Device Performance draft 02]
>
>     Hi all,
>
>     In a number of sections, but specifically '7.8.3.2.  Test
>     Equipment Configuration Parameters', there are requirements to
>     measure with 50% of the maximum connections/ sec measured in the
>     HTTP/S throughput tests
>
>     E.g. "Target objective for scenarios 1 and 2: 50% of the maximum
>     connections per second measured in test scenario..."
>
>     I'm sure this 50% value is the product of much thought and
>     discussion, rather than an arbitrary choice. Is anyone able to
>     explain the reason for the specific '50%' value (as opposed to
>     25%, 75% or whatever) or could you please point to documentation
>     around that decision made by the group?
>
>     I'm asking just to understand. I don't disagree with the decision : )
>
>     Very best wishes,
>
>     Simon
>
>     _______________________________________________
>     bmwg mailing list
>     bmwg@ietf.org <mailto:bmwg@ietf.org>
>     https://www.ietf.org/mailman/listinfo/bmwg
>
>
>
> -- 
>
> Alex Samonte
> Director Of Technical Architecture
>
> FortinetLogosig1550861636.png 
> <http://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=fortinet-logo> 
>
> E: asamonte@fortinet.com <mailto:asamonte@fortinet.com>
> M: +1 408-475-8737
> Skype: asamonteFN
> www.fortinet.com 
> <https://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr-company-url&utm_campaign=sd-wan> 
> Fortinetsocialtwitternew1550866064.png 
> <https://twitter.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-twitter> 
> Fortinetsociallinkedinnew1550866125.png 
> <http://www.linkedin.com/company/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-linkedin> 
> Fortinetsocialfbnew1550866196.png 
> <http://www.facebook.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-facebook> 
> Fortinetsocialinstagramnew1550866291.png 
> <https://www.instagram.com/behindthefirewall/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-instagram> 
> Fortinetsocialblognew1550866342.png 
> <https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog> 
>
>
> Secure Remote Access for Your Workforce at Scale 
> <http://signatures.fortinet.com/uc/5e868f6230170c1cb4a71ffb>
>
> *
> ------------------------------------------------------------------------
> *** Please note that this message and any attachments may contain 
> confidential and proprietary material and information and are intended 
> only for the use of the intended recipient(s). If you are not the 
> intended recipient, you are hereby notified that any review, use, 
> disclosure, dissemination, distribution or copying of this message and 
> any attachments is strictly prohibited. If you have received this 
> email in error, please immediately notify the sender and destroy this 
> e-mail and any attachments and all copies, whether electronic or 
> printed. Please also note that any views, opinions, conclusions or 
> commitments expressed in this message are those of the individual 
> sender and do not necessarily reflect the views of Fortinet, Inc., its 
> affiliates, and emails are not binding on Fortinet and only a writing 
> manually signed by Fortinet's General Counsel can be a binding 
> commitment of Fortinet to Fortinet's customers or partners. Thank you. 
> ***
> ------------------------------------------------------------------------
> *
>
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://www.ietf.org/mailman/listinfo/bmwg