Re: [bmwg] Eric Rescorla's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
bhuvaneswaran.vengainathan@veryxtech.com Wed, 18 April 2018 08:27 UTC
Return-Path: <bhuvaneswaran.vengainathan@veryxtech.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 961E2124C27; Wed, 18 Apr 2018 01:27:20 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level:
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FekY3BjRIxFp; Wed, 18 Apr 2018 01:27:18 -0700 (PDT)
Received: from smtpout4.netcore.co.in (sm2375.nsmailserv.com [202.162.237.5]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0153D12D810; Wed, 18 Apr 2018 01:27:10 -0700 (PDT)
Received: from smtpin4.netcore.co.in (unknown [192.168.2.129]) by cf3.netcore.co.in (Postfix) with ESMTP id 47539120112; Wed, 18 Apr 2018 13:56:44 +0530 (IST)
Received: from cloudmail14.netcore.co.in (cloudmail12.netcore.co.in [202.162.231.3]) by smtpin4.netcore.co.in (Postfix) with ESMTP id E189512E70F; Wed, 18 Apr 2018 13:56:54 +0530 (IST)
Mime-Version: 1.0
X-Intloopheader: 0
Date: Wed, 18 Apr 2018 08:26:55 +0000
Content-Type: multipart/alternative; boundary="----=_Part_943_931896124.1524040015"
Message-ID: <cd573fee93e0ba3050c7b4ee0b685b9d@cloudmail14.netcore.co.in>
X-Mailer: AfterLogic webmail client
From: bhuvaneswaran.vengainathan@veryxtech.com
To: Eric Rescorla <ekr@rtfm.com>, acmorton@att.com
Cc: The IESG <iesg@ietf.org>, draft-ietf-bmwg-sdn-controller-benchmark-meth@ietf.org, bmwg-chairs@ietf.org, bmwg@ietf.org
In-Reply-To: <152399965258.11535.11874306299818806488.idtracker@ietfa.amsl.com>
References: <152399965258.11535.11874306299818806488.idtracker@ietfa.amsl.com>
X-Priority: 3 (Normal)
X-SMTP30-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: E189512E70F.A4C2F
X-SMTP30-MailScanner: Found to be clean
X-MailScanner-From: bhuvaneswaran.vengainathan@veryxtech.com
X-Cloudmilter-Processed: 1
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/lInbhN___7WIqHU4hNLMpeTs1fY>
Subject: Re: [bmwg] Eric Rescorla's No Objection on draft-ietf-bmwg-sdn-controller-benchmark-meth-08: (with COMMENT)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 18 Apr 2018 08:27:21 -0000
Hi Eric, Thank you for your feedback on this draft. Please find below my response inline with tag [Bhuvan] @Al, Thanks for taking care of Eric's last feedback. Best regards, Bhuvan On Wed, Apr 18, 2018 at 02:44 AM, Eric Rescorla wrote: Eric Rescorla has entered the following ballot position for draft-ietf-bmwg-sdn-controller-benchmark-meth-08: No Objection When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html (https://www.ietf.org/iesg/statement/discuss-criteria.html) for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-controller-benchmark-meth/ (https://datatracker.ietf.org/doc/draft-ietf-bmwg-sdn-controller-benchmark-meth/) ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- Rich version of this review at: https://mozphab-ietf.devsvcdev.mozaws.net/D3948 (https://mozphab-ietf.devsvcdev.mozaws.net/D3948) COMMENTS reported. 4.7. Test Repeatability To increase the confidence in measured result, it is recommended that each test SHOULD be repeated a minimum of 10 times. Nit: you might be happier with "RECOMMENDED that each test be repeated ...." Also, where does 10 come from? Generally, the number of trials you need depends on the variance of each trial. [Bhuvan] The RECOMMENDED number 10 was arrived based on our experience during the benchmarking. I will discuss with other authors for changing SHOULD to RECOMMENDED. Test Reporting Each test has a reporting format that contains some global and identical reporting components, and some individual components that are specific to individual tests. The following test configuration parameters and controller settings parameters MUST be reflected in This is an odd MUST, as it's not required for interop.[Bhuvan] The intent of specifying MUST is to capture relevant test parameters to enable Apple to Apple comparison of test test results across two testers/test runs. 5. Stop the trial when the discovered topology information matches the deployed network topology, or when the discovered topology information return the same details for 3 consecutive queries. 6. Record the time last discovery message (Tmn) sent to controller from the forwarding plane test emulator interface (I1) when the trial completed successfully. (e.g., the topology matches). How large is the TD usually? How much does 3 seconds compare to that? [Bhuvan] The test duration varies depends on the size of the test topology. For a smaller topology (3 - 10) the TD was within a minute. So we kept the query interval of 3 seconds to accomodate smaller and larger topologies. Total Trials SUM[SQUAREOF(Tri-TDm)] Topology Discovery Time Variance (TDv) ---------------------- Total Trials -1 You probably don't need to specify individual formulas for mean and variance. However, you probably do want to explain why you are using the n-1 sample variance formula. [Bhuvan] We have added both formulas based on the feedback received in the mailing list. We are using n-1, as it is commonly used variance measure. Do we need an explanation here or providing any reference to this is sufficient? Measurement: (R1-T1) + (R2-T2)..(Rn-Tn) Asynchronous Message Processing Time Tr1 = ----------------------- Nrx Incidentally, this formula is the same as sum_i{R_i} - sum_i{T_i} [Bhuvan] Good suggestion, we will incorporate in the next revision. messages transmitted to the controller. If this test is repeated with varying number of nodes with same topology, the results SHOULD be reported in the form of a graph. The X coordinate SHOULD be the Number of nodes (N), the Y coordinate SHOULD be the average Asynchronous Message Processing Time. This is an odd metric because an implementation which handled overload by dropping every other message would look better than one which handled overload by queuing. [Bhuvan] Believe Al has clarified this feedback. As suggested by Al, we will update the BCP language in the reporting section. _______________________________________________ bmwg mailing list bmwg@ietf.org (mailto:bmwg@ietf.org) https://www.ietf.org/mailman/listinfo/bmwg (https://www.ietf.org/mailman/listinfo/bmwg) DISCLAIMER: Privileged and/or Confidential information may be contained in this message. If you are not the addressee of this message, you may not copy, use or deliver this message to anyone. In such event,you should destroy the message and kindly notify the sender by reply e-mail. It is understood that opinions or conclusions that do not relate to the official business of the company are neither given nor endorsed by the company.
- [bmwg] Eric Rescorla's No Objection on draft-ietf… Eric Rescorla
- Re: [bmwg] Eric Rescorla's No Objection on draft-… MORTON, ALFRED C (AL)
- Re: [bmwg] Eric Rescorla's No Objection on draft-… bhuvaneswaran.vengainathan
- Re: [bmwg] Eric Rescorla's No Objection on draft-… Eric Rescorla